Quick Dockerfile for Python Poetry Projects

      β˜• 4 min read
🏷️
  • #Docker
  • #Python
  • #Poetry
  • Goal

    • Both Poetry and Python work out-of-the-box, dependency-wise especially. No need for entrypoint.sh and stuff.
    • Two images: dev and production.

    Dockerfile

    Say we have the following project structure:

    .
    β”œβ”€β”€ Dockerfile
    β”œβ”€β”€ pyproject.toml
    β”œβ”€β”€ poetry.lock
    β”œβ”€β”€ scripts
    └── my_awesome_ai_project
    

    Copy the following content as your Dockerfile’s start point. The usage is in the next section if in need.

    Also, a Github Gist is available.

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
     47
     48
     49
     50
     51
     52
     53
     54
     55
     56
     57
     58
     59
     60
     61
     62
     63
     64
     65
     66
     67
     68
     69
     70
     71
     72
     73
     74
     75
     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    
    # syntax=docker/dockerfile:1
    # Keep this syntax directive! It's used to enable Docker BuildKit
    
    ################################
    # PYTHON-BASE
    # Sets up all our shared environment variables
    ################################
    FROM python:3.9.17-slim as python-base
    
        # Python
    ENV PYTHONUNBUFFERED=1 \
        # pip
        PIP_DISABLE_PIP_VERSION_CHECK=on \
        PIP_DEFAULT_TIMEOUT=100 \
        \
        # Poetry
        # https://python-poetry.org/docs/configuration/#using-environment-variables
        POETRY_VERSION=1.6.1 \
        # make poetry install to this location
        POETRY_HOME="/opt/poetry" \
        # do not ask any interactive question
        POETRY_NO_INTERACTION=1 \
        # never create virtual environment automaticly, only use env prepared by us
        POETRY_VIRTUALENVS_CREATE=false \
        \
        # this is where our requirements + virtual environment will live
        VIRTUAL_ENV="/venv" \
        \
        # Node.js major version. Remove if you don't need.
        NODE_MAJOR=18
    
    # prepend poetry and venv to path
    ENV PATH="$POETRY_HOME/bin:$VIRTUAL_ENV/bin:$PATH"
    
    # prepare virtual env
    RUN python -m venv $VIRTUAL_ENV
    
    # working directory and Python path
    WORKDIR /app
    ENV PYTHONPATH="/app:$PYTHONPATH"
    
    # pretrained models cache path. Remove if you don't need.
    # ref: https://huggingface.co/docs/transformers/installation?highlight=transformers_cache#caching-models
    ENV TRANSFORMERS_CACHE="/opt/transformers_cache/"
    
    ################################
    # BUILDER-BASE
    # Used to build deps + create our virtual environment
    ################################
    FROM python-base as builder-base
    RUN apt-get update && \
        apt-get install -y \
        apt-transport-https \
        gnupg \
        ca-certificates \
        build-essential \
        git \
        nano \
        curl
    
    # install Node.js. Remove if you don't need.
    RUN mkdir -p /etc/apt/keyrings && \
        curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
        echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list && \
        apt-get update && apt-get install -y nodejs
    
    # install poetry - respects $POETRY_VERSION & $POETRY_HOME
    # The --mount will mount the buildx cache directory to where
    # Poetry and Pip store their cache so that they can re-use it
    RUN --mount=type=cache,target=/root/.cache \
        curl -sSL https://install.python-poetry.org | python -
    
    # used to init dependencies
    WORKDIR /app
    COPY poetry.lock pyproject.toml ./
    COPY scripts scripts/
    COPY my_awesome_ai_project/ my_awesome_ai_project/
    
    # install runtime deps to VIRTUAL_ENV
    RUN --mount=type=cache,target=/root/.cache \
        poetry install --no-root --only main
    
    # populate Huggingface model cache. Remove if you don't need.
    RUN poetry run python scripts/bootstrap.py
    
    # build C dependencies. Remove if you don't need.
    RUN --mount=type=cache,target=/app/scripts/vendor \
        poetry run python scripts/build-c-denpendencies.py && \
        cp scripts/lib/*.so /usr/lib
    
    ################################
    # DEVELOPMENT
    # Image used during development / testing
    ################################
    FROM builder-base as development
    
    WORKDIR /app
    
    # quicker install as runtime deps are already installed
    RUN --mount=type=cache,target=/root/.cache \
        poetry install --no-root --with test,lint
    
    EXPOSE 8080
    CMD ["bash"]
    
    
    ################################
    # PRODUCTION
    # Final image used for runtime
    ################################
    FROM python-base as production
    
    RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
        apt-get install -y --no-install-recommends \
        ca-certificates && \
        apt-get clean
    
    # copy in our built poetry + venv
    COPY --from=builder-base $POETRY_HOME $POETRY_HOME
    COPY --from=builder-base $VIRTUAL_ENV $VIRTUAL_ENV
    # copy in our C dependencies. Remove if you don't need.
    COPY --from=builder-base /app/scripts/lib/*.so /usr/lib
    # copy in pre-populated transformer cache. Remove if you don't need.
    COPY --from=builder-base $TRANSFORMERS_CACHE $TRANSFORMERS_CACHE
    
    WORKDIR /app
    COPY poetry.lock pyproject.toml ./
    COPY my_awesome_ai_project/ my_awesome_ai_project/
    
    EXPOSE 8080
    CMD ["python", "my_awesome_ai_project/app.py"]
    

    Usage

    • Build production Docker image:docker build --progress plain -t registry.example.com/project/production .
    • Build dev Docker image:docker build --progress plain -t registry.example.com/project/dev --target development .

    The dev image is also suitable as CI runtime image to shorten your CI time. Just mount your code into the container.

    Explanation

    • VIRTUAL_ENV lets Python and Poetry knows we are in the virtual environment, so that dependency lookup and installing works as intended.
    • PYTHONPATH guides Python to lookup modules in /app.
    • Modifying PATH so that the executables in VIRTUAL_ENV are callable system-wide.
    • Production image and dev image share many layers so building and storage are efficient.

    References

    Share on

    nanmu42
    WRITTEN BY
    nanmu42
    To build beautiful things beautifully.

    What's on this Page