Docker

You're helping a small film festival collect trailers. Submissions arrive as .mov, .mkv, and enormous camera exports, so you build a Python service that turns them into smaller MP4 previews. FFmpeg does the actual conversion.

A clip converts on your Mac. On the festival's Linux server, the first upload fails with ffmpeg: not found. You install it, only to discover that the server's older version rejects an option your code uses. The next volunteer needs more than your Python files: the right runtime, the FFmpeg binary, and compatible system libraries.

A container image packages that application environment. A container starts a process from the image, with its own filesystem and process view. You can start several containers from one image without installing another copy of Python directly on the host.

Docker brought building, distributing, and running these images into one workflow. It grew out of deployment work at dotCloud; Solomon Hykes demonstrated it publicly in 2013. The Linux isolation mechanisms already existed. Docker made them accessible through a useful application-delivery toolchain.

Packaging Python and FFmpeg

On your Mac, you install and open Docker Desktop, then check that both the client and engine are available:

docker version
docker compose version

Installing the CLI alone does not start an engine. Docker Desktop runs Linux containers inside a Linux VM on macOS. On the Linux server, those containers use its Linux kernel. Their processes are isolated, but each container does not boot a separate operating system.

Your service already has an app.py that listens on 0.0.0.0:8000, a requirements.txt with pinned Python dependencies, and a DATA_DIR setting for uploads and previews. You add a Dockerfile alongside them:

FROM python:3.13-slim
WORKDIR /app

RUN apt-get update \
    && apt-get install -y --no-install-recommends ffmpeg \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

RUN mkdir -p /data/uploads /data/renders \
    && chown -R 10001:10001 /data
USER 10001:10001
EXPOSE 8000
CMD ["python", "app.py"]

The base image supplies Python and its Linux userspace. The package-manager step installs FFmpeg inside the image; your Mac's FFmpeg installation is irrelevant to this build. RUN executes during the build, while CMD sets the command for a new container. The JSON form starts Python directly without putting a shell in front of it.

The app runs as user 10001, with permission to write its media directories. Keeping dependency installation before COPY . . also means a code-only edit can reuse the dependency layers. Changing requirements.txt invalidates that install step and the steps after it.

Before building, you add .dockerignore. The raw trailers, local environment, and credentials have no business being sent into the build context:

.git/
.venv/
__pycache__/
.env
.env.*
!.env.example
uploads/
renders/

The final . in the build command selects the build context. .dockerignore filters it before files reach the builder; .gitignore does not do that job. A secret copied into an earlier image layer is not erased by deleting it in a later layer.

You build the first image and check the FFmpeg installation it contains:

docker build --tag festival-transcoder:0.1.0 .
docker run --rm festival-transcoder:0.1.0 ffmpeg -version

Here, ffmpeg -version replaces the image's default command for this one container. --rm removes it when the command exits; the image remains available. Your newly built image contains the installed packages, but a future rebuild can resolve newer base-image or OS-package versions. Pin those inputs when you need reproducible builds, and retain the actual image you tested.

Keeping trailers between deployments

For the first run, your local .env supplies settings the app already understands:

DATA_DIR=/data
MAX_UPLOAD_MB=200

You create a named volume for the media and start the web process with a CPU and memory budget:

docker volume create festival-media
docker run --detach --init --name festival-web \
  --publish 127.0.0.1:8080:8000 \
  --env-file .env \
  --mount source=festival-media,target=/data \
  --cpus 1 --memory 1g \
  festival-transcoder:0.1.0
curl --fail http://127.0.0.1:8080/health

You open port 8080 on your Mac; Docker forwards it to port 8000 inside the container. EXPOSE 8000 only documents the application's port. The app must listen on 0.0.0.0 inside its container, and publishing on the host's loopback address keeps this preview local.

--env-file supplies runtime configuration without baking it into the image. Environment variables are still visible to people with sufficient access to the container; production credentials need appropriate secret handling. --init adds a small init process to help with signal forwarding and child reaping, useful when Python launches FFmpeg. The app still needs to finish or cancel jobs properly when stopped.

The first small trailers work. A larger encode then exits unexpectedly, so you inspect the process and its resource use:

docker ps -a
docker logs --tail 100 festival-web
docker stats --no-stream festival-web
docker inspect festival-web --format '{{.State.ExitCode}} {{.State.OOMKilled}}'

An exit code of 137 means the process received SIGKILL; it does not by itself prove an out-of-memory kill. OOMKilled, logs, and measurements narrow the explanation. You size the worker's budget and limit concurrent encodes against real files, rather than assuming every video fits the same memory allowance. Logs also need the app to write useful output to stdout and stderr.

Redeploying introduces another question: where do yesterday's uploaded trailers go? A container has a writable layer, but removing the container removes that layer. Stopping it is different:

docker stop festival-web
docker start festival-web
docker stop festival-web
docker rm festival-web

The restart uses the same container, including its original image and configuration. It does not adopt a newer image just because you rebuilt a tag. The later removal discards the container, while festival-media keeps the uploads and previews stored under /data. A replacement container must mount that volume again.

While adjusting the upload page in app.py, you want a shorter edit-and-run loop. You start a temporary development container with that file mounted from your Mac:

docker run --rm --init --name festival-dev \
  --publish 127.0.0.1:8081:8000 \
  --env-file .env \
  --mount type=bind,source="$PWD/app.py",target=/app/app.py,readonly \
  --mount source=festival-media,target=/data \
  festival-transcoder:0.1.0

The bind mount makes that host file visible over the image's copy. Editing it does not rebuild the image, and Python still needs a restart or development reloader to execute the new code. The named volume holds media independently; the bind mount is tied to a particular host path. Ctrl-C ends this foreground container, and --rm cleans it up.

Moving conversions to workers

As submissions increase, a long conversion keeps the web process busy. You move encoding into worker.py and let the API enqueue work in Redis. The new app version reads REDIS_URL, which you add to .env:

REDIS_URL=redis://queue:6379/0

Now there are three processes to coordinate. You describe them in compose.yaml, reusing the app image for the API and worker:

name: festival
services:
  api:
    build: .
    image: festival-transcoder:0.2.0
    init: true
    ports:
      - "127.0.0.1:8080:8000"
    env_file: .env
    volumes:
      - media:/data
    depends_on:
      queue:
        condition: service_healthy

  worker:
    image: festival-transcoder:0.2.0
    command: ["python", "worker.py"]
    init: true
    cpus: 1
    mem_limit: 2g
    env_file: .env
    volumes:
      - media:/data
    depends_on:
      queue:
        condition: service_healthy

  queue:
    image: redis:8-alpine
    command: ["redis-server", "--appendonly", "yes"]
    volumes:
      - queue-data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 10

volumes:
  media:
    external: true
    name: festival-media
  queue-data:

Compose gives the services a shared network. The worker reaches Redis at queue:6379, using its service name, not a container IP. localhost inside the worker would refer to the worker itself. Redis has no published host port here. Both app processes see the same media volume, while Redis keeps its own data in queue-data.

The API keeps the image's default CMD; the worker overrides it. env_file explicitly supplies their environment variables. Compose's automatic reading of .env for substitutions is a separate feature—it does not automatically inject every value into every container.

With the earlier standalone web container removed, you build the app image once for both services, then start this version:

docker compose config --quiet
docker compose build api
docker compose up --detach
docker compose ps
docker compose logs --follow worker

Ctrl-C stops following the logs without stopping the detached services. The health check lets Compose wait for Redis to answer before starting its dependents. That handles initial readiness; the application still needs reconnect and retry behavior if Redis becomes unavailable later. Likewise, restarting a worker cannot decide which interrupted jobs need retrying—that belongs to the queue consumer and job logic.

Once each worker safely claims independent jobs, you can run another instance with docker compose up --detach --scale worker=2. That also increases the total CPU and memory demand. It helps only while the host, storage, and other dependencies have capacity.

docker compose down removes this stack's containers and network, keeping named volumes by default. Adding -v deletes managed volumes such as queue-data. Our media volume is external and has its own lifecycle. Surviving container replacement is useful, but it does not make either volume a backup against losing the server.

Shipping the tested image

The local stack is ready, but your Mac may be ARM-based while the festival server is x86. You create a repository in Docker Hub and use Buildx to publish images for both architectures. Replace YOUR_DOCKERHUB_USER with the account that owns it:

docker login
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --tag YOUR_DOCKERHUB_USER/festival-transcoder:0.2.0 \
  --push .
docker buildx imagetools inspect YOUR_DOCKERHUB_USER/festival-transcoder:0.2.0

The registry stores an image index pointing to the platform-specific images. A compatible engine selects the matching one when pulling. Docker Desktop supports emulated builds, although compiling or encoding under emulation can be slow; native builders are useful for heavier pipelines.

You record the image-index digest from the inspection output and test that exact reference for the server's platform. A tag is a movable name, while a digest identifies particular content. You promote the tested reference instead of installing dependencies or rebuilding on the live server.

On the server, Docker Engine and the Compose plugin are installed, registry access is configured, and the existing HTTPS proxy forwards requests to loopback port 8080. You copy the Compose files and supply the server's runtime .env. Create its festival-media volume too; Docker does not transfer the uploads from your Mac when it transfers an image.

A compose.prod.yaml override selects the tested image for both app processes. Replace the account and sha256:IMAGE_DIGEST with the real image reference:

services:
  api:
    image: YOUR_DOCKERHUB_USER/festival-transcoder@sha256:IMAGE_DIGEST
  worker:
    image: YOUR_DOCKERHUB_USER/festival-transcoder@sha256:IMAGE_DIGEST

The base file still contains a build definition, so you explicitly disable building during deployment:

docker volume create festival-media
docker compose -f compose.yaml -f compose.prod.yaml pull
docker compose -f compose.yaml -f compose.prod.yaml up --detach --no-build
docker compose -f compose.yaml -f compose.prod.yaml ps
curl --fail http://127.0.0.1:8080/health

Compose recreates containers whose configuration or selected image changed. If this release fails, restore the previous tested image references and run the same pull-and-up sequence. That replaces the application processes; it does not undo changes already made to queued jobs or stored media, so old and new versions need a compatible data contract.

Before accepting submissions again, you upload a sample trailer, check its playable preview, and confirm that the file remains available after replacing a worker. A successful /health response on its own has not exercised FFmpeg, job processing, or media persistence.