Okay, disaster is a bit hyperbolic. I did however, end up banging my head against a wall for way too long trying to figure out an issue with a Docker image for use in a GitLab CI / CD pipeline. This had nothing to do with Docker or GitLab and everything to do with me… I hate it when that happens! In this short post I’ll run thru the scenario I found myself in. The issue was pretty obvious, but sometimes those are the hardest things to spot; hopefully I can save someone else some time down the line!

TL;DR

Make sure your Dockerfiles aren’t relying on local files which aren’t part of your source control repository. Noob mistake? Yup, pretty much!

The issue

My goal was to deploy a Phoenix application to a Kubernetes cluster via GitLab’s Kubernetes integration.

Everything went surprisingly smoothly (GitLab is rather fantastic) except none of my static files were showing up on the deployed instance, and looking in the browser console I was getting 404 errors for all the static files. So no images, no CSS and no Javascript. No happy 😞.

The weird thing was when I built and ran the image locally, static files were loading up fine.

Being that I had never deployed to a Kubernetes cluster before, I assumed the issue was something to do with a configuration setting or something else to do with the cluster. After spending way too much time messing around with everything but Docker, I happened to notice a log message appearing on GitLab during the build, something along the lines of The input path "priv/static" does not exist. This was occuring when the Docker build was running mix phx.digest… this being the Phoenix task that compresses and prepares static assets. It made sense that the directory was not available as priv/static is part of the standard .gitignore file for Phoenix applications; therefore it’s not going to be in the source repository.

So the fix was to alter the Dockerfile to build the assets prior to running mix phx.digest. Simple, probably pretty obvious, but it took quite the journey for me to arrive at the destination!

For reference here is the Dockerfile I ended up with, I originally had just mix phx.digest in the # Build assets section.

########################################
# 1. Build stage
########################################
FROM elixir:1.9.1-alpine as app_builder

# Set up build environment.
ENV MIX_ENV=prod

# Create the application build directory
RUN mkdir /app
WORKDIR /app

# Install build tools needed in addition to Elixir:
# NodeJS is used for Webpack builds of Phoenix assets.
# Hex and Rebar are needed to get and build dependencies.
RUN apk update \
    && apk --no-cache --update add nodejs nodejs-npm \
    && mix local.rebar --force \
    && mix local.hex --force

# Copy over all the necessary application files and directories
COPY config ./config
COPY lib ./lib
COPY priv ./priv
COPY mix.exs .
COPY mix.lock .
COPY assets ./assets

# Build the application.
RUN mix do deps.get, compile

# Build assets
RUN cd ./assets \
    && npm install \
    && ./node_modules/webpack/bin/webpack.js --mode production \
    && cd .. \
    && mix phx.digest

# Create the release
RUN mix release

########################################
# 2. Build release image
########################################
FROM alpine:3.9.4

# Install dependencies. Bash and OpenSSL are required for ERTS.
RUN apk update \
    && apk --no-cache --update add bash openssl

# Copy over the build artifact from step 1 and create a non root user
RUN adduser -S app
WORKDIR /home/app
COPY --from=app_builder /app/_build .
RUN chown -R app: ./prod

# Add the start commands bash file
ADD start_commands.sh /scripts/start_commands.sh
RUN chmod +x /scripts/start_commands.sh

# Switch to the non root user
USER app

# set the entrypoint to the start commands script
ENTRYPOINT ["/scripts/start_commands.sh"]

Summary

Sometimes it’s the smallest things that take the most time, but hey… them’s the breaks. A couple of lessons learned, I took from this experience:

• When working on a new Docker build, carefully look through the log output… just because the build “succeeds”, doesn’t mean all is good.
• When building Docker images locally, trash any directories in the project that aren’t included as part of the source repository. Had I done this, I would have seen the static asset issue showing up locally and trouble-shooting would have been much quicker. I would have immediately identified that the problem was with the build image versus some other issue.

Thanks for reading, hope you enjoyed the post and may all your Docker builds be a success!