This is the third part of a series about deploying a non-Rails application with Kamal 2. Read the previous article here. Follow this journey at https://github.com/jjeffers/sherldoc.

Deploying a Queue Worker

I need to provision a “version” of sherldoc that will handle workers to process asynchronous jobs for the web application. The original sherldoc docker compose layout used a shared docker volume and the same sherldoc web container to run the queue workers.

In our configuration the containers don’t share a mounted volume at runtime. Instead, each image uses the same container image with similar post-run commands. The final docker entrypoint commands will diverge, with the queue worker starting the artisan worker process instead of the web application server steps.

Our deploy.yml includes the following additional server entry:

...
image: sherldoc-web 
...  
servers:
  web:
    hosts:
      - 159.203.76.193
    cmd: bash -c "/app/prepare_app.sh && cd /app/public; frankenphp php-server"

  workers:
    hosts:
      - 159.203.76.193
...

Kamal uses every server entry to deploy a container of the image entry, “sherldoc-web”. I can override the “workers” container with a new CMD.

After a another deploy I check kamal app details :

kamal app details -c ./deploy.yml 
  INFO [006857a0] Running docker ps --filter label=service=sherldoc --filter label=role=web on 159.203.76.193
  INFO [006857a0] Finished in 2.108 seconds with exit status 0 (successful).
App Host: 159.203.76.193
CONTAINER ID   IMAGE                                                                                                                          COMMAND                  CREATED              STATUS              PORTS              NAMES
bc7b3029bda6   registry.digitalocean.com/team-james-demo/sherldoc-web:[...]   "docker-php-entrypoi…"   About a minute ago   Up About a minute   80/tcp, 9000/tcp   sherldoc-web-[...]

  INFO [4c339292] Running docker ps --filter label=service=sherldoc --filter label=role=workers on 159.203.76.193
  INFO [4c339292] Finished in 0.229 seconds with exit status 0 (successful).
App Host: 159.203.76.193
CONTAINER ID   IMAGE                                                                                                                          COMMAND                  CREATED              STATUS              PORTS              NAMES
a913b7fcd417   registry.digitalocean.com/team-james-demo/sherldoc-web:[...]   "docker-php-entrypoi…"   About a minute ago   Up About a minute   80/tcp, 9000/tcp   sherldoc-workers-[...]

Both containers are up, but the container with the name “sherldoc-workers” started with the artisan queue:work command.

Why No Supervisor?

I debated adding the supervisor (the process control utility) indicated in the source project’s docker compose configuration.

I suspect if sherldoc-workers container halted kamal-proxy would then restart it. I am not aware of any restart policy set for the docker containers on the server. Any restart would have to be external to the docker engine.

I tested this theory, attaching to the sherldoc-worker container and killing the main process, artisan queue:work. Within a few moments a new sherldoc-workers container was up and running.

Given this, I decide not to install or run supervisor.

Booting Additional Accessories

Next, I stand up Redis and Apache Tika.

...
accessories:
  ...
  redis:
     host: 159.203.76.193
     image: redis:6
     directories:
      - data.redis:/data

  tika:
    image: apache/tika:2.9.2.1-full
    ports:
      - 9998:9998

With that configuration change I start the redis instance:

kamal accessory boot redis -c ./deploy.yml 
  INFO [2f687e13] Running /usr/bin/env mkdir -p .kamal on 159.203.76.193
  INFO [2f687e13] Finished in 1.466 seconds with exit status 0 (successful).
Acquiring the deploy lock...
  INFO [bdb656a9] Running docker login registry.digitalocean.com/team-james-demo -u [REDACTED] -p [REDACTED] on 159.203.76.193
  INFO [bdb656a9] Finished in 0.936 seconds with exit status 0 (successful).
  INFO [34829155] Running docker network create kamal on 159.203.76.193
  INFO [16f49764] Running /usr/bin/env mkdir -p $PWD/sherldoc-redis/data.redis on 159.203.76.193
  INFO [16f49764] Finished in 0.210 seconds with exit status 0 (successful).
  INFO [bf65cb9c] Running /usr/bin/env mkdir -p .kamal/apps/sherldoc/env/accessories on 159.203.76.193
  INFO [bf65cb9c] Finished in 0.171 seconds with exit status 0 (successful).
  INFO Uploading .kamal/apps/sherldoc/env/accessories/redis.env 100.0%
  INFO [f431adff] Running docker run --name sherldoc-redis --detach --restart unless-stopped --network kamal --log-opt max-size="10m" --env-file .kamal/apps/sherldoc/env/accessories/redis.env --volume $PWD/sherldoc-redis/data.redis:/data --label service="sherldoc-redis" redis:6 on 159.203.76.193
  INFO [f431adff] Finished in 1.865 seconds with exit status 0 (successful).
Releasing the deploy lock...

And finally, I spin up the Tika instance:

kamal accessory boot tika -c ./deploy.yml 
  INFO [32435277] Running /usr/bin/env mkdir -p .kamal on 159.203.76.193
  INFO [32435277] Finished in 1.398 seconds with exit status 0 (successful).
Acquiring the deploy lock...
  INFO [42be2d9b] Running docker login registry.digitalocean.com/team-james-demo -u [REDACTED] -p [REDACTED] on 159.203.76.193
  INFO [42be2d9b] Finished in 0.537 seconds with exit status 0 (successful).
  INFO [e9abb23f] Running docker network create kamal on 159.203.76.193
  INFO [b36fa775] Running /usr/bin/env mkdir -p .kamal/apps/sherldoc/env/accessories on 159.203.76.193
  INFO [b36fa775] Finished in 0.207 seconds with exit status 0 (successful).
  INFO Uploading .kamal/apps/sherldoc/env/accessories/tika.env 100.0%
  INFO [94a1e92e] Running docker run --name sherldoc-tika --detach --restart unless-stopped --network kamal --log-opt max-size="10m" --publish 9998:9998 --env-file .kamal/apps/sherldoc/env/accessories/tika.env --label service="sherldoc-tika" apache/tika:2.9.2.1-full on 159.203.76.193
  INFO [94a1e92e] Finished in 16.586 seconds with exit status 0 (successful).
Releasing the deploy lock...

So far everything looks like it’s working, or at least the parts are operational. I’ll dive into the application itself next and see is working under the hood.

The Challenge

After Michael Kimsal released his proof of concept project sherldoc, I was wondering if I could deploy this using Kamal, a deployment tool typically used for Rails web applications. Supposedly Kamal is application and framework agnostic.

Getting this to work is like looking for Bigfoot or that Giant Squid you see in those late night shows on the History Channel. It should be possible!

First, let us look at sherldoc’s description:

Web service endpoint to scan a document for

• existence of keyword or phrase
• absence of keyword or phrase

The project provides a Docker Compose configuration for deployment. Several assumptions are built into the current configuration, which I will outline later. The challenge will be to convert these assumptions into Kamal configuration directives and then get a running sherldoc instance.

Why Kamal?

Kamal promises:

Kamal offers zero-downtime deploys, rolling restarts, asset bridging, remote builds, accessory service management, and everything else you need to deploy and manage your web app in production with Docker. Originally built for Rails apps, Kamal will work with any type of web app that can be containerized.

and

Kamal basically is Capistrano for Containers, without the need to carefully prepare servers in advance. 

Capistrano was the old reliable deployment tool for many years in the Rails world. The idea that we can use the same ease of deployment with containers on a naked VM is attractive.

The whole point is to make it easy to put your application on a low-cost VM, either in some hosted environment or perhaps an on-premise machine. Consider this to be a potent weapon in the crusade against BigVM^TM.

I am not a PHP expert, so getting this to work will push me outside of my comfort zone. I may fail. I may not find the Giant Squid or Bigfoot. In the worst case, I’ll learn something new.

Analyzing the Docker Compose configuration

I start by examining the original docker-compose.yml file for clues:

services:
     nginx:
          image: nginx:stable-bullseye
...
     app:
          image: sherldoc/web:1.0
...
     redis:
          image: redis:6
...
     tika:
          image: apache/tika:2.9.2.1-full
...
     supervisor:
...
     psql:
          image:postgres:16.1-bullseye
...

We start with the entry “app”, which is the PHP web application. This will be the source of our Kamal application image. Kamal creates this image and then stores it in a registry we specify. Once successfully connected to our provisioned virtual machine, Kamal installs docker and then download the image from the registry.

We can see that the original configuration required a docker supervisor process “supervisor”. Since we will only require a single image, the PHP application, the supervisor is not needed. Therefore, we omit this in our Kamal configuration. [Edit: This is not correct. The supervisor image is actually the sherldoc application “jobs” worker process. We will need to replicate this and as we will see, Kamal anticipates this need.]

The “nginx” entry hints that the PHP web application depends on the Nginx application proxy. We can peek inside the associated Nginx dockerfile (docker/app/nginx/conf.d/app.conf) and see that the application directives. Kamal provides an application proxy (“kamal-proxy”, https://kamal-deploy.org/docs/configuration/proxy/) which, in theory, provides the same capabilities.

The “redis”, “tika”, and “postgres” entries indicate additional services that the web application relies on. Each of these services has an associated container image.

Kamal provides configuration options for “accessory” services as well (https://kamal-deploy.org/docs/configuration/accessories/). As long as we can use the same images and apply similar configuration options to match the original values in the docker-compose.yml file it should work.

Preparing for Kamal

I forked the project to avoid bombarding the original project with PR requests. Perhaps my work will be merged in later.

Next I installed Kamal and initialized the local workspace with kamal init.

Next, I edited the default Kamal configuration (config/deploy.yml) with the following, removing all the comments for easier reading:

service: sherldoc

# Name of the container image.
image: sherldoc

# Deploy to these servers.
servers:
  web:
    - 159.203.76.193
registry:
  server: registry.digitalocean.com/team-james-demo
  username: my-user

  password:
    - KAMAL_REGISTRY_PASSWORD

builder:
  arch: amd64
  dockerfile: docker/app/php/php.dockerfile

Let’s review the contents:

service: sherldoc

# Name of the container image.
image: sherldoc

The name is just the name of the original project. No magic here.

servers:
  web:
    - 159.203.76.193

The IP address is the same address as the provisioned in my VM provider of choice at Digital Ocean. This is the cheapest configuration I could find. It might be too small or under provisioned, but we can fix that later.

registry:
  server: registry.digitalocean.com/team-james-demo
  username: my-user

  password:
    - KAMAL_REGISTRY_PASSWORD

Kamal will generate a docker image then push that image into your registry. Because I am using Digital Ocean I can use the Digital Ocean registry service. I could have also used Docker Hub, AWS Elastic Container Store, or any other container registry.

The KAMAL_REGISTRY_PASSWORD is an environment variable set to the credentials (an authentication token) provided by Digital Ocean. For security reasons, I don’t want to commit the actual value to the configuration file. I’ll leave this to be constituted at runtime.

First deployment attempt

All these things in place, we kick off the build with “kamal setup”.

INFO [fe0776d2] Running /usr/bin/env mkdir -p .kamal on 159.203.76.193
INFO [fe0776d2] Finished in 1.702 seconds with exit status 0 (successful).
Acquiring the deploy lock...
Ensure Docker is installed...
INFO [edde3944] Running docker -v on 159.203.76.193
INFO [edde3944] Finished in 0.186 seconds with exit status 0 (successful).
Log into image registry...
INFO [8eb7c038] Running docker login registry.digitalocean.com/team-james-demo -u [REDACTED] -p [REDACTED] as jdjeffers@localhost
... (lots of logs cut out here) 
INFO [9ce887de] Running docker container ls --all --filter name=^sherldoc-web-7ceb4de587a2119c9b007f40973a40cd7eb88b8e$ --quiet | xargs docker inspect --format '{{json .State.Health}}' on 159.203.76.193
INFO [9ce887de] Finished in 0.249 seconds with exit status 0 (successful).
 ERROR null
INFO [54c5ab35] Running docker container ls --all --filter name=^sherldoc-web-7ceb4de587a2119c9b007f40973a40cd7eb88b8e$ --quiet | xargs docker stop on 159.203.76.193
INFO [54c5ab35] Finished in 0.404 seconds with exit status 0 (successful).
  Finished all in 571.2 seconds
Releasing the deploy lock...
  Finished all in 573.5 seconds
ERROR (SSHKit::Command::Failed): Exception while executing on host 159.203.76.193: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy

This result is expected for several reasons. The original application:

1. doesn’t provide a default 200OK to the kamal heartbeat request at “/up”,
2. expects a redis instance,
3. expects an Nginx application proxy,
4. expects a tika server process,
5. expects a PostgreSQL database.

Without these other services, the sherldoc PHP application is probably not going work! We’ll fix these issues next.

Want to follow this quest? Read part 2!