Aegir on Docker: Faster Development, Better Testing, Easier Deployment.

In the last few days We've been able to get Aegir working in a docker container.

See http://github.com/aegir-project/dockerfiles for the code and https://hub.docker.com/r/aegir/hostmaster/ for the images.  There are example docker-compose.yml file in the codebase that you can use to launch aegir really quickly.

The goal is to offer "official" containers, with tags for every OS, different web servers, different versions of PHP, etc.  See https://hub.docker.com/_/php/ for an example.

Advantages

• A better Task Queue!

  The “task queue runner” is really just a continuous process that aegir must always have running so that it can respond to actions on the front-end in a reasonable amount of time. That process is usually handled by Supervisor.

  Docker is all about keeping processes running, so what we did was set the drush command ‘drush @hostmaster hosting-queued’ to be the command that the container runs!

  We didn’t have to worry about starting Apache, because verifying sites in aegir (including hostmaster) does that for us.

  This results in a really fast Aegir site. The tasks start almost instantly.

  I think this is the future for Aegir. Docker kills the container if and when the process stops, and has “Restart” policies to get it going again.  Since every piece of important data is in a volume, the system is back up and running again in seconds.

  We do have more work to do to make aegir truly highly-available: having the webserver and the queue process in the same container is convenient, but not ideal: it would result in the web server going down (albeit only briefly if restart policies are in place) if the queue process died.

  Having the webserver container separate from the hosting queue would also make it easier to pipe webserver logs to docker.

• Faster, easier local development: launching a new aegir takes about 1.5 minutes for me, once the images are downloaded.  We have created a “development” folder in the dockerfiles repo to design a great local development environment for Aegir.  This will also allow us to build a next-gen aegir powered local drupal development tool.

• Behat tests have been implemented, starting with the basics. These tests will make sure everything really works the way we want it to. I created a basic test as a proof of concept. This is an easy place for the team to contribute.

  See http://github.com/aegir-project/tests.  We have Behat tests for devshop that solves the queue problem: Behat can run drush commands.

• Travis (and other) integrations. We now have hostmaster install happening on docker, and we have behat tests, so now we can get better CI.  We will start with Travis for it’s ease of use but the containers and tests can be run anywhere.  Check out a passing travis test run.

• Developer Community Experience: With travis and github and the right scripts in place, It will become much easier for community members to contribute. Every file in GitHub has an “Edit” button, and with a few more clicks, any user can have a fork and a pull request submitted to your project. With good tests, we will know right away if they broke anything.  

• More sharing between DevShop & Aegir: with the Dockerfiles, you can override the environment variables at run time to setup a different flavor of Aegir.  See the docker-compose.yml file we created that launches a devshop server at https://github.com/opendevshop/devshop/blob/1.x/tests/docker-compose.yml. 

I know there is debate about which git platform to use. I have to point out, that from experience, there is no better tool on earth for code review and communication than github pull requests. Combined with the ease of use for new contributors, I think we should focus on github.com for now with gitlab being used for mirrors.

Once the system is in place, we will be submitting our open patches as github pull requests as a test of this new system.

Next Steps

• Create .travis.yml files for all sub projects like provision, hosting, hostmaster.  Each repo should be testable on it’s own. This allows for users to submit Pull Requests on github and have the full test suite run on their code.  We can use some examples from devmaster here, as we were able to write a .travis.yml script that runs on the submitted fork.

• Create Dockerfiles for all the popular base images:

  • Ubuntu 14 (Done)

  • Centos 7

  • Debian

  • Alpine

• Create “remote server” images.  This is 90% done. All we have to do is pass the public key of aegir@server_master as an environment variable, and put it in .authorized_hosts.  

• Create docker-compose.yml templates for High Availability.  Docker-compose was designed to be scalable: docker-compose scale web=5 will make sure there are 5 “web” containers running.

• A scalable drupal docker-compose setup was created for the Terra CLI project: See https://github.com/terra-ops/terra-cli/blob/master/docs/docker-compose-example.yml .  Using the tutum/haproxy image allowed automatic reconfiguration of haproxy when the web containers scaled.

• Create Aegir Docker module for providing docker services to Aegir UI. We already have most of this in DevShop Rancher: https://github.com/opendevshop/devshop_rancher

  • This code works in creating a template docker-compose.yml file and running docker-compose up when the aegir site is installed.

  • The way aegir installs Drupal is hard coded to access files and database locally.  We have a patch going to decouple the install and verify processes so we can run alternate commands to “install” a site: https://www.drupal.org/node/2754069

  • This patch is a blocker for running drupal in anything other than LAMP, and for installing anything other that Drupal. If you want Aegir to be able to do either of those things, come help out on this patch!

So as you can see, there’s a lot going on in the Aegir world!

If you have Docker experience and want to help, please, join us in #aegir on IRC!

Stay tuned for more posts on Aegir & Docker.