A new way to Multisite: migrating Aegir to GitHub for WSU Vancouver

Washington State University at Vancouver set up their 10 websites on Aegir back in 2010 using Drupal 6. 

Thanks to Aegir, our client and friend Aaron Thorne was able to maintain all 10 websites by himself, despite not being a Drupal developer. Eventually, though, it was time for something new.

Last year, they contacted me to upgrade their sites and the hosting platform, but keep it inside their own private server infrastructure. 

We took our time to figure out how we could design a new model for a multisite codebase, including hosting, testing, and deployment. 

How can we implement reliable quality controls and automated delivery across all 11 sites? How can we make it as easy as possible for developers and system administrators to maintain? How can we leave WSU Vancouver with a system that they can use long term, so that they can update their codebase... forever?

The answer? A new model for multisite using DDEV, GitHub Actions, and clever usage of settings.php and Drush aliases.

I gotta be honest: as a developer, working with this system has been a dream. I am moving all of my sites to it.

A new multi-site architecture

In this article we'll cover a few of the components of the multisite codebase and the new "Operations Platform" we created to run it.

1. DDEV for live, local, and preview environments
2. GitHub Actions Matrix for multisite jobs
3. Multisite Drush aliases
4. Unified Settings.php 

DDEV for local, live, and preview environments

We settled on DDEV for developing the project. The project has recently become the "unofficial/official" development environment for Drupal, and the DDEV Foundation non-profit has been launched to support it indefinitely.

We decided to go for it and use DDEV for preview and live environments. Why?

1. Simplicity. With DDEV, there is no server configuration or management needed. Everything is in a pre-configured container.
2. System Parity. Local, live, and preview environments all launch from the same DDEV config.
3. Infrastructure as code. Developers control what services are used and how they are configured. Upgrading PHP version is done in a Pull Request.
4. Parsimony. One DDEV environment for all 11 sites, using a single web and database container for all sites minimizes resource consumption.

To get this to work required some clever tricks. We had to get DDEV to load sites based on a publicly available URL, not the local "ddev.site" url.

By writing a custom .ddev/config.z.yaml file with GitHub workflow, we were able to assign dynamic project names so that DDEV Sites would come up with unique URLs per pull request. We passed those URLs to the workflow "environment" settings, and voila! Running environments with links directly in the GitHub interface. See how this is done on the Operations Site Runner DDEV GitHub page.

Since all of our multisite config in DDEV config, when the Operations Site Runner launches an environment, all of the multisites launch with it.

Example DDEV Config

We used the DDEV Multisite documentation to create a .ddev/config.multisite.yaml file something like this:

# .ddev/config.multisite.yaml
additional_hostnames:
 - www
 - business
 - library
 - medicine
 - nursing
 - studentaffairs
hooks:
 post-start:
   - exec: |
       mysql -uroot -proot -e "CREATE DATABASE IF NOT EXISTS www; GRANT ALL ON www.* to 'db'@'%';"
       mysql -uroot -proot -e "CREATE DATABASE IF NOT EXISTS business; GRANT ALL ON business.* to 'db'@'%';"
       mysql -uroot -proot -e "CREATE DATABASE IF NOT EXISTS library; GRANT ALL ON library.* to 'db'@'%';"
       mysql -uroot -proot -e "CREATE DATABASE IF NOT EXISTS medicine; GRANT ALL ON medicine.* to 'db'@'%';"
       mysql -uroot -proot -e "CREATE DATABASE IF NOT EXISTS nursing; GRANT ALL ON nursing.* to 'db'@'%';"
       mysql -uroot -proot -e "CREATE DATABASE IF NOT EXISTS studentaffairs; GRANT ALL ON studentaffairs.* to 'db'@'%';"
     service: db

Screenshots

GitHub Actions Matrix for multisite jobs

Now that GitHub Actions is powering all of our deployment and testing, running persistent environments on internal servers, we have a lot of power:

1. GitHub Actions Matrix creates separate jobs for each site, allowing admins to view each sites deployments separately, even though they are on the same codebase.
2. Using GitHub Workflows provides seamless integration with GitHub Pull Request test results and Deployments API. No custom API integration is needed. Deployment of all 11 sites, with direct links to the preview environments, is shown directly in the GitHub pull request interface. Each matrix job appears as a separate test, allowing users to prevent merging until all 11 sites pass. 
3. All jobs can take advantage of the matrix feature. For example, every site gets it's own Cron action. This makes it easy to identify problems in a specific site without preventing other sites from running tasks.
4. Actions can be triggered manually through the GitHub interface. That way, when the server admin updates something like an SSL certificate, they can then go trigger the deployment by themselves.
5. GitHub Deployments API has many other integrations, such as Slack. Be notified when deployments start and stop.

The coolest thing for me is the bringing together of automated and manual testing in the Pull Request interface. The results of automated tests appear along side links going to the preview sites. This gives non-technical users an easy way to participate in the review process. You can configure your repo to require reviews from specific people before allowing merging, giving teams a very thorough way to control quality in their project.

There is so much to discuss here. See below for more articles on the topic.

Example GitHub Workflow Config

This small example runs cron against all of the sites listed in matrix:

# .github/workflows/cron.yml
name: Drupal Cron
on:
 workflow_dispatch:
 schedule:
   - cron: '0 * * * *'
jobs:
 cron:
   name: Drupal Cron
   runs-on: platform@server.prod.vancouver.wsu.edu
   strategy:
     matrix:   
       site:
         - www
         - business
         - library
         - medicine
         - nursing
         - studentaffairs
   steps:
     - name: Cron - ${{ matrix.name }}.${{ matrix.site }}
       run: |
         bin/ddrush @${{ matrix.site }} cron -v
         bin/ddrush @${{ matrix.site }} status

Screenshots

Multisite Drush aliases

For any Drupal project, Drush access is a must. With a multisite, you not only need a different alias for each environment, but for each site within each environment.

Since this was a migration, we also needed access to the legacy sites via drush for syncing data to preview environments and for the final migration.

1. We created legacy aliases to connect and sync from existing sites. drush @legacy.nursing sql-dump
2. Site aliases for managing any site in any environment. drush @nursing.pr123 status connects to the pr123 environment, nursing site.
3. To control the active site, only the site name is needed, simplifying scripts and management. Change into the folder, then run drush @nursing status from an environments folder will always connect to the site in that environment.
4. Drush access via ddev: 
   1. A small shell script wrapper to run "ddev drush" was created called "ddrush" for convenience, but also to allow drush remote aliases to work on a server using DDEV. Set "paths.drush-script" to ddrush, and remote calls will use that command instead. This allows users to connect via the host user, removing the need to setup an SSH host in the DDEV containers.

   2. Wildcard drush aliases allowed one alias entry to be used for all sites. We created an alias file like this:

      # drush/sites/live.site.yml
      '*':  
        host: server.vancouver.wsu.edu
        user: platform
        root: /var/platform/vancouver.wsu.edu/${env-name}/web
        uri: http://${env-name}.vancouver.wsu.edu
       paths:

      This allowed users to access any site by name, remotely from any clone of the codebase.

Unified Settings.php.

Most multisite setups follow the standard Drupal multisite setup, with a separate folder and settings.php file for each site.

Instead, we developed a single custom web/sites/default/settings.php file that dynamically set various Drupal settings:

1. Database connection. Set database name = site name.
2. Separate file directories. 
3. Separate config directories. 
4. Separate Hash salt.
5. Common configuration overrides for all sites.

This allowed us to have a single file to configure all sites on the platform.

<?php
# web/sites/default/settings.php
# Load the site name from the first part of the host name.
$hostname = explode('.', $_SERVER['HTTP_HOST'] ?? 'default');
$site = $hostname[0];
$environment = $hostname[1];
$base_domain = implode('.', array_splice($hostname, 1));

$databases['default']['default']['database'] = $site;

$settings['config_sync_directory'] = '../config/' . $site . '/sync';
$settings['file_public_path'] = 'files/' . $site;
$settings['file_private_path'] = '../private/' . $site;
$settings['file_temp_path'] = '/tmp/';
$settings['file_public_base_url'] = ($_SERVER['HTTP_X_FORWARDED_PROTO'] ?? 'https') . '://' . ($_SERVER['HTTP_HOST'] ?? 'default') . '/files';

# Make sure hash salt is different for every site using this codebase or things will explode.
$settings['hash_salt'] .= $site . $environment;

More information

This work has been part of my new effort to simplify Operator Experience, The Operations Experience Project.  

I've begun documenting the work in a book format here: https://operations-project.gitbook.io/operations-experience-project

There are a lot of more things that the Operations Platform does. Check out some more posts about it here:

1. Run CI/CD with preview environments anywhere with self-hosted Git runners.
2. GitHub for Everything: A new model for hosting & operations
3. Monitor and Control your sites with Operations Dashboard and Site.module: Track, organize, and log into your sites from one place. (Coming Soon).

Need Help? Contact Me.

Do you have a multisite setup that could use some of this? 

Get in touch today.