3 Jan 18

all right so John is gonna do a talk on

provision for dot X a developer John is

the man the myth the legend and that is

his intro no John pew is an independent

drupal consultant and code artist it's

kind of like a sandwich artist from

subways that were that's good I like


his company think drop is focused on

business app design DevOps support and

content strategy in New York City he is

an agar maintainer for those you don't

know John can tell you all about ager

and he created devlop and open source

Drupal development platform

can you tell you all about that - I'm

gonna stall real quick while John gets

his computer all set up so this building

was built in 1930 all right you're good

Meet Jon

yes all right John everybody John pew

wow thanks for coming out it's really

cold this is a big crowd I wasn't

expecting okay this is gonna be this is

you know this isn't really building

Drupal huh Mike Symon - yo thanks Frank

like you said this is me the point of

Why Provision

this thing in the point of all the

things that I work on are just trying to

make things simpler it's very very hard

to launch websites did you even if you

know how to do it it's still a thing you

know you may have your scripts ready but

there's plenty of documentation on how

to do it though you can go to Drupal org

and find out right away and learn really

quickly how easy it is to get drupal up

and running there's only these four

steps and then that one's too long you

can read the quick install which is

really short and you can spend you know

it's easy stuff like he just come on

everybody should know this stuff and no

we want to know how to click and build a

website and forget about almost DevOps

stuff right so like I said the problem

is running websites in your own computer

is too hard no matter what provisioner

you use you still have to manage your

own metadata provisioner means like

ansible or chef

for puppet or any of these things you

DevOps people in SystemC will tell you

that they use to provision their servers

like they're still storing somewhere

what git repo am i pulling from where

should I go all that stuff so they still

have to manage that somehow and most of

these provisioning tools are written in

a languages other than PHP which is okay

but there's way too many ways to render

people you can tell from my very

scientific graph that I created here but

it's true go look up how many things

that ways there are to launch Drupal so


in 2007 this started out as a historic

talk right I said oh 10 years of hosting

Drupal because this Egger thing just hit

10 years of commits no one really knows

when it started but the first actual

recorded commit was on December 7 2007

and you know some people were like man

we want to be able to host Drupal really

easy because this Drupal thing is

amazing even 10 years ago it was really

impressive and they were like what if we

could use Drupal to build it's to manage

our Drupal sites and and they did it and

they come created this company called

bright which you can only find on the

archive if you know the domain and they

look for it so I did but they were

really like the earliest kind of Drupal

as a sass like hosting service trying to

automate things and you know who knows

what happened one thing led to another

they were required and transferred to

other people but the point is the source

code for that platform they both came

from open source just like Drupal became

open source and everyone started using

it this thing is just Drupal modules and

they open source that so that we could

click and say add a site type in the

domain name and it runs all the scripts

in the backend needed to actually turn

the site on this is the UI and it's been

the UI since about 2007 hostmaster is a

User Interface

component of that

it's a Drupal distribution so all of

that stuff is Drupal code that you saw

it's and Drupal nodes and fields and

templates and all of those things so

it's really familiar to you and you can

add modules to it to do all the extra

things that Drupal can do Hosting is the

name of the module is Amit self and

provision is the name of these Drudge

commands that are on the backend you may

not know what Russia is so that's a

shell tool for drupal it's a drupal

shell and you can type in brush cache

clear for example and you don't have to

click through all the pages to hit the

button you just type in oshkosh clear it

does all the things

so they extended that to make commands

like provision verify and version

install so that it would actually spin

up all the configuration needed to run

your site Apache configs and database

configs and created database

automatically set all the files

everything in that instructions list a

grid's does automatically for you every

time a little bit of a history graph so

the commits do go back that far

it's intense sometimes digging there

most of mine are like all at the end you

can see the little spike here because

I'm doing crazy on this new project this

is the original commit not terribly

interesting there's only 11 files and

1,300 and whines this is the provision

Inc file as it stands today so get blame

on github is really handy tool if

anybody has an old project very useful

to find that blame button and you can

figure out line by line who did what and

so you can see ten years ago six years

ago ten years ago really really legacy

stuff and that's how it worked in 2007

Whats next

put all these things together and they

were able to build the Drupal hosting

company and also release it open source

so in 2018 what are we gonna do we do

the same thing right now basically it's

basically the same packages it's Drupal

7 front end it's add rush package it

some purple modules it's it's it's

really pretty it's getting old and

showing us age I'm not gonna go through

every detail but like it's got a long a

storied history communities survive


throughout all these things it's older

than Drupal 6 so we're we are still

doing things with it right so what's

next is that the system needs an

overhaul there clearly there's a lot of

smart people involved and they have many

different ideas on how it should go

Roger what it should do so the team

talked together and figured out that if

we break it up into segments segments we

can kind of tackle things elmo herman

van rank is kind of in the current

branch maintainer so he puts up the

releases of ager 3 which is stable right

now for hosting tons of Drupal sites

NASA that's like 500 Drupal sites

running on this I took over what I'm

calling the four point X which is what

I'm working on now we're going to show

and then Christopher Gervase is working

on a totally brand-new next-generation

thing that has no no comment upstream at

all so

Eggar we're calling five for necks and

so we broke it up like this so we can

keep developing the stable version work

on kind of the current gen next-gen

thing on me okay before and then Chris

is looking at the long term with five so

yeah it is a little confusing but this

is all work in progress and we're making

it trying to get through this quick as

possible so yeah like I said Hager for

may or may not be a thing I'm working on

provision for I care most right now

about the back end this command-line

interface which isn't just for hosting

under cloud servers but this should be

easy enough to run on your on your Mac

laptop so you can I'm trying to make

this thing easy up so you're brand-new

to Drupal Oh designer whatever you don't

know how to program you can just call

these little commands and it'll spin up

your Drupal site on any computer once

that's stable and works there's a lot of

features that the main eager thing uses

because it's a huge multi site hosting

machine then we can kind of think about

bringing it bringing the eager front-end

into four point next fully if we do get

there I'm gonna bring in all these extra

tools that we built with like dev shop

and this CI stuff and Drupal that we've

brought all into the bring it all into

the native code base again details don't

really matter but this is way down the

road you know I'm working on focusing on

on the on the backend CLI at first but

hopefully I'm have a little running web

site Drupal on my Mac right now so in

terms of working when VP of turning a

Drupal site on on your laptop this is

gonna be something I put out as like an

alpha in the next few months

I can explain before with five it's a

totally new new rewrite that he's doing

I'm not involved in that five point X

but you can check it out but you really

have to know a lot about it because it's

I haven't even run it yet so it's

Command Line Support

intense stuff but with four point X and

we're gonna back it up a bit

in September rush announced they're not

going to support things the way they

used to support them am I gonna get into

details but basically the command line

thing is like becoming a module you want

to put it right in there with your site

so it's not this external thing so we

realized the team realized okay we got

Total Rewrite

it's finally due time for total rewrite

it's been a long time coming

I've personally built a lot of Symphony

console apps there's a lot of other

people that have built sibling console

apps everything here from composer and

all these things that

once you're in Drupal long enough you

care about are all built on the same

components there are underlying they're

all symphony which is what Drupal 8 is

powered by so there's a million examples

of how to build a really powerful CLI

tool that's really easy to use so

basically I've been dreaming of doing

this for a long time and the opportunity

arose so if I'm only like sitting down

coding it all the details we're going to

store the desktop in yellow files once

we get the commands working well enough

we can put it in there with the the

front-end hosting system but primarily I

needed it to be simple and easy to use

and complete and to do everything for

you like the dress commands were not

easy to use at all you need to know

everything ahead of time basically no


one used the backend command line for

Haggar I would wager they said it was it

was there so you could use it but no one

actually did because it just was not

easy to use so how do we make it easy to

use clean and simple console makes it

things colorful concise not cluttered

you just type in provision and you see

the list of things that you're able to

do ads so it works with context you save

like the site it represents the platform

represents the site you're working on

when you type in save if you don't

specify upfront it asks you for each

question and generates the defaults if

possible like I'm in this folder so if I

just I could just hit enter and it will

use that folder for the path to my

Drupal site I'm trying to minimize the

keystrokes that everyone has to make

we're gonna do a live a live type

through thing here but like like we

don't just doing complex things down to

as simple as possible when you're using

it right in the terminal right so

writing the web config green the data

base something fails it just throws an X

we're not gonna like spit out everything

unless you ask for it being verbose this

is what a successful verify it looks

like so like all those instructions

right create database and your patchy

and all that we're just running them as

tasks and it interactively shows the

tasks checks it if it runs successfully

puts an X if it fails this thing has to

be easy to develop Aker was really hard

to contribute to you know this is just

background but the way it works is it

you're storing things like you

it represents your local computer you

know it knows it has a patching in a


the site is URLs you can put as many

sites on that server and platform as you

can the platform represents the codebase

all this stuff is now stored in these

little simple yamo files we don't you

don't write these like when it's asking

you those questions you're answering

them and it writes it for you

automatically but you can go in there

and read it and see okay that's how my

thing is configured properties like

route that's where are my drupal files

locating the make file is like the thing

that is used to build your site with all

the modules in it I get URL is like

where it's going to clone from so if you

put get URL in there and you verify this

platform it'll automatically clone it

into that root alright so you don't do

any of these manual steps in terms of

programming interactively it's like

tells you this this is this green down

here is what you see this is the default

if you hit enter that's stored and it's

basically I was able to because it's

symphony and all these things that I'm

learning about like chain methods I can

just create this very simple ok platform

has a route that the description is this

the default value is the current folder

you're in whether or not it's required

and then a special function of like

validate that again I'm going over the

heads a lot of people because this is

not stuff but all you developers out

there this is how easy this is gonna be

you know I'm talking to you look how

simple this stuff is this is the

property for get URL when I'll do it in

real time but when you type in a get URL

it then runs the validate hook which

says can I access that get URL and tells

you no you cannot access are you sure

that's the right one you know and so

it's like you you are able to get this

like hand-holding through and it's

really a lot of fun but clean in terms

of programming right so you can walk

through it's like if it's empty don't do

anything tell the user what I'm doing

I'm checking the git remote because if

it just hangs and the user has no idea

what you're doing this is a tax exact

just means run this command get LS

remote just pings the get URL this

basically is just building a command

that runs on the command line if it was

not successful then show this message if

it was successful then just show that

message and that's it

so if you're building DevOps tools this

is really handy stuff services they


represent like your web server were

aware of your web server what type it is

what we need to do to start it what

config file needs to look like the

database all that stuff the way this

whole thing works is through a thing

called verify you run you insert a

server and you say verify server and the

code goes what am I supposed to do and

it looks up all these tasks and it says

okay I need to write a file I need to

change the permissions of this folder

all the stupid stuff that we don't care

about and a robot can do better it will

do for you and that's how eager works

now but it's in like all these drugs X

and she'll execs and it's just this big

mess of drush and functions and it's

extremely confusing now we have these

tasks or little tasks and it's like just

a list and when you saw that checkbox

that's where it's getting these streams

so it says start and then it prints that

message and then it runs the command and

then it just prunes done it just prints

done at the end with the timestamp or

you can send a failure message or all

this other stuff and that's basically

says like 99% of DevOps is like write a

file run a command so whether it's

you're launching kubernetes with this

thing or just running Apache it's like

write a file with some data in it and

want to come in and you get you get a

website or whatever service you're

running so yeah basically every service

you have a class that just says like

database service and you have these

methods for verify site platform and

server and so you do things when someone

calls verify site verify server and it

could be anything like I showed the git

clone one earlier this is the docker

build one so this is a docker web

service that'll actually build the

container for you automatically and

because it's using that this is all Robo

stuff which I'll get into in a minute

but it like fails gracefully and throws

beautiful exceptions in color it's

awesome stuff and it hides all of that

see this silent here rowboat ask docker

build right

what directory this is all Robo stuff

what tag is going to run

and you know what file it's gonna use

options but the silent part here is

universal for Robo tasks which will then

awesome yeah so if it's not verbose it

just hides it so if you to be it prints

out everything again everything in terms

of DevOps is built-in it's like yeah the

gulper grunt or whatever the task runner

is this is a whole new thing rush me

feels a bit of autonomy Robo all these

other tools are using Robo for all sorts

of different things it's got everything

in it we could possibly need for moving

files in doing ops type work so we're

not writing execs and running shell

scripts all the time we're just

leveraging these kind of elegant

object-oriented things to do stuff we're

using this framework it can be put into

a fire file this is all built by this

consolidation team which is like the

drush core maintainer is got together

and they're like all the stuff we wrote

for drush it's useful for other things

so like their libraries are used by

wordpress CLI and all these allotted

different tools so this this is the PHP

PHP Renaissance

Renaissance we're all talking about

right so these all use Robbo

specifically like we had BLT and all

these other tools once this thing works

we can include it in Drupal 8 itself and

just use the interaction to you generate

a form instead


okay terminals annoying right but we're

gonna make it easier so pro is my

shortcut for provision type Pro it tells

you all the commands if I type pro

status it tells me just my config but I

have no context yet so I'm just gonna go

ahead and start making things for a save

it tells me Oh context you could put all

these things in the command line

automatically - so when your if you're

using it for DevOps you can just write a

single line to save your context all at

once give it a name I tell it what type

I want to server these autocomplete

things are awesome I can just hit one

letter or I can hit up and down and it

Scrolls through so it's super easy use

defaults are built-in so I'm just like

hitting Enter is fine script users are

automatically detected because the

properties I like automatically looking

up they're currently running user

automatically looking at my home


this stuff is all just gonna be you

don't even know you're gonna need to

know not even gonna need to know what it

is I hit enter to save that config I

want to add a service yes I do that

server has a database

and another one HTTP we're gonna it's

basically again Universal so there's

gonna be libraries for like whatever

type of system we happen to be running

it on this one demo is running Apache

detects the web group automatically

whether it's W data or one of these

other ones this is gonna set up local

Mac OS built into patchy

there's no docker or p.m. that's gonna

happen here it saves your restart

command that's how I can write this

write it and restart it no I okay so now

that I have the context server master C

it saved everything I put in to those

inputs and now I can verify the server

will tell me if I actually do have a

webserver there I actually put a sleep

in there or one second so you can see

the to do and then it runs because it

goes to pass them but I'm if it's

interactive I'm gonna make it an option

but if I do - uh V for verbose look it a

lot more so it's like checking that

checking to see if I can create the

database you can always think this is

just a server I haven't added the

platform yet or anything like that so

this is just making sure okay I can see

a database I have root access I can

create privileges this is all you know

the goal of this is to create these

databases automatically okay so now I

need to add a platform I already

downloaded atrium so I can save a new

platform and we're going to write it

saved it to the Emma file I want to it

needs a web service so it's list now it

lists the server I just added because it

knows that server is providing web

service to me so I can say yep super


so it just says a little patchy config

platforms don't do much so let's add

another platform because it's fun to do


so that's the route is where I want my

Drupal files to go I'm gonna use a get

here all this time so like if this was

your Pantheon or whatever put it in here

it checks says is it really there it's

really there connected

do I want to save that yes I want to

specify the web server and now when I

call verify on this platform for the

when it does it the first time it'll

actually clone it for me it tells you

it's creating the folder if I hit B you

would see the output if you get cloned

so now we have two platforms

so each site what a site goes onto only

one platform so the site I'm creating

over like do I want to be an atrium yeah

or do I want it to be on the other

platforms I said yes

and then I give it a domain I already

have this URL dot local a computer is my


all redirects to one to seven localhost

feel free to use it I don't care about

the profile atrium or agar installs

automatically so it tracks when install

profile you want to use so it can fully

automate so you it'll actually run the

install profile and have everything done

for you so now it's saying okay it's a

site the site requires a database

service so which server would you like

to use to further your database I hit

server master this will be automatically

generated but it's not right now so I'm

just going to type it in so we can do it

later but it saves that data to the

contacts I can see it here all right so

all the different metadata what users

names needs site and this should greet

database create the virtual host file so

a couple things were mocked in there but

what a go does is actually write your

settings that PHP file so it's already

ready to go it's not already configured

but I should be able to know this

properly worked to load that URL


no no Nazi bugging I mean I we're gonna

check our metadata back ends here so

everything huh

Oh site down local my computer is just a

context name it means nothing the URI is

what matters so local about computers

always resolving to localhost no matter

what anything under local computer

results of low-cost but yeah anyway this

is this this is the I do this on Linux

this is the mag me yeah this yeah it's

the name of the context is the your I

because I can show you the come up here

so it all just goes to Petra configs

yes no it's the restart okay max have a

very strange thing you have to put a

dash K in the restart command thank you

Frank it's very strange

okay so the command is so you can see

these are all the gamma files it's tough

it's trying to restart with this which

works fine it says yeah it's fine

it doesn't actually reload the config

I've learned this through debugging this

is literally the built in Mac OS Apache

so it's very obscure and weird but if I

do this - Kay and then do the provision

to verify I'll do see that one is shown

that doesn't show up unless you use the

kay very confidence anyway right this is

very hard this is a very hard process

but like we just want to build Drupal

sites so if we can simplify it we can

make it easier for everybody and bring

more people on because getting on your

local it was really important and I hate

you know poor new people I spend like

three two weeks to have three months

figuring out how to run the localhost

and stuff like that so questions yeah

that's that that's next

like I'm haven't gotten there with 14x

but that's how eager your core works now

so you add multiple servers and only the

first one is the local one the other

servers are all somewhere else and they

only work if there's an SSH connection

and then it pushes the configs to that

server runs restart on that server

thanks back here so where is the file

saved John today any file or does it

read it right from that's the mo files

you know

default directory but you can change

that too so if you do this is the

console config file what any of these

can be overridden if you put it in that

gamma file I don't have one

right now this is where it's actually

saving the contexts it uses this is

inspired by terminus terminus has it has

a nifty class that loads and all this

config easily it automatically loads

them all and from environment variables

for example and all these other

interesting things so yeah everything is

reconfigurable from these files yeah

short question probably long answer are

you in a virtual environment with this

now this is running directly on Mac OS

Apache I pull it for this use case I

around my Linux machine I run dr.


the point is it's all comes down to

Apache anyway so it's abstracting it out

or doesn't it can work we can put it on

all these different use cases cuz like

there's always gonna be you're always

gonna want to run to your website and

different systems people are gonna have

different you know so if we can get it

as simple as possible I even want to use

I even have run PHP run server playing

plans in there to just to use PHP run

server and like SQL Lite which is just a

file you know and so if I can get that


you will need like zero extra

dependencies on a Mac laptop because you

already have PHP CLI and I found simple

tasks will break major people thing but

yeah you know like the point is like run

a website all the different ways we can

do it turns out are pretty simple

because we have all these different


so like instead of Apache it's just run

this command that's the whole thing

under verifies the server run execute

I'm trying to get this to run in the

background and like pull it later if

it's tricky but you know this is kind of

these aren't these tasks that could be

are totally arbitrary you can have these

classes and just - thanks thank you

Aegir's back-end is getting an overhaul after 10 years of service. We're developing a brand new Symfony console based CLI for all of your website management needs: Provision 4.x.

With the power of Symfony console we've completely re-written Provision to be as easy and clear as possible, and flexible enough to work anywhere: cloud or workstation.

Can this new CLI become the defacto dev-test-production website management tool? Is it wrong to write DevOps tools in PHP? Can we keep Aegir going for another 10 years?