all right so John is gonna do a talk on
provision for dot X a developer John is
the man the myth the legend and that is
his intro no John pew is an independent
drupal consultant and code artist it's
kind of like a sandwich artist from
subways that were that's good I like
that
his company think drop is focused on
business app design DevOps support and
content strategy in New York City he is
an agar maintainer for those you don't
know John can tell you all about ager
and he created devlop and open source
Drupal development platform
can you tell you all about that - I'm
gonna stall real quick while John gets
his computer all set up so this building
was built in 1930 all right you're good
Meet Jon
yes all right John everybody John pew
wow thanks for coming out it's really
cold this is a big crowd I wasn't
expecting okay this is gonna be this is
you know this isn't really building
Drupal huh Mike Symon - yo thanks Frank
like you said this is me the point of
Why Provision
this thing in the point of all the
things that I work on are just trying to
make things simpler it's very very hard
to launch websites did you even if you
know how to do it it's still a thing you
know you may have your scripts ready but
there's plenty of documentation on how
to do it though you can go to Drupal org
and find out right away and learn really
quickly how easy it is to get drupal up
and running there's only these four
steps and then that one's too long you
can read the quick install which is
really short and you can spend you know
it's easy stuff like he just come on
everybody should know this stuff and no
we want to know how to click and build a
website and forget about almost DevOps
stuff right so like I said the problem
is running websites in your own computer
is too hard no matter what provisioner
you use you still have to manage your
own metadata provisioner means like
ansible or chef
for puppet or any of these things you
DevOps people in SystemC will tell you
that they use to provision their servers
like they're still storing somewhere
what git repo am i pulling from where
should I go all that stuff so they still
have to manage that somehow and most of
these provisioning tools are written in
a languages other than PHP which is okay
but there's way too many ways to render
people you can tell from my very
scientific graph that I created here but
it's true go look up how many things
that ways there are to launch Drupal so
History
in 2007 this started out as a historic
talk right I said oh 10 years of hosting
Drupal because this Egger thing just hit
10 years of commits no one really knows
when it started but the first actual
recorded commit was on December 7 2007
and you know some people were like man
we want to be able to host Drupal really
easy because this Drupal thing is
amazing even 10 years ago it was really
impressive and they were like what if we
could use Drupal to build it's to manage
our Drupal sites and and they did it and
they come created this company called
bright which you can only find on the
archive if you know the domain and they
look for it so I did but they were
really like the earliest kind of Drupal
as a sass like hosting service trying to
automate things and you know who knows
what happened one thing led to another
they were required and transferred to
other people but the point is the source
code for that platform they both came
from open source just like Drupal became
open source and everyone started using
it this thing is just Drupal modules and
they open source that so that we could
click and say add a site type in the
domain name and it runs all the scripts
in the backend needed to actually turn
the site on this is the UI and it's been
the UI since about 2007 hostmaster is a
User Interface
component of that
it's a Drupal distribution so all of
that stuff is Drupal code that you saw
it's and Drupal nodes and fields and
templates and all of those things so
it's really familiar to you and you can
add modules to it to do all the extra
things that Drupal can do Hosting is the
name of the module is Amit self and
provision is the name of these Drudge
commands that are on the backend you may
not know what Russia is so that's a
shell tool for drupal it's a drupal
shell and you can type in brush cache
clear for example and you don't have to
click through all the pages to hit the
button you just type in oshkosh clear it
does all the things
so they extended that to make commands
like provision verify and version
install so that it would actually spin
up all the configuration needed to run
your site Apache configs and database
configs and created database
automatically set all the files
everything in that instructions list a
grid's does automatically for you every
time a little bit of a history graph so
the commits do go back that far
it's intense sometimes digging there
most of mine are like all at the end you
can see the little spike here because
I'm doing crazy on this new project this
is the original commit not terribly
interesting there's only 11 files and
1,300 and whines this is the provision
Inc file as it stands today so get blame
on github is really handy tool if
anybody has an old project very useful
to find that blame button and you can
figure out line by line who did what and
so you can see ten years ago six years
ago ten years ago really really legacy
stuff and that's how it worked in 2007
Whats next
put all these things together and they
were able to build the Drupal hosting
company and also release it open source
so in 2018 what are we gonna do we do
the same thing right now basically it's
basically the same packages it's Drupal
7 front end it's add rush package it
some purple modules it's it's it's
really pretty it's getting old and
showing us age I'm not gonna go through
every detail but like it's got a long a
storied history communities survive
somehow
throughout all these things it's older
than Drupal 6 so we're we are still
doing things with it right so what's
next is that the system needs an
overhaul there clearly there's a lot of
smart people involved and they have many
different ideas on how it should go
Roger what it should do so the team
talked together and figured out that if
we break it up into segments segments we
can kind of tackle things elmo herman
van rank is kind of in the current
branch maintainer so he puts up the
releases of ager 3 which is stable right
now for hosting tons of Drupal sites
NASA that's like 500 Drupal sites
running on this I took over what I'm
calling the four point X which is what
I'm working on now we're going to show
and then Christopher Gervase is working
on a totally brand-new next-generation
thing that has no no comment upstream at
all so
Eggar we're calling five for necks and
so we broke it up like this so we can
keep developing the stable version work
on kind of the current gen next-gen
thing on me okay before and then Chris
is looking at the long term with five so
yeah it is a little confusing but this
is all work in progress and we're making
it trying to get through this quick as
possible so yeah like I said Hager for
may or may not be a thing I'm working on
provision for I care most right now
about the back end this command-line
interface which isn't just for hosting
under cloud servers but this should be
easy enough to run on your on your Mac
laptop so you can I'm trying to make
this thing easy up so you're brand-new
to Drupal Oh designer whatever you don't
know how to program you can just call
these little commands and it'll spin up
your Drupal site on any computer once
that's stable and works there's a lot of
features that the main eager thing uses
because it's a huge multi site hosting
machine then we can kind of think about
bringing it bringing the eager front-end
into four point next fully if we do get
there I'm gonna bring in all these extra
tools that we built with like dev shop
and this CI stuff and Drupal that we've
brought all into the bring it all into
the native code base again details don't
really matter but this is way down the
road you know I'm working on focusing on
on the on the backend CLI at first but
hopefully I'm have a little running web
site Drupal on my Mac right now so in
terms of working when VP of turning a
Drupal site on on your laptop this is
gonna be something I put out as like an
alpha in the next few months
I can explain before with five it's a
totally new new rewrite that he's doing
I'm not involved in that five point X
but you can check it out but you really
have to know a lot about it because it's
I haven't even run it yet so it's
Command Line Support
intense stuff but with four point X and
we're gonna back it up a bit
in September rush announced they're not
going to support things the way they
used to support them am I gonna get into
details but basically the command line
thing is like becoming a module you want
to put it right in there with your site
so it's not this external thing so we
realized the team realized okay we got
Total Rewrite
it's finally due time for total rewrite
it's been a long time coming
I've personally built a lot of Symphony
console apps there's a lot of other
people that have built sibling console
apps everything here from composer and
all these things that
once you're in Drupal long enough you
care about are all built on the same
components there are underlying they're
all symphony which is what Drupal 8 is
powered by so there's a million examples
of how to build a really powerful CLI
tool that's really easy to use so
basically I've been dreaming of doing
this for a long time and the opportunity
arose so if I'm only like sitting down
coding it all the details we're going to
store the desktop in yellow files once
we get the commands working well enough
we can put it in there with the the
front-end hosting system but primarily I
needed it to be simple and easy to use
and complete and to do everything for
you like the dress commands were not
easy to use at all you need to know
everything ahead of time basically no
Console
one used the backend command line for
Haggar I would wager they said it was it
was there so you could use it but no one
actually did because it just was not
easy to use so how do we make it easy to
use clean and simple console makes it
things colorful concise not cluttered
you just type in provision and you see
the list of things that you're able to
do ads so it works with context you save
like the site it represents the platform
represents the site you're working on
when you type in save if you don't
specify upfront it asks you for each
question and generates the defaults if
possible like I'm in this folder so if I
just I could just hit enter and it will
use that folder for the path to my
Drupal site I'm trying to minimize the
keystrokes that everyone has to make
we're gonna do a live a live type
through thing here but like like we
don't just doing complex things down to
as simple as possible when you're using
it right in the terminal right so
writing the web config green the data
base something fails it just throws an X
we're not gonna like spit out everything
unless you ask for it being verbose this
is what a successful verify it looks
like so like all those instructions
right create database and your patchy
and all that we're just running them as
tasks and it interactively shows the
tasks checks it if it runs successfully
puts an X if it fails this thing has to
be easy to develop Aker was really hard
to contribute to you know this is just
background but the way it works is it
you're storing things like you
it represents your local computer you
know it knows it has a patching in a
database
the site is URLs you can put as many
sites on that server and platform as you
can the platform represents the codebase
all this stuff is now stored in these
little simple yamo files we don't you
don't write these like when it's asking
you those questions you're answering
them and it writes it for you
automatically but you can go in there
and read it and see okay that's how my
thing is configured properties like
route that's where are my drupal files
locating the make file is like the thing
that is used to build your site with all
the modules in it I get URL is like
where it's going to clone from so if you
put get URL in there and you verify this
platform it'll automatically clone it
into that root alright so you don't do
any of these manual steps in terms of
programming interactively it's like
tells you this this is this green down
here is what you see this is the default
if you hit enter that's stored and it's
basically I was able to because it's
symphony and all these things that I'm
learning about like chain methods I can
just create this very simple ok platform
has a route that the description is this
the default value is the current folder
you're in whether or not it's required
and then a special function of like
validate that again I'm going over the
heads a lot of people because this is
not stuff but all you developers out
there this is how easy this is gonna be
you know I'm talking to you look how
simple this stuff is this is the
property for get URL when I'll do it in
real time but when you type in a get URL
it then runs the validate hook which
says can I access that get URL and tells
you no you cannot access are you sure
that's the right one you know and so
it's like you you are able to get this
like hand-holding through and it's
really a lot of fun but clean in terms
of programming right so you can walk
through it's like if it's empty don't do
anything tell the user what I'm doing
I'm checking the git remote because if
it just hangs and the user has no idea
what you're doing this is a tax exact
just means run this command get LS
remote just pings the get URL this
basically is just building a command
that runs on the command line if it was
not successful then show this message if
it was successful then just show that
message and that's it
so if you're building DevOps tools this
is really handy stuff services they
Services
represent like your web server were
aware of your web server what type it is
what we need to do to start it what
config file needs to look like the
database all that stuff the way this
whole thing works is through a thing
called verify you run you insert a
server and you say verify server and the
code goes what am I supposed to do and
it looks up all these tasks and it says
okay I need to write a file I need to
change the permissions of this folder
all the stupid stuff that we don't care
about and a robot can do better it will
do for you and that's how eager works
now but it's in like all these drugs X
and she'll execs and it's just this big
mess of drush and functions and it's
extremely confusing now we have these
tasks or little tasks and it's like just
a list and when you saw that checkbox
that's where it's getting these streams
so it says start and then it prints that
message and then it runs the command and
then it just prunes done it just prints
done at the end with the timestamp or
you can send a failure message or all
this other stuff and that's basically
says like 99% of DevOps is like write a
file run a command so whether it's
you're launching kubernetes with this
thing or just running Apache it's like
write a file with some data in it and
want to come in and you get you get a
website or whatever service you're
running so yeah basically every service
you have a class that just says like
database service and you have these
methods for verify site platform and
server and so you do things when someone
calls verify site verify server and it
could be anything like I showed the git
clone one earlier this is the docker
build one so this is a docker web
service that'll actually build the
container for you automatically and
because it's using that this is all Robo
stuff which I'll get into in a minute
but it like fails gracefully and throws
beautiful exceptions in color it's
awesome stuff and it hides all of that
see this silent here rowboat ask docker
build right
what directory this is all Robo stuff
what tag is going to run
and you know what file it's gonna use
options but the silent part here is
universal for Robo tasks which will then
awesome yeah so if it's not verbose it
just hides it so if you to be it prints
out everything again everything in terms
of DevOps is built-in it's like yeah the
gulper grunt or whatever the task runner
is this is a whole new thing rush me
feels a bit of autonomy Robo all these
other tools are using Robo for all sorts
of different things it's got everything
in it we could possibly need for moving
files in doing ops type work so we're
not writing execs and running shell
scripts all the time we're just
leveraging these kind of elegant
object-oriented things to do stuff we're
using this framework it can be put into
a fire file this is all built by this
consolidation team which is like the
drush core maintainer is got together
and they're like all the stuff we wrote
for drush it's useful for other things
so like their libraries are used by
wordpress CLI and all these allotted
different tools so this this is the PHP
PHP Renaissance
Renaissance we're all talking about
right so these all use Robbo
specifically like we had BLT and all
these other tools once this thing works
we can include it in Drupal 8 itself and
just use the interaction to you generate
a form instead
Pro
okay terminals annoying right but we're
gonna make it easier so pro is my
shortcut for provision type Pro it tells
you all the commands if I type pro
status it tells me just my config but I
have no context yet so I'm just gonna go
ahead and start making things for a save
it tells me Oh context you could put all
these things in the command line
automatically - so when your if you're
using it for DevOps you can just write a
single line to save your context all at
once give it a name I tell it what type
I want to server these autocomplete
things are awesome I can just hit one
letter or I can hit up and down and it
Scrolls through so it's super easy use
defaults are built-in so I'm just like
hitting Enter is fine script users are
automatically detected because the
properties I like automatically looking
up they're currently running user
automatically looking at my home
directory
this stuff is all just gonna be you
don't even know you're gonna need to
know not even gonna need to know what it
is I hit enter to save that config I
want to add a service yes I do that
server has a database
and another one HTTP we're gonna it's
basically again Universal so there's
gonna be libraries for like whatever
type of system we happen to be running
it on this one demo is running Apache
detects the web group automatically
whether it's W data or one of these
other ones this is gonna set up local
Mac OS built into patchy
there's no docker or p.m. that's gonna
happen here it saves your restart
command that's how I can write this
write it and restart it no I okay so now
that I have the context server master C
it saved everything I put in to those
inputs and now I can verify the server
will tell me if I actually do have a
webserver there I actually put a sleep
in there or one second so you can see
the to do and then it runs because it
goes to pass them but I'm if it's
interactive I'm gonna make it an option
but if I do - uh V for verbose look it a
lot more so it's like checking that
checking to see if I can create the
database you can always think this is
just a server I haven't added the
platform yet or anything like that so
this is just making sure okay I can see
a database I have root access I can
create privileges this is all you know
the goal of this is to create these
databases automatically okay so now I
need to add a platform I already
downloaded atrium so I can save a new
platform and we're going to write it
saved it to the Emma file I want to it
needs a web service so it's list now it
lists the server I just added because it
knows that server is providing web
service to me so I can say yep super
master
so it just says a little patchy config
platforms don't do much so let's add
another platform because it's fun to do
this
so that's the route is where I want my
Drupal files to go I'm gonna use a get
here all this time so like if this was
your Pantheon or whatever put it in here
it checks says is it really there it's
really there connected
do I want to save that yes I want to
specify the web server and now when I
call verify on this platform for the
when it does it the first time it'll
actually clone it for me it tells you
it's creating the folder if I hit B you
would see the output if you get cloned
so now we have two platforms
so each site what a site goes onto only
one platform so the site I'm creating
over like do I want to be an atrium yeah
or do I want it to be on the other
platforms I said yes
and then I give it a domain I already
have this URL dot local a computer is my
domain
all redirects to one to seven localhost
feel free to use it I don't care about
the profile atrium or agar installs
automatically so it tracks when install
profile you want to use so it can fully
automate so you it'll actually run the
install profile and have everything done
for you so now it's saying okay it's a
site the site requires a database
service so which server would you like
to use to further your database I hit
server master this will be automatically
generated but it's not right now so I'm
just going to type it in so we can do it
later but it saves that data to the
contacts I can see it here all right so
all the different metadata what users
names needs site and this should greet
database create the virtual host file so
a couple things were mocked in there but
what a go does is actually write your
settings that PHP file so it's already
ready to go it's not already configured
but I should be able to know this
properly worked to load that URL
Metadata
no no Nazi bugging I mean I we're gonna
check our metadata back ends here so
everything huh
Oh site down local my computer is just a
context name it means nothing the URI is
what matters so local about computers
always resolving to localhost no matter
what anything under local computer
results of low-cost but yeah anyway this
is this this is the I do this on Linux
this is the mag me yeah this yeah it's
the name of the context is the your I
because I can show you the come up here
so it all just goes to Petra configs
yes no it's the restart okay max have a
very strange thing you have to put a
dash K in the restart command thank you
Frank it's very strange
okay so the command is so you can see
these are all the gamma files it's tough
it's trying to restart with this which
works fine it says yeah it's fine
it doesn't actually reload the config
I've learned this through debugging this
is literally the built in Mac OS Apache
so it's very obscure and weird but if I
do this - Kay and then do the provision
to verify I'll do see that one is shown
that doesn't show up unless you use the
kay very confidence anyway right this is
very hard this is a very hard process
but like we just want to build Drupal
sites so if we can simplify it we can
make it easier for everybody and bring
more people on because getting on your
local it was really important and I hate
you know poor new people I spend like
three two weeks to have three months
figuring out how to run the localhost
and stuff like that so questions yeah
that's that that's next
like I'm haven't gotten there with 14x
but that's how eager your core works now
so you add multiple servers and only the
first one is the local one the other
servers are all somewhere else and they
only work if there's an SSH connection
and then it pushes the configs to that
server runs restart on that server
thanks back here so where is the file
saved John today any file or does it
read it right from that's the mo files
you know
default directory but you can change
that too so if you do this is the
console config file what any of these
can be overridden if you put it in that
gamma file I don't have one
right now this is where it's actually
saving the contexts it uses this is
inspired by terminus terminus has it has
a nifty class that loads and all this
config easily it automatically loads
them all and from environment variables
for example and all these other
interesting things so yeah everything is
reconfigurable from these files yeah
short question probably long answer are
you in a virtual environment with this
now this is running directly on Mac OS
Apache I pull it for this use case I
around my Linux machine I run dr.
locally
the point is it's all comes down to
Apache anyway so it's abstracting it out
or doesn't it can work we can put it on
all these different use cases cuz like
there's always gonna be you're always
gonna want to run to your website and
different systems people are gonna have
different you know so if we can get it
as simple as possible I even want to use
I even have run PHP run server playing
plans in there to just to use PHP run
server and like SQL Lite which is just a
file you know and so if I can get that
working
you will need like zero extra
dependencies on a Mac laptop because you
already have PHP CLI and I found simple
tasks will break major people thing but
yeah you know like the point is like run
a website all the different ways we can
do it turns out are pretty simple
because we have all these different
classes
so like instead of Apache it's just run
this command that's the whole thing
under verifies the server run execute
I'm trying to get this to run in the
background and like pull it later if
it's tricky but you know this is kind of
these aren't these tasks that could be
are totally arbitrary you can have these
classes and just - thanks thank you
Aegir's back-end is getting an overhaul after 10 years of service. We're developing a brand new Symfony console based CLI for all of your website management needs: Provision 4.x.
With the power of Symfony console we've completely re-written Provision to be as easy and clear as possible, and flexible enough to work anywhere: cloud or workstation.
Can this new CLI become the defacto dev-test-production website management tool? Is it wrong to write DevOps tools in PHP? Can we keep Aegir going for another 10 years?