Site Audit 4: Track your Drupal sites into the 4th Dimension - DrupalCamp NJ

all right welcome this is uh this talks about an old module that uh I'm is now
my I'm the maintainer of now I guess uh it's a classic I don't know if you've
ever heard of it but it's about 10 years old um it's got but I mean that says a lot by
itself just today it has 555 sites yeah we're using it but you don't remember it's just a drush
module most of it so people have been using it on a dress I remember Pantheon uses them like all the sets so anyway
um there's a lot of teams that worked on it over the years you Pantheon's initial development and
just through some background right one point x is still where we're at just triple seven it was like you know
helpful but uh a lot of information the second version came along for Drupal 8. took
like a couple years but still drush uh still what it is and that's that's fine
you can output that and save it and then eventually three came along and they finally had a web interface so we
finally had this but also this where we could uh look at it and that was about
it though um it gave you this really nice looking thing if you had the bootstrap theme uh
it was it did its job and so it's not like something everybody clicks all the time so it just kind of was one of the
modules that just sat around for a while did what it did and it was good and some people used it
um but all of that is just static so you run the command it processes your site and then it's it prints it out right
then and there and that's all it does all it did so you know if you had your system set up and you ran prawn and say
did any ran other scripts to email it that's great but it was helpful you go to the status page you click you know
audit and you can see what's wrong um but again you'll you know someone will call you
that your site's down before you and then you go check it later so what we wanted to do with uh
with that vardot on things drop um they are not hiring me to help me build some stuff and so they like this module they
build a bunch of extensions to it as well checks like SEO and all sorts of other things
um but it was like we have a hundred websites we're supporting we need to keep track of them so I want to make a
dashboard boss said and I said yeah yeah that's awesome we've made a bunch of those let's make a new one and we were
trying to figure out what's the best way to really like talk through these things have these sites talk to each other and it just occurred to me like this old
site audit module would be pretty work pretty well so and it really was it didn't take very
long but I turned it into an entity so now you can save audit reports just like a node you hit
save save add report and it renders it and it data dump and saves it in there
so you can have timestamps and authors and uh even the save the site title and
the uuid so you can have a whole database of reports and see like where they came from so you
can have reports on this live site you can have reports on the dev site and see like what URLs actually it was running
on what site it actually was and then I put the uuid in there because then everything can change and it doesn't
matter I still know hey this is the same site even though it's a copy from here or here here right so once you start thinking about
running it in different environments like tests since it becomes really pretty interesting okay
um and once we had that ability we could do things like ad recurring savings so it's
very easy just Cron job now you just save one set your setting and it'll save a report every minute month hour
whatever it also this is coming up later but you can send a report now as well
that's why we wanted one site to track all the others
um and that's simple setting Drupal 8 9 10 is so easy you just type
dress generate and hit on it and there it is so I was like trust generate entity site audit report was done dress
generate rest endpoint done this is simple out of the box Drupal stuff you put this URL in there you get
a key and it'll start sending those reports to this other to another Google site and just saving the entities over
there and so when we made a site audit server module that basically can receive the reports and store them in the same
place it's reporting itself so in one place you could have all these Uris and all the site titles and all the
data from all of that stuff and that's like having all of these
all the checks so I didn't have any more slides because I could just dive in and look at these checks and things but
um yeah doesn't even use site audit module
Site Audit Module
classic yeah so it's like it's not a new module
I'm updating it so uh let's just like look at it yeah I
have so many questions shoot shoot them yeah yeah so it's got
the old thing in there still you know oh fast forward is great and unable though you know lots and lots of
recommendations yeah this is good because I need to go off slides because there's a lot to talk about now we have
more capabilities right because not everything here is like a big deal so a
side audit just kind of spits out a percentage point in a kind of arbitrary like are you good or bad
without you and it's kind of it can be overwhelming because it's like not always does it matter yeah you want your
caches in there but there's so much information in here that it's it's hard sometimes to even dig in and pull out
what you really care about you know um like how many blocks I have and stuff
but it's getting this is kind of buried into it so we this is like a call for Community as well because right now it's
just taking a data dump of whatever it was rendering and so some of it's machine readable but some of it's not
like I report here let's go back to the nice looking page um yeah so you see the main page and
then there's uh the saved reports page now and so you can just click on the
individual report and now see a little bit better just traditional field set system and now for example Chrome
that saves it when it ran it so it actually saves the three seconds ago as
a string and so we're like got to mix a bunch of tweaks to these classes but it's actually really simple classes for
all these different checks um can you make the screen bigger yeah
yeah so like this is just a dump of the Drupal status page but you know pretty useful you want to
see like the version and whether it's old or not and so we need to come up with a better way to
like pull it out and so I was what I'm really thinking about is making like an audit facts system where it's like you
would just you could just save it arbitrary thing that says like up to date yes or no or account users and then
because this is in views now so you can make a view but you can't access these individual properties
because it's just all dumped into one one thing so I'm just talking about like the future
but yeah like we need to pull out very specific problem items that should show like red flag or not instead of just a
general you know that 86 is stored as a field and that can be it is actually
yeah so you could do longevity over time to show that you're making criminal improvements yes so it's it's just
uh it's in here somewhere under data yeah so it took whatever it was rendering
before it's not each thing is slightly different so yeah it does actually store the percent actually right but that's actually good
right because yeah that way you can look yeah and so right now because this is not like normalized or whatever you
can't like it would take work to kind of pull that out but if there was a another
thing attached to reports that was just like a fact and you would have like uh cash 100 cash 100 cash 200 over time
with a time stamp and it's just another dribble entity and it kind of becomes like a storage for whatever you might
want to look at uh wouldn't be able to dump the whole
report out into like a Json file well that's the easiest thing to do that's what they did already they did that 10
years ago yeah it's basically render is Json or renders HTML yeah so that's what I did is I took the date renderers Json
and I just saved it yep so I'm really not even doing anything fancy this is already done the data structure was already there okay it just was rendering
into the theme function and then that was it so now it's saving that just as a dump in that one field
um and but I was already so now that there's an entity I have all these Hooks and so for our dashboard we were able to
hook into the audit being saved and actually like do things so we had a feel like in a fill in a
field for example um
soon too big and it's frozen uh so like I added a feel to the site
audit report itself that referenced something else and so it could be anything but we created a site node
actually and so we could have other things and Link that up to other stuff and so every time a site report comes in
it looks at the URL and it does stuff all right um I'm gonna open up another thing here
uh
one of the neat things it does also is it's like it it receives data back and
forth right so we have a custom module
that has a event I believe
uh there's an event for receiving a report
this is actually what I want to show I'll go I'll go into the there we go
okay that's working now
so I have to talk to Siri I actually drew it up on paper I didn't get a chance to put in the slide
the client site sends the remote the whole audit to the remote right there is now a hook to like
alter that payload actually so you can send whatever you want from a custom module say hey site uh this thing is not
working or whatever you can just do it site audit class too but if you want to up that pay or whatever so then when the
server receives it try to open it up that's that's an event
as well so the custom module on your server can receive the report and react to that to do stuff on top of the entity
things but then there's an alter hook for what you send back and that's what's interesting is so
the client site can say send report and that first time it gets back it's okay or 200 or an error you can have this
main site receive the audit and say yes or no and actually pass a string back and forth and so that's what we're doing with with
the the dashboard thing support dashboard is as a support widget
on the top and we're actually going to send back if their license is supported or not you know it's awkward does it all
sorts of other things do it uh on another one browser uh
I made this server separate so I can just so you don't every site doesn't have a server in it
um I should have started with the site
audit itself classes and stuff so you can get familiar yeah let's go there
control so it's really easy to just add arbitrary checks to your site so like if
there's this one corner of your app that always goes down for some reason you could just drop a plug-in
into Drupal 8 and if you're familiar with that it's just to create this class and it's called a check or a checklist
and you can just run any arbitrary PHP to say pass or fail so like this is just
doing the size of the files content Fields counting your Fields
but it's neat because you can even like return like a display of some kind like a steam code so that the audit report
has whatever you want in it you know you want to render a view or something you could put it in there to print out to
send us some boss or whatever and so yeah you can even do the warnings
and things like that so uh one of the little changes we made is like you had to check checks these are
called checks and used to be called a report but now they're saving reports that I changed the code they call the checklist
so like a checklist doesn't do much but it's just a category but
um you can extend these to say like add a checklist and then only run that checklist to say these are my most
required things you know um
so like we just released the public module that what we can put into like every single site even public Drupal
sites going in the the uh
and so they we I mean they just wrote a ton of other additional checks that we can all share right so
alternative caches and then Commerce stuff you know like Devil or DB log like
so much stuff if the Google analytics is working uh there's already like security there's
a Security review module
this integrates with the checklist as well so it provides a checklist to site audit
Site Audit Checklist
I just changed the class name so like we could leave I don't know I guess we could leave it there but I actually
posted a little issue and I was asking like it's it's a quick fix like and now even more there's like Drupal Rector and
other tools that'll upgrade so I would provide like a little I didn't even do it I can submit pull requests now easily
so I would go ahead and do that and it's just because like a report is just this was just like a
group of checks it had no other functionality whatsoever and so from like this information architecture
perspective calling the state identity a report just felt right you know you're making a report a reporter reporter so
like there was no other functionality of the of the checklist as other than being a list of checks so
um but again it's just a class change um from site audit report based to
checklist space so I I want to help like we're going to be using all of them so I'll probably be the one to just go
ahead and do flip the switch but again they've actually evolved very simple so most of these modules they don't go
through a bunch of changes over time right they build their little tests when they leave it so
um yeah I know it's kind of like jumping all over but that's kind of how the module works you know things are other
things that'll be cool if we change the checklist thing is we can make checklists an ending even because I want
to know I want to see make a thing called like a requirements list where these five checks don't pass like you
fail Ci or whatever right so that's kind of our arbitrary thing sometimes as well
because there's just a lot of people we'll get more checks if you want
um anybody else want to like questions I'm going to do an interactive
session here because it's not a long yeah can you show that dashboard yeah I
was just pulling it up actually here uh yeah I mean so it's just like a list of
Site Audit Report
customers we've all maybe done it before and so it's got the site audit
list built in or this is this site if I go to save
reports it's receiving them from different even PR sites and other other customer sites but it's interesting you
can catch things like drush is not configured for the UI it's default you know so that's not that's pain pain for
people to constantly being set and you don't know where your site's being calling Josh from so you can add a check
that says this is not default or this or that um an interesting too you can see the
site titles change over time as well um and this this is all the free open
source stuff and so but what's interesting here is these are extra fields we added right and so the report
uh it doesn't this it doesn't do this by itself actually now but it just stored all the reports but we added a field to
associate the report with another thing called a site so that we can have
multiple sites and list the reports just from that so this is the standard View right where we made we're filtering only
the reports for this site Okay so we've got a list of sites and they all get
uh yeah and so I can even I had one earlier
where I was oh yeah this is from a PR so my bad
so yeah so yeah you can see how the URL changed because it's like we clone that site so
it's still hanging to the same server and Crown's still running so it's actually a good good thing interesting thing to see like this could be used to
audit a developers work for example like put the site on a check in there that they didn't
turn something on wrong or whatever
yeah so but like I said there's there's a lot and some of them are like oh this is
you know check every single view that's not query catch no no you know so I would I think this is kind
of a foundation to implement some like the real best practices kind of checks for people you know because it is kind
of broad right now um but I really want to have like use this for CI stuff too like you have to
pass this if you're gonna but yeah have you considered doing well we
saw percentages again it's like a design feature right I don't know it came it was like this when
I got here but uh I think that's when you'll be able to now that the data is kind of in there you can kind of play
with it and do different things but I kind of agree a percentage is really kind of misleading yeah I'd rather just see a count like 15 out of 16 pass or
something um but then again yeah this is someone has to be the threshold though yeah but
he does have a percentage well yeah actually that's the thing I add on the right so you can see kind of a how many
warnings so but again yeah we need some design help on this for sure I was going to suggest what being able to dip between
Bingo see I was almost tempted to say what if we save each one as a revision because then we instantly built in you
have the dip for your own Drupal because they're entities so yeah that's that would be totally almost possible because
it's still just a it's a Json dump right now so it's still a little tricky but help
you know to join in and it's easy to click edit on contribute modules now and merge stuff there's even like built-in
Ides on drupal.org now you don't even have to clone it or any of that stuff so if you're not entrep.org yet and you
think you just check it out uh if you go to the project page we can show you right now actually
before you leave that page um because there wasn't a lot of hands that went up about people that are actively using
this yeah I one thing I always point out to clients that are questionable whether or not we introduced this to a project
um I'll be like if you're ever doing like a major re-platform it's always good to know the content on your site so
you were showing like the block list and the number of times used yeah how many times they're used so if
there's like one or zero you're just like do not write a migration for this or yeah it's definitely useful
if you're not using a field or a low number and it's got the weight of needing a custom work into a module
attached to it and that's the only place you're using it that gives you an opportunity to like pin the site down before you jump into the migration
yeah that's all great for the recording uh they say he was saying how this is very useful when you're like upgrading a
site or moving from one way to the other because you can tell what not what's in there but yeah like for example
duplicate titles it actually does scan and see if you have any duplicate titles you know against all nodes so whatever
useful that can be um but again these are all simple classes so once you get the module you can kind of open them up and and copy
and paste most of them you don't need to be Advanced programmer or anything uh to do that
uh yeah what about like different
Environments
environments for the same website all those just different websites and that's all you could tell that by the URL so
the uid is the same but hopefully if you're running thrush with the URI attached or whatever you're
getting you can the URL uh the URLs Exchange so that I thought I had it I've been
working on some bugs oh yeah wrong tab uh this one has
yeah so these are actually clones of each other right but they're coming from a different URLs so
um that's part of what we're thinking about too is like I wanna be able to pass the environment variable
or something in there and like uh yeah but for now the main
um fields we have we store are the uuid and the URI
and a label um and but
again coming up with a interesting way to add other fields like that like environment
type or string or you know something we can do now with Fields but
uh because there's so much data you want to play with trying to come up with an interesting way to say to make like the
very like in this view allow me to add like exact you know thing from one of
the checks but yeah that's like data science questions
mongodb stuff no whiskey well move that by customers yeah
everyone so exactly yeah so yeah but right now it works well to just it's an entity hook to just hook any of this
insert field there and it's really uh Triple H is actually pretty easy to do that too
let's see I think it's this one yeah so it's just the entity pre-save
uh looks up another report if it already exists so it matches up against the site
that already has this reference field um
and yeah AAA if you're never caught in the field thing or you're like oh I have to worry about the there's a Duo one or
you can just reach the properties now it's really easy to just save this stuff
all that and uh saves a message
so fun to play with um
that's just the helper functions and stuff so yeah
um yeah any other questions
important project manager not really a developer like there's all these things
that comes out of the box you've got so much information is there anything yeah
actually it's less than the full amount you can yes you can um one of the original settings
is to only run some of them uh but again it's like that's all cash I
want to if you want to show something useful to like a you know stakeholders or something yeah you're going to want
to kind of like piece these together but an interesting thing is you it
started like go back into code or whatever but because it's so simple you could piece together make one custom
checklist out of all the other checks and just like inherit classes and stuff so it wouldn't be hard to kind of make a
mat make a meta uh thing in code right now but I keep thinking about how it
would be awesome was to make that in a web interface anyway we make no types why not just say hey give me this report
type and in this report type it has these three yeah
yeah certain point you don't need it custom should just be
real be reusable across you know a pretty wide swap in the community it's
already this would be so easy let's do it tomorrow I'm going to be at the kind of trip thing but yeah like there's already a site audit thing and a managed
field thing so then it would just be site audit type or report type and then each that each one could say hey this is
the cash one this is developer one this is the the customer one so it really wouldn't be for a single content type
that too all that stuff yeah well at the report thing I think that we're having a report type would be
great because then you would say like this report type is absolute requirements and then you hit add create
report which type you know and then you save that View mode is cool too but it's a lot of
data too you only want I mean I want to store it up sometimes I do want to store a lot like or get it up to date a lot
rather but I don't want to store a giant blob every time so why wouldn't you store as much data as possible because
the site will live forever I don't disagree but it's also
I deal with a lot of very large sites because they don't care about stuff like that and they're like oh I watch Dog Tales two gigs who cares so in terms of
site store design it kind of live forever right so it's up to you it's easily configurable send every minute
but if we had a sometimes you like you want to check if the site's actually online every minute but you don't really
care if the disc is full every 30 you know five minutes or whatever those things that only matter once an hour so
if it was configurable like that then you could say this should run this check once a minute run this check once a day
I guess it depends upon if you have a bunch of other monitoring tools and you're just looking at this one to watch
right configuration change your configuration change isn't going to
be every minute right right yeah it's always it's a balance it's like a one date you know once per day check or once
per hour or something then you it's not like you're gonna have a yeah
an endless date yeah and it's you know any data man you're dealing with that with every other table you know like you
can uh you can back it up you can clear it out but um
yeah so your attach a little extra feels to those reports and you're sending them
back to the server uh you know uh does that mean that um you could add like a
taxonomy field any field at all yeah and then it would be
yeah it just sends everything that's what's nice about the data triple it's already got serialized and
just sends the whole thing over that's cool yeah
all right um yeah and I like I'll be at concert days tomorrow and
um Frontline all the time on the slack and all that
and yeah like I mean I can show you it's really really quite easy if you haven't been on Triple at work a lot uh
all the stuff is there's links to the source code and things but if you like find something wrong with it and there's
90 open issues by the way if you feel like you know
uh so for example yeah here can I rename this please like connect it so I just
describe all of it and then I just did it and so it's in the fork already and you can it's already uh
I don't think and I know I've been yeah
yeah like this one actually is fixed let's get an example
just click create issue for it's my worker that's my co-worker excellent
Moretta yeah in recent too it's like 29 days ago so people are still this is uh
but yeah this actually look at that it creates a whole branch uh and then you can fix it and click
open merge requests and then we can all look at your code and stuff and I think you click that Branch it
goes to the repo yeah so even it makes a whole a separate repo doesn't just to make a branch this is your own personal
copy of the module and yeah so you can really do whatever
you whatever you want um it's like I thought there's some read me
change I can't make here uh obviously it web IDE actually
that's what I want to show you this is if you haven't seen this stuff
it's phenomenal look at that [Music]
full um
so yeah uh during the community I think that's
real quick where's the main documentation .org project site audit
I'm looking at that I just didn't see it either I don't know how
again like I just started maintaining myself uh but that's what this is actually good
for you can go to gitlab so there's the home page they don't have a link to documentation oh there's a link to this
readme
module and I wanted to create a little test where would I look to see where to do that yeah that's well that's almost
like a dribble.org project question but you have a contrib module already you said yeah like so if I wanted to just be
like hey you only use this for migrate you don't need to leave it enabled and pop that report
oh yeah yeah well uh you mean like out of feature of the
core module here yeah it would so it would all be through this you can I would create an issue first to explain
what you want to do you can go to the uh
go to the page again and here on the right all issues and
yeah there's not really good you have to go to open it first that's a good usability note uh you clue in there and
then you can click create new issue and even if it's feature requests whatever
um yeah yeah uh I can actually control this so I can
even add documentation pages and things that'll be great yeah
they give you a template here even if it's a bug or whatever but um
I would do that and then where's that readment here here
yeah see so there you go how to use pipe it to HTML file
so hopefully yeah now we don't have to do that anymore another feature I want to add is like email sending so if you
want to get some picture points come on in help us send the email when we hit
the button instead it's all set up to do that I just haven't had a chance to write the code
uh yeah that's my presentation it's a shorty and if nobody else has any questions and
we'll uh give you time back