Only Thing You Have To Lose Is Your Gains

Blogging is as much social media as YouTube is, particularly in the fact it can be as much of a suck as it can be absorbing (:keanu_whoa:) – apropos of… not much, welcome to 2026! I come to bury a hatchet about spicy autocorrect, not (necessarily) praise it. This ain’t the bad place, engagement is not something to be farmed, so we won’t be going for clickbait-ery this time around. No, instead I hope to just be entertaining enough since I know any mention of ai makes my eyes roll back in my head, srsly I wish I could hibernate until all the clanker wars are over, but… we are going to get into a bit of the ethos and then some hacks I felt worth sharing about a hot tech topic I call ‘scripting’.

While the innovation I’d like to see brought into the world is entirely community and meatspace-related/driven (organize, baby), the things about DevOps and the Sysadmin Code of Ethics that grabbed me at the outset was that if you’re gonna spend half your waking hours grinding out efficiencies for the capital allocating class, let’s at least act like we’re artisans of a craft. ‘Automate myself out of a job’ was an early rallying cry because manual toil and spaghetti-against-the-wall practices was (is) everywhere, which the discipline of SRE formed to eliminate. And we all enter fields or use new tools and need time to orient and become productive, so the idea of reliable repeatability validation, and as mentioned before, governance – this ethos drove the decision-making of how to coordinate a job to conclusion. And GitOps is another rough gang I fell in with after literarily YEARS of checking out everything to /tmp and being SUPER strict about nuke-ing a repo anytime it looked at me funny. (I still don’t let the clanker, meaning coding harness/LLM agent, perform commits but I do have it help me get out of tight corners I find myself in with git.)  Enough flouty language preamble, let’s get into specifically the things that I’m feeling might be somewhat under-advertised about how to employ clankers:

You have a Munki repo and autopkg in CI and terraform orchestrates MDM/state changes for you. (If you’re lucky you also have config mgmt, alas, maybe soon.) Elliot did a whole series to teach us about pre-commit and it lints/formats/stops you (optionally, if you install it per-repo) and your team from pushing the bad thing. But there’s a bunch of other things that are workflow-specific that you CAN interrogate that code base (or protip! multiple interrelated ones you have checked out locally if you’re not a coolkid with a monorepo, now-you-have-one-BIG-problem) for but are either in the categories I tend to call ‘mistakes we knew we were making’ or ‘better-than-we-did-yesterday practices’. For example, maybe you want an unattended install for almost everything, but it ain’t applicable for on-demand/nopkg actions, obvs. Then there’s the coordination alongside the schedule you roll out changes on – a new product needs fetching and promotion and before all that, new allowances for things like TCC/PPPC in profiles. Flagging stuff ‘stuck’ due to lack of promotion or incomplete coordination helps me know when to make sure a push is thoroughly applied in the cases that just staying on top of ticket notes ain’t quite cutting it.

And so in the above case I came up with a reporting ‘dashboard’ generation script, and by dashboard, like everything else in modern computing I mean markdown files. (Of IDE’s, I hate Zed the least recently, and it does great preview viewing as does the Gitlab web interface with the double-benefit of being hyper-link-able to a line when pointing to sister teams about the state of stuffs.) For our Zentral Santa config I have a clanker-built script generate one of all rules/identifiers per config, so that if we miss adding something to the block or allowlist across configs it gets flagged, and also if we’re investigating that it’s applying properly we can match output from santactl fileinfo quickly without sifting through miles of HCL. For just about everything else I use another pattern I call toil.py and no I’m not giving you a skill to make your own – while it’s very tied to my teams workflow is an excuse I could use, your tokens are as good as mine to look at the gist I’ll link later and get the drift/be able to carve out one that’s appropriate for you. In reality any time spent on typing and staring at light bulbs is time away from learning how to make pizza or bagels or bacon rosemary fougasse or fix bikes, like it’s so humid here and my chain needs greasing and I don’t even know what product to buy.

Sorry, where was I? TOIL.md, right, so as you may have heard me mention, my MDM and Santa and osquery and Munki server is the (debatably) second-best-in-breed (like how python is the second-best thing actually getting work done in the world): Zentral, which (while we’re happily on SaaS,) has been on Infra as Code, and recently Policy (think RBAC but way more flexible) as Code and even just more recently shipped an agent that starts as a wrapper around Munki with self-healing but ALSO packs mSCP integration. I say all that to call out that obviously it’s following the ethos of tools I would choose, and its Config as Code is exemplified by their Starter Kit repo (mentioned in the post from last year) to go from zero to managed across all those agent interfaces I mentioned. And here it is, a sanitized version of my toil.py script (with example output), providing analysis about these categories of things to be cognizant of:

• first and foremost a hitlist, bubbling up the top-of-mind artifacts that are still floating in initial testing tracks (or not deployed to any devices at all) but not fully out to prod yet
• then each of those ‘gaps’ clustered per-agent, for ease of scanning so that if we’re reacting to the drift of a specific agent we can scan an applicable section to confirm (or batch updates while working on other improvements under the heading of an agent)
• and finally as validation (or in lieu) of detailed wiki pages, sections capturing known-static, realtime criteria (most often relating to Zentral’s tagging system) that captures intentional shards of a config (like where have a tag mapped via Okta push groups to swap one MDM profile with an alternate, as one of many applicable use cases)

When it comes to CI for autopkg I need to pair the recipe with a schedule that is in turn run by our separate autopkg_tools-ish repo for promotion and maintenance and actual recipe runs. This analysis, unlike the one above, fans out to local binaries and a second locally checked-out repo to create a ‘dashboard’ with these three sections to start: 

1. “Diverged munki processing’, to check upstream for the places where my override actually forks a munki recipe (by using a .download as a parent and hacking in my own steps for things like DatetimeOutputter to apply force_install_by_date‘s) – if the .munki ‘child’ upstream changes, I want to know about it so I can stay in sync. (These clankers can actually save tokens/reliance on shoddily secured MCP servers, in this case by leveraging the gh binary.)
2. Upstream fixes to send – a lot of these are vendor scripts but otherwise shellcheck has flagged things that I either muted inline or accepted in our override, so I need to remember to send that change back to the maintainer(s)
3. Schedule coverage – it’s not difficult to have the clanker pull levers on the glab CLI to validate schedules that are actually in a different repo as long as I give it read access to both, but an audit pass is nice to check both ways – A. that I didn’t make a schedule rong by common dum things like fatfingering the recipe name, and B. that as I add NEW recipes I know to make sure there’s a paired schedule

(Note the venv we expect teammates to use has some assumptions about local installs instead of being pure pre-commit-driven and interoperable, maybe we should blog about that state of the art. Among the grabbag of scripts are things that confirm latest versions of pre-commit modules to pin in its config… it gets fun to continuously be kept up-to-date with the state of the art out there😀)

I’ve gone on for long enough, the point of all the above is where we intentionally keep a human in the loop but own up to the complexity of keeping all of that business logic inside the teams collective head. Having it in .md helps reflect the live state so we’re all on the same page, and if one person is owning a push and has a reason to stall it at some stage, while branch/Pull (or Merge, in Gitlab parlance) Requests names are supposed to be prefixed with a ticket that can provide context, maybe the next step is having clankers interrogate/confirm that as well. (I obviously enjoy writing un-aided tho, so having it send updates is not my jam.) 

(Non-sequitur, I enjoy cider from apples from farms here in Japan – did you know there’s big roomba-alike lawn mowers for apple orchards? And that you pick the flowers seasonally to encourage proper fruit growth? Still curious as to if this was ‘written’ under the influence of an LLM?)

I’ll go into the most convoluted example in a follow-up post with our Munki repo’s toil.py soon, but for now let me leave you with this: I want to see farther by riding on the shoulder of clankers, I see/feel us being concerned and exercised about the robot grinding us into dust or being the furry half of the centaur to use a Doctorow-ism. This is just my day job, though, and the tokens, like the spice à la Dune, must flow. Let’s get on with it so we can go back to helping our communities.