Odds and Ends,Puppet March 5, 2018 at 6:26 am

Stuck in the past? Your time may get warped, too

If the recent kerfluffle among autopkg folks (over GitHub shutting off TLS less-than 1.2) is any indication, a lot of people are still on 10.12.6. If I may editorialize for a second, this is probably contributed to by Apple’s less-than-ideal planning, execution, and communication around the High Sierra release.

What makes me come out of an unintentional blogging sabbatical to talk about 10.12? Well, a pretty widespread issue we saw is time went out of sync all of a sudden recently, across a bunch of networks and even with a known-good image. NTP as a protocol is supposed to be lossy and expensive, so it should be resilient to failure to the point that you may not notice it skewing for quite some time, and querying it even shows a ‘resolution’ or tolerance that is measured across the various command line tools in 5+ digits/decimal places. Some implementations can’t even correct themselves if you’re off by greater than 15 minutes (or 1000 seconds). Apple has had its ups and downs, from the only other time a silent patch was pushed (previous to 10.13.2’s passwordless-root bug) to treating it like DNS resolution and rewriting its mechanisms every couple of releases. (Remember lookupd? AFP548 alumni have blogged about DNS confusion for quite some time.) I was particularly flummoxed when this issue cropped up again because it had previously occurred for all of the signage iPads in one of our offices. You could directly reproduce the issue by trying ntpq -p time.euro.apple.com and get time-ios.g.aaplimg.com: timed out, nothing received. Nothing was being explicitly blocked or… delayed at the firewall, but in that instance Apple’s NTP servers were not able to send a response back to the device due to an issue with how the wireless LAN controller was configured.


In this case, however, we were seeing ntpq: read: Connection refused, as if the process wasn’t running. But a trip to launchctl as sudo would disagree, telling you that the job was loaded/running, so… what gives? There are a few stackoverflows that will tell you about the mindblowingly deep new-math Apple’s been using to just win so hard, dunking on us with the swiftness by never meeting a good-enough that it couldn’t rewrite, but none of it seemed applicable anymore – ain’t no pacemaker in /usr/libexec on my systems, and knowing that timed is a completely new animal as of 10.13 anyway means I wasn’t too hot on the idea of learning older stuff now.

Luckily Mr. Oakley wrote this post, which pointed me to the /usr/libexec/ntpd-wrapper… shell script (:jackie:) which Sierra still relies on. Inspect that loaded launchd job at /System/Library/LaunchDaemons/org.ntp.ntpd.plist and you’ll notice it has a KeepAlive/PathState key with a value that it only runs as long as… /private/etc/ntp.conf is true, meaning the file exists. The contents would be something like server time.euro.apple.com., but for whatever reason over the past few weeks that file went missing entirely across hundreds of computers under our care. Putting it back in place immediately synced the clock. To the fixing machine! We use a lot of ugly Ruby in an elegant way via facter to do site detection, and even already had a ‘location’ fact for use with Simian that told us overall region (although Apple only shows 3 NTP server choices, and we consider India separate). Just detect if the file is missing, and if so, echo the appropriate server into that path, and Bob’s Your Uncle, Muriel’s Your Aunt. But then we hit another stumbling block – SIP!

You can’t echo text to /private/etc. You can mv files to /private/etc. Because, y’know, reasons. Hope this helps whoever out there is still clinging to 10.12 become less crazy. (But really, we just have an appliance-type use case that we haven’t migrated to 10.13 yet, everyone accepts the fact we’re on borrowed time and should upgrade ASAP. Cheers!)

Allister Banks

Allister lives in Japan, has not read the Slack scroll back, and therefore has no idea what is going on.

More Posts - Website

Follow Me:


1 Comment

  • Hi, Allister,

    I just made the jump to light speed, going from 10.6 to 10.13, when I heard the news that Server is dead. So, @*%)@.

    Any thoughts on a migration to similarly-managed (i.e., GUI) macOS-based services or plans to produce some guidelines? I mean, anything would be better than Apple’s “Hey, here are some other things which, you know, might work…”


Leave a reply

You must be logged in to post a comment.