Of the UNIX tools shipped with OS X, curl has become one of the most used in recent years, whether as a simple URL fetcher, or a component of much larger projects. It takes any URL as an argument, and reliably fetches it, as well as giving many additional options for control. With OS X Mavericks, Apple ships a recent version of curl, version 7.30.0. However, the Apple-shipping binary contains some modifications from those in previous OS X releases, breaking some applications. The first issue introduced with recent versions of curl is relatively minor, in that headers are no longer immediately relayed when in certain console interfaces, including the OS X default and with Python calling out to curl. This caused cosmetic issues with Munki and other scripts calling curl, as download status could never be seen. For Munki, this was a relatively simple fix, wrapping curl so it believed it was running in a TTY interface. Thanks to Greg Neagle for finding the fix to the problem and quickly implementing it in build 0.9.2.1852 of Munki. The most significant change impacts the use of client certificates for SSL authentication, commonly used for management software such as the Munki Project. Previous versions of curl allowed for certificate files to be specified with flags at the command line, but this feature is no longer functional. Instead, Apple has modified curl to only accept certificates from the System keychain. Even if a certificate is moved to the System keychain, the system may still refuse to accept it if it comes from a non-Apple shipping certificate authority, which is the case with almost all client certificates. Apple’s additional support for the use of Keychain certificates with curl is welcome, and opens up some interesting options for further applications. However, it is a serious issue for those current applications relying on the ability to pass certificates via flags. Additionally, even if certificates are located in the Keychain, curl will not accept them if they are signed by a non-Apple-shipping certificate authority. Most large organizations run their own internal certificate authorities, and this becomes a serious issue, as OS X in general, and not just curl, treat these CA’s as different, even if they are set to be fully trusted. When this issue was first discovered, there were a few possible idead for fixes floated, generally having do with workarounds for Munki, which makes heavy use of curl for all communication with repositories. The first, and easiest, would have been the worst, replacing Apple’s curl with a self-complied version, slipping it in while Munki was running, and reverting to Apple’s curl when done. This worked in testing, but also would have introduced huge chances of causing issues with the rest of the system. If there were other services using curl at the same time as Munki, or if Munki unexpectedly quit during its run, there would very likely have been issues in other areas of the system which rely on Apple’s changes to curl. This was a very hacky workaround, and should be avoided. After this option was discarded, better options came into play. While at MacTech Conference 2013, it was discussed what the best approach would be to fix this in the short term, and it turned out to be a very easy one. While Munki would still rely on Apple’s curl by default, which works just fine in the vast majority of environments, a new key would be added to Munki’s configuration allowing for the path to curl to be set. With much thanks to Frogor (Michael Lynn), a patch was submitted for Munki that night. To use this new key, a self-compiled version of curl should first be loaded onto the system. Utilizing Tim Sutton’s excellent brew-pkg, it is simple to create an installer package for the new curl binary. While Apple’s curl lives in /usr/bin, the best place for this new binary is /usr/local/bin. After placing this new curl binary, set the key `CurlPath` to `/usr/local/bin/curl` in `/Library/Preferences/Managed Installs.plist` or using MCX/Profiles. Unfortunately, if Munki is the only management system on a Mac and it has already been upgraded to Mavericks, this will have to be done manually, so placing these on older non-upgraded clients would not be a bad idea. This fix has rectified the issue, but it is not ideal, as it means that to use Munki with client certs, it now has dependencies on extra packages. Additionally, as OS X goes on, there may be more issues with Apple’s modified UNIX tools versus expected behavior. Currently in planning and discussion on Twitter and ##osx-server is a complete move away from curl in Munki, relying solely on Python libraries. This will require much work, and a lot of testing, so please hop on IRC and give your thoughts. While many bug reports have already been filed with on this issue, it would be appreciated if more could be filed in order for Apple to see the extent of the problem. With 10.9.1 already being tested, it is highly unlikely to even be touched until 10.9.2, but it would be great for the issues to be truly fixed soon. While the workaround was relatively simple for Munki, it might not be so easy for other pieces of software. If there is one lesson learned here, it should be to never assume that the open-source tools Apple ships with OS X will behave the same as on other UNIX systems, and to avoid hardcoding the paths to these tools. Additionally, it is wise to have multiple systems in place for Macintosh management, even if the second is simply SSH. Other management tools broke with the upgrade to Mavericks, including Puppet. Of course, the best approach is to test extensively and have fixes in place before deployment of new OS X versions, something which especially holds true with Apple’s now-yearly release cycle. One of the most important tools in any Mac admin’s toolbox is an Apple Developer ($99 yearly) or AppleSeed (free, invite only) account, giving early access to software builds. Part of any engineer’s job is to test and fix problems early, as the end-users should not be guinea pigs. Again, special thanks to Michael Lynn (@mikeymikey) and Tim Sutton (@tvsutton) for making this fix so easy to implement.
Samuel Keeley
Samuel Keeley can frequently be found in ##osx-server on Freenode, or on Twitter @keeleysam.
Follow Me:
by 
Why not bundle a workable self-compiled version of curl with Munki as a resource? If this has already been discarded, I would guess licensing incompatibilities…?