Articles,Server October 31, 2014 at 7:53 pm

VMware ESXi 5.5 on the Mac Pro (2013)

Five years ago, it was not uncommon to see many cabinets full of Xserves fulfilling serving roles in the enterprise.  Today those servers are on their last legs and warrant replacement by modern hardware and software design.

Luckily there isn’t much of a need of OS X Server by IT anymore as almost everything can run on Linux or Windows these days.  Aside from Deploy Studio, management tools such as Munki, Reposado, Puppet and even the JSS are platform agnostic.  However, many organizations still need instances of OS X running in the datacenter, whether it be for testing of older operating systems, compiling apps, or running automated AutoPkg.

VMware’s ESXi hypervisior has emerged to be the market leader, but until last week the only supported Mac was the Mac Pro 5,1.  This was a good machine, but took up a massive amount of rack space and while its processors were fast in 2010, they just can’t keep up today.  For those running vSphere, the licensing costs of dual CPUs providing just 12 cores also don’t make sense anymore.

Apple shipped the new Mac Pro ten months ago and no public version of ESXi would boot on it due to various kernel issues.  It is unfortunate that it took so long as many of us had a stack of machines gathering dust for so many months, but the wait is over.  As of ESXi 5.5 Update 2 Patch 1, it is on the HCL and works without issue.

For those wishing to run ESXi on the latest Mac Pro, here are some recommendations:

CPU and RAM

The 4, 6, and 8 core Mac Pros ship with processors from the Intel Xeon E5-1600v2 series, and the 12 core has an Intel Xeon E5-2697v2.  Aside from the clock speed and cache differences, the difference between them is that only the 12 core model supports LRDIMMs.  For those wishing to upgrade the Mac Pro to 128GB, double its supported amount, consider purchasing the 12 core.  32GB LRDIMMS are available at the full speed of 1866MHz, so performance should remain near its maximum potential. While 32GB modules are available which work on the E5-1600v2 series, they are all RDIMMs and capped at 1333MHz and will impose a significant performance penalty for workloads which are memory intensive which is very common when running multiple virtual machines.

An upgrade to 64GB of memory is the most cost effective option, and the best chip for the job is the Hynix HMT42GR7AFR4C-RDT3, available for around $150 each in bulk.  These are the same chips Apple and OWC sell for much more, and they will work in any Mac Pro.

Networking

In a couple of years 10-GBaseT chipset prices will likely come down to reasonable levels, but until then we will be stuck with the same gigabit Ethernet interfaces that have been around since the PowerMac G4.  OS X does not support the VMXNET3 adapter, so virtual interfaces are also capped at gigabit speeds, but those who have their virtual machine storage on a SAN will quickly find that gigabit speeds are not enough for anything but the most basic virtual machines.

Thankfully, the new Mac Pro has three Thunderbolt 2 channels spread across six ports, and these are perfect for expanding the networking capabilities of the machine.  The easiest solution for this is the Promise SANLink2, which is available with either 10GBase-T or 10G SFP+, along with a Fibre Channel model for those who don’t use iSCSI or NFS.  All three models are priced around $900, although the 10G SFP+ version does not come with any modules.

Alternatively, and likely a better option for those with specific networking needs or already have 10 gigabit PCI-e cards is the Sonnet Echo Express SE II, also available in black as the OWC Mercury Helios II.  Add any supported card such as the Intel X520 or X540, and it will work without any configuration required.

If you don’t have a network that supports 10 gigabit connections but need more than 2 gigabits of connectivity, the Apple Thunderbolt to Ethernet adapter does work with the latest patch.  With up to six of these and the two onboard ports, up to 8 gigabits of connectivity can be had.

Take note, ESXi does not support hot-plug of Thunderbolt devices at this time.

Storage

With proper networking as configured above, storage will perform just as any other ESXi host.  However, the Mac Pro does contain an extremely fast PCI-e SSD, and it should be put to use.

There are three options for use of this SSD:

  • Local datastore – this will likely be the fastest place for a VM to live, but it will be space constrained and not redundant at all.
  • Virtual Flash Read Cache – for those using external SANs, this should be enabled.  It will allow the SSD to act as a read cache for VMs, and can significantly speed up read-heavy workloads.
  • VSAN – Virtual SAN is a new product from VMware which replaces the traditional SAN with one running on top of the hypervisior.  Hosts need to contribute at least one SSD and one spinning disk, and the internal one works without issue.
  • Those considering using any of these options should likely choose the 1 TB option – it is an $800 upgrade but this is a great price for the size and speed compared to options like the Intel S3500 or Micron P420m used in many other servers.

    Installation

    Unfortunately Macs cannot PXE boot, so ESXi must be installed onto local storage.  While it can of course be installed onto the internal SSD, it is best to save that to be used as a VMFS5 or VSAN datastore.  Instead plug a

    slimline USB 3.0 thumb drive into one of the Mac Pro’s four ports, and ESXi will happily install and boot from it.  Almost any thumb drive will work, but ESXi will install and boot significantly faster than from a USB 2.0 drive. The ESXi installer will not bless the drive it is installed on, so if the internal SSD still has an operating system on it that will still boot.  Before installing ESXi, erase the internal SSD so that the USB disk is the only bootable option, and the Mac Pro will happily boot it every time.

    Rack mounting As a cylinder, the new Mac Pro isn’t the best fit in a server room or datacenter, but there are quite a few options for mounting the Mac Pro in a standard 19 inch rack.  Even though it does require additional hardware, it thankfully takes up less than the 5 to 11U of the old model Sonnet has the

    RackMac Pro which allows for one or two Mac Pros to be mounted in 4U of space.  While it is a good product, it is incredibly expensive at $869 for the two node model.  If that price included some form of Thunderbolt to PCI-e expansion it would be a good deal, but for a steel box with a few extension cables it just seems quite steep. Instead, consider options from MK1 Manufacturing, a small company from Utah.  MK1 offers the MPR-2x, which like the RackMac Pro mounts two nodes side-by-side in 4U of space but also offers sliding rails for easy access to the units.  Better yet, at $289 it is less than third of the price which Sonnet charges. For those who need to rack many Mac Pros, MK1 has a new product, the MPR-8X.  This allows eight Mac Pros to be mounted vertically in 7U of space, and has an active exhaust unit to keep the units cool.  This is a very recent release, but it is more than double the density of the other options and almost six times the density of the old Mac Pro.

    Conclusion Before purchasing a fleet of Mac Pros to throw in your datacenter, please consider whether they are actually needed.  The need for OS X Server is waning, but if you need to virtualize OS X for any reason the Mac Pro is certainly the best bet.  It is officially supported by VMware, and even though the dual GPUs go unused it is much more cost effective for compute than the latest Mac mini.

    About Samuel Keeley

    Samuel Keeley can frequently be found in ##osx-server on Freenode, or on Twitter @keeleysam.

    3 Comments

    • The need for OS X Server is waning? What are people doing for MCX/Profiles?

    • MCX is deprecated and has been since 10.8. Config profiles are the replacement – essentially the same thing under the hood though. MDM is the solution moving forward think casper/airwatch/absolute manage etc etc all of these will run without the need for osx server. the long standing reason people needed actual mac os x hardware and os x server was netboot but this has now been solved by bsdpy and netsus which run on linux and are solid solutions

    • I just tried this on my 2013 MacPro and It works. I had to use ESXi 5.5 Update 2a. We then purchased a Promise SANLink 2 dual 10 G Ethernet to test. We have an iSCSI back end and we connect our hosts with two network connections for redundancy through two different switches. After installing the drivers for the SANLink 2, Mac OS X sees both ports on the device. But rebooting into ESXi, only one of the two ports is available. I see the two built in ports on the Pro, the one on the TB display, and only one on the SANLink 2. Any one else experience this issue?

      Andrew

      —————————————————–

      Andrew W. Johnson
      Sr Macintosh Systems Administrator
      Desktop and Systems Engineering
      Division of Information Technology
      Stony Brook University
      Computing Center Room 115
      Stony Brook, NY 11794-2400

    Leave a reply

    You must be logged in to post a comment.