Read Product Review CyberStore 136NF1 - 36x NGSFF PCIe NVMe SSD All-Flash Storage Array

Broadberry delivers a heap of the latest ruler form factor NVMe SSDs

When looking to outfit data centre servers and storage arrays, the high performance of NVMe SSDs makes them a natural choice, but the current range of M.2 and U.2 form factors pose significant restrictions on applications and density. The EDSFF (Enterprise & Data Center SSD Form Factor) specification addresses these limitations, however, and in this hands-on review, we look at one of the first production servers to support these new storage devices.

Broadberry’s CyberStore Xeon SP2-136 showcases AIC’s ‘Grays Peak’ FB128-LX 1U rack server, which has room up front for no less than 36 hot-swap EDSFF NVMe SSDs. The system was supplied to us with six 4TB Intel DC P4511 SSDs, so a full house of these diminutive devices delivers an impressive 144TB of raw storage density.

Formerly known as ‘Ruler’ SSDs, Intel’s DC P4511 adheres to the E1.S (short) standard, employs 64-layer, 3D NAND TLC technology and presents a standard PCI-E 3.1 x4 connector. A card width of only 5.9mm allows it to deliver this high storage density and Intel claims it’s three times more thermally efficient than bulkier U.2 SSDs.

Storage

The CyberStore is well designed and built with all SSDs presented in small hot-swap carriers across the front panel. Air flow and cooling are aided by the carrier’s perforated front panel and each SSDs is fitted with slim-line heat spreaders on their upper and lower surfaces.

Internally, everything is just as tidy as a single backplane board with integral switches services all SSD bays and is directly connected to the motherboard’s two PCI-E risers with four ribbon cables. A key advantage of this arrangement is you don’t need to sacrifice any expansion slots to PCI-E switch HBAs.

Other less interesting storage features are the pair of M.2 slots on the motherboard. Supporting both SATA and NVMe M.2 modules, these can be used to run an OS on a mirrored array, leaving all your EDSFF SSDs available for data storage duties.

You have a number of choices as to how you employ the SSDs; with Windows Server 2019 at the helm, we could configure them as individual devices and use Storage Spaces to create pools with simple, mirrored or RAID5 virtual disks. Broadberry also offers an optional Intel VROC (Virtual RAID on CPU) key which plugs into the motherboard and enables hardware-managed stripes, mirrors, RAID10 and RAID5 arrays.

Performance

For the DC P4511 E1.S SSD, Intel quotes sequential read and write rates of up to 2,800MB/sec and 2,400MB/sec, along with random read and write throughputs of up to 610,200 and 75,000 IOPS. We used a range of benchmarking tools including Iometer, ATTO and CrystalDiskMark and saw figures that generally agreed with these.

Using a single SSD, we recorded up to 2,785MB/sec and 2,247MB/sec for sequential read and write operations. We couldn’t reach the top random read throughput claims - with our tests returning best speeds of 481,288 IOPS - but random write rates of 94,590 IOPS were noticeably higher.

We also tested a four-SSD RAID10 array created using the VROC Windows utility. It’s easy to use, but be careful with SSD placement, as arrays that span across multiple controllers can’t be used as bootable volumes.

Performance was very impressive; we recorded top sequential read and write rates of 9,531MB/sec and 4,950MB/sec. I/O throughput was good too, with the array returning high random read and write speeds of 769,300 and 465,400 IOPS.

Features

The CyberStore is also well-specified in other departments, and the price includes a pair of 2.2GHz 10-core Xeon Scalable Silver 4120 CPUs teamed up with a generous 256GB of DDR4 memory. These CPUs can handle up to 1TB of memory - but bear in mind that these Silvers only support maximum memory speeds of 2,400MHz.

There’s plenty of room to expand, as well. Along with dual embedded Gigabit ports, the motherboard offers two OCP 2.0 mezzanine card slots and two free PCI-E slots. The server supports plenty of industry-standard adapters and mezzanine cards, so you can easily boost network speeds to 25GbE, 50GbE or 100GbE.

SSD health monitoring is handled by Intel’s handy MAS (Memory and Storage) Windows utility, which provides valuable information on each SSD such as remaining life and temperatures. It’ll also tell you what partitions are present on each SSD - although we did notice that it is unable to show temperatures for individual members of VROC managed RAID arrays.

Remote server management and monitoring are provided by an embedded AST2500 management controller – the same as used by Gigabyte. It isn’t up to the standards set by Dell EMC’s slick iDRAC9, but does provide plenty of detail on critical components with graphical readouts on hardware sensors, fan speeds, temperatures and voltages.

Cooling is handled by a bank of seven dual-rotor fans spread out between the motherboard and SSD backplane. Noise levels aren’t excessive with our iPad measuring sound levels of 65dB from one metre, although unlike the Gigabyte variant of the AST2500, the AIC version doesn’t include fan profiles to control their speeds.

For centralized multi-server management, you’ll need to source your own SNMP monitoring solution; whereas Gigabyte offers its free Server Management web console, AIC doesn’t provide any extra utilities. However, AIC does include full OS remote control and virtual media services as standard and not as chargeable license upgrades.