For IT infrastructure, performance will never go out of style. Over the last few years, this has been driving the take-up of all-flash storage, or all-flash arrays (AFAs). While the current generation of AFAs meets the needs of mainstream applications, there are other applications, such as analytics, that need even faster access to data.

The majority of existing AFAs are powered by SAS flash drives, and one way to boost their performance is to adapt them to use drives based on the emerging and faster NVMe standard. Another way to boost their performance is to use the NVMe over Fabric (NVMe-oF) standard in the networks that link them to host servers. A small number of existing AFA vendors have already made at least one of those moves. Incumbent vendors say that, in time, they will follow suit and put NVMe to work both inside their storage systems and in the storage networks.

However, more performance can be wrung from NVMe flash by developing storage systems based on new architectures. A number of startups have raised funding to do exactly that, and have been bringing their products to market over the last year or so.

The systems being sold by this group are not competing with existing SAS-powered AFAs, or AFAs that have been converted to NVMe. Instead, they collectively represent a second generation of all-flash, NVMe and NVMe-oF systems that are significantly more expensive than existing AFAs, but are also much faster, and can meet the needs of data-intensive applications.

This report is an overview of the sector and the architectural issues. In a separate report, we will outline the startups that have developed purpose-built NVMe storage systems and the prospect of acquisitions.

The 451 Take
NVMe is important technology. Suppliers of all sizes agree that it will eventually dominate usage of flash in datacenter storage systems, and that it will take some while for that to happen. We think it will also take time for new NVMe-based architectures to be adopted, but that it will happen because of the growth of data-intensive applications, such as analytics. These architectures represent the future of high-end storage systems in the datacenter, and they will change the way storage fits into the overall IT infrastructure. NVMe was designed to work not just with flash, but also with nonvolatile memories that are even faster than flash, which will make the need for new architectures even stronger.

The NVMe and NVMe-oF standards
First issued in 2011, NVMe is a standard for the protocol and software drivers used with PCIe flash drives. Before then, PCIe flash drive makers wrote their own proprietary protocols and drivers, which varied in performance. NVMe levels the playing field by giving all drive makers access to a high-quality, community-owned standard protocol, alongside drivers written to that protocol, which are now included in Linux and Windows. That opens up the market to a new generation of enterprise storage vendors looking to move beyond the legacy of the SCSI command set, on which SAS is based.

NVMe drives are currently about 50-75% faster for reads and writes than the SAS-based flash drives that power most current AFAs. However, it's important to note that the management architecture and command set for NVMe is nowhere near as complete or validated as that of SAS for array-based use of the drives.

The NVMe over Fabric standard was released in 2016, and extends NVMe to run over the Ethernet, Fibre Channel or InfiniBand networks that link storage systems to host servers. That provides another boost to performance. Finally, the significance of NVMe goes beyond flash – the standard was also written to suit other nonvolatile memories, such as Intel's and Micron's 3D XPoint, or HPE's promised ReRAM.

Target markets
Take-up of the current first generation of AFAs began in earnest only about five years ago, and the systems are already in use at about 25% of all enterprises, and well over half of enterprises with more than 1,000 employees, according to 451 Research's Voice of the Enterprise (VotE) service. Those percentages are growing quickly because all-flash storage eliminates the performance issues inflicted by disk storage.

However, there are data-intensive applications that require even faster access to data than provided by current AFAs. Among these, real-time analytics is the biggest single target for vendors of second-generation NVMe storage, not least because it is expected to see sharply increased usage over the next few years. Others include artificial intelligence, genomics, video processing, financial trading, and conventional but performance-sensitive transactional databases.

Because of this need for fast access to data, some IT organizations are currently forced to store application data in flash drives installed in host servers. This use of direct-attached storage (DAS) boosts performance by eliminating the latency imposed by the storage network that links the host servers to AFAs. Also, NVMe drives can be used inside the host servers.

However, DAS suffers multiple drawbacks, which is why IT organizations have moved away from DAS over the last two decades, and have replaced it with stand-alone, shared storage systems such as AFAs (commonly known as SAN storage). The intention of second-generation NVMe storage vendors is to provide the same or similar performance as DAS, but in shared stand-alone systems that don't suffer the cost and operational drawbacks of DAS.

The price of NVMe flash drives
Most of the cost of an all-flash system is in the flash drives that power it (because of the cost of the NAND flash chips in each drive). PCIe flash drives – whether or not NVMe-compliant – have always been more expensive than SAS drives. They are not intrinsically costlier to make, but they currently sell in smaller volumes, and drive-makers can charge more for them because they are faster than SAS drives. However, the price gap has been narrowing, and the rate at which that has been happening has been accelerated by the democratizing effect of the NVMe standard. But it is still significant.

SAS and NVMe enterprise flash drives ship as single-port or dual-port devices, and vendors say NVMe pricing will pass through a watershed when enterprise-grade dual-port NVMe drives reach the same price as equivalent dual-port SAS drives. This is because dual-port drives provide greater overall system availability. While many of the vendors in this report are using single-port drives to achieve the same availability, that involves far higher capacity overheads and, hence, greater costs.

One OEM told 451 Research recently that dual-port NVMe drives are currently about 15% more expensive than SAS equivalents, and added that, for many customers, that is a significant price difference. Exactly how soon price parity will be reached is in the hands of drive makers, such as Samsung and Intel. Some system vendors say it could happen this year. Others predict it won't happen until H2 2019. Drive-makers will not comment.

Architectural challenges
To achieve the ultimate performance of NVMe flash in shared, stand-alone storage systems, system designers face multiple problems. The core issue is that the IO path from the drives, through the systems themselves and across storage networks to the host servers, must be shortened or made faster than in current AFAs. This is even more necessary if the same systems are to be powered by next-generation memories, such as the 3D XPoint memory used in Intel's Optane drives, which are even faster than flash. When those memories are used, the latency added by the rest of the storage infrastructure becomes even more of a problem.

NVMe-oF is part of the solution, but many of the startups in the field are also using new generations of very fast FPGA chips to handle a range of tasks – both in their storage systems and in the network links to host servers.

According to the same startups, a closely related issue is that the latency imposed by software interrupts in the x86 controllers used in conventional storage systems must be eliminated or addressed. Generally, incumbent suppliers tacitly agree with this view, although not always.

Some of the startups in the sector have eliminated x86 controllers entirely from their systems. The resulting hardware is often described as 'just a bunch of flash,' or JBOF. The startups themselves are happy with that label because they want buyers to know that their systems are highly streamlined and unencumbered by x86 controllers. On the other side of the fence, we know of at least one incumbent AFA vendor that disparages the new systems as 'just dumb JBOF.' That is because, when there are no storage controllers, there is nowhere in the system to run software that provides data services, such as RAID, snapshots, replication and data reduction.

This is an important issue for most, but not all, storage use cases. There are applications like NoSQL databases that benefit from the low overhead and high performance of local storage, so it makes sense to build a storage environment that offers more options than traditional SAN or NAS does.

Over the last two decades, the rise of stand-alone shared SAN storage has led IT organizations to expect volume-based services like replication and snapshots to be provided by the storage system, rather than by the application or other software running on host servers. By using the services provided by the storage system, the services are standardized and can be centrally managed as part of the storage layer.

However, this situation is changing, and services such as data mirroring are becoming more common within applications and virtualization platforms. Furthermore, as applications become more complex, they benefit from more closely integrated storage services that are better tailored to their specific requirements. This overall change is what some – but not all – startups in this field are banking on.

Hardware versus software
The startups that have developed purpose-designed NVMe storage systems have taken a wide range of approaches, which fall into two broad camps – one focused on software and the other on hardware.

The suppliers that have taken a software route have moved the data services off the x86 controllers within their systems and onto the x86 processors in host servers. To do that, they have had to develop distributed volume manager software that runs on the host servers. The advantages are a major reduction in the latency imposed by the x86 controllers in the storage system, the ability to base the system entirely on commodity hardware, and the ability to provide full data services. The disadvantage is that customers suffer the operational overheads of needing to install software on their host servers, and the startups supplying that software have to qualify it for multiple operating systems and hypervisors.

In the hardware camp, there is much use of FPGA programmable processors, both within the systems and in network adaptors. To paraphrase the argument behind this practice, 'there has never been a better time to take a hardware approach because of the continual improvements in FPGA performance, capabilities and cost.'

In the past, many storage systems have used ASICs, which are purpose-designed to handle specific tasks. FPGAs share many similarities to ASICs, but rather than being hard-coded to handle specific functions, FPGAs can be coded for a variety of tasks, making them far less expensive to design and make. Indeed, FPGA usage is not entirely new or uncommon. 451 knows of at least two incumbents with established AFAs that include FPGAs.

The disadvantage of using FPGAs is that the resulting storage system is no longer based on commodity off-the-shelf hardware. But FPGAs themselves do not represent proprietary technology; indeed, they introduce flexibility into hardware. Some startups have used FPGAs to implement proprietary alternatives to NVMe-oF. That standard is currently immature – version 1.0 was not released until 2016. But when it settles down, those startups say they will be able to very quickly switch to the official standard, simply by recoding the firmware driving the FPGAs.

Disaggregation
Some of the startups in this sector are using the term disaggregation when they describe their products. Disaggregated storage is physically and logically separated from host servers, or from the 'compute layer' – much in the same way that current SAN storage is separate from compute, and hyperconverged storage is not.

Ideally, storage and compute are kept separate to allow independent scaling, and to simplify operations and allow storage and compute hardware to enjoy different replacement lifecycles. For many purpose-designed NVMe storage systems, the level of disaggregation is less than it is for conventional SAN storage. This is either because host servers in the compute layer have to host storage software to provide data services, or there are no such services at all, in which case applications – again, part of the compute layer – must provide those services. Also, some purpose-designed NVMe storage systems require custom or proprietary storage network adaptors to be installed in compute servers, which reduces the level of disaggregation.

Data reduction
Quite separate from architecture, a simple distinction between existing AFAs and second-wave NVMe storage systems is that the latter do not currently feature any data-reduction functions, and may never do so. This is because their highest design priority has been to maximize performance.

Although de-dupe and compression slash the purchase cost of flash storage systems – typically by a factor of two or more – they also reduce performance. This underlines the fact that purpose-built NVMe systems are not intended to compete with existing AFAs, or even with AFAs that have been converted to NVMe, but represent a new class of high-performance storage.
Tim Stammers
Senior Analyst, Storage

Tim Stammers is a Senior Analyst at 451 Research, where he covers flash- and disk-based primary storage.

Steven Hill
Senior Analyst, Storage

Steven Hill is a Senior Analyst of Storage technologies. He covers the latest generation of hyperconverged systems, cloud-based storage and business continuity/disaster recovery solutions for enterprise customers.

Speaker Name
Speaker Title

Sed ac purus sit amet nisl tincidunt tincidunt vel at dolor. In ullamcorper nisi risus, quis fringilla nibh mattis ac. Mauris interdum interdum eros, eget tempus lectus aliquet at. Suspendisse convallis suscipit odio, ut varius enim lacinia in. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Want to read more? Request a trial now.