The emerging NVMe standard and its networked NVMe over Fabric (NVMe-oF) version promised to deliver a significant boost to the performance of both all-flash storage systems and future storage systems based on other non-volatile memories, such as the 3D XPoint memory that powers Intel's Optane drives, or Samsung's Z-NAND. The majority of existing all-flash arrays (AFAs) are powered by serial-attached SCSI (SAS) drives. Converting these AFAs to be powered by NVMe drives and using NVMe-oF to connect them to host servers will boost their performance. But a handful of startups argue that the full potential of NVMe and NVMe-oF can only be realized in storage systems that have been purpose-designed for the job, and are based on very different architectures than existing AFAs.
In a separate report, we outlined the potential applications and market for such systems, as well as the architectural issues. In this report, we examine the startups that have developed the systems, and discuss the potential for M&A.
The 451 Take
NVMe is revolutionary technology, but it will be a relatively slow burner. OEMs acknowledge its importance, but as a group – and with some exceptions – they are moving at a relatively relaxed pace to adopt it. This is because mainstream needs for high-speed all-flash storage are being met with current technology. As a result of this and other factors, in the near term we do not expect any of the incumbent suppliers to ink any acquisitions in this field. But unless they abandon their long history of entering new technology areas by buying startups, we think that eventually there will be consolidation, probably in 2019 or beyond.
Incumbents suppliers and NVMe
Although all incumbent vendors say NVMe is an important technical development, very few have yet adopted the standard in a significant way. So far, Pure Storage is the only major AFA provider to have launched a system powered entirely by NVMe drives. Called the FlashArray//X, the system began shipping last year and is a version of the company's existing SAS-powered AFA, with a conventional architecture. Pure recently said the FlashArray//X now accounts for over 20% of its sales, and is being used in artificial intelligence and analytics. Later this year, the FlashArray//X will gain support for NVMe-oF networks externally.
In 2017, NetApp launched support for NVMe-oF external networks with its EF-series AFA, but internally the AFA remains SAS-powered. The company says that at some stage, other NetApp storage systems will also support NVMe-oF, and will be powered internally by NVMe drives – but it is not yet disclosing when that will happen.
IBM recently unveiled NVMe-oF network support for its FlashSystem 900 AFA. Big Blue also announced that other IBM all-flash systems will support NVMe-oF later this year, and that it plans to ship systems powered internally by NVMe flash drives in 2018. IBM's portfolio is very unusual in that it includes systems powered by proprietary flash modules that are similar to NVMe drives, but which Big Blue claims are faster (and technically, are not PCIe drives). The systems that use these modules include the FlashSystem 900, which IBM describes as a 'tier zero' device that has no main x86 controllers but does offer services such as compression.
HPE has not said when it will introduce AFAs powered entirely by NVMe internally, or support NVMe-oF externally. Hitachi Vantara (formerly HDS) has said it will ship an all-NVMe version of its flagship USP storage system in 2019. Dell EMC has already made an ambitious but failed move into NVMe storage (see below), and says it will try again with a second system this year. It has given no details of that second system, so 451 Research does not know whether it will be a ground-up-designed NVMe system, or a version of an existing AFA converted to NVMe.
EMC's DSSD D5 was a market failure, not a bad omen
In 2016, pre-merger EMC made a major foray into NVMe storage when it unveiled the DSSD D5. The D5 was a purpose-designed NVMe AFA developed by a company called DSSD that EMC had bought two years earlier. Presumably, the storage giant hoped to repeat the success it enjoyed with its XtremIO AFA, for which a key ingredient had been early-mover advantage.
But the D5 was canceled within a year of its launch, very soon after the merger with Dell. Publicly, Dell EMC said it discovered that customers were happy to use existing AFAs for the applications the D5 was aimed at. In other words, too few customers wanted to pay a premium for the D5's superior performance. However, we do not believe this means this will be true for every purpose-designed or second-generation NVMe storage system.
The D5 was complex to deploy, had scaling limitations, and used proprietary technology from end to end. We estimate it was two to three times more expensive than the systems being sold by the startups in this report. Despite these issues, and its short and apparently very unsuccessful history, some rivals have told us that the D5 was an admirable design. We presume the D5 failed because, at best, its technical content was too much, too soon for the market.
Potential acquirers
We believe the major storage suppliers are unlikely to make acquisitions in this sector in 2018. This is because of various reasons related to individual incumbents, and because the size of the demand for purpose-designed NVMe systems is not yet clear. Another potential source of bids is from enterprise disk and flash drive makers, which have been edging into the storage systems market for some years to boost their profit margins. For this second group of potential suitors, there is reason to believe that one or more might eventually ink a purchase in this sector – but again, that appears unlikely this year.
Last year, giant flash chip and drive maker Micron briefly licensed software from Excelero (one of the startups discussed in this report), which it used to create a purpose-designed NVMe flash system called SolidScale. Such deals can be preludes to acquisitions. However, the project was cancelled within months after Micron replaced the head of its storage business unit. The vendor is now instead pitching a SolidScale reference architecture to channel partners. That makes a purchase by Micron in this segment very unlikely, at least in the short term.
Disk and flash drive behemoth Western Digital (WD) already sells object-based disk arrays, as well as SAS-powered AFAs, and it might want to buy a provider of purpose-designed NVMe storage systems to develop its all-flash portfolio. While this is a possibility, it is compromised by the fact that last year WD picked up Tegile, a privately held developer of midrange to high-end storage systems. Before its sale, Tegile had announced that it was only months away from launching an NVMe storage system. Regardless of the architecture of that system, WD may believe it now has enough NVMe system expertise in-house, and doesn't need to make another acquisition.
Giant flash chip and drive maker Samsung sells no enterprise storage systems, but owns more than 50% of Stellus Technologies, a company that was founded in 2016. Stellus describes itself as a developer of solid-state storage systems for next-generation software-defined centers. If there were any doubt about what that might involve, Stellus has been advertising for software engineers experienced in NVMe. This rules out Samsung for M&A in both the short and long term.
The other two giant flash chip and drive makers are Toshiba and Intel. Toshiba's financial problems have just seen it sell off its flash chip-making business, although it has retained its drive-making business. Intel has taken a stake in Vexata, which is one of the vendors described below. However, Intel's practice is to invest in startups to stimulate developments, rather than acquire them outright, because it does not want to compete with its system-making customers.
Potential targets
Apeiron
Apeiron's system has been in 'active' general availability (GA) since 2017. The company claims multiple customers, including two hosting facilities for Splunk, and has named aerospace and electronics behemoth L3 Technologies and the US Food and Drug Administration as customers, and cited two US Department of Defense deployments involving Hadoop and Splunk. Apeiron says it raised over $10m in initial funding, and is currently closing a series A round.
The startup has adopted a stripped-down hardware approach, and claims the result is extremely low latencies, even by the standards of purpose-designed NVMe systems. Its system features no x86 controllers, is powered by standard NVMe drives, and is linked to host servers via a hardened layer 2 Ethernet network using FPGAs to handle network technology it calls NVMe over Ethernet (NoE). Compared with NVMe-oF, Apeiron says NoE is simpler, faster and better suited to its architecture, in which storage services are provided from the host or application, rather than its storage system.
Attala Systems
Attala's system is slated to enter GA later this year, but the company says it has already won two paying customers, and has proofs of concept in progress. It doesn't disclose how much financing it has raised. Attala describes Intel as a strategic partner that has helped with marketing, as well as with other 'significant resources and funds,' without taking equity in the firm.
Like Apeiron, Attala has developed a stripped-down system for which it claims extremely low latencies. The system includes no x86 controllers or storage services, and employs FPGAs to handle NVMe-oF on Ethernet links to host servers. Attala claims its system can support any OS or hypervisor, and is accompanied by cloud-style self-service provisioning and management software.
E8 Storage
E8 released its products into GA in late 2016. The startup says it has racked up production deployments in several verticals such as the financial sector and the travel industry, and has announced test deployments of Intel Optane- and 3D XPoint-powered versions of its system at the Tel Aviv Stock Exchange. E8 has raised $18m in funding.
The vendor's system is based on commodity x86 hardware, and can be bought as software-only or as an appliance. The system eliminates latency in its x86 controllers by providing services such as RAID, snapshots and replication from software running on host servers. A proprietary protocol is used with Ethernet on the links between the system and host servers, but E8 says it could adopt NVMe-oF in the future, when the standard is more mature.
Excelero
Excelero came out of stealth in early 2017, at which time it said it had already scored about a dozen customers, including NASA, PayPal, GE and Hulu, as well as hyperscalers and banks. The company has raised $17.5m in two funding rounds.
Alongside E8, Excelero is the other vendor that has taken a software approach. Unlike E8, Excelero only sells software and does not sell appliances. Its software runs on commodity hardware and provides data services from the host servers. The software can be used to create either standalone or hyperconverged storage. Most deployments have involved standalone storage. Excelero offers a choice of NVMe-oF or a proprietary protocol across InfiniBand or Ethernet.
Pavilion Data Systems
Pavilion has been shipping beta systems since late 2015, and says it has won nine customers. The vendor is not yet naming any of its customers, but says they come from a range of industries, including hosting, SaaS, social media and other sectors. It has raised $15m in one financing round, which occurred in 2015.
We would describe Pavilion's design approach as hardware-focused. The startup's system provides storage services from x86 controllers in the system itself, hence maintaining the highest levels of separation or disaggregation of storage and compute. It avoids the latency that this might impose by using up to 20 array-side pairs of x86 controllers, alongside patented internal networking to deliver what it says is better or equal performance to rival systems. Pavilion notes that apart from the chassis and some of the boards, all of the hardware is commodity gear.
Vexata
Vexata's offering officially became GA in early 2017, and the company says several customers are employing its system in production, including online service provider Oath (AOL/Yahoo), Sanmina and the Tata Group. It has raised $54m, which includes an investment from Intel Capital.
The startup's system uses conventional x86 main controllers, with processing offloaded to FPGAs. The system requires no host software to be installed, and data services running from the array itself include RAID snapshots and replication. Like other systems, the Vexata product can handle NVMe-oF running on Ethernet. Unusually, however, Vexata has chosen to initially support standard Fibre Channel links between its system and host servers, with no usage of NVMe-oF. It believes its customers will be happier with this approach because FC is more familiar to storage administrators than Ethernet, and because the approach requires no specialist network adaptors.
X-IO
X-IO is a long-established supplier of mainstream storage systems, and is therefore the odd one out in this report, as it is not a startup formed to exploit NVMe flash. However, the vendor expects much of its future growth to come from an NVMe appliance it introduced in mid-2017. X-IO's customer base already included government agencies that were deploying its storage systems for high-performance analytics, and the new appliance is aimed at edge analytics.
When the company unveiled the appliance, it said it had already won sales to a handful of customers. The appliance is a 2U dual-socket device that can house up to 72 NVMe drives. That high density is achieved employing PCIe switching and nontransparent bridging. It uses no custom silicon, and can be configured as conventional standalone, hyperconverged or scale-out storage.
In a separate report, we outlined the potential applications and market for such systems, as well as the architectural issues. In this report, we examine the startups that have developed the systems, and discuss the potential for M&A.
The 451 Take
NVMe is revolutionary technology, but it will be a relatively slow burner. OEMs acknowledge its importance, but as a group – and with some exceptions – they are moving at a relatively relaxed pace to adopt it. This is because mainstream needs for high-speed all-flash storage are being met with current technology. As a result of this and other factors, in the near term we do not expect any of the incumbent suppliers to ink any acquisitions in this field. But unless they abandon their long history of entering new technology areas by buying startups, we think that eventually there will be consolidation, probably in 2019 or beyond.
Incumbents suppliers and NVMe
Although all incumbent vendors say NVMe is an important technical development, very few have yet adopted the standard in a significant way. So far, Pure Storage is the only major AFA provider to have launched a system powered entirely by NVMe drives. Called the FlashArray//X, the system began shipping last year and is a version of the company's existing SAS-powered AFA, with a conventional architecture. Pure recently said the FlashArray//X now accounts for over 20% of its sales, and is being used in artificial intelligence and analytics. Later this year, the FlashArray//X will gain support for NVMe-oF networks externally.
In 2017, NetApp launched support for NVMe-oF external networks with its EF-series AFA, but internally the AFA remains SAS-powered. The company says that at some stage, other NetApp storage systems will also support NVMe-oF, and will be powered internally by NVMe drives – but it is not yet disclosing when that will happen.
IBM recently unveiled NVMe-oF network support for its FlashSystem 900 AFA. Big Blue also announced that other IBM all-flash systems will support NVMe-oF later this year, and that it plans to ship systems powered internally by NVMe flash drives in 2018. IBM's portfolio is very unusual in that it includes systems powered by proprietary flash modules that are similar to NVMe drives, but which Big Blue claims are faster (and technically, are not PCIe drives). The systems that use these modules include the FlashSystem 900, which IBM describes as a 'tier zero' device that has no main x86 controllers but does offer services such as compression.
HPE has not said when it will introduce AFAs powered entirely by NVMe internally, or support NVMe-oF externally. Hitachi Vantara (formerly HDS) has said it will ship an all-NVMe version of its flagship USP storage system in 2019. Dell EMC has already made an ambitious but failed move into NVMe storage (see below), and says it will try again with a second system this year. It has given no details of that second system, so 451 Research does not know whether it will be a ground-up-designed NVMe system, or a version of an existing AFA converted to NVMe.
EMC's DSSD D5 was a market failure, not a bad omen
In 2016, pre-merger EMC made a major foray into NVMe storage when it unveiled the DSSD D5. The D5 was a purpose-designed NVMe AFA developed by a company called DSSD that EMC had bought two years earlier. Presumably, the storage giant hoped to repeat the success it enjoyed with its XtremIO AFA, for which a key ingredient had been early-mover advantage.
But the D5 was canceled within a year of its launch, very soon after the merger with Dell. Publicly, Dell EMC said it discovered that customers were happy to use existing AFAs for the applications the D5 was aimed at. In other words, too few customers wanted to pay a premium for the D5's superior performance. However, we do not believe this means this will be true for every purpose-designed or second-generation NVMe storage system.
The D5 was complex to deploy, had scaling limitations, and used proprietary technology from end to end. We estimate it was two to three times more expensive than the systems being sold by the startups in this report. Despite these issues, and its short and apparently very unsuccessful history, some rivals have told us that the D5 was an admirable design. We presume the D5 failed because, at best, its technical content was too much, too soon for the market.
Potential acquirers
We believe the major storage suppliers are unlikely to make acquisitions in this sector in 2018. This is because of various reasons related to individual incumbents, and because the size of the demand for purpose-designed NVMe systems is not yet clear. Another potential source of bids is from enterprise disk and flash drive makers, which have been edging into the storage systems market for some years to boost their profit margins. For this second group of potential suitors, there is reason to believe that one or more might eventually ink a purchase in this sector – but again, that appears unlikely this year.
Last year, giant flash chip and drive maker Micron briefly licensed software from Excelero (one of the startups discussed in this report), which it used to create a purpose-designed NVMe flash system called SolidScale. Such deals can be preludes to acquisitions. However, the project was cancelled within months after Micron replaced the head of its storage business unit. The vendor is now instead pitching a SolidScale reference architecture to channel partners. That makes a purchase by Micron in this segment very unlikely, at least in the short term.
Disk and flash drive behemoth Western Digital (WD) already sells object-based disk arrays, as well as SAS-powered AFAs, and it might want to buy a provider of purpose-designed NVMe storage systems to develop its all-flash portfolio. While this is a possibility, it is compromised by the fact that last year WD picked up Tegile, a privately held developer of midrange to high-end storage systems. Before its sale, Tegile had announced that it was only months away from launching an NVMe storage system. Regardless of the architecture of that system, WD may believe it now has enough NVMe system expertise in-house, and doesn't need to make another acquisition.
Giant flash chip and drive maker Samsung sells no enterprise storage systems, but owns more than 50% of Stellus Technologies, a company that was founded in 2016. Stellus describes itself as a developer of solid-state storage systems for next-generation software-defined centers. If there were any doubt about what that might involve, Stellus has been advertising for software engineers experienced in NVMe. This rules out Samsung for M&A in both the short and long term.
The other two giant flash chip and drive makers are Toshiba and Intel. Toshiba's financial problems have just seen it sell off its flash chip-making business, although it has retained its drive-making business. Intel has taken a stake in Vexata, which is one of the vendors described below. However, Intel's practice is to invest in startups to stimulate developments, rather than acquire them outright, because it does not want to compete with its system-making customers.
Potential targets
Apeiron
Apeiron's system has been in 'active' general availability (GA) since 2017. The company claims multiple customers, including two hosting facilities for Splunk, and has named aerospace and electronics behemoth L3 Technologies and the US Food and Drug Administration as customers, and cited two US Department of Defense deployments involving Hadoop and Splunk. Apeiron says it raised over $10m in initial funding, and is currently closing a series A round.
The startup has adopted a stripped-down hardware approach, and claims the result is extremely low latencies, even by the standards of purpose-designed NVMe systems. Its system features no x86 controllers, is powered by standard NVMe drives, and is linked to host servers via a hardened layer 2 Ethernet network using FPGAs to handle network technology it calls NVMe over Ethernet (NoE). Compared with NVMe-oF, Apeiron says NoE is simpler, faster and better suited to its architecture, in which storage services are provided from the host or application, rather than its storage system.
Attala Systems
Attala's system is slated to enter GA later this year, but the company says it has already won two paying customers, and has proofs of concept in progress. It doesn't disclose how much financing it has raised. Attala describes Intel as a strategic partner that has helped with marketing, as well as with other 'significant resources and funds,' without taking equity in the firm.
Like Apeiron, Attala has developed a stripped-down system for which it claims extremely low latencies. The system includes no x86 controllers or storage services, and employs FPGAs to handle NVMe-oF on Ethernet links to host servers. Attala claims its system can support any OS or hypervisor, and is accompanied by cloud-style self-service provisioning and management software.
E8 Storage
E8 released its products into GA in late 2016. The startup says it has racked up production deployments in several verticals such as the financial sector and the travel industry, and has announced test deployments of Intel Optane- and 3D XPoint-powered versions of its system at the Tel Aviv Stock Exchange. E8 has raised $18m in funding.
The vendor's system is based on commodity x86 hardware, and can be bought as software-only or as an appliance. The system eliminates latency in its x86 controllers by providing services such as RAID, snapshots and replication from software running on host servers. A proprietary protocol is used with Ethernet on the links between the system and host servers, but E8 says it could adopt NVMe-oF in the future, when the standard is more mature.
Excelero
Excelero came out of stealth in early 2017, at which time it said it had already scored about a dozen customers, including NASA, PayPal, GE and Hulu, as well as hyperscalers and banks. The company has raised $17.5m in two funding rounds.
Alongside E8, Excelero is the other vendor that has taken a software approach. Unlike E8, Excelero only sells software and does not sell appliances. Its software runs on commodity hardware and provides data services from the host servers. The software can be used to create either standalone or hyperconverged storage. Most deployments have involved standalone storage. Excelero offers a choice of NVMe-oF or a proprietary protocol across InfiniBand or Ethernet.
Pavilion Data Systems
Pavilion has been shipping beta systems since late 2015, and says it has won nine customers. The vendor is not yet naming any of its customers, but says they come from a range of industries, including hosting, SaaS, social media and other sectors. It has raised $15m in one financing round, which occurred in 2015.
We would describe Pavilion's design approach as hardware-focused. The startup's system provides storage services from x86 controllers in the system itself, hence maintaining the highest levels of separation or disaggregation of storage and compute. It avoids the latency that this might impose by using up to 20 array-side pairs of x86 controllers, alongside patented internal networking to deliver what it says is better or equal performance to rival systems. Pavilion notes that apart from the chassis and some of the boards, all of the hardware is commodity gear.
Vexata
Vexata's offering officially became GA in early 2017, and the company says several customers are employing its system in production, including online service provider Oath (AOL/Yahoo), Sanmina and the Tata Group. It has raised $54m, which includes an investment from Intel Capital.
The startup's system uses conventional x86 main controllers, with processing offloaded to FPGAs. The system requires no host software to be installed, and data services running from the array itself include RAID snapshots and replication. Like other systems, the Vexata product can handle NVMe-oF running on Ethernet. Unusually, however, Vexata has chosen to initially support standard Fibre Channel links between its system and host servers, with no usage of NVMe-oF. It believes its customers will be happier with this approach because FC is more familiar to storage administrators than Ethernet, and because the approach requires no specialist network adaptors.
X-IO
X-IO is a long-established supplier of mainstream storage systems, and is therefore the odd one out in this report, as it is not a startup formed to exploit NVMe flash. However, the vendor expects much of its future growth to come from an NVMe appliance it introduced in mid-2017. X-IO's customer base already included government agencies that were deploying its storage systems for high-performance analytics, and the new appliance is aimed at edge analytics.
When the company unveiled the appliance, it said it had already won sales to a handful of customers. The appliance is a 2U dual-socket device that can house up to 72 NVMe drives. That high density is achieved employing PCIe switching and nontransparent bridging. It uses no custom silicon, and can be configured as conventional standalone, hyperconverged or scale-out storage.
Tim Stammers
Senior Analyst, Storage
Tim Stammers is a Senior Analyst at 451 Research, where he covers flash- and disk-based primary storage.
Steven Hill
Senior Analyst, Storage
Steven Hill is a Senior Analyst of Storage technologies. He covers the latest generation of hyperconverged systems, cloud-based storage and business continuity/disaster recovery solutions for enterprise customers.
Speaker Name
Speaker Title
Sed ac purus sit amet nisl tincidunt tincidunt vel at dolor. In ullamcorper nisi risus, quis fringilla nibh mattis ac. Mauris interdum interdum eros, eget tempus lectus aliquet at. Suspendisse convallis suscipit odio, ut varius enim lacinia in. Lorem ipsum dolor sit amet, consectetur adipiscing elit.