IBM is working on both super-fast and super-dense storage media that should reach enterprises before the end of this decade, and demand in some industries looks likely to keep pace with the advances.
The amount of data that enterprises have to deal with is growing so fast that they are coming up against power and space limits and grappling with how to manage the information, according to analysts, users and IBM officials at a press event that the company hosted in San Francisco on Wednesday.
Storage Class Memory
IBM thinks “racetrack” technology will pave the way for Storage Class Memory, which will be nearly as fast as today’s memory but be able to scale up to enterprise storage capacity. It’s so dense that it could allow a portable music player to hold 500,000 songs.
With Storage Class Memory, the amount of space and power required to store large enterprise data sets could shrink dramatically by 2020. What would take 1,250 racks of hard disk drives today could be stored in one rack, with less than one-third the energy, Hillsberg said. The medium will also last longer than the flash SSDs (solid-state drives) now used for fast storage, he said.
A Petabyte in a Rack Unit
The other device IBM is working on, which it called simply the “petabyte storage device,” would be designed to store data for as long as 50 years without the need for migration to another medium, Hillsberg said. Moving content from one device to a newer generation of hardware costs money and time and can introduce record-keeping errors, he said.
Without giving more details about the device, Hillsberg said it would have some moving parts but not as many as a tape library, the typical solution for long-term archiving today. He envisions it being used not as a replacement for tape but in conjunction with it, possibly to provide access to archived data over a cloud infrastructure.
Hollywood’s Storage Woes
The movie industry is now grappling with dramatically larger data sets on the scale that the petabyte device is designed to address, according to Peter Ward, a digital entertainment consultant and former senior vice president of Sony Pictures Entertainment. As filmmakers gradually swap reels of film for SSDs, a day of shooting can generate hundreds of terabytes of data, Ward said. The 2011 Facebook drama “The Social Network” is one recent film that was shot digitally, he said.
That data has to be stored and secured on location and sent each day to post-production facilities, which is done by transferring it to tape and physically shipping it, because networks aren’t fast enough, Ward said. The footage and associated metadata is used throughout the editing of the movie, and is then compressed into formats for use in theaters and homes. But studios want to keep all the original footage, just as they used to keep the uncut film, to use years later for sequels and other projects, he said. A 3-D movie may take up 1 petabyte of capacity, and the industry is still working on how to handle that.
“There isn’t a digital archival media that meets the archivists’ standards,” Ward said.
Health-care Data Demand
The health-care industry is facing the prospect of even more data than the entertainment business, according to Paul Markham, vice president of global strategy and marketing at TeraMedica. On Thursday, his company is introducing a medical archiving system based on IBM hardware and software. The medical records of a patient in the U.S. represent on average about 1 terabyte of data today, and one 300-bed hospital may generate 30T bytes of data per year, Markham said. Those figures will only grow with higher-resolution medical imaging, he said.
“It’s such a tsunami of data, they just don’t know what to do with it,” Markham said. He believes the future technologies IBM envisions will be necessary to keep up.
Speed Is a Lower Priority
IBM’s proposed long-term storage device may be more crucial for enterprises than will Storage Class Memory, said Forrester Research analyst Andrew Reichman. Many IT departments have more speed than they need, because they want to make sure storage access doesn’t slow down applications. The bigger problem is how to deal with the data they need to keep for a long time, he said. That involves sunk costs in real estate and energy, as well as management.
“Being able to store a petabyte in 1U is going to be massively valuable,” Reichman said.
However, the biggest challenge of the moment for most enterprises is knowing what data to put in what form of storage, for the benefit of application users, he said. Application and storage managers need to communicate better to make that possible.
“There’s always going to be a faster bucket. … But what will never change is that it’s hard to organize your stuff,” Reichman said.