MIT demonstrates more efficient big data solution

Researchers at the Massachusetts Institute of Technology (MIT) have unveiled a new network design that they claim could make big data analytics operations cheaper and less power intensive, without compromising on performance.

The technique involves integrating flash memory into big data applications in such a way as to overcome the speed deficit that the technology has when compared to traditional RAM-based in-memory computing.

MIT observed that flash memory is typically about a tenth as expensive as RAM and consumes around a tenth as much power. But the trade-off of this is that it is only around a tenth as fast.

However, researchers at the university presented a new system at June's International Symposium on Computer Architecture in June that should make servers using flash memory as efficient as those using conventional RAM for several common big-data applications, while preserving their power and cost savings. 

The researchers were able to make a network of flash-based servers competitive with a network of RAM-based servers by moving some of the computational power off the servers and onto the chips that control the flash drives. "By preprocessing some of the data on the flash drives before passing it back to the servers, those chips can make distributed computation much more efficient," MIT stated.

Arvind, the Johnson Professor of Computer Science and Engineering at MIT, said that while the process – called BlueDBM – is not a replacement for solutions such as dynamic RAM, it offers up many new opportunities for managing large volumes of information.

"There may be many applications that can take advantage of this new style of architecture which companies recognise," he said "Everybody's experimenting with different aspects of flash. We're just trying to establish another point in the design space."

Jihong Kim, a professor of computer science and engineering at Seoul National University, added that the architecture may be particularly appealing for big data applications that require very fast or real-time responses.

"The main advantage of BlueDBM might be that it can easily scale up to a lot bigger storage system with specialised accelerated supports," Mr Kim continued.