Why is Spark on the rise? Does it meet business needs?

With Strata + Hadoop World soon upon us it is clear that Spark is the hot topic – see our #AnalyticsNews article

But Why? What does it represent?

I would posit that this is driven by business wanting a lot more from Hadoop… essentially a lot more throughput. Whether it be better concurrency or reduced latency or better job efficiency business is seeing value and thus demanding more use. Hooray say the engineers! More APIs to play with and code to develop. But this drive will be tempered by business impatience – “Sorry? How long to code that analytical model efficiently so it does not dominate the cluster resources?”

Business managers mentally-extrapolate new tech to commodity in a thought. They just want to switch on and use the analytics capability on their swelling lake of data. They just want to plug in standard tools and run studies…”What? I have to wait for you to develop this?”

Spark is young and developing and will become a formidable force…in time. I liken it to a box of Lego(TM). The developer sees this


The business user wants this


and this


The business manager wants pre-packaged; ready to use out of the box; flexible; scaleable; highly efficient; low latency; high throughput; engineered for analytics? It’s so much easier for the business if they can just switch on and use. Sorry Hadoop developers, there’s an easier way. Get to know Kognitio.

Paul Groom, COO