Big Data and the Cloud: A Far from Perfect Fit

Big data seems to be all the rage these days. It's big, it's new, it's Hadoop-related, and it's typically in the public cloud. New startups and cloud offerings show up weekly, promising they'll finally get your data issues under control. They all promote the same idea: the migration to huge petabyte databases with almost "unlimited scalability" through the elasticity of the public cloud.

The reality is different than the hype lets on. As organizations try to consolidate their enterprise data into large databases that exist in the public cloud, they may be overlooking a few technical realities.

First, big data means big integration challenges. Thus, the ability to get the data from the enterprise to the public cloud may be problematic. Although you can certainly ship up a couple hundred thousand data records each day over the open Internet, in many cases we're talking millions of data records that must be transformed, translated, and synced from existing enterprise systems.

You bump your head on the bandwidth limitations quickly. Indeed, many enterprises are actually shipping USB drives via Federal Express to their public cloud providers to pump their big data system with current data.

Second, although security is indeed what you make of it in the cloud, it's typically cheaper to deal with data-level security in on-premise systems or in private clouds. In many cases, their security models and technology are less expensive. For example, you may have to encrypt data at rest in the public cloud -- but not in your old data center. Compliance is typically easier and cheaper when keeping data locally as well.

By the way, before you send in the hate mail, I'm not saying that big data is never a fit for public clouds, but you need to consider all the issues with the technologies. As with other architectural problems, you have to judge them on a case-by-case basis.

This article, "Big data and the cloud: A far from perfect fit," originally appeared at Read more of David Linthicum's Cloud Computing blog and track the latest developments in cloud computing at For the latest business technology news, follow on Twitter.

This story, "Big Data and the Cloud: A Far from Perfect Fit" was originally published by InfoWorld.

To comment on this article and other PCWorld content, visit our Facebook page or our Twitter feed.
Shop Tech Products at Amazon