Big Data. All You Wanted to Know but Were Afraid to Ask

"Imagine that you are able to see with only one eye. Then suddenly there appears an opportunity to see with two or even three eyes. You'll be amazed not with the increase of information on the world around you, but with the realization that everything around you is connected and one thing continues another one..."

On<U+0441>e again we can say hello to the Big Brother, the one that is spying on us. Recently the whole world was simply astonished by Argus IC the new camera that’s watching US citizens.

That’s the eagle eye of Big Brother, fear him!

When talking about things that concern everyone, Big Data is definitely something to talk about. Probably you’ve heard about it, if not, here is a fragment from Wiki.

"In information technology, big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data .processing applications. The challenges include capture, curation, storage, search, sharing, analysis, and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions."

The notion of Big Data was first applied when discussing the decoding of human DNA. Scientists needed excessive computing power to store these volumes of info. As a result, we can cram 700 terabytes of data into 1 gram of DNA. Looks like all your Blu Ray movie collection can be stored in 1 gram of DNA memory, pretty awesome; I’m looking forward to the day when such data carriers will be on sold in shops.

***

---

---

In the next few years the world market of solutions and services related to Big Data, will grow 31.7%, that’s 7 times more than the global IT and telecommunications market. This is according to the research firm IDC.

According to IDC, by 2016 the world market of solutions and services related to Big Data will reach $23.8 billion.

Earlier, Gartner company analysts predicted that by 2015, Big Data will create 4.4 million new vacancies (more data requires more workers). Experts believe that starting from 2014 Big Data will play the role of locomotive in the industry. Big Data was referred to as one of the key technologies in 2013.

In addition to the total market, analysts talked about the key trends. They said lack of necessary skills for working with Big Data will force organizations to focus on automated solutions and outsource research services.

Experts: Big Data Keeps on Making More "Noise"

All vendors, without exception, on the market of data management are developing technologies for managing Big Data. This new technological trend is being actively discussed by professional community, developers, industry analysts and by potential users of these solutions.

As it turned out the wave of discussion around Big Data exceeded all imaginable sizes. By analyzing the number of mentions of Big Data in social networks, Datashift found out that during 2012, the term was used about 2 billion times in posts created by 1 million different authors around the world. This is the equivalent of 260 posts per hour, with a peak of references of 3070 per hour.

The Global Technology Market of Big Data

According to IDC’s forecast (Worldwide Big Data technology and services forecast), released in March 2012, the market of technologies and services for processing large volumes of data will grow from $3.2 billion in 2010 to $16.9 billion in 2015. This corresponds to an average annual growth rate (CAGR) of 40%, which is about seven times more than the average annual growth rate of the ICT market in general.

According to IDC, in 2012 there has been about 2.43 zettabytes of digital content generated, which is 48% more than in 2011. Ninety per cent of this information is unstructured data (images, video, mp3 files, and social media), i.e. data that is very difficult to analyze:

Image Source: Big Data

A study provided by LogLogic, showed that 49% of IT managers believe their companies are ready to deal with Big Data. However 38% admitted they haven’t fully acknowledged the essence of this phenomenon.

Leading Players on the Market

All of the leading IT companies show interest towards tools aimed at collecting, processing, managing and analyzing big data, which is quite natural. First of all, big data directly interferes with their business, and second of all big data offers excellent opportunities for development of new niche markets and attracting new customers.

• Amazon
• Dell
• eBay
• EMC
• Facebook
• Fujitsu
• Google
• Hitachi Data Systems Corporation
• HP
• IBM
• LinkedIn
• Microsoft
• NetApp
• Oracle
• SAP
• SAS
• SGI (Silicon Graphics Inc)
• Teradata
• VMware
• Yahoo

The New Wave of Start-ups

Recently there appeared lots of startups that make money by processing huge amounts of data. Some of them use a ready-made cloud infrastructure provided by big players like Amazon. If you use Google you’ll be able to find dozens of other companies besides the ones featured below.

• Acunu
• Apigee
• Aspera
• Aster Data (consumed Teradata)
• Cloudera
• Couchbase
• Datameer
• DataStax
• Factual
• GoodData
• Greenplum (consumed <U+0415><U+041C><U+0421>)
• Hortonworks (has general roots with Yahoo)
• MapR Technologies
• Metaweb Technologies (consumed Google)
• Netezza (consumed IBM)
• nPario
• Palantir Technologies
• ParAccel
• ParStream
• SenSage
• Socrata
• Splunk
• Sybase (consumed SAP)
• TellApart
• Vertica (consumed <U+041D><U+0420>)

Who Benefits from Big Data?

It’s difficult to find an industry for which the issues of big data would be irrelevant.

• the ability to handle large amounts of data, analyze the relationship between them and make informed decisions has the potential for companies from different verticals to increase profit and effectiveness.
• this is a great opportunity for additional income for vendor partners - integrators and consultants.

How Customers Will Benefit from Big Data?

Sean Madden described the best example of how Big Data can help the regular consumer. Sean talked about his experience with Amazon support. He had to call them because of his malfunctioning Kindle. The way he was treated amazed the guy and made him write the article. Due to the practical implementation of Big Data the support agent was able not only to help Sean by predicting some of his actions and questions, but to HELP HIM and fix the Kindle.

With the help of Big Data you can not only have your Kindle fixed, there are dozens of other possible ways to use it. Here is a list of the most important:

Healthcare and medicine, public services, retail trade, various manufacturing industries, telecommunications, IT, finance and insurance, energy, marketing, publishing, media and entertainment, biotechnology, transportation, logistics, etc.

You can continue this list as long as your imagination allows you to.

Big Data Spans in Four Dimensions: Volume, Velocity, Variety, and Veracity

Volume: Enterprises are awash with ever-growing data of all types, easily amassing terabytes—even petabytes—of information.

• Turn 12 terabytes of Tweets created each day into improved product sentiment analysis
• Convert 350 billion annual meter readings to better predict power consumption

Velocity: Sometimes 2 minutes is too late. For time-sensitive processes such as spotting fraud, big data must be used as it streams into your enterprise in order to maximize its value.

• Scrutinize 5 million trade events created each day to identify potential fraud
• Analyze 500 million daily call detail records in real-time to predict customer churn faster

Variety: Big Data is any type of data - structured and unstructured data such as text, sensor data, audio, video, click streams, log files and more. New insights are found when analyzing these data types together.

• Monitor 100s of live video feeds from surveillance cameras to target points of interest
• Exploit the 80% data growth in images, video and documents to improve customer satisfaction

Veracity: One in three business leaders don’t trust the information they use to make decisions. How can you act upon information if you don’t trust it? Establishing trust in big data presents a huge challenge as the variety and number of sources grows.

***

Big Data is positioned as a tool to create new competitive advantages. No doubt that application of analytical tools in the context of Big Data will lead to profound "mutations" of existing methods and development of new analytical approaches. Analysis of Big Data will strongly affect the accuracy of decisions and at the same time significantly accelerate decision-making.

In a world where the amount of data doubles every two years, competitive advantage will be on the side of those who are able to manage the info effectively.

SPEAK UP! So, what would you say. Did you have any experience with Big Data. If non, share your thoughts about this trend. How do you think, when analysts will finally get some ultimate tools for scanning those data how will it effect humanity in general?