Big data: what it is and why now

"Big" is the least of it: the four V's

Before defining anything, the professor shows that survey (p. 4); Fig. 1 is the chart as the slide prints it.

"Large volumes of data" is sixth with 10 %; scope, new kinds of analysis and real time come first. Size is one ingredient among several, and the next four slides do not define big data either: they characterise it along four dimensions, the V's, each with a two-word gloss printed in red and a one-line explanation under it.

Two details of the wording matter. Velocity is about analysis fast enough to decide, "within fractions of a second" — not about fast disks or networks (the course notes say "a time frame of a few seconds"; the slide's wording wins). And Veracity comes with a tilted note on p. 8, "the one certainty about uncertainty is that it is not likely to go away", and on the summary slide (p. 9, Fig. 2) it is drawn as a band under the other three: uncertainty is not a fourth kind of data but a property of all of it.

The same slide adds "More Vs: Value, Volatility, Validity, …" without defining them. Working definitions, mine and not the teacher's: Value, the business worth you can extract; Volatility, how long the data stays relevant and must be kept; Validity, whether the data is correct and fit for its intended use — Veracity seen from the purpose rather than from the noise.

The V's characterise the problem, not the product: Volume "data at scale", Variety "data in many forms", Velocity "data in motion", Veracity "data uncertainty".

Why now: three prices collapsed

"Why now?" is the deck's second question, and the professor answers it with prices, not processors. Page 11 first shows the rate: three infographics of one internet minute — "639,800 GB of global IP data transferred", "204 million emails sent", "2+ million search queries", "30 hours of video uploaded" — and the caption "And Future Growth is Staggering". The data is being produced; the question is why it became worth keeping.

Three charts from Mary Meeker's Internet Trends 2014, with Deloitte data, give the answer (pp. 12–14), each with its endpoints labelled:

• Global data storage costs, 1992–2013, "In US$ per gigabyte of storage": $569 in 1992, $0.02 in 2013 — on a logarithmic axis, gridlines from $0.01 to $1,000.00.
• Global smartphone costs, 2008–2013, "Average price per unit, in US$": $430 to $335, flat until 2011, then a drop — on a linear axis that starts at $300.
• Global bandwidth costs, 1999–2013, "In US$ per 1,000 megabytes per second": $1,245 to $16, again on a logarithmic axis.

The order is a chain: cheap storage makes it reasonable to "cumulate" data (the verb of p. 5), cheap devices produce it, cheap bandwidth moves it. The fourth chart (p. 15) is the consequence, "Digital content universe generated by consumers": more than 4 ZB in 2013 ("+50% Y/Y"), 13 ZB projected for 2016 ("+40% Y/Y"); its footnote spells the unit "zetabyte" — zettabyte, \(10^{21}\) bytes.

On a logarithmic axis equal vertical steps are equal factors, so a straight line means the same percentage fall every year — an exponential collapse. Switch axes in the widget and see the difference.

How fast is "exponential"? From the two printed values you can compute the halving time, the years after which the price is half. The derivation is mine — the slide prints no formula — but every number in it is on the slide.

1. Model a price that loses the same fraction every year, with halving time \(T\): \[ p(t) = p_0 \cdot 2^{-(t - t_0)/T} . \] Why this law: a straight line on a log axis is exactly a constant factor per year.

2. Divide by \(p_0\) and take logarithms, so that the unknown \(T\) comes down from the exponent: \[ \ln\frac{p_0}{p(t)} = \frac{t - t_0}{T}\,\ln 2 . \]

3. Solve for the halving time: \[ T = \frac{(t - t_0)\,\ln 2}{\ln\bigl(p_0 / p(t)\bigr)} . \]

4. Storage (p. 12): \(p_0 = 569\) in 1992, \(p(t) = 0.02\) in 2013, so \(t - t_0 = 21\): \[ \frac{569}{0.02} = 28\,450 , \] \[ \ln 28\,450 \approx 10.26 , \] \[ T \approx \frac{21 \times 0.693}{10.26} \approx 1.4 \text{ years}. \]

5. Bandwidth (p. 14): \(1245 / 16 \approx 77.8\), \(\ln 77.8 \approx 4.35\), fourteen years: \[ T \approx \frac{14 \times 0.693}{4.35} \approx 2.2 \text{ years}. \] On average storage halved every 1.4 years, bandwidth every 2.2 — neither rate was constant, as the widget shows.

Page 16 closes the loop: the "Virtuous cycle of Big Data content", five arrows in a circle, entered where a yellow arrow points — at the prices.

"Virtuous" because nothing in the loop brakes it: each turn makes the next one easier. The deck uses the same word again on p. 38, for the data of a single organisation. One caveat: all these charts stop in 2013, the last year of the 2014 report they come from; the slides give no later figure, so neither does this lesson.

What the three curves bought is the subject of the whole course: once a terabyte costs twenty dollars, spreading data over many ordinary machines beats buying one very big one — horizontal scalability, lesson 3. Keep the price of a gigabyte in mind whenever a later lesson says "replicate it three times".

What it is worth, and who is missing

Pages 17–18 put a price on the opportunity: two posters credited "[source: McKinsey]" with six big numbers, each given a yellow label by the professor.

Two readings. "60 % potential increase in retailers' operating margins" is an increase of the margin, not sixty percentage points: 5 % becomes 8 %. And the two figures about people are almost ten times apart — 140,000–190,000 analysts against 1.5 million managers able to use what they produce. Value is realised when somebody acts on an analysis; keep that ratio for the capability chart of section 4. (Background: 2011 McKinsey Global Institute estimates of potential.)

Pages 32–33 show where the money went. Page 32 is a wall of logos (Matt Turck's 2016 "Big Data Landscape", as background); p. 33 is the professor's abstraction of it (Fig. 4): four layers — Data Sources, Infrastructure, Analytics, Applications — and two vertical bands across all four, "Full data stack providers" and "Opensource platforms" (the slide's spelling). Much of this course sits in the open-source band of p. 32: Hadoop, YARN, Spark, Pig, Hive, Cassandra, MongoDB, HBase, Storm, Lucene, Elasticsearch. Page 34 adds the enterprise AI market, organised by business function rather than by technology.

The last market slide (p. 35) is Gartner's hype cycle for big data, "As of July 2012": expectations against time through five phases — Technology Trigger, Peak of Inflated Expectations, Trough of Disillusionment, Slope of Enlightenment, Plateau of Productivity. Learn the five names. The positions are a snapshot: in July 2012 "noSQL Database Management Systems" were still climbing towards the peak, "MapReduce and Alternatives" were sliding into the trough, "Column-Store DBMS" and "Predictive Analytics" were almost on the plateau. Quote them as 2012 positions, never as today's.

Why a market map and a hype curve in a lecture about data? Because there is no single big-data product: the past papers' one-point table asks you to pick a specialised database technology for each part of the data. And 8 % of the answers on p. 4 called big data "the latest buzzword": the hype cycle shows that part of it was.

Value is a loop, not a pile

Page 37 is a Dilbert strip. The boss: "I need you to assemble a huge amount of totally incomprehensible data. Make it boring so no one looks at it too closely. I'm aiming for quantity over quality." The slide's title calls the result "The Big Data Deluge": Volume and nothing else — no analysis, no decision, no value.

The alternative is a loop. Page 38, "The Data-driven Virtuous Cycle", is three circles joined by arrows: Data Ingestion → Data Analysis → Value Generation → back to ingestion; the value pays for more ingestion, the pile never does. Pages 40–42 turn the loop into a working method — collect, analyse, build value — on a city-analytics example (Fig. 7).

Between the two sits the chart that carries the argument (p. 39, Fig. 6): "Data vs. Analytical vs. Execution Capability".

The argument, in the words of the course notes (section 2.1): "The available data is an upper bound for the analytical capability of a company, which in turn is an upper bound to the execution capability." You cannot analyse what you do not have, nor act on what you have not analysed. Why does the gap widen? Because the supply of data follows the exponential price collapse of section 2, while the two capabilities depend on trained people and organisational change, which grow linearly at best — the 1.5 million managers of p. 18. The implication to state in an answer: value comes from investing in the two lower curves, not from collecting more data.

One pair in Fig. 7 is worth knowing by name: the two kinds of analysis on p. 41, Description and Prediction — the deck's only taxonomy of analytics. Description tells a stakeholder what happened; prediction lets them act before it happens, which is where execution capability comes in.

Two roles: the data scientist and the data engineer

The exam guideline lists "Data Science and Data Engineering" among the Foundations topics, but no slide of this deck defines the roles; the course notes do (section 2.3, "The Roles in the Data World"), so this part is from the notes.

Mapped onto Fig. 7: the data engineer is the "collect" circle and the plumbing between the circles — the lever that lifts the analytical-capability curve by making data "always available and usable"; the data scientist is the "analyse" circle. This course is mostly data engineering, with data-science touchpoints (vectors, feature stores, Spark): deep-learning models sit at the top of the pipeline, and everything below them is this course.

Available data bounds analytical capability, which bounds execution capability; the gap widens because data grows exponentially and people do not.

Netflix: the worked example

Pages 43–56 are the "Motivational Use Case", and most ideas of the deck come back in it. Scale (p. 44): "Netflix streaming service accounts for 1/3 of peak time internet traffic in the US"; "Last year, Netflix announced that it signed on 50 million accounts worldwide". Variety of streams (p. 45): "Personalization analytics", "Messaging analytics", "Content delivery analytics", "Device analytics".

Then the before and after (Fig. 8). "Back in 2006 …" (p. 46) shows a DVD sorting centre and a red envelope in a mailbox. Page 47: "Netflix was a DVD delivery in need for data analysis but with only 4 data points: Customer ID, Movie ID, Rating, The date the movie was watched". Page 48: "As streaming video became the primary focus → more insights on the customers" — time of day, "User gender and age (based on individual logins)", "Time spent selecting movies", "How often a movie or program was paused".

With the richer row, analysis (p. 49): "Netflix can now analyze how these factors impacted viewers' enjoyment"; "The happier the customers, the longer they stay"; "Models to predict the "perfect situation" for the customer experience" — the Data Analysis circle of p. 38 with a business reason, retention. The products of the analysis:

• Categories (p. 50): "Tagging", "Preferences", "76,897 ways to describe types of movies", "alt-genres" such as "Comedies Featuring a Strong Female Lead".
• A content bet (pp. 51–52): Netflix took House of Cards "outbidding major cable networks with a massive two-season order", because "by analyzing its data, Netflix saw that a large majority of its viewers enjoy programs directed by David Fincher … starring Kevin Spacey. House of Cards was exactly that!" No pilot was shot (background, not on the slide): the data played its role.
• Promotion (p. 53): "the company modelled the show's cover image on the colours and styles for successful, similarly tagged programs, to help draw new viewers in" — one cover, designed from data.
• Cover personalisation (pp. 54–55): nine artworks for one series; the same film shown to a romance viewer through its kiss and to a comedy viewer through its comedian. The slide repeats the sentence of p. 53, but the mechanism differs: one of several covers is chosen per viewer, not one cover designed for everybody.

The conclusion (p. 56): "Netflix disrupted the TV industry using data science to provide viewers with exactly the content they want"; Uber, Airbnb and Amazon do the same — "They are all "data companies"". A data company is one whose product is the loop of p. 38 running on its own users. The "perfect situation" models are supervised learning of the kind a deep-learning course trains, fed with implicit feedback; finding the titles closest to what a viewer watched is the similarity search of lesson 27; storing and moving all that data is everything in between.

Big problems, and the puzzle resolved

The deck does not end on the success story. "Big Data = Big Problems?" (p. 57) is followed by a Netflix screen (p. 58) asking "Notice anything?" with no answer: a row titled "PICKS FOR YOU" — The Hunger Games, Pirates of the Caribbean, Arrow … — with House of Cards enlarged in its centre, wearing a "NETFLIX ORIGINAL" badge. My reading, which the slide leaves to you: a political drama sits in the middle of an action-and-adventure row because it is the platform's own product. The recommender serves the company's interest as well as your taste — the third item of the last slide.

"Problems" (p. 59) lists four, learn them as printed:

1. Data access — the data exists but is owned, siloed, regulated or priced; the sources of p. 40 (telco, transport, WiFi logs) are not public.
2. Noise and trustworthiness — Veracity (p. 8) as a problem; the wrangling lessons (35–37) are its treatment.
3. Algorithmic (and many other types of) bias — the platform's interest in p. 58, but also who is on the platform at all ("all of Twitter" is not all of the population) and what the labels encode: the algorithm is only the last link.
4. Cost of maintenance / problem solving — the pipeline is never finished: sources change, schemas drift, models decay. Much of that is the data engineer's job.

Now the hook. What did the other nine answers in ten mean? The rest of this deck. "A greater scope of information" is analysing more of what is captured; "new kinds of data and analysis" is Variety and exploration; "real-time information" is Velocity; "data influx from new technologies" is the devices and bandwidth of section 2; "non-traditional forms of media" and "social media data" are Variety again; and the 8 % of "buzzword" answers fit the hype cycle. Netflix shows them all at once, and the bias problem that comes with them.

Three things from the deck are deliberately not here: the answer to "How?" — the slides "Solving problems in new ways", the "4 paradigmatic shifts" of the exam guideline — is lesson 2; scalability, the engineering consequence of cheap storage, is lesson 3; schema-on-read is lesson 4.