Four ways of solving problems differently

Big Data. How? Two panels, four times

Lesson 1 asked what big data is and why it happened when it did. The next divider of the deck asks "Big Data. How?" (p. 19), and the answer fills twelve pages under one title, "Solving problems in new ways" (pp. 20–31). Every shift is built the same way. A sub-title names the change. On the left, a grey arrow-shaped panel is labelled "TRADITIONAL APPROACH"; the next build adds a panel labelled "INNOVATIVE APPROACH" on the right; a short caption under each panel states the approach. For three of the four changes, one more slide repeats the captions under two photographs: an everyday version of the same contrast, something done by hand, a little at a time, against something that runs on everything, all the time.

The four sub-titles are the four paradigm shifts. The deck's agenda calls them "Paradigm shifts enabled" (p. 2), and the exam-guideline deck lists "4 paradigmatic shifts" among the theory questions on the foundations. This deck is the only place in the course that lists exactly four, so learn them as the sub-titles print them, each with its two captions.

The capitals inside the captions ("Before", "And", "Generated") mark the slide's line breaks; in an answer, write ordinary sentences.

Four shifts: leverage more of the data being captured; data-driven exploration looking for correlation; reduce effort required to leverage data; leverage data as it is captured.

Shift 1 · Leverage more of the data being captured

The first pair is the picture of the hook (Fig. 1). Both panels hold the same forty dots, "All available information", as the label over the traditional grid says. The traditional approach puts a dashed box around six of them and analyses only those: "Analyze small subset of information". The innovative panel draws the box around all forty: "Analyze all available information".

Why did anyone analyse six dots out of forty? Because keeping and processing everything used to be expensive, and statistics offers a sound shortcut: a well-chosen sample estimates the properties of the whole. Lesson 1 showed what removed the reason for it: storage fell from $569 per gigabyte in 1992 to $0.02 in 2013 (p. 12). Once keeping everything costs almost nothing, the question becomes what the shortcut was losing.

The photo slide (p. 22) gives the answer a face. On the left, a yellow measuring gauge lies across the rails in front of a maintenance train: a crew measures a stretch of track on the day it visits. On the right, a black sensor unit is fixed to the rail and shines red beams across it: it measures whatever passes, every time. Between the two photos sits the shield of the American railway Union Pacific, and no other word.

Shift 1 in one sentence: a sample is good at what is common and blind to what is rare. The widget measures how blind.

The readout's number comes from a counting argument, not on the slide but elementary:

1. Swap the roles. Scattering the defects at random under a fixed box gives the same chance of a miss as fixing the defects and drawing the \(k\) items of the sample at random. Choosing \(k\) items out of \(N\), without order, can be done in \(\binom{N}{k}\) ways, all equally likely.

2. Count the samples that see no defect. All \(k\) items must come from the \(N - m\) good ones: \(\binom{N-m}{k}\) ways.

3. Divide the favourable cases by all cases: \[ P(\text{miss}) = \frac{\binom{N-m}{k}}{\binom{N}{k}} . \]

4. Put in the slide's numbers, \(N = 40\), \(k = 6\), and three defects: \[ \frac{\binom{37}{6}}{\binom{40}{6}} = \frac{2\,324\,784}{3\,838\,380} \approx 0.61 . \]

5. Put in the innovative panel, \(k = 40\): you cannot choose 40 good items out of 37, so \(\binom{37}{40} = 0\) and \(P(\text{miss}) = 0\). In fact any \(k \ge 38\) leaves room for at most two defects outside the box: the census cannot miss.

So the miss probability falls slowly (a box holding half the data still misses all three defects 11.5 % of the time), and the position of the box does not matter when the defects could be anywhere. There is no clever sample, only a bigger one, and the biggest is "all available information".

Leverage more of the data being captured: analyse all available information instead of a small subset, because a subset misses what is rare.

Shift 2 · Data-driven exploration looking for correlation

With all the data at hand, the order of the work can be turned around. The second pair (p. 25, Fig. 2) draws each approach as four steps joined by three arrows. Traditionally you start from a hypothesis, turn it into a question, select the data that can answer it and get an answer: "Start with hypothesis and test against selected data". The innovative path starts from the data, explores it, identifies correlations and ends in an insight: "Explore all data and identify correlations".

The traditional path is the one a statistics course teaches: state the hypothesis, collect data designed to test it, run the test, reject or not. Its strength is that the answer means something, because the data were chosen to answer that question. Its limit is that you only get answers to the questions you thought of asking. The innovative path looks for structure nobody expected; its output is not "yes" or "no" but an insight, a relation that someone must then interpret.

This shift needs the first one: a pattern found by searching a small sample is easily a fluke of that sample, while the same pattern across all the captured data is worth a closer look.

The third build of the slide pastes a green call-out onto the innovative path: a grey box labelled "nature", an arrow into it from "x", the "independent variable", and an arrow out of it to "y", the "response variable", drawn right to left. The slide does not explain it. The picture comes from Leo Breiman's 2001 paper "Statistical Modeling: The Two Cultures" (background, not on the slide), where it frames two cultures of statistics that match the two paths of Fig. 2, as Fig. 3 shows.

In the first culture you open the box: you assume a mechanism, say a linear model, and test its parameters. That is the traditional path. In the second you leave nature closed and look for whatever relation predicts y from x. That is the innovative one, and neural networks belong to it: they predict y from x without stating why.

The risk has a name you know: correlation is not causation. Two stories from outside the slides: in 2008 Chris Anderson wrote in Wired, in an essay called "The End of Theory", that "with enough data, the numbers speak for themselves"; a few years later Google Flu Trends, which estimated flu activity from search data, overestimated it by about a factor of two in 2012–13 (Lazer and colleagues, Science, 2014). Searching many variables for correlations is bound to turn up some that mean nothing. The remedy is to send an insight back through the traditional path: treat it as a hypothesis and test it on new data or with an experiment. The two paths complement each other.

Data-driven exploration: start from all the data and look for correlations, instead of testing one hypothesis against selected data. Then treat what you find as a hypothesis, not as a cause.

Shift 3 · Reduce effort required to leverage data

The third pair (p. 27, Fig. 4) is about the work that comes before any analysis. In the traditional panel two small piles of mixed dots go up two arrows into a dashed box holding nine identical dots: "Small amount of Carefully organized information", with the caption "Carefully cleanse information Before any analysis". The innovative panel holds forty dots, clean and odd ones mixed: "Large amount of messy information", and the caption "Analyze information as is, Cleanse as needed".

The photo slide (p. 28) puts a scientist in a white coat and blue gloves, pouring a blue liquid from one flask into another, on the left, and an industrial chemical plant of towers and pipes on the right: purification by hand, a little at a time, against raw material processed in bulk.

Why give up cleaning first? It is the price of shift 1. Cleansing is expensive work: fixing formats, removing duplicates, filling gaps, reconciling sources. When you keep everything, cleansing everything up front costs more than most of it will ever be worth. Worse, you do not yet know which analyses will come, so you cannot know what "clean" means for them; as p. 169 puts it, "Some analysis can no longer be performed because the data were lost at writing time". "Cleanse as needed" moves the work to the moment an analysis runs, and limits it to the data that analysis reads.

The same idea returns in lesson 4 as a property of the data model, schema-on-read, which pp. 169–170 call a "Paradigmatic shift introduce by Big Data" and sum up as "Load data first, ask question later": shift 3 is the method, schema-on-read the mechanism that lets a database store data before anyone has decided their final shape.

Reduce effort required to leverage data: analyse information as is and cleanse it as needed, instead of carefully cleansing a small amount of it before any analysis.

Shift 4 · Leverage data as it is captured

The last pair (p. 30, Fig. 5) is the simplest picture of the four. The traditional panel is a chain of four circles climbing to the right, Data, Repository, Analysis, Insight, with the caption "Analyze data after it's been processed And landed in a warehouse or mart". The innovative panel has three circles: Data, Analysis, Insight. The repository, drawn as a database cylinder, is simply gone, and the caption reads "Analyze data in motion as it's Generated, in real-time".

A warehouse or a data mart is a store built for analysis; lesson 23 opens one. Data reach it through scheduled loads, processed and landed, and the analyses run after the load. The photo slide (p. 31) shows the difference in time: a seismograph pen tracing a wave on paper, a record made now and read later, against a phone running a trading app: a live price chart and two buttons, CALL and PUT, for a decision taken on the price of the moment.

What does the cylinder cost? Watch it fill up:

The widget's averages follow from one line of reasoning:

1. Batches run every \(T\) seconds. An event that arrives at a random moment of a window waits until the window closes: its wait \(W\) lies anywhere between \(0\) and \(T\), all values equally likely.

2. So the average wait is half a window, \(\E[W] = T/2\); the worst case, an event that arrives just after a batch, waits almost \(T\).

3. The batch then takes the processing time \(p\), so the average time to insight of the traditional chain is \[ \bar{L}_{\text{batch}} = \frac{T}{2} + p . \]

4. In motion there is no window to wait for: \(L_{\text{stream}} = p\). The difference, \(T/2\), is the price of landing first.

The delay of the traditional chain is therefore set by the schedule, not by the volume. This is the method behind Velocity in lesson 1, "Data in motion": "Analysis of streaming data to enable decision within fractions of a second" (p. 7). The same two panels return as "Batch data" and "Real-time streaming" in the Spark exercise session (p. 1719, lesson 39), and the streaming tools of the Hadoop world, Storm and Flume, in lesson 33.

Leverage data as it is captured: analyse data in motion, as they are generated, instead of after they have been processed and landed in a warehouse or mart. The repository is the delay.

One answer in four parts

Read in order, the four shifts form a chain. Cheap storage made it possible to keep everything, so you analyse all of it (shift 1). With all of it, you can explore instead of only confirming (shift 2). Keeping everything means you cannot afford to clean everything first, so you analyse it as it is and cleanse as needed (shift 3). And when a decision cannot wait for the next scheduled load, you analyse the data as they arrive (shift 4).

Each shift also answers one V of lesson 1. The pairing is a memory aid of this notebook, not a slide: it helps you recall the two lists together and keep them apart.

Nothing on the slides says the left panels are obsolete, and they are not. A bank can analyse every card payment as it happens to block fraud, and still load the same payments every night into a warehouse for the monthly report: both chains of shift 4, each for its own question. A correlation found in all the data still gets tested on a sample or in an experiment, the traditional path of shift 2. The innovative approach complements the traditional one: it adds options where the traditional one was too slow, too narrow or too expensive.

Two related statements appear elsewhere. Schema-on-read is called a "Paradigmatic shift" on p. 169, the data-model face of shift 3 (lesson 4); the course notes (section 3.1.1) describe the hardware behind keeping everything, "going from the typical vertically scalable, on premise mainframes to cloud based, horizontally scalable commodity machines" (lesson 3). The guideline's "4 paradigmatic shifts" are the four sub-titles of this lesson, the only place that lists four; the other two make a good closing sentence, not a replacement.

The puzzle, resolved

The hook asked how often a six-dot box misses three defective dots among forty. The answer is 61 %, three times in five, wherever the box stands, because the defects could be anywhere: \(\binom{37}{6}/\binom{40}{6} \approx 0.61\). That is why the slides' answer is not "sample better" but "Analyze all available information". Once you do, the other three shifts follow: you explore what all the data say, you stop cleaning everything first, and you stop waiting for the data to land. The Union Pacific story can be read through all four: every wheel is measured; the software looks for the patterns that come before a failure; the readings of microphones and infrared sensors are messy data used as they come; and the warning is worth something only if it arrives before the derailment.