Getting data from the web: APIs and scraping

An API is a counter where programs ask for data

The course notes call data ingestion "the first and fundamental step of any Data Analysis Pipeline" (§5.1.1) — the bottom layer of the architecture of lesson 4, and the step that a ready-made dataset does for you in most courses. From the web there are two ways in: ask an API, or scrape the page.

Picture a take-away counter: a form with fixed fields goes in, a labelled box comes out, and you never see the kitchen. The first slide defines this counter for software (p. 1510):

The definition is general: the functions of a Python library are an API too. "Remotely" narrows it to the web API, called over the network with HTTP — in the course notes, "A Web API is a API which is based on the HTTP protocol, and it's usually used to give programmatic access to data and platforms" (§5.1.2). "API without Libraries", the title of the first two slides, is not explained on them; read it as calling the API directly — a URL in the browser, in Postman or with curl — rather than through a client library that wraps the calls.

Every call names an HTTP method and a URL: "APIs can be invoked through HTTP methods (e.g., GET, POST, etc.) depending on the kind of operations to be performed", and "As an end-user, the most used requests for APIs are GET requests" (p. 1511). GET reads, and whoever collects data mostly reads; it also fits in one URL, which is why every example of the session is a line you could paste into a browser.

To develop and test APIs, "Postman is one of the most renowned ones" (p. 1511). The next slide is a screenshot of it with four red callouts, redrawn in Fig. 1.

Read the four parts in the order you use them. The API Collection keeps saved requests; "API 101" has one HTTP method per operation on customers — GET to read all or one, POST to add, PUT to update, DEL (DELETE) to remove. The Query callout points at the request line, method and URL. Request Parameters lists the key=value pairs after the ?. The Result holds status 200 OK, time, size and a JSON body — and since the server is an echo, the body shows what really travelled: your parameters as args, plus headers the client added on its own.

A web API answers a method and a URL with data for programs — usually JSON — not with a page for people.

Reading a request: the iTunes Store examples

The session's first worked example asks Apple's iTunes Store to "Retrieve the first 10 artists whose name contains "maroon"" (p. 1513). No app, no login: the whole question is one URL.

GET https://itunes.apple.com/search?term=maroon&entity=allArtist&attribute=allArtistTerm&limit=10

Read it left to right: https is the scheme, the protocol; itunes.apple.com the host, the server that answers; /search the path, the service you call. After the ? comes the query string: key=value pairs separated by &, each an instruction for the search. The course notes print the general shape (§5.1.2); the bracketed parts are optional:

URL format (RFC 2396):
scheme:[//[user:password@]host[:port]][/]path[?query][#fragment]

The answer is not a page but a JSON document: an object with resultCount and results, an array of artist objects. Its start, as the slide prints it:

{
  "resultCount": 10,
  "results": [
    {
      "wrapperType": "artist",
      "artistType": "Artist",
      "artistName": "Maroon 5",
      "artistLinkUrl": "https://music.apple.com/us/artist/maroon-5/1798556?uo=4",
      "artistId": 1798556,
      "amgArtistId": 529962,
      "primaryGenreName": "Rock",
      "primaryGenreId": 21
    },
    … nine more artist objects ("Maroon" is the second)
  ]
}

resultCount is 10 because limit=10 allowed at most ten; each element of results is a record with named fields — structure an API gives you for free, and scraping must rebuild from a page layout.

The second call, "Retrieve all the artist whose surname is "Stallone"" (p. 1514), is GET https://itunes.apple.com/search?term=Stallone&entity=movieArtist: no attribute, no limit. Its answer has "resultCount": 6, and its first two records do not have the same fields:

"artistType": "Artist",
"artistName": "Frank Stallone",
"artistLinkUrl": "https://music.apple.com/…",
"artistId": 91734703,
"amgArtistId": 5513,
"primaryGenreName": "Pop",
"primaryGenreId": 14

"artistType": "Movie Artist",
"artistName": "Sylvester Stallone",
"artistLinkUrl": "https://itunes.apple.com/…",
"artistId": 186052826,
"primaryGenreName": "Drama",
"primaryGenreId": 4406

Frank Stallone has an amgArtistId and a link on music.apple.com; Sylvester Stallone has no amgArtistId and a link on itunes.apple.com. Records of one answer need not share their fields — the freedom a document store gives the documents of one collection (lesson 13) — so a program must check that a field exists before it uses it.

A URL cannot carry a space, and it reads &, ? and # as separators. Inside a value such characters are written with percent-encoding: % plus the character's code in two hexadecimal digits, so a space is %20 and an & is %26. Otherwise the server would read an & inside a value as the start of a new pair. The warm-up URL has nine %20.

In Python (background), requests builds and encodes the query string from a dictionary, as Postman does from its table:

# the request of p. 1513, in Python
import requests

url = "https://itunes.apple.com/search"
params = {"term": "maroon",
          "entity": "allArtist",
          "attribute": "allArtistTerm",
          "limit": 10}
r = requests.get(url, params=params)
data = r.json()       # a dict
data["resultCount"]   # 10

Parameters in the path, days in milliseconds: ViaggiaTreno

Trenitalia's ViaggiaTreno service works differently: its parameters travel inside the path, one per segment, and it takes two calls — the first returns the key the second needs. First, the autocompletion of train number 798 (p. 1515):

GET http://www.viaggiatreno.it/infomobilita/resteasy/viaggiatreno/cercaNumeroTrenoTrenoAutocomplete/798

798 - SALERNO - 04/11/25|798-S09818-1762210800000
798 - SALERNO - 05/11/25|798-S09818-1762297200000

Two lines of plain text, one per run of train 798: the run that departs on 4 November and the one that departs on 5 November. Left of the | is a label for people; right of it, a key for programs — train number, station code (S09818 is SALERNO, as the line says) and departure day, joined by dashes. The second call puts the key's parts into the path (p. 1516; its answer is cut down here to a few of its fields):

GET http://www.viaggiatreno.it/infomobilita/resteasy/viaggiatreno/andamentoTreno/S09818/798/1762210800000

{tipoTreno: "PG", orientamento: null, codiceCliente: 4, fermateSoppresse: [], …}
  arrivato: true
  categoria: "ICN"
  circolante: true
  compClassRitardoLine: "regolare_line"
  compDurata: "14:35"
  compImgCambiNumerazione: "  "
  compImgRitardo: "/vt_static/img/legenda/icone_legenda/regolare.png"
  …

The second answer is the train's status, with Italian keys: categoria: "ICN", compDurata: "14:35" (a duration), arrivato: true. The comp keys carry pieces of the website — CSS class names such as "regolare_line", image paths, the HTML entity   — the opposite of the "Separation between model and presentation" that the course notes list as the first advantage of a web API (§5.1.2).

What 1762210800000 means

The day is a number: epoch time in milliseconds, the count of milliseconds since 1 January 1970, 00:00:00 UTC — the usual way computers store an instant. The steps check that 1762210800000 is the midnight of 4 November 2025 in Italy.

1. Fix the instant: in November Italy is on CET, UTC + 1, so local midnight of 4 November is 23:00 UTC on 3 November.

2. Whole years 1970–2024: 55 years of 365 days, plus one day for each of the 14 leap years 1972 … 2024.

   \[ 55 \times 365 + 14 = 20\,089 \text{ days} \]

3. In 2025, January to October have 304 days; 1 and 2 November make 306.

   \[ 20\,089 + 306 = 20\,395 \text{ days} \]

4. A day has \(24 \times 3600 = 86\,400\) s; convert, then add the 23 hours of 3 November.

   \[ 20\,395 \times 86\,400 = 1\,762\,128\,000 \] \[ +\ 82\,800 = 1\,762\,210\,800 \text{ s} \]

5. The API counts milliseconds: multiply by 1000 and you get the key of p. 1515, digit for digit.

   \[ 1\,762\,210\,800\,000 \text{ ms} \]

6. One day later is 86 400 000 ms more — the second key of p. 1515.

   \[ 1\,762\,297\,200\,000 \text{ ms} \]

The third call asks for all departures from a station at a given moment (p. 1517) — the long date of the warm-up. Decoded, the URL reads:

http://www.viaggiatreno.it/infomobilita/resteasy/viaggiatreno/partenze/S00219/Wed Nov 05 2025 20:43:00 GMT+0100 (Central European Standard Time)

The answer is a JSON array of twelve departures. Element 0, compNumeroTreno: "REG 3143", has codOrigine: "S00219" and, side by side, dataPartenzaTrenoAsDate: "2025-11-05" and dataPartenzaTreno: 1762297200000 — the same day for people and for programs, and the number the autocompletion gave for 5 November.

Providers and their rules

Three more slides present data providers reached through APIs.

The Twitter slide shows the Twitter logo crossed out by a red no-entry sign: read it as a warning that this API no longer works as described. Indeed (background), the version the slide calls v1 was v1.1, and since 2023 X, as Twitter is now called, sells read access to its API in paid tiers. At the exam, describe the two versions as the slide prints them; what lasts is the idea — an API is a contract its owner can change (who may call it, which data it reaches), and every pipeline built on it depends on that contract.

No API? Scrape the page — and meet the DOM

An API hands you records with named fields; a web page hands you text in tags, laid out for human eyes. When a site has no API, you take the page itself. The scraping deck opens with the definition (p. 1523):

The slide sets scraping apart from web crawling, "which is used by search engines to index web content": a crawler moves from page to page along links, a scraper extracts data from them — and Scrapy, below, does both.

What can be scraped (p. 1527)? "Social networks: Facebook, Instagram, etc.", "Retailers: Amazon, Tripadvisor, Google Maps, etc.", and "Any website with valuable contents and no official API (e.g. events, train schedules, etc.)". The last item is the rule — scrape when there is no API; "a last resort", say the course notes (§5.1.3). Train schedules are on the list, yet the first deck reached them through an API: when one exists, it is the better door.

The page is a tree

What is a page to a program? The slides answer with the W3C definition (p. 1524):

"When a web page is loaded, the browser creates a Document Object Model of the page" (p. 1525), which defines "The HTML elements as objects", "The properties of all HTML elements", "The methods to access all HTML elements" and "The events for all HTML elements". The third DOM slide is a picture (p. 1526), redrawn in Fig. 3; this is the HTML behind it (the slide names the attribute, not its value):

<html>
  <head>
    <title>My title</title>
  </head>
  <body>
    <a href="page2.html">My link</a>
    <h1>My header</h1>
  </body>
</html>

Why a scraper cares: a parser such as BeautifulSoup builds this same tree from the HTML text; the patterns that pick data out of a page are paths in it; and the page's own scripts can change it after loading — "dynamically access and update", the subject of the next section.

Pointing at a node: three kinds of pattern

Once the page is in hand, "Another basic step is parsing the page, extracting data that respects specific patterns" (p. 1530): CSS selectors, regular expressions and XPath. All three reach the link of Fig. 3 (background: the slides name the techniques, not the calls):

# html holds the page of Fig. 3 as text
import re
from bs4 import BeautifulSoup
from lxml import html as lh

# 1. CSS selector, with BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
link = soup.select_one("body > a")
link.get_text()     # 'My link'
link["href"]        # 'page2.html'

# 2. XPath, with lxml
lh.fromstring(html).xpath("//a/@href")
# ['page2.html']

# 3. regular expression, on the raw text
re.findall(r'<a href="([^"]*)">', html)
# ['page2.html']

CSS selectors and XPath walk the tree: body > a is "an a that is a child of body", //a/@href "the href attribute of any a" — reached with @ or ["href"], not one level down (Fig. 3). A regular expression reads the raw text, blind to nesting, and works only while the HTML is written exactly as the pattern expects. Hence p. 1529: "Since HTML code is tagged text, any tool that allows to manipulate and process text could be potentially used to parse a page" — potentially; a tree-building parser is sturdier.

Scraping rebuilds structure from layout: the DOM is the tree it walks, and a selector is a path in that tree.

Static or dynamic: when downloading the page is not enough

For a static page, scraping takes two steps (p. 1530): "retrieving a static page from the Web" — with "curl, using command line" or "requests, using Python scripts" — then parsing it with the patterns above (the other group's deck lists only XPath and regular expressions, p. 1542). Take a small restaurant page whose reviews you want — Trattoria Rossi, invented for this lesson:

# step 1, from the command line: download the page into a file
curl -s "https://example.com/trattoria-rossi" -o page.html

# steps 1 and 2 in Python: download, then parse
import requests
from bs4 import BeautifulSoup

r = requests.get("https://example.com/trattoria-rossi")
soup = BeautifulSoup(r.text, "html.parser")
# one element per review in the page
reviews = soup.select("div#reviews > p.review")

Then comes the sentence on which the second half of the session turns: "Most websites have dynamic-generated content: downloading and parsing the HTML page could not be sufficient" (p. 1529). The browser shows you the DOM after the page's JavaScript has run; requests downloads the text the server sent, before any script ran — it is not a browser and runs nothing. When a script fills the page after loading, the reviews you see are simply not in the download (Fig. 4).

The next slide asks the two questions that a plain download cannot answer (p. 1531): "What happens if we want to obtain pages in sequence (e.g.: pages of reviews from Amazon or TripAdvisor)?" and "What we need to do if the page loads the content we need only after clicking a specific field or while scrolling the page itself (e.g.: Facebook and Instagram posts)?"

Three modules, three jobs

"Python is one of the best languages to build scrapers: it is easy to learn, fast to code and it has plenty of libraries to get anything done efficiently" (p. 1532). The deck presents three of them.

Selenium's origin explains why it answers both questions of p. 1531: a testing tool must do what a user does, so it drives a real browser — the browser runs the scripts, Selenium clicks "next" or scrolls, and the DOM it reads at the end is the rendered one. The handover to BeautifulSoup is one line, driver.page_source:

# the dynamic page: a real browser loads it, BeautifulSoup parses
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

# start a real browser and open the page
driver = webdriver.Chrome()
driver.get("https://example.com/trattoria-rossi")
# wait (at most 10 s) until the script has added a review
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, "p.review")))
# hand the rendered page to the parser
soup = BeautifulSoup(driver.page_source, "html.parser")
driver.quit()
reviews = soup.select("div#reviews > p.review")

The pipeline, and why scrapers break

"The image below represents the interactions between the three presented modules: their simple combination is the basis for a web scraper" (p. 1536). Fig. 5 redraws it, with the two return loops of the other group's version (p. 1545).

Read it as fetch, parse, store: Selenium fetches when the page needs a browser, Scrapy when many pages must be crawled, and BeautifulSoup turns the HTML page into parsed data. The loops answer the two questions of p. 1531 by name: "get dynamic elements" (click or scroll, then read again) and "get another page". Two simplifications: the static path of p. 1530 (requests, then BeautifulSoup) is not drawn, and Scrapy's page goes through BeautifulSoup although Scrapy can extract by itself.

Fragile by design

An API is a contract with named fields; a page is a layout that its owner may redesign any night. "Most of the times, the HTML changes over time, especially the classes and sometimes the structure of the HTML. Therefore, it is really important to keep the scraper updated" (p. 1537, "Dynamic Websites" — the third sense of the word). Rename the class in div#reviews > p.review and the scraper returns an empty list without any error, which is why the course notes call scrapers very difficult to maintain (§5.1.3).

Sites also defend themselves: "Some websites have some protections against scraping. They are able to detect weird behaviours (e.g., repeated scrolling) and prevent you from scraping" (p. 1528). And Selenium needs a "chromedriver", the program through which it drives the browser, so the two versions must match: "Check out your Google Chrome/EDGE version", then "Download and put the correct chromedriver version within the folder" (p. 1538). Since version 4.6, Selenium fetches the matching driver itself (background).

Who wrote the URL of the warm-up

Back to the warm-up. Its decoded tail is exactly the text JavaScript produces for a date — try new Date().toString() in the console of a browser that runs in English on Italian time — and a %20 for every space, with : and + left bare, is exactly what JavaScript's encodeURI makes of it. So a program running JavaScript wrote this URL: most likely the ViaggiaTreno web page itself, asking its server for the departures after it had loaded. The session shows the call, not how it was found; this reading is ours.

If this reading is right, it is the dynamic page of p. 1529 seen from inside: a board filled this way reaches the browser as an empty frame, and requests.get receives only the frame. Two ways out: drive a browser with Selenium and parse what it renders, or call the API that the page itself calls, as the session did. When you can see the call, the second is simpler and sturdier — named fields instead of a layout, and nothing breaks when a CSS class is renamed.

Prefer an API when one exists; when you scrape, fetch with the simplest tool that sees the data, parse with BeautifulSoup, and expect to maintain the scraper.