Inside an Elasticsearch index — and the rest of ELK

Who does what: the ELK stack

Page 569 stacks four programs as coloured bars: Kibana, "Visualize and Manage"; Elasticsearch, "Store, Search and Analyze"; Logstash + Beats, "Ingest". The initials of the first three name the ELK stack; p. 571 calls it the Elastic Stack.

Page 570 shows who talks to whom (Fig. 1). Beats are small programs on the machines where data is produced, each collecting one kind of data and shipping it, through their shared library libbeat, straight into Elasticsearch or to Logstash, which parses and enriches it first. Kibana reads what Elasticsearch holds and draws it.

The arrows carry a design decision: in this picture no data is born in Elasticsearch. It arrives from elsewhere — a web server's log, a database's rows — and is indexed to be searched, while the system it came from keeps the original. That is why two of the four programs exist only to bring data in.

Elasticsearch itself is the "Core of the Elastic Stack", a "Search & Analytic engine" with four properties (p. 571, again on p. 575):

• "Based on Apache Lucene which enables full-text search": every text and keyword field ends up as terms in an inverted index, term → documents (lesson 24); the mapping and the analyzers decide which terms. (Numbers and dates get their own structures for range queries — background.)
• "Distributed": an index is cut into shards on several machines, with copies.
• "RESTful": you talk to it with HTTP requests carrying JSON.
• "Near real-time: the time for a document to be searchable after indexing is very short" — short, not zero: by default new documents become searchable about once a second (background).

Documents, indices, shards and replicas

Start from one record: six fields of the inspection that p. 590 fetches from the index nyc-restaurants.

{
  "DBA" : "BENTO HOUSE",
  "BORO" : "Queens",
  "CUISINE DESCRIPTION" : "Chinese",
  "INSPECTION DATE" : "02/17/2021",
  "ZIPCODE" : 11354,
  "VIOLATION CODE" : "22G"
}

That is a document: "Elasticsearch stores data structures in JSON documents" (p. 576), JSON objects like MongoDB's (lesson 13), nesting included. "When stored, new documents are indexed and made fully searchable", and they "can be accessed from any ES node when in a cluster".

The documents of one kind form an index: a "Data organization mechanism", "Partitioned in shards", whose structure is defined "via mappings"; and "Elasticsearch is based on the concept of Inverted Index" (p. 577). Page 586 lines the words up with the relational ones:

The analogy holds for words, not mechanisms: a document nests, an index is cut into pieces, and "index" means the whole collection, not SQL's helper structure on a column.

Shards and replicas

An index is split into primary shards — "Each document in an index belongs to one Primary Shard" (p. 1621) — spread over the nodes of a cluster: the partitioning of lesson 3. A replica is a copy of a primary shard, with one condition: "Replicas: copies of shards stored on a different node" (p. 578). The same slide gives the traffic rules:

• "Write operation → Primary → Replica (to copy)": the primary writes the document, then copies it to every replica.
• "Read operation → Routed to primary or replica": any copy may answer.

Why one writer? If every copy accepted writes, copies could disagree; one primary per shard orders the writes and the replicas follow. Reads change nothing, so any copy will do — more replicas, more reads at once. But every write is done once per copy, so replicas never speed up writing. Per index this is two numbers, number_of_shards and number_of_replicas (per primary), plus a name and a location per node (p. 579).

Page 579 draws the result, and is easy to misread: its five boxes labelled "Node" cannot be one cluster — a primary cannot exist twice. Read them as one index, three primaries with one replica each, laid out on two different clusters (Fig. 2), with \(3 \times (1 + 1) = 6\) copies in each. In general \(P\) primaries with \(R\) replicas make \(P\,(1 + R)\) shard copies.

Shards are not free: "The more shards you have, the more the management overheads", and "The larger the shard size, the more it takes to move them (if needed)" (p. 1622). The number of primaries is fixed when the index is created; the number of replicas can change at any time (background).

Talking to Elasticsearch: one REST request at a time

"Interactions with Elasticsearch happen through requests to REST endpoints", and "Actions depend on HTTP verbs" (p. 582): GET to read, POST and PUT "often used to create new documents, indices, etc.", and DELETE. You send them from a command line, from Postman, or from the "Developer Tools built in Kibana", the console of the exercise sessions. A request is a verb, an endpoint and, often, a JSON body. The shortest creates an index (p. 587):

# create an empty index
PUT /my_index

The answer, "acknowledged" : true, says that the index exists. Now add a document (p. 584):

# add a document; Elasticsearch picks its _id
POST /my_index/_doc
{
  "author":"Andrea",
  "title":"Set up Elasticsearch and Kibana in 15 minutes!",
  "date":"Tue, 23 Feb 2021 11:40:00 +0000",
  "categories": ["elasticsearch","tutorial",
                 "data science","bigdata"],
  "lang":"en-US"
}

Read it aloud: verb POST; endpoint /my_index/_doc, the documents of my_index; body, the document. No identifier is given, so "Elasticsearch takes care of creating and assigning IDs to documents" (p. 583): the answer carries "_id": "jYpU_XQBkGeBYJKltRh8" and "result": "created" (p. 585). With PUT you choose the identifier:

# create or replace document my_id
PUT /my_index/_doc/my_id
{
  "author":"Andrea",
  "lang":"en-US"
}

Hence the slide's warning (p. 583): "POST doesn't require the ID of the resource → duplicates", "PUT requires the ID of the resource → create or update the same one". Send the POST twice and the post is stored twice, under two ids; send the PUT twice and one document has _version 2. A PUT to a known address is idempotent; a POST to a collection is not.

A GET on a document's address reads it back (p. 590):

# one document by its _id
GET /nyc-restaurants/_doc/pZpozncBDaS7kSRanan2

The answer wraps the document in metadata — _index, _id, _version, _seq_no, _primary_term, "found" : true — and returns under _source the JSON exactly as it was sent. Searches look in the inverted index; _source is what they hand back. The last verb removes:

# delete one document; DELETE /my_index drops the whole index
DELETE /my_index/_doc/my_id

The mapping: what each field becomes

Ask the restaurant index for its structure:

# the mapping this index is using
GET /nyc-restaurants/_mapping

Nobody typed the answer of p. 581, yet every field has a type: "@timestamp" : { "type" : "date" }, "ACTION" : { "type" : "text" }, "BBL" : { "type" : "long" }… "Is Elasticsearch really schemaless?" the slide asks, with an arrow labelled "Proof" to this answer. No: "Elasticsearch tries to guess the structure of the documents" — dynamic mapping — when it first meets a field, and keeps the guess.

A mapping is the schema of an index: with it "you can tell Elasticsearch the types of your documents' fields", "Decide which fields are searchable" and "Enable full-text search, time and geo-based queries" (p. 580). The type decides how a value is indexed, so it decides which questions can be asked of it. Page 589 teaches four types; the session's table adds more (p. 1635):

The hard choice is text against keyword. A text field is cut into words by an analyzer and searched word by word; a keyword field is one exact value, matched whole, counted and sorted. Decide by the question: "violations that mention mice" wants words, so text; "only Queens" and "how many per cuisine" want whole values, so keyword. Mind the slide's own list: a postcode is keyword even when it arrives as a number, like "ZIPCODE" : 11354 — you match postcodes, you never add them.

The guess is "risky (e.g. dates could be parsed as strings)" (p. 581): "02/22/2021" is not a date format Elasticsearch recognises by default, so it becomes a string — a text field with a keyword sub-field (background) — and sorting by it compares characters, not days.

What may change once documents are in

Adding a field (p. 588):

# add a text field to the mapping
PUT /my_index/_mapping
{
  "properties": {
    "field_name": {
      "type": "text"
    }
  }
}

The exam's question: the complete mapping of the index

Safer than any guess is a mapping written before the first document. The session shows the full form, settings and mapping side by side (p. 1663, "Define the full mapping describing a Person with 10 shards and 3 replicas"):

# an index with settings and mapping
PUT /people
{
  "settings": {
    "number_of_shards": 10,
    "number_of_replicas": 3
  },
  "mappings": {
    "properties": {
      "personal_id": { "type": "keyword" },
      "name": { "type": "text" },
      "surname": { "type": "text" },
      "birth date": { "type": "date", "format": "yyyy-MM-dd" },
      "address": { "type": "text" },
      "eye_color": { "type": "keyword" },
      "height": { "type": "integer" }
    }
  }
}

Read it from the outside in: PUT /people creates the index; settings gives it 10 primaries with 3 replicas each; mappings → properties holds one entry per field with its type and, for a date, its format. Seven of the nine past papers open their Elasticsearch exercise with this request, for 1 point: "Provide the complete mapping of the index for the considered entity (i.e., field name, field type, the structure of the mapping, etc.)" — the three 2024 papers word it without "for the considered entity". mappings is the answer; settings may be added.

Solved on the paper of 30 May 2025 (its header prints "June 30, 2025"), exercise 3. The ER scenario tracks puzzles' production and sales; the entity is Puzzle — Puzzle ID, Name, Collection Name (e.g., Museum Collection), Number of Pieces, Width, Height, Description — and "All sizes are in cm". The next sub-question (2 points) asks for "the total number of puzzles for each collection name". "Consider the Puzzle entity. Provide the complete mapping of the index for the considered entity (…) (1 PT)". List every attribute, inherited ones included (Puzzle has no ISA parent), and type each by the question it will answer:

# the answer: mappings → properties
PUT /puzzles
{
  "mappings": {
    "properties": {
      "puzzle_id": { "type": "keyword" },
      "name": { "type": "text" },
      "collection_name": { "type": "keyword" },
      "number_of_pieces": { "type": "integer" },
      "width": { "type": "double" },
      "height": { "type": "double" },
      "description": { "type": "text" }
    }
  }
}

Choose each field's type by the question that will be asked of it: words → text, whole values to match or count → keyword, numbers and dates → their own types.

Analyzers: how a text becomes terms

A text field is "pre-processed" by an analyzer (p. 589), the program that turns a string into the terms of the inverted index. You can run one on any text (p. 594):

# the terms of a text
POST /_analyze
{
  "analyzer": "standard",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

The answer lists tokens with their offsets and positions; the slide shows the first, "token" : "the" at position 0. The table runs the same sentence through every analyzer of pp. 591–593:

Check the slide's definitions against the rows. The standard analyzer "divides text into terms on word boundaries, as defined by the Unicode Text Segmentation algorithm. It removes most punctuation, lowercases terms, and supports removing stop words" (p. 592). The simple analyzer cuts "whenever it encounters a character which is not a letter" and "lowercases all terms": the 2 vanishes, "dog's" falls apart. The whitespace analyzer "does not lowercase terms": QUICK and Brown-Foxes survive. The stop analyzer "is like the simple analyzer, but also supports removal of stop words"; the keyword analyzer, a "noop", "outputs the exact same text as a single term"; the pattern analyzer splits "according to a custom regexp pattern" (p. 593) — the pattern marks the separators, by default every run of non-word characters, \W+, and the terms are lower-cased (background). A language analyzer "removes stopwords of a given language" and performs "stemming" (p. 591): english makes foxes fox and lazy lazi — a stem need not be a word, only shared by related words.

The analyzer runs twice: on every document when it is indexed, and on the query text of a match. Both sides must end with the same terms, or "Brown-Foxes" never meets "brown" — the training–serving skew of machine learning, where preprocessing that differs at serving time feeds the model inputs it never learned.

And a keyword field? "Borough (BORO) was mapped as a keyword. A keyword does not have any analyser!" (p. 598). Its value enters the index whole, as written, and so does the query text. The widget replays the hook on three documents from the slides.

The rest of ELK: Beats, Logstash and Kibana

Ingestion takes two programs because it has two jobs: "Beats focus on data collection and shipping", "Logstash focuses on processing and data normalization" (p. 617). Beats, a "Platform for data shippers" — Filebeat, Metricbeat, Packetbeat, Heartbeat — "Collect and ship logs and metrics from hosts or containers" (p. 616), light enough for every machine. Logstash, the "Streaming ETL engine of the ELK stack", does the heavy work once, centrally: "centralized data collection, processing and enrichment on the fly", "Data agnostic", with a "Wide range of integrations and processors" and "Ready-to-use monitoring and administrative panes built in Kibana" (p. 615). ETL is the warehouse word of lesson 23; streaming means event by event.

Logstash also takes data without Beats, "Via TCP, UDP, HTTP protocols" or "Pool-based inputs like JDBC" that fetch it (p. 617), and emits it "to Elasticsearch, to other data stores or via TCP, UDP and HTTP protocols" (p. 619); in between, filters "structure, normalize and enrich data" (p. 618) — the three columns of p. 620.

Events and the pipeline

What flows is an event: the "Primary unit of data in Logstash", "Similar to JSON documents", which "Flow through pipelines" (p. 623). The slide prints one as Logstash's debug output shows it, with => where JSON has a colon:

{
  "@timestamp" => 2021-02-16T01:01:01.000Z,
  "message" => "hello",
  "other_field" => {
    "nested_field" => 5678
  }
}

The pipeline, the "Logical flow of data" (p. 624, Fig. 3), "Supports multiple inputs", has a "Single queue to buffer data", "Either in-memory or persistent", and "Workers ensure server-side scalability".

Why a queue? Inputs and filters run at different speeds: bursts wait in the queue, and a slow Elasticsearch fills it until the inputs wait, without dropping events. Why workers? Parsing and enriching are expensive, so several threads do it at once, each on its own batch — scaling inside one server (background).

Codecs and filters

A codec changes "the data representation of an event" — "Serialization and de-serialization" — "as part of an input or output" (p. 625):

# parse the JSON lines of file.log
input{
  file {
    path => "file.log"
    codec => json
  }
}

# pretty-print events on the console
output{
  stdout {
    codec => rubydebug
  }
}

They are the slide's "Parse JSON data in file.log" (a real file input wants an absolute path, like /var/log/file.log) and "Format console output with Ruby Amazing Print", the => form above. A codec works at the edges, on bytes; a filter in the middle, on fields (pp. 626–627):

One filter almost every pipeline uses is missing: grok, which cuts an unstructured line into named fields with patterns (background) — the "structure" of p. 620, which geoip needs before it can enrich (Fig. 4).

The slide's own pipeline is the smallest complete one (p. 628):

# Beats in, Elasticsearch out
input {
  beats { port => 5043 }
}
filter {
  mutate { lowercase => ["message"] }
}
output {
  elasticsearch {}
}

The slide reads it: "Receive data from a host that uses Beats" (port 5043; a Beat sends to 5044 unless told otherwise — background), "Process that input via a filter" (lower-case message), "Output directly to Elasticsearch". Type straight quotes, not the slide's curly ones.

Delivery: at least once, and the dead letter queue

Page 629 promises "“At Least Once” message delivery": "In most conditions messages are delivered exactly once", but "Unclean shutdowns could lead to duplicates". With the persistent queue, an event leaves the queue only after the outputs accept it; if Logstash dies in between, the event is processed again at restart — twice in Elasticsearch, never zero times.

Some events can never be delivered: Elasticsearch rejects them, typically when a field contradicts the mapping. Retrying forever blocks the pipeline; dropping them breaks the at-least-once promise. The slide squeezes this into "Due to “at least once” policy these are undeliverable": the refusal makes them undeliverable, the policy forbids dropping them. The dead letter queue (DLQ) sets them aside on disk for later inspection: it "Avoids losing those events and halting the pipeline, freeing processing resources for subsequent events" (p. 629). It is off by default, and only the Elasticsearch output writes to it (background).

Modules, finally — "Provided by Logstash and Beats", "Available only for specific data types" — package the whole chain: "Automated parsing and enrichment in Logstash", "Custom schema in Elasticsearch", "Default dashboards in Kibana" (p. 621).

Kibana: every panel is a question to Elasticsearch

Kibana "is an open source data visualization dashboard for Elasticsearch" that "provides visualization capabilities on top of the content indexed on an Elasticsearch cluster" (p. 631): it shows what an index holds, nothing else. With it "you can aggregate, organize, filter and represent data" (p. 632) in dashboards of "basic representations, location-based analyses, time series, machine learning and graphs/networks" (p. 633), some paid.

A first dashboard (p. 635) is an editor with four labelled zones: "Dashboard Management", "Time and Fields Filters", "Tools to create a new Graph" and the space where "The Graphs you will create will be displayed". Its "Create Panel" (p. 636) offers Lens, Maps, "Aggregation based" charts, TSVB, a "Custom visualization" in Vega, Text and Controls: "easy to learn … but really hard to master" (p. 637).

Back to Queens

The hook resolves. BORO is a keyword, so "Queens" entered the index as one term, capital Q included. The query "Queens" became the same term; "queens" became a term with no posting list, hence "hits" : [ ]. A text field with the standard analyzer would have lower-cased both sides and found both.

Was keyword a mistake? No. A borough is a category to filter and count by, and "term aggregations only work on keywords" (p. 1658): the Kibana pie of inspections per borough needs this very mapping. Exactness costs spelling the value as stored. To have both, map text with a keyword sub-field, as dynamic mapping does, and query BORO for words, BORO.keyword for exact values. The mapping did not fail: it answered the question it was built for.