Welcome to the follow up of Running Loki as a monolith. In the previous article we saw what's Loki and how to quickly deploy it as a service in your VM. Now its time to explore how we can run Loki split into different components hence obtaining better distribution/reliability.

The architecture is based on Cortex, a project built to run Prometheus in an horizontally scalable and distributed fashion.

Loki Architecture

Components

There are 4 components: distributor, ingester, querier and query-frontend.

Distributors are responsible for handling incoming streams. Each stream is validated for correctness and valid chunks are then split into batches and sent to multiple ingesters in parallel.
Ingesters are responsible for writing data to long-term storage backends (S3, Filesystem, etc.) and returning log data for in-memory queries. Streams are built into “chunks” in memory and flushed to the backing storage backend at a configurable interval.
Queriers handle queries using the LogQL query language, fetching from all ingesters for in-memory data before falling back to running the same query against the backend store.
Query-Frontend is an optional service. When in place, incoming query requests should be directed to the query frontend instead of the queriers. It performs some query adjustments and holds queries in an internal queue. Queriers act as workers which pull jobs from the queue, execute them, and return them to the query-frontend for aggregation.

The configuration documentation states we can chose to deploy a Loki instance as one of these 4 components by selecting the target. When looking for HA setups there is a common complaint about the lack of documentation about it. The closest thing to it is this k8s configuration used by Grafana Labs themselves.

"One ring to rule them all"

When running distributed Loki we need to store information about the different nodes so Loki knows how to distribute/retrieve the data properly. It requires a Key-Value (KV) store to store what we call the ring. We can use traditional KV stores like Consul or Etcd, but we can also use Loki's own KV store based on the memberlist library using a gossip algorithm.

...

<aside> 💡 Note from the article writer: I took too long to type this down and no longer have the work in my head. It may take a while until I find time to complete this.

</aside>

Made with Notion, Fruition and some love