ELK Stack Introduction

ELK Stack Introduction

ELK is Elasticsearch + Logstash + Kibana, the most widely-used open-source log management platform. Elasticsearch stores and indexes logs for fast querying. Kibana is the web UI for searching and visualizing. Logstash processes and transforms logs before indexing; for simple setups it can be replaced with Filebeat (a lightweight log shipper). Together they let you search gigabytes of logs in milliseconds and build dashboards showing error rates, top attacking IPs, and response time distributions across your entire infrastructure.

ELK components

ELK Stack architecture:

  Application servers (log shippers)
    Filebeat: /var/log/nginx/access.log
    Filebeat: /var/log/auth.log
         ↓  (ships to)
  Logstash (optional transformation layer)
    Parse, filter, enrich logs
    (can skip Logstash for simple setups — Filebeat → Elasticsearch directly)
         ↓  (indexes into)
  Elasticsearch (storage + search)
    Stores documents indexed for full-text search
    REST API on port 9200
         ↓  (queries from)
  Kibana (web UI)
    Search, visualize, build dashboards
    Web interface on port 5601

Installing Elasticsearch

# Add Elastic repository
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update && sudo apt install -y elasticsearch

# Configure for single-node setup:
sudo nano /etc/elasticsearch/elasticsearch.yml
# node.name: node-1
# network.host: localhost
# discovery.type: single-node

sudo systemctl enable --now elasticsearch

# Verify it's running:
curl -s http://localhost:9200 | python3 -m json.tool

Elasticsearch health check response

{
  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "version" : { "number" : "8.x.x" },
  "tagline" : "You Know, for Search"
}

Installing Kibana

# Install Kibana (same repository as Elasticsearch)
sudo apt install -y kibana

# Configure:
sudo nano /etc/kibana/kibana.yml
# server.port: 5601
# server.host: "0.0.0.0"     ← or localhost if using nginx proxy
# elasticsearch.hosts: ["http://localhost:9200"]

sudo systemctl enable --now kibana

# Access at http://YOUR_SERVER_IP:5601
# Note: Protect with SSL and authentication in production

Shipping logs with Filebeat

# Install Filebeat on the servers you want to collect logs from:
sudo apt install -y filebeat

# Configure /etc/filebeat/filebeat.yml:
# filebeat.inputs:
#   - type: log
#     enabled: true
#     paths:
#       - /var/log/nginx/access.log
#       - /var/log/nginx/error.log
#     tags: ["nginx"]
#
#   - type: log
#     enabled: true
#     paths:
#       - /var/log/auth.log
#     tags: ["auth"]
#
# output.elasticsearch:
#   hosts: ["elasticsearch-host:9200"]
# setup.kibana:
#   host: "kibana-host:5601"

# Load pre-built dashboards and index patterns:
sudo filebeat setup --index-management -E output.logstash.enabled=false -E output.elasticsearch.enabled=true

sudo systemctl enable --now filebeat

Practical ELK queries

# In Kibana Discover (KQL query language):
# Find all SSH authentication failures:
# message: "Failed password" AND tags: "auth"

# Find high-status-code nginx responses:
# http.response.status_code: >400 AND tags: "nginx"

# Elasticsearch REST API queries (direct):
curl -s "http://localhost:9200/filebeat-*/_search?q=Failed+password&size=10" | python3 -m json.tool

Conclusion

ELK is powerful but resource-intensive: Elasticsearch needs at least 4GB RAM for a minimal single-node setup and prefers SSDs for performance. For small environments (under 10 servers), the simpler rsyslog centralization approach is often better. ELK's strengths are full-text search across billions of log events, visualization dashboards for metrics and trends, and anomaly detection. Once Filebeat is running on your servers and Elasticsearch is indexed, searching for events like "all 500 errors in the last hour across all services" takes milliseconds.