The Elastic Stack is primarily a collection of Elasticsearch, Logstash, Kibana and log shippers. At the time of this writing, they are maintained by Elastic.
Definitions
- Elasticsearch: This a NoSQL database that is based on the Lucene search engine.
- Logstash: This a log pipeline tool that accepts inputs from various sources, executes different transformations, and exports the data to various targets.
- Kibana: This a visualization layer that works on top of Elasticsearch.
- Beats: These are technically log shippers.
Why Elastic
When setting up and administering systems at scale, it is important to keep an eye on your logs. The logs tell you how well your system is doing. Depending on your configuraton settings, you can find errors, warnings, information etc. just about anything you want. The Elastic stack helps in aggregation of your logs. Beyond systems data, this can also be used for processing data in general. Other use cases includes Business intelligence, compliance, security, analytics etc.
Setup
| Component |
Size |
Systems Specification |
| Elasticsearch |
3 |
2 CPU 4GB RAM |
NOTE:
You can make changes as needed. The above is just a base systems spec.
Manual Installation
Update Hosts File: Update /etc/hosts on all nodes
sudo bash -c 'cat <<EOF>> /etc/hosts
192.168.1.170 elknode-01
192.168.1.159 elknode-02
192.168.1.138 elknode-03
EOF'
Package Update: Upgrade and install required packages on all nodes
sudo apt upgrade -y
sudo apt-get install -y jq net-tools npm ntp ntpdate ca-certificates curl software-properties-common openjdk-14-jdk
Install and configure Kibana and Elasticsearch on elknode-01
# Download Elastic Packages
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.3-amd64.deb
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.17.3-amd64.deb
# Install Elasticsearch & Kibana
sudo dpkg -i elasticsearch-7.17.3-amd64.deb
sudo dpkg -i kibana-7.17.3-amd64.deb
sudo bash -c 'cat <<EOF> /etc/elasticsearch/elasticsearch.yml
cluster.name: pocstack
node.name: elknode-01
network.host: 0.0.0.0
node.master: true
node.data: true
node.ingest: true
node.ml: true
node.remote_cluster_client: true
node.transform: true
http.port: 9200
discovery.seed_hosts: ["elknode-01", "elknode-02", "elknode-03"]
cluster.initial_master_nodes: ["elknode-01"]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
EOF'
sudo bash -c 'cat <<EOF> /etc/kibana/kibana.yml
server.host: 0.0.0.0
server.name: elknode-01
elasticsearch.hosts: ["http://localhost:9200"]
server.port: 5601
EOF'
# Enable and start services
sudo systemctl enable kibana
sudo systemctl start kibana
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
Install and configure Elasticsearch on elknode-02
# Download Elastic Packages
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.3-amd64.deb
# Install Elasticsearch & Kibana
sudo dpkg -i elasticsearch-7.17.3-amd64.deb
sudo bash -c 'cat <<EOF> /etc/elasticsearch/elasticsearch.yml
cluster.name: pocstack
node.name: elknode-02
network.host: 0.0.0.0
node.master: true
node.data: true
node.ingest: true
node.ml: true
node.remote_cluster_client: true
node.transform: true
http.port: 9200
discovery.seed_hosts: ["elknode-01", "elknode-02", "elknode-03"]
cluster.initial_master_nodes: ["elknode-01"]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
EOF'
# Enable and start services
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
Install and configure Elasticsearch on elknode-03
# Download Elastic Packages
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.3-amd64.deb
# Install Elasticsearch & Kibana
sudo dpkg -i elasticsearch-7.17.3-amd64.deb
sudo bash -c 'cat <<EOF> /etc/elasticsearch/elasticsearch.yml
cluster.name: pocstack
node.name: elknode-03
network.host: 0.0.0.0
node.master: true
node.data: true
node.ingest: true
node.ml: true
node.remote_cluster_client: true
node.transform: true
http.port: 9200
discovery.seed_hosts: ["elknode-01", "elknode-02", "elknode-03"]
cluster.initial_master_nodes: ["elknode-01"]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
EOF'
# Enable and start services
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
Test Cluster
Log into elknode-01 and type curl -XGET http://localhost:9200/_cluster/health?pretty. You should see this
{
"cluster_name" : "pocstack",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 6,
"active_shards" : 12,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
List Nodes with this command curl -XGET http://localhost:9200/_cat/nodes?v. You should have this
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.1.170 25 95 2 0.11 0.06 0.01 cdfhilmrstw * elknode-01
192.168.1.138 13 96 2 0.20 0.07 0.03 cdfhilmrstw - elknode-03
192.168.1.159 49 97 2 0.04 0.03 0.02 cdfhilmrstw - elknode-02
List indices with this command curl -XGET http://localhost:9200/_cat/indices?v. You should have this
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana_task_manager_7.17.3_001 yLLLy6DAT1qv8yOBnJQvQw 1 1 10 772 333.8kb 188kb
green open .apm-custom-link rgLZ_MziRbqp-ySSyz4Gag 1 1 0 0 416b 208b
green open .apm-agent-configuration 4c81IGbkS3aWF4b6M1CzgA 1 1 0 0 416b 208b
green open .kibana-event-log-7.13.1-000001 O7A8guR5SPmprppycbe9Sg 1 1 1 0 11.2kb 5.6kb
green open .kibana_7.17.3_001 jSRgvGcZTN-AasYTsJa2-g 1 1 13 9 4.2mb 2.1mb
Get Shard allocation per node with this command curl -XGET http://localhost:9200/_cat/allocation?v. You should see this:
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
4 140.4kb 9.2gb 10.3gb 19.5gb 47 192.168.1.159 192.168.1.159 elknode-02
4 2.3mb 9.2gb 10.3gb 19.5gb 47 192.168.1.138 192.168.1.138 elknode-03
4 2.1mb 10.4gb 9.1gb 19.5gb 53 192.168.1.170 192.168.1.170 elknode-01
Reference Links