Set Up Multi-node Elasticsearch Cluster

What is Elasticsearch Cluster? 

An elastic-search cluster is a group of nodes that have the same cluster.name attribute. As nodes join or leave a cluster, the cluster automatically syncs itself to evenly distribute the data across the available nodes. 

Each node in an Elasticsearch cluster serves one or more purpose:

• Master-eligible node – A node that has node.master set to true (default). It is responsible for lightweight cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes.
• Data node – A node that has node.data set to true (default). Data nodes hold data and perform data related operations such as CRUD, search, and aggregations.
• Coordinating node – Its main role is to route search and indexing requests from clients to data nodes. It behaves as smart load balancers.

In this guide, we are going to set up a two node Elasticsearch cluster with each node being master eligible.

Muvi Environment:

• Node 1
• Node 2

Installation

• sudo apt-get update

• sudo apt-get -y install apt-transport-https curl wget

• sudo wget -qO – https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add –

• sudo echo “deb https://artifacts.elastic.co/packages/7.x/apt stable main” | tee -a /etc/apt/sources.list.d/elastic-7.x.list

• sudo apt-get update && apt-get install elasticsearch=7.5.2

Set Elasticsearch Cluster name

On each node, open the Elasticsearch configuration file and set the name of your Elasticsearch cluster.

Step1:

vim /etc/elasticsearch/elasticsearch.yml

# Use a descriptive name for your cluster:

cluster.name: muvi-cluster

Set Descriptive names for Elasticsearch Nodes

Step2:

Node 1

…

# ———————————— Node ————————————

# Use a descriptive name for the node:

node.name: muvi-node-01

Node 2

# ———————————— Node ————————————

# Use a descriptive name for the node:

node.name: muvi-node-02

Step 3:

Disable Memory Swapping

Swapping affects the stability of Elasticsearch cluster as it can cause nodes to respond slowly or even to disconnect from the cluster. Once of the ways of disabling memory swapping is by enabling memory lock. Hence, uncomment the line bootstrap.memory_lock: true.

# ———————————– Memory ———————————–

# Lock the memory on startup:

bootstrap.memory_lock: true

I recommend that you disable swapping using systemd by editing Elasticsearch service and adding the content below;

Step:4:

systemctl edit elasticsearch

[Service]

LimitMEMLOCK=infinity

Step:5:

Whenever a systemd service is modified, you need to reload the systemd configurations.

#sudo systemctl daemon-reload

Step: 6:

One of the recommended ways to disable swapping is to completely disable swap. This is fine if Elasticsearch is the only service running on the server.

swapoff -a

Step: 7:

Define the Roles of each Elasticsearch Node

As stated above, you can assign each node a respective role as master, data node, ingest node, coordinating node.. In this setup, we will configure all the three nodes to act as both master and data node.

# ———————————- Cluster ———————————–

# Use a descriptive name for your cluster:

cluster.name: es-nodes

node.master: true

node.data: true

Step: 7:

Discovery and cluster formation settings

There are two important discovery and cluster formation settings that should be configured before going to production so that nodes in the cluster can discover each other and elect a master node;

• discovery.seed_hosts and cluster.initial_master_nodes.

discovery.seed_hosts setting Provides a list of master-eligible nodes in the cluster. Each value has the format host:port or host, where port defaults to the setting transport.profiles.default.port. This setting was previously known as discovery.zen.ping.unicast.hosts. Configure this setting on all Nodes as follows;

# ——————————— Discovery ———————————-

# Pass an initial list of hosts to perform discovery when this node is started:

# The default list of hosts is [“127.0.0.1”, “[::1]”]

discovery.seed_hosts: [“60.0.7.29”, “60.0.7.46”]

Step 8:

# Bootstrap the cluster using an initial set of master-eligible nodes:

#cluster.initial_master_nodes: [“muvi-node-1”, “muvi-node-2”]

cluster.initial_master_nodes: [“60.0.7.29”, “60.0.7.46”]

Step 9:

Set JVM Heap Size

Elasticsearch sets the heap size to 1GB by default. As a rule of thump, set Xmx to no more than 50% of your physical RAM but not more than 32GB.

vim /etc/elasticsearch/jvm.options

-Xms1g

-Xmx1g

Step 10:

Set maximum Open File Descriptor

Open below file and LimitNOFILE 

/usr/lib/systemd/system/elasticsearch.service

process

LimitNOFILE=65535

You also should set the maximum number of processes.

LimitNPROC=4096

Step 11:

Virtual Memory Settings

Open below file  and add vm.max_map_count as shown below.

/etc/sysctl.conf

vm.max_map_count=262144

Or

#echo “vm.max_map_count=262144” >> /etc/sysctl.conf

#sysctl vm.max_map_count

vm.max_map_count = 262144

Step 12:

Running Elasticsearch

Reload the systemd manager configuration.

#systemctl daemon-reload

Enable Elasticsearch to run on system boot.

#systemctl enable elasticsearch

Start Elasticsearch

#systemctl start elasticsearch

You can check the Elasticsearch status running the command below;

#systemctl status elasticsearch.service

Step 13:

Check Elasticsearch Cluster Health

curl -X GET “<elbname>.elb.us-east-1.amazonaws.com:9200/_cluster/health?pretty”

root@ip-60-0-7-29:~# curl -X GET “<elbname>.elb.us-east-1.amazonaws.com:9200/_cluster/health?pretty”

{

  “cluster_name” : “muvi-cluster”,

  “status” : “green”,

  “timed_out” : false,

  “number_of_nodes” : 2,

  “number_of_data_nodes” : 2,

  “active_primary_shards” : 0,

  “active_shards” : 0,

  “relocating_shards” : 0,

  “initializing_shards” : 0,

  “unassigned_shards” : 0,

  “delayed_unassigned_shards” : 0,

  “number_of_pending_tasks” : 0,

  “number_of_in_flight_fetch” : 0,

  “task_max_waiting_in_queue_millis” : 0,

  “active_shards_percent_as_number” : 100.0

}

Step 14:

Check the Cluster Nodes

#curl -X GET “<elbname>.us-east-1.amazonaws.com:9200/_cat/nodes?v”