Muvi Blogs

Beyond The Buzz- Latest Insights From Muvi

Set Up Multi-node Elasticsearch Cluster 11 November 2021

 

 

What is Elasticsearch Cluster? 

elastic search cluster

An elastic-search cluster is a group of nodes that have the same cluster.name attribute. As nodes join or leave a cluster, the cluster automatically syncs itself to evenly distribute the data across the available nodes. 

 

Each node in an Elasticsearch cluster serves one or more purpose:

  • Master-eligible node – A node that has node.master set to true (default). It is responsible for lightweight cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes.
  • Data node – A node that has node.data set to true (default). Data nodes hold data and perform data related operations such as CRUD, search, and aggregations.
  • Coordinating node – Its main role is to route search and indexing requests from clients to data nodes. It behaves as smart load balancers.

In this guide, we are going to set up a two node Elasticsearch cluster with each node being master eligible.

 

Muvi Environment:

  • Node 1
  • Node 2

Installation

 

 

  • sudo apt-get update
  • sudo apt-get -y install apt-transport-https curl wget
  • sudo wget -qO – https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add –
  • sudo echo “deb https://artifacts.elastic.co/packages/7.x/apt stable main” | tee -a /etc/apt/sources.list.d/elastic-7.x.list
  • sudo apt-get update && apt-get install elasticsearch=7.5.2

 

 

Set Elasticsearch Cluster name

On each node, open the Elasticsearch configuration file and set the name of your Elasticsearch cluster.

Step1:

 

vim /etc/elasticsearch/elasticsearch.yml

 

# Use a descriptive name for your cluster:

cluster.name: muvi-cluster

Set Descriptive names for Elasticsearch Nodes

Step2:

 

Node 1

# ———————————— Node ————————————

# Use a descriptive name for the node:

node.name: muvi-node-01

Node 2

 

# ———————————— Node ————————————

# Use a descriptive name for the node:

node.name: muvi-node-02

Step 3:

Disable Memory Swapping

Swapping affects the stability of Elasticsearch cluster as it can cause nodes to respond slowly or even to disconnect from the cluster. Once of the ways of disabling memory swapping is by enabling memory lock. Hence, uncomment the line bootstrap.memory_lock: true.

 

# ———————————– Memory ———————————–

# Lock the memory on startup:

bootstrap.memory_lock: true

 

I recommend that you disable swapping using systemd by editing Elasticsearch service and adding the content below;

 

Step:4:

 

systemctl edit elasticsearch

 

[Service]

LimitMEMLOCK=infinity

 

Step:5:

 

Whenever a systemd service is modified, you need to reload the systemd configurations.

 

#sudo systemctl daemon-reload

 

Step: 6:

One of the recommended ways to disable swapping is to completely disable swap. This is fine if Elasticsearch is the only service running on the server.

 

swapoff -a

 

Step: 7:

Define the Roles of each Elasticsearch Node

As stated above, you can assign each node a respective role as master, data node, ingest node, coordinating node.. In this setup, we will configure all the three nodes to act as both master and data node.

# ———————————- Cluster ———————————–

# Use a descriptive name for your cluster:

cluster.name: es-nodes

 

node.master: true

node.data: true

 

Step: 7:

Discovery and cluster formation settings

There are two important discovery and cluster formation settings that should be configured before going to production so that nodes in the cluster can discover each other and elect a master node;

  • discovery.seed_hosts and cluster.initial_master_nodes.

discovery.seed_hosts setting Provides a list of master-eligible nodes in the cluster. Each value has the format host:port or host, where port defaults to the setting transport.profiles.default.port. This setting was previously known as discovery.zen.ping.unicast.hosts. Configure this setting on all Nodes as follows;

# ——————————— Discovery ———————————-

# Pass an initial list of hosts to perform discovery when this node is started:

# The default list of hosts is [“127.0.0.1”, “[::1]”]

 

discovery.seed_hosts: [“60.0.7.29”, “60.0.7.46”]

 

Step 8:

# Bootstrap the cluster using an initial set of master-eligible nodes:

#cluster.initial_master_nodes: [“muvi-node-1”, “muvi-node-2”]

cluster.initial_master_nodes: [“60.0.7.29”, “60.0.7.46”]

Step 9:

Set JVM Heap Size

Elasticsearch sets the heap size to 1GB by default. As a rule of thump, set Xmx to no more than 50% of your physical RAM but not more than 32GB.

vim /etc/elasticsearch/jvm.options

-Xms1g

-Xmx1g

Step 10:

Set maximum Open File Descriptor

Open below file and LimitNOFILE 

/usr/lib/systemd/system/elasticsearch.service

process

LimitNOFILE=65535

You also should set the maximum number of processes.

LimitNPROC=4096

Step 11:

Virtual Memory Settings

Open below file  and add vm.max_map_count as shown below.

/etc/sysctl.conf

 

vm.max_map_count=262144

 

Or

#echo “vm.max_map_count=262144” >> /etc/sysctl.conf

 

#sysctl vm.max_map_count

vm.max_map_count = 262144

 

Step 12:

 

Running Elasticsearch

 

Reload the systemd manager configuration.

 

#systemctl daemon-reload

Enable Elasticsearch to run on system boot.

 

#systemctl enable elasticsearch

 

Start Elasticsearch

 

#systemctl start elasticsearch

 

You can check the Elasticsearch status running the command below;

 

#systemctl status elasticsearch.service

#systemctl status elasticsearch.service

Step 13:

 

Check Elasticsearch Cluster Health

curl -X GET “<elbname>.elb.us-east-1.amazonaws.com:9200/_cluster/health?pretty”

 

root@ip-60-0-7-29:~# curl -X GET “<elbname>.elb.us-east-1.amazonaws.com:9200/_cluster/health?pretty”

{

  “cluster_name” : “muvi-cluster”,

  “status” : “green”,

  “timed_out” : false,

  “number_of_nodes” : 2,

  “number_of_data_nodes” : 2,

  “active_primary_shards” : 0,

  “active_shards” : 0,

  “relocating_shards” : 0,

  “initializing_shards” : 0,

  “unassigned_shards” : 0,

  “delayed_unassigned_shards” : 0,

  “number_of_pending_tasks” : 0,

  “number_of_in_flight_fetch” : 0,

  “task_max_waiting_in_queue_millis” : 0,

  “active_shards_percent_as_number” : 100.0

}

step 13

Step 14:

Check the Cluster Nodes

#curl -X GET “<elbname>.us-east-1.amazonaws.com:9200/_cat/nodes?v”

step 14

Don't forget to share this post!


Uma Shankar Pradhan
Uma Shankar has 9+ years of experience in EC2, VPC, S3, Glacier, IAM, SES, SNS, Cloud Watch, EBS, EFS, Elastic Load Balancer, Auto Scaling, RDS, Dynamo DB, AWS CLI, Cloud Front, Cloud Formation, Cloud watch, Route53, Amazon Aurora & AWS. He is an integral member of the Muvi DevOps team.

Related Posts

White Papers
blog_opt_6
Tags

Latest Tweets

Join Us On Facebook

Subscribe to our Newsletter

Take a Free Trial to experience
how easy it is to 

Launch an OTT Platform
Try for Free
No Credit Card required!
close-link