Complete Guide to Run Elasticsearch on Kubernetes

Follow the step by step guide below to deploy Elasticsearch on Kubernetes.

Why Elasticsearch/EFK on Kubernetes?

Elasticsearch is an open-source tool that ingests application data, indexes it then stores it for analytics.

Since it gathers large volumes of data while indexing different data types, Elasticsearch is often considered write-heavy. To manage such dynamic volumes of data, Kubernetes can make it easier to configure, manage, and scale Elasticsearch clusters. Kubernetes also simplifies the provisioning of resources for Elasticsearch using Infrastructure-as-Code configurations, abstracting cluster management, and increasing the automation and repeatability of your deployments.

The Elasticsearch operator by Elastic, called Elastic Cloud on Kubernetes, or ECK, has many capabilities to assist in the ongoing operations of Elastic on Kubernetes including:

  • 1. Managing & monitoring many clusters
  • 2. Scaling cluster capability up & down
  • 3. Scheduling backups

And can be found at: https://github.com/elastic/cloud-on-k8s or on the Operator Hub at: https://operatorhub.io/operator/elastic-cloud-eck

Deploying ElasticSearch / EFK on Kubernetes

mountainmountainmountainmountainmountainmountainmountainmountain
1. Before starting
2. Preconfiguration
3. Getting Started with OpenEBS
4. Installing Kudo Operator
5. Installing ElasticSearch using Kudo
6. Installing Kibana
7. Installing Fluentd-ES
8. Monitoring Elastic search

Elasticsearch is used in production on OpenEBS by companies like

cncf
ByteDance (Creator of Tiktok)
KubeSphere
Lockheed_Martin
optoro

Elasticsearch Operators

Mayadata supports multiple operators to run Elasticsearch on Kubernetes

Kudo Operators

Kudo Operators

Kubernetes Universal Declarative Operator is a toolkit that helps to manage stateful applications after they are deployed on Kubernetes Cluster. While Kubernetes already comes in with a lot of built in automation to run simple workloads, KUDO is designed to help complex scenarios.

OpenEBS Dynamic LocalPV Benefits for Elasticsearch

OpenEBS together with Elasticsearch gives a complete logging solution. Many OpenEBS users have shared their experience of using OpenEBS for local storage management in Kubernetes for Elasticsearch, including the Cloud Native Computing Foundation, ByteDance (TikTok), and Zeta Associates (Lockheed Martin).

Dynamic Cassandra

Persistent Volumes

While Kubernetes alone cannot store data generated by a cluster, persistent volumes can be used to sustain it for future use. To help with this, OpenEBS provisions local persistent volumes or LocalPV and allows for data to be stored on physical disks.

Integrations with Cassandra

Highly available logs

Even if the node fails or rebooted during upgrades, persistent volumes from OpenEBS continue to be highly available.

Monitoring Cassandra

Monitoring Elasticsearch

OpenEBS provides in-depth monitoring and storage for logs. You can check metrics and monitor Elasticsearch instances using Prometheus and Grafana.

Disaster Recovery for Cassandra

Backup & Restore

OpenEBS allows you to take backup of Elasticsearch to any object storage and restore it to the same or any Kubernetes cluster.

Resources to help you get up and running with ElasticSearch / EFK and OpenEBS:

Blog

Quickly benchmarking Elasticsearch ingestion on Kubernetes

In this blog we look at running Elasticsearch on OpenEBS in an easy way

Solution Guide

Deploying Elasticsearch on Kubernetes using OpenEBS LocalPV

This step by step solution guide explains the steps and important considerations for deploying Elasticsearch clusters on Kubernetes using OpenEBS Persistent Volumes.

Blog

Elasticsearch and OpenEBS LocalPV

In this blog, we explore how OpenEBS Local PV can provision data storage for Elasticsearch clusters.

Enterprise Support

Need enterprise support for OpenEBS under ElasticSearch/EFK and other workloads?

Talk to us