How to Install Elasticsearch, Logstash and Kibana (ELK Stack) on Ubuntu

Objective

ELK Stack, is a dynamic trio of open-source tools designed for unparalleled log management and data analytics. ELK, an acronym for Elasticsearch, Logstash, and Kibana, comprises three essential components, each playing a pivotal role in seamlessly collecting , processing , and visualizing data.

Elasticsearch:

Role : A distributed search and analytics engine.

Functionality : Elasticsearch excels in storing and indexing data, offering swift searches, advanced analytics, and comprehensive visualization capabilities. Its robust features support full-text search, structured queries, and real-time analytics.

Logstash:

Role : A versatile data processing pipeline.

Functionality : Logstash takes centerstage in collecting, processing, and transferring data from diverse sources to Elasticsearch. Its prowess lies in handling various input formats, applying filters for parsing and enriching data, and seamlessly transporting it to Elasticsearch for indexing.

Kibana:

Role : An intuitive data visualization and exploration tool.

Functionality : Kibana provides a user-friendly web interface to interact with data stored in Elasticsearch. Users can effortlessly create dashboards, employ charts and graphs for data visualization, and explore indexed data. Kibana is the go-to solution for monitoring, troubleshooting, and analyzing log data.

This powerful synergy of Elasticsearch, Logstash, and Kibana forms an indispensable stack for organizations seeking superior log management, real-time analytics, and visualization. Widely embraced across industries such as IT operations, security, and business intelligence, the ELK Stack empowers users to derive valuable insights from vast datasets generated by systems, applications, and devices. Furthermore, its inherent flexibility allows for seamless customization and expansion by integrating additional plugins and components tailored to specific needs . Elevate your data analytics game with the ELK Stack!

Let’s Get Started

Before embarking on the installation journey, ensure you have the following prerequisites in place:

• Ubuntu 22.04Server Setup :
  Have an Ubuntu 22.04 server at your disposal equipped with a robust configuration , boasting 4GB RAM and a dual-core processor. Ensure it's configured with anon-root sudo user for enhanced security.
• OpenJDK 11Installation :
  Make sure OpenJDK 11 is installed on your Ubuntu server. If you haven't done this yet, refer to the comprehensive guide on installing OpenJDK 11 on Ubuntu for step-by-step instructions. This ensures that your server is equipped with the Java Development Kit necessary for the upcoming tasks.
• Nginx Configuration :
  Nginx, the high-performance web server, is a crucial component. Ensure it's installed and configured on your server. Nginx will serve as the gateway, enhancing the performance and security of your applications.

Step 1 — Elasticsearch Installation and Configuration

When installing Elasticsearch on Ubuntu ,note that its components aren't in the default repositories. To set it up securely, import the Elasticsearch GPG key using cURL, ensuring package authenticity. Use the Elastic package source list to install Elasticsearch via APT, bolstering your system's security.

Code

                $ curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg
            

The Elastic source list should then be added to the sources.list.d directory, which is where APT will look for new sources:

Code

                $ echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main" | 
                sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
            

Next, make sure APT can read the updated Elastic source by updating your package lists:

Code

                $ sudo apt update
            

Now, let’s install Elasticsearch with this command:

Code

                $ sudo apt install elasticsearch
            

With Elasticsearch successfully installed, the next step is configuration. Utilize your preferred text editor, like nano, to edit the main configuration file, elasticsearch.yml:

Code

                $ sudo nano /etc/elasticsearch/elasticsearch.yml
            

By default, Elasticsearch listens on localhost and port 9200. Modify settings such as network.host and http.port if needed.

Initiate the Elasticsearch service using systemctl and allow a brief startup period to avoid connection errors:

Code

                $ sudo systemctl start elasticsearch
            

Code

                $ sudo systemctl enable elasticsearch
            

With Elasticsearch operational, proceed to install Kibana, the subsequent component in the Elastic Stack.

Step 2 — Kibana Dashboard installation and configuration

As per official guidelines, installing Kibana after Elasticsearch is essential to ensure the proper setup of dependencies. Following this sequential installation guarantees that each component relies on the correct foundations.

With the Elastic package source already integrated in the prior step, effortlessly install the remaining Elastic Stack components using apt:

Code

                $ sudo apt install kibana
            

Activate and initiate the Kibana services wiftly:

Code

                $ sudo systemctl enable kibana
            

Code

                $ sudo systemctl start kibana
            

To enable external access to Kibana, configured to listen on localhost, a reverse proxy using Nginx is essential. Follow these steps, assuming Nginx is already installed on your server.

Firstly, create an administrative Kibana user for secure web interface access. Use the openssl command to generate a username and password, storing them in the htpasswd.users file. Opt for anon-standard, secure username for enhanced security.

Command:

Code

                $ echo "kibanaadmin:`openssl passwd -apr1`" | sudo tee -a /etc/nginx/htpasswd.users
            

This creates the administrative user and password, paving the way for Nginx configuration. The next steps involve configuring Nginx to require this username and password for secure access.

Enter and confirm a password at the prompt; remember it for accessing the Kibana web interface later.

Moving forward, create an Nginx server block file. As an example, we'll refer to it as "your_domain," but feel free to choose a more descriptive name. If you have an FQDN and DNS records set up, consider naming the file after your FQDN.

Use your preferred text editor, like nano, to craft the Nginx server block file:

Code

                $ sudo nano /etc/nginx/sites-available/your_domain
            

Insert the following code block into the file, making sure to update "your_domain" to match your server’s FQDN or public IP address. This code directs Nginx to route your server’s HTTP traffic to the Kibana application on localhost:5601. Additionally, it configures Nginx to read the htpasswd.users file, enforcing basic authentication.

If you've completed the Nginx tutorial previously, you may already have this file. In that case, clear any existing content before adding the following:

Code

                server { listen 80; server_name your_domain; auth_basic "Restricted Access"; auth_basic_user_file /etc/nginx/htpasswd.users; 
                location / { proxy_pass http://localhost:5601; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; 
                proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } }
            

When you’re finished, save and close the file.

This configuration enhances Nginx to securely manage access to the Kibana application.

To activate the new configuration, create a symbolic link to the sites-enabled directory. Skip this step if you've already created a server block file with the same name during the Nginx prerequisite:

Code

                $ sudo ln -s /etc/nginx/sites-available/your_domain /etc/nginx/sites-enabled/your_domain
            

Then check the configuration for syntax errors:

Code

                $ sudo nginx -t
            

If any errors surface in the output, revisit your configuration file to ensure accurate content placement. Once you verify that the syntax is correct, proceed to restart the Nginx service: