VK Cloud logo
Updated atMarch 27, 2024   02:33 PM

Installing the ELK Stack on Ubuntu 18.04

This article describes the installation of the ELK stack on a Linux operating system — Ubuntu 18.04.

The ELK stack is a powerful set of tools for efficiently solving a wide range of data collection, storage and analysis tasks:

  • Elasticsearch is a full text search solution built on top of Apache Lucene with added convenience.
  • Logstash is a utility for collecting, filtering and then redirecting to the final data storage. This mechanism provides a real-time pipeline. It can take data from multiple sources and convert it into JSON documents.
  • Kibana is an application that allows you to take and search data from Elasticsearch and build visual graphs.

Requirements

  • Operating system Ubuntu version 18.04.
  • Installed Nginx web server.
  • Installed Java virtual machine.
  • User with access to the sudo command.

Installing the Nginx web server

Compared to the Apache web server, the Nginx web server uses fewer resources to host large, high-traffic sites. Thanks to the Nginx architecture, you can easily scale up to hundreds of thousands of concurrent connections.

To install and perform the initial configuration of the Nginx web server:

  1. Open a terminal window.

  2. Update the package indexes by running the command:

    sudo apt update
  3. Install the Nginx web server by running the command:

    sudo apt install nginx -y
  4. To test the operation of the web server, launch a web browser and enter the IP address of the web server in the address bar.

    If the installation is successful, the following web server page will open:

Installing the Java Virtual Machine

The ELK stack requires a Java virtual machine to run. To install JVM:

  1. Open a terminal window.

  2. Install the JVM software package by running the command:

    sudo apt install default-jre -y

    This will install the Java Runtime Environment (JRE) package.

  3. Install the JDK software package, which includes the Java compiler, standard Java class libraries, examples, documentation, and various utilities. To do this, run the command:

    sudo apt install default-jdk -y

Installing and configuring Elasticsearch

To install and perform the initial configuration of Elasticsearch:

  1. To check the current version of Elasticsearch, go to: https://www.elastic.co/downloads/elasticsearch.

  2. Open a terminal window.

  3. Import the GPG Elasticsearch public key, which is used to protect Elastic packages, by running the command:

    sudo wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt key add
  4. Add the Elastic packages to the sources.list.d system repositories directory by running the command:

    sudo echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
  5. Update the package indexes by running the command:

    sudo apt update
  6. Install Elasticsearch by running the command:

    sudo apt install elasticsearch
  7. Make changes to the elasticsearch.yml configuration file. For this:

    1. Open this file for editing by running the command:

      sudo nano /etc/elasticsearch/elasticsearch.
    2. Find the line:

      #network.host: 192.168.0.1

      Replace it with the line:

      network.host: localhost

      After editing the .yml config file, make sure it doesn't have extra spaces and/or indents!

    3. Save your changes using the keyboard shortcut CTRL+O and finish editing using the keyboard shortcut CTRL+X.

  8. Start the Elasticsearch service by running the command:

    sudo systemctl start elasticsearch
  9. Check the startup status of the Elasticsearch service by running the command:

    sudo systemctl status elasticsearch
  10. If an error is displayed:

    Do the following:

    1. Open the file containing the Java virtual machine settings by running the command:

      sudo nano /etc/elasticsearch/jvm.options
    2. Find the parameters that define the minimum and maximum amount of RAM for Java:

    3. Specify the required values ​​in the -Xms1g and -Xmx1g parameters. For example, for an operating system with 1 GB of RAM, you can specify:

      1-Xms128m
      2-Xmx128m
    4. Save your changes using the CTRL+O key combination and finish editing using the CTRL+X key combination.

    5. Start the Elasticsearch service and check the status. If there are no errors, the following will be displayed:

  11. To start the Elasticsearch service automatically when the operating system is restarted, run the command:

    sudo systemctl enable elasticsearch
  12. To test access to the Elasticsearch service, send an HTTP request by running the command:

    curl -X GET localhost:9200

    If the installation of Elasticsearch was successful, the following information will be displayed:

Installing and configuring Kibana

To install and perform the initial configuration of Kibana, do the following:

  1. Make sure you have successfully installed Elasticsearch.

  2. Open a terminal window.

  3. Install Kibana by running the command:

    sudo apt install kibana
  4. Start Kibana by running the command:

    sudo systemctl start kibana
  5. To start the Kibana service automatically when the operating system is restarted, run the command:

    sudo systemctl enable kibana
  6. To check the status of Kibana, run the command:

    sudo systemctl status kibana
  7. Make changes to the kibana.yml configuration file. For this:

    1. Open this file by running the command:

      sudo nano /etc/kibana/kibana.yml
    2. Find the line:

      #server.port: 5601

      And replace it with the line:

      server.port: 5601
    3. Find the line

      #server.host: "localhost"

      And replace it with the line:

      server.host: "localhost"
    4. Find the line:

      #elasticsearch.hosts: ["http://localhost:9200"]

      And replace it with the line:

      elasticsearch.hosts: ["http://localhost:9200"]
    5. Save changes using CTRL+O and finish editing using CTRL+X

  8. Create an administrator account to access the Kibana web interface. To do this, run the command:

    echo "mcskibadmin:\`openssl passwd -apr1\`" | sudo tee -a /etc/nginx/htpasswd.users

    where mcskibadmin is the login of the administrator account, htpasswd.users is the file where credentials are stored.

    Then enter the password.

  9. Create a file with a virtual site for the Nginx web server by running the command:

    sudo nano /etc/nginx/sites-available/elk
  10. Add the following information to this file:

    1server {
    2listen 80;
    3
    4server_name <web server external IP address>;
    5
    6auth_basic "Restricted Access";
    7auth_basic_user_file /etc/nginx/htpasswd.users;
    8
    9location / {
    10proxy_pass http://localhost:5601;
    11proxy_http_version 1.1;
    12proxy_set_header Upgrade $http_upgrade;
    13proxy_set_header Connection 'upgrade';
    14proxy_set_header Host $host;
    15proxy_cache_bypass $http_upgrade;
    16}
    17}

    Save your changes using the keyboard shortcut CTRL+O and finish editing using the keyboard shortcut CTRL+X.

  11. Activate the new Nginx configuration by running the command:

    sudo ln -s /etc/nginx/sites-available/elk /etc/nginx/sites-enabled/
  12. Restart Kibana by running the command:

    sudo systemctl restart kibana
  13. Restart the Nginx web server by running the command:

    sudo systemctl restart nginx
  14. Make sure that the syntax of the nginx configuration file does not contain errors by running the command:

    sudo nginx -t

Installing and configuring Logstash

To install and perform the initial setup of Logstash:

  1. Install Logstash by running the command:

    sudo apt install logstash
  2. Create and configure a configuration file containing rules for receiving information from beats agents. For this:

    1. Create the file 02-beats-input.conf by running the command:

      sudo nano /etc/logstash/conf.d/02-beats-input.conf
    2. Add the following lines to this file:

      1input {
      2beat {
      3port => 5044
      4}
      5}
    3. Save your changes using the CTRL+O key combination and finish editing using the CTRL+X key combination.

  3. Create and configure the 30-elasticsearch-output.conf configuration file containing the rules for storing beats in Elasticsearch information. For this:

    1. Create the 30-elasticsearch-output.conf file by running the command:

      sudo nano /etc/logstash/conf.d/30-elasticsearch-output.conf
    2. Add the following lines to this file:

      1output {
      2elasticsearch {
      3hosts => ["localhost:9200"]
      4sniffing => true
      5manage_template => false
      6template_overwrite => true
      7index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      8document_type => "%{[@metadata][type]}"
      9}
      10}
    3. Save your changes using the CTRL+O key combination and finish editing using the CTRL+X key combination.

  4. Create a file containing rules for filtering and structuring incoming data. For this:

    1. Create the file 10-system-filter.conf by running the command:

      sudo nano /etc/logstash/conf.d/10-logstash-filter.conf
    2. In the file that opens, place the following lines:

      1input { stdin { } }
      2filter {
      3grok {
      4   match => { "message" => "%{COMBINEDAPACHELOG}" }
      5}
      6date {
      7   match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
      8}
      9}
      10output {
      11elasticsearch { hosts => ["localhost:9200"] }
      12stdout { codec => rubydebug }
      13}
    3. Save your changes using the CTRL+O key combination and finish editing using the CTRL+X key combination.

  5. Check the Logstash configuration by running the command:

    sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t
  6. Start Logstash by running the command:

    sudo systemctl start logstash
  7. To start the Logstash service automatically when the operating system is rebooted, run the command:

    sudo systemctl enable logstash

Installing and configuring Filebeat

Filebeat allows you to collect data (beats) from various sources and transfer them to Logstash or Elasticsearch on Linux-like systems.

To install Filebeat:

  1. Open a terminal.

  2. Install Filebeat by running the command:

    sudo apt install filebeat
  3. Set up the filebeat.yml configuration file. For this:

    1. Open this file:

      sudo nano /etc/filebeat/filebeat.yml
    2. Prevent Filebeat from sending data directly to Elasticsearch. To do this, find the lines:

      1output.elasticsearch:
      2# Array of hosts to connect to.
      3hosts: ["localhost:9200"]

      And replace them with the lines:

      1#output.elasticsearch:
      2# Array of hosts to connect to.
      3#hosts: ["localhost:9200"]
    3. Tell the Filebeat service to use Logstash as a log collector. To do this, find the lines:

      1#output.logstash:
      2# The Logstash hosts
      3#hosts: ["localhost:5044"]

      And replace them with the lines:

      1output.logstash:
      2# The Logstash hosts
      3hosts: ["localhost:5044"]

      Save your changes using the CTRL+O key combination and finish editing using the CTRL+X key combination.

  4. Enable the Logstash module. To do this, run the command:

    sudo sudo filebeat modules enable logstash
  5. To view the included modules, run the command:

    sudo filebeat modules list
  6. Download the Elasticsearch index template by running the command:

    sudo filebeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
  7. Dashboards allow you to visualize the Filebeat data sent to Kibana. To enable the dashboard, run the command:

    sudo filebeat setup -e -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601
  8. Start Filebeat by running the command:

    sudo systemctl start filebeat
  9. To start the filebeat service automatically when the operating system is rebooted, run the command:

    sudo systemctl enable filebeat
  10. To verify that Elasticsearch is receiving data, query the Filebeat index with the command:

    curl -XGET 'http://localhost:9200/filebeat-\*/_search?pretty'

The installation of the ELK stack is complete.

In the address bar of your web browser, enter the IP address of your Elastic server. Use your administrator credentials to sign in. After successful authorization, you will be redirected to the Kibana main page.