Forem: Yusuf Ganiyu

How to Setup Kibana and Elasticsearch on Windows

Yusuf Ganiyu — Sat, 08 Oct 2022 16:21:42 +0000

Hi guys, in this article, I will be showing you how to install, configure and connect Kibana to elasticsearch on Windows. In case you missed it, you can install elasticsearch on windows by following the steps defined in the previous article here.

Downloading Kibana

You can download Kibana from the official website here. Choose windows from the dropdown and click on the download button.

Configuring Kibana

Once downloaded, extract kibana and navigate into the bin directory inside kibana-8.4.3 folder then Click on kibana.bat.

You should see Kibana has not been configured. Click on the URL displayed in your terminal to get started.

Once you navigate to the URL, you should see something like this

Copy and paste the enrolment token from elasticsearch (you can get it by following the steps here)

If your enrolment token has expired or you get this error message while configuring with enrolment token

Use the command below to create a new token then copy and paste the newly generated token into kibana.

bin/elasticsearch-create-enrollment-token -s kibana

Configuring Elasticsearch manually

You can click on Configure Manually

Check address, enter the username and password, check the trust certificate and click on Configure Elastic to continue.

I strongly recommend you paste the enrolment token for smooth configuration.

If you made it to this point, well done! You are done with the configuration.

You will be asked to enter username and password (this is the username and password generated for you while setting up elasticsearch.

In case you forgot your elasticsearch password use the command below to reset it (you should run this command in elasticsearch bin directory)
elasticsearch-reset-password -u elastic

Copy and paste the new password and login. Voila! You are in!

If you made it to this point, you have successfully connected Kibana and Elasticsearch.

If you have any questions, you can drop them in the comments section or visit elastic forum to dicuss with other great minds.

Thanks for reading :).

How to Setup elasticsearch on Windows

Yusuf Ganiyu — Sat, 08 Oct 2022 15:35:44 +0000

Hi guys and welcome to another episode of the Elasticsearch series. In case you missed my previous series on How to Create a Full Autocomplete Search Application with Elasticsearch and NestJS, you can catch up here.

In this article, I will be showing you how to setup elastic search on Microsoft Windows. PS: This process works on Microsoft Windows 8 and above :).

Running Elasticsearch on Windows

Navigate to the elasticsearch download page and choose windows from the dropdown, then click on the Download button (as shown below)

Extract the downloaded file and navigate into the elasticsearch-8.4.3 folder (your version may be higher, but the process remains the same!)

In the bin folder, click on elasticsearch.bat

Once you've done that, you should see something like this:

On your browser, navigate to https://localhost:9200, you will be asked for username and password. By default username is elastic and password is shown on your terminal above

Once you input the right username and password, you should see this welcome page. This page shows you some information about elasticsearch installed on your computer.

That's it!

Elasticsearch is up and running on your windows machine! Yay!

Thanks for reading.

How to Create a Full Autocomplete Search Application with Elasticsearch, Kibana and NestJS - The Concluding Part

Yusuf Ganiyu — Sun, 11 Sep 2022 21:12:45 +0000

Hello and welcome to the concluding part of this series. In the previous articles, we walked through the installation and configuration of Elasticsearch, Kibana as well as importing data into Elasticsearch and querying them with NestJS (in case you missed them check them out here.

In this article I will be walking you through how to connect a simple react application with autocomplete feature that leverages NestJS backend with elasticsearch.

Setting up a react project

You can setup a simple react app using this command (or checkout this detailed react documentation

$ npx create-react-app nest-elastic-frontend

Once you app is setup, open it in your favorite IDE, mine is VSCode.

We will need to install axios as a dependency. If you prefer npm package manager, you will have to run npm install axios but if you preferred yarn, yarn add axios.

We need to update three files

1. public/index.html

You need to add bootstrap CDN. (PS: I have removed the comments to reduce the length of the file).

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <link rel="icon" href="%PUBLIC_URL%/favicon.ico" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta name="theme-color" content="#000000" />
    <meta name="description" content="Web site created using create-react-app" />
    <link rel="apple-touch-icon" href="%PUBLIC_URL%/logo192.png" />
    <link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
    <link
      href="//maxcdn.bootstrapcdn.com/bootstrap/4.1.1/css/bootstrap.min.css"
      rel="stylesheet"
      id="bootstrap-css"
    />
    <title>React App</title>
  </head>
  <body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="root"></div>
    <script src="//maxcdn.bootstrapcdn.com/bootstrap/4.1.1/js/bootstrap.min.js"></script>
    <script src="//cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
  </body>
</html>

2. src/App.js

import './App.css';
import axios from 'axios';
import { useState } from 'react';
const App = () => {
  const [searchResponse, setSearchResponse] = useState([]);
  const [totalValue, setTotalValue] = useState();

  const handleChange = async e => {
    const { data } = await axios.post('http://localhost:8000/movies/search', {
      data: {
        title: "e.target.value,"
      },
    });

    setSearchResponse(data.results);
    setTotalValue(data.total.value);
  };
  return (
    <div className='App'>
      <div className='container search-table'>
        <div className='search-box'>
          <div className='row'>
            <div className='col-md-3'>
              <h5>Search All Fields</h5>
            </div>
            <div className='col-md-9'>
              <input
                type='text'
                id='myInput'
                onChange={handleChange}
                className='form-control'
                placeholder='Search IMDB movies'></input>
            </div>
          </div>
        </div>
        <div className='search-list'>
          <h3>
            {totalValue ?? 0} {totalValue > 1 ? 'Records' : 'Record'} Found
          </h3>
          <table className='table' id='myTable'>
            <thead>
              <tr>
                <th>Title</th>
                <th>Overview</th>
                <th>Revenue:Budget ($)</th>
              </tr>
            </thead>
            <tbody>
              {searchResponse.map((res, idx) => (
                <tr key={idx}>
                  <td className='title'>{res.title}</td>
                  <td>
                    <p>{res.overview}</p>
                    <sub>"{res.tagline}"</sub>
                  </td>
                  <td>
                    <p>
                      <sub>
                        {res.revenue.toLocaleString()}:{res.budget.toLocaleString()}
                      </sub>
                    </p>
                  </td>
                </tr>
              ))}
            </tbody>
          </table>
        </div>
      </div>
    </div>
  );
};

export default App;

3. Lastly, src/index.css

body {
  margin: 0;
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu',
    'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
}

code {
  font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace;
}

.search-table {
  padding: 10%;
  margin-top: -6%;
}
.search-box {
  background: #c1c1c1;
  border: 1px solid #ababab;
  padding: 3%;
}
.search-box input:focus {
  box-shadow: none;
  border: 2px solid #eeeeee;
}
.search-list {
  background: #fff;
  border: 1px solid #ababab;
  border-top: none;
}
.search-list h3 {
  background: #eee;
  padding: 3%;
  margin-bottom: 0%;
}

.title {
  word-wrap: normal;
  width: 200px;
}

Running your app

Start your react app with yarn start or npm start depending on your preferred package manager.

Testing your app

Summary

In this article, we are able to visualize the result of our backend app running on elasticsearch queries in our react application.

Thanks for staying tuned!

Here is the link to the source code

How to Create a Full Autocomplete Search Application with Elasticsearch, Kibana and NestJS - Part 3

Yusuf Ganiyu — Sat, 10 Sep 2022 20:27:34 +0000

Hi guys, welcome to back to the Part 3 of the Elastic, Kibana and NestJS series. In the part 1 of this series, we installed and configured elasticsearch (check it here if you missed it), in the part 2, we connected elasticsearch with Kibana and ran a few queries (check here for part 2).

In this article, we will be writing the NodeJS code that will connect and query elasticsearch.

Loading data into elasticsearch

To enable us write our code effectively, we need data loaded into our elasticsearch. We will be using a sample dataset from Kaggle (Download it here.

Follow the FOUR steps below to load it up into elasticsearch

Open up Kibana (http://localhost:5601)
Under Get started by adding integrations click on Upload a file
Click on import and enter the name of the index you want to put the data in.
Click on import (final page)

If you made it to this point, you have successfully imported data into elasticsearch.

Querying for sample

Goto DevTools (Hamburger on the top left corner of the screen > Scroll down to Management > DevTools)

Run the query below (select it and click on the play button)

GET tmdb_movies/_search

If you see this, we are good to go!

Now, Let's dive into coding, shall we 😊?

NestJs

NestJS is a progressive Node. js framework that helps build server-side applications. Nest extends Node. js frameworks like Express or Fastify adding modular organization and a wide range of other libraries to take care of repetitive tasks. It's open-source, uses TypeScript, and is a very versatile Node.

Creating a NestJs Application

Run the command below to install nestcli and create a new NestJs Application (in the article, the name of the app will be nest-elastic).

$ npm i -g @nestjs/cli
$ nest new nest-elastic

You will be asked to select a package manager, you can select npm, yarn or pnpm. I will be selecting yarn (you can choose any other one you want 😉). Your project will be setup and we should be ready to code!

Adding elasticsearch to your app

Run the command below to add elasticsearch to your nest-elastic and other dependencies.

yarn add @elastic/elasticsearch @nestjs/elasticsearch @nestjs/config

In your root folder add your .env file with the following contents

ELASTICSEARCH_NODE=https://localhost:9200
ELASTICSEARCH_USERNAME=elastic
ELASTICSEARCH_PASSWORD=elasticPasswordGoesHere
ELASTICSEARCH_MAX_RETRIES=10
ELASTICSEARCH_REQ_TIMEOUT=50000
ELASTICSEARCH_INDEX=tmdb_movies

Let's create a separate module that handles only search using elasticsearch. A simple shortcut is to use the command below (you are welcome to do it manually, if you want too)

nest g resource search

Update the search.module.ts to have the content below

import { Module } from '@nestjs/common';
import { SearchService } from './search.service';
import { SearchController } from './search.controller';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { ElasticsearchModule } from '@nestjs/elasticsearch';

@Module({
  imports: [
    ConfigModule,
    ElasticsearchModule.registerAsync({
      imports: [ConfigModule],
      useFactory: async (configService: ConfigService) => ({
        node: configService.get('ELASTICSEARCH_NODE'),
        auth: {
          username: configService.get('ELASTICSEARCH_USERNAME'),
          password: configService.get('ELASTICSEARCH_PASSWORD'),
        },
        maxRetries: configService.get('ELASTICSEARCH_MAX_RETRIES'),
        requestTimeout: configService.get('ELASTICSEARCH_REQ_TIMEOUT'),
      }),
      inject: [ConfigService],
    }),
  ],
  controllers: [SearchController],
  providers: [SearchService],
  exports: [SearchService],
})
export class SearchModule {}

Update search.service.ts with the content below:

import { ConfigService } from '@nestjs/config';
import { Injectable } from '@nestjs/common';
import { ElasticsearchService } from '@nestjs/elasticsearch';

type dataResponse = {
  UnitPrice: number;
  Description: string;
  Quantity: number;
  Country: string;
  InvoiceNo: string;
  InvoiceDate: Date;
  CustomerID: number;
  StockCode: string;
};

@Injectable()
export class SearchService {
  constructor(
    private readonly esService: ElasticsearchService,
    private readonly configService: ConfigService,
  ) {}

  async search(search: {key: string}) {
    let results = new Set();
    const response = await this.esService.search({
      index: this.configService.get('ELASTICSEARCH_INDEX'),
      body: {
        size: 50,
        query: {
          match_phrase: search
        },
      },
    });
    const hits = response.hits.hits;
    hits.map((item) => {
      results.add(item._source as dataResponse);
    });

    return { results: Array.from(results), total: response.hits.total };
  }
}

Now, let's add movies modules

nest g resource movies

Update movies.controller.ts with the content below:

import { SearchService } from './../search/search.service';
import { Body, Controller, Post } from '@nestjs/common';

@Controller('movies')
export class MoviesController {
  constructor(private readonly searchService: SearchService) {}

  @Post('search')
  async search(@Body() body) {
    return await this.searchService.search(body.data);
  }
}

Then movies.module.ts

import { SearchModule } from './../search/search.module';
import { Module } from '@nestjs/common';
import { MoviesService } from './movies.service';
import { MoviesController } from './movies.controller';

@Module({
  imports: [SearchModule],
  controllers: [MoviesController],
  providers: [MoviesService],
})
export class MoviesModule {}

Finally, update app.module.ts

import { MoviesModule } from './movies/movies.module';
import { ConfigModule } from '@nestjs/config';
import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { SearchModule } from './search/search.module';

@Module({
  imports: [MoviesModule, ConfigModule.forRoot(), SearchModule],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

Your package.json should look like this

{
  "name": "nest-elastic",
  "version": "0.0.1",
  "description": "",
  "author": "Yusuf Ganiyu",
  "private": true,
  "license": "UNLICENSED",
  "scripts": {
    "prebuild": "rimraf dist",
    "build": "nest build",
    "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
    "start": "nest start",
    "start:dev": "nest start --watch",
    "start:debug": "nest start --debug --watch",
    "start:prod": "node dist/main",
    "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
    "test": "jest",
    "test:watch": "jest --watch",
    "test:cov": "jest --coverage",
    "test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
    "test:e2e": "jest --config ./test/jest-e2e.json"
  },
  "dependencies": {
    "@elastic/elasticsearch": "^8.4.0",
    "@nestjs/common": "^9.0.0",
    "@nestjs/config": "^2.2.0",
    "@nestjs/core": "^9.0.0",
    "@nestjs/elasticsearch": "^9.0.0",
    "@nestjs/mapped-types": "*",
    "@nestjs/platform-express": "^9.0.0",
    "reflect-metadata": "^0.1.13",
    "rimraf": "^3.0.2",
    "rxjs": "^7.2.0"
  },
  "devDependencies": {
    "@nestjs/cli": "^9.0.0",
    "@nestjs/schematics": "^9.0.0",
    "@nestjs/testing": "^9.0.0",
    "@types/express": "^4.17.13",
    "@types/jest": "28.1.4",
    "@types/node": "^16.0.0",
    "@types/supertest": "^2.0.11",
    "@typescript-eslint/eslint-plugin": "^5.0.0",
    "@typescript-eslint/parser": "^5.0.0",
    "eslint": "^8.0.1",
    "eslint-config-prettier": "^8.3.0",
    "eslint-plugin-prettier": "^4.0.0",
    "jest": "28.1.2",
    "prettier": "^2.3.2",
    "source-map-support": "^0.5.20",
    "supertest": "^6.1.3",
    "ts-jest": "28.0.5",
    "ts-loader": "^9.2.3",
    "ts-node": "^10.0.0",
    "tsconfig-paths": "4.0.0",
    "typescript": "^4.3.5"
  },
  "jest": {
    "moduleFileExtensions": [
      "js",
      "json",
      "ts"
    ],
    "rootDir": "src",
    "testRegex": ".*\\.spec\\.ts$",
    "transform": {
      "^.+\\.(t|j)s$": "ts-jest"
    },
    "collectCoverageFrom": [
      "**/*.(t|j)s"
    ],
    "coverageDirectory": "../coverage",
    "testEnvironment": "node"
  }
}

Running the app

You can fire up the app in dev environment using yarn start:dev

Testing

You can access the full source code here

Summary

In this article, we are able to import data into elastic using kibana and also connect our NestJs backend app to the leverage the power of elasticsearch.

In the next article, we will be building a simple frontend to query and visualize results from elasticsearch in realtime.

Thanks for reading!

How to Create a Full Autocomplete Search Application with Elasticsearch and NestJs - Part 2

Yusuf Ganiyu — Thu, 08 Sep 2022 22:05:56 +0000

Hello and welcome back to the part 2 of this series, in the previous article, I showed you how to setup and configure elasticsearch on your PC, in case you missed it, no worries, check it out here.

In this article, I will be walking you through how to setup and connect to elasticsearch through Kibana and NestJs application to create a backend server.

Hold up, what is Kibana?

Kibana is a source-available data visualization dashboard software for Elasticsearch, whose free and open source successor in OpenSearch is OpenSearch Dashboards. Read more

How does Kibana help us in this case you may ask?

Kibana is a great visualization tool that helps us with our elasticsearch queries and manipulation. It is part of B.E.L.K (Beats, Elasticsearch, Logstash and Kibana) stack. In case you don't know much about it, follow along and you will be just fine! (A separate article on Kibana can be written as an addendum to this series if you want, let me know in the comment section!)

Setting up and Configuring Kibana

In order to download Kibana, goto Kibana Download Page, click on the download button.

After downloading, extract the zip file using this command

$ tar -xzvf kibana-8.4.1-darwin-aarch64.tar.gz
$ cd kibana-8.4.1

There are two ways to configure and connect Kibana to elasticsearch, one way is to manually input the required keys into kibana.yml (in config/kibana.yml) or run Kibana and configure on the UI. I'm choosing the latter 😉!

Run kibana using this command:

bin/kibana

When starting Kibana for the first time, you will be presented with this screen

There are two ways to tackle this;

Use the elastic enrollment token (usually generated on your terminal when starting elastic for the first time) see below;

Copy and paste the enrollment token above to Kibana and click Configure Elastic. You will be fine

Manually configure Kibana with elastic using the elastic URL. Click on check address and click Configure elastic.

Enter kibana_system password generated. Or click forgot password (if you can not recall the password), paste the command below to your terminal to reset the password.

bin/elasticsearch-reset-password --username kibana_system

You will be asked to enter a verification code, check your terminal (where you are running kibana to retrieve it)

That's it! You are all set up.

Enter your elastic username and password to login to kibana and you are in!

Querying elastic using Kibana

Let's run a sample query to test our connection, shall we?

Click on the hamburger (three lines on the top left corner of your screen)
Scroll down to the bottom of the page
Click on Dev tools

You should see something like this:

On the left pane is where queries are written, the right pane is where the outputs are presented.

Ensure you click on any part of the query and select the play button.

The output of the query will be presented on the right pane.

Summary

In this article, we are able to connect successfully to elasticsearch and run sample queries using Kibana.

In the next article, I will show you how to Import data to elastic using Kibana and walk you through how to use the imported data in your NestJS application.

How to Create a Full Autocomplete Search Application with Elasticsearch and NestJS - Part 1

Yusuf Ganiyu — Tue, 06 Sep 2022 15:59:38 +0000

Elasticsearch is a distributed, RESTful search and analytics engine that can handle an expanding range of use cases. It offers a full-text search engine with a multitenant capability, an HTTP web interface, and schema-free JSON documents, all with simple installation.

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Read more

In this article, I will be walking you through how to setup elasticsearch on your PC.

How to setup, install and configure Elasticsearch for Mac

In order to download Elasticsearch, goto Elasticsearch Download Page, select the respective operating system and click on the download button as shown below:

After downloading, extract the compressed file using the command below

$ tar -xzvf elasticsearch-8.4.1-darwin-aarch64.tar.gz

Configuring Elasticsearch

After extracting elasticsearch, depending on your use case, you may want to configure elasticsearch to suit you. You can take a look at the config file using the command below:

$ vi elasticsearch-8.4.1/config/elasticsearch.yml

Cluster: You can have multiple nodes in your cluster, in order to do this, you need to ensure that the cluster name on your nodes matches.

# -------------------------------- Cluster ----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: air-elastic
#

Node: This is an identifier for the individual nodes in your cluster

# -------------------------------- Node ----------------------------------
#
# Use a descriptive name for your cluster:
#
node.name: airscholar-node
#

For other configurations, check elasticsearch.yml to fine tune the configs to suit your use case.

Starting up Elasticsearch

In order to start up elasticsearch engine use the command below:

$ bin/elasticsearch

When starting up elasticsearch for the first time, you will be presented with a password that needs to be changed

Changing your password for the first time

You should change your password after starting up the elasticsearch for the first time, of course, you can do that using this command:

bin/elasticsearch-reset-password -u elastic

NOTE: ENSURE YOU KEEP YOUR ELASTIC PASSWORD SAFE, IT WILL BE REQUIRED IN ORDER TO CONNECT TO ELASTICSEARCH

Testing Elasticsearch

In your browser, goto https://localhost:9200 or the specified hostname and port in your elasticonfig.yml.
You will be asked for username (elastic) and password (the password you changed to when starting elasticsearch for the first time)

You should see something like this.

{
  "name" : "airscholar-node",
  "cluster_name" : "air-elastic",
  "cluster_uuid" : "7g3LmMYFRmyxTyR6ZI-b7A",
  "version" : {
    "number" : "8.4.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "2bd229c8e56650b42e40992322a76e7914258f0c",
    "build_date" : "2022-08-26T12:11:43.232597118Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

If you have any challenge setting up elasticsearch, feel free to drop a comment, I will try to respond. You can also check official elasticsearch documentation for a more detailed information about elasticsearch.

Summary

Congratulations our elasticsearch guru, you made it!

Now that elasticsearch is up and running, your Elasticsearch node should be completely functional. The process is pretty much the same if you have multiple nodes and you do not have to worry about the nodes in your cluster because elasticsearch fully supports automated routing and load balancing.

Thanks for reading!

In the next article, I will walk you through how to setup and connect your NestJs application with Elasticsearch. Stay tuned!