Forem: molamk

Monitor your work station with Elastic Stack

molamk — Sun, 11 Oct 2020 08:21:17 +0000

The pitch

Unify logs, metrics, traces, and more in a scalable stack that's free, open, and built for speed. Monitor every element in your infrastructure to quickly resolve issues and consistently deliver exceptional digital experiences.

Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease.

# Import GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

# Install apt-transport-https
sudo apt-get install apt-transport-https

# Add repository to sources.list
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | \
  sudo tee /etc/apt/sources.list.d/elastic-7.x.list

# Install Elasticsearch
sudo apt-get update && sudo apt-get install elasticsearch

# Start the Elasticsearch service
sudo service elasticsearch start

# Verify that it's running
curl localhost:9200
# {
#   "name" : "parrot",
#   "cluster_name" : "elasticsearch",
#   "cluster_uuid" : "dYmxXrrJSIqF24FE7UcO7A",
#   "version" : {
#     "number" : "7.9.2",
#     "build_flavor" : "default",
#     "build_type" : "deb",
#     "build_hash" : "d34da0ea4a966c4e49417f2da2f244e3e97b4e6e",
#     "build_date" : "2020-09-23T00:45:33.626720Z",
#     "build_snapshot" : false,
#     "lucene_version" : "8.6.2",
#     "minimum_wire_compatibility_version" : "6.8.0",
#     "minimum_index_compatibility_version" : "6.0.0-beta1"
#   },
#   "tagline" : "You Know, for Search"
# }

Kibana

Kibana is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack. Do anything from tracking query load to understanding the way requests flow through your apps.

# Install kibana
sudo apt install kibana

To verify that Kibana is running, head over to localhost:5601. You should be able to see the dashboard right in your web-browser.

Actual monitoring

Metricbeat

Collect metrics from your systems and services. From CPU to memory, Redis to NGINX, and much more, Metricbeat is a lightweight way to send system and service statistics.

# Install metricbeat
sudo apt install metricbeat

# Start the service
sudo service start metricbeat

Heartbeat

Monitor services for their availability with active probing. Given a list of URLs, Heartbeat asks the simple question: Are you alive? Heartbeat ships this information and response time to the rest of the Elastic Stack for further analysis.

# Install heartbeat
sudo apt install heartbeat-elastic

# Start the service
sudo service start heartbeat-elastic

Filebeat

Forget using SSH when you have tens, hundreds, or even thousands of servers, virtual machines, and containers generating logs. Filebeat helps you keep the simple things simple by offering a lightweight way to forward and centralize logs and files.

# Installation
sudo apt install filebeat

# Configuration
sudo vim /etc/filebeat/filebeat.yml

In the /etc/filebeat/filebeat.yml go to the filebeat.inputs section and set enabled to `true.

The last step to get Filebeat up and running is to configure an "index pattern". To do this head over to localhost:5601 and follow these instructions:

Go to left-menu
Click on Discover
Create index pattern
In index pattern name: Write filebeat-*
Click on create index pattern
Same text box: Write filebeat-*
Click on Next Step
Select @timestamp as Time Field
Click on Create index pattern

How to use OpenAI's GPT-2 with React and Flask

molamk — Sat, 21 Sep 2019 22:27:44 +0000

Introduction

In this tutorial, we'll build a Flask & React app with GPT-2 capabilities. We'll go step by step, by tweaking the generator's "interface", then we'll build the Flask server and finally the React frontend.

By the end of this tutorial, here's what our app should look like:

Generating text with GPT-2

Thanks to pytorch-transformers, it's actually really easy to play with state of the art NLP models. We'll use a recipe found in pytorch-transformers/examples/run_generation.py as a template for our app.

First, let's install our dependencies

# Download the transformers package
pip3 install pytorch-transformers

# Get only the text generation file from the repository
wget https://raw.githubusercontent.com/huggingface/pytorch-transformers/master/examples/run_generation.py

Now that we have our generation script, we need to change it a bit so it plays nice with our Flask app. The script itself normally as a CLI tool with arguments, like this.

python3 ./run_generation.py \
    --model_type=gpt2 \
    --length=20 \
    --model_name_or_path=gpt2 \
    --promt="Hello world"

But since we want to call the text generation function from our Flask app, some changes are needed. First let's rename main() to generate_text() and give it some arguments. Those arguments are exactly the same as the ones we normally give if we run it directly in a shell. Here's what it looks like

def generate_text(
    padding_text=None,
    model_type='gpt2',
    model_name_or_path='gpt2',
    prompt='',
    length=20,
    temperature=1.0,
    top_k=0,
    top_p=0.9,
    no_cuda=True,
    seed=42,
):
    # Set the seed manually
    np.random.seed(seed)
    torch.manual_seed(seed)
    if n_gpu > 0:
        torch.cuda.manual_seed_all(seed)

    # The rest of the old main() code
    # We just need to replace args.* with
    # the corresponding function's arguments
    ...

That's it! Now we're ready to expose our feature through a REST API with Flask.

Building the Flask app

Our server will be pretty minimalistic, with only one endpoint that handles a POST request. In the body, we will provide the text which will serve as a "prompt" for GPT-2 to generate stuff. We'll also give a model which can be one of the 3 GPT-2 models, namely the small (117M), medium (345M) and large (774M).

from flask import Flask, abort, jsonify, request
from flask_cors import CORS, cross_origin

from .run_generation import generate_text

app = Flask(__name__)
cors = CORS(app)
app.config['CORS_HEADERS'] = 'Content-Type'


@app.route("/generate", methods=['POST'])
@cross_origin()
def get_gen():
    data = request.get_json()

    if 'text' not in data or len(data['text']) == 0 or 'model' not in data:
        abort(400)
    else:
        text = data['text']
        model = data['model']

        result = generate_text(
            model_type='gpt2',
            length=100,
            prompt=text,
            model_name_or_path=model
        )

        return jsonify({'result': result})

Front-end work

Now that we set up our Flask server, it's time to build to front-end. We'll have a simple form that takes a model and a textarea that inputs the text (prompt). I also used material-ui to have fancy form controls. Alright, let's set up our React app

# Create the app
create-react-app gpt2-frontend
cd gpt2-frontend

# Add some dependencies
yarn add @material-ui/core node-sass axios

We'll also use React Hooks to handle the state. I shamelessly copy-pasted API related boilerplate from this very thorough article. Now here's what our App.js looks like

function App() {
  const [text, setText] = useState("");
  const [model, setModel] = useState('gpt2');
  const [generatedText, postGenerateText] = postGenerateTextEndpoint();

  const generateText = () => {
    postGenerateText({ text, model, userId: 1 });
  }

  return (
    <div className='app-container'>
      <form noValidate autoComplete='off'>
        <h1>React GPT-2</h1>
        <SelectBox model={model} setModel={setModel} />
        <TextBox text={text} setText={setText} />
        <Button onClick={generateText} />
      </form>

      {generatedText.pending &&
        <div className='result pending'>Please wait</div>}

      {generatedText.complete &&
        (generatedText.error ?
          <div className='result error'>Bad Request</div> :
          <div className='result valid'>
            {generatedText.data.result}
          </div>)}
    </div>
  );
}

Grab the code from Github

https://github.com/molamk/gpt2-react-flask

Running a Node.js app with Docker, Kubernetes, and Minikube

molamk — Sat, 31 Aug 2019 00:51:07 +0000

Set up your tools

# Virtualbox (for virtualization)
brew cask install virtualbox

# HyperKit
brew install hyperkit

# Docker
brew cask install docker

# Kubernetes CLI & kubectl
brew install kubernetes-cli

# Minikube => Local Kubernetes
brew cask install minikube

# Helm => Chart management (optional)
brew install kubernetes-helm

Writing a (minimal) Node.js app

We won't dive into the details of how to write a good Node.js app in this tutorial. Our app will have a minimal server with only one route and one method, namely GET /. Of course, we can add as many features as we want, but for the purposes of this tutorial, we'll focus more on infrastructure with Docker, Kubernetes, and Minikube. Here's what our app will look like:

const express = require('express');

// Constants
const PORT = process.env.PORT || 3000;
const HOST = '0.0.0.0'

// App
const app = express();
app.get('/', (req, res) => {
    res.send('Hello world\n');
});

app.listen(PORT, HOST);
console.log(`Running on http://${HOST}:${PORT}`);

We only need one npm package, which is express. To install it, run:

npm install --save express

Dockerizing the app

We can dockerize our app by writing a Dockerfile, which is a set of steps Docker will run to bundle it. It looks like this:

FROM node:10

# Create app directory
WORKDIR /usr/src/app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./

RUN npm install
# If you are building your code for production
# RUN npm ci --only=production

# Bundle app source
COPY . .

EXPOSE 80
CMD [ "node", "index.js" ]

We'll also ignore some files, like the locally installed node_modules. To do that we create a .dockerignore file:

node_modules
Dockerfile
.dockerignore
npm-debug.log

Now that we're set, we need to actually build our Docker image then run the container. Since I always forget the exact commands to do so, I prefer to put those in a Makefile. Here's what it can look like:

image-name="molamk/node-app"

build:
    docker build -t $(image-name) .

run:
    docker run -p 3000:80 -d $(image-name)

Now we'll build the image, then run our container. It should give us "Hello World" response with a 200 status when we call it with curl.

# Build the image
make build

# Run the container
make run

# Call our API
curl -v localhost:3000

# HTTP/1.1 200 OK
# X-Powered-By: Express
# Content-Type: text/html; charset=utf-8
# Content-Length: 12
# ETag: W/"c-M6tWOb/Y57lesdjQuHeB1P/qTV0"
# Date: Sat, 24 Aug 2019 21:00:43 GMT
# Connection: keep-alive

# Hello world

Cool! Now that our app is dockerized, we can tag & push the image to Dockerhub. We'll add some stuff to our makefile to do that:

tag:
    docker tag molamk/node-app molamk/node-app:latest

push:
    docker push molamk/node-app

Local Kubernetes with Helm & Minikube

We'll use Helm to bundle our application as a package, ready to be deployed on Kubernetes. Here's a little more info about Helm:

"Helm helps you manage Kubernetes applications — Helm Charts helps you define, install, and upgrade even the most complex Kubernetes application.
Charts are easy to create, version, share, and publish — so start using Helm and stop the copy-and-paste." - The Helm Team

First, we need to initialize Helm, here's how we do that

# Fire up Minikube
minikube start
minikube addons enable ingress

# Initialization
helm init

# Update the repositories to their latest versions
helm repo update

After that, we create what's called a Chart which will contain the manifest files for our Kubernetes deployment and service.

helm create node-app

Now let's go into the chart generated folder node-app and edit some yaml. We'll set the repository to be our own Docker image that we pushed previously.

# Chart.yaml

apiVersion: v1
appVersion: "1.0"
description: Running a Node.js app with Docker, Kubernetes, and Minikube
name: node-app
version: 0.1.0

# values.yaml

replicaCount: 1

image:
  repository: molamk/node-app
  tag: latest
  pullPolicy: Always

env:
  containerPort: "80"

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

service:
  type: ClusterIP
  port: 80

ingress:
  enabled: true
  annotations: {}
  hosts:
    - host: minikube-node-app.local
      paths: ["/"]

  tls: []

resources: {}

nodeSelector: {}

tolerations: []

affinity: {}

We'll also modify the deployment.yaml file a little bit to infer our custom port.

# deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "node-app.fullname" . }}
  labels:
{{ include "node-app.labels" . | indent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ include "node-app.name" . }}
      app.kubernetes.io/instance: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: {{ include "node-app.name" . }}
        app.kubernetes.io/instance: {{ .Release.Name }}
    spec:
    {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
    {{- end }}
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:
            - name: "PORT"
              value: "{{ .Values.env.containerPort }}"
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
    {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
    {{- end }}

Now let's deploy that on Kubernetes. We'll use the Helm CLI to do so, then we'll verify that everything is set up correctly.

# Deploy
helm install node-app

# Verify that it's been set-up
helm ls

# NAME                    REVISION        UPDATED                         STATUS          CHART           APP VERSION     NAMESPACE
# wondering-cricket       1               Sun Aug 25 18:24:51 2019        DEPLOYED        node-app-0.1.0  1.0             default

Testing the whole set-up

Now let's tell our /etc/hosts file about our custom host minikube-node-app.local so we can call our endpoint. We'll the call it with curl which should return a Hello world response with a 200 status code.

# Putting our custom host into the host file
echo "$(minikube ip) minikube-node-app.local" | sudo tee -a /etc/hosts

# Calling our endpoint
curl -i minikube-node-app.local

# HTTP/1.1 200 OK
# X-Powered-By: Express
# Content-Type: text/html; charset=utf-8
# Content-Length: 12
# ETag: W/"c-M6tWOb/Y57lesdjQuHeB1P/qTV0"
# Date: Sat, 24 Aug 2019 21:00:43 GMT
# Connection: keep-alive

# Hello world

Grab the code

molamk/minikube-node-app

Sneaky Fingerprint and IP Tracker with React

molamk — Tue, 14 May 2019 15:53:21 +0000

Introduction

Most popular tracking methods involve persistent identifiers, like the famous cookies. But what if we delete those cookies? What if we go incognito?

It turns out that there other ways to track users without relying on persistent identifiers. Browser fingerprinting is one of them. Since it doesn't need to store anything, there's nothing to delete and going private won't do the trick.

We'll write a small React app to demonstrate how to collect a device's fingerprint and IP address metadata. The app will run entirely in the browser, and no data will be stored.

Motivation

By gathering that information together and storing it on its own servers, a site can track your browsing habits without the use of persistent identifiers stored on your computer, like cookies. Fingerprinting can also be used to recreate a tracking cookie for a user after the user has deleted it. Users that are aware of cookies can remove them within their browser settings, but fingerprinting subverts the built-in browser mechanisms that allow users to avoid being tracked. (EFF | The GDPR and Browser Fingerprinting: How It Changes the Game for the Sneakiest Web Trackers).

What we're building

To build the app, we'll use:

Fingerprintjs2 to collect the fingerprint. This will gives us the device's platform, RAM, number of cores and other juicy info.
The Extreme IP Lookup API to get the IP address metadata. This provides info such as the device's location, ISP, etc...
React Hooks to manage the application's state while minimizing boilerplate. Since we're going for a light-weight approach, we'll favor this option over something like Redux.

How it works

Getting the device's fingerprint

Fingerprintjs2 automatically detects the running browser's features, then selectively query the parameters that are available. Some of the identification methods it uses include:

Canvas fingerprinting. By drawing text and adding background colors, a script then calls the Canvas API to get a Base64 encoded representation of the binary pixel data. That representation is then turned into a hash, which will serve as a fingerprint.
Font detection. Based on what fonts you have, and how they are drawn. By measuring the dimensions of the filled, we can model a fingerprint.
Browser plugin probing.
Audio sampling.
WebGL fingerprinting.

To actually get the fingerprint we'll use the get function of Fingerprintjs2, and it looks like this:

import fp from "fingerprintjs2";

// We re-write the callback into a Promise style,
// so it plays nice with React Hooks
export const getFingerprint = () =>
  new Promise(resolve => {
    fp.get(components => {
      resolve(components);
    });
  });

Getting the IP address metadata

IP-based Geolocation is a mapping of an IP address or MAC address to the real-world geographic location of an Internet-connected computing or a mobile device. Geolocation involves mapping an IP address to the country, region (city), latitude/longitude, ISP and domain name among other useful things. More about here.

No magic involved here. Geo-location works by simply using a pre-populated database with relevant information. There are also many free (and paid) services that do this for you, like the one we're using in this tutorial.

We'll use a standard HTTP GET request to the extreme-ip-lookup REST API. We'll also specify that we want json as the returning response format.

// Standard HTTP GET using "fetch"
fetch("https://extreme-ip-lookup.com/json")
  .then(res => res.json())

Creating the component

Now let's bring everything together and create our main component. Since our data fetching is asynchronous, we'll use the useEffect hook to populate our component.

import React, { useEffect, useState } from "react";
import { getFingerprint } from "./utils";   // The method we wrote above
import DataTable from "./dataTable";        // Custom component to render our data

function App() {
  const [fingerprint, setFingerprint] = useState(null);
  const [ipData, setIpData] = useState(null);
  const [showReport, setShowReport] = useState(true);

  useEffect(() => {
    if (showReport) {
      fetch("https://extreme-ip-lookup.com/json")           // Get the IP data
        .then(res => res.json())
        .then(ip => Promise.all([ip, getFingerprint()]))    // Get the fingerprint
        .then(([ip, fp]) => {
          setFingerprint(fp);                               // Update the state
          setIpData(ip);
          setShowReport(false);
        });
    }
  }, [showReport]);

  return (
    <div>
      {ipData && fingerprint ? (
        <div>
          <DataTable title="IP Data" data={ipData} />
          <DataTable title="Fingerprint" data={fingerprint} />
        </div>
      ) : (
        <div>
          <h2>Please wait...</h2>
        </div>
      )}
    </div>
  );
}

export default App;

Notes

The app will not store any collected info, but we can easily change it and give it storage capabilities. We could:

Use the localStorage API to store the results in the browser.
Send the collected data to a server that we built that will store it in a database for future use.

Github repository

molamk / fingerprint

Sneaky fingerprint and IP address tracker

Sneaky fingerprint and IP tracker with React

Introduction

The most popular tracking methods involve persistent identifiers, like famous cookies. But what if we delete those cookies? What if we go incognito?

We'll write a small React app to demonstrate how to collect a device's fingerprint and IP address metadata. The app will run entirely in the browser, and no data will be stored.

Motivation

By gathering that information together and storing it on its own servers, a site can track your browsing habits without the use of persistent identifiers stored on your computer, like cookies. Fingerprinting can also be used to recreate a tracking cookie for a…

View on GitHub

Running it

git clone git@github.com:molamk/fingerprint.git
cd fingerprint

yarn install
yarn start

Live Demo Link

https://fingerprint.molamk.com