Forem: aakhtar3

Build and host your own observability solution

aakhtar3 — Sun, 12 May 2024 02:22:25 +0000

Build and host your own observability solution with docker and open source software.

Network Traffic

Review source code @ https://github.com/osirislab/observability

High level overview of how the control node will run SIEM, while hosts are sending telemetry data.

Portainer

Create install portainer script, set execute permission, and run with docker

./install-portainer.sh

#!/bin/bash

docker volume create portainer_data

docker run -d \
    --name portainer \
    --restart=always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v portainer_data:/data \
    -p 8000:8000 \
    -p 9443:9443 \
    portainer/portainer-ce:latest

Head over to https://localhost:9443 and create admin credentials.

Agent

Separately you can install an agent on a seperate host. This will allow you to manage multiple docker hosts from a single portal.

Stack

Build your siem stack on portainer using an .env and docker-compose.yml

Create

Env

siem.env

Identify your username and replace $(whoami)

PUID=1000
PGID=1000
TZ=America/New_York
DOCKER_DATA=/home/$(whoami)/siem/data
DOCKER__CONFIG=/home/$(whoami)/siem/config

Compose Common Attributes

These attributes are meant to apply Do Not Repeat Yourself (DRY) templating.

version: "3.9"

x-common: &common
  restart: unless-stopped
  security_opt:
    - no-new-privileges:true
  labels:
    - 'logging=promtail'
    - 'logging_jobname=containerlog'

x-environment: &environment
  TZ: $TZ
  PUID: $PUID
  PGID: $PGID

Agents

These service are intended to be applied to every host, which allows the portainer control node access to docker deployments and adding observability agents for metrics and logging.

services:
  ######## Docker ########
  portainer-agent:
    container_name: portainer-agent
    image: 'portainer/agent:2.19.4'
    <<: *common
    volumes:
      - '/var/lib/docker/volumes:/var/lib/docker/volumes'
      - '/var/run/docker.sock:/var/run/docker.sock'
    ports:
      - 9001:9001
    environment:
      <<: *environment

  ######## Metrics ########
  cadvisor:
    depends_on:
      - portainer-agent
    container_name: cadvisor
    image: gcr.io/cadvisor/cadvisor:v0.47.1
    platform: linux/aarch64
    devices:
      - /dev/kmsg:/dev/kmsg
    <<: *common
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
    ports:
      - 8080:8080
    environment:
      <<: *environment

  node-exporter:
    depends_on:
      - portainer-agent
    container_name: node-exporter
    image: prom/node-exporter:latest
    command: 
      - '--path.procfs=/host/proc' 
      - '--path.sysfs=/host/sys'
      - --collector.filesystem.ignored-mount-points
      - "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)"
    <<: *common
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    ports:
      - 9100:9100
    environment:
      <<: *environment

  ######## Logging ########
  promtail:
    depends_on:
      - portainer-agent
    container_name: promtail
    image: grafana/promtail:2.9.3
    command: -config.file=/etc/promtail/config.yml
    <<: *common
    volumes:
      - $DOCKER__CONFIG/promtail/config.yml:/etc/promtail/config.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - 9080:9080
      - 1514:1514
    environment:
      <<: *environment

  dozzle:
    depends_on:
      - portainer-agent
    container_name: dozzle
    image: amir20/dozzle:latest
    <<: *common
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - 8082:8080
    environment:
      <<: *environment
      DOZZLE_LEVEL: info
      DOZZLE_TAILSIZE: 300
      DOZZLE_FILTER: "status=running"

Metrics

Prometheus will scrape promtail for new metrics.

services:
  prometheus:
    depends_on:
      - node-exporter
      - cadvisor
    container_name: prometheus
    image: prom/prometheus:latest
    user: root
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.enable-admin-api
      - --web.enable-lifecycle
    <<: *common
    volumes:
      - $DOCKER__CONFIG/prometheus/config.yml:/etc/prometheus/prometheus.yml:ro
      - $DOCKER_DATA/prometheus:/prometheus
    ports:
      - 9090:9090
    environment:
      <<: *environment

Logging

These services will handle syslog/container logs and store on mini/loki.

services:
  minio:
    container_name: minio
    image: minio/minio:latest
    user: root
    entrypoint: sh
    command: -c 'mkdir -p /data/loki && /usr/bin/docker-entrypoint.sh minio server /data'
    healthcheck:
      interval: 30s
      retries: 3
      test:
        - CMD
        - curl
        - -f
        - http://localhost:9000/minio/health/live
      timeout: 20s
    <<: *common
    volumes:
      - $DOCKER_DATA/minio:/data
    environment:
      <<: *environment
      MINIO_ACCESS_KEY: minio123
      MINIO_PROMETHEUS_AUTH_TYPE: public
      MINIO_SECRET_KEY: minio456

  loki:
    depends_on:
      - minio
    container_name: loki
    image: grafana/loki:2.9.3
    user: root
    command: -config.file=/etc/loki/loki-config.yml
    <<: *common
    volumes:
      - $DOCKER__CONFIG/loki/config.yml:/etc/loki/loki-config.yml:ro
      - $DOCKER_DATA/loki:/tmp
    ports:
      - 3100:3100
    environment:
      <<: *environment

  syslog-ng:
    depends_on:
      - promtail
    container_name: syslog-ng
    image: ghcr.io/axoflow/axosyslog:latest
    command: -edv
    <<: *common
    volumes:
      - $DOCKER__CONFIG/syslog-ng/config.conf:/etc/syslog-ng/syslog-ng.conf:ro
    ports:
      - 514:514/udp
    environment:
      <<: *environment

Alerting

Monitor your services and get notified if something goes down.

services:
  uptime-kuma:
    image: 'louislam/uptime-kuma:1'
    container_name: uptime-kuma
    <<: *common
    volumes:
      - $DOCKER_DATA/uptime-kuma:/app/data
    ports:
      - 3001:3001
    environment:
      <<: *environment

Dashboard

Grafana Dashboard for displaying metrics and logs.

services:
  grafana:
    depends_on:
      - prometheus
      - loki
    image: grafana/grafana:latest
    container_name: grafana
    <<: *common
    volumes:
      - $DOCKER__CONFIG/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro
      - $DOCKER__CONFIG/grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards:ro
      - $DOCKER__CONFIG/grafana/dashboards:/var/lib/grafana/dashboards:ro
    ports:
      - 3000:3000
    environment:
      <<: *environment
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
      GF_AUTH_BASIC_ENABLED: "false"
      GF_AUTH_DISABLE_LOGIN_FORM: "true"
      GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/no_folder/syslog_overview.json
      GF_INSTALL_PLUGINS: "grafana-clock-panel,grafana-simple-json-datasource,grafana-worldmap-panel,grafana-piechart-panel,cloudflare-app"

Mine XMR on Raspberry Pi

aakhtar3 — Wed, 08 May 2024 03:03:17 +0000

I have been playing around with xmrig to see if I could make any money with a few raspberry pi v3 that have been collecting dust. I choose XMR (Monero) due to it being ASIC resistant. Which allows the little guys a chance to get rewards for contribution to the blockchain. Also, I am a big fan of the privacy coin.

After testing for 5 months, the conclusion is that the Raspberry pi v3 is very old and cannot compete with new powerful cpu/gpus. However, the Raspberry pi v5 is new and has potential to output more computation. In addition, I packaged the software on alpine container to run native to the ARM architecture and the results were very surprising.

I am still in the red due to electricity cost, but ARM helps with the cost due to being energy efficient. The learning in this project is intangible and was worth the minor expense.

Pi Cluster

Building a compact cluster has advantages, by sharing electricity over USB hubs or Power Over Ethernet and segmenting the cluster into its own network.

ARM is energy efficient and can share power over USB hub of 5v
Stackable nodes in a segmented network
Egress over privacy VPN to remain anonymous

Upgrade Hardware

Switched from Pi v3 to Pi v5.

Pi v3 - Over 5 months I made $0.00 with a 1 H/s
Pi v5 - In a day was hitting consistent ~150 H/s and finally made $0.01

Use ARM architecture

Avoid cross architecture emulation on docker, which causes performance bottlenecks. Instead use native ARM Architecture when packaging your containers.

Pi v3 arm - 1 H/s
Pi v5 amd -> arm - ~150 H/s
m3 arm - ~1 KH/s
Pi v5 arm - ~1.2 KH/s

Summary

Final summary of current setup

Peaked at 3.5 KH - Improved by 3,500%
Now make $0.05 per day

1 m1 2 core arm
1 m2 2 core arm
5 Pi v5 20 core arm

Quick Setup with Wallet



# https://github.com/aakhtar3/xmrig
docker run -d \
    --name xmrig \
    -e URL=${POOL} \
    -e WALLET=${WALLET} \
    -e NODE_NAME=$(hostname) \
    --cpus="2" \
    --cpu-quota=400000 \
    --cpu-period=100000 \
    aakhtar3/xmrig

Create ${WALLET} and run software on docker
Join a pool ${POOL} - pool.hashvault.pro:443
Tweak CPU usage based on your cores

Kubernetes TLDR

aakhtar3 — Mon, 18 Mar 2024 02:30:13 +0000

Kubernetes is a dense topic, so here are some key topics to review.

Kubernetes	Component	Application	Istio
Terraform	Pod	Service	Control Plane
Cluster	Config Map	Service Registry	Data Plane
Control Plane Node	Secrets	Service Discovery	Traffic Management
Worker Node	Persistent Volume	Deployment	Security
Namespace	Policies	Stateful Sets	Observability

Kubernetes

Orchestration service to automate application deployments at scale.

Terraform

Infrastructure as code to deploy cloud resources that will be used to run Kubernetes.

Lifecycle

terraform init - Set up backend and pull dependencies from providers
- .tfstate - s3
- .lock - dynamodb
terrafrom validate - Validate Configs
terraform fmt - lint suggestions for styling code
terratest - Testing framework in Go
terraform plan - Show plan to build desired state and observed state
terraform apply - Build infrastructure to be in desired state
terraform destroy - Clean up infrastructure

Files

provider.tf - terraform block, backend, provider config, alias
main.tf - Resource blocks which define resources to be created
- resource - declare new component
- data - pull resource from cloud provider
- module - pull resource from external source
variables.tf - variables declarations for resources
output.tf - Key values generated on successful apply
*.tfvars - environment specific default variables

Helpers

meta arguments - depends on, count, for_each, locals
expressions - operators (arithmetic, equality, comparison, logical), conditionals, splat, constraints
functions - String/Number/Collection types, encoding/crpyto/hash, IP networking, filesystem, date/time

Cluster

Cluster - Multiple Control plane nodes for High Availability
Traffic
- North <-> South - Client <-> VPC <-> Ingress
- East <-> West - AZ A <-> AZ B <-> AZ C

Control Plane Node

Api Server - Front end of cluster on RESTful API over HTTPS
Cluster Store (etcd) - Holds desired state of all components of cluster
Controller (Manager) - Cluster intelligence, watching cluster to reconcile observed state and desired state
Scheduler - Handles tasks and assign to capable nodes for work

Worker Node

Kublet - Agent handles workload from cluster and reporting
Runtime - kernel tasks using containerd to manage lifecycle
Kube-proxy - Cluster networking and load balancing

Namespace

Default Namespace - Components will be placed here if not specified
Named Namespace - Isolate components into segmented namespaces
Isolation
- Soft - Segment by namespace
- Hard - Segment by cluster

Component

Key atomic componentes to build applications on Kubernetes.

Pod

Init container - Runs once before main container
Main container - Core functionality of service
Sidecar container - Optional containers to side load computation
- Proxy - Handles ingress traffic
- Ambassador - Handles egress traffic
- Adapter - Handles telemetry traffic

Config Map

Environment Variable - Data can be injected to container environment
Container Argument - Pass arguments into a container status up command
Files in Volume - Mount config file into container environment

Secrets

Native Secrets - Does not encrypt in transit or flight, stores in tmpfs as base64 and decode to plain text
Vault Secrets - Require 3rd party integration to mange secrets

Persistent Volume

Persistent Volume Claim (PVC) - Pod need to request/claim volume and grant access to PVs
Persistent Volume (PV) - Maps to external storage providers
Storage Classes (SC) - Proxy between PVC and SP and automates the connectivity
Container Storage Interface (CSI) - Interface plugin to connect to external/cloud SPs
Storage Providers (SP) - Physical storage disk drives

Policies

Role-based access control (RBAC) - Regulate access to groups when invoking the API
Pod Security Standards (PSS) - Privileged, Baseline, Restricted
Pod Security Admission (PSA) - enforce, audit, warn

Application

Package applications into segmented resources on Kubernetes cluster.

Service

Ingress - Handle many services on a single load balancer
Load Balancer - Cloud load balancer to handle traffic for service
Node Port - Port exposed on port to access service
Cluster IP - Ip obtained from service registry for service

Service Registry

Kube-system (NS) - Cluster namespace for control plane
Kube-dns (SVC) - Service registry and discovery
Coredns (POD) - Application to handle dns traffic

Service Discovery

Cluster Store - Stores DNS records from coredns
Fully Qualified Domain Name (FQDN) - {service}.{namespace}.svc.cluster.local

Deployment

Deployment - Stateless application
Replica Set - Deploy multiple pods in parallel

Stateful Sets

Stateful Sets - Stateful application that needs to mount data
Replica Set - Deploy multiple pods in sequence
Persistent Volume - Requires volume for application

Istio

Use Service Mesh to mange Kubernetes traffic, security, and observability.

Control Plane

Pilot - Service Discovery and configuring sidecar proxies
Citadel - Automated key and certificate management
Galley - Validate and deploy Istio configuration
Container Network Interface (CNI) - Network initialization, init container to sidecar proxy
Core DNS - DNS resolution
Policy - CDA Policy enforcer
Cert Manger - Issue and renew TLS certifications
Prometheus - Metrics collector
Sidecar Injector - Namespace enabled sidecar poxy
Ingress Gateway - Handle incoming traffic into mesh
Egress Gateway - Handle outgoing traffic from mesh

Data Plane

Envoy - Reverse Proxy Sidecar managed by Traffic management rules

Traffic Management

Virtual Services - Configure how requests are routed to service in mesh. Timeout, retry, and fault injection
Destination Rules - Policies on routing, load balancing, TLS, circuit breaker, etc
Gateways - Handle Ingress/Egress traffic to/from mesh
Service entries - Register services to be added and discovered in Mesh
Sidecars - Add additional proxy rules on in/e-gress

Security

Peer Authentication (Mutual TLS) - Client and Server Envoy sidecar establish two TLS tunnel
Request Authentication (JWT) - Envoy validate JWT with Public JSON Web Key Set (JWKS)
Authorization Policies - Envoy engine handle Custom, Deny, Allow policies on traffic rules

Observability

Metrics
- Proxy - In/E-gress Traffic: volume, errors, response times
- Service - Latency, errors, saturation
- Control plane - Service regulations, Certs
Distributed Traces - Sample traffic for monitoring/debugging
Access Logs - Envoy access logs

Earn Max Leetcode Coins

aakhtar3 — Sun, 02 Jul 2023 05:12:58 +0000

Goal	Tasks	Tips
T-shirt	13	8,435 coins

Hate it or love it, but leetcode.com has become the de facto learning tool to prepare for software engineering coding interviews. Other skills are required to do the job, but this interview standard has its benefits by providing candidates a fair chance at solving a timed problem in a stressful environment.

Investing time to master and retain data structures and algorithms will net you a great ROI by helping you become a top earner. In addition, having a deep understanding of the material will make you a subject matter expert and a top performer.

Goal

Leetcode has gamified this egregious grind by providing a store, where you can redeem your earned coins for prizes. The prizes are nothing spectacular, but earning the t-shirt is an exciting challenge since 6,000 (t-shirt) or 7,800 (t-shirt, keychain, and coaster) coins will require a lot of time, discipline, and commitment.

Tasks

Below is a breakdown to earn the maximum amount of coins to get the t-shirt in less than a year.

Type	Tasks	Per Year
Social Media	4	40 (One time)
Daily	2	4,015
Weekly	2	2,080
Bi-Weekly	2	1,040
Monthly	3	1,260
Total	13	8,435

8,435 coins from doing all 13 tasks.

Social Media

Easy coins if you want to share your social media on your profile.

Task	Coins	Per Year
Linkedin	10	10 (One time)
Google	10	10 (One time)
Github	10	10 (One time)
Facebook	10	10 (One time)

40 coins from connecting social media accounts.

Daily

View the daily challenges from the problem page.

Task	Coins	Per Year
Daily Check-in	1	365
Daily Challenge	10	3,650

4,015 coins from daily check-in and solving daily challenge.

Weekly

View the weekly challenges from the problem page.

Task	Coins	Per Year
Weekly Challenge	35	1,820
Weekly Contest	5	260

2,080 coins from solving the weekly challenge and attending the weekly contests.

Bi Weekly

Task	Coins	Per Year
Bi-Weekly Contest	5	130
Attend Bi/Weekly Contests	35	910

1,040 coins from attending bi-weekly contests and attending both contests.

Monthly

Earn badges for keeping your streak.

Task	Coins	Per Year
30 Day Check-in Streak	30	360
25 Challenges Streak	25	300
Entire Month of Daily Challenges	50	600

1,260 coins from solving all daily challenges and keeping up your monthly streak.

Tips

Here are some tips to stay consistent and retain:

Set up re-occurring calendar blocked time slot (one hour) for daily/weekly challenges and bi/weekly contests.
Refactor the solution with cleaner and descriptive code.
Add Time O(…) | Space O(…) complexities in your solution.
Share your solutions on Leetcode and GitHub.
Buy and use a Time Travel Ticket from the store if you miss a day (Long flights 😡).

Multi-thread with node.js

aakhtar3 — Thu, 15 Jun 2023 04:24:21 +0000

Node.js is a very powerful server-side javascript framework that excels in single-thread asynchronous I/O and is constantly updating with new features.

This guide is meant to be a resource to understand the multi-threading APIs and provide:

High-level computer science concepts
- Synchronous vs Asynchronous
- Concurrency
- Sequential vs Parallel
- Distributed
- Process vs Thread
- Javascript Engine
Code examples of node.js API and performance metrics
- Thread blocking operation
- Split operation into async tasks
- Invoke multi thread operations

Synchronous vs Asynchronous

Synchronous - Operations are expected to start and finish in a time span.
Asynchronous - Operations can be deferred till a later time.

Concurrency

Concurrency - Multiple operations are handled at the same time.

Sequential vs Parallel

Sequential - Multiple operations are done in a linear fashion.
Parallel - Multiple operations are done at the same time.

Distributed

Distributed - Multiple operations are done independently on segmented resources.

Process vs Thread

Process - A program can use single or multiple threads.
Thread - Code execution is invoked and uses less resources than a process.

Javascript Engine

Javascript Engine - Node.js process runs on top of the google chrome v8 engine.
- Heap - Memory management of variables.
- Call Stack - Code running on main thread.
- Thread Pool - Async code pending to be executed.
- Callback Queue - Async code ready to be executed.
- Event Loop - Async code is pushed on to empty call stack.

Thread Blocking Operation

Here is an example of a long running operation that will increment a counter variable from zero to ten billion.

The snippet of code will block all other operations on the thread since it is a synchronous operation.

ex: web browser freezes after clicking a button.
ex: locking a database record when making an update.

Prerequisite

node -v @12+
echo '{ "type": "module" }' > package.json

const blockingOperation = (operations = 10_000_000_000) => {
    let counter = 0;

    while (counter < operations) counter += 1;

    console.log({ counter }); 

    return counter;
};

(() => blockingOperation())();

time node index.js

{ counter: 10000000000 }
node index.js 7.80s user 0.04s system 99% cpu 7.849 total

Executing this program takes around ~8 seconds.
- Running on a single CPU core (single-thread).
- Sync tasks are blocking the thread.

Split Operation into Async Tasks

Building on the example, we introduce Promises to invoke asynchronous operations.

In addition, the large operation will be split based on the number of CPU cores on the machine.

This chunking strategy will shorten the task's size of operations.
tasks = (operations / threads)

Async will allow the program to handle concurrent operations.

ex: web server accepting multiple API requests.
ex: video game accepting multiple clicks on a keyboard.

import { cpus } from 'os';

const getUTCSeconds = () => new Date().getUTCSeconds();

const blockingOperation = (operations) => new Promise((resolve) => {
    let counter = 0;

    while (counter < operations) counter += 1;

    console.log({ counter, utcSecond: getUTCSeconds() });

    resolve(counter);
});

const splitOperationIntoAsyncTasks = async () => {
    const operations = 10_000_000_000;
    const threads = cpus().length;
    const tasks = (operations / threads);

    const blockingOperations = new Array(threads).fill()
        .map(() => blockingOperation(tasks))

    const countOperations = await Promise
        .all(blockingOperations);

    const totalCount = countOperations
        .reduce((sum, counter) => (sum + counter), 0);

    console.log({ operations, threads, tasks, totalCount });

    return totalCount;
};

(async () => splitOperationIntoAsyncTasks())();

time node index.js

{ counter: 1250000000, utcSecond: 21 }
{ counter: 1250000000, utcSecond: 22 }
{ counter: 1250000000, utcSecond: 22 }
{ counter: 1250000000, utcSecond: 23 }
{ counter: 1250000000, utcSecond: 23 }
{ counter: 1250000000, utcSecond: 24 }
{ counter: 1250000000, utcSecond: 24 }
{ counter: 1250000000, utcSecond: 25 }
{
  operations: 10000000000,
  threads: 8,
  tasks: 1250000000,
  totalCount: 10000000000
}
node index.js  4.35s user 0.02s system 99% cpu 4.372 total

Executing this program takes around ~4 seconds.
We see a 50% performance improvement.
- Running on a single CPU core (single-thread).
- Async tasks are processed in sequential.

Invoke Multi Thread Operations

The final step is to introduce the multi-thread API worker threads.

There are two concepts that are at play:

Parent (main thread)
- if (isMainThread) return new Worker(__filename)
Child (worker thread)
- parentPort.postMessage(await blockingOperation())

The parent will create multiple children and wait for the messages to be returned.

import { cpus } from 'os';
import { dirname } from 'path';
import { fileURLToPath } from 'url';
import { isMainThread, Worker, parentPort } from 'node:worker_threads';

const __filename = dirname(fileURLToPath(import.meta.url));

const createWorker = async () => new Promise ((resolve, reject) => 
    new Worker(__filename)
        .on('message', (data) => resolve(data))
        .on('error', (error) => reject(error))
        .on('exit', (code) => { if (code !== 0) reject(new Error(`Thread exit code: ${code}`)); }));

const getUTCSeconds = () => new Date().getUTCSeconds();

const blockingOperation = (operations) => new Promise((resolve) => {
    let counter = 0;

    while (counter < operations) counter += 1;

    console.log({ counter, utcSecond: getUTCSeconds() });

    resolve(counter);
});

const runTaskInParallel = async (operations, threads, tasks) => {
    const blockingOperations = new Array(threads).fill()
        .map(() => createWorker(tasks));

    const countOperations = await Promise
        .all(blockingOperations);

    const totalCount = countOperations
        .reduce((sum, counter) => (sum + counter), 0);

    console.log({ operations, threads, tasks, totalCount });

    return totalCount;
};

const invokeMultiThreadOperations = async () => {
    const operations = 10_000_000_000;
    const threads = cpus().length;
    const tasks = (operations / threads);

    if (isMainThread) return runTaskInParallel(operations, threads, tasks);

    parentPort.postMessage(await blockingOperation(tasks));
};

(async () => invokeMultiThreadOperations())();

time node index.js

{ counter: 1250000000, utcSecond: 59 }
{ counter: 1250000000, utcSecond: 59 }
{ counter: 1250000000, utcSecond: 59 }
{ counter: 1250000000, utcSecond: 59 }
{ counter: 1250000000, utcSecond: 59 }
{ counter: 1250000000, utcSecond: 59 }
{ counter: 1250000000, utcSecond: 59 }
{ counter: 1250000000, utcSecond: 59 }
{
  operations: 10000000000,
  threads: 8,
  tasks: 1250000000,
  totalCount: 10000000000
}
node index.js  6.38s user 0.05s system 711% cpu 0.903 total

Executing this program takes around ~1 second.
We see a 87% performance improvement.
- Running on a multi CPU core (multi-thread).
- Async tasks are processed in parallel.

Build multi-cloud & hybrid networks

aakhtar3 — Mon, 22 May 2023 23:36:01 +0000

Goals

The purpose of this guide is to create four distributed networks and combine them into one highly available logically connected network.

Build with the top three cloud vendors (aws.amazon.com, azure.microsoft.com, cloud.google.com) and one On-Premise (pfsense.org) network
Scale Mesh network topology to allow additional point-to-point connections
Dynamic routing between Autonomous Systems (AS) using Border Gateway Protocol (BGP)
Encrypt network traversal over Virtual Private Network (VPN) tunnels using Internet Protocol Security (IPSec)

AWS	Azure	CGP	PFSense
Network	VNET	VPC	VLAN
Gateway	VWAN	NCC	ISP
DNS	Private Resolver	Cloud DNS	Pihole
S2S	Connection	Peer VPN	IPSec
SSM	Azure vm	GCP vm	Mac

Network

Final mesh network topology architecture

AWS

Network

Create a Virtual Private Cloud Network in AWS


Resource to Create	VPC and more
Name	vpc-aws
IPv4 CIDR block	172.16.11.0/24
Num of AZs	2
Public	0
Private	2
NAT	None
Endpoint	None

Gateway

Identify the IP address of the ISP
Point to Point Identification and traffic passthrough

Customer Gateway


Name	pfsense
BGP ASN	65000
IP address	4.4.4.4

Transit Gateway


Name	tg-aws
Description	tg-aws
ASN	64512

Route Table

Update routes to TGW

DNS

AWS will dedicate a reserved IP address x.x.x.2 for a VPC resolver
Outbound Endpoints will allow you to forward DNS requests for resolvers on other networks
Inbound Endpoints will allow resolvers on other networks to forward requests to AWS

Outbound Endpoint


Endpoint Name	oe-aws
VPC	vpc-aws-vpc
Security Group	Default
Endpoint Type	IPv4
IP Address #1	AZ us-east-1, subnet 1, IPv4
IP Address #2	AZ us-east-2, subnet 2, IPv4
Rule Name	onpremise
Rule Rule Type	Forward
Domain Name	firewall.lan
VPC Rule	vpc-aws-vpc
Target IP #1	10.0.1.2:53
Target IP #2	10.0.4.2:53

Inbound Endpoint


Endpoint Name	ie-aws
VPC	vpc-aws-vpc
Security Group	Default
Endpoint Type	IPv4
IP Address #1	AZ us-east-1, subnet 1, IPv4
IP Address #2	AZ us-east-2, subnet 2, IPv4

Site to Site

Use IPsec tunnels to connect AWS to another datacenter
Have a failover connection for High availability


AWS	s2s-aws-pfsense
Target gateway type	Transit Gateway
Transit Gateway	TGW
Customer Gateway	CGW
Routing Options	Dynamic
Tunnel inside IP	IPv4
Inside IPv4 CIDR for tunnel 1	169.254.11.0/30
Pre-shared key for tunnel 1	strong password
Inside IPv4 CIDR for tunnel 2	169.254.12.0/30
Pre-shared key for tunnel 2	strong password

Status

When BGP session is established, the status will go from down to up

SSM

Using AWS System Manager will allow remote access without opening any ssh ports
Use to keep your network private
Use to debug any connectivity issues

IAM Role Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ssm:DescribeAssociation",
                "ssm:GetDeployablePatchSnapshotForInstance",
                "ssm:GetDocument",
                "ssm:DescribeDocument",
                "ssm:GetManifest",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:ListAssociations",
                "ssm:ListInstanceAssociations",
                "ssm:PutInventory",
                "ssm:PutComplianceItems",
                "ssm:PutConfigurePackageResult",
                "ssm:UpdateAssociationStatus",
                "ssm:UpdateInstanceAssociationStatus",
                "ssm:UpdateInstanceInformation"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "ssmmessages:CreateControlChannel",
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenControlChannel",
                "ssmmessages:OpenDataChannel"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "ec2messages:AcknowledgeMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:FailMessage",
                "ec2messages:GetEndpoint",
                "ec2messages:GetMessages",
                "ec2messages:SendReply"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Endpoint

SSM Endpoint


Name	ssm-endpoint
Service Category	AWS Service
Service	SSM
VPC	vpc-aws-vpc
Subnets	us-east-1, us-east-2
Security Group	Default
Policy	Full Access

SSMMessage Endpoint


Name	ssmmessage-endpoint
Service Category	AWS Service
Service	SSMmessages
VPC	vpc-aws-vpc
Subnets	us-east-1, us-east-2
Security Group	Default
Policy	Full Access

EC2Message Endpoint


Name	ec2message-endpoint
Service Category	AWS Service
Service	ec2messages
VPC	vpc-aws-vpc
Subnets	us-east-1, us-east-2
Security Group	Default
Policy	Full Access

Azure

VNET

Create a Virtual Network on Azure

Resource Group

Resource Group provides a single detailed view of all resources in a groups stack


Resource group	rg-aws-azure
Region	East US

Virtual Network

This Iaas will build a virtual network similar to a VPC
Create 1 network /24 CIDR to create 4 subnets with /26 CIDR


Resource group	rg-aws-azure
Name	vnet-aws-azure
Region	East US
Bastion	Disabled
Firewall	Disabled
DDoS	Disabled
Adress Space	172.16.12.0/24
Subnet	172.16.12.0/26

VWAN

TODO: Azure VWAN
Have a AWS site-to-site connection config to populate data

Local Network Gateway

IP of the customer/data center Gateway


Resource group	rg-aws-azure
Region	East US
Endpoint	IP Address
IP Address	1.1.1.1
Address Space(s)	None
ASN	64512
BGP	169.254.21.1

Reserved APIPA


AWS	169.254.0.0/16
Azure	169.254.21.0/24 - 169.254.22.0/24

Virtual Network Gateway


Resource group	rg-aws-azure
Region	East US
SKU	VPNGw2AZ
Generation	2
VNET	vnet-aws-azure
Subnet	172.16.12.64/27
Gateway Type	VPN
VPN Type	Route Based
Active-active	Disabled
BGB	Enabled
ASN	65000
Custom APIPA	169.254.21.2, 169.254.22.2
Public IP adress	vng-aws-azure-pip

Private Resolver

TODO: Azure DNS

Connection

Use to create an IPsec connection using BGP
Create a second connection for failover


Resource group	rg-aws-azure
Region	East US
Connection type	Site-to-site(IPsec)
Connection name	conn-1-aws-azure
Virtual Network Gateway	vng-aws-azure
Local Network Gateway	lng-aws-azure
IKE Protocol	IKEv2
IpSec / IKE policy	Default
Use Policy based traffic selector	Disable
DPD timeout	45
Connection Mode	Default
BGP	169.254.21.2

BGP

Verify Connection is enabled
Create second connection for failover

Verify Route propagation from BGP

Azure VM

Azure -> AWS

GCP

VPC

Create a Virtual Private Cloud on Google


Name	vpc-gcp-aws
Description	VPC
IPv6	Disabled
Subnet	Custom
Subnet Name	Private
Subnet Region	us-east-1
IP stack	IPv4
IP range	172.16.13.0/24
Private Google Access	off
Flow Logs	off
IPv4 Firewall Rule	Ingress Apply to all 0.0.0.0/0 ICMP Allow
Dynamic Routing	Regional

Network Connectivity Center

TODO: GCP Network Connectivity
Have a AWS site-to-site connection config to populate data

Cloud Router


Name	cr-gcp
Description	route
Network	vpc-gcp-aws
Region	us-east-1
ASN	65000
Interval	20
Routes	Advertise all subnets to CR

VPN Gateway


Name	vpn-gcp-aws
Network	vpc-gcp-aws
Region	us-east-1
IP stack	IPv4

Cloud DNS

TODO: GCP DNS

Peer VPN

Set up the infrastructure for GCP VPN

Repeats these steps on interface 1 (failover)


Name	vpng-gcp-aws
Interfaces	two interfaces
Interface 0	3.3.3.3
Interface 1	3.3.3.2
Peer VPN Gateway	On-Prem or Non Google
Peer VPNG Name	vpng-gcp-aws
High Availability	Create a pair of VPN tunnles
Cloud Router	cr-gcp
Associated Peer VPNG interface	0: 1.1.1.1
Name	conn1-gcp-aws
pre-shared key	strong password
Peer ASN	64512

BGP


Name	conn1
Peer ASN	64512
BGB IPv4 address	Manually
Cloud Router BGP	169.254.250.138
BGP Peer Address	169.254.250.137

Verify Dynamic Route update

GCP vm

GCP -> AWS

PFSense

VLAN

Check out this write-up on how to configure VLANs with pfsense

ISP

TODO: Check out this write-up on how to configure a VPN Server with pfsense

PiHole

TODO: Check out this write-up on how to configure a DNS server with PiHole

IPSec

Phase 1

Start by creating a primary tunnel and repeat the below steps for the failover connection tunnel 2


Description	conn1-aws-pfsense
Key Exchange version	IKEv2
Remote Gateway	1.1.1.1
Pre-Shared Key	strong password key token
Algorithm	AES
Key Length	128 bits
Hash	SHA256
DH Group	14 (2048 bit)
Max failures	3

Phase 2

Start by creating a primary tunnel and repeat the below steps for the failover connection tunnel 2


Description	conn1-aws-pfsense
Mode	Routed (VTI)
Local Network	address: 169.254.11.12
Remote Network	address: 169.254.11.11
Encryption Algorithm	AES256-CGM 128bits
Ping Host	172.16.11.11
Keep Alive	Enabled

Status

Both primary and failover tunnels connected with IPSec

BGP

FRR Global Settings


Enabled	true
Master Password	strong password

FRR Route Maps


Name	Allow-all
Action	Permit
Sequence	100

FRR BGB


Enabled	true
Local AS	65000
Router ID	10.0.1.1
Networks to distrbute	10.0.1.0/28, 10.0.2.0/29, 10.0.4.0/28

FRR Neighbors

Start with the primary tunnel and repeat the steps for the failover tunnel


Name/Address	169.254.11.12
Description	conn1-aws-pfsense
Remote AS	64512
Inbound Route Map Filters	Allow-all
Outbound Route Map Filters	Allow-all

FRR Status

Verify Dynamic Routes have been updated

View the BGP Summary

Mac

Mac -> AWS

Stack Cloud/Network/Security Certifications

aakhtar3 — Thu, 20 Apr 2023 22:15:41 +0000

I recently went on a certification binge to get 12 certs in 5 months (Nov/4/23 - Apr/20/22). Below are some strategies to stack multiple certifications to stack your resume and LinkedIn profile.

Scrum	AWS	CompTIA
resources	resources	resources

Scrum

ScrumMaster

Foundation -> Advanced -> Professional -> Elevated

Developer

Foundation -> Advanced -> Professional

Product Owner

Foundation -> Advanced -> Professional

My journey started back in 2016 when I first got my Certified ScrumMaster certification through scrumalliance.org. The certification needs to be renewed before the two-year renewal date. Typically, I would obtain the scrum education units (SEU) and pay the renewal fee. However, in late 2020, I wanted to pick up a new Scrum Developer certification. I plan on getting Product Owner and then moving onto the Advanced level where the certs start to stack on renewal.

Scrum Resources

AWS

DevOps

Cloud Practitioner -> Developer Associate -> DevOps Engineer Professional

Cloud Practitioner -> SysOps Administrator Associate -> DevOps Engineer Professional

Solution Architect

Cloud Practitioner -> Solutions Architect Associate -> Solutions Architect Professional

Speciality

Speciality

I skipped the Cloud Practitioner and plan on taking the Speciality certs at some point. I had already taken the three Associate and two Professional certifications through aws.amazon.com in early 2020, but they needed to be renewed before the three-year renewal date. Luckily, these certifications stack, so I only needed to take two Professional exams to receive all five certifications in late 2022.

AWS Resources

Course	Time	Practice Exam
Developer Associate	33.5 Hours	DVA
Sysops Administrator Associate	26.5 Hours	SOA
Devops Engineer Professional	41 Hours	DOP
Solutions Architect Associate	26.5 Hours	SAA
Solutions Architect Professional	70 Hours	SAP

CompTIA

Comptia.org was new to me but they offer a few tracks to allow stackable certifications. I skipped A+ and Server+ but instead went straight for Infrastructure and Cyber Security certifications.

Infrastructure

A+ -> Server+

A+ -> Network+ -> Security + -> Linux+ -> Cloud+

Cloud+ can be renewed by either retaking the test before the three years renewal date or obtaining 50 Continued Education Units (CEU). The cloud renew list has the AWS certifications that I already obtained. Going forward I will only need to take an AWS exam to recertify the CompTIA infrastructure certifications.

Cyber Security

A+ -> Network+ -> Security+ -> CySa+ -> CASP+

A+ -> Network+ -> Security+ -> PenTest+ -> CASP+

CASP+ can be renewed by either retaking the test before the three years renewal date or obtaining 75 Continued Education Units (CEU). I plan on taking college courses, working, and writing blogs to renew this certification. Down the road, I will start taking other non-CompTIA certs on the CASP renew list.

CompTIA Resources

Course	Time	Practice Exam
Network+	26 Hours	n10
Linux+	21 Hours	xk0
Cloud+	10.5 Hours	cv0
Security+	21.5 Hours	sy0
CySa+	32.5 Hours	cy0
PentTest+	32.5 Hours	pt0
CASP+	31.5 Hours	cas

System Design

aakhtar3 — Sun, 26 Mar 2023 02:50:04 +0000

A high level break down of key components when doing a system design problem.

Requirements	Estimations	Network	Application	Database
Functional	Calculations	WAN	Load Balancer	SQL
Non-Functional	QPS	Payload	Service Discovery	NoSQL
Architecture	Max QPS	Gateway	Consensus	CAP PACELC
Key Performance Indicators	Storage Bytes	VPC	Compute	Partition
-	Retention Bytes	-	Message Queue	Index
-	In E Gress Bytes	-	Cache	ACID

Requirements

Get requirements and estimations to scope your system design.

Functional

Identity what you are trying to build.

Features	System
1 on 1 chat	Mobile/Web
News feed	Notifications

Non Functional

Identify how your system will behave.

User Experience	System
Full end to end encryption	High Availability
See new post fast	Reliability

Architecture

High Availability
Failover

Reliability
RPO/RTO

Key Performance Indicators

Observability
Metrics

Estimations

Value	Number	Byte
Hundred	-	B
Thousand	K	KB
Million	M	MB
Billion	B	GB
Trillion	T	TB
Quadrillion	Q	PB
Quintillion	Qu	EB
Sextillion	S	ZB
Septillion	Se	YB

Estimate the scope of your users and data points.

Users (DAU)	File Type (File Bytes)	Storage (Retention Time)
100 B Message/day	1 KB (avg)	10 years
10 B Post/day	10 KB (max)	10% new data per year

Calculations

Using your estimations do some back of envelope.

QPS	Max QPS	Storage-Bytes	Retention Bytes	In/E Gress Bytes
DAU / ~100K	2 * QPS	DAU * FileBytes	StorageBytes * Retention Time	(QPS * FileBytes) / 8 Bit

QPS

Queries Per Second
Daily Active Users
Time = Round up (86,400 seconds in day)

Max QPS

Double Queries Per Second
Calculate (Read % : Write %) Ratio
- 1 Read : 10 Write
- Read % = 1 * Queries Per Second
- Write % = 10 * Queries Per Second

Storage Bytes

Storage Bytes
Daily Active Users
File Bytes

Retention Bytes

Retention Bytes
Storage Bytes
Retention Time
- 365 days in year

In E Gress Bytes

In-gress
E-gress
1 Byte / 8 Bits
Queries Per Second
File Bytes

Network

Wide Area Network

Clients
- Mobile
- Browser
- CLI
DNS
- DNSSec (Signed Registries)
- DNS over HTTPS (Privacy)
CDN
APN

Payload

API
Request Lines
Authentication/Authorization
Protocols

Public Gateway

Firewall
Rate Limiter
Authentication/Authorization
Reverse Proxy
Forward Proxy

Virtual Private Cloud

Access Control List
Security Group
Route Table
Classless Inter-Domain Routing
Gateway
- Internet
- Network Adress Translation
- VPC Peering

Application

Load Balancer

Layer 3/4/7 OSI
Strategy
Use cases

Service Discovery

Auto Scale
- DevOps
Protocols
Zeekeeper/etcd

Consensus

Raft Consensus
Replicated State Machine

Compute

Session(less)
State(full)/(less)
Server(less)

Message Queue

Producer
Consumer

Cache

Eviction Strategy

Database

SQL

Relational Data
Transactions

NoSQL

Key Value
Document
Wide Column
Graph
TimeSeries
Spatial

CAP PACELC

CAP
- Consistency
- Availability
- Partition
PACELC
- Partition == true
  - IF
- Availability
  - or
- Consistency
  - ELSE
- Latency
  - or
- Consistency

Partition

Consistent Hashing
Range Partition
Vertical Partition

Index

B Tree
Skip List
Hash
Inverted

ACID

Atomicity

Transactions
Rollbacks

Consistency

SQL Consistency

Schema
Constraint
Fan Trap

NoSQL Consistency

Eventual Consistency
Sequential Consistency
Strong Consistency

Strong Eventual Consistency
Quorum Replication
(Dotted) Vector Clocks
Chain Replication

Isolation

Race Conditions
Lock
Concurrency

Durability

High Watermark
Checkpoint
Write Ahead Log

Build Networks with VLANs

aakhtar3 — Mon, 30 Jan 2023 02:21:31 +0000

Why	What	How
1. Infrastructure	2. Administrator	3. Game
4. Laboratory	5. Internet of Things	6. Network Attachment Storage
7. Dematerialized Zone	8. Work	9. Guest

Why

Having all devices on the same network introduces risk and network congestion. Physically separating devices onto subnets and then bridging them together is complex. Virtual Local Area Networks (VLANs) provide a way to keep all devices on the same physical network, but will logically isolate traffic.

You wouldn’t let strangers take a squat in your home so why should you let rouge devices on your network? Properly identifying which device is who and what are they doing is important to reduce risks. Each device can be identified by it’s Media Access Control (MAC) address and be assigned a static Internet Protocol (IP) address and hostname. You can easily spoof these, but this is a layer in the defense in depth strategy.

Your firewall will identify VLANs through interfaces and can be used to quickly filter and debug network issues. Grouping devices together into an interface will reduce broadcasts and improve network traffic by splitting into separate lanes 802.1P. Each interface will have its own firewall rules on ingress and egress traffic.

There are many risks introduced when using a single Local Area Network (LAN), but here are a few examples.

Attack	Risk
Traffic Sniffing	Unencrypted traffic can be read
On Path	Man in the middle can be passive or malicious
DNS/ARP Poisoning	Corrupted data on important network protocols
Internet of Things (IOT)	Devices that have weak security or rouge services

What

802.1Q is known as VLAN tagging. Which adds a VLAN ID tag on the data frame. When moving across the network, this frame is tagged and untagged.

Be aware of VLAN hopping attacks, this can be achieved by misconfigured VLAN tables and adding additional VLAN tags on the data frame, but the data can not return in this situation.

networkacademy.io

How

The goal of this guide is to learn about different types of networks by building the example network architecture using open source software.

When following this guide, you should work on ONE network at a time. Be patient, patch to latest version of software/firmware, use different web browsers, use commands (ping, ssh, netstat, route), read logs, and restart devices as a last resort.

Physical and Logical Network Diagram

14 Networks
- 1 WAN
- 1 LAN
- 8 VLANs
- 4 PANs
4.5 Wireless Frequencies
- 5.0 GHz Wifi
- 2.4 GHz Wifi + ZigBee Channel
- 2.45 GHz Bluetooth
- 300 GHz Infrared

INFRA

This is your most critical network because it will logically separate your network. You should put physical safeguards on these devices. You can achieve this by storing in a secure location, disabling unused port, and applying MAC address allow lists.

ADMIN

This network should be limited to an administrator who will build, monitor, debug, test, and fix the network.

The admin's access should still be limited to a few ports. You can do most task over Ping, HTTP(S), and SSH (SSH Tunneling).

Bluetooth devices such as wireless headphones will create a Personal Area Network (PAN).

GAME

You want to take advantage of prioritizing networks by moving this network to a lower priority using quality of service 802.1P.

Bluetooth controllers are on PAN with the consoles. Infrared remotes also create a PAN with the receiver.

LAB

This network is physically separated over a distance between four switches. You should have a laboratory (dev) network to test on before making changes on other networks.

IOT

IOT devices should be on 2.4 GHz, since they do not need high data caps and to lower broadcast interference. In addition, these devices can have poor security and can be exploited.

2.4 GHz
metageek.com

ZigBee is using the 2.4 GHz frequency, but has multiple channels that overlap Wifi channels. This frequency should be placed on a channel that won't get overpowered from wifi devices.

ZigBee
metageek.com

NAS

Data is your Gold (Network Attached Storage (NAS)), so it need to be protected. This is a private network, meaning it has no access to any network, including the internet. Only a few devices should have access to the data ports. To perform updates on the system and software, you can forward proxy the requests through a NAT using filtered allow list of endpoints.

DMZ

There are many ways to connect to a dematerialized zone (DMZ) a.k.a screened subnet and one wrong misconfiguration can be disastrous.

This type of network is open to the WAN (Internet). This can be used when you want to host a website, file server, or VPN concentrator.

WORK

Sometimes you need to work from home and you should avoid mixing network traffic with your personal network. Your work network should be connected to your works network through a VPN.

GUEST

Guests will need access to your internet, so having a wifi only connection will serve this purpose.

You can send all egress WAN (Internet) traffic through a VPN. So that you can provide them privacy and prevent your IP address from being associated with their network activity.

Installation

firewall	dns	siem	wap	TL-1	TL-2	TL-3
10.0.1.1	10.0.1.2	10.0.1.3	10.0.1.4	10.0.1.11	10.0.1.12	10.0.1.13

firewall

	10.0.1.1
1. Specs	2. Config	3. VLANs
4. Interfaces	5. DHCP	6. Aliases
7. Rules	8. Log	9. Diagnostics

firewall Specs

Software: pfSense				Hardware: Protectli Vault
Quad Core	2 Switch Port	AES-NI	8GB RAM	120GB mSATA SSD

There are many hardware alternatives
You can also run pfSense on a virtual environment

General Config

You should already have pfSense installed and configured on your Local Area Network (LAN) and connected to a Wide Area Network (WAN).

Click General Setup

Add your local DNS

Save

VLANs

Add your VLANs with a priority and ID.

Click Interfaces/Assignments

Add VLANs

Add Interfaces

Interfaces

You will do this for each interface.

Edit Interface

Save

Apply

Dynamic Host Configuration Protocol

You will need to do this for each interface. This protocol will assign IP address for each VLAN.

Click Services/DHCP Server

Enable Dynamic Host Configuration Protocol (DHCP)

Add DNS

Use MAC address to assign static IP and hostname

Save

Aliases

These are used to point to more one or more devices on the the network. These can be edited and applied across all firewall rules.

Click Firewall/Aliases

Add IPs

Add Ports

Save

Firewall Rules

Rules are stateful (Ingress traffic will be able to return egress traffic with out any rules), read from top to bottom, and have a implicit deny.

Click Firewall/Rules

WAN Rules

Wide Area Network (WAN) is used to connect to remote networks over your ISP or VPN connection.

Block Everything by using the implicit deny

Floating Rules

These can be applied across multiple interfaces that share the same logic. The alternative is to add ingress and egress rules for all VLANs, which is prone to errors.

The LAN internal traffic will allow the ADMIN_HOSTS to perform debugging tasks on the LAN by using these protocols (PING, SSH, HTTP, HTTPS). In addition, all LANS on the network will have access to these protocols (DNS, NTP, SYSLOG), which are on the INFRA LAN.

The LAN Egress traffic is set to WAN only. The NAS interface does not have this rule because the data on this network needs to be tightly locked down.

Add Floating Rules

Infra Rules

Anti-Lock is added by default to prevent from being locked out. The SIEM (Security Information Event Management) host will be the only host on the entire network to connect to the NAS network Data Ports.

Add rule with aliases

NAS Rules

Add Rules

Route Table

You can run this command on the firewall netstat -nWr to see the route table on the firewall.

Save

Firewall Logs

For short term log viewing, you can view from status

Click Status/System Logs

Add Filter

For long term log viewing, you can send to a SIEM.

Click Status/System Logs

Enable

Add SIEM
Save

Diagnostics

Use these services to help troubleshoot.

Click Diagnostics

Type Host IP to see active connections

Filter on interface

Restart if all else fails

dns

10.0.1.2
1. Specs
2. Software

DNS Specs

Software: dietpi				Hardware: raspberry Pi 3B+
Quad Core	1 ETH Port	4 USB	1GB RAM	32GB Micro SD

DNS Software

You can either use a remote DNS servers, but your network will need to make a round trip through the WAN. Local DNS can cache, provide insights on your requests, and filter out request with DNS sink.

pi-hole

Adguard

siem

10.0.1.3
1. Specs
2. Software

SIEM Specs

Software: macOS				Hardware: Mac Mini M1
8 Core	1 ETH Port	4 USB	16 GB RAM	256 SSD

Siem Software

This device will collect logs and metrics from the network and use this data to trigger events. There's a lot on this topic, so I do a separate write up and update at a later point.

GrayLog

Using docker compose you can run a local installation of the application.

version: '3'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongo:
    image: mongo:5.0.13
    networks:
      - graylog
  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
    environment:
    - http.host=0.0.0.0
    - transport.host=localhost
    - network.host=0.0.0.0
    - "ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true -Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
         memory: 1g
    networks:
      - graylog
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:5.0
    environment:
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
    entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
    networks:
      - graylog
    restart: always
    depends_on:
      - mongo
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 514:514
      # Syslog UDP
      - 514:514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp

networks:
  graylog:
    driver: bridge

Add input

Syslog

Port 514

Grafana

Use Graylog data to visualize on dashboard

Wireshark

Use this for packet capturing on a particular interface.

wap

	10.0.1.4
1. Specs	2. Administration	3. Config
4. Services	5. Wireless	6. Security
7. VLAN	8. Networking	9. Debug

Make sure you are on the latest firmware to enable this feature. Use Firefox or chrome.

There are 2 different frequencies to understand.

5.0 GHZ

Provide higher speeds, has multiple channels to use, and radio waves dissipate in shorter distances.

2.4 GHZ

Can provide reasonable speeds for IOT devices, has 3 usable channels in the USA, and radio waves can travel further distances.

wap Specs

Software: dd-wrt				Hardware: linksys wrt3200acm
Dual Core	4 Switch Port	2 USB	802.11a/b/g/n/ac	4 Dual-Band Antennas

There are many hardware alternatives
The WAN port will be untagging the INFRA network to access admin console
The LAN port will be carrying the tagged traffic
Once VLANS are set up, you can remove WAN port and plug into LAN

Administration

Click Administration/Management

Enable Protocols

Save

Basic Config

Click Setup/Basic Setup

Disable WAN

Assign IP information
Disable DHCP

Save

wap Services

Click Services/Services

Disable Services

Enable SSH

Enable Syslog

Save

wap Wireless

Your wireless connection might be different.

Click Wireless/Basic Setting

Add 5.0 GHZ Virtual Access Points

Add 2.4 GHZ Virtual Access Points

Disable Mixed Connection

Save

wap Security

Click Wireless/Wireless Security

Set Mode

Save

wap VLAN

You will need to use 2 ports on wap (WAN and LAN) when configuring your VLANs. You can use one, but you will be unplug and plug between these interfaces. Once configuration is complete you will only need to connect LAN port, but you will lose access to the router console.

Click Setup/Switch Config

Configure your VLAN Table

Save

wap Networking

Click Setup/Networking

Add Bridges and map to your VLANs

Save

wap Debug

Use the SIEM to read SYSlogs and Wireshark to analyze traffic

Reboot if all fails

TPLink Switch

These switch help expand the network by carrying VLAN IDs. You will enable VLANS and set PVIDs through the admin console.

802.1Q

Enable

Use when a port needs to be untagged and allow DHCP to assign IPs for that VLAN

TL-1

10.0.1.11

VLAN Table

PVID

TL-2

10.0.1.12

VLAN Table

PVID

TL-3

10.0.1.13

VLAN Table

PVID

Fast Ethernet -> Gigabit Ethernet

aakhtar3 — Fri, 30 Dec 2022 23:10:18 +0000


1. Identify the Problem	2. Develop a Theory	3. Test the Theory
4. Plan of Action	5. Implement the Solution	6. Verify System Functionality
	7. Document the Issue

I wanted to hard wire all of my devices on my Local Area Network using Ethernet. I noticed one of my connections was much slower than expected and wanted to fix it. This write up goes over the 7 steps that a network administrator should use to troubleshoot network issues.

Terms	Definition
Fast Ethernet	100 Megabits per second
Gigabit Ethernet	1000 Megabits per second
Half Duplex	Transmit data in one direction at a time
Full Duplex	Transmit data in two directions at the same time
ISP	Internet Service Provider
VPN	Virtual Private Network
WAN	Wide Area Network
LAN	Local Area Network

1. Identify the Problem

TP-Link 5 Port Gigabit Switch identifies port as 100 Mbps Full duplex

fast.com test shows 92 Mbps

2. Develop a Theory

Cat5e cable is damaged or misconfigured

3. Test the Theory

Use another wire and connection

4. Plan of Action

Fix Cat5e Ethernet with a Network Tool Kit

Update Male Cat5e Ethernet Jack

Update Female Ethernet Keystone Jack

5. Implement the Solution

Update Cat5e Ethernet Jack

Use crimping tool

Cut Male Cat5e Ethernet jack

Expose Wires

Inspect wiring arrangement

Use wire stripping knife

Cut covering

Remove covering

Separate wire pairing

Insert wires into RJ45

Tighten wires with crimping tool RJ45 slot

Update Ethernet Keystone Jack

Use Punchdown tool

Push down into Female Ethernet Keystone Jack and trim exposed wires

Use B layout

Use B layout

Install on wall plate

6. Verify System Functionality

TP-Link 5 Port Gigabit Switch identifies port as 1000 Mbps Full duplex

fast.com test shows 220 Mbps
In theory, I should get higher speed if I purchase more bandwidth from my ISP

7. Document the Issue

This write up is the documentation and is intended for sharing on the public internet

Kali Linux on iPad

aakhtar3 — Wed, 21 Oct 2020 04:15:20 +0000

This guide will go over how to run a virtual instance of Kali Linux on your iPad using a two iPad applications and provisioning a server on Digital Ocean


1. Blink	2. Digital Ocean	3. Mosh
4. Kali	5. VNC	6. JumpDesktop

Cost

	SSH	VNC	Kali
Port	22	5900	22
Site	Blink	JumpDesktop	Digital Ocean
Cost	$19.99	$14.99	-
Cost Per Month	-	-	$15.00

Blink SSH Key

# Use blink to create a SSH key
ssh-keygen -t rsa -b 4096 -C “your@email.com”

# Copy public key
cat ~/.ssh/id_rsa.pub | pbcopy

# Copy private key
cat ~/.ssh/id_rsa | pbcopy

Digital Ocean Server

Choose Debian 9 for the OS

Choose Basic plan $15.00/mo so that you can properly install the OS without errors

Choose Datacenter

Add SSH key from the previous blink step

Get IPv4 address

Mosh

mosh is like ssh, but allows intermittent connections

Host

Click on Hosts

Enter the hostname values and use the SSH key that you created in the previous steps

Install

# Log into your host with ssh2
ssh2 kali

# Update and install mosh
apt-get update -y & apt-get install -y mosh

# Log out of your host
exit

# Log into your host with mosh going forward
mosh kali

Kali

Setup Source

# Update source.lists by adding kali reference
echo "deb http://http.kali.org/kali kali-rolling main contrib non-free" >> /etc/apt/sources.list

# Update
apt-get update -y

You will get this error if you do not have permission

W: GPG error: http://kali.mirror.garr.it/mirrors/kali kali-rolling InRelease: The following signatures couldn't be verified because the public key is not available:
  NO_PUBKEY ED444FF07D80BF6
E: The repository 'http://kali.mirror.garr.it/mirrors/kali kali-rolling InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

# Fix Error
apt-get -y --allow-unauthenticated install kali-archive-keyring

# Re-run update
apt-get update -y

# Search for Kali installation
apt-cache search kali-linux

Install

# Install everything, which will be larger than 25GB
apt-get -y install kali-linux-everything

You will be prompted a few questions during the installation

Select Language

Install Samba

Enable Packet Capture

Enable Mac Changer

Kismet is a resource hog, but can be turned off with
systemctl stop kismet

Add user

Standalone

Restart services after installation

Restart services

Use sshd config

Validate

Cleanup

# Finalize/cleanup Linux Kernel
apt-get -y update
apt-get -y upgrade
apt-get -y dist-upgrade
apt-get -y autoremove

# Restart
shutdown now -rf

Check Kernel

uname -r

VNC

Enable SSH Port Local Forwarding

VNC is very insecure because the traffic is sent over an insecure connection
- Unencrypted connection that can be sniffed
SSH Port Local Forwarding creates an encrypted session over port 22
- Reverse proxy the connection to localhost:5901

# Enable TCP forwarding
nano /etc/ssh/sshd_config

############ START OF FILE ############
AllowTcpForwarding yes
############ END OF FILE ############

# Restart SSH
systemctl restart sshd

# Check status
systemctl status sshd

Install

This startup script will enable a VNC server that listens on localhost for SSH Port Local Forwarding
The password will be used to login through VNC

# Install tight vnc server
apt-get install -y tightvncserver

# Create start up script
nano /etc/init.d/tightvncserver

############ START OF FILE ############
#!/bin/sh

USER=root
HOME=/root
export USER HOME

case "$1" in
 start)
   echo "Starting VNC Server"
   /usr/bin/vncserver :1 -geometry 1280x1024 -depth 16 -localhost -nolisten tcp
   ;;

 stop)
   echo "Stopping VNC Server"
   /usr/bin/vncserver -kill :1
   ;;

 *)
   echo "Usage: /etc/init.d/vncboot {start|stop}"
   exit 1
   ;;
esac

exit 0
############ END OF FILE ############

# Update the permission
chmod +x /etc/init.d/tightvncserver

# Update defaults
update-rc.d tightvncserver defaults

# Configure password
tightvncserver

# Restart
shutdown now -rf

Validate

# Verify that VNC is listening on localhost or 127.0.0.1 and port 5901
netstat -tupln | grep vnc

JumpDesktop

Set the hostname as localhost or 127.0.0.1 and the port as 5901

Edit

Add SSH Tunneling

Add sever by adding the IPv4 address from Digital Ocean

You will need to either copy and paste or load iTunes and store the key in the jump desktop app

Add the private key

Update the icon and title

Enable SSH

Enter VNC password that you created in the previous step

Interact with GUI

Dynamic and Abstract Nginx

aakhtar3 — Thu, 01 Oct 2020 00:53:12 +0000

Dynamic and Abstract Nginx

Nginx is a powerful webserver and can built dynamically by injecting environment variables with envsubst and importing modular configs with the include directive.

envsubst

Create /etc/nginx/nginx.conf.template with environment variables ${VAR}
Use envsubst to dynamically generate a new /etc/nginx/nginx.conf

# Define the environment variables
export SERVER='example' LOG='example'

# Run command to generate new nginx.conf
cd /etc/nginx
envsubst '${SERVER},${LOG}' < nginx.conf.template > nginx.conf

brew install

Mac users will need to install gettext which includes envsubst.

brew install gettext
brew link --force gettext

Include

From the example above using the sample http config.

* is a wildcard and will match on example.com.conf.

http {
    ...
    include /etc/nginx/conf.d/example.*.conf;
    ...
}

Here is an example folder structure for your Nginx modules.

/etc/nginx/
    nginx.conf                  # HTTP
    /conf.d                     # Server(s)
        example.com.conf        # Can contain multiple servers
        default.conf            # Comes with nginx
        *.conf
    /sites-enabled              # Includes
        /location               # Location Includes
            assets.inc          # /location blocks to static assets
            *.inc
        /SSL                    # SSL Includes
            example.com.inc     # Contains SSL directives for example.com
            *.inc
        /*                      # You can continue to modularize your code base
            *.inc

Check out this repo which has more include examples.