Forem: Daniel Caballero

How to: Deploy Angular Website to Github Pages

Daniel Caballero — Mon, 18 Sep 2023 06:00:50 +0000

With the following pages you can host an Angular website in github pages. I did this to host my personal website, which I created using Angular, in github pages:

https://danielcaballero88.github.io/

Step 1: Create GitHub Repository

If you haven't done this yet, create a GitHub repository for your Angular website project, it's a pre-requisite to deploy the website to github pages.
Commit all your changes and push your project files to this repository.

Step 2: Create gh-pages Branch

Create a gh-pages branch from your master or main branch. ```

$ git branch gh-pages
$ git checkout gh-pages


**Step 3: Build Your App**
- Build your Angular app in the `gh-pages` branch.

$ ng build --configuration-production --base-href https://[username].github.io/[repo]/

  Note: Replace `[username]`, `[repo]`, and `[project-name]` with your GitHub username, repository name, and project name respectively. Make sure the project name matches what's in your `angular.json` file under `defaultProject`.
  Note 2: If this is your default github pages repo (`[username].github.io`) then you can skip `[repo]` in the `base-ref` above.

- Push the code and configure the repository to pick up the files from the build path (relative path to your local `/dist` directory in your repo, or whatever configuration you have in `angular.json`).

**Step 4: Visit the App Page**
- Visit your Angular app on GitHub Pages at the URL: `https://UserName.github.io/RepoName/`

Your application should now be live on GitHub Pages.

How to implement a simple scraper with Python

Daniel Caballero — Wed, 21 Dec 2022 06:29:21 +0000

Introduction

Sometimes we need to get some information from a website as soon as possible after it is published, and just manually visiting the website every hour to check if the information is already there is just not a practical way to do it, specially if you are a programmer and have the ability to do it automatically for you.

In my case, I need to know the date when new slots for appointments to complete some paperwork in the Consulate of Spain in Buenos Aires.

They usually provide the date with a very short notice (or none at all) and the available slots for the appointments are
filled sooner than I have time to even realize that the slots were available in the first place.

So, I implemented a very simple but yet effective scraper that visits the website every hour, and parses the
information. If a new date is found for the slots, it will notify me via GMail and Telegram, so I don't miss it.

NOTE: The full code can be found in this Github repo, so in this post I'll only share code snippets
without many very important features that are not essential to understand the key concept, and that can also get in the
way of it on a first read (e.g., file path resolution, error handling, logging, tests, etc). If you want to check out the program with all those features, do check the repo.

How to Implement the Scraper

The website is just HTML with no need for rendering dynamic content or providing credentials of any kind. The objective
is to scrape a table element, looking for the information contained in a single row that we can identify for the content
of the first column.

Therefore, there is no need to use scraping frameworks like Scrapy, which are awesome but, for this particular case,
also a huge overkill. Simply using the Python requests library to get the website response will suffice, and parsing
the information with the also amazing Beautiful Soup will achieve the results beautifully (pun
intended).

To get the response and, specifically, the target row, we just need to use a few lines of code:

# Fetch the response.
req = requests.get(<url>, timeout=10)
soup = BeautifulSoup(req.text, features="html.parser")

# Get the target row from the table.
target_row = soup.find_all(lambda x: x.name == "tr" and "Registro Civil-Nacimientos" in x.text)
target_cells = row.find_all("td")

# Parse information into a dictionary.
result = {
    "servicio": cells[0].text,
    "ultima_apertura": cells[1].text,
    "proxima_apertura": cells[2].text,
    "solicitud": cells[3].a.get("href"),
}

That's it, now we have the information. Most tutorials I've found online for simple scrapers end here, but this is still
completely useless (like those tutorials) unless we actually do something with that data.

Set up Notifications and Deploy to Linux VM in GCP

Add notifications

To make it actually useful in practice, we need to add notifications. You can check my other post on how to do that:

[How to deploy your program to run periodically in a Linux VM in GCP Compute Engine][hot-to-deploy]

We need to implement the following simple logic:


# Code to scrape the website.
...

if result["proxima_apertura"] != "fecha por confirmar":
    # Code to send the email.
    ...

Of course in your case you might need to adapt the code to the expected result that you'll have. And, also, instead of checking for equality or inequality, you might want to use a regex or something more flexible than a simple inequality (but for the purpose of this post, that is beyond the scope).

Deploy to VM

Finally we can deploy it to a VM so it can run there periodically in the cloud. To see how to do that, you can check my other post:

How to set up email and message notifications for a Python application

Once that is done, make sure that the cronjob that you set up to run your scraper is not aggressive on the target website, there is no need to put strain on their servers and force them to take measures against your scraper. Running it every hour or 30 min, in my case, is more than enough, and it's not something that will case the website any stress.

Conclusion

And done! I showed you how to set up a Python scraper to automate the task of checking a website for you, and notify you whenever a desired change happens.

Do check my github repo to see the full code with logging, error handling, etc...

And as usual, feel free to ask any questions in the comments and I'll check them out when I can.

Cheers!

How to deploy your program to run periodically in a Linux VM in GCP Compute Engine

Daniel Caballero — Sun, 18 Dec 2022 05:02:01 +0000

Introduction

Ok, so you have been developing a nice script or program, and it's now working just you we want it to, that's great! But it's also not enough... you need to host it somewhere!

I'll show you here how to deploy your program (be it a script, a scraper, or whatever) to a Linux VM in Google Cloud Platform's Compute Engine service.

If you don't have a Linux VM in GCP, you can follow this other tutorial of mine on how to spin up a free VM in GCP.

If you followed that, you should have a GCP project (I'll refer to it as $PROJECT) and a VM (with a name that I'll reference as $INSTANCE). Of course, you may already have that without having followed my other post, that's just as good.

NOTE: This tutorial assumes that you are also working from a Linux machine (as I do). If you use Windows (please don't) or Mac, the same concepts apply but some commands may differ. Don't worry, there will be links to the official documentation where you can see the exact differences and adapt to your case.

Set up gcloud locally

If you haven't done this yet, it's important to have gcloud installed locally to be able to connect your local machine to all your GCP services (in this case, it will be just the VM).
To do so, just follow the official docs for the gcloud sdk. All the steps are very easy to follow, and all the default configurations are good enough.

To test that it's working properly, let's try to ssh to our VM. To get the gcloud command to do it, go to the Compute Engine service in the [GCP console], see your VM listed and click on the 3 dots and then on "View gcloud command".
This will prompt a dialog that displays the command that we will need to use to connect to the VM.

Copy that command and run it locally in a terminal, you should see something like this:

And that's it! Now we're ready to deploy our code to our VM using the gcloud command.

Deploy the code

I'll assume a simple structure for the code, because I want to focus on the concept of deploying the code and not on the nitty-gritty details of production-ready code (which is incredible important but not essential to explain how to perform a deployment, and will get in the way of the concept I want to explain here).

.
├── src
│  └── ...
├── docs
│   └── ...
└── tests
    └── ...
├── main.py
├── deploy.sh
└── ...

This means that there is a directory src where the code is, some other directories not essential but usually present like docs and tests, an entry-point main.py (this is the file that you execute to run your program, in this case it's a Python file but it could be any other kind), and a deployment script deploy.sh.
We'll focus, naturally, on the deployment script.

Copy the code over to the VM

The first part is to copy the code over to the VM, so it runs there instead of on our local machine.

Let's assume we have our program contained in the directory /path/to/local/app in our local machine, and we want to put it in the directory /path/to/remote/app in the VM.

First we can zip the needed files into a temporary file temp.tar.gz:

$ cd /path/to/loca/app
$ tar czf /tmp/temp.tar.gz src main.py

In this case I'm only taking the src directory and the main.py entry-point, but of course that you can include more directories and/or files if you need it for your program to run properly.

After doing so, we can send that temporary file to the VM (you'll need to have $ZONE, $PROJECT, and $INSTANCE as environment variables or simply replace them in the command):

gcloud compute scp --zone $ZONE --project $PROJECT --recurse /tmp/temp.tar "$INSTANCE:/tmp/"

And finally, extract the contents to the correct directory in the VM:

gcloud compute ssh --zone $ZONE $INSTANCE  --project $PROJECT --command="cd /path/to/remote/app && tar xzf /tmp/temp.tar"

And done! Now the code is in the VM ready to run there.

Set up the VM so the code can run

Ok, so... the code is in the VM ready to be executed, but some configurations will be needed. For instance, if it's a Python program, it will need a Python interpreted of the right version installed in the VM and then the correct requirements installed with pip.

To do so, we need to ssh to our VM, and apply the proper configurations. For example, I'll assume our program uses Python 3.8 and the requirements are saved in a requirements.txt file.

First we connect via SSH:

$ gcloud compute ssh --zone $ZONE $INSTANCE  --project $PROJECT

Now we install Python 3.8 in the VM using apt.

$ sudo apt install python3.8

And install the requirements:

$ python3.8 -m pip install -r requirements.txt

And finally test our program:

$ cd /path/to/remote/app
$ python3.8 main.py

And done! If it runs (and it should if it was running locally) it means that the VM is really ready to execute our program.

Set up a cronjob

Now the program runs whenever we execute it, but that is hardly useful since we are still running it manually as we do in our local machine.
The whole idea is to have it run periodically to do whatever it does hourly, or daily, or every 20 minutes, or whenever you need it to.

For instance, I sometimes need some information from a website as soon as it's published, so I set up a simple scraper and have it run hourly and notify me via email and telegram whenever there is a change.

To achieve that, we'll set up a cronjob, which is quite easy to do. First we SSH to the VM and then we use the following command:

$ crontab -e

This will open up an editor, where we need to add a line into it:

0 * * * * cd /path/to/remote/app && python3.8 main.py

In this case it will run hourly according to the cron expression that we used. If you're not familiar with cron expressions, it's quite simple, and you can understand it quickly here.

Finally, we can check that the new cronjob is correctly set by checking the syslog:

$ tail -f /var/log/syslog

You should see some CRON entries whenever your code is executed. If you're not sure, just change the cron expression so it runs every minute (*/1 * * * *) and inspect the syslog again.

And done!

Now your code is running periodically in a Linux VM in GCP Compute Engine.

Deployment script

After having explained all the steps, it's easy to put them all together into the deployment script (deploy.sh). The only addition is that we remove the current files in the remote directory before copying over the new files:

#!/bin/bash
set -e

# Project base path to reference files.
LOCAL_PATH="/path/to/local/app"

# Delete the current script and requirements files
echo "Deleting remote files."
REMOTE_PATH="/path/to/remote/app"
FILES="src main.py"

$ZONE=...
$INSTANCE=...
$PROJECT=...

gcloud compute ssh --zone $ZONE $INSTANCE  --project $PROJECT \
--command="cd $REMOTE_PATH && rm -rf $FILES"

# Send the current files
echo "Sending new files."
tar czf /tmp/temp.tar.gz $FILES

gcloud compute scp --zone $ZONE --project $PROJECT \
--recurse "/tmp/temp.tar.gz" $INSTANCE:/tmp/"

gcloud compute ssh --zone $ZONE $INSTANCE  --project $PROJECT \
--command="cd $REMOTE_PATH && tar xzf /tmp/temp.tar"

# Install the latest requirements
echo "Installing requirements."
gcloud compute ssh --zone $ZONE $INSTANCE  --project $PROJECT \
--command="cd $REMOTE_PATH && python3.8 -m pip install -r requirements.txt"

And then whenever we want to deploy our code we simply execute our deployment script:

$ cd /path/to/local/app
$ bash deploy.sh

Conclusion

I showed you how to copy over your code from your local machine to a remote Linux VM in GCP Compute Engine, set it up to run the code properly and then set up a cronjob to execute it periodically.

All the steps are quite simple and easy to follow, but if I overlooked anything or you are having trouble with any step, feel free to leave a comment and I'll reply as soon as I can.

Cheers!

How to set up email and message notifications for a Python application

Daniel Caballero — Tue, 13 Dec 2022 10:01:20 +0000

Summary

So, you've prepared a nice Python application that is now running (or almost there), but you still need to manually check the results, or see that it is actually running smoothly... so what happens if you're away and suddenly something stops working as it should? It will continue to be broken until you decide to check on it and fix it, but that's not very practical.

It would be MUCH better to implement some kind of notification system so you're alerted when something happens. It might be something bad like your application being incapable of properly handling an unexpected error, or something good like some heavy program finishing running having the results ready for you to see.

Here we'll go over a simple way to implement a quite good and reliable notification system using Python, GMail, and Telegram.
You might want to ask "why two channels?". Well, a notification system might fail, and since the lack of a notification is not something that will be noted, then it might be completely overlooked and then you might miss an important notification. That would be a big fail.

To avoid that case, it's important to have some redundancy. Of course that it's also possible for both notification systems to fail, but to do so simultaneously is way less likely, so we can feel a lot more secure on our notifications if we have more than one system implemented in parallel. If you're application is of critical importance to you, it's also possible to add more channels in parallel, but here we'll limit ourselves to two.

Also, if on a given occasion we only receive the notification via one of the channels, we know that we need to check on the other one because it might be broken.

NOTE: In this post I'll only share code snippets but I'll ignore very important features that are not essential to understand the key concept, and that can also get in the way of it on a first read (e.g., classes and functions definitions, file path resolution, error handling, logging, tests, etc). By all means, you should try to implement these code snippets together with all these other ingredients in a way that makes sense for your application.

Email notifications with GMail

To send notifications using GMail we need, naturally, a Google account.
I think that the best approach is to create a new account to send these emails, to avoid using one's personal Google account in case there is a security breach and someone gets access to the password app we'll create.
Having a new account created just for these purposes means that even if someone gets access to the password app, there's no much harm that they can do, or non at all.

So let's create a new Google account, and call it notifier.bot@gmail.com (this exact address is probably taken but the actual account name is not important, you can create it with whatever name you prefer).

After going over all the steps, we need to enable 2 step verification first, because it is a pre-requisite that Google has before we are able to create a Password App.
Simply go over the steps and use one of the methods provided for 2 step verification.

Once that is set up, we can now create a App Password.
The steps are quite simple, just a few clicks and we get a new App Password (let's call it <app-password>) that we can use to authenticate to our account using it instead of our regular passowrd.
The obvious benefit is that we can use this App Password in our app and we don't need to include our actual account password, so if by mistake it is filtered and someone gains access to it, we can simply revoke it and we never lose the control of the account.

Now that we have a notifier Google account and we created a Password App for it, the code to create and send an email is quite simple:

import json
import os
import smtplib
from email.message import EmailMessage

from dotenv import load_dotenv
_ = load_dotenv()

# Read email address and app password from environment variables.
gmail_account = os.environ.get("GMAIL_ACCOUNT")
gmail_password = os.environ.get("GMAIL_PASSWORD")

# Create message.
message = EmailMessage()
message["To"] = "danielcaballero88@gmail.com"
message["From"] = "danielcaballero88@gmail.com"
message["Subject"] = subject
message.set_content(content)

# Send email.
with smtplib.SMTP_SSL("smtp.gmail.com", 465) as smtp:
    smtp.login(gmail_account, gmail_password)
    smtp.send_message(message)

Note that the secrets (GMail account and password) are read from environment variables that are previously loaded from a .env file. It's VERY important that these secrets be kept out of the code for security reasons.
If you're not familiar with this, the official documentation of the python-dotenv package is quite easy to follow.

Message notifications using Telegram

To send notifications via Telegram we'll create a new Telegram bot, add it to the Telegram chat where we want to receive the notifications, and implement the code to send the notification message to that channel.

The best approach is to follow the official tutorial for creating and setting up a bot.
Nonetheless, here I'll put a summary of the steps I followed.

Create a Telegram bot

Simply look for the @BotFather in the Telegram messaging app, and ask to create a new bot by sending a /newbot
message to it.
It will prompt for the bot name, we can give something like NotifierBot, but this one will most likely be taken, so
use a unique one (and by all means try to use CamelCase, just for consistency sake).
The Bot Father will then provide a token for the bot (let's refer to it as <token>), which is meant to be kept secret,
just save it for now.

Now you can look for your bot and start interacting with it, but since there is nothing implemented for it, it will be
like talking to a wall (in the search bar look for @NotifierBot, or whatever name you used preceded by a @ sign, and you should see it there).
What we can do, however, is to get the chat id for this chat, or add the bot to another chat if we already have a chat dedicated to notifications, or we can create a new chat for it, whatever you choose is up to you, and it won't affect the next steps.

We do need to get the chat id however, because we want the bot to send messages to that particular chat, so we need to
visit the following url:

https://api.telegram.org/<token>/getUpdates

Just replace <token> with your actual bot token and visit it in your browser.
You'll get a JSON response displayed in the browser window and somewhere in it, something like this:

...
"my_chat_member":{"chat":{"id":<chat_id>,"
...

where <chat_id> is a positive or negative number.

Copy that number and save it too.

Implement the notifications

Now with both the token and the chat_id we can construct the call to send the message that we want, using the
sendMessage method of the Telegram API:

import requests
from dotenv import load_dotenv
_ = load_dotenv()

# Read token and chat_id from environment variables.
telegram_token = os.environ.get("TELEGRAM_TOKEN")
telegram_chat_id = os.environ.get("TELEGRAM_CHAT_ID")

headers = {
    "Content-Type": "application/json",
    "Proxy-Authorization": "Basic base64",
}
data_dict = {
    "chat_id": telegram_chat_id,
    "text": <message>,  # Replace <message> with the intended message.
    "parse_mode": "HTML",
    "disable_notification": True,
}
data = json.dumps(data_dict)
url = f"https://api.telegram.org/bot{telegram_token}/sendMessage"

response = requests.post(
    url, data=data, headers=headers, verify=False, timeout=10
)

Again note that we are reading the secrets from environment variables that are previously loaded from a .env file.

Conclusion

Here I showed how we can set up notifications using GMail and Telegram, which can be incredibly useful for many practical applications when we need to be notified or alerted immediately when something (either good or bad) happens.

Everything is explained and links to the official documentation for each case is also provided.

You should be able to adapt these code snippets to your use case, and also feel free to ask questions and I'll do my best to answer them.

How to spin up a free Linux Virtual Machine on Google Cloud Platform

Daniel Caballero — Tue, 06 Dec 2022 10:59:38 +0000

Summary

Ok, so you have been developing a nice script or application, and it's now working just you we want it to, that's great! But it's also not enough... you need to host it somewhere!

There are many options to host your app, but we'll focus on GCP Compute Engine because it offers a lot of freedom.

An important remark that is always valid is that we don't want to spend money unnecessarily, so keeping costs as low as possible is absolutely desirable, with extra points for keeping costs to actually zero.

Many tutorials (pretty much all the ones that I found so far) propose to use the free trial that GCP offers, but that is a limited amount of money during a limited period of time, so it's only useful for first time users and for short timed projects, so if you've used that up already (I certainly have), it's not really an option for you anymore.

Therefore, in this post I'll describe how to spin up a VM using GCP Compute Engine, and keep it within the free tier so it's completely free. And I don't mean using your free trial credits to pay for the VM, which is virtually free but only for a given time until those credits run out. I mean a VM that will cost nothing to keep it running for as long as we need it to.

NOTE: I'll assume that you're starting from scratch regarding Google Cloud Platform. For instance, I'll include the creation of a new GCP Project as a step, but of course you can just use an already existing project if you have one. Same for other things, but you should be able to adapt to each case only needing a bit of common sense.

Why a Linux VM

Like I said, there are countless options to host your app, way too many to even mention here a meaningful fraction of them, but whenever possible I try to always use a Linux VM.
The reasons for my choice are also many, but I can enumerate some:

It's Linux, and I love Linux. Every other solution or service will be a virtualization on top of a Linux machine, so being able to manage a Linux VM is a great skill to have.
I can host many applications in the same VM, so I don't need to have many services running in GCP or any other cloud, just one VM is good enough for many solutions (until it runs out of CPU, RAM, or disk space, of course).
I can have complete control over the machine, something that's not available with other services with high virtualization like Cloud Functions, App Engine, Cloud Run, etc.
I can still implement different levels of virtualization if needed using Docker, so I don't lose anything in that regard.

Billing

Even when using only the free tier, you need to create a GCP Project and a Billing Account, because that is a simply pre-requisite to enable the Compute Engine (VMs) service in GCP.
On top of that, the Billing Account will require that you add a Credit or Debit card to it, because that is a pre-requisite to create a Billing Account.

No need to worry, however, because everything is prepared to avoid being charged accidentally in case we make a mistake when setting up the VM.

Create a Billing Account

The first step is to create the billing account.
Just go to "Billing" > "Billing Accounts", click on "Add Billing Account" and follow the given steps (it's quite straight forward, but note that here you'll need to add your credit or debit card details):

Again: we'll set up a Billing Account with a credit or debit card because it's a requisite to use some GCP services (like Compute Engine), but we'll remain within the free tier limits so we won't be charged.
And even if we make a mistake and surpass the free tier limits, we'll assign a very low budget to the billing account, and set up alerts so we get notified immediately if we start spending any money.

After creating the Billing Account, you should see it listed as follows:

Let's set up a budget and some alerts to avoid spending money by mistake.
Clicking on the Billing Account takes us to the Billing Account overview:

Then go to "Budgets & alerts", click on "Create Budget" and follow the steps to set up a very low budget (10 USD is low enough in my opinion, but you can choose something different matching your idea of "very low") and some alerts (I use 50%, 90%, and 100%):

This way if during any future month we start spending money by mistake, and reach 5 USD, we'll get a notification via
email and we can go to our GCP Project and change or stop whatever service is spending money.
Besides, we can always just go to our Billing Account overview and manually check if it's spending any money or not.

Now we have to assign this billing account to the project, so go to Billing > Projects, and you'll see your GCP project
there, which will have the billing disabled because so far no Billing Account has been assigned to it:

To assign the billing account to it, click on the 3 dots, and click on Change Billing, and follow the steps:

And that's it!
We can now enable services that require Billing to be active, like Compute Engine.

Spin up a VM

The GCP free tier specifications for Compute Engine might be changed by Google eventually, so by all means do check them
out in the official docs.
At the moment they are like this:

So we can spin up an e2-micro VM with a 30 GB standard persistent disk.

First, we need to enable the Compute Engine service:

And then create a VM by clicking on "Create Instance":

Follow the steps by selecting all the free options according to what is currently free,
and once that is finished the VM Instance will be listed in "Compute Engine" > "VM Instances" (as can be seen in the
picture above, blurred).

And that's it! The VM will be created and listed in the Compute Engine overview.

Conclusion

Here I showed you how to, starting from scratch, create a GCP project, add a Billing Account to it, and spin up a free Linux VM using GCP Compute Engine.
The steps are quite easy and simple to follow, but do take into account that Google is prone to change things here and there as time passes, so if some time has passed between me writing this (December 2022) and you reading it, go and check the official documentation to see that everything is still valid (I left links to GCP official docs here and there throughout the post).
And that's it, the VM is ready to be used!