Forem: Ramzi A.

Bucket Storage on Railway with Go

Ramzi A. — Tue, 14 Mar 2023 03:50:49 +0000

A little about Railway and what we are trying to solve

Railway has changed how I will build any side projects (and future start up!). One of the best things about building software is actually writing the code and solving the problems you intended with it. But a lot of times I feel like I am wrestling configurations for AWS or GCP to get things to work in an automated way. Thats time lost into building your actual application.

Railway has changed that for me. Deploying and adding resources like a database is seamless and serverless. I haven't been this excited about a new product in a long time. Thats why I decided to write this blog post. Currently Railway doesn't have native object storage, so I wanted to share one way to use some sort of object storage for your project. It's the least I can do for their generous free tier.

Pre-requisites & Setup.

You have a google cloud account you can sign up here with a gmail account.Google Cloud
Why Google cloud? Well I thought there is a lot s3 and AWS guides out there to try something different. (If you are interested in AWS guide let me know!).
You have a Railway.app account. Their free tier is really generous and should be sufficient for this guide.
The way I like to use Railway is to connect to an existing project from Github to it. In this case it is the object storage railway project https://github.com/ralaruri/object-storage-railway. Every-time you push a PR you can run a development deployment and when you merge to your main branch you can push to production Add Project
If you want to skip the guide check out the repo here you can deploy to Railway on your personal account. https://github.com/ralaruri/object-storage-railway

Google Cloud setup

We will create a project & service account in order to give our code access in our project to create, write, delete and update buckets along with data.

Create a new Project or use a current project you have in the UI. I created a Temporary project for this guide called object-storage-railway
Go to IAM & Admin Panel and create a service account. Click + CREATE SERVICE ACCOUNT
Name the Service account whatever you like mine is named: bucket-creator-and-writer
Add Permissions to your service account:(Remember to give it the least amount of access needed i.e. reading, writing and creating buckets) In this guide I will keep it simple and give it storage admin access. This allows it to create, update, read, write and delete storage. Go to Permissions

Then add storage admin access.

Create a private JSON key and download it locally. Do not commit this publicly to your github or share this anywhere what I am sharing in this guide is just a test account I have deleted these and are no longer valid!

Setting up the Service Account Locally or on Railway.

Convert your JSON keyfile into a base64 key. using this function in your terminal cat object-storage-railway-f5a8a25dd590.json|base64 it will return the base64 key Again don't expose this publicly this is a demo account for the purpose of this blog post and has already been deleted. Save the output

This allows us to keep the code in our .env locally or inside of Railway's variable storage for the project and decode it as needed at runtime.

base64 decoding in Go part of the bucket_creater.go file.



func CovertStringToJSON(env_details string) []byte {
    decoded_json, err := base64.StdEncoding.DecodeString(env_details)
    if err != nil {
        panic(err)
    }

    return decoded_json

}

For your project you can access variables here we will add the encoded base64 key as a variable in our Railway Project.

Adding Key

Key added

Additionally you will want to create a variable called ENV=PROD this allows us to use Railway as our production env and locally we can use our .env if we choose to.

Understanding the Code:

Establishing a connection to GCP We are using the Google Cloud library in Go to create a bucket depending if you are running locally or on Railway we use either the .env you have local variables or the variables on Railway. This allows us to test locally if we need to.

This client is the underlying method to use any GCP service in our case we are using to create, read and write to a bucket.

in create_bucket.go:



func clientConnection() *storage.Client {
    ctx := context.Background()

    // Set up credentials to authenticate with Google Cloud
    var bucketVar string

    if env := os.Getenv("ENV"); env != "PROD" {
        bucketVar = utils.GoDotEnvVariable("./.env", "BUCKET_CREATOR")
    } else {
        bucketVar = os.Getenv("BUCKET_CREATOR")
    }

    new_creds := CovertStringToJSON(bucketVar)
    creds := option.WithCredentialsJSON(new_creds)

    // Create a client to interact with the Google Cloud Storage API
    client, err := storage.NewClient(ctx, creds)
    if err != nil {
        log.Fatalf("Failed to create client: %v", err)
    }

    return client

}

Creating a Bucket

We can now use the client connection to create a bucket just providing the bucket name and the project we are using.

in create_bucket.go:



func CreateBucket(projectID string, bucketName string) {

    ctx := context.Background()

    client := clientConnection()

    // Create a new bucket in the given project with the given name if the bucket
    // already exists then it will just print (bucket already exists)
    if err := client.Bucket(bucketName).Create(ctx, projectID, nil); err != nil {
        fmt.Printf("Failed to create bucket %q: %v", bucketName, err)
    }

    fmt.Printf("Bucket %q created.\n", bucketName)
}

in main.go:



buckets.CreateBucket("object-storage-railway", "food-bucket-dev")

Reading and Writing Data

We have two operators we care about writing and reading. So the two functions take the name of the bucket and the file path of the data you are writing to the bucket. You can change the path of the file you want to write in the bucket it does not need to match the file path you have when reading the file.

in bucket_operator.go:




func WriteToBucket(bucket_name string, file_path string) {

    ctx := context.Background()

    // Set up credentials to authenticate with Google Cloud

    client := clientConnection()

    bucketName := bucket_name
    filePath := file_path

    file, err := os.Open(filePath)

    if err != nil {
        log.Fatalf("Failed to open file: %v", err)

    }

    defer file.Close()

    // timestamp the file
    today := time.Now().Format("2006_01_02")

    // write the date part of the file path name
    filePath = fmt.Sprintf("%s_%s", today, filePath)

    // Create a new writer for the file in the bucket
    writer := client.Bucket(bucketName).Object(filePath).NewWriter(ctx)

    // Copy the content
    if _, err := io.Copy(writer, file); err != nil {
        log.Fatalf("Failed to write file to bucket: %v", err)
    }

    // Close the writer to flush the contents to the bucket
    if err := writer.Close(); err != nil {
        log.Fatalf("Failed to close writer %v", err)

    }

    log.Printf("File %q uploaded to bucket %q. \n", filePath, bucketName)

}

Note the date is currently hardcoded and will need to be changed.

in main.go:



buckets.WriteToBucket("food-bucket-dev", "cheese.json")
buckets.ReadFromBucket("food-bucket-dev","2023_03_12_cheese.json")

Creating Mock Data:

The purpose of this is jsut to create some mock data to use for the purpose of the project. we create a json of different cheeses.



// write the cheeseMap to a json file

func WriteCheeseMapToJSONFile() {

    cheeseMap := map[int]string{100: "chedder", 200: "swiss", 300: "gouda", 400: "mozzarella"}

    // create a file

    file, err := os.Create("cheese.json")

    if err != nil {

        log.Fatal(err)

    }

    defer file.Close()

// create a json encoder
encoder := json.NewEncoder(file)
encoder.SetIndent("", "  ")

// write the cheeseMap to the file
err = encoder.Encode(cheeseMap)
if err != nil {
    log.Fatal(err)
}



}

func DeleteJsonFile() {

    err := os.Remove("cheese.json")

    if err != nil {

        log.Fatal(err)

    }

}

Deploying and Checking out Deployment

As mentioned before deploying in Railway is as simple as PR push or manually clicking the deploy button on a project.

From there your deployment will run and you can check the logs that everything ran successfully.

Logs should show the JSON and the bucket creation:

And we can check in GCP in the cloud storage panel which looks like it was a success!

Conclusion

Now we have established a way to setup Object storage that you can use with you Railway project. I created this for a personal project I am working on where I needed to store some files. As I mentioned before I am a huge fan of Railway and looking forward to creating more content and sharing my next personal project with everyone on Railway!

If you enjoyed this post let me know! I am happy to create more guides using different languages (Python, Typescript, even Rust!)

Local Development Environments for Python using asdf & Poetry

Ramzi A. — Mon, 06 Mar 2023 21:04:58 +0000

In this blog post I introduced my general setup of how I use asdf for local development for languages. That was meant as a dip into the methodology in order to ramp us up for where I see the real benefit using this for Python & virtual environments.

We will go through almost the exact same setup until we add Poetry and i'll walk you through step by step.

Assumptions:

You have homebrew homebrew install
You are using MacOS or Linux.

Step 1: Pre-reqs:

asdf: Installing asdf this is how you can manage different versions of languages and use them across projects
Installing Poetry (for Python only) is a way to manage dependencies & virtual environments in Python. This was shown to me by Data Scientist at work and I would never go back using any other method.

Step 1: Setup for both install asdf & add to your shell (I use zsh, so adjust if you use bash or something else).

brew install asdf
echo -e "\n. $(brew --prefix asdf)/libexec/asdf.sh" >> ~/.zsh_profile

Step 2: Python Setup

In asdf the order of operations is adding a 1. language plugin, 2. adding a version(s) and 3.then setting the language locally or globally for your projects

Setup Langauge Plugin

Setting up asdf for go run asdf plugin add python
adding different go versions if you want the latest run asdf install python latest
OR install a specific version running asdf install python 3.10.0
For Python in my organization we recommend using Python version 3.n-1. So if the latest is 3.11 we would use 3.10 in production.
In the case below I already have both installed

Setup Local Version

In this case I have a hello-python directory that i want to use Python 3.10 in order to do that you would run

asdf local python 3.10.0 This is really important you want to keep your python versions separate for repos so you are using the correct version for your project this makes it easier to manage dependencies when we introduce Poetry.

An additional note you will only have to do this once in the directory unless you want to change the version number.

Step 2: Setting up Poetry

We should have poetry installed & the path setup in your shell.
Make sure you're in the directory of the project you want to setup the virtrual environment.
Poetry is like pyenv or venv for Python. the difference is it also manages dependencies issues building a .lock & .toml file
run poetry init and start the walkthrough in the CLI.
You can add the naming of the project, the license, the python versions and even the packages. The packages can be added after the generation too.

once you went through everything you can confirm the generation this will create a .toml file
You can now install all the dependencies from the pyproject.toml by running
poetry install
this creates a virtual environment in the python version you have locally and all the dependencies.

The beauty of Poetry is that if you have a version conflict with different packages it will attempt to resolve it by upgrading/downgrading the package based on the constraints you set for the project. An example is if you have numpy version 1.20 and pandas 0.16 they might be compatiable it will resolve the issue by upgrading pandas based on the constraints you set on the pyproject.toml

Step 3: Using the Virtual Env & Poetry

to add packages you can run poetry add pandas
to remove packages you can run poetry remove pandas
to enter the virtual environment run poetry shell and then running python hello.py or run you files by running poetry run python hello.py

Tips:

Dont name your project the same as a python project there can be naming conflicts. i.e. your project is named Pandas and you are adding Pandas.
You can adjust the pyproject.toml and reinstall the packages in your virtural env

Local Development Environments for Go using asdf.

Ramzi A. — Mon, 06 Mar 2023 01:40:49 +0000

Over the years I have dealt with different ways to setup a project for Python & Go. I am sharing how I setup my personal and work projects for extensibility and modularity.

There are a lot of ways to setup Python dev environments and Go but this is what works best for me and how I get higher adoption at work for my projects. (This can be extended for any language btw Node or Rust even).

I will write up how I setup Go projects in this post & then follow up with a Python version which adopts a lot the same concepts.

Assumptions:

You have homebrew homebrew install
You are using MacOS or Linux.

Step 1: Pre-reqs:

asdf: Installing asdf this is how you can manage different versions of languages and use them across projects ~~2. Installing Poetry (for Python only) is a way to manage dependencies & virtual environments in Python. This was shown to me by Data Scientist at work and I would never go back using any other method. ~~ this will be relevant for the Python blog post

Step 1: Setup for both install asdf & add to your shell (I use zsh, so adjust if you use bash or something else).

brew install asdf
echo -e "\n. $(brew --prefix asdf)/libexec/asdf.sh" >> ~/.zsh_profile

Step 2: Go Setup

In asdf the order of operations is adding a 1. language plugin, 2. adding a version(s) and 3.then setting the language locally or globally for your projects

Setup Langauge Plugin

Setting up asdf for go run asdf plugin add golang
adding different go versions if you want the latest run asdf install golang latest
OR install a specific version running asdf install golang 1.19

Setup Local & Global Version

In this case I have a hello-go directory that i want to use Go 1.19 in order to do that you would run

asdf local golang 1.19

Init a baseline project.

Get the current directory and init a go project

run pwd which will output the directory /Users/ramzi/documents/projects/hello-go in my case.
run go mod init documents/projects/hello-go

Create hello.go file by running touch hello.go and add in the following code:
Run go run hello.go and you should get your output printed!

Closing Thoughts

The purpose of this is when you need to have multiple version of a language across different projects and you want to keep this binaries separate. With asdf you can set a global version (which I do not recommend) but to do so you will run asdf global golang 1.19 for example.

This is especially important as a Python developer which I will share in an upcoming blog post.

Get Paid To Train Magic

Ramzi A. — Sat, 25 Sep 2021 02:11:54 +0000

Grab a wizard mind bomb

Let's talk about magic training in old school Runescape.

In old school Runescape
a typical way to train your magic level is to use a high alchemy spell.

The High Alchemy spell turns an item into gold pieces (denoted as gp in Runescape's currency).

You find the item that will return the most profit based on the player economy. In order to optimize your magic training generally you train at a loss and that is normally accepted.

Loss Scenario:

The item you buy is 500gp, a nature rune is 200gp, and when you complete the high alchemy spell it turns the item into the average market value of 680gp.

Your cost is 700gp
Your return is 680gp
Your loss is -20gp

Gain Scenario:

The item you buy is 500gp, a nature rune is 200gp and when you complete high alchemy spell the average market value of the item is 750gp.

Your cost is 700gp
Your return is 750gp
Your gain is 50gp

The market is dynamic and can change in an instant though so you have to watch and find items that will be ideal to gain a profit. This can be a full time job in itself if you really want to commit to it.

But not with my new startup that eliminates the guess work and saves you time

hialchemy.com

Hialchemy is one of the first movers in the virtual fintech world. Actually Hialchemy coined the term virtual fintech (all puns intended).

I sort the top 100 items with the highest return value based on last purchase price. Or sort on item name, price whatever you desire. We always show the most recent Nature rune price to help you gauge how much they will cost.

Or you can search for any item and see its return value. On top of that it calculates the cost, experience and time it will take of the number of alchs you want to perform!

The Tech Stack

Frontend:

Streamlit.io a really cool python library I found that helps create data tools really quickly to put on the web. It was the quickest way to make an MVP for me so I chose this and focused primarily on the backend.

Backend:

Databases/Storage:

The primary database is NoSQL Firebase. The reason I chose this was because it was something I never used before and wanted to explore how it worked. It is a NoSQL document storage database.

I use GCP buckets for historical json storage of daily snapshots of data.

In addition I send all my data to BigQuery from Firebase (there is a direction built in connection). All my data is transformed using dbt. The data that gets fed into the frontend is actually transformed data from dbt!

Orchestration:

Pub/Sub is the main method of orchestration for all data. Where all my data Pipelines are built in Python and trigged as cloud functions. They are simple json requests to pull api data or publish data in firebase or a gcp bucket.

Deployment:

I use primarily Terraform to deploy everything, there are some pieces that I use cloudrun for that was easier to setup than Terraform.

Whats Next?

More frequent purchase updates
Create an API using FastAPI
Publish a deeper dive into the tech stack
Rewrite frontend in Flask (maybe a javascript library?)
Move backend historical data from gcs bucket to Postgresql

Credits:
I am a big 10k diver fan and took inspiration from his threads for this post. https://10kdiver.com/twitter-threads/

Should I Take the Data Engineer Nano Degree?

Ramzi A. — Wed, 12 Aug 2020 18:48:28 +0000

I just completed the Udacity Data Engineering program and wanted to share my insights about the program to other people that were thinking of taking it. You will use S3, Redshift, Spark, EMR, EC2, Airflow, Cassandra, & Postgres throughout the course. With a focus on the end on Airflow, Spark, S3 & Redshift.

The Five Modules you will go through

Data Modeling in Apache Cassandra & Postgres
Cloud Data Warehouses (AWS, S3, Redshift)
Building a Data Lake with an ETL pipeline in Spark (EMR, Spark)
Using Airflow as a data pipeline to schedule data sends.
An open ended project similar to project 3+4 or using their template project with their own data.

Why I took it:

I wanted to get better at data governance, data quality, and data pipelines so I can introduce these concepts at my current role, along with being more familiar with Airflow and AWS. Overall take more ownership of the data engineering I am doing now.

Our current stack includes data loaders like Stitch and Fivetran but there has been more instances where Stitch and Fivetran don't meet our needs of integrating all our data sources. This is where I saw a need to leverage Airflow into the picture to bring data into our warehouse but also into other sources to be used by our business users.

My Experience:

Coming into this program I felt really confident with my SQL skills and Python skills to take this nano degree. I don't write python as much as I liked to (maybe once a week), and there were some cases I struggled with the content because of its involvement with Python in unfamiliar scenarios. Especially near the Airflow portion but its manageable if you just search examples and the Airflow documentation.

There were a lot of issues with the course after module 2, to the point a large chunk of my peers were complaining. They eventually added an extension to module 3 with new videos and sources that makes the completion of it a lot easier but should have been there from the start. Unfortunately I really struggled here to complete this portion before the introduced the new videos and content, so I was stuck googling and reading through the AWS documents to figure it out.

This was a common theme after the 2nd module. A lot of the information and resources were outdated, I felt like I was troubleshooting and learning AWS IAM roles, or AWS security permissions than I was learning data engineering. These issues were only remedied when enough people had an issue. It felt like a half finished program. Although through the hardship I got a lot better at understanding AWS and Airflow because of limited information I had to pass each module.

Overall though I really enjoyed the program to get exposure to these tools and concepts. The highlights for me was learning Spark, and Airflow. I am comfortable enough now I can stand up my own ecosystem where I can send data to a warehouse from different sources along with having tests on the data to ensure its quality. This was my overall goal taking this nano degree.

Do I recommend this program?

Do I recommend it? It depends on the work you do or want to do.

With tools like dbt, dbt Cloud, Stitch and Fivetran you can avoid a lot of headaches and stand up an analytics ecosystem relatively quickly. I don't think this is the nano degree is for you if that is your goal. Buying over building makes more sense 9/10 times especially if you are a startup trying to gain insights quickly.

But if you're building a backend for an application or purposes outside of an analytics warehouse. Then I would potentially take this course and proceed with caution.

I think a better step is to choose one of these technologies and focus on them. (Take a Udemy Course on Airflow first for example).

Overall Pros:

Learned AWS, Airflow, Spark
Good background information on data modeling, traditional data schemas.
Highlighted data quality and data governance how to introduce tests within your data pipeline.
Great community of help

Overall Cons:

Unclear directions and modules become a huge time sink
Can burn through free AWS credits quickly. You spend roughly $200 USD a month on the course and then maybe an additional $100-150 USD Month on AWS products trying to figure out the projects due to the outdated documentation and resources provided. (Overall I had to spend an additional $320 over 3 months)
Spending more time troubleshooting problems with the course or filling gaps of it then learning about data engineering.

Overall Rating 3/5