Forem: Rafael Lourenço

Setting Up a PySpark Cluster with Docker: Guide

Rafael Lourenço — Thu, 02 May 2024 06:46:01 +0000

Diving back into Python with a focus on expanding my knowledge in data processing, which has sparked my interest in creating a Proof of Concept (POC) for a data playground. With help of @caiocampoos we started this repo here.

Even with almost zero knowledge in the area, we began discussing ideas about what we want to do. The main idea is to create knowledge through learning, and one of the best ways to learn is by creating something. In this project, we're going to develop a basic data analysis platform. Throughout the process, we documented the code.

Talk is cheap, let's code!

Let's start with a PySpark. PySpark is essentially Apache Spark tailored to integrate smoothly with Python. Python dominance in the data science realm makes PySpark an ideal choice for our business-oriented project. While Apache Spark offers support for various languages, our emphasis on Python stems from its versatility and efficiency in our business-focused endeavor. Therefore, we'll proceed with PySpark to ensure seamless integration with our Python-centric workflow.

Pyspark has a convenience Docker container image you can also browse around Apache Spark website to check others version.

We starting our project creating a Dockerfile and with this code.

FROM python:3.10-bullseye as spark-base
ARG SPARK_VERSION=3.5.1

The first line of code set the base image to a Python image with version 3.10 on the Debian "Bullseye" distribution, naming it spark-base.

The second line of code defines a version of spark we gonna use, you may change that in the future.

Then we write this.


RUN apt-get update && \
    apt-get install -y --no-install-recommends \
      sudo \
      curl \
      vim \
      unzip \
      rsync \
      openjdk-11-jdk \
      build-essential \
      software-properties-common \
      ssh && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

With this next lines of code we setup the directories for our Spark and Hadoop installations.

ENV SPARK_HOME=${SPARK_HOME:-"/opt/spark"}
ENV HADOOP_HOME=${HADOOP_HOME:-"/opt/hadoop"}

RUN mkdir -p ${HADOOP_HOME} && mkdir -p ${SPARK_HOME}
WORKDIR ${SPARK_HOME}

This command downloads and installs Spark and Hadoop.

RUN curl https://dlcdn.apache.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz -o spark-${SPARK_VERSION}-bin-hadoop3.tgz \
 && tar xvzf spark-${SPARK_VERSION}-bin-hadoop3.tgz --directory /opt/spark --strip-components 1 \
 && rm -rf spark-${SPARK_VERSION}-bin-hadoop3.tgz

The next lines of code in the Dockerfile import and install all Python dependencies that we are going to cover in another folder.

FROM spark-base as pyspark

COPY requirements/requirements.txt .
RUN pip3 install -r requirements.txt

Now we gonna set all the enveriroment variables.


ENV PATH="/opt/spark/sbin:/opt/spark/bin:${PATH}"
ENV SPARK_MASTER="spark://spark-master:7077"
ENV SPARK_MASTER_HOST spark-master
ENV SPARK_MASTER_PORT 7077
ENV PYSPARK_PYTHON python3

The next lines of code will copy the configuration files, set permissions, and configure the environment variables accordingly.

COPY conf/spark-defaults.conf "$SPARK_HOME/conf"

RUN chmod u+x /opt/spark/sbin/* && \
    chmod u+x /opt/spark/bin/*

ENV PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH

And finally, copy the entrypoint script and set it.

COPY entrypoint.sh .

ENTRYPOINT ["./entrypoint.sh"]

Here the full code of dockerfile.



FROM python:3.10-bullseye as spark-base

ARG SPARK_VERSION=3.5.1

# Install tools required by the OS
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
      sudo \
      curl \
      vim \
      unzip \
      rsync \
      openjdk-11-jdk \
      build-essential \
      software-properties-common \
      ssh && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*


# Setup the directories for our Spark and Hadoop installations
ENV SPARK_HOME=${SPARK_HOME:-"/opt/spark"}
ENV HADOOP_HOME=${HADOOP_HOME:-"/opt/hadoop"}

RUN mkdir -p ${HADOOP_HOME} && mkdir -p ${SPARK_HOME}
WORKDIR ${SPARK_HOME}

# Download and install Spark
RUN curl https://dlcdn.apache.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz -o spark-${SPARK_VERSION}-bin-hadoop3.tgz \
 && tar xvzf spark-${SPARK_VERSION}-bin-hadoop3.tgz --directory /opt/spark --strip-components 1 \
 && rm -rf spark-${SPARK_VERSION}-bin-hadoop3.tgz


FROM spark-base as pyspark

# Install python deps
COPY requirements/requirements.txt .
RUN pip3 install -r requirements.txt

# Setup Spark related environment variables
ENV PATH="/opt/spark/sbin:/opt/spark/bin:${PATH}"
ENV SPARK_MASTER="spark://spark-master:7077"
ENV SPARK_MASTER_HOST spark-master
ENV SPARK_MASTER_PORT 7077
ENV PYSPARK_PYTHON python3

# Copy the default configurations into $SPARK_HOME/conf
COPY conf/spark-defaults.conf "$SPARK_HOME/conf"

RUN chmod u+x /opt/spark/sbin/* && \
    chmod u+x /opt/spark/bin/*

ENV PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH

# Copy appropriate entrypoint script
COPY entrypoint.sh .

ENTRYPOINT ["./entrypoint.sh"]

Now we need create a entrypoint.sh file for our script, this Bash script is designed to start different components of Apache Spark based on the value passed to the environment variable SPARK_WORKLOAD. We are using a flag --memory 1g to limit the use of memory for workers, you can change that if you want.


#!/bin/bash

SPARK_WORKLOAD=$1

echo "SPARK_WORKLOAD: $SPARK_WORKLOAD"

if [ "$SPARK_WORKLOAD" == "master" ];
then
  start-master.sh -p 7077
elif [ "$SPARK_WORKLOAD" == "worker" ];
then
  start-worker.sh spark://spark-master:7077 --memory 1g
elif [ "$SPARK_WORKLOAD" == "history" ]
then
  start-history-server.sh
fi

And a folder requirements with requirements.txt file inside.

ipython
pandas
pyarrow
numpy
pyspark

Our Docker Compose configuration is divided into three key components: the Spark Master, responsible for orchestrating the Spark cluster; the Spark History Server, which provides historical data on completed Spark applications; and the Spark Worker, representing a worker node within the cluster.

This is all the code we use in the docker-compose file.

version: '3.8'

services:
  spark-master:
    container_name: da-spark-master
    build: .
    image: da-spark-image
    entrypoint: ['./entrypoint.sh', 'master']
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:8080" ]
      interval: 5s
      timeout: 3s
      retries: 3
    volumes:
      - ./book_data:/opt/spark/data
      - ./spark_apps:/opt/spark/apps
      - spark-logs:/opt/spark/spark-events
    env_file:
      - .env.spark
    ports:
      - '9090:8080'
      - '7077:7077'


  spark-history-server:
    container_name: da-spark-history
    image: da-spark-image
    entrypoint: ['./entrypoint.sh', 'history']
    depends_on:
      - spark-master
    env_file:
      - .env.spark
    volumes:
      - spark-logs:/opt/spark/spark-events
    ports:
      - '18080:18080'

  spark-worker:
#    container_name: da-spark-worker
    image: da-spark-image
    entrypoint: ['./entrypoint.sh', 'worker']
    depends_on:
      - spark-master
    env_file:
      - .env.spark
    volumes:
      - ./book_data:/opt/spark/data
      - ./spark_apps:/opt/spark/apps
      - spark-logs:/opt/spark/spark-events


volumes:
  spark-logs:

We also need to create a folder named 'config' with a file named 'spark-defaults.conf' to store all configurations for Spark.


spark.master                            spark://localhost:7077
spark.eventLog.enabled                  true
spark.eventLog.dir                      /opt/spark/spark-events
spark.history.fs.logDirectory           /opt/spark/spark-events

And a ssh_config file with.


Host *
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no

And a .env.spark file with.

SPARK_NO_DAEMONIZE=true

And finally our makefile file.


build:
    docker-compose build

build-nc:
    docker-compose build --no-cache

build-progress:
    docker-compose build --no-cache --progress=plain

down:
    docker-compose down --volumes --remove-orphans

run:
    make down && docker-compose up

run-scaled:
    make down && docker-compose up --scale spark-worker=3 

run-d:
    make down && docker-compose up -d

stop:
    docker-compose stop

submit:
    docker exec da-spark-master spark-submit --master spark://spark-master:7077 --deploy-mode client ./apps/$(app)

submit-da-book:
    make submit app=data_analysis_book/$(app)

rm-results:
    rm -r book_data/results/*

Now we can just build and run how many workers we need!

Just one more time

Rafael Lourenço — Thu, 25 Jan 2024 14:27:55 +0000

"Just one more state, bro! I promise this is the last one."

Perhaps you've already heard a technobro (technology enthusiast) saying something like that. Well, that technobro is me, just mentioning one more challenge. I promise this is the last one.

I'm back to the Woovi challenge again, and this time I'll try to make something simpler. I'll attempt to create a droplist wiki for a game fueled by players with a few moderators on Discord.

The idea is pretty straightforward—a CRUD with GraphQL, Relay, and React. I'll use Next.js version 13 because I'm more familiar with it, and every new version of Next.js adds overhead to understand with tons of new documentation. So, I hope this time things work better.

I have talked with people who want to use the wiki, and the design should be something very simple. The interface should be as simple as possible—perhaps this is the most minimalist project that I have done.

The end of a project!

Rafael Lourenço — Tue, 23 Jan 2024 17:30:26 +0000

Today, I found myself pondering a question: When can we genuinely declare a project as complete? This question becomes even more complex in the programming world, where even minor projects can undergo refactoring over an extended duration. The reason is straightforward: with an expanding understanding, there’s always an opportunity for enhancing and optimizing your code.

In my view, a project can be deemed complete when specific objectives are established. After all, we wouldn’t want to be perpetually coding one thing – the universe of programming is vast! Setting clear objectives not only steers the process but also serves as a roadmap to finish a project without any regrets.

Now, if you find the outcome unsatisfactory after some time, don’t hesitate to establish new objectives. Just be wary not to plunge into coding without a plan; you might lose track of your initial motivation and end up feeling somewhat lost in the process.

As for my latest project, the Leaky Bucket with Redis and Node, it’s complete, and I’m satisfied with it for the time being. Perhaps in the future, equipped with more knowledge and new tools, I’ll revisit and revamp it. But for now, I believe the initial objective has been accomplished.

The next step is to return to the Woovi Challenge, but I’m contemplating doing something simpler than an e-commerce. Since my friend has already established a page with no-code tools, I just need to think what I can do. There are numerous problems in the world that need solutions, but your personal issues should be the top priority.

What I have learned in the last project

Rafael Lourenço — Thu, 11 Jan 2024 22:02:48 +0000

My last project is almost done, it's a Leaky Bucket algorithm with Node and Redis, and a simple front end showing requests and data. Here are a few things that I learned while building this.

First of all, it's almost impossible to build things without a list of tasks. I tried skipping a to-do list in the beginning because I thought it would be faster, but I encountered a few problems that I had to think about. Most of the problems are easy to solve, but they appear in abundance when you are very inexperienced.

A to-do list can help you stay focused on what matters most.

The second thing that I really learned is how easy it is to use tools when they are very well documented. Good documentation can be a real game-changer when choosing technologies to use. If the technology is poorly documented, no matter how good it is, at the end of the day, you want to choose the simplest and well-documented tech.

The most valuable weapon is the one that you can use

The third and last, but not least, is the importance of writing down what you have learned. Programming is not easy; it starts to become easy because you have practiced so many times. Writing can help you solidify that practice. You can write the same code three times or write it once and create a text explaining how and why you have done it. When you explain what you have done, you are learning more, because explaining something is the ultimate level of acquiring knowledge. If you don't know how to explain something, you really don't know that thing, you only kinda know.

You only really know when you can explain. If you don't explain, you kinda know.

How am I using AI to get the best of myself

Rafael Lourenço — Thu, 28 Dec 2023 17:31:11 +0000

In this text, I'll try to explain how I am using AI to bring out the best in myself, and I think I can start with one of my most significant weaknesses.

Writing can be quite a challenge for me sometimes due to my dyslexia, and it's not uncommon to encounter misspellings in my personal handbook, like 'wrods.' Thankfully, with the help of AI, I don't struggle as much in the virtual world. All of my texts pass by AI to catch misspellings and errors. Thus, my most crucial and effective use of AI is in enhancing my writing.

My second most important use of AI is for searching. The Bing search engine is actually more powerful than Google. It's easy for me to find anything on the internet using Bing. Even if I don't fully understand, Bing can summarize and assist me in the search.

Thirdly, I consider it to be the most frequent use case, although not necessarily the most important. I use AI to generate code blocks. I spend a significant amount of time writing and incorporating these code blocks into my projects. The reason for using AI is the convenience it provides in reviewing the meaning of the code block and identifying necessary corrections to achieve my desired outcome. Creating code blocks with AI not only streamlines the process but also saves a considerable amount of time

All three of these use cases are very important in my life for bringing out the best in myself. Without them, it would probably be much harder for me to be where I am right now

I made a logical mistake

Rafael Lourenço — Tue, 19 Dec 2023 14:51:24 +0000

I made a logical mistake in my code, but that's okay. Making mistakes is entirely normal. In fact, every attempt involves a few failures, and my error is a straightforward logical resolution in a problem that involves a leaky bucket algorithm with Redis.

Redis must have a bucket to count how many tokens we have, and only API requests should reduce our tokens. However, we can only call the API if we have enough tokens to spend in our bucket. There are two errors that I made here. First, my Redis calls are depleting the buckets too, and secondly, all my calls are going to the API first, and only after that, the tokens are reduced. All these implementation ideas come from me, but when implemented, things just go wrong.

Practical learning is not easy; you're going to be wrong most of the time. However, every mistake you solve contributes to solid knowledge. If you want to be one of the best, you should practice more.

The Bubble

Rafael Lourenço — Mon, 18 Dec 2023 12:47:22 +0000

This weekend, I started thinking about what non-coding problems I could solve. After spending some time reflecting, I realized there are many things I can do to help solve problems. First of all, there's basic accountability. I just helped a friend who doesn't know much about managing money by creating a spreadsheet template for him. Then I realized he doesn't know how to search for things for his business, so I introduced him to the Bing search AI.

I just realized there are so many people who don't use AI engines and others who don't even know about the existence of powerful AIs. And courses like 'How to use AI in your job' are starting to make sense to me right now. It's curious how we live in a bubble of knowledge, and sometimes reality breaks through that bubble.

How to Export Global Variables in Node.js

Rafael Lourenço — Fri, 15 Dec 2023 15:27:01 +0000

Imagine you are using Koa.js to create a route, and within that route, you retrieve a key that you intend to use in another router, function, or elsewhere. In such cases, you may want to use the easiest method to export that key for subsequent use.

One of the easiest case are export that key with global variable, to do that you may try weird thinngs likes me

let key = ''
const router = new Router();

router.get('/createkey', async (ctx: ParameterizedContext) => {
  const tokenData = await getToken();
  key= tokenData.access_token ;
  await client.set('createkey', tokenData.access_token, { EX: 2399 });
  ctx.body = 'Key created'
  console.log('Key created')
  console.log(key)
});

export { key, router as default };

Trust me, that doesn't work. The easiest way to achieve this in Node.js is by using the global object.

global.key

const router = new Router();

router.get('/createkey', async (ctx: ParameterizedContext) => {
  const tokenData = await getToken();
  global.key= tokenData.access_token ;
  await client.set('createkey', tokenData.access_token, { EX: 2399 });
  ctx.body = 'Key created'
  console.log('Key created')
  console.log(global.key)
});


export default router;

With that object, you can access and modify your global variable in every part of your project.

Understanding the Leaky Bucket Problem with Redis

Rafael Lourenço — Thu, 14 Dec 2023 12:45:14 +0000

In my path to become a software engineer, I was challenged by my mentor @sibelius to demonstrate how I implement the leaky bucket with this API documentation.

To work on that i started drawinng on Excalidraw and got this design.

Understanding this picture is pretty simple; all the queries are going to hit Redis to check if the information already exists in memory. If the information exists in memory, we get the answer back. If not, we have to check if we have 'coins' in our bucket to make a query to the DICT API and get the answer back to the client. We then save that in Redis. If our bucket is empty, we don't call the DICT API and respond with a rate limit.

This initial application will have a very simple calculation for bucket usage:

Status 200: Remove 1 coin
Status 400: Remove 20 coins
Bucket refill: 2 coins per minute
Bucket size: 100 for individuals and 1000 for legal entities

The Hydra Knowledge Theorem

Rafael Lourenço — Wed, 13 Dec 2023 13:42:57 +0000

I start feeling something that I am considering good. I wake up every day with doubt, and I go to sleep with more doubts than I had when I woke up.

So, it's like a Hydra, where you cut down one head, and two more sprout in its place. The path I'm taking in software development is always expanding my doubts.

Yesterday, I woke up trying to figure out how to implement the leaky-bucket algorithm with Node and Redis, and I went to sleep with the doubt of how to configure and connect mutual TLS. The difficulties in programming seem to escalate every time I take a step up and they don't seem to have an end.

What problem am I solving?

Rafael Lourenço — Mon, 11 Dec 2023 14:26:58 +0000

I read a tweet today and started asking myself, What problem am I solving?

Solving other people's problems is the way to be valued. If I want to be highly valued, I should solve problems. The manner in which I solve them doesn't matter too much, whether a problem can be resolved with code or a call doesn't matter at the end of the day. As long as the problem is solved, the result should be the same.

Custom Paths for mocks Folder in Monorepo using Jest

Rafael Lourenço — Fri, 08 Dec 2023 11:16:46 +0000

Imagine you have a monorepo with tons of repositories. For each repository, you encounter the same issue: a need to maintain an identical mock in each repository. Whenever you update one, you must manually update all the others. This can result in a significant waste of your time.

You have a good idea to place your mock at the root level for importing into all tests as needed. While this approach works, it introduces a new small problem: Your imports can become like Morse code, especially if your tests are deeply nested.

To solve that, we're going to create a custom path for your mocks folder inside your Jest configuration for each repository in your monorepo.

 moduleNameMapper: {
    '^@mocks/(.*)$': '<rootDir>/<your_path_to_your_mock_folder>/$1', 
  },

And now, we can import our mock inside each test using this syntax:

const mockAdd = require('@mocks/math');


test('uses mockAdd to add 1 + 2 to equal 3', () => {
  expect(mockAdd(1, 2)).toBe(3);
});

And our mock tests are well-exported.

const mockAdd = jest.fn((a, b) => a + b);

module.exports = mockAdd;

If your tests are correct, this should be the result.