Forem: Cup of Code

RSUs — What if you turned them into ETFs?

Cup of Code — Sun, 01 Jun 2025 15:08:28 +0000

Thinking of cashing out the RSUs and investing the money elsewhere? Let’s run the numbers first!

In the previous blog post, we talked about the RSUs portion of an employee’s compensation. At first, they are granted to you, and when the time comes, they are vested: The shares are yours!

Once you can vest your shares, you need to pay income tax on them. For example, if you get 10 shares worth $100 each at the time of vesting, you need to pay 52% of their value to taxes (that number depends on where you live). This means that for those 10 shares, you need to pay $520 in taxes.

We compared three options to handle the stocks given to you by the company:

Sell all when received: sell all 10 shares and take $480 home.
Sell to cover taxes: sell 6 shares, invest 4 shares, and take $80 home.
Keep all: add $520 to your brokerage account before the shares vest, and then invest 10 shares.

You can find the full comparison of these options here.

Today, we will explore a fourth option: sell all and direct the money towards an investment of your choice!

Remember Aoife? She works at a company called Nile, and as part of her compensation plan, she received 20 RSUs in 2020 that were vested throughout the years. Somewhere after January 2022, her company did a 1-to-20 split of stocks, so the original 4 shares vesting became 80 smaller shares.

In today’s blog post, I’ll present the profits that Aoife would’ve gained if she were to invest the extra income in S&P 500 ETFs and sell in January 2025. Aoife might not be real, but the numbers are!

Before we pull out the calculators, we need to review some popular terms from the investment world.

S&P 500

The Standard and Poor’s 500, or simply the S&P 500, is a stock market index tracking the stock performance of 500 leading companies listed on stock exchanges in the United States. This is one of the first terms you hear when you start looking into investments because the funds tracking it are considered to be a beginner-friendly, safe option due to its broader scope.

ETFs

An ETF, or exchange-traded fund (ETF), is an investment fund that allows investors to purchase a variety of different stocks and/or bonds at once. In our case, our ETF is tracking the S&P 500 index. Like with other competitive markets, there are several ETF providers you can choose from.

In the screenshot below, I attached the graphs of 4 different ETFs tracking the S&P 500. You can see they have similar trends, meaning they are all tracking the index pretty accurately! On the left, I have listed the top 8 companies, including the weight in %, represented in the Vanguard S&P 500 ETF.

To explore this new scenario, Aoife will cash out her RSUs and invest that money in the S&P 500 ETF.

Note that there are three important differences between investing in the S&P 500 ETF shares and investing in Nile shares:

The S&P 500 ETF holds multiple stocks: Nile is one of those 500 companies, representing ~3.85% of those assets. So yes, she is selling Nile stocks and later purchasing some Nile stocks.
You can purchase fractional ETF shares. In the previous blog post, when Aoife was selling to cover taxes, she was sometimes left with cash. That’s because you can’t buy a fraction of a stock. But with ETFs, you don’t need to reach the whole value of a share to purchase!
If you couldn’t tell by her name, Aoife is an Irish resident, and therefore, her ETF’s capital gain is taxed at 41%, unlike the 33% that stocks are taxed at. We will calculate that into the formula.

Before we start, note that we are comparing stock profit to ETF profit. Meaning, in a normal setting, Aoife would’ve invested all the cash received from selling the RSUs. For our experiment, we will only invest the money that was invested in the stocks, and any leftover cash (that didn’t accumulate to a full stock price) won’t be invested in ETFs either.

Now we can start calculating :D

Let’s look at the S&P 500 index in the last 5 years (link):

The table on the right shows us the amount of money invested in Nile shares by vesting period.

January 2021: Aoife got her first share! But she had to sell it to cover taxes. As mentioned earlier, this means we won’t be investing in any ETFs this time.
January 2022: 1 Share vested, worth $3240, so we put $3240 in our ETF!
July 2022: 38 shares (post 1-to-20 share split) worth $110 each, meaning $4180 goes to ETFs!
January 2023: 38 shares worth $86 each, meaning $3238 goes to ETFs!
July 2023: 38 shares worth $130 each, meaning $4940 goes to ETFs!
January 2024: 38 shares worth $145 each, meaning $3800 goes to ETFs!
January 2025: Selling the ETF shares. But how much did Aoife gain?

We will calculate the Capital Gain from each batch of purchase:

Unlike stocks, because we buy fractional ETF shares, you can’t just multiply the number of shares by the delta between the buy and sell values. Instead, I used a different method: In the link I shared earlier, you can get the percentage of value difference in a certain period.

For example, from January 2022 to January 2025, there was a 27% increase in value. This means the capital gain was 3240*0.27= $874.8. Now, because the capital gain tax in Ireland is 41%, Aoife’s profit is 874.8*0.59= $516.1.

Now it’s time to compare! If Aoife were to take the stock route, she would’ve made $33942! This number is a bit lower than the one presented in the previous blog post, because this does not include the leftover cash from covering taxes in any of the vestings.

If Aoife were to invest her RSUs money in an ETF, she would’ve made $25878! This is a $8064 difference!!

There are two reasons for this big gap:

First of all, the tax on ETFs is 8% more than on stocks. Second, the S&P 500 tracks 500 companies, so it’s more stable than an individual stock. Some shares go up more, some go down, and they balance themselves. The value of the ETF share increased by 27–55%, while the individual stock’s value fluctuated between 44%-172%! Of course, this is nice when you buy the share at $86 and sell at $235, but not so much when it’s the other way around.

As I concluded in the previous blog post, the stock market is kind of a gamble. I once heard some advice: “Don’t invest money you are not willing to lose”. At first I didn’t get it — I mind every money loss! But after a while, I understood: They mean you shouldn’t invest money you can’t afford to lose. So it shouldn’t be any money for a nearby purpose: emergency fund, vacation, wedding, house, etc. Be responsible!

This is not financial advice


https://cupofcode.blog/

Blogging is my hobby, so I happily spend time and money on it. If you enjoyed this blog post, putting 1 euro in my tipping jar will let me know :) Thank you for your support!

Understand Your Paycheck: RSUs

Cup of Code — Mon, 03 Feb 2025 12:50:05 +0000

Explaining the terms and simplifying the calculation process — with a real-life example!

If you’ve been ignoring the RSUs part in your paycheck — This blog post is for you! I have a walkthrough of all the calculations :D

I have a confession to make: I used to disregard the RSUs part of my compensation. It’s not “real” money coming into my checking account, so it doesn’t count. Recently, I decided to learn more about personal finances and gained a new appreciation for them!

Unfortunately, the stock portion of your compensation is not straightforward: RSUs vs shares, vesting options, taxed once, taxed twice…With all that, how much money do you end up with? This is a complex calculation, that I’m here to help with!

We’ll start with some general information, and then go through a real-life example!

Of course, nothing here is financial advice. I’m a software developer, not an accountant. Also, let me clarify — I am only using information available on the internet.

What are RSUs?

RSUs are Restricted Stock Units and are a common part of the employee’s compensation. They are called restricted because it is not fully transferable until certain conditions have been met (like tenure).

RSUs are a retention tool for employers: The employees have an incentive to stay in the company longer until the next RSUs are vested.. and the next ones.. and the next ones. Also, the employee might perform better knowing the success of the company will lead to an increase in share value, which will be worth more money for them (compared to a fixed base salary that will come in regardless of the company’s performance).

RSUs have a life cycle: First, they are granted to you (saved for you, kind of like a reserved table at a restaurant), and when the time comes — they are vested: The shares are yours! But… don’t get too comfortable, because you need to pay taxes on that income!

(Note that I use the words “share” and “stock” to describe the same thing, but if you want to be precise — “stock” is the financial instrument a company issues, and a “share” is a single instance of that financial instrument. — source)

Vesting Options

When it comes to vesting, there are three routes you can go: Selling all the new stocks (sell-for-profit), selling just enough to cover their taxes (sell-to-cover), or paying the taxes in advance (which are 52.1% today in Ireland). The choice depends on how much cash you can/will invest in it and whether you think the stock will increase in value. Some might prefer cashing the stocks and investing in other things by themselves. The default is sell-to-cover.

Lastly, if you choose to keep some stocks and they gain profit — once you sell them — you need to pay the capital gain tax of 33% (yes, EVERYTHING is taxed).

Share split

At some point, a company might decide to split the shares into smaller shares. In Amazon’s case, if you had a single share with a value of $3200 before the split — after the split you’ll own 20 shares of $160, which totals the same amount. That way, the shares are more affordable.

If you check the history of a certain stock online, you will see only the new (smaller) value — which can confuse you when you check the history of your RSUs. Don’t worry, it will be clearer with a real-life example!

Brokerage account

A brokerage account allows one to invest in publicly traded assets such as stocks. You probably can choose between brokerage companies your employer works with. Once you set up your vesting preferences, they will do the rest.

There are two fees you should be aware of: The first is the Trading Commission, which anyone who trades pays, regardless of which broker they choose to go with. The second one is the Supplemental Transaction Fee, which is an additional fee you pay your broker.

All the calculations we will see today do not include those fees.

Real Life Example

Meet my AI-generated friend, Aoife. She started working in a company called Nile 5 years ago (in January 2020). Her vesting approach from the beginning was to sell-to-cover. Let’s help her calculate the “real money” that will come out from the RSUs granted upon signing the contract if she were to sell them today (in 2025).

Aoife signed a compensation plan containing 20 RSUs, and her employment started in January. In her contract, it says that “RSUs follow a 4-year vesting schedule: 5% on the 15th day of the month in which you reach your first anniversary of employment, 15% on the 15th day of the month in which you reach your second anniversary of employment, and 20% every six months thereafter, until fully vested.”

Therefore, in Aoife’s case:

*Note that Amazon’s RSUs (Oops! Nile’s RSUs, I mean) went through a 1 to 20 split in June 2022, which will make our calculation even more complex!

Now, let’s add the value of the stock at the time to each line. I’ll be rounding the values for easy calculation, but you can see the exact share value in your periodic stock documents. Note that the value is post-split!


Nile’s share value in the last 5 years

Here are the numbers we are working with:

RSUs granted:

January 2021: 1 (=20) RSU, share value: $160

January 2022: 3 (=60) RSUs, share value: $162

July 2022: 4 (=80) RSUs, share value: $110

January 2023: 4 (=80) RSUs, share value: $86

July 2023: 4 (=80) RSUs, share value: $130

January 2024: 4 (=80) RSUs, share value: $145

January 2025: ------, share value: $234 <- value when selling

Pull Out Your Calculators!

Each vesting has its calculation — different share values, different tax amounts to pay, and different shares left post-taxes. There are three cases I’d like to go over:

Vesting (pre-split) where tax requires selling all shares.
Vesting (pre-split) with shares left post taxes.
Vesting post-split.

Lucky for us, these are the first three vestings (January 2021, January 2022, and July 2022). I will cover those scenarios within the blog post, and the rest of the calculation slides I’ll attach at the end of the blog post.

Lastly, we will calculate the profit if Aoife were to sell all her shares in 2025.

1. January 2021 — A Single Share Vesting

Here we have 1 (pre-split) RSU, with a value of 160*20= $3200. Taxes for this income are 3200*0.52= $1664. Because Aoife is selling-to-cover + there is only 1 share = she sells the share and gets an extra 3200*0.48= $1536 in her next paycheck. By the way, she will also see a line in the deduction column named “RSU TX WH” — which indicates money withholding for taxes from that vesting.

Note that when Aoife logs into her brokerage account, she will see that out of 20 shares vested in January 2021 — all were sold, which doesn’t fit the sell-to-cover approach. If it was, she would’ve sold only 11 shares and get to keep 8. This mismatch is because the number of shares presented is post-split, while the vesting was pre-split. Very confusing, I know!

Let’s review the other options for vesting on this example:

Selling for profit: In this case, sell-for-profit = sell-to-cover, because there was only one share.
Paying tax in advance: In that case, Aoife would’ve paid $1664 before the vesting, and then instead of getting $1536, she would’ve received 1 share.

2. January 2022 — A Pre-Split Vesting

Here we are still with the old shares. The value of the income (the granted 3 stocks) is 3*20*162= $9720. Therefore, income tax is 9720*0.52= $5054.4. Because Aoife is selling-to-cover, she has to sell 2/3 of her shares: This gives her 2*20*162=$6480. Lastly, after covering taxes, she is left with 6480-5054.4= $1425.6 extra cash in her next payslip.

To conclude, in January 2022, Aoife gets 1 share (pre-split, worth $3240) and $1425.6 cash.

Here as well, when Aoife logs into her brokerage account, she will see that out of the 80 shares she received in January 2022–60 were sold, which doesn’t fit the sell-to-cover approach. If it was, she would’ve sold only 42 shares and get to keep 38. This mismatch is because the number of shares presented is post-split, while the vesting was pre-split.

Let’s review the other options of vesting on this example (just to ensure we understand!):

Sell-for-profit: In this case, Aoife would’ve gotten 9720 * 0.48 = $4665.6 and 0 shares.
Paying tax in advance: In that case, Aoife would’ve needed to pay $5054.4 from her own money, and then she would’ve gained 3 shares.

3. July 2022— A Post-Split Vesting

This is the first vesting post-split! Originally Aoife was granted 4 RSUs with the original value, but after the split, those 4 shares became 80 shares, with the total value remaining the same: 80*110= $8800.

With smaller shares, we can sell-to-cover taxes more accurately — leaving us with more shares and less cash: Taxes for this income are 8800*0.52= $4576. To cover taxes, Aoife needs to sell 4576/110=41.6 -> 42 shares. Therefore, out of the 80 shares — Aoife keeps 38 shares, pays $4576 tax, and gets 42*110-4576= $44 extra cash in her next paycheck.

This was the calculation of the third vesting out of six. You can find the calculation slides for the rest of the vestings in the appendix of the blog post.

4. January 2025 — Selling All Shares

Ok, now Aoife wants to sell her stocks! She checked the current share value, and was happy to see that it increased since her RSUs vested! But… how much cash will she get? Let’s calculate!

For each vesting period, we need to calculate the difference between the total shares value on vesting and at the time of selling ($234 at the time of writing). That difference is the profit — and Aoife needs to pay capital gain tax on that.

Note that there are only 5 vestings in the image above, but there were 6 vesting dates. That is because the first vesting (January 2021) didn’t result in any shares.

Let’s look at the numbers: Aoife accumulated 172 shares that at the time of selling are worth 172*234= $40248. Out of that, 3240+4180+3268+4940+5510= $21138 is income Aoife already paid tax on because that was the value at the time of vesting. That’s what I call “Cash from original value”. The profit from the shares invested is 40248-21138= $19110. This amount is the capital gain — which requires a tax of 19110*0.33= $6306 (I rounded down), leaving us with 19110-63006= $12804.

Therefore, the total amount of cash entering Aoife’s account is 21138+12804= $33942.

To Conclude

Now, we can take all the cash earned over all the years, and sum it up:


Look how much Aoife paid in taxes!

Now, Aoife is wondering what would the profit be if she were to choose another vesting option:

Sell-all is easy to calculate: the value of shares vested * 0.48 (assuming 52% taxes). This means there is no income in 2025.
Keep-all calculation: keep the original number of shares vested, and extract the cash paid to cover taxes from the gain made in 2025.

Note that in Aoife’s case, if she were to not invest the cash in anything else in the “sell-all” option, and if Nile’s share value goes as in the example: keep all would’ve been the most profitable option.

Buying shares is a gamble because you don’t know if they will go up or down when you need to cash them out. Therefore, there is no “best approach” with your RSUs: Do what fits you, your financial situation, and your comfort (enjoyment?) with risk-taking.

For your convenience, I created a Google sheet with Aoife’s RSU calculations: you can make a copy and put your numbers in!


https://bit.ly/4aFInOQ

That’s it for today! Interested in digging more into your paycheck? Check out my blog post about Turning PDF Payslips Into a Single CSV Report!

Blogging is my hobby, so I happily spend time and money on it. If you enjoyed this blog post, putting 1 euro in my tipping jar will let me know :) Thank you for your support!

Appendix

The slides for the rest of the vesting calculations:

Turn PDF Payslips Into a Single CSV Report

Cup of Code — Sun, 22 Dec 2024 21:03:27 +0000

Ever programmed with PDF files? Write a Python script with me!

As you will see in the upcoming blog posts, I am in my financial literacy era. With the end of the year coming, I wanted to look at my numbers: How much taxes did I pay? How much did I earn for on-call shifts? Multiple PDF files are not the most comfortable way to see this data, and I wanted a single CSV file I could play with in Excel.

Like many good developers, I was too lazy to insert the numbers manually, so I wrote a script. If you enjoy programming — join me on an adventure! And if you are not in the mood — I’ll show you how to tweak the code to match your payslip structure :D


This script receives a directory with payslip PDFs and returns a CSV file with the desired data

The Plan

main.py:
# translate 1 pdf to 1 dict
# loop over the pdf dir
# save all dicts to 1 json file
# translate json report to csv report

We will start by writing the code to read a PDF, including deciding which fields we want in our report. This is the part you’ll need to tweak to match your payslip structure. Once that is figured out, we’ll iterate the whole payslips directory.

In the third step, I chose to add an extra step between the PDF and the CSV — a JSON report. Once we see everything works, we will remove the use of that file.

Lastly, we’ll translate that JSON data into a CSV file. That CSV can then be easily converted to Google Sheets (just click "open with") or to Excel (instructions can be found here).

That’s an easy nice plan, but you know how it goes — challenges are discovered along the way… Can you guess where things might get complex?

Before we start — an important note: KEEP YOUR PAYSLIPS PRIVATE! If you upload your project to GitHub — be sure not to share those personal details! You can use .gitignore for this:

/payslips_pdf
pdf_rows.txt
report.json
report.csv

Shall we begin?

Read PDF File

We’ll start by reading the PDF and printing all the rows. That way we will know what appears in each row. This only needs to be done once (while a report will probably be created once a month or once a year), and it’s not part of the report -so we will create that in a separate file.

Start by creating a new Python file (I called mine pdf_to_txt.py) and write a function that reads the pdf and prints the result into a .txt file:

This will work, but there are 3 changes to make it more user-friendly:

Get the file path as a command line argument, so the user can run it without touching the code.
Add some error-catching with instructions in case the user runs the command wrong. (I added a warning color but you don’t have to)
We will read PDF files in the main script as well, so moving this function there is better.

Give it a go! Try running the command with the right and the wrong arguments :)

When you run py .\pdf_to_text.py you will see a new file in the project directory called pdf_rows.txt.

(Don’t worry, this is not my data — it is based on the example payslip from the image below)

Process The PDF Data

Now that we know the structure in which the PDF is read — we can fish the desired values. In my case — here is the information I was interested in:

Notice there is data within the table (categories that might vary every month) and data outside the table.

Data outside the table:

Pay Period — Can be found on row 19

Gross Pay — This one was tricky to find a rule for because it appears after the payments list and doesn’t have the title “Gross Pay”.

As mentioned earlier, payments and deductions can vary, and not every month is the same. Therefore, gross pay might appear in a different row in different months.

I did notice it appears right after the employee name — so that is what I used. Start by adding it hard-coded, and later we will get it externally.

Nett Pay: This one is easy — it appears in line 17.

I gathered those out-of-table values into a function:

Data inside the table

Payment and Deduction Details: This is the juicy part! We’ll start by cutting the rows array to save a few milliseconds in the for loop coming up. Then, I needed to differentiate between list items and other rows.

I’ve noticed that within the whole file, list items are the only ones that match this rule: Start with an alphabetic character and end with a numeric character and contain a space (the last condition is to filter out wrong rows in my payslip, you might not need that).

Now that we have the group of lines, let’s process them to save in the JSON object. At this point, we don’t mind if it’s a payment or a deduction.

For example, we’ll look at the pension item:

PENSION     G   150.00   587.49

I don’t care about the balance (the number on the right), but I do care about the code (G means it’s deducted from the Gross pay — before taxes — and N means it is deducted from the Nett pay — after taxes). So ideally, we’ll have json_obj["Pension (G)"]=150.00.

We’ll use the spaces to split the line. It’s good there are duplicate spaces — that way we can differentiate between space splitting between a couple of words and space splitting between a couple of fields.

The description:

We will find the first double-space and split by it.

The code:

The amount of spaces is dependent on the length of the description, so we can’t know in advance how many are there — that’s why I’ll use lstrip() as well. Now the rest of the line starts with a non-space character.

Not all list items have a code, so we want to check if the line starts with a code or a digit. If it’s a code — I wrap it in () (including a space before the opening parenthesis) , and attach it to the description string. and if not — add nothing.

The amount:

If there was code — we’ll have more spaces to strip. If not, our line might contain two amounts: The monthly and the balance.

There are 4 cases I’ve noticed:

# SALARY      Y   1234.67
# AVC PRCTG   G   1234.00   2345.00
# RSU TAX WH  N   -1234.00  -2345.56
# RSU TAX WH  N             -2345.56

After extracting the category and code, we are left with:

# 1234.67
# 1234.00  2345.00 (we want the amount on the left)
# -1234.00  -2345.56 (we want the amount on the left)
# -2345.56 (we want the amount on the left, which is none)

To cover cases 2–3, we’ll find the index of the spaces separating the amounts and cut the tail. It also works for the first case, where there is no space (aka no tail).

To cover case 4, I’m relying on the difference between two types of categories with a single amount in the row: The first one is like the salary — where we want to save the amount, and the second type is like the tax withholds — where we want to ignore it. The difference is that only deductions keep track of the annual balance in the table — so I am checking for -.

All together, that’s how it looks like:

Write to JSON File

This is not a mandatory step — we can work with a JSON object without exporting the values. I prefer seeing what it looks like, at least for the coding stage.

At this point, you can run py ./main.py and see a new report.json pop up.

Scale to Multiple PDF Files

The only reason this step is getting a dedicated section is because wrapping the pdf_to_dict in a for loop reveals an unpleasant surprise. To demonstrate it, I created a function called iterate_over_pdfs():

This is happening because the list of files is sorted by alphabetic order, so 10 appears before 2. Having the report entries in chronological order can be considered crucial, and not just a nice-to-have feature. Therefore, we need to fix it!

Originally, I thought I’d have to rename the files (Payslip1.pdf -> Payslip01.pdf), but there is a better solution: Sort the list by length. That way, 10 will appear after 2. Then we can also tackle the b'<STRING>' structure by decoding. Lastly, after running the script on Mac, I discovered the directory includes hidden files – so we need to filter them out.

After adjusting the main function to loop over the new list we created – the report.json file will contain an array of payslips (give it a try!).

Create the CSV Report

Because the items in payments and deductions might vary from payslip to payslip, this section is more than just direct translation. CSV is a relational dataset, which means we need to know in advance all the categories in payments and deductions and keep the entry empty for a payslip where it doesn’t exist. JSON, on the other hand, is non-relational and each entry specifies its keys.

With that in mind, the first step in our CSV report is to collect the categories. All the categories.

Collect the categories:

Now, at first glance, you might think to use Set for that — because we want all categories to appear only once. I’ve tried that. The problem with this is that sets are unlisted, and I find it important to match the order of items that appear in the original payslips. When using lists, don’t forget to check if the item exists in the list before appending it:

Now that we have this figured out: Remember earlier when we said we don’t care about which item list is a payment and which is a deduction? Well, we do care now! We don’t have to separate, but I’d expect a payslip report to have all payments on the right and all deductions on the left, not mixed.

Although each payslip might have different list items — some will always exist (because you’ll always pay your taxes ;) ). We can use this to our benefit — and flag PAYE as the start of the deductions! (I'm pretty sure PAYE is only in Ireland, so you'll need to change it to match your payslip)

Finally, I return a single list, because there is no use in separating the payments from the deductions — the split was to assure payments will appear on the right and deductions will appear on the left.

Populate the CSV table:

Now that we have the categories, we can start populating the CSV table:

Each payslip will be a row, and each row will have the fields in a specific order split by a comma. I find it easier to organize the fields in a list, and then join them. Fields that appear in the categories but not in the payslip — will remain empty:

Lastly, we will write to the CSV file:

After this, you’ll have a nice CSV report with all your payslips!

You can make it easier to read by downloading the VS extension RainbowCSV (or any parallel of another IDE)

Remove the use of the .json file

Once we know things work, we don’t need to write to the JSON file: Remove the section of # save dict to json file.

Add a success message

It’s nice to notify the user that the script is done successfully. for that, we will need to expand print_warning() to print_in_color(), and add the option for green color:

Don’t forget to update the call in pdf_to_txt.py as well!

Get the employee’s name as an argument

Once you have your script ready — you’ll want to share it with your colleagues and friends! For a nice user experience, we’ll export the employee name to come from the command line, rather than ask them to open the code.

Read argument

We’ll start with the happy path — assuming the user entered the employee name — and add code that uses it:

In pdf_to_dict(), instead of hard coding EMPLOYEE_NAME = "IFAT NEUMANN", we'll read it from the arguments: employee_name = sys.argv[1]. Don't forget to import sys!

Now we’ll think of other scenarios:

No employee name was given

What if the user did not enter any employee name? We’d wanna catch it as soon as possible, and notify them!

Therefore, we’ll add a check in the first line of the main function. Now, the intuition is to initialize the employee_name variable there - but this will cause bubbling function properties until it reaches the function that uses this variable - and I don't find it a very clean approach.

Lastly, I’ll just try accessing this field — and catch if it’s not there:

Note that adding exceptions means the print_warning() function moves to main.py. Otherwise, you’ll get an error:

ImportError: cannot import name 'print_warning' from partially initialized 
module 'pdf_to_text' (most likely due to a circular import)

Employee name without quotes

We are asking the user to wrap the name with quotes because command line arguments are split by spaces. The only argument we expect is the employee name, so if there is another argument — we know they did not use quotes and we can notify them:

You can skip the quotation requirements and loop over arguments, collecting all the parts of the user name — but I find this approach adds unnecessary complexity.

The employee’s name doesn’t appear on the payslip

We won’t be able to find gross pay if we don’t have the employee’s name as it appears on the payslip.

The earlier spot to identify the mismatch is when we read the pdf. Therefore — well add the check at the beginning of the pdf_to_dict() function:

Note that I moved the employee_name variable to this function because it’s easier to read (rather than if sys.argv[1] not in rows). After that, I pass it to get_fixed_values().

Lastly, we catch the error back in the main function:

To wrap it all up — don’t forget to update your README file with the new instructions.

The full script

Here is the full code of main.py:

And here is pdf_to_txt.py (note it’s calling functions from main.py):

Hope you enjoyed this one! Interested in another weekend coding project? Check out my automated email-sending blog post!


https://cupofcode.blog/

Blogging is my hobby, so I happily spend time and money on it. If you enjoyed this blog post, putting 1 euro in my tipping jar will let me know :) Thank you for your support!

Spotify API: How To Create a Data Set of Songs

Cup of Code — Mon, 18 Mar 2024 21:33:55 +0000

A Fun Tutorial using Python, JSON, and Spotify API!

You might find it more comfortable to read this in my website.

As you might know, I’ve recently written a Taylor Swift-themed Wordle game in React. To focus on the game logic, I downloaded a Taylor Swift data set from the Internet. Sounds good, right?

I integrated this data set into the project and happily continued coding. Suddenly, I realized that the song I was currently listening to wasn’t on the list! This means that the list was too old (or that Tay-Tay is releasing new albums at a rapid rate). Right then and there I decided that I could trust no one — and should create my own data set!

Not only that, but I saw here a good opportunity to get hands-on experience with Spotify API :D.

So, what is on the agenda today?

Requirements
What data do I need for my Wordle game?
Theoretical
Because we came here to learn!
Connecting to Spotify API + Performing API calls
And important things to notice!
Data Processing
With all due respect to Taylor’s Version, of course.
Modifications for future use
How to update the data set without running everything again.

Shall we begin?

Requirements

I wrote a Wordle game themed with Taylor Swift song titles. To accommodate that, I am interested in a JSON file containing information about all Taylor Swift songs, with these restrictions:

No duplicates: This is a requirement that the queen made very challenging with her album re-recordings.
Song titles no longer than 13 characters: This is so I have a reasonable amount of characters to present in a word-guessing game. Six tries are probably not enough to guess “We Are Never Ever Getting Back Together”, not to mention the width of a mobile view!
Every object needs to have a split array: This array would indicate after which indexes there were spaces, in a space-less string of the song title. For example, “shakeitoff”-> [4,6] because:

Shake it off
01234 56 789
Every object needs to have an album_title property, to let the user of the game know where they can find this song.
The list has to be shuffled because we don’t want a guessing game such as Wordle to have a pattern in the solutions!

Why JSON and not CSV?

JSON is JavaScript Object Notation, and CSV stands for Comma Separated Values. Both are great ways to save data, and each has its advantages.

In the screenshot above you can see dataset examples I made to show you the difference between the formats’ syntax. Note that CSV, as the name implies, is sensitive to commas, so you’ll need to wrap every comma-containing string in your data with quotation marks (the split array is a good example of that). On the other hand, everything is wrapped in quotation marks in JSON files.

In my case, the data is pretty simple (string, string, and a short array) and both formats will do the job. I chose JSON because I wanted this data set file to be reachable from outside the game. That way, when I update the file I don’t need to redeploy the game. JSON is the recommended way for communication between web applications and APIs.

Theoretical

We’ll start with the new terms we touch on in this blog post.

If you read the Spotify for developers documentation, you’ll see that “Spotify implements the OAuth 2.0 authorization framework”. What is OAuth 2.0?

OAuth 2.0

OAuth 2.0, which stands for “Open Authorization”, is a standard designed to allow a website or application to access resources hosted by other web apps on behalf of a user. (source)

A nice analogy for this is a coffee shop restroom. In some restaurants, there is a code on the bathroom door you can get only if you buy with the cashier. The code is printed on the receipt and is replaced daily. The restaurant does this to authorize only the paying customers to use the restroom.

This approach is similar to OAuth 2.0. The customer has the access they need, but it’s the owner of the service who withdraws the access every day (and changes the code). If, for example, the access to the restroom would have been done with a key, it’s the key holder that has the access. With a key to control access, customers come to the cashier and ask for the key, use it, and then return the key. A customer with a key has access that can not be revoked (not as quickly and easily as a code).

When looking at Spotify, as you will soon see, we won’t be getting any permanent password to access the data. Instead, we receive an access token that gives us temporary access.

Access Token

In general, there are two types of information you can pull from Spotify: General data (artists, albums, songs) and user-specific data (liked songs, “for you” playlists, etc). We want the first category — which is what Spotify calls Client credentials workflow.

Our application will be requesting the Spotify accounts service for an access token. With that access token, we will make requests to Spotify Web API. Note that there are two players here: Spotify accounts service is the coffee shop cashier and Spotify Web API is the restroom ;)

The response from the Spotify accounts service is returned in this format:

{
   "access_token": "NgCXRKc...MzYjw",
   "token_type": "bearer",
   "expires_in": 3600
}

This means that we can use this token in the next hour, with the bearer token type. What is this bearer?

Bearer Authentication

Bearer authentication (also called token authentication) is an HTTP authentication scheme that involves security tokens called bearer tokens.
The name “Bearer authentication” can be understood as “give access to the bearer of this token”. The client must send this token in the Authorization header when making requests to protected resources. (source)

This means that when we build our request to Spotify API, we will need to add a header of authorization, using the word “Bearer” and attach the access token: {“Authorization”: “Bearer “ + access_token}.

Let’s see it in the code!

Connecting to Spotify API

Getting The Access Token

You might’ve noticed in the Client credentials workflow above that to get an access token, we need client_id and client_secret values. The client in this case is a Spotify project. Therefore, the first thing you need is to create a Spotify project/application in developer.spotify.com:

Log into your account. It is the same as your “regular” Spotify account.
Go to the dashboard and Create a new project. This is the configuration I used:

Click Save.
Under Settings, You will find your Client ID and Client Secret. Grab those and open the VS code! (or another IDE of your choice)

Using the Access Token

We will start by saving the client variables in the ENV.env file. I suggest you also create example.env and .gitignore files now so you won’t accidentally upload your client secret to GitHub!

Let’s go back to the IDE. We’ll start by importing the environment variables:

Now we need to create the request in a very specific structure. We’ll do it exactly as the documentation says:

- Send a POST request to the /api/token endpoint.
- grant_type: Set it to client_credentials.
- Authorization: Basic <base64 encoded *client_id**:**client_secret*>,
- Content-Type: set to application/x-www-form-urlencoded.

On line 18 we are sending a Post request to the /api/token endpoint. We can see this endpoint in the code on line 12.
We set grant_type to client_credentials on line 17.
We create the header with authorization and content type on lines 13–16. To create string, we first create the string client_id:client_secret on line 10, then we encode it in base 64 on line 11.
Lastly, we decode the JSON data on line 19.

This looks good but it’s not yet working.

If you run this (with py ./main.py in the terminal), you’ll get the following error:

TypeError: a bytes-like object is required, not ‘str’. This error arises while trying to encode to base64 (base64.b64encode(client_id_and_secret)). This means we need to encode client_id_and_secret to be in bytes, not in string. The solution is to add a “translation“ to bytes between lines 10–11 above:

client_id_and_secret = client_id + ":" + client_secret
client_id_and_secret_bytes = client_id_and_secret.encode("utf-8")
client_id_and_secret_base64 = base64.b64encode(client_id_and_secret_bytes)

After doing that, we reach the next error: TypeError: can only concatenate str (not “bytes”) to str”. When we try to compose the authorization string (“Basic “ + client_id_and_secret_base64). Now the problem is that we don’t have a string!
There is an easy fix for that: Wrap our client_id_and_secret_base64 with str(...).

Cool, now it runs! But hey, the result isn’t the JSON we expected: {‘error’: ‘invalid_client’}. Another bug! Looks like there is a problem with the client ID and secret we send. I test it by printing: print(“Basic “ + str(client_id_and_secret_base64)) and I see Basic b’MjY5N…’ . Wait, where did this b'..' came from?

Bytes literals are always prefixed with ‘b’ or ‘B’; they produce an instance of the bytes type instead of the str type. (source)

This means we are stringifying the object, but not accurately enough (‘A’ != b’A’). For the Authorization header, we want a UTF-8 string — so we need to specify that: str(...,"utf-8")

Now it works! Note that we need to extract the access_token from the JSON (line 20 below):

Performing API Calls

Now that we have the access token, we can make API calls! But which APIs do we need?

In my project, I want all of Taylor Swift’s songs. After looking at the documentation, it looks like my script should do the following calls:

Search for Tay-tay’s artist ID
With that ID, get artist’s albums
For each album, get the album’s tracks

Cool! I’m ready, are you ready?

GET Artist ID

I played with the “try it” tool on the website and got this:

Let’s implement something similar in our code:

Looks like we need two variables to create a GET request: URL and header. We saw those two in the POST request above (result = post(url, headers=headers, data=data))

We’ll write the URL similar to the one in the Spotify tool: https://api.spotify.com/v1/search?q=TaylorSwift&type=artist
The header has bearer authorization, using the access token from the previous function: {“Authorization”: “Bearer “ + token}
Lastly, we need to pass the token as a parameter. I decided to pass the artist's name as well, to keep the function more generic: What if one day I’ll get sick of Taylor Swift? JOKING! I will never.

Our function will look like this:

Let’s look at the result we got:

Notice there are 13 Spotify artists with “Taylor Swift” in their name (total: 13 on line 10), but there is only one Queen T! Therefore, limiting the result items to be of size 1 will get us the ID we need:

Add &limit=1 to the end of the URL.
get to the id property by using json_result[“artists”][“items”][0][“id”].

Now that we have the ID, we don’t need to perform this call anymore. Our main code will look like this:

...
# main
TAYLOR_SWIFT_ARTIST_ID = "06HL4z0CvFAxyc27GXpf02"

access_token = get_access_token('ENV.env')
artist_id = TAYLOR_SWIFT_ARTIST_ID # get_artist_id(access_token, "Taylor Swift")

GET Artist Albums

Now that we understand how an API call works, let’s look at the next one because it has a complexity we didn’t need to think about before: There is only one Taylor Swift, but there are multiple albums!

We’ll start with a simple scenario:

When you run this code, you’ll see the result you get is pretty big for the human eye and includes a lot of information we don’t need. As much as it is tempting to do processing here, let’s keep this function true to its name — getting the albums from Spotify. Therefore, we will just return return json_result[“items”].

The next step is to create a dictionary of album names to album IDs:

The reason I’m grabbing both name and id is because the name is used in my data set, and the id is used later to grab the album tracks!

Running this code shows that this small amount of processing resulted in a much smaller JSON:

Wait. This list is too small — it does not contain all the albums! Any guess why is that? Let’s look at the fields of the response we got:

We sent a short and sweet URL (https://api.spotify.com/v1/artists/{artist_id}/albums), and on lines 2–3 above it looks like the URL has some default fields that affect our response:
?include_groups=album,single,compilation,appears_on&offset=0&limit=20

According to the documentation, we can filter the response to include only studio albums: .../albums?include_groups=album. That will drastically lower that 'total':387 on line 10!
The limit is the maximum number of items to return (line 5). The default is 20, the minimum is 1 and the maximum is 50. Let’s do 50, shall we?
The offset is the index of the first item to return (line 8). The default is 0, which means we start with the first item. We don’t need to touch this one.

With all that new information, our new URL will be: https://api.spotify.com/v1/artists/{artist_id}/albums?limit=50&include_groups=album, and the response coming back is:

Therefore, the output of create_album_name_to_id_dict() is:

GET Album Tracks

Like with the albums, we’ll start with a simple call:

Given an album id, we pull the tracks. In a separate function we’ll create a song-to-album dictionary:

We will loop over these functions for each album in the album_name_to_id dictionary and the result will be:

{
    "Welcome To New York (Taylor's Version)": '1989',
    ...
    'Teardrops on My Guitar - Pop Version': 'Taylor Swift'
}

(332 songs, the full list can be found here)

This wraps up the section of the API calls. Next, we will start processing the data, which might require us to come back for some adjustments.

Before we do that, let’s make sure we are on the same page regarding main.py:

Data processing

Remove Special characters

If you peek at the songs we gathered, you will see some song titles that are not Wordle-ready. Wordle solutions should not have special characters like commas, parentheses, single quotation marks, and digits.

Because of that, we need to create a function that removes those characters. Hey, do you know what would be a fun way to do that? Test-driven development!

We’ll start by exporting song_to_album into a file.

...
    song_to_album.update(songs_to_one_album)
data_json_format = json.dumps(song_to_album)
file = open("song_to_album.json", "w")
file.writelines(data_json_format)
file.close()

Then, we’ll create a name-processing function and run the names through that function. After that, we’ll loop over the list with another function checking if the names are alphabetic, and print the remaining “bad ones”.

That way, we’ll adjust our name-processing function until all the names are valid!

On lines 4–6, we open the file and save the JSON object in a variable.
On line 9 we loop over the songs (the keys of the dictionary) and on line 10 we call the processing function. We start with this function empty.
On line 11 we check if the song name without spaces is alphabetic. If not, we print that name.

This is our starting point:

wordlestruck-spotify-api> py -m src.test_process_name_func
Welcome To New York (Taylor's Version)
...
"Slut!" (Taylor's Version) (From The Vault)
...
Is It Over Now? (Taylor's Version) (From The Vault)
Anti-Hero
You're On Your Own, Kid
Question...?
...
the lakes - bonus track
Miss Americana & The Heartbreak Prince
Soon You’ll Get Better
ME!
...

---
 190 songs

Looks like we have 190 song titles to fix! Don’t worry, this amount will drop pretty quickly.

At this point, you might wonder why we are “wasting time” on all song titles, instead of working on the ones that are no longer than 13 characters. This is because we have to “clean” the song titles to know which are in the correct length.

Let’s start writing the function!

With all due respect to Taylor’s version, we’ll start by trimming the parenthesis. It doesn’t matter if it’s round or square — we don’t need it in our Wordle solution. I created a dedicated function for this part:

We are down to 95 song titles to fix!

The next thing that pops up in the output is the suffix after the “-“ character. Let’s add code for that:

def process_name(name):
  ...
  index_of_hyphen = name.find(" - ")
  if(index_of_hyphen > 0):
    name = name[:index_of_hyphen]

Now we are down to 43 “bad” songs! At this point you might notice some duplicates — this is ok, we have a dedicated section for that later.

The next step is to start using the replace function! The first replacement will be of the single quote character, on all its varies:

name = name.replace("'", "").replace("’", "").replace("‘", "")

This brought us down to 24 “bad” songs!

The next character to remove is “…” but note that we can’t just remove it because it might function as a separation between words (Come Back…Be Here). So, we replace it with white space. This means the other songs in this group will have a trailing whitespace, which we will resolve with .strip().

def process_name(name):
  ...
  name = name.replace("...", " ")
  return name.strip()

Now I’ll add the replacement for the characters ?/!/,/. which will resolve for us another 9 songs! NOTE that this should be added after the ... trimming.
Next, we will replace 1 with one, 22 with twenty two and & with and.

wordlestruck-spotify-api> py -m src.test_process_name_func
"Slut"
Anti-Hero
Back To December/Apologize/Youre Not Sorry

3 songs

The last thing to do is remove the double quotes and the -. Note the difference between - in a song name like Anti-Hero and the one trailing a song name (the lakes — bonus track). The trailing one is padded by whitespaces, and we do it before!

I’m ignoring the song title Back To December/Apologize/Youre Not Sorry because it is definitely longer than 13 characters.

That’s it! This is our final function:

At this point, we have a beautiful name-processing function, but we haven’t called it yet!

We’ll go to where we create the song_to_album dictionary, and add 2 things: First, process the name. Then, if the name is no more than 13 characters — add it to the dictionary!

Make sure you use the processed name as the key!

The new song_to_album list can be found here.

{
  "Blank Space": "1989",
  ...
  "New Romantics": "1989 (Deluxe)",
  "Slut": "1989 (Taylor's Version)",
  ...
}

I like that there is a difference between the original album, the deluxe version, and the new songs from the vault. If you don’t like it, you are now fully capable of trimming the album titles :)

It’s important to mention that the list isn’t final! But it’s good enough for this stage of the project. After we finish coding the simple version, we’ll come back and fine-tune the code. Don’t worry, you’ll leave this blog post with a full end-to-end project!

The Split Array

For the wordle game, each song should have 4 fields: id, song_title, album_title, and split array. We already have the song title and album title, so the next step is adding the split!

Let’s start with a reminder of what we are trying to achieve:

The split array would indicate after which indexes there were spaces, in a space-less string of the song title. For example, “shakeitoff”-> [4,6] because:
Shake it off
01234 56 789

To do that, we will find the first occurrence of space, add it to the split array, remove it, and find the next occurrence, until there are no more spaces.

Note that I’m removing the space that I “checked in” so that we won’t stop on it in the next find. Also, you need to specify you want to replace only 1 occurrence: song_name.replace(" ","",1)

Now, we need to create objects containing the properties we have so far:

I chose to stringify the split array because it would be easier to read later.

Lastly, we call the function in the main code:

...
arr_with_split = create_arr_with_split(song_to_album)

Creating The Data Set

For this to be a legit data set for the Wordle game, we are missing three actions: Shuffle the list, add ids, and save to a file

We use an additional variable because shuffle() is a function that affects the input object. The replace() function, for example, did not. This is why we needed to save the modified string: str = str.replace(...)
The use of the function list(..) is to make the copy deep. Otherwise, shuffled would’ve pointed at the same array, and using shuffle(...) would’ve mixed our original list. It won’t be a problem now, but for future functionality, we need to keep the original list as it is.
In the for loop we use enumerate(..) to get access to the loop index (for i..).
Fun fact: the lists are independent (arr_with_split is not shuffled and shuffled is shuffled), but the objects they contain are shared — the new id property we added for shuffled appear in arr_with_split objects as well!

The last thing left to do is write our data set to a JSON file. I created functions for writing and reading from files. It’s very useful for testing!

That’s it! This is our main.py:

This was intense! We deserve a break!

While waiting for my API calls to populate my dictionaries, I thought it would be cool to create a progress bar. I always wondered how the line gets rewritten in the terminal — so this was a good opportunity!

Feature: Progress Bar

Here is what it would look like:

We’ll start with the printing progress function:

On lines 2–3, I define the characters I’ll use to represent full and empty parts of the bar.
Line 4 is building the bar: The length of the bar will always be len_range, and the amount of █ and - are determined by i.
printEnd needs to be \r for the line to override itself in updates, but I added the option \n to wrap it up (when i==len_range). This is because otherwise the next line I print after gets swallowed by the progress bar.
Lastly, the print. I started with /r but that’s a personal preference. The main thing to notice is the structure of the string: 'Progress: |%s| %s/%s albums complete' % (bar, i, len_range)

Note that if your terminal window is not wide enough to contain the print in one line — the override won’t work!

Now let’s call it in main:

Note that in line 3 we start the progress bar empty, and inside the for loop (line 9) we call the function with i+1, because the album on index 2 is the 3rd album.

Also, to reach i in the for loop, we iterate over enumerate(album_name_to_id*)*.

Modifications For Future Use

Tay Tay announced she is releasing a new Album on April 19th, and we need to prepare our code for that!

After many trials and errors, I’ve decided to create a new data set on every update. This does not mean I will make all the API calls to Spotify and string processing on every update! You can, it will lead to the same result, but it’s not optimized.

For the updating feature, I will be creating 4 files in the code:

1_album_to_name_id.json
2_song_to_album.json
3_arr_with_split.json
4_dataset.json

After every pull, we save the relevant information, ensuring we execute only the necessary parts. Let’s see it in the code!

Adding External Files

The easy part is to add the write functions:

Note that the file path should be relative to the terminal’s directory! My terminal is open on the project’s main folder so the writing file paths start with src/ !

Soon we will be adding read functionality for those files. It is too easy to update the write file path and forget the read path. Therefore, we will save those as variables:

Now we need to add read-from-files and ensure it works with empty files. What exactly do we want? It’s easier to figure out when we focus on a small group of albums, so we’ll look at the 1989 albums.

The 1989 Test

If you look at the list of albums, you’ll see 1989 (Taylor’s version) is the most recent album. I considered the chronological order because I thought of making the actual API call. Then I realized I could just mock it. Anyway, 1989 it is!

We’ll start by modifying album_to_name_id to contain only the original 1989 albums:

# album_name_to_id = create_album_name_to_id_dict(json_result_album_items)
album_name_to_id = {
  "1989 (Deluxe)": "1yGbNOtRIgdIiGHOEBaZWf",
  "1989": "5fy0X0JmZRZnVa2UEicIOl"
}

Now we’ll execute the program and let it populate our files.

So far so good. Now, let’s mock an update call:

# album_name_to_id = create_album_name_to_id_dict(json_result_album_items)
album_name_to_id = {
  "1989 (Taylor's Version) [Deluxe]": "1o59UpKw81iHR0HPiSkJR0",
  "1989 (Taylor's Version)": "64LU4c1nfjz1t4VnGhagcg",
}

And think about what we want the program to do.

1_album_name_to_id.json should contain a total of 4 albums. We want to make sure we process only the new albums.
2_song_to_album.json should contain an additional 3 songs.
3_arr_with_split.json should also contain an additional 3 songs, but we want to make sure we process only those!
4_dataset.json will be overridden so no extra action is required from us.

Updating 1_album_name_to_id.json

With the code as it is now, the new album dictionary will override the previous one:

...
json_result_album_items = get_albums(access_token, artist_id, "album")
album_name_to_id = create_album_name_to_id_dict(json_result_album_items)

write_to_json_file(ALBUM_NAME_TO_ID_FILE_PATH, album_name_to_id)
...

So let’s execute the code as usual, but read the previous dictionary and update the variable before writing to the file:

On line 7 we merge the two album_name_to_id dictionaries. Note that we updated the new album with the old album so that the list will be LIFO = last in first out. This is critical for assigning the original album to each song.
On line 9 we update the file. Note that this line was originally before the for loop on line 3 but now it is after!
On line 6 we save the previous object of albums. This requires modifying the read function to accommodate the first execution of the program (when the file doesn’t exist):

Note that with these changes, we only execute the songs section on the new albums!

Updating 2_song_to_album.json

Look at the code as it is now and consider that we want to keep the original album names (and have "Blank Space": "1989" instead of "Blank Space": "1989 (Taylor’s Version)"):

song_to_album = {}
for album_name in album_name_to_id:
  ...
write_to_json_file(SONG_TO_ALBUM_FILE_PATH, song_to_album)

There are 2 ways to achieve this:

Let the old dictionary override the new dictionary values:
1. Extract the new songs. This means the second dictionary will only have new keys, and add those to the new list:
  
  We’ll go with option number 2 because it will be useful in the next section!

Updating 3_arr_with_split.json

As a reminder, this is the code as it is now:

...
arr_with_split = create_arr_with_split(song_to_album)

write_to_json_file(ARR_WITH_SPLIT_FILE_PATH, arr_with_split)
shuffled = list(arr_with_split)
...

In this case, the logic will be similar. Just note that this is an array we are updating, so we use the .extend() function instead of .update().

I think it can be confusing to call the updated list “prev”, so I created a new variable:

updated_arr_with_split = list(prev_arr_with_split) # new var for readablity
write_to_json_file(ARR_WITH_SPLIT_FILE_PATH, updated_arr_with_split)

For this to work, we need to modify the read function to be able to return an empty array! The function as it is now returns only an empty dictionary if the file doesn’t exist. This can be achieved by a small tweak:

def from_file_to_json_obj(file_path, array = False):
    if Path(file_path).exists():
        ...
    return [] if array else {}

This means that when we read from the array file, we add the array flag:

prev_arr_with_split = from_file_to_json_obj(ARR_WITH_SPLIT_FILE_PATH, array=True)

Now we’ll see it in action!

Here is the updated main file:

There are 3 tests to perform:

album_name_to_id = {
"1989 (Deluxe)": "1yGbNOtRIgdIiGHOEBaZWf",
"1989": "5fy0X0JmZRZnVa2UEicIOl"
}
album_name_to_id = {
"1989 (Taylor’s Version) [Deluxe]": "1o59UpKw81iHR0HPiSkJR0",
"1989 (Taylor’s Version)": “64LU4c1nfjz1t4VnGhagcg"
}
album_name_to_id = {
"1989 (Taylor’s Version) [Deluxe]": "1o59UpKw81iHR0HPiSkJR0",
"1989 (Taylor’s Version)": “64LU4c1nfjz1t4VnGhagcg",
"1989 (Deluxe)": "1yGbNOtRIgdIiGHOEBaZWf",
"1989": "5fy0X0JmZRZnVa2UEicIOl"
}

We want to start with an empty files directory and run test #1. Then, with those new files, run test #2 and see if the files look as we expected. After that, we want to move those files aside and run test #3 on a clean slate. Lastly, compare those files: This is our one pull vs update test.

Oops

The same thing happened to me again! I was working on this project, listening to Tay Tay (as one does), and I suddenly realized “You’re losing me”, which I’ve been listening to on repeat 1, is not in the data set!

When I started working on this project, I decided to include only the studio albums because the singles list was full of duplicates (Lavander Haze, Anti-Hero, …). Turns out I was too quick to dismiss that album type. Back to the API call!

Pulling the single songs

There are two ways to gather the single albums along with the studio albums. One way is to make a single API call with include_groups=album,single. The other way is to make two separate API calls, one for include_groups=album and one for include_groups=single.

I would’ve chosen the first option if the response would’ve brought the albums in chronological order. As it is now, the response contains the studio albums in chronological order, and then the singles in chronological order. I’ve tried sorting them by date but things got messy quickly.

Therefore, I’ll be making two API calls. I will treat the singles API call as an update to the dataset. That way, all the duplicates won’t mess up the song dictionary, and the songs that are only singles will be added.

Nice to meet you, Pagination

If you make the API call with include_groups=single, you will see there is a total of 64 albums. This is above the maximum limit of items in a response — which means we need to paginate!

I didn’t address pagination prior, because we used the API call with the include_groups=albums parameter and the total was 26. This is half of the maximum amount of items coming back in the call (limit=50). Therefore I made the executive decision not to paginate on those.

So, let’s talk about pagination!

If you look at the response JSONs I shared earlier (for the call https://api.spotify.com/v1/artists/06HL4z0CvFAxyc27GXpf02/albums for example), you will see that one of the values in the response is next. This contains the URL to the next page of items (or None if there isn‘t one).

This URL means that not all the data we asked for is coming in one response, and to get the next page of the data, we need to make another API call. The difference between those calls is the offset, which is the first item index appearing in the response.

To make it clear, we’ll change our limit to be 5. With a total of 26 albums, we will need a total of 6 API calls to gather all the albums!

Now, we want at least one call to get albums, and then keep calling until next==None. This is a perfect candidate for a do-while loop, which unfortunately does not exist in Python. I saw online you can create a version of it with While True:...break but it itches me to use While True. And then I had an epiphany: You know what will be fun? RECURSION :D Have you done those since your last job interview?

The core of the recursion is to make an API call to the given URL, gather album names, and return that list after we concat that to the list returning from the recursive call.

Our stopping condition will be if url == None.

Let’s see it in the code!

Lastly, the rest of our program doesn’t need to know what is happening under the hood, so we will wrap the call to the function and gather the parameters it needs:

Note that I’m adding a variable for album type, to reuse this function for the two API calls!

This is a good place to add limit as an optional parameter because when we update our dataset we don’t need to pull all the albums, we can take a quick look in Spotify and eyeball estimate the amount.

If we didn’t paginate, limit would’ve worked by itself: One function call — one API call — one response. With recursion, limit is just telling how thin we want the slices. Therefore, we need to add a flag to indicate to our function that we don’t want the other pages!

This is how I implemented this feature:

On line 9, I added to the function 2 parameters: limit and an optional offset. The offset is for testing.
On line 14 I created a boolean that is true if the limit is smaller than 50 (which is my program’s default limit).
In the recursion, on line 4 I check if this is an update call. If it is: I return the items and exit. if not, we continue to the next iteration.

Lastly, we integrate this in the main code:

I added an optional argument for the limit, which I checked for on line 1.
If I want to pull the latest 5 albums, I run py -m src.main 5.
If I need an initial pull, I don’t add anything, and the default is set to 50.
On line 5 I pass that variable to get_albums().

Test your code:

Add an offset to get_albums(..., limit, offset=5), run py -m src.main, remove the offset get_albums(..., limit) and then run py -m src.main 5 - This is a dataset update!

Compare the lists coming back from the update to the lists returning from a single pull. If you are pleased with the result — we can move on!

Merging Singles in The Code

As I mentioned, we will treat this list like an update, and only add the unique keys.

A good way to achieve that is by looping over the album type:

On line 3 we add a for loop. The loop block contains all the code until the shuffle.
Now that we have two album_name_to_id dictionaries. Therefore, we insert the album_name_to_id_file_path variable inside the loop (line 4).

Skip Albums

We have just a tiny issue to solve before closing this project. As a Swifty, when I look at the songs dictionary I see some strange pairings:

{
  "Better Man": "reputation Stadium Tour Surprise Song Playlist",
  "Love Story": "Live From Clear Channel Stripped 2008",
  "Santa Baby": "The Taylor Swift Holiday Collection",
  "September": "Spotify Singles",
  "I Want You Back": "Speak Now World Tour Live",
  ...
}

Better Man should be linked to Red (Taylor’s Version), Love Story is from the Fearless album, and the rest are covers, which might sound nice but are useless in a word game.

These oopsies don’t mean our code is wrong, but some albums will inevitably need to be skipped. In this list belong all the albums that should be ignored completely.

My list is:

How do we add this feature to the code?

We start by reading the file and saving it into a variable. Then, instead of looping over all the albums received from the API call, we filter out the albums that appear in the skip dictionary. Let’s see it in the code:

Note that we still save the full albums dictionary in our files, to keep track of the albums we’ve processed.

After this change, the song dictionary is correct:

{

  "Better Man": "Red (Taylor's Version)",

  "Love Story": "Fearless (International Version)",

  "Santa Baby": "The Taylor Swift Holiday Collection",

  "September": "Spotify Singles",

  "I Want You Back": "Speak Now World Tour Live",

  ...

}

Well, almost.

Tay Tay decided to rerecord SuperStar but not keep the original name — which for our dictionary means “new song!”. I can solve it by unifying the dictionary keys — all upper-case or all lower-case.

I decided not to do so because I wanted to preserve the original song titles. Therefore, my game will have 2 occurrences of the song Superstar. I can’t recall the last 5 solution words in Wordle, so hopefully my users won’t notice either :D

Now we are done!

What a journey! For a second there, I thought this blog post would never end!

Today we learned about OAuth2.0, bearer authentication, and Spotify API. We created API calls, processed data, and created a beautiful clean dataset.

This was the fourth and last part of my Wordle series. See you in the next one!

Blogging is my hobby, so I happily spend time and money on it. If you enjoyed this blog post, putting 1 euro in my tipping jar will let me know :) Thank you for your support!

React Tutorial: Create Your Twist on Wordle

Cup of Code — Sat, 24 Feb 2024 09:01:33 +0000

There are many React tutorials to create Wordle clones… but why stop there? This is how I created a Taylor Swift-themed Wordle!

I’ve recently decided I would like to improve my React skills, and I knew the way to do it was to create a fun project. Therefore, I found a Wordle tutorial! It’s called Make a Wordle Clone with React by Net Ninja. That was a great start.

With this basic version of Wordle implementation (which from now on I will refer to as “the base code”), I chose to invest my best efforts in creating a game I would like to see in the world — “Guess the Taylor Swift Song Title”!

In addition, I developed the base code to include all the cool features the original Wordle game has to offer! You can read about it in my previous blog posts. To clarify, this blog post does not depend on them. Fun fact, the first thing I did after finishing the base code tutorial was adjust it to longer solution words.

So, grab the code base from Net Ninja’s GitHub (link), and join me in this tutorial! Instructions on how to clone the GitHub project can be found here. Which theme are you planning to give your Wordle? Let me know in the comments!

The agenda for this blog post is:

Overview of the base code components
Adjust the game to words longer than 5 letters
Adjust the game to multi-word solutions
Bonus — use an external data set

Are you ready for it? ;)

Overview of The Base Code

No need to show here how to do those steps because the YouTube tutorial does it very well. I’ll detail the state of the project post the tutorial, to connect you to the rest of the blog post.

The base code from Net Ninja’s GitHub can be found here. You can either clone his code and come join me now (instructions can be found here), or revisit this blog post after following his tutorial!

Let’s take a look at the files:

db.json contains 2 JSON objects: letters and solutions. The letters are used for the keypad functionality and the solutions are currently 5 letter words, with an ID attached:

    {
      "letters": [
        {"key": "a"},
        {"key": "b"},
        ...
      ],
      "solutions": [
        {"id": 1, "word": "apple"},
        {"id": 2, "word": "hello"},
        ...
      ]
    }

The base code pulls the solution words from the JSON file by running json-server in a separate terminal, which simulates a RESTful API.

node_modules is where all your dependencies sit. If you don’t have this folder yet, you’ll have it after the next section (“Running the code”). We won’t be touching this directory.
public folder contains static files such as index.html, images, and other assets. This is where we’ll set our title, favicon, and themed images!
src folder is divided into two sub-folders: Components and hooks.
components directory has the different building blocks of the code:

hooks directory has our only custom hook: useWordle.js.
App.js is the starting point of our game. This is where we fetch the solutions JSON.
index.css is where we make our game pretty :D
package.json is used to run the project. This is also where you build it if you want to upload your game to the World Wide Web ;)
README.md is where you explain your project, provide instructions on how to run it, and give a shout-out to Net Ninja thanking him for his base code :)
There are more files in the project, but we won’t touch them.

Running the code:

You’ll need to install all the dependencies by running these 2 commands (if you haven’t already):

npm i
npm i json-server

This is done only once in the project.

Run the json-server:
i. Open a new terminal
ii. Run the command json-server .\data\db.json --port 3001
Run the project: Open package.json, hover above "start" and click "Run Script".

When you run the code you will see the page with the grid and the keypad. The keypad is not clickable and is here to show you all the letters and their colors. Above the grid there is the solution for this round, and the current guess the user is typing (for testing purposes).

Every time the user types a guess and presses enter, the tiles flip and color with green, yellow, or grey. The game ends when the user guesses the solution word or when the user does not manage to guess the solution word within 6 tries. Then, a modal pops up with a winning/losing message, the solution word, and the number of guesses it took.

Please note that every time you refresh the page, there will be a different solution word.

The overview is done… Let’s start coding! :D

Adjust to Words Longer Than 5 Letters

The game as it is now is hardcoded for 5-letter words, in all parts of the application. Start by modifying your dataset to contain 6-letter words (so “apple” becomes “apples”, etc). Note that changing the data set will require re-running the json-server: ctrl+c if you have it currently running, and then json-server .\data\db.json --port 3001. Otherwise, you won’t see your changes.

The next step is to swap the hard-coded 5 with a length variable. This information, at the moment, is easy to gather with solution.length , but this won’t be the case when we add functionality for multi-word solutions later on ;)

We shouldn’t just use “replace all” in the editor because not all hard-coded appearances are with the character 5, as you will soon see. Instead, we should think logically — which components need to know the word size? The first one that comes to mind is the row — this component is the one dictating how many tiles are in a single row in the grid. It’s not the only place to change, but it’s a good start!

The Row Component

We’ll start by observing the existing logic. For us to do that, I attached here all the places that reference the five letters:

The current row (lines 6–13 above)

This div creates the row that the user is typing into. It means the map iterates over the characters that are currently in the guess and completes the rest of the row with empty tiles.

What does the code iterate on to finalize the empty tiles? The iteration is done on the array […Array(5-letters.length)] (line 10). This structure, […Array(5)], is equivalent to [0,1,2,3,4]. Considering we now have words longer than 5 letters, this value has to be modified:

For now, we will add length as a parameter of the Row component (line 1 below), and later on deal with passing this value. On line 5 we are now mapping the array [...Array(length-letters.length)].

The empty row (lines 17–23)

This code returns the empty rows that appear after the current played row. The five divs are the five tiles in each row. We can’t return 5 hard-coded divs anymore, we need the number of divs to depend on the length of the word! We do that by using a map:

In the new version, the divs have a key parameter (line 5 above). This is because it’s a requirement when creating HTML elements via a map (“Warning: Each child in a list should have a unique ‘key’ prop”).

Passing the length parameter

The next step is to pass Row the length parameter. As a reminder, the row is a part of the grid component, and the grid is a part of the Wordle component. The grid doesn’t know what the final solution is, it just receives guesses. The Wordle component has the solution, and therefore — access to solution.length.

Technically, we can use currentGuess.length and get the same information, with no dependency on the Wordle component passing the parameter. You can still do so. I preferred having one source of truth — solution.length, because I think it’s cleaner.

Note that there are two instances of Row in the grid component! (lines 4,6 above)

Cool, that was simple. The next place that has an explicit 5 reference is useWordle.js.

The useWordle Hook

Here we have a pretty easy swap as well because this hook also has access to the solution variable. Note that on line 6 above, the 5 is for the amount of turns, not the amount of characters! See? I told you replace-all wouldn’t work here ;)

For the rest of the occurrences, we carefully replace them with a new variable: let solution_length = solution.length. Notice that on line 14 below, you’ll need to swap the message string from ‘word must be 5 chars.’ to word must be ${solution_length} chars.. This includes both inserting a variable and changing from single quotes (‘) to backtick characters.

Last but not least — the styling!

The CSS file

If you skip this part, you’ll notice it pretty quickly, when only the first 5 tiles flip beautifully and the others just hitchhike with the first one.

To fix this, we need to add animation to the siblings that come after 5:

You might consider changing the delay. If, for example, you have 15 characters — the flipping as it is now (0.2 seconds for each tile) will take 3 seconds, and that’s a long time to wait. You can modify the delay gap to be 0.1 seconds between tiles.

Moreover, it’s not straightforward to determine the amount of .row > div:nth-child(X){…} to put. Even when we unlock the wonderous world of longer solutions, there is a limit to how many characters you’ll give your user. Don’t forget we want this game to be mobile-friendly! For my game, I chose a limit of 13 characters (and the Swifties among you know why ;)).

By the way, the length limit on the word won’t be done here — it will be done in the data set. By that I mean your data set should contain only “correct” solutions, and you shouldn’t do checks for appropriate length in the game’s code. The only hint of the character limitation in the game would be here, in the number of tile animations in the CSS file.

That’s it! The rest of the code fits this feature “for free”. Tiny changes gained you the cool ability to have longer words. This is setting the ground for the juicy part of the game (both for coding the game and playing it) — the multi-word solutions!

A comic relief:

Adjust to Multi-Words Solutions

We currently can contain an “unlimited” amount of characters for our solutions, so what difference does it make if the solution is one word or more? Ooh! It makes a big difference, which lies in the split!

To make it easier to understand, let’s follow this tutorial with the example solution “Shake it off” in our minds. Shall we start?

Define The Split

There are several approaches to achieve the same goal. I chose the one that would be less disruptive to the existing game logic. By that, I mean that I chose to look at the solution term as one long word (with no spaces) and save next to it an array of the splits in the word, based on this new representation. It sounds complex, I know, so let’s look at an example:

“Shake it off” -> “shakeitoff”, [4,6]

Think of it this way: If I wanted to go back from “shakeitoff” to the real song title, after which indexes would I need to enter spaces? After indexes 4 and 6:

    Shake it off
    01234 56 789

Needless to say, song titles with a single word, will have an empty split array [].

Hold on! No need to manually start inserting split arrays into your data set!! At this point, you can choose one of two options: Prepare your data set to contain split, or calculate it “on the spot” when you gather the solution. I chose the first approach, and in my next blog post, I’ll share how I did it 😀

Introducing The Split Property

As I mentioned earlier, our dataset structure changes from this

    "solutions": [
        {"id": 1, "word": "cupofcode"}
      ]

to this

    "solutions": [
        {"id": 1, "word": "cupofcode", "split":"[2,4]"}
      ]

To introduce the split in the code, we’ll go to the same file where we get our solution value: App.js. We’ll process split the same way we did solution:

Now, keep in mind that removing whitespaces from a string is easier than inserting them, even if you have a split array. This is the reason why I chose to save the solution term with spaces, and then in the Wordle component have a variable solutionWithoutSpaces = solution.replace(/\sg/,””);. This is the value I use in the grid component and the useWordle hook.

The grid component needs the split parameter just so it can pass it to the row component. The row component is where the magic happens!

The Row Component

This component can return three different types of rows: One occupied with a past guess, one filled (fully or partly) with the current guess the user is typing, or one empty from letters.

We’ll start with the initial case: The empty row.

The empty row (lines 26–30 above)

With the solution “Shake it off” in mind, let’s think about what we want this Row to return. Our split array is [4,6]— This means we want space after the 4th and 6th tiles. This space will be achieved by another type of div class, which I named space-div with a small width and a zero-sized border (to override the border set for .row > div class).

    .row > .space-div {
      width: min(2vw,20px);
      border: 0px solid #bbb;
    }

We already have a map iterating over the character tiles, so all we have left to do is add the new div only if it fits the condition “Is the index in the split array?”

This is how it will look like:

    {
      split.includes(i) &&
      <div 
        key={`${i}_seperator`}
        className=”space-div”>
      </div>
    }

Is that it? Almost! If you try to add this block below the existing <div key={i}></div> (on line 28), you will encounter an error: “JSX expressions must have one parent element”. This is easily solved by wrapping those two divs with . Now they have a parent!

Most times you don’t even need to use the full Fragment syntax, and can just use the shorter version: **<>**..code..**</>**. In our case, we do need to use the full Fragment syntax because we need to add a key (remember from earlier? The map function requires adding a key!), and you can’t do it with the short syntax.

With all that, our code looks like this:

Note that we have two divs and a fragment in the map, so we give each one a unique key: ${i}_frag, {i} and ${i}_seperator.

This was the first Row case — an Empty row. We took the initial map callback function, and wrapped it in a React fragment that contains a separator div when needed:

   <React.Fragment key={`${i}_frag`}>
     <INITIAL_DIV_RETURNED>
     {
      split.includes(i) && 
      <div key={`${i}_seperator`} className=”space-div”>
      </div>
     }
    </React.Fragment>

We will use the same fragment structure in the other row scenarios as well. The next case, in chronological order, is the current row (that the user is typing into).

The current row (lines 15–22 above)

In <div className=’row current’> , there are two arrays: One for the letters the user entered in the current guess, and one for filling in the empty tiles to match the length of the solution word.

In that second array, where it iterates length-letters.length times, make sure you use letters.length + i value to check if the index is split or not. Let’s see it in the code:

The past row

At this point, it’s easy peasy. We take the React fragment, insert the original div, and get:

Cool, all the rows are modified. Off to the next component!

The useWordle Hook

You’d expect this file to contain some big changes. However, because we passed the solutionWithoutSpaces parameter, the hook doesn’t even know there were spaces in the original solution!

The CSS file

This only needs to be updated to accommodate the changes made in the Row component. In my case, it means adding animation delays up until the 13th child (the spaces from the split array are also .row > div). This is, of course, in addition to adding the style for the “space-div” class:

The Share String

This section is only relevant to those who followed the tutorial in my previous blog post. There, we created the squares string that the user gets when they click the “share” button after the game is done. This string represents the solution characters, which means we need to add spaces!

To do so, all we need to add is a tiny if condition. For each guess, we assign to every letter the appropriate square color. With the letters, we can also check the index!

That way, after adding the right colored square, we check: If the index is in the split array, it means there should be a space after this square, so we add it to the string:

Comic relief: Because life is too short to pretend you don’t like Taylor Swift songs :)

Bonus: External Data Set

I really wanted the data set to sit outside the project. Firstly, I wanted to place an extra step for the curious user who checks the browser’s developer tools to see the bank of words. Secondly, and most importantly, I didn’t want to rebuild and upload the whole project every time Taylor Swift released a new album (who else is excited about The Tortured Poets Department releasing in April??).

By having the data set in a different location, I decouple it from the main project, which is always a good practice — why create a dependency where there isn’t one?

Officially,

Decoupled architecture is an architectural approach that allows each computing component to exist and perform tasks independently of one another, while also allowing the components to remain completely unaware and autonomous until instructed. — source

In Wordlesturck’s case, it’s even better than just generic loose coupling because the data set is created in a different project (more about it in my next blog post :D), so I can update the file (and therefore the game) without touching the Wordle project!

Creating The External Data Set

Currently, our data set file is called db.json and it sits in the data/ directory. We’ll start by creating a new project, outside of the Wordle one, that will have that same db.json file.

besides the JSON file, you’ll need a netlify.toml file with the following content:

The netlify.toml is a configuration file that specifies how Netlify builds and deploys your site. More about it here.
Lines 1–4 redirect https://react-wordle-db.netlify.app to https://react-wordle-db.netlify.app/db.json. Your code will work without it, I just find it nicer to see something and not an error when I click the link, as the developer.
Lines 5–8 are needed to make it work :D Otherwise, when you load your game (the Wordle project), you’ll see a CORS error.

Now you can create a new site on Netlify using this project!

Calling The External Data Set

In our current code, we interact with db.json in two files: App.js for grabbing the solution, and and Keypad.js for pulling the letters.

As I mentioned in a previous blog post, I am strongly against saving the letters in the db file (more about it here). I prefer having them in the keypad component:

    export default function Keypad() {
      const topRow = ["q","w","e","r","t","y","u","i","o","p"]
      const middleRow = ["a","s","d","f","g","h","j","k","l"]
      const bottomRow = ["z","x","c","v","b","n","m"]
    ...
}

So, I’ll be ignoring that part and focusing on the occurrence in App.js. But don’t worry, after this one, the keypad modification will be easy for you to do by yourselves.

Let’s start by observing the current code:

We’ll start by changing line 4 to fetch the file from our new location: https://react-wordle-db.netlify.app. Note that we are not pulling the solutions JSON specifically (AKA we don’t end with /solutions), but the whole file. Therefore, we need to adjust line 8 to address the solutions part of the JSON response:

    const randomSolution = json.solutions[
      Math.floor(Math.random()*json.solutions.length)]

This will work. With that said, to keep our code quality high, I recommend adding some error-catching to this fetch call. We didn’t need it before because the file was local, but that’s not the case anymore.

So, to make the project as durable as possible, I chose to keep the original db.json file inside the project and gracefully fail into it if my db website had an oopsie.

Note that the original file will soon become “the old version”. This is ok because due to the game’s nature, the user probably won’t notice it: They receive only 1 solution per day, in random order.

This is how we’ll do it:

Just like the .then(...) function, we’ll add a .catch(...). This catch will contain the same code block that is in lines 5–12 above.
For this to work, we’ll need to declare randomSolution outside the fetch, in line 4.

Before we test our error handling, there are two additional changes we should make.

We Don’t Need Fetch

Please note that you don’t need to use fetch(...) for local files. The base code had fetch because the developer was using the json-server package which acts like an external API server. As a result, The code was using the fetch function to grab the json.

json-server won’t work when you upload your website to Netlify, so I decided not to use it. Instead, we can simply do the following:

We’ll start by importing the JSON using a simple import on line 1.
Then, just like before, because we don’t have a specific API call with /solutions, we turn any json occurrence to json.solutions.

The next thing that would be nice to do is export the duplicate code to a function:

Notice that even though both calls to the setSolutionRelatedStates() function are with the json.solutions parameter, one is referencing the json variable from .then(json => {...} (line 17 above) and one references the json variable from import json from ‘./data/db.json’ (line 1).

Lastly, let’s see the error handling in action:

An easy error would be to add /solutions to https://react-wordle-db.netlify.app in the fetch.

And we’re done! In this blog post, we created a Wordle version that can have solutions longer than 5 characters, and multi-word solutions. We also set up an external location for the data set, so it will be decoupled from the main project.

To theme your wordle, all there is left to do is choose an appropriate name and logo, and gather the right data set. I will show you how I created mine in the next blog post :)

And for the Swifties out there, my game will launch VERY SOON! Connect with me on Linkedin to get notified when it’s online! :D

See you soon!

Blogging is my hobby, so I happily spend time and money on it. If you enjoyed this blog post, putting 1 euro in my tipping jar will let me know :) Thank you for your support!

Wordle in React: Picking Up Where We Left Off!

Cup of Code — Sun, 11 Feb 2024 23:48:28 +0000

The flipping tiles are cool, but why stop there? In this tutorial, we’ll create all the other cool features that are in the original game!

Welcome to part 2 of my Wordle tutorial! In the previous blog post, I gave beginner’s tips and tricks in React, and we worked on creating guess distribution, landing page, and game index. You should start there because the features we will work on today depend on those!


https://cupofcode.blog/wordle-in-react-part-1/

In this blog post, we will continue developing the features that exist in the original game, but for some reason were left out of the common “Wordle in React” tutorials online.

I am also planning a last, but not least, part for this series — how to modify the game to play with words different than 5 letters long!

The agenda for today is:

The share button
A different view depends on the user’s game state
The progress modal — adding streak!
Bonus: A shaky row!
Final touches

Are you ready? :D

The Share Button

We are going back to the progress modal!

After they finish the game, our users want to show off their success and send their friends their game process. In the original game, when you click the share button, it copies to your clipboard a string of squares that are a visual representation of the guesses in the game. Then, you can paste that string anywhere you want: WhatsApp, Twitter, etc.


Users want to show off their success and send their friends their game process

We will build this feature in 2 steps: First, create the string of squares along with the game name, game number, and the number of guesses it took. Then, we add the “copy to clipboard” functionality.

1. Create The Squares String

There are two places you can create that squares-string in, and they both make sense. The first one is in the useWordle.js hook, right next to where we add a new guess to the guesses array: Adding a new guess— adding a new string of squares.

The second option is adding this code where the user requests the string: In the Share button’s “onClick” function. That way, we execute this code only when we need it. This approach is cleaner, but the problem with it is that we don’t currently have access to the guesses via the modal component, and passing them just for this feature seems like a major modification.

So, I went with the first approach. In the useWordle.js hook, similar to guesses state, I created squearesStrings state with a very similar logic: Based on the previous squares-strings array, create a new array and add the new guess.

Translating the guess into squares string is pretty straightforward, so we’ll move on and focus on the juicy parts instead :D

Now that we have an array of the squares representing the guesses, we’ll pass it to the modal and stringify it there. At this point, we will create the resultString variable, which includes the rest of the information that is shared in the message: The Game name, the game number (which we need to pass as a parameter as well), and the number of turns it took.

This worked just fine, so what made me change to the second approach? The answer to that depends on future functionality, so we’ll revisit this one later.

2. “Copy to Clipboard” Functionality

This is a cool feature that it’s implementation is easier than I thought!

The w3schools tutorial is very clear, the only modification we need is to copy our resultString variable and not a text box like they are showing in their demo. Our line of code will be: navigator.clipboard.writeText(resultString);

Our code will look like this:

And in index.css:

I usually don’t share CSS code, because styling is an individual taste. However, this time it contains functionality. For example:

v.tooltip .tooltiptext::after {content: ""; ...} will initialize the text after the copy, so if the user clicks the button twice, they won’t get double the content.
The visibility property will ensure the tooltip appears when the share button is clicked and disappears when the mouse is not hovering above the button anymore.


Copy to clipboard functionality

A Different View Depends on The User’s Game State

In the previous blog post, we created the grey “Welcome” landing page. Have you ever tried going to the Wordle website after you already played that day in that browser? If you do so, you will see a “Hey, great job today!” page.

While working on this project, I visited the Wordle website a lot. I had to see what’s the scope of every feature and test how it responds to edge cases. It was by accident that I discovered there is a third landing page — the “you’re mid-game, come continue” page!

The different view is not only on the landing page. When the user clicks the button, they see their previous guesses on the grid and the colored keys on the keyboard.

A quick recap:

Our game is a single-page application.

“A single-page application is a website that interacts with the user by dynamically rewriting the current web page with new data from the web server, instead of the default method of a web browser loading entire new pages.” Wikipedia

In our case, we don’t use a server-side, but you get the idea. This means that depending on booleans, we show and hide different components: We start with showWordle=false , which makes the user see a grey landing page with two buttons - Play and How to Play. When they click “Play”, we set the Wordle boolean (showWordle) to true (line 9 below). This makes the Wordle component (the page with the grid and the keyboard) show up and the landing page hides (lines 6,15 below).

When the user clicks “How to Play”, they will see the instructions modal above the Wordle component. This means that you can see the grid page in the modal’s background and closing the modal will reveal the page fully. To achieve that, we need another boolean (line 3 below) to indicate the instructions modal should show as well:

Whether the wordle boolean or the instructions boolean are true (depending on which button the user clicked) — we show the wordle component (line 16 above).

Recap is over, now we can start!

The Post-Game Welcome Page

If we look at the welcome component, the only difference between the two grey pages is the title, description, and button(s).

The indication of which page to show depends on whether the user played today, which is a boolean! We’ll call our boolean gameWasPlayedToday and for now — set it to false (line 3 below). We use this boolean to wrap our existing landing page code (line 8) and the post-game landing page code (lines 14–20).

The post-game welcome page has one button — see stats. Similar to showing the instructions modal, we need to create a boolean and use it for the logic:

On line 2 we initialize showProgress to false.
On line 17 we set it to true on the button’s onClick function.
On lines 6,25 we add this boolean to the page’s conditions.
On line 29 we pass the boolean to the Wordle component, which is the one showing/hiding the modal.

Now, all that is left here is to set gameWasPlayedToday correctly (line 3). How do we know if the user played today? We have local storage for statistics and guess distribution— could that be useful? Yes and no.

The data we save in statistics do not currently indicate if the user played today. We can modify this object to save more information, but I think it’s already big enough, and prefer creating a new local storage object: localStorage.wordlestruckLastGamePlayed.

Creating a new local storage object

There are five things we need to know about the last game played:

gameNum will help us know if the last game played was today.
turn, isCorrect will be used to color the correct guess distribution bar in green.
guesses, usedKeys will be used to populate the grid and the keyboard.

Where does our code determine when a game is over? In the Wordle component’s useEffect! That’s where we will set our new local storage object:

Now we can go back to the landing page component and set gameWasPlayedToday dynamically: We start by setting lastGame to our new localStorage variable (or undefined if there isn’t any). Then we determine if the user played today by comparing the saved game index with today’s game index.

Note that there are two ways to populate gameWasPlayedToday, and you’ll see both in my code. When it’s only one variable dependent on whether last game is defined, I use the inline if, and add lastGame && ...:

var gameWasPlayedToday = lastGame && lastGame.gameNum == index ? true : false;

If there are several variables, I prefer initializing them first and then modifying them in an if block:

var gameWasPlayedToday = false;
If (localStorage.wonderstruckLastGamePlayed){
  let lastGame = JSON.parse(localStorage.wonderstruckLastGamePlayed)
  gameWasPlayedToday = lastGame.gameNum == index ? true : false;
  ...
}

So, we used the stored game index, and we have 4 more variables to utilize: turn, isCorrect, guesses, and usedKeys. We’ll apply the turn and isCorrect variables to update the statistics component.

The post-game landing page has a “see stats” button that opens the progress modal (which is calling the statistics component).


Post-game functionality. (Don’t worry, I’ll shuffle the solutions before launching the game ;) )

Up until this point, the turn and isCorrect variables were passed through the component’s parameters, ‘in real time’ when the game was over. We will now pull that data from the local storage variable:

We’ll start with an if condition to check if there is stored data for a previous game played, and initialize the variables for a case there isn’t (this is the “second way to initialize variables based on lastGame" that I mentioned earlier :D).

Did you notice something extra on line 12?

Even with the new landing page, we only reach the progress modal when the game is over. So, why did I add gameWasPlayedToday && ... to the condition for coloring barColor? This is not part of this feature, but part of a new one that I will give you the pleasure of adding yourselves: The header buttons.

The original Wordle has buttons in the main page’s header, that give you access to the instructions and the statistics modals at any time. This means the user can start a new game, and decide they want to check their statistics before the game is over.

In that case, we want all the bars to be grey (and not green), and in my game — I also hide the solution word. In addition, I chose to hide the share button, but this is a personal preference.


The Progress modal before the game is over: All the bars are grey and there is no share button

We already have the functionality to show those modals thanks to the various landing pages, so all that is left to do is create two buttons and setShowProgressModal(true)/ setShowInstructionsModal(true) on click!

Showing guesses and used keys

We made nice progress, but we still didn’t cover all the functionality required for the post-game landing page.

When the user clicks the See Stats button, it will open the progress modal, which we took care of. When the user closes that progress modal, they should see all their past guesses on the grid, but we don’t currently have that information. So… we’ll add it now!

We already have guesses and usedKeys saved in localStorage.wordlestruckLastGamePlayed object, so all that is left to do is add a view option so that if the user already played today, we’ll populate the data with the stored object.

On lines 3–10 below, we populate gameWasPlayedToday, gameData.
On lines 23–31 is the good old Wordle main page — the grid and keypad components, wrapped with the condition of !gameWasPlayedToday && ....
On lines 14–22, we also have the grid and keypad components but this time we populate the properties guesses, turn, and usedKeys with the data from the stored object. And of course, we wrap this block with gameWasPlayedToday && ...

Huston, we have a problem

Another issue that comes up now is that our new shiny share button will be accessible but without a squares-string! This is because we create the squares-string in the useWordle.js hook, but we don’t interact with this part when opening the progress modal directly from the landing page.

I know, we are jumping back and forth, and it must be confusing! No need to scroll up, I’ll give you all the context you need: To let our users show off their game skills, we created squares string representing the game guesses (without exposing the word!).


Please don’t take away my Swiftie badge

We talked about two places we can build that string and chose to go with the useWordle.js hook, right after we add a new guess to the guesses array. This is because we dismissed the second option, which is within the share button’s on-click function. We didn’t have access to the guesses via the progress modal component, and passing them just for this feature was too much hustle for the same result.

Well, guess what? We have access to the guesses array from wherever we want, thanks to the localStorage.wordlestruckLastGamePlayed object! Let’s modify the code to create the share-string in the progress modal component.

A few things to note:

The guesses array in the useWordle.js hook is initialized to be of size 6 (because there are max 6 guesses), and the guesses are added every turn. This means we can’t simply iterate over the guesses array (guesses.forEach((formattedGuess) => { ...), we need to ensure we iterate only over the populated cells (if(formattedGuess) { ...).
Now we can initialize resultString directly with the prefix, instead of adding it after the squaresString creation.
At this point, you can remove any code related to the previous version of squaresString we added in useWordle.js, Wordle.js and ProgressModal.js.

A well-deserved meme break:

The Mid-Game Welcome Page

If the user loads the page after entering guesses but before finishing the game, they will see the mid-game welcome page.

This page has one button — Continue. When the user clicks it, they see the main page with the guesses they tried so far and the keys they used. To achieve that, we can’t save the game data only when the game ends — we need to save the relevant information after every round ( = after every guess the user entered)! We’ve seen usage of local storage before, but this one is different!

You can either trust me or try yourselves (I encourage both approaches ;) ), but updating local storage directly won’t work. This is because the Wordle.js component renders a lot more times than you might think —in fact, we trigger page rendering at every key press (due to the handlekey listener). For that reason, if we were to try to continue the game, the typing would cause the data of previous guesses to initialize to an empty array.

The right approach is to add to useWordle.js hook a new state variable: storageData. We will set this variable after every guess entered, and then in Wordle.js we will update local storage with it. That way, in those handlekey renders, the data we set in the useWordle.js hook stays as it was. It only updates on the submission of valid guesses, when the code reaches return {turn, currentGuess, guesses, ...}.

Let’s start coding!

We will begin by initializing variables for each data needed in the hook: initialX, when X is game number, turn, etc. The value will be pulled from gameWasPlayedToday or restarted, if there isn’t any data saved (lines 3–8 below). Then, we will create the new variable storageData and initialize it with those variables (lines 10–17).

Please note that to continue the game at any point, we need to update our localStorage.wordlestruckLastGamePlayed object to contain two more properties: history and isCorrect. This is because we need those for the useWordle hook functionality.

A quick reminder: The difference between history and guesses is that history is a simple array of strings, while guesses is an array of formatted guesses, where each character in each guess is colored.

All that is left to do is add the setter for storage data to addNewGuess() function. We will set the properties just like we set their original counterparts (lines 11–49 below).

Lastly, we will return storageData to the Wordle component (line 52).

Are you seeing what I am seeing?

This code is a bit painful to read because setStorageData contains all the previous sets’ code: setIsCorrect, setGuesses, setHistory, setTurn and SetUsedKeys.

Code duplication is bad practice because now you have multiple places to update code and you might not remember all of them. How about modifying useWordle.js to have only setStorageData? We will do that later, once storageData is up and running, and the application is using it exclusively. We still need to use the data we collected!

In Wordle.js , just above the end of the useEffect scope, we update the local storage with storageData:
localStorage.wordlestruckLastGamePlayed = JSON.stringify(storageData);

Lastly, we need to put the correct boolean to show the mid-game landing page.

A quick reminder: If the game hasn’t started yet — we want to show the “welcome” landing page. If the game started but is still in progress, we want to show the “welcome back” page, and if the user finished today’s game, we will show the “good job!” page:

We already set gameWasPlayedToday variable when we worked on the post-game landing page:

gameWasPlayedToday=
lastGame && lastGame.gameNo === gameNo ? true : false;

This now needs to change. We modified the code to save state after every turn, therefore the game index in lastGame indicates this game has started, not necessarily that it's done.

If gameWasStartedToday is the new gameWasPlayedToday, how do we gather the value for gameWasPlayedToday? No need to save additional variables in local storage, we can just calculate it from the existing stored data:

We know the game started today if the last game’s index is today’s game index. To check if the game was done — we use the same condition as in the Wordle.js component — if isCorrect is true or if the user played their 6th guess.

In case you wondered, I am saving the turn value in a local variable because I need it for the work-in-progress landing page — we want to show the user how many tries they already made.

Back to the Progress Modal!

In the previous blog post, we created the progress modal: It called the statistics component, which shows the user data of games played and % of wins, and below that their guess distribution. This is where we stopped because we needed today’s context to take the modal to the finish line!

Bug Fix — Update Statistics Only Once

With today’s changes, we have unlimited access to the Wordle component, and more specifically — access to the end-of-game condition: isCorrect===true || turn>5:

We wrote this code in the previous blog post, go check it if you need a reminder!

This code has become problematic now because every time we go from the “you-already-played-today” landing page to the grid and keyboard page, the if statement (line 5 above) is correct, and we update the statistics (again) on lines 14–15.

Therefore, we need to make sure we update the statistics data only once per game. We can do that by wrapping it in another if condition. That, together with the existing if condition (“if the game is finished”), creates a sweet spot that happens only once in the daily game’s life cycle: The state where local storage’s isCorrect or turn>5 is changing value for the first time!

I created two variables: player_just_won and player_just_lost. Let me remind you we are updating the local storage after every turn. Now, I know the player has just won if isCorrect is true but JSON.parse(localStorage.wordlestruckLastGamePlayed).isCorrect is still false. Same for when the player just lost the game: turn>5 but also JSON.parse(localStorage.wordlestruckLastGamePlayed).turn===5.

It’s also very important to move out the initializing code (lines 7–10 above) to be before the useEffect hook. If you don’t move it, you will encounter a bug later, when you add the statistics button to the header. The bug occurs when the user enters the game for the first time and clicks the statistics button: It will be undefined! Logically, we want localStorage.statistics to be defined when we reach the progress modal. So we’ll define it first thing in the Wordle component.

Our code will now look something like this:

Feature — Adding Streaks Data

To give the progress modal the final touch, we can collect the data for the current strike and the maximum strike.

We’ll start by adding those values to our statistics object, along with currentStreakLastIndex — this is needed to know if the strike continues or not. This means the array of statistics will sit deeper in the object— localStorage.wordlestruckStatistics.stats. We will initialize all the new properties with zero.

Updating the values can be tricky: It’s very important to update them in the correct order! Let’s start with the easier one: maxStreaks is updated only if currentStreak is bigger. we need to check currentStreak‘s value only after we update it today.

currentStreak will be increase only if today’s game continues the streak, which means if today’s game index is currentStreakLastIndex + 1. Otherwise, it will initialize to be 1. This means we should check currentStreakLastIndex‘s value before we update it today.

Lastly, currentStreakLastIndex‘s value should be updated to be today’s game index. Remember that we reach this code only when the game is over, so this game is the current streak’s last index.

Let’s see it in the code:

Important note! The original game is counting streaks of winning, not streaks of playing. I decided to count the streak of playing the game, regardless if the user managed to guess the correct answer or not. If you wish to count streaks for winning, make sure to check if isCorrect is true before updating the new three variables (currentStreak, maxStreak and currentStreakLastIndex).

If you already published your website and sent it to your friends (I’ve mentioned Netlify as a good platform to host your website for free, in the previous blog post), then you should update the name wordlestruckStatistics to something else. Otherwise, you are referencing in your code the developed version of the object (wordlestruckStatistics.stats for example), but your users have the previous version saved in their local storage, so running wordlestruckStatistics.stats will break the application. When you rename that variable, the code will enter the initializing block (if(!wordlestruckStatistics){…}), and you will save yourself a bug.

Bonus Feature — Shaky Rows!

There is one feature from the original game that I’ve decided to not recreate — the “not a real word” feature. When that happens, the row shakes, and a black tooltip appears, notifying you that this word doesn’t exist.

I am still wondering “How did they create a data set of all possible 5-letter words?”. Once you have this data set, it’s pretty easy to notify the user when they have invented a word.

In my game, as you will see in the next blog post, I have a small data set — less than 200. I can check if a guess is within this array, but I don’t think it fits that game, so I decided to not implement it.

With that said, I did want to shake my rows! This is the first turn I took from the original game — I am shaking the current guess row when the user is trying to enter a guess they already made today! In the original game, they let you waste a guess on a duplicate word — so watch out for that!

Well, how do we shake the row?


Shake shake shake

For this, we will need CSS animation. This is not the first time we have used animations in the project. We bounce tiles when the user types in characters, and flip tiles to reveal the characters’ colors! Let’s start by creating the shaking animation in the CSS:

All animations have the same structure: @keyframes and division to the percentage of progress. In our case, we want back-and-forth of left and right (lines 2–11 below). Then, we create a new class that will use it (lines 13–15):

Now we need to give the appropriate row div this class name. Which is the appropriate row? The shaking will be of the row that is currently in typing. So, we’ll modify Row.js:

In the code above, we pass a property of shaky to the row component (line 1). The current row will shake (line 8 above) only if the shaky bool is set to true (line 2). Where do we set that value? At the same place we “find out” the user already tried this word — in useWordle hook!

The fun part is that this code was already in the base code, so no extra work was needed from us, just to add the setShaky() function in the right places:

We set shaky to true in the block where we check if the history includes this word (line 8). We also need to set shaky back to false so it will stop shaking after the first character deletion (line 15).

Lastly, as line number 2 implies, we need to add the shaky variable to Wordle.js:
const [shaky, setShaky] = useState(false)

Note that this time we pass the setting function, not the variable:

const useWordle = (…, setShaky)

Look at it shake! Lastly, we need to add a tooltip: We should notify the user why the guess didn’t enter. We already have a tooltip in the code so this will be easy for you to implement by yourselves.

With this done, we successfully finished our list of features!


Shaky row in action!

Final Touches

Now that we have a cool game, there are a few steps to make it even cooler!

Refactoring the code to use only storageData:

Remember earlier, when we noticed a disgusting amount of duplicate code in useWordle.js? We will fix it now! Start cautiously by going to Wordle.js and adding variables instead of returning them from the useWordle hook and see if anything breaks.

Once that looks ok, go to useWordle.js and remove those variables and their setters. You will need to use
storageData. instead of those variables in some places, but overall this is a simple change.

Phone screen capability:

Make sure the website view is neat from the phone as well. Check both Android and iPhone, because they don’t present the same way. A useful trick I learned in this project is to set font size or div width with this structure: min(9vw,40px);. Pixels are fixed, but vw is viewport width, which works like a percentage. Keeping the pixel option is good because vw gets too big on the laptop view, and this sizing structure keeps it proportional.

Fonts:

The whole goal of this project is to wink at the original Wordle. What better way to do so, than gather similar fonts as well? My search showed the original Wordle uses a special New York Times font (that I assume is out of reach), so I used Abhaya Libre and AbrilFatface, and I am pleased with the result :)

More Refactoring:

At first glance, yes — why bother if the user doesn’t see the code? But, refactoring has two main advantages: First, optimizing the code and using better practices, and names. This will minimize things that could cause you to make oopsies. Second, it will sharpen your React skills. You’ll learn additional (better) ways to produce the same output.

The Only Feature I Couldn’t Figure Out:

In the original game, when the user already played today and clicks the See stats button from the landing page, there is a small delay before showing the modal. This delay exposes the tiles beautifully flipping with the previous guesses. The code, as it is now, has flipping tiles animation, but I didn’t manage to delay the modal.
If you decide to work on this feature, please remember: The modal should be delayed only in two situations: When the user is winning, which is when we have the line setTimeout(() => setShowProgressModal(true), 2000), or when the user arrives from the landing page. There should be no delay when the user clicks the header button!

Wow, what a journey! In this blog post, we worked on the share button, set different views based on the user’s game state, and shook some rows!

In the next blog post, we will transfer our game from a boring 5-letter word to anything you want! Mine is Taylor Swift songs :D

Exciting news to all the swifties out there — my Wordlestruck game will launch at the end of this blog post series!

See you soon!


https://cupofcode.blog/

Blogging is my hobby, so I happily spend time and money on it. If you enjoyed this blog post, putting 1 euro in my tipping jar will let me know :) Thank you for your support!

Wordle in React: Picking Up Where The Others Left Off!

Cup of Code — Sat, 27 Jan 2024 17:09:51 +0000

Learn how to create the guess distribution, the landing page, and all the cool features that are in the original Wordle game!

I’ve recently decided I would like to improve my React skills. Sure, I’ve followed the official React tic-tac-toe tutorial before, but things still didn’t click for me. I knew the solution was to create a fun project, so I found a Wordle tutorial! It’s called Make a Wordle Clone with React by Net Ninja, and that was a great start.

Unfortunately, this project wasn’t close enough to the real game. Don’t get me wrong, it had the main functionality, which is very cool to implement, but I was passionate about adding the missing features that appear in the original Wordle!

In this tutorial, we will grab the Wordle-clone project that many people on YouTube created, and take it to the finish line! In a future blog post, I will show you how to theme your game with more exotic topics than ‘5 letter words’ :D


So many React Wordle tutorials and they all stop at the same stage

To make things clear, I will refer to the code I produced by following Net Ninja’s tutorial as “the base code”. Here is a link, in case you want to check it out.

The agenda for this blog post is:

Useful React tips for beginners
Quick overview of the base code we are starting with
The virtual keyboard
The post-game modal
The landing page
Calculating game index

Shall we begin?

Useful React tips for beginners

I know it all looks overwhelming when you are just starting with React, so let me make things a bit clearer!

There are two Reacts out there: React Native and React/ReactJs. ReactJS is for making front-end websites, and this is the one we are using today.
Creating a new project: Super simple, follow this paragraph and you are good to go. If everything goes well, your browser will open a new tab for localhost:3000 with the default starting page:


The starting page

Running the app locally: You can either use npm start in the command line or hover above start in package.json and then click Run Script.


Running the app locally

You don’t need to restart when you want to see your code changes, it will update automatically.
You should try refreshing the browser page when you are fixing an error — because those sometimes stay on the screen even when the issue is fixed.

There is a useful VS code extension called ES7+ React/Redux/React-Native snippets. With that installed, you can create a new component file, type in rfc and it will auto-complete the structure of the component.

Here are some solutions for problems you will probably encounter:

Double rendering: When you start printing to the console you will quickly notice that everything is rendering twice. This is not a bug in your code! This is a React thing. This does not happen in Production, just locally in the development environment. You can read more about it here.
Syntax tips and tricks:

The error “JSX expressions must have one parent element” means you need to wrap your HTML elements with empty tags: <><your code></>.
The syntax for mapping over a JS array to produce HTML elements is as follows:

The syntax for styling inside the HTML: <h1 style={{paddingLeft: “70px”}}> . Notice the double curly brackets, the camelCase, and the stringifying of the value only (but not the key).

Updates via the UseState hook do not reflect immediately, which means setting a value and then immediately getting it, won’t work as you intended. You can read more about it here.

setMovies(result);
console.log(movies) // movies!==result

Deploying to an App Deployment Platform

Excitement to share your final product is an appropriate outcome of following this tutorial, and Netlify is the way to do so. Netlify is a website that hosts your project for free!

Uploading to Netlify:

You can watch the full tutorial here, but the TLDR is:
build the project (you can find it in the package.json file, just below start). This will create a build directory with all the needed files. Then, sign up to Netlify, and upload the build directory. It’s that simple!

I recommend you give it a try now and deploy the project you have as it is currently, before adding code. This way, you’d know if the blank page you see is a code issue or a deployment issue due to it being your first time using the platform.

Speaking of a blank page, let me introduce you to the white screen of death:

This refers to a situation where our React application appears to have crashed and only displays a blank white screen. This is very frustrating, especially when the app works fine on the laptop but not on the phone (and how do you debug there?!).

This is usually caused by JS code such as Date formatting, regex expressions, etc. I quickly learned that Chrome and Android phones are more forgiving than iPhones and some browsers on Mac (Safari, Firefox) so I highly recommend checking your website there periodically. If you see an error when opening the website on an iPhone, you will probably encounter the issue in Firefox on the Mac as well, where you have developer tools and can see the error.

The definition of done of any feature should include testing the iPhone view. Checking it via the developer tools’ responsive view isn’t enough, because it doesn’t give an accurate display. You’ll see examples of that later in the blog post.

Now you are ready to start!

The Base Code

No need to show here how to do those steps because the YouTube tutorial does it very well. I’ll detail the state of the project post the tutorial, to connect you to the rest of the blog post.

The base code from Net Ninja’s GitHub can be found here. You can either fork his code and come join me now, or revisit this blog post after following his tutorial!

If you want to pull the base code:

Pull the code from Net Ninja: a. Open a terminal in your projects directory. b. Run the following commands one by one:

git clone https://github.com/iamshaunjp/React-Wordle.git
cd .\React-Wordle\
git pull origin lesson-16 --allow-unrelated-histories

You might need to resolve conflicts in README.md

c. Edit the README.md to be yours, and don't forget to credit Net Ninja for the base code!

d. Now you have the code locally!

Make sure it runs smoothly: a. Open your React-Wordle project in your IDE and open a new terminal b. Install dependencies by running the following commands one by one:

npm i
npm i json-server

c. Run the json-server:
i. Open a new terminal
ii. Run the command json-server .\data\db.json --port 3001

d. Open package.json, hover above "build" and click "Run Script"

e. A new tab will open and you’ll see the game!

Connect the project to your GitHub repository: a. Create a new repo at GitHub. b. Open the terminal in your project directory. c. Run the following commands one by one:

git remote set-url origin <URL_TO_GITHUB_REPO>
git add .
git commit -m "Starting with NetNinja's base code"
git push origin main

d. Now your code is in your GitHub repository!


Run the base code locally

Overview of the base code

When you run the code you will see the page with the grid and the keyboard. The keyboard is not clickable and is here to show you all the letters and their colors. Above the grid there is the solution for this round, and the current guess the user is typing (for testing purposes).

Please note that every time you refresh the page, there will be a different solution word. The original Wordle game gives one word per day, and we will be coding this feature in today’s blog post! :D


Screenshots from Net Ninja’s YouTube tutorial

My favorite part was the flipping tiles — I’ve never made CSS animations before — and now I know how! :D

This won’t work on Netlify!

If you try to deploy the base code as it is now in Netlify it won’t run! This is because the base code is pulling the solution words from the JSON file by running json-server in a separate terminal, which simulates a RESTful API. Then, the line fetch(‘http://localhost:3001/solutions') breaks in Netlify.

Using the json-server package doesn’t work in Netlify because Netlify can only host static pages (with no servers running on parallel terminals).

The solution: Uninstall this package (npm uninstall json-server) and modify the code to simply call the local file: import solutions from './data/db.json';

The Virtual Keyboard

The keyboard in the original game has two roles: Visual and functional.

First, show the color of the letters:

Letters you haven’t tried yet — light grey
Letters you guessed correctly in the right place — green
Letters you guessed that are in the solution word but not in the right place — yellow
Letters you have tried but do not appear in the word at all — dark grey

The second purpose of the keyboard is to populate the grid tiles. The user can click the virtual keyboard and it will function the same way as a regular keyboard, which is useful for users on mobile and Padlets.

The keyboard we start with from the base code does not have any functionality, and it doesn’t look like a regular keyboard, so we are now going to fix that!

Making It Look Like A Real Keyboard

First thing first — the keyboard is in lowercase letters! The original Wordle’s keyboard (and any keyboard in the world) has uppercase letters, so let’s change that!

In the base code, the keyboard is created using a JSON object called letters:

{
  "letters": [
    {"key": "a"},
    {"key": "b"},
  ...
}

Then the Keypad component fetches that object:

We can simplify this! Remove it altogether and create an array of letters inside the component. No need for fetching and no need for useEffect hook! Tip: Don’t forget to order them in the qwerty keyboard layout.

Note that even though it’s tempting, you shouldn’t change the letters to uppercase. The user is typing in lowercase (like the letters you are reading now), and the function will compare them to what would now be uppercase letters, and will always return a mismatch. So how do we get the uppercase visuals? With the CSS file!

.keypad > div {
  ...
  text-transform: uppercase;
}

If you run your app now, you might see the keyboard’s keys spilling to the wrong places: P goes to the second row, and L goes to the third row, depending on the width of the window. I fixed that by dividing my array of letters into three.

Now, to build the keyboard, I’m looping over the 3 arrays. This solution makes it very easy to add “Enter” and “Backspace” buttons: I just add them separately before and after the third loop!

Making the Virtual Keyboard Clickable

The most straightforward approach here is to add “handleKeyup” function as the keys’ “onClick” event. Unfortunately, it didn’t work.

I couldn’t get the virtual keyboard to give the “handleKeyup” function the right parameters to make it work (something about it being an event handler and this {key} parameter). This caused me to use a workaround:

I took the content of handleKeyup and wrapped it in a new function, one that doesn’t have a weird {key} parameter. That way I had one source of truth to handling the user typings!

const useWordle = (solution) => {
  ...
  const handleKeyup = ({ key }) => {
    handleKeyupVirtual(key)
    return
  }
  ...
  return {
    ...
    handleKeyupVirtual
  };

To use this new function, we need to output it from the useWordle hook and pass it to the Keypad component:

export default function Wordle({...}) {
  const { ..., handleKeyupVirtual} = useWordle(...);
  ...
  <Keypad usedKeys={usedKeys} handleKeyupVirtual={handleKeyupVirtual} />
  ...
}

And then in Keypad.js:

return (
  ...
  <div key="Enter" style={{width: "15%", fontSize:"13px"}}
     onClick={() => handleKeyupVirtual("Enter")}>Enter</div>
...

Fixing a Tiny Bug

When I opened the game on the iPhone I discovered a problem: The backspace character is very small, regardless of the font size.

How do we solve this? I checked how the original game did it: I opened the NY Times Wordle with the developer tools and saw they used an image instead of a character, so I did the same.

After finding an appropriate image online and placing it on my keyboard, the image was stuck at the top of the key and I wasn’t able to resolve it with direct CSS modification: When I tried to modify the margin or the padding, it affected the entire row. It was not as easy as it initially looked!


When I added a margin to the backspace image — it affected the entire keyboard row

Eventually, I did find a trick on Stack overflow: Add the “no wrap” white-space property to the backspace div, and add inside it a helper span. Now it’s in a perfect size!

Final Touches

The keyboard is functioning and it looks good, but there are a few more things we can do to make it perfect!

Adding delay to the key color change

You might have noticed that when you play the original game, the keyboard key color updates only after the grid row is colored. In our game at the moment, the keyboard updates quicker because of the tiles’ animation.

I was looking into delaying components in react when I suddenly realized there must be an easier approach via CSS — and my hunch was right! All you need to do is go to the .keypad > div.green (and yellow and grey) and add transition-delay: 1s; — That’s it! :D

Addressing CapsLock in handleKey

At some point, I’ve noticed that if the CapsLock is on, it messes up the entered guess. To solve this, I added an if condition to catch that key, and ignore it (just return):

const useWordle = (solution) => {
  ...
  const handleKeyupVirtual = ( key ) => { 
    if (key === 'CapsLock') {
      return;
    }
    if(key === 'Enter'){
      ...

Position the keyboard at the bottom of the page

When the user is on the mobile, it’s more comfortable to have the keyboard lower on the screen (closer to the thumbs). The keyboard is currently right below the grid, which is closer to the center.


Positioning the keyboard at the bottom is better for mobile users

With that said, I like that the keyboard is just below the grid when it’s the desktop view. This means we need different styles for different screen sizes!

We start by adding a @media rule with the condition of max-width: 700px, so it would affect only a narrow screen view. Inside that, we add the second version of the keypad style.

We need two modifications to make it work: First is adding to the keypad position: absolute; along with bottom, right, and left = 0. Then, we need to add to the body’s style position: relative; and set the height to be 90vh.

If you set it to 100vh, it will look beautiful on your screen, in any size, but will require scrolling on the iPhone (I encourage you to check for yourself).

Now, after all these modifications, we have a beautiful functioning keyboard:


The new keyboard — both pretty and smart!

Post Game Modal

When the game is over, pops up a modal with some interesting information.

In the original game, there are statistics including the number of games played and the percentage of games won.

Below that, you can find past games’ guess distribution: How many times did the user guess the right word in how many tries.

Lastly, there is a share button that produces a string of squares, representing the game, that the user can share with their friends. We will create all those cool features!

Add a Close Button

I’ve started with the thing that annoyed me most: I couldn’t close the modal. I had to add an exit button.

This is super easy to implement in pure HTML and JS but I wasn’t sure how to do it correctly in React. According to the base code, the modal shows up with the boolean showModal, and its value is set in the “Wordle” component:

Add Close Button

I’ve started with the thing that annoyed me most: I couldn’t close the modal. I had to add an exit button. This is super easy to implement in pure HTML and JS but I wasn’t sure how to do it correctly in React. According to the base code, the modal shows up with the boolean showModal and its value is set in the “Wordle” component:

Even if I add an X button, how do I change the value of that showModal boolean from within the Modal component? The trick is to pass setShowModal as a parameter to the component!

Fun fact: × is a multiplication sign in HTML. It looks a bit different than “x”.

Now the grid page is accessible post-game

It’s important to mention that adding the functionality to close the modal means the user can see the grid of guesses after the game is over. This is fine, but we need to make sure he can’t type in any more guesses, both on the physical and virtual keyboards.

For the physical keyboard, we can simply add else to the “if-user-won” inside the useEffect, and only there add the keyup listener.

Please note that in the base code, there are different modal versions for winning and losing (and therefore different if conditions), but I merged them — so in my code, it’s one if condition: if the guess is correct or if it’s not correct but we reached the 6th try (if (isCorrect || turn > 5))

For the virtual keyboard, we want to override the clickability of the keys on that last turn. Let’s see it in the code:

We start by passing an argument notifying the keypad component that it should lock the keys. Considering we want to indicate the game is over, we can use the same boolean we use in the if-the-game-ended statement: isLocked = (isCorrect || turn > 5). Now, the trick is to use pointerEvents property: We set it as “none” or “auto” based on the value of isLocked. Lastly, we add it to the style of the keyboard’s keys (you should have a total of 5 occurrences: top, middle, and bottom rows, Enter, and Backspace).

Guess Distribution

Now we can move on to the coolest feature of the modal: Let’s show the user their statistics! In the total games they played, what was their guess distribution? This is divided into 2 parts: collecting the statistics, and presenting them.

Collect the statistics

The base code doesn’t save how many guesses the player used, so we need to add that ourselves… but how?

We want this information to be saved even when the user closes the website. An easy way to do so is to use local storage.

Local storage’s stored data is saved across browser sessions, with no expiration time. So, if there is no statistics object at all — create one. Then, read the previous statistics, update them with this game’s result, and save them in the local storage.

Important tip! When coding with local storage, you need the ability to restart it, for testing purposes. I have this line of code commented out and easy to reach in App.js: //localStorage.removeItem(“wordlestruckStatistics”);. Comment it in, refresh, comment it out, refresh — and you get a clean slate.

Let’s write some code!

We will create an array of length 7, to represent the possible statistics: User guessed the solution after 1,2,3,4,5,6 tries, or they lost. 0 will represent the loss.

The tricky part is that when the user loses, the turn is set to 6 (because it was their 6th try), so we need to adjust the index based on whether the user was correct: let index = isCorrect ? turn : 0;

Now — where should the code sit? It can sit in the Modal component, or in the Wordle component. It’s best to update the statistics when the game is finished, so we will add our new code in the “If the game finished” condition in the Wordle component.

Present the statistics

I encourage you to create a new React component dedicated to this section (< Statistics />) and call it from the Modal component.

First, you need to pull the data from localStorage. Then, save the number of total games in a variable. I present the data using a table element: The left column is the possible number of guesses (0–6), and the right column is the grey bar charts representing the distribution.

Now, those aren’t just any boring bar charts! In the original game, the distribution is written inside the bar, and the bar’s size depends on the distribution amount (but 0 isn’t empty). Also, the bar representing the number of guesses it took the user for the current game is colored green, not grey.

How do we achieve all that? For our bar charts, we’ll be using divs, so it’s a no-brainer to write in them. For the bar length, here is the math I did: Math.floor((stat / totalGames) * 80) + 8.

I divided stat-for-specific-guess-number by the total number of games to reach a proportional size. I multiplied by 80 because I didn’t want the biggest size to reach 100% width (but this is my personal preference). Lastly, I added 8 because I set the size of the 0 distribution to be 4, and I wanted 1 to be visibly bigger than that, even if there were many games played.

What about the green color? That property depends only on the turn number: let barColor = (index === turn) ? "green" : "rgb(107, 107, 109)";

Full code for the guess distribution section:

Screenshot:

In the screenshot above you can see I also presented information on total games played and % of wins. After completing the guess distribution, this part will be easy for you to add by yourselves.

Maybe we shouldn’t show the user his losses

Later in the development, I decided I didn’t want to present the number of losses. This can be achieved by wrapping the code inside the map in if(index!==0). Not the prettiest but the most minor change. If you do insist on prettier, there is another solution: Create a copy of objStatistics and .shift() it — this will remove the first item. There is no critical harm in shifting the original array, but it’s not a good practice because it can confuse the human eye reading this code. objStatisticsWithout0 is not confusing anyone :)

// to remove first element - number of loses.
let objStatisticsWithout0 = objStatistics;
objStatisticsWithout0.shift();

If you choose to go this route, you’ll need another index variable: We shift our array, which means array[turn] does not match the guess distribution in turn. I called mine guessDistIndex and set it to be index +1.

Lastly, before we continue, I found it useful to make the modal scrollable. You can easily achieve that by adding in index.css:

.modal {
  ...
  overflow-x: hidden;
  overflow-y: auto;
}

This is it for now. We will come back to the Modal after we go over the features that the rest of it depends on :D

Ok, remember when I said that in a future blog post, I’ll show you how to give your Wordle a cool theme? In the next section, I expose the theme of my Wordle because I can’t hide it anymore — it will appear in the screenshots.

My Wordle game is about Taylor Swift songs! The juicy part of a themed Wordle is creating the data set and adjusting the functionality to also work with solutions longer than 5 characters, and solutions containing more than one word. I will touch on those cool features in a future blog post, so stay tuned!

The Landing Page

All the Wordle clone projects start the game with the main game page (the one with the grid and the keyboard), but the true Wordlers (“Hi Wordler”) among you know that there is a grey landing page with “How to play” and “Play” buttons. This is what we will work on in this section!


The landing page

The structure of the page is pretty clear: Image, title, text, 2 buttons, more text. The tricky part is clicking the Play button and “moving between pages” while staying on the same page with the same URL, without routing!

We’ll start by creating a new component, which I called <Welcome />, and we add all the needed data there. You can present the date in this format by using:

var today = new Date();
var displayDate = today.toLocaleDateString("en-US", options);

Calculating the index, on the other hand, is a tricky part which I will touch on later! Keep it hard-coded for now.

Adding this new component requires some refactoring because it sits between App.js and Wordle.js.
App.js used to call Wordle.js, but now it will call Welcome.js, and Welcome.js will call Wordle.js.

Now, we want the page to show the Wordle component if the button was clicked. This means the landing page content disappears and the Wordle content appears. We achieve that by using React’s useState. That way we trigger a re-render to replace a value:

const [showWordle, setShowWordle] = useState(false)

So, on button click we set showWordle to be true, and we wrap the corresponding code sections so that they are dependent on the boolean value!

{!showWordle && (<landing page code>)} {showWordle && (<Wordle component>)}

Let’s see it in the Welcome component:

The Instructions Modal

It’s time to address the second button on the landing page. The modal itself is nothing we haven’t already seen, but the tricky part is showing it above the grid page!

In the original game, clicking the how to play button leads you to the same page as the play button, just with the extra instructions modal. Once you click the X button you’ll see the grid and the keyboard.


Wordlestruck’s instructions modal

So, you create a new component (<Instructions />) and put your code there. Where should you call it from? The structure will be similar to the landing page/wordle page structure, but from inside the wordle component!

We established earlier that both buttons will lead to the grid page, it’s just a question of whether or not the instructions modal will be shown on top. To achieve that, when we call the wordle component, we will pass a boolean parameter for the instructions page: {showWordle && <Wordle solution={solution} showInstructions={showInstructions} />}

That’s a good start, but it’s not enough. We also need to update the condition of showing the landing page vs showing the wordle component to take into consideration that new boolean:

Note that the initial state is to see the landing page, which means not showing the Wordle component, and not showing the instructions modal on top of the Wordle component. Then, if one of those booleans is set to true, we want to remove the landing page code and show the Wordle component.

So, we finished with the changes in the Welcome component, and we passed an additional parameter to the Wordle component. Let’s adjust the code to consider that new parameter!

Inside the Wordle component:

This time, different from the landing page logic, we don’t need to attach the Wordle code (grid, keyboard) to the !showInstructionsModal condition because we want it to exist underneath the instructions modal. That way, when we close the modal we will see the game!

I didn’t show any code for the Instructions modal, because it’s easy to implement considering it’s not our first modal in the project. You can do it! :D

Calculating the Game Number (Index)

At first glance, it may seem like the index is a tiny feature that appears only once on the landing page. In reality, this is a crucial part of this game, and it will be used throughout the whole game flow!

The index is accessed on the landing page, in the data set (to pull the word of the day), in the share button, and in various components to indicate if the user has played today or not. I know, most of those we didn’t do yet — which is why we need the index at this point!

When I started planning this function, I got overwhelmed: Every month has a different number of days, some years are leap years, and it even gets more complicated — Date.now() function returns the number of milliseconds since January 1, 1970, but there is no function to calculate the number of milliseconds since January 1, 1970, to another date that is not now!

At this point, I used the successful method “drop it for now and come back to it later”, and it worked! I googled it, understood all the parts (I can’t find the exact link but it was from Geeks for Geeks), and was determined to write it myself!

How to calculate the number of days between two dates?

First of all, the trick is not to calculate days between two dates, but to calculate the number of days in each date (# of days in 2023 years, # of days in 3 months, etc) and subtract the two!

Note that when you calculate years and months, you don’t use the direct value, but subtract 1: If the date is March 13th, 2023, we don’t have all days of March, nor all days of 2023. This is why you see in the code above year-1 and month-1. The for loop that is calculating full months in days is running from 0 to month-1 (excluding), which matches the monthDays array: January’s amount of days is in cell number 0, not 1.

Addressing Leap Years

The leap years aspect is not as complicated as it initially looks. A leap year is a year that has one additional day at the end of February. So, once you know how many years are leap years — that’s the amount of days you add.

What makes a year a leap year? According to Wikipedia,

“This extra leap day occurs in each year that is an integer multiple of 4 (except for years evenly divisible by 100, but not by 400)”.

In code, it simply means if(year % 4 === 0 && !(year % 100 === 0 && year % 400 !== 0)).

This is how I wrote the function:

Don’t forget that leap years have their extra day in February, so keep the last year out of this calculation, and add a day only if the month is after February (month > 2).

Just to make sure we tackled all possible edge cases, let’s think about what would happen on Feb 28th and Feb 29th.

Feb 28th: month is not >2 then leap year doesn’t impact our calculation.

Feb 29th: month is not >2 then leap year doesn’t impact our calculation in this part. When we add the day to the sum of total days, we will add 29, so that’s ok.

This caused a bug!

I’ve discovered the hard way that Mac’s Safari and Firefox browsers do not agree to render the starting date in the current format. Chrome is more forgiving, which is why it took me a while to discover it.

The acceptable format is new Date(2023,11,8) — This means the month range is 0–11! (and only the month, not the other variables, I double-checked!)

It’s more human-friendly to address the months as starting from 1 so it’s better to adjust the month variable (and leave yourself a comment as a reminder!):

let startingDate = new Date(2023, 11, 8); // YYYY-(MM-1)-DD
...
let month = date.getMonth() - 1;

Now, you can optimize this by hard coding the start date. We set it only once, no need to calculate it again and again. I chose to keep the calculation because I haven’t launched my game yet, so the starting day is about to update.

If you are passionate about optimizing the program, you can calculate the first 2023 years in days only once and save those in variables! I won’t do it here, so let’s call this your homework ;)

Now that you have the game index calculated, you can modify the word pulling so that there will be only one word per day, based on the date!

The base code has the line: const randomSolution = solutions[Math.floor(Math.random() * solutions.length)], which gives a new word every refresh of the page. If we change it to be const todaysSolution = solutions[game_id], where game_id is the index calculated by our function, you could add the exciting sentence “A new puzzle is released daily at midnight” to your Instructions modal! :D

Last but not least

We need to take a closer look at the index ranges: Let’s say we have 200 solution words. The solutions data set indexes will range between 0–199, because of the convention that an array starts at 0. The game indexes (numbers) should range between 1-infinity. This means we have a mismatch both at the start and at the end of the index ranges. Also, if we want the game index to start at 1, we need to revisit our getGameIndexofToday() function. We will fix it all in this section!

App.js:

This is our code at the moment:

const game_id = getGameIndexofToday(); // 0-infinity, but we don't want "game no.0"
...
const todaysSolution = solutions[game_id]; // 0-199

At some point, your game index will be higher than your dataset size. If you have 200 words in your data set, and it’s day 201, your app will crash. You can cover this case simply by adding const solution_id = game_id % solutions.length; instead of using the value coming from the index calculation function directly.

That is not enough. Once we fix game_id to start at 1, the first solution word will be the second item in the array (solutions[1]). We need to correct that! Therefore, we'll use const solution_id = (game_id-1) % solutions.length; instead.

num_of_days_from_date.js:

Lastly, let’s look at getGameIndexofToday() function. We built it so that we calculate amount of days in each date, and then subtract. What happens on the first day? The result will be 0, which is not a good starting index for our game (game number 0?). This happened because usually when we calculate something from start to finish, we exclude the end value. In our case, we need to include it. Let’s fix the code!

Not only do I add +1 before I return the value, but I make sure to rename the function to make it explicit that it includes the end day.

Now our indexes are matching!

const game_id = getGameIndexofToday(); // 1-infinity
...
const solution_id = game_id % solutions.length; // 1-200
const todaysSolution = solutions[solution_id]; // 1-200

This is a good place to close today’s tutorial. There are more features to add, but we’ll do them in the next blog post!

Let’s give a quick reminder of what we created today:

A beautiful and functional virtual keyboard
A cool statistics modal with the user’s guess distribution
An instructions modal
A landing page with the real date and game index!

This is a lot! But a lot of fun as well. 😀

In the next blog post, we will create the following features:

The share functionality
Different screens for whether the user played today or not
Header buttons for statistics and instructions

Exciting news to all the swifties out there — my Wordlestruck game will launch at the end of this blog post series!

Blogging is my hobby, so I happily spend time and money on it. If you enjoyed this blog post, putting 1 euro in my tipping jar will let me know :)

Thank you for your support!


https://cupofcode.blog/

Weekend Coding: Daily Email Sending Through Code

Cup of Code — Sun, 31 Dec 2023 14:53:17 +0000

A fun end-to-end project, from requirements to the final product

A few weeks ago I went to visit my parents and we sat down for dinner. While we were catching up, my dad mentioned that every evening he goes on his computer, and sends my mom an email with several Excel files she has been updating that day, for backup. I almost choked on my food! Every evening? Manual backup? In 2023?!

My first optimization idea was to swap Excel for Google Drive’s Sheet. This idea was declined because Excel’s user experience is so much better, that it is non-negotiable. Then my next suggestion was to set up a Google Drive syncing directory on the desktop. This idea was rejected as well because apparently, my dad had a resiliency threshold so high, that he did not want to put all his eggs in one basket: He wanted a multi-company backup, and he is achieving it by sending the files to two different email addresses, hosted by two different email providers.

Now, let me be clear: I know that Google Drive is good enough for my mother’s precious spreadsheet files. This double-company backup approach is cracking a nut with a sledgehammer. With that said, this is what the customer wants, and I was excited to discover how to automate email sending, so we started laying down the requirements!

(Initial) Requirements

I was naive to think that we had everything covered at our first discussion. Later on, we realized that every little adjustment we decided to make along the way, resulted in additional, more specific, requirements, which I will cover later.

Now, we know the final goal is to have a daily email sent from my father’s email address to my mother’s email address but to be exact, we want:

Daily automatic email from X@hotmail.com to Y@gmail.com.
Easy manual send upon request as well.
Attaching multiple Excel files, while the names of files may vary.

Therefore, I had a few things to figure out, with which I could create the whole program:

Send an email through code.
Attach files to that email.
Extract the code to .exe file.
Automate a daily send.

You might wonder why step 3 is necessary. There are two reasons for that: First, I want the program to be user-friendly for non-developers (like my parents). Second, I want to hide the code because, as you will see soon, it contains a Gmail app password. I am assuming .exe files can be disassembled back to code, but at least this is an extra layer of protection.

Luckily, I wasn’t the first one on the internet trying to code email sending, even with files attached! Nor I was the first to extract a .exe file from Python, and I was definitely not the first to want to automate an action on Windows. I will link useful videos at the end of the blog post. Now, let’s get started!

1. Send an Email through Code

I will admit that when thinking of automation and scripts, the first thing that came to mind was the shell — using the command line. But a quick YouTube search showed Python is a more friendly way, and that is what I went with.

Whatever coding language you are choosing, there is one thing you’ll need to use in order to send out emails: SMTP, or Simple Mail Transfer Protocol. This is how you send email messages between servers. If you will be using a Hotmail email address, you will need to authenticate through the Hotmail SMTP server.

This is the first point where our initial plan shifted. Setting up an SMTP connection to Gmail, which is what I used, requires you to set up a 2-step verification in your account. A 2-step verification is verifying who you are with more than just your password (usually sending code through email/SMS). Higher security for your account is never a bad thing, but the fact that this is a requirement shows that connecting to an SMTP server using your email address credentials is compromising your account.

I want an email sent, and I don’t mind from where (well, this will come back to bite us, stay tuned ;) ) — so I can create a new account, one I don’t mind compromising its security, which only purpose will be my new cool software!

The way you set up your SMTP is by using app passwords. An app password is a 16-digit passcode that gives a less secure app or device permission to access your Account. Here are instructions for setting up an app password on Google.

Let’s start coding

If I’m being honest, the first thing I did was refresh my memory about working with environment variables file. Those are variables whose value is set outside of the program. This is a critical step because the customers will use their own email recipients and their own file paths, and they shouldn’t need to open the code in order to set those.

ENV_VARS.env:

shoe-maker.py:

So what do we see in the screenshots above? First of all, we see that I called my program shoe-maker.py, and you might wonder why. I will tell you at the end of this blog post.

lines 6–7 are for directing the program to the location of the .env file. By the way, you can call it whatever you want, but ENV_VARS is the standard.
lines 9–12 are saving the values we extracted from the file into local variables. Grabbing the values externally every time we want to use them is expensive.
Lastly, lines 1–4 are all the imports the IDE complained you are missing. ;)

Wait a minute — PASSWORD in the env vars?

Let’s reflect on that screenshot for a minute. Despite being worried about compromising the email account, the app password is out and about in the env vars file. How come?

This is because I was mixing up who my audience is. I knew I was going to write this blog post, and I wanted to give you (people from the software field) the software with which you can set up your own email sender (and app password).

The problem is, that this is a different goal than providing a final product aimed at “real” users. Real users shouldn’t be exposed to any password, and it’s better to have as few env vars as possible so that the user will have fewer opportunities to make oopsies in the configuration.

So, from this point forward, I am setting the email address and password within the code.

Straightaway, after figuring out the env vars, I stopped and moved to step 3. I did it because I wanted to ensure I could successfully export the .py file into a .exe file before I invested any additional time writing in Python. As I mentioned before, .exe file was an essential goal of this project, and I was willing to switch to another programming language if I couldn’t achieve it there.

3. Extract The Code to .exe File

Thanks to YouTube, I found this tutorial. Long story short, you need to install PyInstaller. Pretty easy to do, I’ll just mention that for me, when I used the regular pip install pyinstaller I got an error (CommandNotFoundException) and was able to bypass it with py -m pip install pyinstaller.

Could I’ve gone to figure out how to make pip the command work directly without the py -m? Yes. Was I going to do it while I was in the fun flow of my email-sending program? Nope. :D

Speaking of errors, when I was trying to verify the installation went smoothly with the -v flag, I got an “I don’t know what are you talking about” error, and it took me too long to realize it was because it requires a camel case in the word PyInstaller:

Once you have it installed, you can create the .exe file by running py -m PyInstaller --onefile --icon=image.ico my_file.py. This way you create a my_file.exe file with the icon of your choice, using the code from my_file.py. Note that it has to be a .ico file. I used this website to convert my image from jpg. After a few seconds, you’ll see a new dist folder in your directory with your new .exe file.

Now that step 3 is behind us — we can continue with our code!

Back to Sending an Email Through Code

Sending a simple email

To focus on email sending, I temporarily set the variables in the program, instead of moving between files. You can see that in line 2 in the screenshot below. Actually, at this stage, I didn’t even include files — I just wanted to see that I could send the most simple email.

Lines 4,17 are for catching exceptions, which I will touch on later.

I used the email-sending code from this video. You might noticed I renamed my sender email from EMAIL to GMAIL_ADDR. This is a good reminder that the code is set up to use the Gmail server (line 5). If my program was meant for people from the software world to use, setting up their own server (that might not be Gmail), I wouldn’t have hard-coded it. But for my use case, even if I have more users within my family, Gmail is good enough.

So, I started the connection (lines 6,7), logged in with the credentials (line 8), wrote a simple email (lines 10–13), and sent it (line 14). Lastly, on line 16, I notified the software user that the email was sent successfully (if we were on the happy path).

This is what it looks like on the recipient’s side:

Sending email to multiple recipients

This is nothing complex, I am just showing the order in which I progressed. Sending the email to multiple recipients is easily done by a simple for loop, over the ‘recipients’ variable, which is an array of email addresses (strings).

Testing usage of env vars file

I know the pieces work well separately, but it doesn’t guarantee they will work well together too. So, at this stage, the program runs smoothly when I set the recipients in the env vars file, but later on, the env vars are going to cause one of my biggest bugs. Any guess at which point?

Error handling

I know that adding error handling is a bit excessive, considering I only plan to have between 1–2 customers (who are obligated to love me unconditionally). What can I say? I have a standard for coding, and I can’t half-bake it. Now that I think about it, graceful failure is important in this case because the user might update the env vars file incorrectly and will need some guidance in order to fix it.

So, what do we have here?

In lines 3,5 I added colors because I wanted to differentiate between the error message and the instructions for the user.
In the instructions (lines 6–8), I informed the user that this process failed and that they would need to send an email themselves. Also, I invited them to contact support, and lastly — to close the program.
I split the instructions into 3 lines because it’s easier to read for the developer (rather than to scroll right).

The reason I added the input() in the end is so that the user will have as much time as they need to read the messages.

I’ve added more error handling later on, but let’s continue the story in chronological order.

Refactoring

This refactoring is coming from the tutorial I linked earlier. We have two changes to make:

First, If you swap line 10:
with smtplib.SMTP(‘smtp.gmail.com’, 587) as smtp: with
with smtplib.SMTP_SSL(‘smtp.gmail.com’, 465) as smtp:
then you can remove lines 11,12 because they now happen behind the scenes.

What exactly is happening behind the scenes? SMTP_SSL connects to the server using a secure encrypted SSL protocol (SSL stands for Secure Sockets Layer,), while SMTP doesn’t, which is why we need to add the starttls() function right after connecting. ehlo() function is for the server to say hello and identify :)

Note that the different connections use different ports (587 vs. 465).

The second refactoring is happening in the message content: Swapping
msg=f’Subject: {subject}\n\n{body}’
in line 18 with a more comfortable structure. This is done by using EmailMessage import.

Using the new import, we set the message object like this:
msg = EmailMessage()
msg[‘Subject’]=subject
msg.set_content(body)

The most important thing in refactoring is checking we didn’t botch anything that used to work fine :D

Still working! Now we can progress to step 2.

2. Attaching Files to The Email

Let me remind you, that the whole purpose of this project was to send Excel files. Adding an attachment to the email is pretty straightforward: Given the file path, you gather the variables needed for the msg.add_attachment(…) function:

And for multiple files — just wrap this code in a loop.

That’s cool if we have the paths in the correct form. Unfortunately, it turns out that gathering the paths from the env vars file wasn’t as easy as I thought it would be:

Encountering My First Major Bug

What is this error saying? The error is rising from the beautiful ast.literal_eval(..) function. This function parses strings representing types, into the actual type (here we use it for arrays). In this case, the function can’t parse my file paths string from the env vars file because of Unicode escape errors: My file path location (C:\Users\…) triggers the function to convert the string to the format of a 4-digit hexadecimal code point (for example, “ABC” becomes “\u0041\u0042\u0043”), which makes the path absurd.

One solution for this problem is adding the r prefix to all the paths (r"myfriend@gmail.com"). This r stands for “raw”, and with it, the string will be read exactly as it is written. I didn’t want to go that route because I don’t think it’s a good practice to “force” the user to adjust to my program's limitations. The users should add file paths as they see it in the file explorer and the conversion to the right format should be done inside the code.

With that in mind, this is the solution I wrote:

First, I save the env var as a string (including brackets, commas, ditto marks (“), and white spaces). Then, I removed the brackets and split the string by the commas. Now I have an array of strings, but they still begin and end with ditto marks. This brings me to my final line of code: Strip each of the strings of any ditto marks and white spaces. In case you wondered, I’m handling white spaces because some people like to write lists with white spaces after the commas.

Note that the paths’ backslashes have converted from \ to \ automatically, in order to avoid those escape characters I mentioned before.

So far we have sent an email, added attachments, and exported the program to .exe… At this point, we can put the .exe file on the desktop and send emails at the click of a button! All we have left to do is to automate that task.

4. Automate a Daily Send

Luckily for us, Windows has a program called Task Scheduler. It’s already on your laptop! It took me a bit of learning to understand how to get it to do what I wanted it to do, but I made it :D

This is what you need to know: First, it is good practice to create a directory for your tasks (“My Tasks”) because other scheduled tasks are running on your computer and you don’t want to mess with them!

Once you are in your new directory -> click the Create Task button.

Create a new task

Now you will see a window with 5 tabs, we will go over them one by one:

General: Give the task a good name, and tick off the “Run whether the user is logged on or not”. This is because we want the task to run daily, regardless of whether you are on your laptop or not.
Triggers: What will initiate our .exe file? The time of the day. Choose your most comfortable hour, set it to “daily”, and don’t forget to tick off the “Stop task if it runs longer than” option. Our program only takes a few seconds and if it’s running longer it means it’s stuck — and we should kill it.

Actions: This is the juicy part! Here you put the path to your .exe file. Let me save you some precious time, and tell you this is where the program, as it is now — will fail.

The issue is that the task scheduler doesn’t run the program from the same directory of the file. This is what I mean: Instead of running the command like this
C:\Users\ Desktop\shoe-maker > py .\shoe-maker.py

Imagine running it like this:
C:\Users> py .\ Desktop\shoe-maker \shoe-maker.py

We coded the env vars file location relatively, and we don’t know where the task scheduler is executing the program from, therefore — we need to add the full path as an argument.

Conditions: It’s important to tick off the “Wake the computer to run this task” option so that the automation doesn’t depend on the user’s laptop usage.

And that’s it! I recommend setting it first to 1 minute instead of 1 day just to test that it works :)

Are we done now? Not exactly.

5. Privacy Gap Discovered

Remember earlier, when I easily made the decision to open a dedicated Gmail account, in order not to compromise any existing Gmail account’s security (due to usage of app passwords)?

It looked like a no-brainer solution, and I didn’t realize that using an external Gmail account meant exposing the data (the precious Excel files) to anyone with access to this account (like me!)

Did you notice how WhatsApp keeps reminding you that your messages are end-to-end encrypted, and your message content stays between you and your conversation partner? I aim to provide the same.

Unfortunately, I couldn’t find a way to do so. This is a compromise I’m not happy about, but I still prefer that over risking an actual active email address that has gathered years of data.

A nice settlement had been setting on Gmail a filter to send every email that is not part of the development (aka not “from me to me”) directly to “Trash”. Messages will sit there for 30 days and then will be automatically deleted. In the real world, this is not a good enough solution, but my 2 customers were okay with the terms and conditions :)

6. Final Touches

Now that we have the functionality, we just need to add some final touches, to make it cool:

Icon: That is the first thing I added, as soon as I set up the .exe export, but maybe for you, this step is coming only now.

Meaningful email title and body:

I started by creating a new env var, to let the user title the email with something meaningful. In addition to that, I created a default title, in case they don’t want to set it up (lines 2,4).
On line 6 I set a fun email body.
On lines 8–16, I used the option to add alternative text, in HTML format.

Additional error handling: First, I considered what errors the user might be creating. Then I caught them, and added instructions:

Once you’ve added the code to print the error, it’s pretty easy to start covering specific cases: Make a mess and copy the error that comes out. Then, add it as a case in the catching :D

Adding a progress bar: It’s nice to keep the user in the loop and show them the progress. Why should we message the user only when things are bad? ;)

7. Developer Troubleshooting

Here is a short list of errors you might encounter that didn’t come up in the blog post flow:

(334, b’UGFzc3dvcmQ6'): This is an SMTP authentication error, and it comes up when the email or password you entered is incorrect. The reason behind it might be a typo in the actual email and password, but it can also be a typo in the env vars variable names for those fields! So watch out for that.
PermissionError: [WinError 5] Access is denied: exe window was open: This one happens when you try to create a new .exe version (which overrides the previous one), but you don’t exit the program. Is this the reason I added the You may exit this program. instruction to my error handling? Maybe :)

8. Feature request: Add directories (as well as direct file paths) to be sent

Our program is already able to send files given their path on the computer.
The plan is to loop over the given directory paths and create file paths from those. Then those file paths can work with the rest of the program.
We'll start by creating a temporary file, which will be our playground to get from a directory path to a list of file paths.

Now, we'll go to the section in the code where we attach the files and check if the path is a directory (there is a function for that!), and if it is, get the file names, and for each: create a full path and attach the file.
This will require exporting the code for attaching the file to a separate function, and calling it in each of the scenarios: if the path is a file or a directory.

Why Shoemaker?

First of all, I’m glad that you made it this far in the blog post! Hopefully, this inspired you to start your own little project. It surely made me :D

I named this project The Shoemaker because of the saying “The shoemaker’s children go barefoot”. I found it fitting to the scenario of the software engineer's parents doing daily tasks that can be automated via code. ;)

Useful links:

If you read this, you’ve reached the end of the blog post, and I would love to hear your thoughts! Here are the ways to contact me:
Facebook: https://www.facebook.com/cupofcode.blog/
Instagram: https://www.instagram.com/cupofcode.blog/
Email: cupofcode.blog@gmail.com

Create Your Own Website — Is It Worth It?

Cup of Code — Sat, 26 Aug 2023 20:31:57 +0000

All the pros and cons, all the whys and the hows — in one place :D

Before the How — let’s start with the Why: Why should you create a website? Because it doesn’t matter what you do for a living, even if you don’t need to attract customers in your profession, it’s pretty cool to be able to send people to a space that represents you. So, whether it’s a one-pager resume with contact information, or a collection of blog posts, recipes, or any hobby that you want to share with the world — a website of your own is a nice way of doing that.

A Bit of Background

I started Cup of Code blog on Aug 2019 (yes, it just turned 4 years old :D). I wrote blog posts on Medium, and a year later I saw I loved it, and was ready to take it to the next level — Create my own external website! So, without any previous experience but with lots of motivation, I maneuvered my way to a functioning website:

One day, while I was dealing with moving apartments — it turned out my website needed to move too! I suddenly got a surprising $400+ charge from my hosting service! This made not just me worried, but my bank as well — which resulted in them blocking my credit card! (You know how it goes, trouble never comes alone)

Why did I not expect this $400 charge? This is a wonderful segue for diving into my website hosting tips and tricks!!

Important note: While doing the research for my own case, I found it hard to reach unsponsored content — which made me doubt the integrity of the reviews. Let me assure you — this blog post (and my blog in general) is NOT sponsored. I blog as a hobby, and I appreciate the autonomy of writing whatever I want, whenever I want.

Now we can start :D

Tip #1: Know All The Costs!

Owning a website is not just pure hosting. It is composed of several parts that I will go over here in the blog post. Learn all the moving parts, and the total cost of your website before deciding if you are willing to go that route!

I will divide it into 3 parts:

Hosting: HTML pages, Pictures… Where do your files sit? There are many hosting services out there, and it can be overwhelming. I’ll talk more about it in tip #2.
Domain: What do people type in the URL to reach your website? Are you willing to spend money on a “clean” domain, or are you okay with a domain like https://cupofcode.medium.com/ (under Medium) or https://graphcsv.netlify.app/* (hosted with Netlify)? More about it in tip #3.
*I created this website for my Alicecode class to demo a project with CSV files ;)
Website features: If you decide to use WordPress, you will see there are a bunch of plugins you can purchase. It is tempting, and you should count it in when calculating the total costs of your website. I’ll explain it all in tip #4.

It is important to mention that all those parts are subscriptions— you’ll be charged periodically forever! Well, not really forever, but as long as you want to keep your website up. In the end, I’ll share my yearly costs.

Tip #2: How To Choose The Right Hosting

This is where you can really get decision fatigue: Not only there are so many hosting services, but each of them has multiple plans! TOO MANY OPTIONS!

All plans have an introductory price and then a full price after the first period. My most important tip here is to not be fooled by the introduction price - Pick a plan that you are willing to pay the full price for. Yes — This is where the surprise $400 bill came from!

I first started with Bluehost, because they are big, reliable, and beginner-friendly. I chose the Basic Web Hosting *and paid ~$100 for the 3-year plan. That’s a nice price. The problem comes with *“Auto renews at the regular rate”, and $11/mo was too much (for me).

So, despite the inconvenience, and after some research, I migrated my website to the Single hosting plan in Hostinger, because they are also big and reliable, and $4/mo is the cheapest I could find.

I attached here only 2 hosting services with a total of 8 plans, and it is already looking very confusing. You may be wondering why don’t I pay an extra tiny $1/mo and get 100 websites — Well, because I don’t need 100 websites! Also, don’t forget how quickly this $3/mo turns into $7/mo forever. Lastly, even if you do eventually need additional websites, you can always upgrade.

Whatever hosting service you choose, you’ll be able to save some more coins with promo codes. You can find plenty on YouTube: Just search for a review of the hosting service of your choice and most chances you’ll land on a sponsored video ;) In Bluehost, I got free Domain name registration + domain privacy&protection for 1 year (which saved me ~$50), and in Hostinger I got 10% off.

Besides The Plans

So I talked about costs, but do you know what is even more important than saving some money? Good customer service! No doubt you are all very smart, but it doesn’t mean you need to figure out things by yourselves: When you are tackled with a surprise charge, or when you have problems migrating your website to another hosting service… The smartest thing you can do is save time.

I don’t know much about others, but I had a wonderful experience both with Bluehost and Hostinger. I got a human being in the chat within 3 minutes every time!

The last thing to mention when it comes to hosting is choosing the right server location. It is hard to predict where will your traffic come from, but a good start will be your own location (because your audience will probably begin with your circle of acquaintances, right?). I don’t remember an option to choose when I set up my hosting with Bluehost, but with Hostinger I chose Europe and it’s loading faster :D

Tip #3: Don’t cheap out on domain extras!

Extras? Wait, what are the domain basics? Don’t worry, I’ll cover everything you need to know about the domain here.

When you choose your domain provider, you’ll see a text box where you fill in your future website’s name, and next to it you’ll have a drop-down with possible suffixes: .com, .blog, .org, etc.

I’m no expert, but my experience shows that the more popular suffixes (.com for example) will be more expensive. I liked the idea that you can tell my URL is a blog before you even click (https://www.cupofcode.blog) but be aware that people trust “.com” more. Invest efforts in choosing the right URL — because this is where people will go when they want to reach your website, and it’s not something you should plan on changing.

Lastly, as I mentioned before — some hosting services will give you a free domain for a year, or something similar — so keep an eye out for that!

Now that we finished with the basics — let’s talk about the extras: You choose your perfect URL, you accept the price it costs, click next — and discover there is an optional extra: Domain privacy+protection.

All the domains (no matter the provider you choose) are managed by an organization called ICANN. They are required by law to publicly list the contact information of every domain owner in a big database called WHOIS. Your domain provider can mask this information and keep your privacy. I think this is important, but you do you!

Transferring Domains

(Feel free to skip this part if you are a first-time user)

When you move from one hosting service to another, you can (but don’t have to) move your domain registerer as well. I decided to do so because I find it convenient to have it all under one roof. Remember I told you I moved apartments? Thanks to this video I learned you can not migrate your domain if you changed your contact info (that includes your address) in the last 60 days! So I changed the address after… Phew! Another piece of information he noted in that video was that migrating automatically adds a year to your expiration date (which means you pay money) — I don’t find this critical, but good to know.

Tip #4: You can write code, but should you?

Now that you have hosting and a domain name, you need to take care of the content! When I started — everywhere I looked, people used WordPress. WordPress is a content management system (CMS), which means it looks like this:

When you log into your WordPress account, you’ll see this dashboard. The magic happens when you add good plug-ins, some are free and some are worth the money.

There are many WordPress tutorials out there, you don’t need me to add another one. I’ll just tell you that for some annoying reason, there are 2 WordPress-es out there: WordPress.org and WordPress.com. .org is the open-source, free-to-download management platform I was talking about, and .com is a fully hosted platform, meaning you get everything in one package. If you are getting hosting separately — WordPress.org is your answer.

Now, I know how to build web pages but chose not to. I wanted to invest my time in the content. Therefore, I’ve decided to use Elementor, which is nice because you can easily set it up with drag&drop elements. They offer a free tier, but I got the Pro Essential plan for ~$50/yr because of widgets such as forms and posts. The comparison between Elementor plans can be found here. With Elementor Pro, after I followed this tutorial, my website turned out beautiful and I no longer need to make adjustments every time I publish a new blog post.

Now, I know how to build web pages — but I chose not to. I wanted to invest my time only in the content. I’ve decided to use Elementor because I got a recommendation. Maybe they have a free tier, I use the pro version for ~$50/yr. The fun part here is that you can easily set it up with* drag&drop *elements, and after I followed this tutorial, My website turned out beautiful, and ever since doing that, I don’t need to make any adjustments every time I publish a new blog post.

Free Alternatives

With that said, I have a friend who now asked for my help with Jekyll, which is a free platform to build clean static pages. Who knows, maybe I’ll discover this is a better option for a blog (Stay tuned!). Note that this requires a learning curve and time investment compared to Elementor.

Yup, it always boils down to time vs. money ;)

Tip #5: Don't fall into “first-year free” traps!

Soon enough after the migration from Bluehost to Hostinger was done, I received an email encouraging me to use my “free email address”.

Their email was written in a pretty convincing way: “Hi there, It’s tough to be taken seriously with a personal email address such as proconsulting1@gmail.com. … The good news is that your hosting order comes with a free business email! Here are a few reasons why it pays to use a business email at your domain…”

So you probably think “Hey, why not try? It’s free”… Because this is a trap!! Think about it, an email address is an actual address — it is where people go when they want to reach you. It is hard, maybe even impossible, to redirect all the emails to a free alternative once the free year is over. I wouldn’t want to be trapped with renewing this email address forever, even for a small price.

Tip #6: You will spend a “learning fee” — and it is ok!

This is wonderful advice from my Mom. She wasn’t in on my Website endeavors, she was saying it about my relocation— but it fits here as well: You are doing something new, and you don’t know what is the correct/fastest/cheapest way yet — and it is ok! This is your learning fee, embrace it and have compassion for yourself. :)

To conclude... Is it worth it?

Unfortunately, you are a grown-up and therefore this is a decision you will need to make for yourself. Many hobbies will require costs if you want to step it up, and blogging isn’t different. My current yearly cost is ~$120, broken down as follows:

Hosting: ~$24/yr for the next 4 years, and then ~$48/yr
Domain+privacy+protection: ~$45/yr (the domain alone is ~$23)
Elementor: ~$50/yr

You can also pick and choose where you spend your money — I gave free alternatives throughout the blog post.

My guideline is — if you are curious about it, and you can afford it — SURE! Give it a go. If you don’t like it, it isn’t really forever, despite my using this word a lot today. You can always cancel the automatic renewal -> Here is another tip for you, so you won’t receive an unexpected $400 charge!

Introduction to AWS — Cloud Best Practices

Cup of Code — Mon, 08 Aug 2022 00:00:00 +0000

The cloud is useful for flexibility, efficiency, scalability, and security — especially when you use it correctly ;)

From the person who gave you varieties of only part-ones (Angular, Networking), I’m very excited to present the seventh and last blog post of the AWS introduction series!!

Today we will talk about a variety of techniques to use AWS services to meet your needs in the best way, including scalability, automation, and more! Here is a table of content for easy access:

The content of this blog post is based on the course AWS Certified Cloud Practitioner 2020 by a cloud guru and on the whitepapers Well architected framework and AWS overview.

High Availability and Scalability

Let’s start with scalability and see how it connects to high availability.

Scalability means that a system can handle greater loads by adapting. There are two types of scalability: vertical and horizontal (=elasticity).

Vertical Scalability is scaling up/down: Upgrading/downgrading the instance, like getting a new fancy phone.
It is common for non-distributed systems, which makes sense because this is when you can not add a machine and split the data.

—

A side note about distributed systems:

A distributed system is a computing environment in which various components are spread across multiple computers on a network. In a non-distributed system, all the parts of the system are in the same physical location.

As I see it, if you need to use the internet to communicate between your components — it’s distributed. That means, that almost all of the systems we know today are distributed, maybe besides some projects you did as a freshman in College.

If you wanna learn more about distributed systems, I stumbled upon the lecture Why are Distributed Systems so hard? A network partition survival guide — Denise Yu and it’s fun and full of cats!

—

Horizontal Scalability is scaling out/in: Adding/ removing instances, like getting extra 3 phones! (identical to your original phone)
This one is common for distributed systems, and easier than vertical scaling mainly because there is no hardware limit to reach (you can always add another instance).

High Availability is using different locations and that’s why it goes with horizontal scaling. It means running your system in at least 2 distinct data centers (=availability zones), so you could survive a data center loss.

Automation

There are 3 ways to automate in AWS: Serverless management and deployment, infrastructure management and deployment, and lastly — alarms and events.

Serverless Management and Deployment

Ideally, you would like to get everything serverless because then you don’t need to worry about infrastructure. I have to be a geek for a second and mention that “Serverless” is a misnomer (inaccurate name) in the sense that servers are still used by cloud service providers to execute code for developers, but if a server goes down it’s not on you to replace it — all you worry about is deployment.

Infrastructure management and deployment

AWS offers services that will scale automatically, and all you have to worry about is your code. For example:

Elastic Beanstalk is a service for deploying and scaling web applications and services. The deployment, from capacity provisioning, load balancing, and auto-scaling is done automatically.
EC2 auto-recovery: Automatic recovery migrates the instance to another hardware during an instance reboot while retaining its instance ID, private IP addresses, Elastic IP addresses, and all instance metadata.
AWS systems manager, which allows you to safely automate common and repetitive IT operations and management tasks.
WAF (Web App Firewall) security automation: Automatically deploys a set of AWS WAF rules that filter common web-based attacks.

Alarms and events

This is the first thing that pops into my head when I think about automation. A few AWS examples:

CloudWatch Metric Alarms: Classic automation, get notified when what you’re waiting for/ afraid of happens, instead of constantly checking.
Cloudwatch Events: Those are system events that describe changes in Amazon Web Services (AWS) resources. For example, get a lambda function to do something when someone uploaded a picture to your S3 bucket. It’s a way of having your environment proactively respond to events.
Lambda scheduled events: Schedule events for every defined period of time. For example, run the lambda function at midnight, or every Monday.

Loose Coupling

It’s easier to talk about loose coupling after explaining tight coupling, so let’s start there. Tight coupling means classes and objects are dependent on one another. So, when you change one component it could break your system quickly. With loose coupling, you design independent components, and you increase the flexibility and the resilience of the code.

There are three ways to achieve loose coupling:

Well-defined interfaces: For example, the AWS API gateway allows you to create your own APIs and expose them to the public internet. Interfaces are loosely coupled because the only knowledge that class A has about class B, is what class B has exposed through its interface.
Service discovery: Amazon ECS services can be configured to use Service Discovery. Service discovery uses AWS Cloud Map API actions to manage HTTP and DNS namespaces for your Amazon ECS services.
Asynchronous integration: When your components are loosely coupled, you can work with queues, and when one instance fails, for example, another one can come up and pull requests.

Anti-Patterns in DBs

I have a blog post dedicated to various options of databases in AWS, but when speaking of cloud best practices, it’s also important to mention when should you use what.

Relational DBs

Visually, they look similar to excel sheets: Tables, rows, columns, and fields. You interact with your DB using SQL = Structured Query Language.

Just a reminder, AWS’s Relational DB is Amazon RDS(Relational Database Service), and it lets you choose between 6 different DB services, one of them being Amazon’s Aurora.

You should NOT use relational DB when there is no need for joins or complex transactions. For example, If I plan on pulling information per customer (like latest purchases of X), or if the fields between entries are prone to vary (entries have some different fields), non-relational DB is the way to go.

Non-Relational DBs

Non-relational DBs, such as AWS’s dynamo DB, are using NoSQL. Fun fact, NoSQL actually means “not only SQL”, and not “no SQL”.

The difference from relational DB is that the columns in the table can vary, and this will not affect other rows in the DB.

You should NOT use non-relational DB when the work requires joins or complex transactions. By that, I mean queries like “Get the concatenation of fields A and B for every customer whose last name starts with C and purchased in the last D days.”

Data Warehouse

Amazon’s data warehouse is called Redshift. A data warehouse is an information system that stores historical and collective data from single or multiple sources. While the database is designed to record data, the data warehouse is designed to analyze data.

Now, to reach our anti-pattern, we need to introduce two more terms: OLTP vs OLAP. OLTP (OnLine Transaction Processing) pulls the entire row as RDS does, and OLAP (OnLine Analytic Processing) pulls in large numbers of records.

You should NOT use the data warehouse when you want to perform OLTP queries! You should also NOT use a data warehouse when you can use a data lake. What is a data lake? I’m glad you asked!

Data Lake

A data lake is an architectural approach that allows you to store massive amounts of data in a central location so that it’s readily available to be analyzed. Since data can be stored as-is, you do not have to convert it to a predefined schema, and you no longer need to know what questions to ask about your data beforehand.

Data lake vs data warehouse

Reason for storing data: undefined vs pre-defined.
Data is left raw until needed vs processed and ready to be queried.

Amazon S3 is a great place to create data lakes, you can use Amazon Athena (serverless interactive query service) and run SQL queries on your data.

Removing Single Points of Failure

A single point of failure refers to one fault or malfunction that can cause an entire system to stop operating. Hopefully, nothing will ever break, but that’s not realistic. The approach here is that when something does break, it does not shut down the whole system.

Here are a few ways to prevent a single point of failure in AWS:

Introducing Redundancy

Redundancy is a system design in which a component is duplicated so if it fails there will be a backup. A good basic example of it is in the AWS infrastructure — Availability Zones!

Availability zones (AZs) consist of one or more discrete data centers, each AZ with its own power, networking, and connectivity, and is housed in separate facilities. Each region has multiple AZs, so if something happens to one of them, it’s far enough from the rest, and they can be used as a backup!

Detect Failure

Introduce a mechanism to detect failure. For example, health checks! Network Load Balancers use active and passive health checks to determine whether a target is available to handle requests. That way you can respond to the event earlier before there is any major impact.

Durable Data Storage

You should always aspire for durable data storage. Amazon S3, for example, is designed to provide 99.999999999% durability of objects over a given year.

99.999999999% (11 nines) durability means that if you store 10,000 objects, on average you may lose one of them every 10 million years or so. Impressive!

Automated Multi Datacenter Resilience

This one is on the list but is a sub-category of the previous one mentioned. S3 achieves its durability by automating backups creation, which is transparent to the user. It’s not the users who made sure to have 3 backups, they just know that he can always (well, almost) reach their data.

S3 is not a DB, it is a simple storage system (kind of like Google Drive). A more specific example of automated multi-data center resilience is Amazon Aurora. This is Amazon’s in-house DB service, and it stores copies of the data in a DB cluster across multiple Availability Zones in a single AWS Region.

Fault Isolation

Fault Isolation means isolating the component, device, or software module causing the error. This topic connects beautifully to the scaling and high availability we talked about prior! More specifically, horizontal scaling — adding more instances.

Sharding

Sharding, also known as horizontal partitioning, is a popular scale-out approach for relational databases. Amazon RDS provides great features to make sharding easy to use in the cloud.

Sharding increases redundancy both because each shard is a replica set, and because even if an entire shard becomes unavailable, the database as a whole still remains partially functional, with part of the schema on different shards.

Optimizing Costs

There is a dedicated blog post for Billing & Pricing in AWS, but here we’ll emphasize the best practices for optimizing costs:

Right-Sizing

Right-sizing is the most effective way to control cloud costs. Examples of size options in AWS services are:

EC2 instance types, various combinations of CPU, memory, and network capacity
S3 storage classes, depending on the frequency of access
RDS instance type, also various combinations of CPU, memory, and network capacity

Elasticity

Elasticity is the ability to fit the resources needed to cope with loads dynamically usually in relation to scale out. It sounds very similar to scalability, right? So I went to check the difference!

Scalability is being capable to expend for a little additional cost, more of a long-term. For example, using a template. That way, the creation of the template itself will require some work, but adding a new instance using that template will be quicker than creating every new instance from scratch.

Elasticity, on the other hand, is supporting short-term changes, like your stomach getting bigger when you eat, and smaller after a few hours. If you insist on a cloud example, it’s like adding another server to receive requests, for the period when people go Christmas shopping.

Elasticity optimizes costs because you can grow or shrink capacity for CPU, memory, and storage resources to adapt to the changing demands, without long-term commitments and contracts!

Purchasing Options

The third way to optimize costs is by taking advantage of the variety of purchasing options in AWS! For example, Amazon EC2 Instances have four purchasing options: On-Demand Instances, Reserved Instances, Spot Instances, and Savings Plans. Each one is best for different scenarios.

Using Cache

Caching helps data to be served faster and with lower latency. There are two types of caching: App caching and edge caching.

App caching: Apps can load faster by keeping temporary files such as thumbnails, scripts, and video snippets on your phone instead of loading them from the web each time. In AWS, Elasticache is a web service that makes it easy to deploy, operate, and scale an in-memory cache in the cloud.
Edge caching: This is the practice of using intermediate storage between data centers and end users accessing the resource. In AWS, Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers content to customers globally with low latency and high transfer speeds.

Security Best Practices

Here we will touch on security in a nutshell, and I recommend reading my AWS security dedicated blog post for more coverage. Let’s go over AWS’s security best practices:

Use AWS features for defense-in-depth: Defense in depth is a strategy that leverages multiple security measures to protect an organization’s assets. The thinking is that if one line of defense is compromised, additional layers exist as a backup to ensure that threats are stopped along the way. AWS provides security features including IAM, firewalls (WAF), port filtering (security groups), and network protection.
Share security responsibility with AWS: Security and Compliance are a shared responsibility between AWS and the customer. this differentiation of responsibility is commonly referred to as Security of the Cloud versus Security in the Cloud. The best practice will be for the customer to do their part.

Reduce privileged access: Users need to get only the access they need. Every user, role, or group shouldn’t have full access to resources and actions, just enough privileges in order to do their job.
Security as code: Codify your security by creating a golden environment, for example, an EC2 launch template with security patches. That way, every new EC2 instance will be created from that template and therefore be more secure than a basic EC2 instance.
Real-time auditing: An audit gives you an opportunity to remove unneeded IAM users, roles, groups, and policies, and to make sure that your users and software have only the permissions that are required. Examples of auditing services in AWS are AWS inspector and CloudTrail.

That’s it! Today we talked about best practices in the cloud, and this was the last blog post in the introduction to AWS series.

Now with this one behind us, all that’s left is to practice and take the AWS cloud practitioner exam. Best of luck to all of us! :D

My Relocation Diaries — The 2-Years Mark!

Cup of Code — Wed, 09 Mar 2022 00:00:00 +0000

When does it stop being “a relocation” and start being “the new home”?

As you can understand from the title — I am now celebrating my 2 year anniversary in Dublin, and soon enough so will Covid-19. The difference between 2020 and 2021 is that this year we got vaccines and looser restrictions every once in a while. Furthermore— and this is a big one — this year I finally went to visit home, and that was a meaningful experience.

Before we dive into my Instagram-beautiful experiences, let’s put things in perspective: After a certain period of time, it stops being an exotic relocation, and it becomes.. a location. Work, relationships, good days, bad days… it’s just life, with a bit of a different spice.

I’m not here to sell how everyone should relocate. It’s not a journey that everyone needs to go through. Some don’t want it, and some don’t have the privilege to do it, and it’s ok! For those of you who are curious about the experience, and for those of you who are curious about my experience — here are my thoughts.

If you read my previous relocation journeys, you know this took a while, but I finally started to feel more comfortable at work, and things eventually clicked. I should’ve seen it coming because I felt the exact same way after the exact same time elapsed in my previous job as well.

For those of you who don’t know, I am a software developer who works at Amazon’s Payments Security department. This huge department’s job is to make sure your details are stored correctly and securely. To be honest, it is pretty amazing how those multiple teams work together… Anyway, back on track:

My team’s job is networking. By networking, I mean the foundation of the various components of this big complicated system. This is a crucial part of security, and something I didn’t have any experience in.

Starting to work in networking is kind of like being an actor in Grey’s anatomy: You are not a doctor, never intended to be, but your job is to memorize all those complicated lines and look like you know what is Hemopneumothorax or Pancreaticoduodenectomy (source). So, it’s very interesting, but also requires a lot of learning.

Dublin is full of ex-pats, which means you are surrounded by people who are different from you. There is so much to learn about divergent lives and cultures! I can’t write them all down, but here is a cool example: For those of you who don’t know, there is such thing called “Irish goodbye”, which is when a person is ducking out of a party or social gathering without bidding farewell. I’ve seen it happen several times, and it looks like someone is just going to the restroom, but then never comes back. This is very different from the Israeli goodbye, which is not an official term but it means giving a warm hug to each and every person at the party before you go.

Speaking of people, I’ve discovered that building a social life, especially outside of the workplace — requires effort. You need to actively seek out friends. Sure, this is a struggle for every person in their late 20s, when there is no social structure with crazy hours like school and military service. When you’re in your home country and want company, even if you just moved to a new city — you can always go visit your friends and family, even if it’s a bit further now. It does not work like that here.

When you’re in a new country, and you don’t want to be alone on a Friday night, you need to find friends, which means you need to go on dates, and let me tell you — first dates are awkward, even when they are not romantic. When you find someone you like, it will take several embarrassing engagements in order to get to a comfortable level. To be honest, not every week do I have the tolerance for that. When having a night off, sometimes it’s just easier to go meet this one person you already feel comfortable with, rather than go sweat on leveling up a potential friendship.

This year I moved out of a shared apartment and got my own place for the first time. I do recommend sharing a flat when you just arrive in a new country — it makes the landing softer, you’ll definitely be less lonely, and make fewer mistakes. Some people love being surrounded by friends all day, but I appreciate my me-time — so it was a natural move.

A rent lease is one year long, so it’s not a big commitment. This change didn’t shake my core. Deciding to study for a driving license did — way more than you’d assume! This process costs money and takes months, and you don’t do it when you have a visible expiration date for your stay. Do you know what that means?

This is when I realized what a relocation really is. You think it will be like going on a trip and making a detour to grab a cup of coffee. It will take a bit of extra time but is totally worth it — because of the adventure! In reality, you don’t go back to the original route you planned for your trip. This detour propels in a different direction. I’ve seen it relevant even when one decides to relocate back home. This experience changes your life, and you can never know in advance what will be coming back with you.

Do you know what else suddenly happened this year? I am not the newbie anymore! There are people who seek me out and my advice! The other day, I was on the phone with someone who got an offer to relocate to Dublin. He had so many questions, most of which I couldn’t answer because our situations are different: Martial status, job description, Visa requirements… I told him: “Why not get answers only for the crucial things you need now, for the start, and then, when you arrive and meet your colleagues — ask them? I’m sure they’ll be very helpful”, and he answered: ”Because I trust only Israelis”. Obviously, there is no reason to be skeptical, but I get why he thinks that. I used to think that as well.

Before I moved here, when it was just me with my two big suitcases, I was absolutely sure that getting on this plane means I wouldn’t date for 2 whole years. This made sense to me because I trust only Israelis. Then you arrive and quickly realize people are people, and that fear is just false.

Lastly, I want to talk about Israel. The Israeli-Palestinian conflict is an unpleasant topic to discuss, both here online and outside in the big world. I have had several attempts in writing here, that eventually did not make it to the final draft. This is what I did choose to put: Relocation to another country means you are going to a more heterogenic environment than you had back home. You are most likely to encounter people with different opinions than yours. Remember that you represent your origin country when you are abroad, wherever you are from, especially when there are not many others next to you.

Speaking of home, I finally went to Israel for the first time after a year and a half! Apparently, there are two parallel timelines, and coming back feels like I’ve been away only for one month. The problem is, that everyone else did spend their months in this timeline, and I need to catch up. I wasn’t here when it was a slow gradual change. Friends are living in different locations and have different priorities. And by the way, good luck trying to convince ANYONE in Tel Aviv to come to meet you anywhere else.

On one hand — your friendships from your new home don’t have the comfort and ease of the years-long friendships you have left behind. On the other hand — you weren’t there when those friendships shifted and tightened. Eventually, this is a bit of a lose-lose situation, a sacrifice one makes when deciding to move away.

With that said— I am really enjoying the friendships I was able to maintain. There is something about this deadline, “I’m only here for a month”, that motivates people to make the effort. Sometimes, when your parents are 2 hours drive, you’ll see them less often than when they are on another continent.

90% of the people who hear I moved away, say “I will never be able to leave my family”, which I understand and respect. But after 2 visits of month-long each, I can assure you this method has its benefits as well. Some of my friends, who are in the same country, haven’t met each other as often as they met me!

When coming home, it is inevitable to think about returning to your country: Would I want to work here? Would I want to live here? What will my evenings look like? What will my weekends look like?

The only thing that is irreplaceable is the holidays. All I have is to hope the pandemic will end soon, and I’ll have the luxury to plan vacations in advance. Another problem is that after 2 years in Dublin, every arrival at the airport means you will be missing someone for a while.

Before we go, I want to dedicate a section to your questions:

How do you find a job abroad? By looking for a job abroad. Every job description has a location. You just need to be open to the idea, and work on your English ;)

What is the partner doing? I did not arrive with a partner, but I did notice there is always one side who looked for a job abroad and one side who came along. Be sure that being the partner is harder than being the initiator of the move, so they deserve extra-kindness! Some of them keep their previous job remotely (which can be very isolating), and some of them find a job as well. The lucky ones find a job in their field.

What are the logistics? How do you actually move? The logistics can be a headache. It really depends on your situation and on your destination.
By situation, I mean family, visa, what you left behind, etc.
By destination, I mean things are very different in each country.
If Dublin is your target, I’ll be happy to help!

Is it hard? Is it lonely? It can be sometimes. It’s a fresh start. With that said, you have colleagues and communities and Facebook groups full of people who also want to make friends :)

Did you manage to make meaningful relationships? Yes, I have. I was surprised to discover how two different people from two completely different backgrounds can have such a strong connection and laugh at the same jokes.

A question I frequently get asked when coming home is “when will you come back for good?”. Now, that I passed my 2 years mark, I’ve noticed I’m referring to Dublin as going back home as well, and the only answer I can give is “not now”.

Introduction to AWS — Billing & Pricing

Cup of Code — Fri, 31 Dec 2021 00:00:00 +0000

All you need to know about pricing for AWS services and AWS services for billing

Welcome to the 6th blog post in the AWS series! This time we will talk about money, and it’s an important topic in the Cloud Practitioner exam! So what will we talk about today? Here is a table of content for easy access:

AWS Pricing Introduction
4 Key Principles
The AWS Free Tier
The Different Support Levels
Pricing for individual services
AWS services that are related to billing
Resource Groups & Tagging

Shall we begin?

AWS Pricing Introduction

Every AWS topic has a whitepaper, and the link to the pricing whitepaper can be found here.

Before we dive into the pricings in AWS, let’s recap two fundamental terms we got introduced to in the first blog post:

Different Pricing Models: CAPEX vs OPEX

CAPEX stands for Capital Expenditure, which is where you pay upfront. It’s a fixed, sunk cost because you need to accurately predict how much resources you are going to need, and it doesn’t give you flexibility for peaks and lows.

OPEX stands for Operational Expenditure which is where you pay for what you use, like electricity bill. Some of you would think it’s also like a water bill but here in Ireland, there is no such thing ;)

For example — Do you need 100 servers for daily traffic and another 50 for the holiday season? With CAPEX, your only way to support that need is to rent 150 servers for the whole year. OPEX, on the other hand, gives you flexibility.

Now that we established that Operational Expenditure is way better for us, let’s dive into what AWS has to offer.

The Pricing Philosophy of AWS

Here is a quote from the whitepaper:

“While the number and types of services offered by AWS have increased dramatically, our philosophy on pricing has not changed. You pay as you go, pay for what you use, pay less as you use more, and pay even less when you reserve capacity.”

Let’s talk about them a bit:

Pay as you go —in contradiction to paying in advance.
Pay for what you use — and not a pre-defined fixed price.
Pay less as you use more — (per unit), like when a pack of 6 bottles of 1.5L coke is cheaper per bottle than purchasing a single 1.5L
Pay less when you reserve capacity — Like with EC2 instances, for example.

Speaking of EC2 instances, here is a quick recap of the EC2 Pricing Models:

On-demand: This allows you to pay a fixed rate by the hour (or by the second) with no commitment.
Reserved: Provides you with a capacity reservation, and offers a significant discount on the hourly charge for an instance (like booking a hotel room a year in advance). Contract terms are 1 year or 3 years terms.
Spot: Enables you to bid whatever price you want for instance capacity, providing for even greater savings if your applications have flexible start and end times.
Dedicated hosts: A physical EC2 server dedicated for your use. Dedicated hosts can help you reduce costs by allowing you to use your existing server-bound software licenses.

This will be useful when you’ll get asked “Which of the following is an EC2 pricing option?”

Key Principles

While pricing models vary across services, it’s worthwhile to review the 4 key principles and best practices that are broadly applicable.

1. Understand the fundamentals of pricing: There are three fundamental drivers of cost with AWS: Compute, Storage, and Data outbound (data going out). Remember that when you get asked “Which of the following is not a fundamental AWS charge?” *— The answer is *Data Inbound**.

2. Start early with cost optimization: Whether you started in the cloud, or you are just starting your migration journey to the cloud, AWS has a set of solutions to help you manage and optimize your spending.

3. Maximize the power of flexibility: One of the key advantages of cloud-based resources is that you don’t pay for them when they’re not running. By turning off instances you don’t use. you can reduce costs by 70% or more compared to using them 24/7. You can choose and pay for exactly what you need — no minimum commitments or long-term contracts are required unless you choose to save money through a reservation model.

4. Use the right pricing model for the job: AWS offers several pricing models depending on the product. These include:

On-demand instances: you pay for compute or database capacity by the hour or second
Saving Plans: a flexible pricing model that offers low prices in exchange for a commitment to a consistent amount of usage
Spot Instances: in EC2, which we learned about in a previous blog post.
Reservations: which grants you up to 75% discount.

The AWS Free Tier

The AWS Free Tier enables you to gain free, hands-on experience with more than 60 products on the AWS platform. Why give services free? To attract new customers, of course! And more officially: To help new AWS customers get started in the cloud.

AWS Free Tier includes the following free offer types:

12 Months Free —12 months following your initial sign-up date to AWS. When your 12-month free usage term expires, or if your application use exceeds the tier, you simply pay standard, pay-as-you-go service rates.
→ Under this category, you can find (to some extend*) EC2, S3, RDS, and CloudFront.
Always Free — These free tier offers do not expire and are available to all AWS customers.
→ Under this category, you can find (to some extend*) DynamoDB, S3 Glacier, and Lambda.
Trials — This tier’s offers are short-term free trials starting from the date you activate a particular service. Once the trial period expires, you simply pay standard, pay-as-you-go service rates.
→ Under this category, you can find (to some extend*) Redshift and GuardDuty.

This section lists some of the most commonly used AWS Free Tier services. The full list of AWS Free Tier services can be found here.

When I write *(to some extend)*** it means for the first X hours / Y GB or Z requests.

Remember when we talked about account types in the first blog post, under Available Regions? So, the AWS Free Tier is not available in the AWS GovCloud (US) Regions or the China (Beijing) Region at this time. The Lambda Free Tier is available in the AWS GovCloud (US) Region.

Speaking of free, remember those for when you’ll get asked “Which of the following AWS services are free? Choose 3”

Please keep in mind that with VPC, Elastic Beanstalk, CloudFormation, and Auto-Scaling, the underlying provisioned resources will incur charges.

The Different Support Levels

How many support levels are there? Depends on who you ask.

In the course I took, they mention four support levels: Basic, Developer, Business, and Enterprise.
According to the pricing whitepaper, there are 3 support levels: Developer, Business, and Enterprise.
In that whitepaper, they send you to a page called AWS Support Plan Pricing, where they introduce four support levels: Developer, Business, Enterprise On-Ramp, and Enterprise.

I understand why “Basic” isn’t technically a support plan (and not counting it fits well with my Pokèmon reference), and Also — there is a good chance Enterprise On-Ramp is too recent to be asked about in the exam — so I’m not going to talk about this one, but you should remember those names, because you can be asked “Which of the following is an AWS plan?” *and they give made-up options like *“Individual”.

What Is The Difference?

You will soon see a beautiful table with the contrast between the available support levels, but to summarize:

Basic — Free, but that’s the only good thing about it.
Developer — Start with $29, and has support that an individual developer needs: Case opening with support, support only on business hours — nothing urgent.
Business — Start with $100, and they need more availability — tech support 24x7 through chat and phone as well as email, and the ability for anyone in the account to open cases.
Enterprise — Start with $15,000, very expensive, compared to the Business level, which looks similar but has 2 main differences: 15 minutes response (that’s fast, you gotta admit) for business-critical systems down, and a 1:1 with a designated Technical Account Manager (TAM) to proactively monitor your environment and assist with optimization and coordinate access to programs and AWS experts. Fancy.

Important Notes:

All the costs are monthly and start at the price mentioned, but scale based on the usage! Also, not mentioned in the course but is mentioned in the AWS Support Plan Pricing —* “Greater of $5,500.00 - or - 10% of monthly AWS charges”.* Could be too new to be asked about — but worth mentioning.
The response times under the Developer level (24b, 12b) are in business hours — so 24 hours is actually 3 days, not 1.

Questions on this topic are tricky: “Which of the following AWS Support Plans provide Enhanced Technical Support ONLY during business hours via email?”. The answer is — Developer.

Pricing Details for Individual Services

This section is tricky as well! It requires memorizing (and common sense) because questions in this section go something like this: *“Which of the following components are billed for Amazon RDS instances?”. *Don’t worry, When mentioning a service, I’ll give a reminder of what it is!

Let’s start with EC2: Elastic Compute Cloud (EC2) is a virtual server in the cloud.

*EIP = elastic IP.

But hey, who says you even need a server? There are also serverless services in AWS, like lambda and S3. Let’s look at how they are priced:

And also:

Don’t remember what is EBS? S3? Here is a quick reminder:

EBS:

EBS is a virtual disk in the cloud. Inside the EC2 service, you will see a tab saying Elastic Block Store. EBS allows you to create storage volumes and attach them to EC2 instances. Once attached, you can create a file system on top of these volumes, run a DB, or use them in any other way you would use a block service. EBS volumes are placed in a specific AZ, where they are automatically replicated to protect you from the failure of a single component.

Introduction to AWS: EC2, related services, and AWS through the CLI

CloudFront:

Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers content to customers globally with low latency, high transfer speeds, all within a developer-friendly environment.

Introduction to AWS: IAM, CloudFront, and S3

S3, S3 Glacier:

Amazon S3 is a Simple Storage Service that stores and retrieves any amount of data from anywhere on the web. S3 Glacier is used for archiving solutions. Here are some more details about its pricing:

Snowball pricing:

AWS Snowball is A PB-scale data transport solution. PB stands for petabyte, which is equal to 2 to the 50th power of bytes! So, Snowball is like a giant USB key.

Here you can see what we talked about in the key principles — Data inbound is free, data outbound is not.

Now we are reaching the Databases section. RDS is the Relational Database Service, and the non-relational DB is called DynamoDB.

RDS pricing:

Dynamo DB pricing:

So, before you go to the next section — pop quiz!
Which of the following components are billed for Amazon RDS instances?
Choose 3:

Storage
DB instance hours
I/O requests for Amazon RDS magnetic storage
Data transfer incurred in replicating data between your primary and standby, in a Multi-AZ DB instance deployment
Standby time

Ok, I think we earned our right for a meme break!

Now, we’ll go over the AWS services related to billing (there are 9 of those!) and then Tagging and Resource Grouping, and that’s it!

AWS Services Related to Billing

In this section we’ll talk about:

The difference between AWS Budgets and AWS Cost Explorer
What is AWS organizations and how it integrates with CloudTrail
The difference between AWS QuickStart and AWS LandingZone
What is the AWS Partner Program and how much does it cost?
The two AWS cost calculators and the difference between them

AWS Budgets vs AWS Cost Explorer

AWS Budgets gives you the ability to set custom budgets that alert you when your costs or usage exceed (or are forecasted to exceed) your budgeted amount.

AWS Cost Explorer has an easy-to-use interface that lets you visualize, understand, and manage your AWS costs and usage over time.

So, AWS Budgets is used to budget *costs **before* they have been incurred, and AWS Cost Explorer is used to explore *costs **after* they have been incurred.

AWS Organizations

AWS Organizations is an account management service that enables you to consolidate multiple AWS accounts into an organization that you create and centrally manage.

To clarify: you create an organization and add multiple AWS accounts into it. This is a global service, and you reach it by clicking on your username->my organizations. This service is available in two feature sets: Consolidated billing and All features.

Consolidated Billing

Consolidated billing has the following benefits:

One bill — You get one bill for multiple accounts.
Easy tracking — You can track the charges across multiple accounts and download the combined cost and usage data.
Volume pricing discount — You can combine the usage across all accounts in the organization to share the volume pricing discounts, Reserved Instance discounts, and Savings Plans. This can result in a lower charge for your project, department, or company than with individual standalone accounts. Unused reserved EC2 instances for EC2 are applied across the group.
No extra fee — Free!

All Features

All features give you full access:

When you create an organization, enabling all features is the default. With all features enabled, you can use the advanced account management features available in AWS Organizations such as integration with supported AWS services and organization management policies.

Policies in AWS Organizations enable you to apply additional types of management to the AWS accounts in your organization.

When you create organizations and then put AWS accounts behind those org units, you can apply policy either to the ou or the AWS accounts

AWS Organizations Best Practices

Enable multi-factor authentication on the root account.
Use a strong and complex password on the root account.
Paying account should be used for billing purposes only. Do not deploy resources into the paying account.
Linked accounts: 20 linked accounts by default, there is an ability to add more.
Billing Alerts:
When monitoring is enabled on the paying account, the billing data for all linked accounts is included.
You can still create billing alerts per individual account.

Pop quiz!

True/False: With Consolidated Billing, the Paying Account can make changes, like altering resource access, to any of the resources owned by a Linked Account.
Choose the features of Consolidated Billing (choose 3):
Charging is based per VPC
A single bill is issued containing the charges for all AWS Accounts
Multiple standalone accounts are combined and may reduce your overall bill
Account charges can be tracked individually.

CloudTrail

This one can be easily confused with CloudWatch, so let’s talk about the differences:

CloudTrail vs CloudWatch

CloudWatch monitors performance.
CloudTrail monitors API calls in the AWS platform: Changes to AWS env, creation of resources, etc.

How to use CloudTrail with AWS Organizations

CloudTrail is on a per account per region basis but can be aggregated into a single bucket belonging to the paying account.
CloudTrail can consolidate logs using an S3 bucket:
Turn on CloudTrail in paying account.
Create a bucket policy that allows cross-account access.
Turn on CloudTrail in the other accounts and use the bucket in the paying account.
A best practice is to use a separate account for logging.

AWS Quick Start vs AWS Landing Zone

AWS Quick Start is a way of deploying environments quickly, using CloudFormation templates built by AWS Solutions Architects and AWS Partners. The resources cost money, of course — but this service is free!
https://aws-quickstart.github.io/

AWS Landing Zone —This allows you to set up a secure, multi-account AWS environment (as opposed to using *AWS Quick Start *which is just for individual accounts). It is based on AWS best practices and starts with four AWS Accounts: AWS Organisations account, Shared Services account, Log Archive account, and Security account.

Landing Zone is a solution that helps customers more quickly set up a secure, multi-account AWS environment based on AWS best practices.

AWS Landing Zone is currently in Long-term Support and will not receive any additional features. Customers interested in setting up a new landing zone should check out AWS Control Tower (but for our purposes, there is no need to dive into what AWS Control Tower is).

AWS Partner Program

The AWS Partner Network (APN) is a global community of partners that help you build, market, and sell your offerings.

As you validate your offerings with AWS and pay the APN annual fee of $2500, you can unlock access to differentiation programs, go-to-market resources, funding benefits, and more to gain recognition with customers and grow your business.

Together, partners and AWS can provide innovative solutions, solve technical challenges, win deals, and deliver value to our mutual customers.

Types of partners:

Consulting: These partners design, architect, build, migrate and manage customer workloads and applications on AWS.
Technology: These partners provide hardware, connectivity services, or software solutions that are either hosted on or integrated with, the AWS Cloud.

Consulting Partners:

Partner  |Practitioner Certs|Associate Certs|Prof/Specialty Certs
------------------------------------------------------------------
Select   |        2         |        2      |          2
Advanced |        4         |        4      |          4
Premier  |       10         |       10      |         10

More information can be found at https://aws.amazon.com/partners/programs/

Different AWS Cost Calculators

AWS helps you to calculate your costs using a couple of different calculators. You won’t find them in the options of the services in the AWS console, but these come up a lot in the exam.

There are two AWS pricing tools, both have an old name and a new name (just to be sure — remember both names!):

AWS Simple Monthly calculator (aka AWS Pricing Calculator)
AWS TCO (Total Cost of Ownership) calculator (aka Migration Evaluator)

AWS Simple Monthly Calculator is used to calculate your running costs on AWS on a per month basis. It is not a comparison tool. https://docs.aws.amazon.com/pricing-calculator/latest/userguide/what-is-pricing-calculator.html pricing calculator https://calculator.aws/#/

AWS TCO Calculator is used to compare the costs of running your infrastructure on-premise vs in the AWS cloud. It will generate reports that you can give to your C-level execs to make a business case to move to the cloud.

More information can be found here.

Resource Groups & Tagging

AWS Tagging: A tag is a label that you or AWS assigns to an AWS resource. Each tag consists of a key and a value. Tags are metadata (= data about data) and you can use them to organize your resources, and cost allocation tags to track your AWS costs on a detailed level.

For each resource, each tag key must be unique, and each tag key can have only one value.
Tags can sometimes be inherited. Tags from a CloudFormation stack are inherited to all the resources.

For example: {env: prod}. So, If our CloudFormation stack has this tag — all the resources that are created in that stack will have this tag as well. Also, You can’t have two tags with the key env.

Resource groups make it easy to group your resources using the tags that are assigned to them. You can group resources that share one or more tags. Resource groups contain information such as Region, Name, EmployeeID, Department.

Tags can contain specific information:

For EC2 — Public & Private IP Addresses
For ELB — Port Configurations
For RDS — Database, Engine, etc

Resource groups can be found in the AWS console header, next to Services.

Using Resource Groups, you can apply automation to resources tagged with specific tags. For example, stop all EC2 instances in the Ireland region. Resource Groups in combination with ***AWS Systems Manager*** allow you to control and execute automation against entire fleets of EC2 instances, all at the push of a button.

Tag Editor is a global service that allows us to discover resources and add additional tags to them as well (newer regions may take some time to be compatible with the tag editor). To clarify, the tag editor is good for finding all your tags.

Pop Quiz!
Which two of the following options best describe a Resource Group? Choose 2.

A resource group is a collection of resources of the same type (EC2, S3, etc.) that share one or more tags or portions of tags.
A resource group is a collection of resources that share one or more tags.
A resource group is a collection of resources of the same type (EC2, S3, etc.) that are deployed in the same Availability Zone.
A resource group is a collection of resources that are deployed in the same AWS Region, and that match the criteria specified in the group’s query.

That’s it!

We’ve learned a lot today! And this time, there is a lot to memorize as well!

So, a quick recap on what we talked about today, and then you can test yourself!

We started with AWS Pricing Introduction, the 4 Key Principles and the AWS Free Tier. After that, we got introduced to the Different Support Levels and the differences between them. The main and juicy part of the blog post was the pricing for individual services and the AWS services that are related to billing. Lastly, we got introduced to Resource Groups & Tagging.