Forem: Rijul Rajesh

Understanding Transformers Part 12: Building the Decoder Layers

Rijul Rajesh — Thu, 23 Apr 2026 19:23:30 +0000

In the previous article, we just began with the concept of decoders in a transformer.

Now we will start adding the positional encoding.

Adding Positional Encoding in the Decoder

Now, for the decoder, let’s add positional encoding.

Just like before, we use the same sine and cosine curves to get positional values based on the embedding positions.

These are the same curves that were used earlier when encoding the input.

Applying Positional Values

Since the <EOS> token is in the first position and has two embedding values, we take the corresponding positional values from the curves.

For the first embedding, the value is 0
For the second embedding, the value is 1

Now, we add these positional values to the embedding:

As a result, we get 2.70 and -0.34, which represent the <EOS> token after adding positional encoding.

Adding Self-Attention

Next, we add the self-attention layer so the decoder can keep track of relationships between output words.

The self-attention values for the <EOS> token are -2.8 and -2.3.

Note that the weights used in the decoder’s self-attention (for queries, keys, and values) are different from those used in the encoder.

Adding Residual Connections

Now, we add residual connections, just like we did in the encoder.

What’s Next?

So far, we have seen how self-attention helps the transformer understand relationships within the output sentence.

However, for tasks like translation, the model also needs to understand relationships between the input sentence and the output sentence.

We will explore this in the next article.

Looking for an easier way to install tools, libraries, or entire repositories?
Try Installerpedia: a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 11: How Decoding Begins

Rijul Rajesh — Wed, 22 Apr 2026 19:31:56 +0000

In the previous article we wrapped up the encoder part, In this article, we will start building the second part of the transformer: the decoder.

Just like the encoder, the decoder also begins with word embeddings.

However, this time the embeddings are created for the output vocabulary, which consists of Spanish words such as:

ir
vamos
y
<EOS> token

Starting the Decoding Process

To begin decoding, we use the <EOS> token as the input.

This is a common way to initialize the decoding process for an encoded sentence.

In some cases, people use a <SOS> (Start of Sentence) token instead.

Creating the Initial Input

We represent the <EOS> token as a vector by assigning:

1 to <EOS>
0 to all other words in the vocabulary

From this we can see that 2.70 and -1.34 are the numbers that represent the value for the EOS token.

Now that we have the initial input for the decoder, the next step is to add positional encoding.

We will explore this in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 10: Final Step in Encoding

Rijul Rajesh — Tue, 21 Apr 2026 19:36:28 +0000

In the previous article, we explored the use of self-attention layers, now we will dive into the final step of encoding and start moving into decoders

As the final step, we take the positional encoded values and add them to the self-attention values.

These connections are called residual connections. They make it easier to train complex neural networks by allowing the self-attention layer to focus on learning relationships between words, without needing to preserve the original word embedding and positional information.

At this point, we have everything needed to encode the input for this simple transformer.

These four components work together to convert words into meaningful numerical representations:

Word embedding
Positional encoding
Self-attention
Residual connections

Now that we have encoded the English input phrase “Let’s go”, the next step is to decode it into Spanish.

To do this, we need to build a decoder, which we will explore in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 9: Stacking Self-Attention Layers

Rijul Rajesh — Fri, 17 Apr 2026 20:50:25 +0000

In the previous article, we explored how the weights are shared in self-attention.

Now we will see why we have these self-attention values instead of the initial positional encoding values.

Using Self-Attention Values

We now use the self-attention values instead of the original positional encoded values.

This is because the self-attention values for each word include information from all the other words in the sentence. This helps give each word context.

It also helps establish how each word in the input is related to the others.

If we think of this unit, along with its weights for calculating queries, keys, and values, as a self-attention cell, then we can extend this idea further.

To correctly capture relationships in more complex sentences and paragraphs, we can stack multiple self-attention cells, each with its own set of weights. These layers are applied to the position-encoded values of each word, allowing the model to learn different types of relationships.

Going back to our example, there is one more step required to fully encode the input. We will explore that in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 8: Shared Weights in Self-Attention

Rijul Rajesh — Thu, 16 Apr 2026 21:08:46 +0000

In the previous article, we started calculating the self-attention values.

Let’s now calculate the self-attention values for the word “go”.

We do not need to recalculate the keys and values.

Instead, we only need to create the query that represents the word “go”, and then perform the same calculations as before.

After completing the calculations, we get the self-attention values for “go” as:

2.5 and -2.1

Key Observations About Self-Attention

The weights used to calculate queries are the same for both “Let’s” and “go”.
- This means that regardless of the number of words, we use one shared set of weights.
- Similarly, the same sets of weights are reused to calculate keys and values for every input word.
- No matter how many words are given as input, the transformer reuses the same weights for queries, keys, and values.
We do not need to compute queries, keys, and values sequentially.
- All of them can be computed at the same time.
- This allows transformers to take advantage of parallel computation, making them very efficient.

We will continue building our transformer step by step in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 7: From Similarity Scores to Self-Attention

Rijul Rajesh — Wed, 15 Apr 2026 02:24:37 +0000

In the previous article, we calculated the similarities between Queries and Keys.

We can use the output of the softmax function to determine how much each input word should contribute when encoding the word “Let’s”.

Interpreting the Weights

In this case, “Let’s” is much more similar to itself than to “go”.

So after applying softmax:

“Let’s” gets a weight close to 1 (100%)
“go” gets a weight close to 0 (0%)

This means:

“Let’s” contributes almost entirely to its own encoding
“go” contributes very little

Creating Value Representations

To apply these weights, we create another set of values for each word.

First, we create two values to represent “Let’s”

Then, we scale these values by 1 (since its weight is 100%)
Next, we create two values to represent “go”

These values are scaled by 0 (since its weight is 0%)

Combining the Values

Finally, we add the scaled values together:

The result is a new set of values that represent the word “Let’s”, now enriched by its relationship with all input words.

These final values are called the self-attention values for “Let’s”.

They combine information from all words in the sentence, weighted by how relevant each word is to “Let’s”.

We can now repeat the same process for the word “go”, which we will explore in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 6: Calculating Similarity Between Queries and Keys

Rijul Rajesh — Mon, 13 Apr 2026 20:02:45 +0000

In the previous article, we explored the concepts of Queries and Keys. Now we will see how to calculate the similiarities

Calculating Similarity Using Dot Product

One way to calculate the similarity between a query and the keys is by using the dot product.

Query vs Key for “Let’s”

Let’s first compute the dot product between the query and key for the word “Let’s”.

We multiply each pair of values and then add the results. This gives us a similarity score of 11.7.

Query for “Let’s” vs Key for “go”

Now, let’s compute the dot product between the query for “Let’s” and the key for “go”.

This gives us a similarity score of -2.6.

Understanding the Result

The similarity score for “Let’s” with itself (11.7) is much higher than its similarity with “go” (-2.6).

This tells us that “Let’s” is much more similar to itself than it is to “go”.

As a result, when encoding the word “Let’s”, it should be influenced more by itself and less by “go”.

What’s Next?

To turn these similarity scores into meaningful weights, we pass them through a softmax function.

We will explore how softmax works in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 5: Queries, Keys, and Similarity

Rijul Rajesh — Sat, 11 Apr 2026 19:26:53 +0000

In the previous article, we explored the self-attention concept for transformers, in this article we will go deeper into how the comparisons are performed.

Building Query and Key Values

Let’s go back to our example.

We have already added positional encoding to the words “Let’s” and “go”.

Creating Query Values

The first step is to multiply the position-encoded values for the word “Let’s” by a set of weights.

Next, we repeat the same process using a different set of weights, which gives us another value (for example, 3.7).

We do this twice because we started with two position-encoded values representing the word “Let’s”.

These resulting values together represent “Let’s” in a new form.

In transformer terminology, these are called query values.

Creating Key Values

Now, we use these query values to measure similarity with other words, such as “go”.

To do this, we first create a new set of values for each word, similar to how we created the query values.

We generate two values for “Let’s”

And two values for “go”

These new values are called key values.

What’s Next?

We will use these key values along with the query values to calculate how similar “Let’s” is to “go”.

We will explore how this similarity is calculated in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 4: Introduction to Self-Attention

Rijul Rajesh — Thu, 09 Apr 2026 18:56:06 +0000

In the previous article, we learned how word embeddings and positional encoding are combined to represent both meaning and position.

Now let’s go back to our example where we translate the English sentence “Let’s go”, and add positional values to the word embeddings.

Now, let’s get the positional encoding for both words.

Understanding Relationships Between Words

Now let’s explore how a transformer keeps track of relationships between words.

Consider the sentence:

“The pizza came out of the oven and it tasted good.”

The word “it” could refer to pizza, or it could potentially refer to oven.

It is important that the transformer correctly associates “it” with “pizza”.

Self-Attention

Transformers use a mechanism called self-attention to handle this.

Self-attention helps the model determine how each word relates to every other word in the sentence, including itself.

Once these relationships are calculated, they are used to determine how each word is represented.

For example, if “it” is more strongly associated with “pizza”, then the similarity score for pizza will have a larger impact on how “it” is encoded by the transformer.

We have now covered the basic idea of self-attention. We will explore it in more detail in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 3: How Transformers Combine Meaning and Position

Rijul Rajesh — Wed, 08 Apr 2026 21:18:57 +0000

In the previous article, we learned how positional encoding is generated using sine and cosine waves. Now we will apply those values to each word in the sentence.

Applying Positional Encoding to All Words

To get the positional values for the second word, we take the y-axis values from each curve at the x-axis position corresponding to the second word.

To get the positional values for the third word, we follow the same process.

Positional Values for Each Word

By doing this for every word, we get a set of positional values for each one:

Each word now has its own unique sequence of positional values.

Combining Embeddings with Positional Encoding

The next step is to add these positional values to the word embeddings.

After this addition, each word embedding now contains both:

semantic meaning (from embeddings)
positional information (from positional encoding)

So for the sentence:

"Jack eats burger"

we now have embeddings that also capture word order.

What Happens When We Change Word Order?

Let us reverse the sentence:

"Burger eats Jack"

The embeddings for the first and third words get swapped.
However, the positional values for positions 1, 2, and 3 remain the same.

When we add the positional values to the embeddings again:

The final vectors for the first and third words become different from before.

This is how positional encoding helps transformers understand word order.

Even if the same words are used, changing their positions results in different final representations.

We will explore further in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 2: Positional Encoding with Sine and Cosine

Rijul Rajesh — Mon, 06 Apr 2026 20:51:19 +0000

In the previous article, we converted words into embeddings. Now let’s see how transformers add position to those numbers.

The numbers that represent word order in a transformer come from a sequence of sine and cosine waves.

Each curve is responsible for generating position values for a specific dimension of the word embedding.

Understanding the Idea

Think of each embedding dimension as getting its value from a different wave.

For example:

The green curve provides the positional values for the first embedding dimension of every word.

For the first word in the sentence, which lies at the far left of the graph (position 0 on the x-axis):

The value taken from the green curve is 0 (the y-axis value at that position).

The orange curve provides the positional values for the second embedding dimension.

At the same position (first word):

The value from the orange curve is 1.

The blue curve provides the positional values for the third embedding dimension.

For the first word:

The value is 0.

The red curve provides the positional values for the fourth embedding dimension.

For the first word:

The value is 1.

Final Positional Encoding for the First Word

By combining the values from all four curves, we get the positional encoding vector for the first word:

We will apply the same process to the remaining words in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 1: How Transformers Understand Word Order

Rijul Rajesh — Sun, 05 Apr 2026 18:42:00 +0000

In this article, we will explore transformers.

We will work on the same problem as before: translating a simple English sentence into Spanish using a transformer-based neural network.

Since a transformer is a type of neural network, and neural networks operate on numerical data, the first step is to convert words into numbers. Neural networks cannot directly process text, so we need a way to represent words in a numerical form.

There are several ways to convert words into numbers, but the most commonly used method in modern neural networks is word embedding. Word embeddings allow us to represent each word as a vector of numbers, capturing meaning and relationships between words.

Before going deeper into the transformer architecture, let us first understand positional encoding. This is a technique used by transformers to keep track of the order of words in a sentence.

Unlike traditional models, transformers do not process words sequentially. Because of this, they need an additional way to understand word order, and positional encoding solves this problem.

Let us take a simple example sentence:

"Jack eats burger"

The first step is to convert each word in this sentence into its corresponding word embedding.

For simplicity, we will represent each word using a vector of 4 numerical values.

In real-world applications, embeddings are much larger, often containing hundreds or even thousands of values per word, which helps the model capture more detailed meaning.

We will continue building on this example in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here