Forem: Christos Alexiou

Using secrets stored in AWS Secrets Manager as environment variables for ECS container definitions. With Terraform.

Christos Alexiou — Sat, 10 Dec 2022 14:25:15 +0000

Hello readers. I hope this piece of writing finds you healthy, sitting in a comfortable sofa or desk chair, and enjoying the wintry weather of your country.

Today, we are going to discuss environment variables in task definitions.

The scenario

Given a managed cluster in AWS ECS, that was created using AWS Terraform provider, you are challenged to securely configure environment variables in the task definitions of each one of your cluster tasks, which you are also managing with Terraform - because Terraform all the things.

The manifesto

There are countless ways of handling application secrets today, probably as many as the people who write software. I am not trying to preach on whether this is the right way to do it, or if it's the most secure or comfortable. But, I know for a fact that many people will end up in situations where this piece of writing will become a useful guide.

The implementation

Directory structure

I was either lucky or farsighted when I was deciding on the structure of my Terraform code. This wonderful occasion allowed me to have the resources for the tasks, the services, and all of the paraphernalia in the same module letting me make direct references between them without worrying about inheritance or inclusion.

For this article, it is enough for the reader to be aware of the following directory structure:

├── services.tf
├── task_env_vars.tf
└── tasks. tf
├── definitions
│   └── template.json

Secrets in AWS Secrets Manager

AWS Secrets Manager helps you manage, retrieve, and rotate database credentials, API keys, and other secrets throughout their lifecycles. They encrypt the keys using the AWS Key Management Service. I suppose it is secure enough. If you are working for VISA or MasterCard and you think that the level of security they provide is not enough, you probably ought not to read this piece of writing.

I am not managing the secrets that are stored in AWS Secrets Manager with Terraform. I chose to manage those secrets using the AWS Console interface, and this is the path I've taken, although not ideal.

If the reader wants to go the route of managing the secrets with Terraform, the required documentation is here and it's extremely straightforward.

For us, plebes, I am going to continue writing with the standpoint that the secrets are created through the interface, as I did, for instance, for the secret values that I want to be used as environment variables in a service that I wrote that's responsible for mailing.

Creating a secret

When prompted by the wizard of the AWS Secrets Manager, we shall select the option "Other type of secret". This will enable us to store secrets in a key/value structure, that will prove particularly useful when handling this data for use with our task definitions.

We can fill the text boxes with whatever we fancy.

Once we are done filling all the key/value pairs for the secret, we hit "Next" to finalize and name our secret, ignoring all the other options - at least for our case.

Since I am planning to use this secret in a service that does mailing, I named it mailing-env.

Now, the secret is available to us through the AWS API and thus, the Terraform provider.

Retrieving the secret

With the secret available in AWS Secrets Manager, we can use the data source aws_secretsmanager_secret_version to retrieve the secret.

data "aws_secretsmanager_secret_version" "mailing" {
  secret_id = "mailing-env"
}

Formatting the secret's values for further use

Because the received data are encoded and do not have the proper naming conventions for us to use right away in our task definitions, we have to do some local processing, to get them into an appropriate state.

locals {
  mailing_secrets = jsondecode(data.aws_secretsmanager_secret_version.mailing.secret_string)

  mailing_secrets_list = [
    for name, value in local.mailing_secrets : {
      name  = name
      value = value
    }
  ]
}

We are using the jsondecode function to get a representation of the result as Terraform language values.

The next endeavor is creating a list of environment variables in a way that we benefit from.

Since AWS expects the environment variables to be passed as key/value pairs with name - value notation, we are using terraform's for looping functionality to craft a list of objects.

Modelling the task definition with the help of templates

To make life easier and code cleaner, I've abstracted the task definition to a separate template file and stored it in definitions/template.json.

The bit that should be interesting to the reader of this essay is the following. The rest of the definition is boilerplate code that can be found in the appropriate documentation.

[
    {
        "environment": ${jsonencode(environment)}
    }
]

To use the template, we are relying on Terraform's templatefile which renders the template for us and contains it in a single variable.

locals {
  mailing_definition = templatefile("${path.module}/definitions/template.json", {
    environment = local.mailing_secrets_list
  })
}

Using the task definition in a task resource

The last step, although not strictly relevant to this article, is taking advantage of the task definition we created.

Using aws_ecs_task_definition we can assign the value of the local.mailing_definition to the container_definitions attribute.

resource "aws_ecs_task_definition" "mailing" {
   container_definitions = local.mailing_definition
}

The gist

The reason this article exists is that I didn't find a similar one. But, seldom is parthenogenesis in software, and this solution is no different.

These are the resources that led me to this solution.

https://stackoverflow.com/questions/67913171/how-to-pass-a-list-in-template-file-var-section-instead-of-string-in-terraform

https://github.com/hashicorp/terraform-provider-aws/issues/6503

The multiple faces of WebRTC N-peer calling: Mesh, MCU and SFU

Christos Alexiou — Sun, 25 Sep 2022 21:39:42 +0000

It was approximately 458BC when Aeschylus wrote the play Agamemnon. The Greek commander has just returned from the long war in Troy, only to be murdered by his wife Clytemnestra.

The play starts and ends in tragedy, but that's what it is after all. Don't fret though, reader, because the chorus suggests that nothing goes unpunished and that the gods may send Orestes, Agamemnon's son, to restore justice.

Aeschylus will aid the gods with the completion of the scheme Hubris, Ates, Nemesis, and eventually Tisis with the help of his masterful writing. While telling the story, he goes into the trouble of describing to the audience one of the ingenious ways the Greeks used to move information across really large geographical areas; fryctories

The system of fryctories or άγγαρον πυρ (meaning, the telling fire) consisted of Greek soldiers lighting fires on top of mountains and hills across a vast geographical region to transport a message. Unfortunately, the system was binary, so only a certain pre-agreed message could be communicated.

One might say, it was one of the first peer-to-peer systems that relied on technology to transfer information that was used for communication purposes.

WebRTC

The need for communication, and especially peer-to-peer, didn't end with the Trojan war. Eventually, people would have to develop solutions that wouldn't rely on fire, weather conditions, and the terrain of a region. With these prerequisites in mind (or not) Justin Uberti and Peter Thatcher wrote a little piece of software that is called WebRTC.

What it does, essentially, is allow direct peer-to-peer communication between parties, particularly for video and audio.

The collection of codecs, encoders, and processing software is usually bundled in APIs that become available to programming languages through browsers or third-party libraries.

As adoption grew for WebRTC (for example, all major video conferencing software in the world takes advantage of it - Zoom, Slack, etc) base level peer-to-peer communication became a bottleneck for developing features that would enable more complex software features to be written. So, the WebRTC community found alternative ways of utilizing it by setting up the topology differently.

Mesh

Supposing a multiparty where you want to have a group call between the n members. We will abstract the calculations in the end, but for now, let's assume that n = 5.

The 5 participants of the group are trying to communicate with each other, sharing audio and video. The first and most naive approach to enable this communication is to use a mesh network.

In this topology, all participants are directly connected with peer-to-peer connections. So, in this case, we have 5 participants with each one having 4 connections to the rest with a total number of 10 connections. We assume that each connection that will be presented in this article, either upstream or downstream, will be a 1mbps connection.

The total upstream in such a case will be the multiplication of 1 connection with the number of upstream connections we have. So, that would be 4 x 1 Mbps = 4mbps total upstream.

The downstream, in a similar fashion, will be 4 x 1 Mbps = 4mbps total downstream. The total bandwidth requirements are then 5 participants x 4 Mbps = 20mbps.

Taking a step back we can see that:

total bandwidth required = number of participants x number connections x connection bandwidth

TBR = n * (n-1) * abr where abr is average bandwidth requirement

Assuming an average connection bandwidth of 1mbps we can see that for 15 people (which is the average number of students in a school class in the country where the author resides) a typical COVID19-era class would need:

TBR = 15 * 14 * 1 = 210 Mbps

Since the user is picking the entire tab this is extremely impractical.

Participants	Connections (Total)	Downstream	Upstream	Total
5	4 (10)	4mbps	4mbps	20mbps

MCU

Assuming the same scenario as above, with a 5-member multiparty group call we can improve upon the naivety of the mesh topology by introducing a centralized composition system, or as engineers usually refer to them; media servers.

One might refer to this topology as "the star" approach since the media server sits "in the middle" composing the upstream videos of each participant into a single stream that then sends to the participants who receive it downstream.

This topology requires extra CPU as the media server controlled by the provider needs to provide a composite video to each of the participants. At least, it seems like this approach has a 1-1 match as each one of the members sends 1 upstream and receives 1 downstream in return.

This means that each user is sending 1 Mbps of data while also receiving an equivalent amount back.

In this case for the user of this topology:

total bandwidth required = n x (average bandwidth requirement x 1 + average bandwidth requirement x 1)

TBR = n * (2 * abr)

Supposing the 15 people classroom example, a call between them would require:

TBR = 15 * (2 * 1) = 30 Mbps

The only concern is now CPU time, but that is something that we can mitigate either by scaling horizontally or vertically, provided that we can run the media servers in a distributed fashion.

MCU seems promising both for the individual users and also for the network.

Participants	Connections (Total)	Downstream	Upstream	Total
5	1 (5)	1 mbps	1 mbps	10 mbps

SFU

The most modern approach to date is to use an SFU (Selective Forwarding Unit), something that could be resembled a routing service. With the SFU, each participant sends its upstream towards the SFU media server and then that will selectively route this to all other participants.

How much information is going to send, what will that be, and to who depends on the specific way the SFU was programmed along with the network policy defined within the SFU.

In the case of 5 participants, there are going to be 5 connections for each one of them, which translates to 25 total connections across the network.

Each participant has 1 upstream connection and 4 downstream connections, all originating from the SFU media server. Supposing we use the same Average Connection Bandwidth with the other topologies, the upstream bandwidth is 1 Mbps and the downstream is 4 Mbps.

This makes the total bandwidth requirement across the network to be 25 Mbps.

The approach is costlier for the user because while the upstream bandwidth remains at 1 Mbps, the downstream bandwidth is now in direct relationship with the number of participants that the call group will have.

But, the CPU cost of the SFU architecture is far lesser than the equivalent cost required from the MCU architecture.

Increasing the level of abstraction we can see that:

total bandwidth required = number of participants x (upstream bandwidth + downstream bandwidth)

TBR = n * (abr * (n - 1) + abr * 1)

In the example of 15 students attending a video call, the total bandwidth requirement would be

TBR = 15 * (1 * 14 + 1 * 1) = 225 mbps

Participants	Connections (Total)	Downstream	Upstream	Total
5	5 (25)	4 mbps	1 mbps	25 mbps

Conclusion

There is no one solution that fits all sizes in the case of multiparty video/audio streaming or calling. Depending on the specific application requirements the programmer can make decisions on how to architect the network topology so that she gets the most value out of the available resources, including virtual network bandwidth, CPU time, and user resources.

Each topology has its benefits and drawbacks in different use cases.

As a rule of thumb, it seems that the mesh network approach fits solutions that limit the number of participants and want to avoid centralized infrastructure costs but bearing in mind that users will have to pick up the network costs.

The MCU is resource intensive and requires a centralized infrastructure that is responsible for compositing media. Encoding is an expensive process that will require increased resources if the solution needs to be scaled. On the bright side, the bandwidth requirements for the user are minimal.

The middle ground is covered by the SFU architecture. Although still requiring centralized infrastructure, the SFU media server is essentially a byte shifter, forwarding media streams to each participant, making it way less CPU intensive than the MCU approach. This keeps the upstream bandwidth requirements for the user minimal but is increasing the downstream requirement.

With some clever programming and the use of efficient codecs and compression, it's possible to decrease the upstream and downstream bandwidth requirements and eliminate some connections in certain cases. We won't get into details this time as it would be out of scope.

A streamlined view into data integrity and confidentiality in modern Linux environments

Christos Alexiou — Wed, 29 Sep 2021 13:33:14 +0000

Current state of encryption on Generic Linux Distributions

Linux has had support for Full Disk Encryption and technologies such as UEFI Secure Boot and TPMs for a long time. They are usually set up suboptimally in distributions.

Supported:

[x] Full Disk Encryption
[x] UEFI SecureBoot
[x] TPMs

Tools & Mechanisms

We can separate the mechanisms and tools that we have to perform confidentiality and integrity operations in our systems in two large categories; disk encryption and data authentication.

Before we dive into the direction of listing the technologies and tools that are available for each aformentioned category, it would be useful to explain the two concepts.

Disk encryption translates into the process of transforming the data contained within a disk in such a way that reading them in clear-text form is only possible if you possess a secret of some form, usually a password/passphrase.

Data authentication means that there are mechanisms in place verifying that no one can make changes to the data on disk unless they have a secret of some sort.

Technologies empowering disk encryption:

LUKS/cryptsetup
dm-crypt

Technologies empowering data authentication:

LUKS/cryptsetup
dm-verity
dm-integrity

Since we have stated that both disk encryption and data authentication require a secret of some form, we can talk about TPMs. We will focus on one facet of TPMs capabilities, that of protecting secrets. TPMs, in general, release the secret keys only if the code that booted the host can be authenticated. The whole process is roughly explained below:

every boot component is hashed with a cryptographic hash function before it is used
resulting hashes are written to TPM's Platform Configuration Registers, essentially a small volatile memory.
- each step of the process writes the hashes of the resources needed for the next boot step
- PCRs are not freely written. The hashes written are combined with what is already stored in PCRs and the result of that is written to PCRs.
- this process is called "measuring"
secrets are protected not only by these PCR hashes but are also encrypted using a "seed key" that is generated by the TPM chip itself.
TPMs will enforce a limit on unlock attempts per time ("anti-hammering")

Distribution exploitation of these practices

In order to understand how these technologies are used in practive in modern Linux distributions we can take a look at the typical boot process of a distribution today (we are assuming - wrongfully so - that every system at hand is UEFI powered):

the UEFI firmware invokes "shim", a trivial EFI application stored in the EFI System Partition that, when run, attempts to open and execute another application. The "shim" is signed with a Microsoft key, built into all PCs/laptops. The "shim" is measured by the TPM.
The "shim" then loads the boot loader (often Grub) that is signed by a private key owned by the vendor. The boot loader is stored in the ESP or even a separate boot partition. The components of the boot loader are measured by the TPM.
The boot loader calls the kernel and passes an initial ramdisk image (initrd) which is the first userspace code encountered in the boot process. The kernel is also signed by the vendor and is validated via the "shim". The initrd remains unvalidated. Sometimes, the kernel is also measured by the TPM.
The kernel unpacks the initrd image and invokes what is contained in it. This is the first point that the system will interact with the user, asking him for a password for the encrypted root file system. The initrd then uses that to setup the encrypted volume. No TPM measuring takes place at this stage.
The initrd moves into the root file system.
The OS itself is up. It will ask the user for a username and a password. At this point no code authentication, no TPM measurements and no data decryption takes place. The username/password combination is only used for unlocking a certain account.

Encryption strategies

Provided that we are seeking to enforce an disk encryption and/or data authentication policy we have three operating system areas to include into our encryption strategy. We can, of course, choose to only focus on a specific area rather every one of them.

Authentication of OS binaries

Most Linux distributions store system-related binaries under /usr/. Given that it generally contains no secret data - anyone can download the binaries off the Internet anyway, and the sources too - by encrypting this you'll waste CPU cycles, but beyond that it doesn't hurt much. What you can do, although, is use some form of data authentication to verify the integrity of the binary files that your operating system is using to perform tasks. This can be achieved by:

making /usr/ a dm-verity volume. dm-verity is a concept implemented in the Linux kernel that provides authenticity to read-only block devices: every read access is cryptographically verified against a top-level hash value. It makes the /usr/ tree entirely immutable in a very simple way. However, the traditional rpm/apt based update logic cannot work in this mode.
making /usr/ a dm-integrity volume. dm-integrity is a concept provided by the Linux kernel that offers integrity guarantees to writable block devices, i.e. in some ways it can be considered to be a bit like dm-verity while permitting write access. There are multiple ways to use dm-integrity but the one that's most interesting in this use case would be using it in "stand-alone" mode, but with a keyed hash function (e.g. HMAC). This provides authenticity without encryption: if you make changes to the disk without knowing the secret this will be noticed on the next read attempt of the data and result in IO errors.

Encryption/Authentication of OS configuration and state

The OS state and configuration stores, a.k.a stuff in /etc/ and /var/ can be considered as the root file system. The root file system should be both encrypted and authenticated since it might contain secret keys, user passwords, sensitive logs and similar. The encryption of choice here is dm-crypt (LUKS) + dm-integrity. This provides both authenticity and encryption. The secret key must be provided somehow, ideally by the TPM.

Encryption/Authentication of the User's Home Directory

The data in the user's home directory should be encrypted as they usually contain personal and confidential information about the user.

We've seen the boot process of a system in a previous chapter and we now know that during boot, the data in the disk are decrypted after the invocation of the initrd image. In order for user specific encryption to make sense we need to get away from the concept of a system wide key and move to a per-user key. That will ensure that the user's password is what unlocks the user's data.

systemd provides a service called systemd-homed that implements this behavior in a safe way: each user gets its own LUKS volume stored in a loopback file in /home/, and this is enough to synthesize a user account. The encryption password for this volume is the user's account password, thus it's really the password provided at login time that unlocks the user's data.

Wrapping up

So, what is the state of encryption in modern day Linux distributions anyway; it's definitely complicated and fragmented.

And to be honest, I wouldn't expect it to be any other way. Vendors rarely agree on which package manager to ship, little could be done in the direction of a unified encryption strategy. It's up to the user, as is everything in the Linux way of doing things.

Speaking of idealized systems, I believe that simplicity is the key so that vendors can find a unified pattern of empowering their users to encrypt their data.

The ideal OS would be simpler without so many moving parts - especially during the boot process. Since UEFI is there and so is ESP, writing everything there would be the simplest solution. That would allow the firmware authenticate the boot loader/kernel/initrd without any further component for this in place.

In the end, the Linux ecosystem has always been that of a diasporic community and it is this dispersion that feeds creativity and diversity.

DISCLAIMER ⚠️
This work is inspired by Lennart's Poettering publication on "Authenticated Boot and Disk Encryption on Linux". My goal was to streamline the information so that it is digestible and usable in fast-paced research.

The Wooden Walls of Athens: the effect of perceived security

Christos Alexiou — Sun, 03 Jan 2021 14:54:23 +0000

In 492 BC the Persian empire marched against the Greek world. Weeks before the first drop of blood would stain the Greek land, Athenians took the road to Delphi, to consult Pythia about the upcoming battles.

Athena cannot appease Olympian Zeus
With her pleading words and shrewd mêtis,

Yet I speak this word, firm as adamant.
Though all else within Attica’s border shall be taken

Even the secret places on divine Mount Kithairon,
Far-sighted Zeus will grant to Athena a wooden wall.

It alone shall come through uncaptured: good fortune for you and your children.

But do not wait for the host of foot and horse coming overland!

Do not remain still! Turn your back and retreat.
Someday you will yet oppose them.

O divine Salamis, you will destroy many women’s children
When Demeter is scattered or gathered in.

Back in Athens, the words of the oracle were made public and an Assembly was convened to debate them.

If the Athenians obeyed the oracle to the letter, they would flee their land, avoid all contact with Xerxes’ forces, and found a new city far away, at “the ends of the earth.” Some professional diviners and older citizens indeed urged the people to abandon hope and emigrate. According to their interpretation, the gods had promised to protect their own temples behind the thorny hedge that encircled the Acropolis. This, they claimed, was the Wooden Wall of the prophecy.

Nothing could have been more disastrous for Themistocles and his aggressive naval policy than a sudden Athenian resolution to “turn their backs” on the Persians. It would be up to Themistocles himself to bend the prophecy to his purpose.

And when the Assembly met to debate the oracle’s meaning, he did just that. The Wooden Wall was not the palisade around the Acropolis, Themistocles said, but the navy. Its triremes, by now numbering two hundred, would be a wooden bulwark for the people’s defense. Apollo had revealed that this floating Wooden Wall would endure and bring benefits for generations to come. The Athenian citizens should man their ships, not to flee, but to face the Persians at sea.

Themistocles marched and sailed. In Salamis, the Persian fleet was destroyed. Salamis is considered one of the battles that have shaped the modern world.

Had the Persians successfully invaded Greece in 490 BC, EMEA history would be different.

Now, I hear the people that clicked this post wanted to learn about security.

Allegories were particularly favourable in Ancient Greece and I find myself charmed by the level of intricacy and witting that they require. I practice their art from time to time and this post seemed like a good excuse for me to do so.

Security engineers in virtually every organisation train their colleagues on the importance of securing their secrets. Sec engineers are like the oracles of Delphi.

Trying to predict the attack vectors of a system, constantly evaluating the situation that the system will be operating at, trying to assess the security of the system while always trying to please the manager and not cause a lot of discomfort.

Exactly like an oracle wants to please the priests while not causing a lot of discomfort to the king that came asking for a prophecy.

I have preached countless times on the importance of keeping one's secrets, well, secret.

Engineers listened. They implemented key stores. Or at least, they thought they did. Just like Athenians thought they understood Pythia, and they almost caused the Western civilisation to collapse under the pressure of Eastern forces.

Hadn't it been for Themistocles.

I want security engineers to stop acting like oracles and start working like Themistocles.

Do not preach of security, rather start working towards it. Make engineers understand how to implement security mechanisms and get to the extent of teaching them. Do not be a contributor to the "perceived security" effect.

In addition to allegories I also enjoy parables.

Parables are literary creations that allow one to transfer a message to her audience with an expressive and figurative way.

Let's examine the parable of "the insecure key store."

Software applications that implement cryptography need to create and store cryptographic keys and possibly certificates to properly operate and service their clients.

These keys and certificates might be stored in memory while the application uses them or stored in a permanent store for later use.

In either case, developers must take the appropriate security measures to limit the access to this store also known as keystore.

An insecure keystore bug allows an attacker to read cryptographic material such as keys and certificates from the keystore to use them during a cryptographic attack.

Once, in the city of Sparta, there was an engineer who was responsible for crafting a saferoom, so king Vrasidas can store all of his valuable pieces of information that his spies were collecting from all over Greece and Persia.

The engineer worked day and night and in a week, he had made the most fancy saferoom that any king has ever seen. A door so big and tough that it would take one thousand Persian horses ramming with their heads against it to even crack it. And walls so thick and tight that you couldn't even hear a thunder when you were inside the room.

On the door of this masterpiece, a large golden lock was installed. Crafted with the shinier gold which the engineer was able to find in all of Sparta.

The lock was created by the finest locksmith in the entire province of Lakonia. The key was so heavy that only a true king would be able to lift it and unlock the huge door of the saferoom.

When king Vrasidas saw the saferoom he was amazed. He gave to the engineer so much gold that he could build a house out of it.

The king went to his room for the night and he slept peacefully after all this time. He finally knew that his secrets were safe.

The next morning, a farmer walking by the side of the road passed by the saferoom, early in the morning.

The huge door was open! He walked with cautiousness towards the building thinking that the king has woken up with the first light of the day to study his documents.

The saferoom was empty. Completely empty. Not even a single piece of parchment was on the floor or hanging in the walls.

He started shouting until the city guards heard him and urged him to the king.

- "My king! The door! It was open. The saferoom. There was nothing inside, my king. My king, I know nothing more, please spare me!"

Vrasidas was pale. He was ready to collapse at his throne. He whispered to the ear of the guard that was standing next to him to fetch the engineer.

The engineer was sleeping in his house. Around him there were many hetaire. The guards dragged him, naked as he was, in front of the king.

- "What's the meaning of all this ?", the engineer managed to spell, naked and disorianted as he were.

- "I have lost all my parchments. Everything is GONE!", yelled the Spartan king.

- "That's impossible.", said the engineer. "The walls, the door, the lock would be admired even by the gods, my king.", shouted the engineer trying to excuse himself.

- "Do you have the key, my king ?", continued.

- "I have bedded with the key, like it was my wife, you fool. I have kept it under my pillow, along with my knife so that anyone that came for it would meet my blade!" the king raged.

- "May I see it, my King?" said the engineer.

King Vrasidas reached for the key in the chest that one of the guards were carrying up to this moment. he opened it with care, and gave the key to the engineer to inspect.

- "That's impossible. This can't have happened. Your saferoom can't be breached. This key is exactly like the one Aristarchos, the Lakonian locksmith gave to the Athenian lords, and their saferoom has NEVER been breached".

Now, modern engineers are no different species than the engineers of the past. They make mistakes and they can cause a lot of trouble.

The ancient Spartan engineer thought that by acquiring the same key as his Athenian counterpart, the saferoom for King Vrasidas would be unbreakable.

As this Java engineer thought that by creating a key store his secrets would be safe.

KeyStore keyStore = KeyStore.getInstance("JKS");   
String fileName = System.getProperty("java.home") +    
"/lib/security/myKeyStore.jks";   
FileInputStream stream = new FileInputStream(new File(fileName));   
keyStore.load( stream, "storeit".toCharArray());

He also forgot that java.home/lib/security may be readable by everyone.

Mistakes are commonplace. If you want to have a secure system, be like Themistocles. Act, before someone puts your city's defence under test.

A story on indexing and why it is important for modern applications

Christos Alexiou — Sat, 04 Apr 2020 14:38:34 +0000

Modern web applications need performance

The majority of modern web applications, if not all of them, are utilising some kind of data management and storing solution. Most of these applications are relying on modern database systems such as MySQL or MongoDB to name a few.

As more and more users are moving their workloads online, either in the form of performing simple word processing and light image editing or in heavily CPU and data retrieval bound applications such as BPM solutions for large corporations, "speed and power" (sic) are indispensable.

Modern database solutions have a vast amount of perks for even a seasoned web developer to utilize in order to make her application fly.

Utilizing indexes for faster data retrieval

Indexing stuff for the sake of finding it faster is not a new kid in the block. For example, scholars in the great Library of Alexandria in Egypt have been using a lookup system for quickly finding papers that were interesting to them as back as 280BC. That system was heavily relying on indexing and was designed by Calimachus, a poet, critic and scholar of the time. The name of the system was "Pinakes", translating in English as "Tables".

Is indexing still relevant today ?

Let's start our short journey in indexing by explaining what indexing actually is in a context that may actually be useful for modern day data retrieval solutions developers and not for ancient Alexandria scholars.

Typical database systems are storing data on slow disks where I/O bottlenecks are almost innevitable. Even with modern solid state drives the speed of retrieving raw data out of the disk has not given us yet that "whoosh" moment.

In-memory database solutions are becoming a thing today with Redis and other caching mechanisms and message brokers but they are still a long way for being a fully fledged database system that we can actually store all of our data - largely due to hardware limitations and cost.

Even if we utilize such caching solutions like Redis, there is a certainty that on some occassions the application will have to query the database and then, the database retrieve data that they are stored on disk. So, I/O lag is a thing and we have to make up for that by making our database searches more performant.

Indexes are an incredibly powerful feature of relational databases. In fact, using indexes properly can often improve the performance of your queries by an order of magnitude.

Indexes allow you to “ask questions” without having to search the entire table on disk. They’re really useful for answering “needle in a haystack” type of questions, where you’d otherwise have to look through many rows to pick out only the one that you actually need.

An index is a data structure that’s stored persistently on disk, and is typically implemented as either:

"B-Tree" -- fast for all sorts of comparisons (>, <, ==)

"Hash Table" -- fast only for equality comparison (==)

Without an index, performing a SELECT ... FROM table WHERE ...; statement requires the DBMS to search through every single row in the table to see if it matches that exact WHERE clause. If you have more than a few hundreds rows in a table - and most modern web applications have millions if not billions - this takes a non-negligible amount of time.

But by adding an index to a column that you frequently run lookups on, you can decrease this amount of time required. Instead of having to scan n rows (where n is the total number of rows in the database), the database can return your query by looking at either log(n) rows (if indexes are implemented as a b-tree) or simply one row (if indexes are implemented as a hash table).

Once you’ve set an index on a particular column, you don’t need to do anything to make subsequent queries use it. The query planner in the DBMS will use that index if it needs to.

Is it all that sugary and sweet ?

What was mentioned above are the stellar reasons of using indexes in your relational database. But, as with everything in this world, there are a few downsides to be aware of. Since the index is stored as a separate data structure, building too many indexes will result in additional space on disk.

One other thing to keep in mind when using indexes is the overhead of index creation and maintenance.

Whenever a new row is added to the table, the index’s data structure must also be updated so that the index remains accurate.

Sacrificing a bit of performance on writes to get much better performance on reads seems a good deal for me.

Thanks for your time,
stay safe and sane.