Forem: Maow

Computers, explained by a nerd.

Maow — Thu, 04 Apr 2024 12:50:40 +0000

Today, a close friend of mine asked me a compelling question:

How the F*** did we make tiny patterns of electricity do all of THIS.

I answered with a tangent about the foundations of computer systems. This is that, but I felt it necessary to be more eloquent.

All jokes aside, welcome to the world of computing! It's the closest thing we have to magic in this world; it's just that we use numbers instead of mana... lots of numbers.

Circuitry

Everything complex starts from something simple. Everything simple hides a deep complexity. The fundamental components of computers are electronic circuits, devices comprised of their own components, such as logic gates.

Logic gates allow electrical signals to make decisions. An "AND" gate is referred to as such because it receives two signals and produces a signal if given both signal 1 AND signal 2. A "NOT" gate will produce the opposite of what it is given. And so on.

Complex devices, such as latches and multiplexers, can be formed through logic gates. Logic gates and signals are, in fact, at the core of how computers work with numbers and arithmetic.

Numbers and Arithmetic

As you're likely aware, Western cultures use the decimal number system, also known as "base 10." This means we use ten unique digits to represent our numbers: 0 to 9. Computers, on the other hand, use binary, or "base 2." They only have 0 and 1.

The reason is simple. Think about a computer; it has only two states: On or off. This is a binary state. An electrical signal can be present or absent.

In binary, we form weightier numbers by joining bits. Each bit represents a number, and we add the number of every "present" bit together. Specifically, we start with the least significant bit, which has the value of one when present and is the rightmost bit; therefore, we read from right to left.

With computers, we group bits into octets, pairs of eight, and we refer to these as bytes. A byte is the tiniest unit of information we can save or retrieve, but not the only unit. By grouping bytes, higher capacity numbers are made.

Since our numbers are bits, we can manipulate them like electrical signals. Through logic gates, we can create adders. Adders, as the name implies, add bits.

Components of a Computer

Central Processing Unit (CPU), or processor: The CPU is the brain and nervous system of the computer. It does most of the complex work and coordinates most components together.
Motherboard: The motherboard is a large circuit board that serves as the "body" of the computer. It allows parts to communicate with one another, like nerves, and it provides system information and firmware.
- Firmware is a bundle of code given to the processor by the motherboard when the computer powers on. It sets up the machine so the operating system, e.g. Windows or macOS, can begin operating.
Random Access Memory (RAM) or memory: Cards that comprise the computer's working memory. Memory is a huge list of bytes, where each byte has a position, also known as an "address," like a person's particular home address. When visiting certain, special addresses, signals may be redirected to other components by the motherboard. This is known as "memory-mapped input/output" and is how the processor communicates with much of the system. RAM is volatile, meaning when it loses power, its contents are lost.
Storage Disk, or disk: Disks contain memory, similar to RAM, but with the added benefit that their memory is persistent. However, disks are often slower for the processor to access than RAM since the signals must travel farther.
Graphics Processing Unit (GPU), or graphics processor: The GPU helps display graphics on your screens by performing many operations simultaneously. It isn't as sophisticated as the CPU, but it is faster at simple operations. It has its own memory, "video RAM."

Instructions

For a computer to be useful, we need the ability to give instructions to the processor. To do so, we provide a group of bytes subject to a specialized format known as an instruction set architecture. For educational purposes, I'll be working with a fictional, simplified architecture.

At the beginning of our instruction, we have a byte specifying the instruction we're dealing with. This is known as the "opcode." Let's introduce the basic opcodes:

0 = Add      ("add")
1 = Subtract ("sub")
2 = Multiply ("mul")
3 = Divide   ("div")

Next, these instructions need some numbers to work with. Addition is a binary operation, meaning it requires two numbers. Let's put them directly after the opcode:

"1 + 1" correlates to: [0, 1, 1]

We can now provide this to our processor and it will handle our instruction. Where does the sum go, though? Simple!

Registers

The processor has several "registers," which are memory cells capable of keeping a single value. When the processor performs an operation, the result of that operation is likely to go into a register. Some registers are also used for special purposes, but we'll cover those more soon.

Let's say that the sum of our previous addition operation ended up in an "accumulator" register. We'll label it ACC. We can use registers within instructions, extracting their values.

"2 * ACC" correlates to: [2, 2, ACC]

Since the product is stored in ACC, ACC is now 4.

If we want to store more than just a few bytes but rather a collection of subsequent bytes, we'll have to use something with a much higher capacity, like memory.

Memory

Let's say we have a person and this person has a name and age. We can represent this person with bytes by breaking them down into a "name" and "age" value.

First, to store their name, we need a text encoding. A text encoding is a format of storing letters, numbers, symbols, etc. as bytes for use in computer programs.
Commonly, we use ASCII or Unicode, but let's just say 0 to 25 represents the letters of the Latin alphabet, where a byte corresponds to a letter, i.e. 0 = A, 1 = B, and so on. This series of bytes is usually referred to as an "array."

Additionally, we need some way to say how long our text is. So, let's just store the length directly as a prefix!

"MAOW" correlates to [4, 12, 0, 14, 22]

For the person's age, let's just use a normal byte for simplicity.

And so, with all that in mind, if we were to store the person in memory we would just store their name and then subsequently their age, right? Correct. Let's give the age the label of "age" and the name "name."

But wait... if we have the address of "name," the byte at its beginning, then we actually have the memory address of the person, since name is at the beginning of the person.

This means that this person we've been talking about is a struct. A struct, short for structure, is a group of fields, which are named values. By composing structs, we can represent complex concepts with bytes!

Jumping!

If the instructions we give to a processor are bytes, then does that mean we can store instructions in memory? Yes, absolutely. In fact, operating systems copy-paste programs into memory and then provide the processor with their addresses, allowing the processor to read instruction by instruction like a list.

Tangent: Any data that the program might need to operate correctly is also stored as part of the program if not as a separate file. The data just lives outside of the "code section" of the file, which has our instructions, and instead in the "data section." This is what an .exe file facilitates, if you've ever wondered.

Each instruction begins at a certain address, and we can use that to our advantage by introducing new "jump" instructions. A jump simply tells the processor to jump forward or backwards to a different instruction using the address of that instruction.

4 = Jump ("jmp")
5 = Jump If Not Zero ("ifnz")
6 = Jump If Zero ("ifz")
7 = Test ("test")

Hold on, wait. What're those "if" and "test" instructions?

Let's back up just a bit (badum tss). First, we covered logic gates a while ago, but I neglected to mention we can use operations like "AND" with bytes. These are referred to as "bitwise operations," since they work with the bits of the byte, rather than the byte itself. AND-ing a byte with another byte simply lines the bytes up and applies the AND to each pair of bits.

Secondly, one of the special registers that a processor might have is the FLAGS register, which states a bunch of useful "flags" usually related to an operation or the processor itself. To set these flags, we'll be using the test instruction.

The test instruction ANDs two numbers and produces flags from the result of that AND. If we provide the test instruction with the same number twice, we can simply get the flags for that number as the AND won't affect it.

The main flag we're after is the zero flag, which states whether the tested number was zero. This is how "jump if not zero" works! That instruction will only jump to the specified instruction address if the zero flag didn't return positive. This is called a conditional branch, or just condition, and is essential for complex tasks.

(Sub)routines

Somewhere in our journey, we migrated from the realm of circuitry to the realm of software. Everything you run on your computer is software, including your operating system which helps manage your applications and system resources, and as defined before, a program is a list of instructions, but the proper term for this would be a "routine."

Therefore, what's a subroutine?

A subroutine is a routine that within a routine, usually to isolate some code so we can jump to it to re-use it at any time, rather than repeating the same code over and over.

Some subroutines expect us to provide them with values prior to running them, much like instructions do, and some subroutines will provide us with values once they've finished running.

To provide these values we store them either in general-purpose processor registers or in memory. Once a subroutine is done, it will do the same with its return values and then jump back to where we jumped from.

Programming Languages

To sum them up briefly: Code written in a language is simply specially formatted bytes that get interpreted by another program which generates a program the processor can actually read. By representing these complicated "features" in text, we make it much more approachable and understandable for humans, as well as potentially safer and faster.

All programming languages are different in some way, just like all natural languages.

Closing

Aren't computers really f***ing cool? We've gotten to a point in history where we don't even realize everything happening in our metal boxes is just electrical signals.

These signals represent numbers, which are used to represent abstract concepts and data. Images and sounds become numbers, which we can transmit to people hundreds of kilometers away through the internet.

Hope you had fun :D

Alternative Views of Software Ecosystems

Maow — Wed, 20 Sep 2023 11:07:58 +0000

For the longest time, software ecosystems have been dominated by one concept. Libraries. There is nothing wrong with the concept of redistributable packages. Such an idea is what allows any developer to build almost any application...

but is there a better way? That's what I wish to share my thoughts on: the idea that we can build any application with ease, as long as we think about our ecosystems the right way.

Abstractions

Abstractions are one of the most core concepts when it comes to software, the idea of taking complex, sometimes unintuitive logic and providing a view of it which is simple and intuitive.

This is most prevalent as programming languages, which give us developers the ability to fluently, easily express our logic and data types, minimizing errors, oversights, and difficulties.

I believe there are multiple types of abstractions in today's environment.

Hardware Abstractions: Abstracting over low-level hardware calls. Examples include graphics drivers, kernels, etc.
Platform Abstractions: Abstracting over the user's platform elements, e.g. architecture and operating system. Examples include graphics APIs (DirectX, Metal), windowing libraries (SDL, winit), programming languages, etc. These can allow developers to write cross-platform software.
Software Abstractions: Abstracting over platform abstractions or, in a recursive fashion, software abstractions. Examples include GUI libraries (Qt, GTK), HTTP servers, etc. These can allow developers to write simple software, and also serve cross-platform software.

Monolithic Libraries

Abstractions are often monolithic. This means they are singular units that cover a large surface area. The most prevalent form of monolithic library is a framework.

Problems

Say you wanted to write your own HTTP server that supports HTTP/1, HTTP/2, HTTP/3, TLS, etc.? This is quite the feat! You would need to write much code yourself, and HTTP grows larger and larger by the day, so your library would eventually grow too outdated if left unmaintained.

It is much more convenient to work with libraries that provide a fully-fledged, albeit somewhat complex HTTP server, and to build a software abstraction over that server.

This is the approach followed nowadays by libraries like Express (built on top of Node.js's http module), and Rust's own HTTP ecosystem, most of which is built on the Tower middleware system and Hyper client/server implementation.

But... what if we decided we wanted to customize the HTTP parser?

Depending on the library, this is possible, but wouldn't it just be easier to tweak the existing parser a little, or perhaps replace it with one that is more suited to our needs? Unfortunately, this means we would either need some type of plugin system (which can cause our program to become slower and possibly more complex) or we would need to fork the project which incurs a higher maintenance cost as we would need to be up-to-date with upstream!

Component Libraries

Component libraries are a concept that would help us achieve this goal. Rather than create an HTTP framework that includes parsing, writing, transport, routing, etc. We split each feature into a component library, i.e. a non-monolithic software abstraction, meaning it does one thing, does it well, and can be swapped out by a library consumer.

Rather than creating frameworks, we are essentially delegating the work of combining each component to the library consumer, i.e. a person developing a monolithic library (monolithic software abstraction) or an application developer.

This may sound complicated, as we are requiring our library consumers to put in more work, but it's actually the contrary. As implied in the last sentence, monolithic libraries exist for general-purpose needs, i.e. when someone just needs something to work.

Additionally, the combination of these component libraries should be simple enough! Almost comparable to just using a regular monolithic library, even. If we need more complexity, i.e. the existing options just don't cut it in terms of features, we can replace a component library with a lower level abstraction or even our own.
As software abstractions can be abstractions of software abstractions, we can even make a component more simple rather than more complex if we really want to.

In theory, you could create a lightweight, simple, and perfect (for your needs) HTTP server by just importing the required components, e.g. router, parser, writer.

Swappable Libraries

In the Linux ecosystem, there exists multiple different sound server projects (ALSA, JACK, PulseAudio, and PipeWire). It's common for a user to need multiple of these sound servers as different programs require different APIs, although...

It is possible to replace ALSA, JACK, and PulseAudio with a PipeWire-based implementation of their APIs. This unifies all APIs to one sound server, meaning your system is more cohesive and you can benefit from PipeWire's improved latency regardless of the program.

This is all possible due to C/C++'s "header" system, something dropped in more modern programming languages. This is, in a way, an example of the facade design pattern. Facades are also commonly used by logging libraries, i.e. Java's SLF4J and Rust's log.

Logging facades allow developers to use one API while swapping out the implementation dependent on their program's needs, like how SLF4J has a "no-op" logger that doesn't actually log messages, a "simple" logger for those who don't need a heavily customizable logger, and etc.

It is inevitable that we will run into a situation where you want to use a monolithic library but it uses an abstraction you can't because it just conflicts with another library in your dependencies. So, rather than have a library dependent directly on that abstraction, why not have it depend on an interface? This would allow program developers to select which implementation they want for that interface across all dependencies, keeping the project cohesive.

Applications

Nowadays, we run into an issue where software we're using is lacking features we want... or has features we really don't want. There is no perfect software, just good enough, and to get to something that is perfect (a subjective matter), we need to write our own.

But this is not always an easy task. Look at digital audio workstations, integrated development environments, browsers, game engines, command-line shells/terminal emulators, and operating systems! These are all hard projects to work on because there exists many features that a developer is forced to implement, sometimes complex features (especially with OSes), even though they exist in other projects anyway.

It is simply not possible for someone to make a set of software that is perfect for them within a reasonable set of time... but what if it was?

The ideal software ecosystem is one where we can have each feature for each of these different applications all existing as their own component libraries, so developers can take what they need, avoid what they don't, reimplement what they require, and perhaps even make monolithic libraries so other developers can benefit.

Closing Notes

What if we lived in a world where it was possible to make your own Unix-like within a week? Where you could have a program that perfectly fit your workflow, needs, wants, ideals, etc. because you made it?

As time goes on, we become more dependent on the simple options regardless of whether or not they are the right ones. We also become more dependent on software that belongs to organizations that we do not agree with, which strive for profit rather than consumer happiness. Things go closed-source out of nowhere, horrible pricing models are introduced, and management conflicts with community. We need to take back control of our software.

I do not suggest that this "alternative view" is perfect, but I do think it's a step in the right direction.

What Websites Have Become (and what they could be)

Maow — Wed, 03 May 2023 18:22:05 +0000

Overview

The World Wide Web, or just "the Web," is an increasingly complex set of technologies that allows us to communicate and share information all over the world.
It's existed for generations, and only gotten more and more advanced as time has gone on due to various safety, performance, and usability additions.

In the modern day and age, websites have slowly transitioned from being focused around sharing text, files, and links, and more towards intricate services; web applications.
In this article, I'd like to illustrate what makes web applications convenient, what we could learn from desktop applications to improve web applications, and what my personal proposal is for the future of the web.

Web Applications vs. Desktop Applications

Discovery, Bookmarking, and Package Managers

Finding an app is the first step to using it. Web applications are usually quite easy to find, only requiring you to use a search engine. People can also share links to the app. While these things are still true with desktop apps, as they'd have their own site, there's the problem of having to download, possibly install, and then run a program.

Although, desktop apps benefit from package managers, app launchers, and start menus which all help the user install, uninstall, update, and access desktop apps.
The only issue is that package managers and app launchers are primarily found in the Linux community, as package managers are usually command-line apps (that's not to say Windows doesn't have one, it's just scarcely used) and other operating systems do not need app launchers. In the browser world, the closest thing to a package manager is the bookmarks bar, as that allows you to save links for quick access.

Sandboxing, Safety, and Malware

One of the immediate benefits of the Web is that many websites are secured with TLS, meaning all HTTP requests sent to those websites are encrypted. In a desktop app, this is something a developer has to go out of their way to implement, and many independent developers have limited experience with cybersecurity.
Adding to this, desktop programs have the ability to execute any code. There are plenty of viruses that have circumvented the need for permissions, and it's not even necessary to have permissions to wreak havoc. Attackers find new ways to exploit processors and operating systems to circumvent safety patches, precautions, and anti-malware programs. There is a constant arms race.

With new web technologies, such as WebAssembly (WASM) and WebGPU, we have the ability to make performant and modern apps in a completely sandboxed environment. WebAssembly has to be safe, considering it takes advantage of the end-user's processor. This is effectively remote code execution, which would be terrible with unregulated, malicious code, but WebAssembly is designed to not be capable of such code.
Better yet, you can compile to WebAssembly from various desktop programming languages, like C, C++, and Rust. This is not only beneficial for developers who want to work with lower level code than JavaScript, usually for performance benefits, but it also means we can develop applications for both web and desktop platforms with relative ease.

That's not to say that websites are perfectly safe. Many will make you download malware, steal credentials by posing as real sites, or scam you with misleading pop-ups. Cautious users will learn to recognize fake looking URLs and suspicious websites, and browsers combat these by showing warnings, though not all of them are blocked. The issue here is that search engines will show anything they can find. A lot of these malicious sites are not exactly front page, but it's still harmful to display them to unsuspecting users in any way.

Bundling and Binaries

One of the issues many front-end web developers run into is making their site load quickly. The cause of this is usually due to sites serving large files or, more often, many files. Many tools and services were designed to combat this issue; CDNs cache commonly used CSS and JS libraries, bundlers minimize and bundle code and assets into easily cache-able segments. WebAssembly even aids smaller download sizes as it is a bytecode language, meaning it is compiled and stored as a series of bytes, rather than as text, like JavaScript. There's even been changes made to HTTP to help with this problem.

The issue here is that we are putting a lot of work into something that desktop apps have no issue with, as many do not need to load anything, and if they do, it's usually just from the filesystem, which is a lot faster than downloading multiple files from an HTTP connection. Where desktop apps fail is in downloading them. A lot of apps come with heavy file sizes, usually due to the libraries they require, or their assets. This means you might have to download a couple hundred megabytes for one app. A lot of these apps are actually Electron instances. Electron is a framework that lets you make desktop apps with web technologies using Node.js and a version of the Chromium browser, the backbone of Chrome, Edge, Opera, and various other browsers. Many apps you know of, including but not limited to Discord, Steam, and Spotify, all render their interface using browsers, either partially or fully. This allows for flexible, modern visuals with CSS, but it has a major impact on memory usage and file size.

A Proposal for the Web

As you can tell, both desktop and web have pros and cons. There isn't much we can do to help with the issues surrounding desktop, but we can improve the web, we do it all the time. Not only could we make it even safer, but we could make it easier to discover and manage apps as well as easier to load them. We could even decouple HTML, CSS, JS, and more from browsers, so developers have a choice over what they can use, and users can be given the most effective, fast software.

Ditch Web Browsers!

This is going to spark a lot of mixed reactions: I personally think that we need to abandon the old web if we want to truly improve. Web browsers are insanely complex pieces of software, they are focused on many different things from accessibility to responsive web apps to video and audio playback to high-performance web games. We could simplify things. Backwards compatibility is a strange focus of web browsers. Many developers take the time to make their sites compatible with Internet Explorer, a heavily outdated, slow, and all around terrible browser you can't even use anymore.

We can keep the good ideas, but there will be a lot of rewriting bad ideas if we want things to be vastly improved.

Web Executables

Instead of web browsers, I propose we have an app that has the ability to load web bundles (AKA web executables). They would use an archive format that is easy to compress and verify but also fast to traverse, allowing our new web browser to easily find and read resources. When our browser receives a bundle, it will find specifically WebAssembly code to execute on a new thread. This code will handle rendering, page events, and various other things. The "WebAssembly first" approach decouples code execution from JavaScript. JS would stop being a primary focus of browsers, perhaps having its own runtime alongside WebAssembly, but more likely, it or a derivative of it would be compiled to WebAssembly.

Rendering

Don't be worried, this concept doesn't entail forcing everyone to write low-level WebGPU code. We will allow that, but frameworks will exist to render pages in a more traditional way, possibly even allowing developers to load CSS resources. This completely decouples rendering from CSS and gives people more freedom to make their own formats and frameworks. No longer would you need a tool for transpiling SASS or LESS, just the library for it.

App Discovery

Not only can you access apps via a URL, but our browser will have first-party support for "app discovery portals." Think app stores but without the monetization. The keen amongst you will notice the plural, "portals." Yes, decentralization.
Now, an issue arises when you allow people to create their own portals. It goes from an app store to more of a repository, which is alright, but it does complicate the discovery part. I don't want to think about an "app discovery portal discovery portal," so the alternative is going to derive from the old web: Word of mouth! If you want your portal to be found, make it worth finding. Link to your portal from your high-quality web app.

Access Protocol

The pattern of decoupling the web continues as HTTP will stop being the primary way to retrieve resources. It would still be possible to use HTTP, but you would require an HTTP client/server library.

The alternative to HTTP is a very simple protocol designed just for requesting and retrieving bundles as well as accessing app discovery portals. You visit a web:// link and it either brings you to an app (via downloading a bundle) or a discovery portal (via downloading some metadata). In our browser, we'd even allow for saving portals and apps to our homepage.

Sockets

With the removal of HTTP, you gain the benefit of sockets, allowing you to send packets, rather than just make requests to an HTTP server.

A bit of work would have to go into making these sockets safe for the end-user. We obviously can't use plain TCP sockets since an attacker could make an app that makes the end-user send malicious packets. Regardless, the work is worth it. The amount of potential this gives developers is astonishing.

Closing

There's a lot we can do to improve the web. Speed, safety, and usability are the primary factors in designing an application, and as such, are the focus of my set of improvements. I do also believe in developer experience. Modern programming languages have not only shifted towards better user experience, but better dev experience too, and it's why I designed a lot of this around giving developers more choices and power.

You don't have to agree with all of the ideas I've presented. In fact, if you have anything to say, I'd highly recommend doing so! Feel free to comment, tell me this is stupid, or whatever. I thank you for sticking to the end.

A Deep Dive into Lambdas

Maow — Sun, 05 Feb 2023 16:44:36 +0000

Introduced in Java 1.8, the lambda expression (a.k.a. closure, anonymous function) is an important piece of syntax for code cleanliness, reduced repetition, and functionality.

It provides the language—and its siblings, like Kotlin and Scala—with the ability to store a function as a value, which can also then be used to implement higher-order functions, i.e. functions that take or return other functions.

But how does it all work underneath the surface?

Functional Interfaces

As Java is a statically typed language, a lambda needs to have a type. The type of a lambda includes its parameters and its return type, which can be seen with functional interfaces like Function<String, String>, analogous to String function(String).

What exactly makes a functional interface so special?

They have a single restriction: They may contain multiple default or static methods, but only one abstract method. This is why functional interfaces are sometimes referred to as "SAM (Single Abstract Method) interfaces."

This restriction exists precisely because the compiler needs to know which method will be implemented to create the lambda, for instance, get in Supplier or apply in Function. With multiple abstract methods, any of them could be implemented when writing a lambda.

Small example of a custom functional interface:

// The @FunctionalInterface annotation is
// used to tell the compiler to enforce the SAM restriction.
@FunctionalInterface
interface TokenProvider {
    ApiToken get();
}

The Low-Level

In order to talk about how lambdas really work, we need to give a brief explanation of the Java Virtual Machine (hereby referred to as the JVM).

When a JVM-based language (Java, Kotlin, etc.) is compiled, it outputs a file with the .class extension. Class files contain what is called bytecode. Bytecode is a term used to describe any compiled code made of raw bytes that is executed by a virtual machine rather than your CPU.

A virtual machine (not to be confused with VirtualBox, VMware, QEMU, and etc.) is a program that processes and executes bytecode, with the optional step of transforming that bytecode into optimized machine code, which is ran on the CPU. Each virtual machine handles a different bytecode language, i.e. the JVM handles JVM bytecode, which is designed specially for the Java platform and contains information about classes, fields, methods, and etc.

Bytecode Instructions

In the JVM, code is split up into single-purpose functions called "instructions." Instructions have a "mnemonic" and set of arguments, the mnemonic being the name of that specific instruction in the bytecode language.

For a simple example:

ldc "Hello, World!"

This is a single instruction with the mnemonic ldc, short for "load constant."
When it's run, it takes the first argument (which must be some kind of constant value, such as a string, int, or float) and pushes it onto the "operand stack," a stack data structure used for performing operations on values.

When your code is compiled, it is turned into a list of instructions. When your code is run, the JVM starts at the first instruction and makes its way down until it reaches the end or reaches a jump (like an if statement) which makes the JVM jump to a different instruction and begin reading down the list.

Lambda Bodies

You would probably be surprised to learn that the body of each lambda you declare in a class actually becomes a private method in that class. These lambda body methods are generated by the compiler and contain all the code placed in your lambda body.
They follow a specific naming scheme: lambda$<originating method>$<index>
So, the first lambda in your main method would have its body become the lambda$main$0 method.

Keep all of this in mind! It will be very important later on.

Method Handles and `java.lang.invoke`

Java's Invocation API (java.lang.invoke) was introduced back in Java 1.7. It allows you to create "method handles," which are invokable references to a method or constructor. These differ from the Reflection API in many ways, the most well-known difference being that access checks are performed on method handle creation, rather than on each call like with Reflection.

An additional type provided by the Invocation API is CallSite, which holds a method handle. Despite the simplicity of this, CallSite is very important for reasons that will be explained shortly.

`invokedynamic`

The core component of lambdas is the invokedynamic instruction. It, like all other invoke- instructions, invokes/calls a method.

However, the difference between it and other invoke- instructions is that the first time that invokedynamic is run, it will invoke a "bootstrap method." This is a method that returns a CallSite object that is "bound" to that invokedynamic instruction, meaning it only needs to be retrieved once. After the CallSite is retrieved, its underlying method handle is invoked and, assuming the method handle returns a value, the returned value is pushed to the operand stack.

This instruction allows us to essentially receive functions that are to be invoked while avoiding performance downsides and compatibility issues.

But, invokedynamic is not the only piece of the puzzle.

The Lambda Metafactory

Under java.lang.invoke exists a highly low-level class called LambdaMetafactory. This class is used to create method handles that return "function objects," which are objects that implement functional interfaces.

The way this is done is somewhat complicated, so bear with me.

LambdaMetafactory has a function named metafactory that returns a CallSite after being given the following parameters (with only a few excluded for simplicity):

The name of the method to be implemented, for instance, "get" when implementing a Supplier.
The descriptor of the method to be implemented, meaning the return type and parameter types. For a Supplier<String>'s method, there would be a String return type but no parameter types.
The descriptor of the CallSite, where the parameter types are the captured local variables and the return type is the interface that should be implemented.
A handle to the implementation method, which is called in the interface's method. This is equivalent to the lambda body.

The most interesting part is what the method handle the lambda metafactory creates is.

It generates bytecode at runtime. This bytecode instantiates a new function object that calls the lambda body method with any necessary parameters, and then creates a new method handle that calls that bytecode before returning it in a call site.

The bytecode generation is done via the help of a third-party library called ObjectWeb ASM that allows you to parse and write JVM bytecode very easily. The standard library uses it in this situation due to its performance and feature-completeness.

Finally, a real example of what one of these function objects looks like:

final class Test$$Lambda$1 implements Supplier<String> {
    private final String arg$1;

    private Test$$Lambda$1(String arg$1) {
        this.arg$1 = arg$1;
    }

    public String get() {
        return Test.lambda$main$0(arg$1);
    }
}
// Oh no, it's hideous.

That function object was generated from the following code:

public class Test {
    public static void main(String[] args) {
        String value = "Hello, World!";
        Supplier<String> msg = () -> value;
        msg.get();
    }
}

End

That is basically all there is to the creation and invocation of lambdas. I'm personally compelled by the unexpected complexity of Java's lambdas and the wonderful design that went into its dynamic invocations, so I felt the need to share this information in a (somewhat) easy to digest way.

I hope you found this either fun, interesting, or useful. Bye!

Forem: Maow

Computers, explained by a nerd.

Circuitry

Numbers and Arithmetic

Components of a Computer

Instructions

Registers

Memory

Jumping!

(Sub)routines

Programming Languages

Closing

Alternative Views of Software Ecosystems

Abstractions

Monolithic Libraries

Problems

Component Libraries

Swappable Libraries

Applications

Closing Notes

What Websites Have Become (and what they could be)

Overview

Web Applications vs. Desktop Applications

Discovery, Bookmarking, and Package Managers

Sandboxing, Safety, and Malware

Bundling and Binaries

A Proposal for the Web

Ditch Web Browsers!

Web Executables

Rendering

App Discovery

Access Protocol

Sockets

Closing

A Deep Dive into Lambdas

Functional Interfaces

The Low-Level

Bytecode Instructions

Lambda Bodies

Method Handles and java.lang.invoke

invokedynamic

The Lambda Metafactory

End

Method Handles and `java.lang.invoke`

`invokedynamic`