Forem: sheosi

A delightful OS - The kernel

sheosi — Thu, 26 Sep 2024 20:40:33 +0000

Hi, my name is Sergio, and...

I have a dream, I'm here to tell, about a fantastic new OS...

So yup, I'm one of those that will perfectly spend hours on end to make something just so that it is ✨perfect©️✨. And yeah, this has been for years on my head, so it is almost time to at least write somewhere.

So what do we want to build? An OS, for desktop usage, like the one you are using, which is probably either Linux or Android (let's be honest you are reading a guy's rambling about which pieces would make for a better OS, chances are you are a nerd too). Now, how is this OS? as I like to put it delightful💫, an OS that's snappy, secure, private ... Wait maybe it is better if we define what it is:

Performant: The OS must feel snappy, fast as it can be, extracting the juice from your computer, whether it's old with cobwebs or shiny new.
Easy to use: It is a desktop, and we've had enough of OS for the savy, if we want people to move here it's gotta be easy (besides, there's a point that even I find it tiring to seSo arch the internet the commands needed for something).
Solid as metal (the material not the API): It's not funny when your computer won't boot up because itself thought a good idea to update, or because in the middle of the update stupid electrity went off.
Secure and private: My computer, my data, thank you.

Most probably I'll be updating those.

So yup, we know what we want, in a nutshell pretty much what every single of the dozens of user-friendly Linux distributions out there want.

So how are we going to build it?

One option could be to make everything from scratch, but since that effort takes several years of hundreds of engineers to achieve such state and my possible group of engineers ammounts to ... let me count ... one, taking as much done as possible seems like a great idea.

Let's look at the contenders

macOS and Windows

I know, I know, already there, and might not be an option, but they exist, and might be the solution for a lot people, which is fine. I feel like they have their own set of problems, namely tied to a company trying to sell you as much as they can, macOS which was regarded as a holy grail for privacy and non-intrusiveness definetely integrates a lot their own services, and Microsoft ... is trying to put ads everywhere on their OS, plus the whole Recall fiasco ... etc ...

No use then, but let's not forget that macOS puts out an open source of their kernel called Darwin, which could be useful if we didn't had ...

Linux

You knew this was going to happen, Linux is the platform of freedom by default. Most people that are running from macOS and Windows will go into Linux (into one of the thousands available).

You can argue that its bazaar model (in which everyone develops the **ck they want and someone integrates it, which are known as distribution) is broken, that the whole DKMS (third party kernel modules) as a bit of a mess, that Wayland is slow to being develop ...

But it has a lot of good, out of the open source kernels it is the one with the best driver support, it is being tackled by tons of developers (improving it making it better), heck it even has the wonderful, incredible and blazingly fast Rust language in it. There's a lot of interesting things going on there, and it's improving at a nice pace.

FreeBSD and **BSD in general

As I said I don't like leaving stones unturned. So, I'm not just considering Linux but literally everything out there, so what about the BSDs?

The BSDs have in their favour a much more organized model, supposedly more stability, and depending on the flavour better performance on different areas. Focusing on FreeBSD (the one closest to desktop usage), we have some nice things there: people speak of a faster OS, faster network IO, less "dissonance" when using their tools, more stability, compatibility with Linux binaries... also there's ZFS, the end all of filesystems, I'm not going to write about it's coolness here, you can duckduckgo it (am I using duckduckgo instead of google as a verb? yes, we need to raise awereness for other engines), but trust me, IT. IS. COOL.

It's pretty good, however, while the benchmarks do support some of those claims, for me FreeBSD is like another-linux, and has a lot of missing drivers (which is already a problem with Linux, and I can say that confidently, since I am writing this from my Surface device) and the Linux compatiblity layer is yet another layer of things that can (and will) go wrong when people execute their programs, and I'm not saying this out of some theoric possibility, I've seen some reports of people failling to execute programs on more complex scenarios (like Steam or Proton), scenarios which are super useful today.

Regarding ZFS we have other options like btrfs that although not as great, exist and are an option, and we also have openZFS, the Linux version of ZFS (AFAIK not as good, but again an option).

Some might point out the license (GPL vs BSD), and how they consider one to be more free than the other. This is a gasoline topic (one that can sause flames) and I'm not ready for any war, but I'm going to not take it into account, for the actual user there's little difference on those, what's more is that GPL has made some companies to actually make open source software, with great results (like AMD, whose open source driver has better performance and works overall better than it's closed source conterpart), while unfortunately, greed shows no limits, and companies like Sony or Apple have taken BSD software and returned AFAIK, nothing to those code bases.

Finally, I'm ruling out other BSDs, since they aren't targeted for the desktop.

So FreeBSD is not a bad option, but definetely would fragment more the space and/or need much more work, and we are willing to put in work...

Microkernels

As I said no stones unturned. My feeling is that you are proposing people to change something either what they have is broken and doesn't satisfy them of the new thing must have a good compelling reason.

And microkernels have this reason.

For those who dont know what microkernels are. All the other OS are what we call monolithic (or mostly monolithic) and they embed every driver (and other modules) inside of the kernel program itself, even if those modules are loaded per users request they still loaded into the kernel itself, a single bad line in any of them, can make your whole computer say "thanks, enough for today" and ... crash.

Microkernels fix this by making all of those modules regular programs, which present several advantages:

A crash is a minor problem: A crash will only bring down that part of the computer not anything else. Plus it can be recovered by realoding the program itself.
No difference in between os-provided drivers and third party ones: Since they are all programs and use the same interface, there's not gonnna be any if some third-party offers a kernel module for their driver, it's gonna work easily since the interfaces are stablished.
Live updates: Since everything is a normal user space program. Updates could be done live, no more rebooting. The only thing that would require a reboot would be the microkernel itself, which is pretty small (some ammount to 3k lines), so it is reasonable to think that the updates here after a couple of versions would rare, besides live patching could be possible, Linux does it, and I argue that for such a small program it would be even easier.

Which ones to focus on then? There are two names out there that i find most interesting nowadays: RedoxOS (written in Rust, great momentum, interesting concepts, some good projects beneath it) and SculptOS (ability to use multiple, different kernels, interesting sandboxing system ...), I still have to try them out before I can say which one I find more promising as far as desktop goes, but those two are the best candidates I know.

Again, just for the completion there are others like HelenOS, but either they aren't desktop-focused at all or they are too far behind to be even considered.

Is there anything else?

Yeah, there's a lot out there (MenuetOS, HaikuOS, ReactOS, ...) and they are widly interesting, if you are curious, do take a look at them. But for me they fall under the same category as FreeBSD, they would require a ton of work to make them consumer-grade desktop friendly.

The only other concept that's interesting is the one of the distributed OSs, where a bunch of machines are linked together and they can use each other's resources, imagine taking a picture from your computer using your phone's camera, or accessing their files transparently. This super interesting, but the level that we are interested in can be implemented in standard OSs, the only thing that we wouldn't be able to replicate would be things like RAM sharing and programs that work on both devices at the same time, things which are hardly useful since they require connection systems unavailable to consumer-grade devices.

TL;DR

Linux is the one closest to being able to be used, with newer microkernels (Redox and Sculpt) showing promise.

Now, there's still a ton of other questions to be asked on our marvelous OS but for the time being we know how we are going to base it.

Don't agree with something? Would love hear your opinion on a comment. Have a great day!

Making a game (engine) with ECS and in OCaml

sheosi — Sun, 11 Jul 2021 23:06:40 +0000

Hi!👋🏻️ My name's Sergio, and I had kind of a dream: how would a game engine look if it were made in a functional style? 🤔️ Also, every project implementing a framework needs something to do, so I went with the game style of Megaman Battle Network.

Language

The language I wanted to use was OCaml, the ⭐️ star of the show in the ML family. I know F# exists (and have used it too), but wanted to try OCaml in itself, functors, object model and their venerable compiler.

Unfortunately, that is hard 😥️... Entering OCaml means you need to understand:

OPAM: 📦️ The package manager
Dune: 🔨️ The build system
OCaml: 📋️ The language itself (which is unique and somewhat quirky)
Standard library: 📚️ The included one is pretty basic, and according to some, aging. The problem is that there are three alternatives out there and they are incompatible with each other.

As you see OCaml has opted to have a tool for each need. And while each of those tools are 👌🏼️ top-notch as they are fast and will cover almost any case you can imagine, they also do very little in terms of integrations and do require some setup.

Truth being told, saying this is just unfair. This is all true, but also is true that the community is making great ammends in fixing OCaml holes. Which brings me to...

One program to rule them all

After some tries of using OCaml and having little succes I found drom. You see, drom is to OCaml what cargo is to Rust, that is an unified interface for the whole set of tools. Install it with OPAM:

opam install drom

You can make a new program with just:

drom new my_program --skeleton program

That command will also tell which files to edit.

🙋‍♂️️ Hey, I've changed some of those configuration files, how I make the project react to it? Easy, in the root of your project do:

drom project

And to see you program in action 🏃‍♀️️:

drom run

Which will build it and run it.

Not everything are roses though, there some thorns as documentation is scarce and some topics like loading a debugger or profiling are not covered yet. But it is a marked improvement over having to learn opam, opam's switches, dune, and using all of them. Also, it feels more integrated and less intrusive than esy.

The game

As I said, the test project I made was a Megaman Battle Network. Though made is a little big as there's little more than 4 sprites, but already gives an impression of how building the project would feel.

For those who don't know what Megaman Battle Network is, let met tell you that is an action RPG (since there aren't traditional turns in it) where you control what could be called a blue Alexa deleting viruses and other badies that want to take over the internet. For what I've made is enogh to say that battles are played in a grid. Here, let me show you:

The engine

As I wanted to make this from scratch I wanted to keep it modern and use an ECS. ECS stands for Entity, Component, System and is all the rage these days. The premise is that instead of following the tradition of making an object which contains a Sprite, it's position, it's health and other parameters, all the data of one kind is held together (there's a table with all the positions from all objects) this brings huge performance wins for a large quantity of objects (100 times better cache usage and possible usage of SIMD) and improves extensibility.

As for the technology underneath I used Raylib. Raylib is a library with pretty much everything you need to make a game, think of Allegro, SFML or SDL, but:

is available on anything you can think of
it covers everything, yeah, even 3D, fonts, networking ...
is dead simple to use, and have really good documentation
there are bindings for any language in existance, really.

The code

As you know ECS is divided into Entities, Components and Systems.
Entities are here nothing more than an ID, a key that we will use to access the hash tables where all the data broken into components is stored, then the systems access any number of components they might need.

Components

To get a better example: to render a sprite we need a position and a sprite to render:

module Position = struct
  type s = {
    x: int;
    y: int
  }

  include(val (Component.create ()): Component.Sig with type t = s)
end;;

module Sprite = struct
  type s = Raylib.Texture2D.t' Raylib.ctyp

  let load path = (
    let tex = Raylib.load_texture path in
    Gc.finalise Raylib.unload_texture tex;
    tex
  )

  include(val (Component.create ()): Component.Sig with type t = s)
end;;

As you can see, we include a base module here (using first-class modules) which actually does much of the heavy lifting. Altough the include gets a bit convoluted it means that the same module is a component now (it even responds to Component.Sig the interface to components). Here, you can see the functions that our new modules have:

  module type Sig = sig
    type t

    val set: t -> key -> key
    val s: t -> key -> key
    val fold: (key -> t -> 'a -> 'a) -> 'a -> 'a
    val iter: (key -> t -> unit) -> unit
    val get: key -> t
    val get_opt: key -> t option
    val remove: key-> unit
  end;;

Systems

In order to render the sprite we need to get info from both of those components:

module Rendering = struct
  let on_process _key (pos: Position.t) spr =
    Raylib.draw_texture spr pos.x pos.y Raylib.Color.white

  include (val (System2R.create on_process Position.iter Sprite.get_opt))
end;;

Again, I'm using first-class modules to fill in all the boilerplate and make us fill like we are working with just one instance (easier to think about too 😚️).

What happens if some entity lacks either Position or Sprite that entity will be ignored and nothing will be rendered for it.

Actually, my Component system is quite basic, and I'm ignoring things like hierachies, incredibily useful for position where something will always be at an offset of something else (like a sword a character is carraying and swinging around).

Input

An special case is input. Every game has input and is the main way through which the game state changes, it also shouldn't be directly tied to keys (so that it can either be ported easily or be remapped by the player in case they want to).

Here I borrowed from FRP frameworks like Elm:

module Commands = struct
type command =
  | GoUp
  | GoDown
  | GoLeft
  | GoRight
  | Attack
end;;

module Input = InputFrom(Commands)

The input componnent, which holds data and registers an entity as accepting input.

module InputHandling = struct  
  let on_input cmd (pos: Position.t) = 
    let x_size = 128 in
    let y_size = 60 in
    match cmd with
    | Commands.GoDown -> {pos with y=(Stdlib.min (pos.y + y_size) ((2 * y_size)))}
    | Commands.GoUp -> {pos with y=(Stdlib.max (pos.y - y_size) 0)}
    | Commands.GoLeft -> {pos with x=(Stdlib.max (pos.x - x_size) 32)}
    | Commands.GoRight -> {pos with x=(Stdlib.min (pos.x +  x_size) ((2 * x_size)+32)) }
    | Attack -> let key = Entity.by_name "Enemy" in Health.set ( (Health.get key) - 1 ) key|> ignore; pos

  let on_process key pos inp =
    Array.iter (
      fun (kb_key, cmd) -> 
        if Raylib.is_key_pressed kb_key then 
          Position.set (on_input cmd pos) key |> ignore 
        else ()
    )
    inp

  include (val (System2R.create on_process Position.iter Input.get_opt ))
end;;

The input system, which does all the processing, some sugar could be added to make it more amenable (maybe putting each command in a function and making on_process dissapear), but that could be added in the future. When we make an entity with input we get to link each key to each action:

    Entity.by_name "Player"
    |> Position.s {x=32;y=0}
    |> Sprite.s (Sprite.load "assets/player.png")
    |> Health.s 100
    |> Script.s (can_die)
    |> Input.s [|
      (Raylib.Key.Up, Commands.GoUp);
      (Raylib.Key.Down, Commands.GoDown);
      (Raylib.Key.Left, Commands.GoLeft);
      (Raylib.Key.Right, Commands.GoRight);
      (Raylib.Key.Space, Commands.Attack);
    |] |> ignore;

See that the argument to Input.s is an array of tuples with a key and what command does it execute. About the rest of the syntax we'll talk more about it in a minute.

So now, we can assign each entity different keys and we can filter entities (in the InputHandling system we could say the we needed another component which registered this entity as a player, for example).

Entities

Thanks to the fact that now entities are nothing but an ID (a number in my implementation) and everything is in the component, nothing restrains me of:

a) Using OCaml's power to make a mini DSL
b) Adding any characteristics to any entity in a super simple way:

    Entity.by_name "Enemy"
    |> Position.s {x=32+384;y=0}
    |> Sprite.s (Sprite.load "assets/enemy.png")
    |> Health.s 10
    |> Enemy.s ()
    |> Script.s (can_die)
    |> ignore;

See? I've easily marked the entity as:

Having a position
Rendering a sprite
Having health
Being an enemy
Being able to die

Now, while everything from there is easy to understand if you know that Component.s is nothing more than a shortcut to set and that it returns the ID to be used again later (that's why we need the ignore) the Script part may be weird to you.

And yes, ECS has an incredible limitation, it is as verbose as it can get. A simple game with both imperative old-style direct programming and OOP-like game (with something like nodes) gets to have an order of magnitude less of lines than it's full-ECS counterpart (maybe of about 200 to 400, and without taking into account all the engine behind the scenes), when a lot of what you want in the game is just a one time behavior (like the UI which there's only one of which, maybe an attack, lots of small things ...) ECS is just plain overkill, you need either another system or a escape hatch.

My scape hatch is Script a Component which will trigger a function each frame (most probably it can improved and filtered for performance) but for the time being it proved a great addition (which was even better thanks to OCaml partial application).

You can find the whole code here https://github.com/sheosi/Ocaml2D/tree/first_iteration

My impressions

This was a great project, it gave me some interesting impressions on all technologies.

OCaml

In one word.

Quirky*

Don't get me wrong, it has some really strong points to it: marlin (the autocompletion engine) is a beast, it is fast as hell, the compiler (and the build system) is so 🔥️ blazing fast that it felt more as if I was working with an scripting language 🐍️, this makes iterating a joy.

But as I hinted not so subtly earlier, the integration is ... subpar. I have only two files, everytime I modified engine.ml the whole project needed to be recompiled so that the new definitions were available to marlin for main.ml, not just that but at first it seems that drom kinda mixed versions and got a nasty error saying that I was using OCaml "4.10" while the version the tools expected was "4.11", I had to clear all caches and packages storages so and upgrade to "4.11" in the project files (just upgrading the version in the files wasn't enough). This is isn't my first time dabbling with OCaml either, and everytime I come seems like something new pops up (either that I have forgotten the kinda complicated set of instructions that I need to do).

And finally, we have the language itself, using VS Code and OCaml's plugin means having something that tells me which types have been inferred whic is 👌🏼️ just the best, but OCaml's fixation in not making it dependent on indentation means that the syntax gets weird at times (after all of this I still don't know 100% where to use ;; and sometimes it seems it is not needed, though that is good, actually).

My OCaml tips

So, you want to give OCaml a try? First, let me remind you of F#, which is pretty close to OCaml, is based on .NET and you can mix it with C# projects, which means you have a ton of libraries out there, you have a ton of documentation available (most of it with C#, but with some expertise you'll be able to understand both and translate) and you'll have a better time convincing your boss to use it on work.

What's worse is that one of the most compelling reasons up until now to learn OCaml, Reason, has now transformed into ReScript and most definetely they see like they want to get rid of OCaml from it. That sucks.

But this language is still something to see by yourself, and learning functors made my head go...

If you still feel like giving OCaml a try then here are some resources and tips:

OCaml guide: https://dev.realworldocaml.org/ -> 🎊️ Hands-down best resource to learn OCaml structures
IDE: VSCode + OCaml Platform -> There used to be several plugins (including the one from Reason) but nowadays the best is this one, made by the OCaml Platform itself (a group making OCaml more amenable)
Standard library: Batteries or Jane Street's Core: You know that OCaml's standard library is a bit... 😑️ lacking, and old, so there are several attempts to replace it. On the one hand you have the battle-proven Core by the folks on Jane Street, a big brand using exclusively OCaml but is incompatible with OCaml's standard library and the other hand you have Batteries a community effort pretending to extend OCaml's standard library.

ECS

Finally, I understand what I've read about using ECS. While it feels awesome defining an entity and giving it any property I feel like, trying to implement the a small label that showed our health (and another for the enemy) meant that I needed to make a Component and a System just for that, at that time I understood that trying to shove everything inside the ECS is going to make your code verbose as hell and you may end with several times the number of lines you might had otherwise.

In my case, I made the Script Component as an escape hatch, and honestly it felt 😀️ good. OCaml's partially applied functions really helped too. Honestly, I can see this design getting some sugar and being used more.

Functional programming

The code here is not super functional, I acknowledge that, yet the part where it is more functional, that is the Input, while still verbose feels pretty good, the Command idea taken from FRP and Elm is a great way of abstracting input hardware.

But I can see shoving everything inside like an object called World and passing that down on all functions for them to do wathever they feel like, something like what the Nu game engine does.

Raylib

If I take something really good from here is Raylib. As I said you can use Raylib pretty much wherever you use SDL, but is really easy, with functions with understandable names, modern and easy-to-parse web with tons and tons of examples. But what does it have? Well, in Raylib you'll find: 3d (with it's own abstraction layer), audio (no 3d audio, but it does support streaming), physics, texts, 2d, GUI and even networking seems to be on the works: https://github.com/raysan5/raylib/issues/753

Basically, is an engine but as a library.

My only gripe is that the 3d part lacks Vulkan, and while raysan5 (the person behind Raylib) is not against it, does not have a lot interest in it.

I would take raylib for a ride if your project needs low-level control.

Final thoughts

TL;DR: ECS is good but tedious and verbose, OCaml is quirky but with people putting some real effort on it, and raylib has chances of becoming my go-to low level game lib.

I might further develop this architecture on the future, most probably not to use it by itself, but as research and to port it to other frameworks later.

Some frameworks that I'm interested in and that I might test are:

Rust + Bevy + maybe? gluon (An ML for Rust)
F# + monogame or raylib
F# + godot
Maybe Swift...?

If you have had experiences on those, let me know in the comment. Also, have you had any experience on functional games and/or ECS?

Links

I based my ECS design on this: https://cranialburnout.blogspot.com/2013/09/database-radiation-hazard.html