Forem: borislav nikolov

tty only

borislav nikolov — Thu, 12 Aug 2021 21:35:45 +0000

I used only the tty (no X installed) for 5 nights. It was relaxing, and now I go back to it every time I am overwhelmed.

I have to seriously re-think the way I spend time on my
computer. Working only on a tty is completely calming experience, there are no ads, no hundreds of tabs open, no pressure, only code and text. I also noticed I read way way way less news, hacker or not.

BTW, browsing was actually way better than I thought, between eww, w3m, lynx, links2 and links2 -g, and sometimes just reading the html dump, I was able to navigate the modern web with reasonable success.

Laptop

I bought used t440p laptop, they go from 100 to 300E. Super sturdy machine, like the opposite of my xps. Very easy to open, and very hackable.

Setup

Install debian and do some basic fixes to make the tty usable.

rate (whoa slow keyboard rate pisses me off)

add 'kbdrate -r 30 -d 0' to /etc/rc.local

ctrl+/ etc

super annoying that ctrl+/ sends Delete and I want to bind it to undo, to do that you have to:

$ showkeys

press ctrl + /

check out the keycode, (in my case it is 53)

$ dumpkeys > keymap

change
    control keycode  53 = Delete
    shift   control keycode  53 = Delete

to
    control keycode  53 = Control_underscore
    shift   control keycode  53 = Meta_underscore

and 51 and 52 to
    shift   control keycode  51 = Control_asciicircum
    shift   control keycode  52 = Meta_asciicircum
    (I use S-C-. and S-C-, for cursor)

add 'loadkeys path/to/keymap' in /etc/rc.local

then the bindings:

(define-key global-map (kbd "C-_") 'undo)
(define-key global-map (kbd "M-_") 'undo)
(define-key global-map (kbd "M-^") 'mc/mark-next-like-this)
(define-key global-map (kbd "C-^") 'mc/mark-previous-like-this)

mouse

install consolation or gpm

cursor

instead of block cursor this will show the tty setup cursor (blinking underscore)

(setq visible-cursor nil)

otherwise emacs has blinking (HZ/5 ~200ms) block cursor which is horrible, so replace it with blinking underscore to take less attention from your eyes. I tried all kinds of ways to stop the blinking but none worked.

brightness

max=$(cat /sys/class/backlight/intel_backlight/max_brightness)
echo -n $max | sudo tee /sys/class/backlight/intel_backlight/brightness

font

install terminus and then add the setfont command to your bash/zshrc

setfont Uni3-Terminus20x10 # 12x6 14 16 22x11 24x12 28x14 32x16

emacs

init.el:

(package-initialize)
(require 'go-mode)
(require 'multiple-cursors)
(require 'tramp)
(load-library "view")
(require 'cc-mode)
(require 'ido)
(require 'compile)

(setq tramp-default-method "ssh")
(setq undo-limit 20000000)
(setq undo-strong-limit 40000000)

(defun delete-word (arg)
  "Delete characters backward until encountering the beginning of a word.
With argument ARG, do this that many times."
  (interactive "p")
  (delete-region (point) (progn (backward-word arg) (point))))

(define-key global-map (kbd "C-h") 'delete-backward-char)
(define-key global-map (kbd "M-C-h") 'backward-kill-word)
(define-key global-map (kbd "C-_") 'undo)
(define-key global-map (kbd "M-_") 'redo)

(define-key global-map (kbd "<M-backspace>") 'delete-word)
(define-key global-map (kbd "C-M-h") 'delete-word)


(define-key global-map (kbd "<f2>") 'compile)
(define-key global-map (kbd "<f1>") 'next-error)
(define-key global-map [C-tab] 'indent-region)
(define-key global-map (kbd "M-^") 'mc/mark-next-like-this)
(define-key global-map (kbd "C-^") 'mc/mark-previous-like-this)

(global-unset-key (kbd "C-t"))

(ido-mode 1)

(menu-bar-mode -1)
(tool-bar-mode -1)
(scroll-bar-mode -1)

(setq inhibit-startup-message t)

(global-linum-mode 0)
(display-time-mode 1)
(global-font-lock-mode -1)
(gpm-mouse-mode -1)
(setq backward-delete-char-untabify nil)

(display-battery-mode 1)
(setq make-backup-files nil)
(setq auto-save-deault nil)

(show-paren-mode 1)
(setq show-paren-delay 0.0)
(setq show-paren-style 'parenthesis)
(transient-mark-mode t)
(fset 'yes-or-no-p 'y-or-n-p)

(defun custom-go-mode-hook ()
  (setq gofmt-command "goimports")
  (add-hook 'before-save-hook 'gofmt-before-save)
  (if (not (string-match "go" compile-command))
      (set (make-local-variable 'compile-command)
           "go generate && go build -v && go test -v && go vet &&  golangci-lint run"))
  (if (not (string-match "go" compile-command))
      (set (make-local-variable 'compile-command)
           "go build -v && go test -v && go vet"))
  (local-set-key (kbd "M-.") 'godef-jump)
  (local-set-key (kbd "M-,") 'pop-tag-mark)
)

(add-hook 'go-mode-hook 'custom-go-mode-hook)

.bashrc:

export VISUAL='emacsclient -ct'
export EDITOR='emacsclient -ct'
alias e='emacsclient -ct'
alias emacs=e
alias vi=e

also start emacs daemon:

~/.config/systemd/user/emacs.service

[Unit]
Description=Emacs text editor
Documentation=info:emacs man:emacs(1) https://gnu.org/software/emacs/

[Service]
Type=forking
ExecStart=/usr/bin/emacs --daemon
ExecStop=/usr/bin/emacsclient --eval "(kill-emacs)"
Environment=SSH_AUTH_SOCK=%t/keyring/ssh
Restart=on-failure


[Install]
WantedBy=default.target

setup zsh with shared history and fzf

lock

vlock -all on pm suspend

email (personal)

I use forgotten (github.com/jackdoe/forgotten) to manage encrypted list of passwords, so just use it with mutt:

.muttrc

source "/usr/bin/forgotten -key gmail -mutt |"
set from = "xyz@example.com"
set realname = "aa bb"
set use_from = yes
set envelope_from = yes
set smtp_url = "smtp://xyz@example.com@smtp.gmail.com:587"
set smtp_pass =$my_pass
set imap_user = "xyz@example.com"
set imap_pass =$my_pass
set folder = "imaps://imap.gmail.com:993"
set spoolfile = "+INBOX"
set ssl_force_tls = yes
bind index G imap-fetch-mail
set editor = "emacsclient -ct"
set charset = "utf-8"
set record = ''
set header_cache = "~/.mutt/cache/headers"
set message_cachedir = "~/.mutt/cache/bodies"

Day 1:

After installing and doing the basic setup it was pretty late, so I didnt do much more.

links2 works nice, but too graphical, its ok to see images though
non ips display is total shit for console; bought new one from amazon, we will see this saturday
replace gpm with consolation

Day 2

patched consolation to support right touchpad click because myh middle button is not very good
fixed the ips display; it is a total gamechanger. the black is so much more black than before; amazing for tty
fix alsa default card

cat > /etc/asound.conf <<EOF
defaults.pcm.card 1
defaults.ctl.card 1
EOF

download some royalty free music and played it with mplayer
battery lasts quite a lot

I wish eww worked better, I hate going out of emacs
I am so happy with the new display.

setup tlp; cap max freq at 1000mhz, I should've gotten i5 instead of i7, this i7 is just hot for no reason. Will disable hyper threading, maybe it will feel better. Or maybe I should open the laptop and clean it. Even with 1 core and no hyperthreading and cap on 1ghz it still gets hot, I guess its cleaning time; will actually check on amazon for i5, shouldnt be very expensive. Though despite the temperature, battery lasts for 6 hours (possibly more)
went back to gpm; having mouse in links2 -g seems better

Day 3

The new screen arrived, took exactly 5 minutes to replace, I love working on machines that are easy to repair.

wow that is a good screen haha, I forgot how much nicer it is
i5 cpus are super expensive; no way I am buying one
removed thermald and started using thinkfan with lower threshold (had to add options thinkpad_acpi experimental=1 fan_control=1 in modprobe.d/thinkfan.conf), so using fan level 1 at 40C is much better because when it gets to 50C it is just hot on my palm
still thinking of what to write
it is incredibly calming having no windows
I did open and clean up the turbine a bit, but wasnt much to clean it is however really nice to have a laptop that is meant to be open

Day 4

made copy paste http service on top of unix socket, so I can copy between emacs and zsh without using ansi-term (https://github.com/jackdoe/pasta)
There is no way to programatically get the current selection from the linux kernel, so I made a patch to add new ioctl to be able to get the selection, so i can use it with M-w and C-y.

# build custom kernel on debian:

cd /usr/src && \
git clone --depth=1 \
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

# change what you want, and then
# copy your config from linux-config-5.xx into the trunk and then:

make -j8 bindep-pkg LOCALVERSION=-xyz

working without X for the whole day, was pretty fun.

This is the getsel ioctl patch

commit 5cf882d8b74747bbc08463d83cf80509c920edca
Author: borislav nikolov <jack@sofialondonmoskva.com>
Date:   Sat Mar 21 23:42:22 2020 +0100

    patch for GETSEL
    add copy_selection_to_user

diff --git a/drivers/tty/vt/selection.c b/drivers/tty/vt/selection.c
index d54a549c5892..9b26dec762dd 100644
--- a/drivers/tty/vt/selection.c
+++ b/drivers/tty/vt/selection.c
@@ -6,6 +6,7 @@
  *                struct tty_struct *)'
  *     'int set_selection_kernel(struct tiocl_selection *, struct tty_struct *)'
  *     'void clear_selection(void)'
+ *     'int copy_selection_to_user(char __user *)'
  *     'int paste_selection(struct tty_struct *)'
  *     'int sel_loadlut(char __user *)'
  *
@@ -71,6 +72,45 @@ sel_pos(int n, bool unicode)
    return inverse_translate(vc_sel.cons, screen_glyph(vc_sel.cons, n), 0);
 }

+/**
+ * copy_selection_to_user      -   get current selection
+ *
+ * Get a copy of current selection, console lock does not have to
+ * be held
+ */
+int copy_selection_to_user(char __user *arg)
+{
+   int get_sel_user_size;
+   int ret;
+
+   if (copy_from_user(&get_sel_user_size,
+              arg,
+              sizeof(vc_sel.buf_len)))
+       return -EFAULT;
+
+   mutex_lock(&vc_sel.lock);
+
+   if (get_sel_user_size < vc_sel.buf_len) {
+
+       mutex_unlock(&vc_sel.lock);
+
+       return -EFAULT;
+   }
+
+   ret = copy_to_user(arg,
+              &vc_sel.buf_len,
+              sizeof(vc_sel.buf_len));
+   if (ret == 0)
+       ret = copy_to_user(arg+sizeof(vc_sel.buf_len),
+                  vc_sel.buffer,
+                  vc_sel.buf_len);
+
+   mutex_unlock(&vc_sel.lock);
+
+   return ret;
+}
+EXPORT_SYMBOL_GPL(copy_selection_to_user);
+
 /**
  * clear_selection     -   remove current selection
  *
diff --git a/drivers/tty/vt/vt.c b/drivers/tty/vt/vt.c
index 309a39197be0..2b7eb55aafa3 100644
--- a/drivers/tty/vt/vt.c
+++ b/drivers/tty/vt/vt.c
@@ -3061,6 +3061,9 @@ int tioclinux(struct tty_struct *tty, unsigned long arg)
        case TIOCL_PASTESEL:
            ret = paste_selection(tty);
            break;
+       case TIOCL_GETSEL:
+           ret = copy_selection_to_user(p+1);
+           break;
        case TIOCL_UNBLANKSCREEN:
            console_lock();
            unblank_screen();
diff --git a/include/linux/selection.h b/include/linux/selection.h
index 5b890ef5b59f..7cb971795013 100644
--- a/include/linux/selection.h
+++ b/include/linux/selection.h
@@ -15,6 +15,7 @@ struct tty_struct;
 struct vc_data;

 extern void clear_selection(void);
+extern int copy_selection_to_user(char __user *arg);
 extern int set_selection_user(const struct tiocl_selection __user *sel,
                  struct tty_struct *tty);
 extern int set_selection_kernel(struct tiocl_selection *v,
diff --git a/include/uapi/linux/tiocl.h b/include/uapi/linux/tiocl.h
index b32acc229024..055ebda041d4 100644
--- a/include/uapi/linux/tiocl.h
+++ b/include/uapi/linux/tiocl.h
@@ -20,6 +20,7 @@ struct tiocl_selection {
 };

 #define TIOCL_PASTESEL 3   /* paste previous selection */
+#define TIOCL_GETSEL   18  /* get current selection */
 #define TIOCL_UNBLANKSCREEN    4   /* unblank screen */

 #define TIOCL_SELLOADLUT   5

to use the patch you need something like:

#include <stdio.h>
#include <sys/ioctl.h>
#include <linux/tiocl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <strings.h>


struct getsel {
  char code;
  int size;
  char data[0];
} __attribute((__packed__));

struct getsel * get_selection(int size) {
  struct getsel *d = (struct getsel *) malloc(size + sizeof(struct getsel));
  if (d == NULL) {
    perror("malloc");
    exit(1);
  }

  bzero(d, size + sizeof(struct getsel));

  d->code = 18; // TIOCL_GETSEL
  d->size = size;

  int fd = open("/dev/tty",O_RDWR);
  if (ioctl(fd, TIOCLINUX, d) < 0) {
    perror("paste: TIOCLINUX");
    exit(1);
  }
  close(fd);
  return d;
}

int main(void) {
  int size = 200;
  struct getsel *d = get_selection(size);

  printf("size: %d\n",d->size);
  for (int i = 0; i < size; i++) {
    if (d->data[i]) {
      printf("data[%d] = %d\n",i, d->data[i]);
    }
  }
  d->data[d->size-1] = '\0';
  printf("string: %s\n", d->data);
  free(d);
}

anyway, there are still issues to be solved, for example when I press ctrl+y while cat is open I cant past inside, because cat hoards the input.. I should patch the usb driver to take it before cat..

moved man to be within emacs

in .zshrc:
man() {
  emacsclient -ct -e '(man "'$1'")'
}

still need to get man9

Day 5

fucking cursor blinking is annoying the hell out of me
I tried all kinds of tricks to disable it from tty and emacs
it is always fucking blinking.

(setq visible-cursor nil)

and patch it in the kernel!

commit afda0f8175fe560d86e1f2ec0b33a9f25b3bf13f
Author: borislav nikolov <jack@sofialondonmoskva.com>
Date:   Wed Apr 1 09:13:55 2020 +0200

    fuck blinking underline

diff --git a/drivers/tty/vt/vt.c b/drivers/tty/vt/vt.c
index 2b7eb55aafa3..b8a7478c9f98 100644
--- a/drivers/tty/vt/vt.c
+++ b/drivers/tty/vt/vt.c
@@ -2306,13 +2306,6 @@ static void do_con_trol(struct tty_struct *tty, struct vc_data *vc, int c)
                set_mode(vc, 0);
            return;
        case 'c':
-           if (vc->vc_priv == EPdec) {
-               if (vc->vc_par[0])
-                   vc->vc_cursor_type = vc->vc_par[0] | (vc->vc_par[1] << 8) | (vc->vc_par[2] << 16);
-               else
-                   vc->vc_cursor_type = cur_default;
-               return;
-           }
            break;
        case 'm':
            if (vc->vc_priv == EPdec) {
diff --git a/drivers/video/fbdev/core/fbcon.c b/drivers/video/fbdev/core/fbcon.c
index bb6ae995c2e5..721f326b01e6 100644
--- a/drivers/video/fbdev/core/fbcon.c
+++ b/drivers/video/fbdev/core/fbcon.c
@@ -173,7 +173,7 @@ static const struct consw fb_con;

 static int fbcon_set_origin(struct vc_data *);

-static int fbcon_cursor_noblink;
+static int fbcon_cursor_noblink = 1;

 #define divides(a, b)  ((!(a) || (b)%(a)) ? 0 : 1)

@@ -3527,13 +3527,8 @@ static ssize_t store_cursor_blink(struct device *device,

    blink = simple_strtoul(buf, last, 0);

-   if (blink) {
-       fbcon_cursor_noblink = 0;
-       fbcon_add_cursor_timer(info);
-   } else {
-       fbcon_cursor_noblink = 1;
-       fbcon_del_cursor_timer(info);
-   }
+   fbcon_cursor_noblink = 1;
+   fbcon_del_cursor_timer(info);

 err:
    console_unlock();
diff --git a/include/linux/console_struct.h b/include/linux/console_struct.h
index 24d4c16e3ae0..b21061f8aad7 100644
--- a/include/linux/console_struct.h
+++ b/include/linux/console_struct.h
@@ -166,7 +166,7 @@ extern void vc_SAK(struct work_struct *work);
 #define CUR_HWMASK 0x0f
 #define CUR_SWMASK 0xfff0

-#define CUR_DEFAULT CUR_UNDERLINE
+#define CUR_DEFAULT CUR_BLOCK

 bool con_is_visible(const struct vc_data *vc);

applying it..

Whoaaa, this is so beautiful.. just a block █, absolutely amazing.

I did not expect the blinking to put so much mental pressure, having █ just sitting not doing anything, just telling you where you are, its like meditation for the eyes.

Conclusion:

I have to seriously re-think the way I spend time on my other
computer. Working only on a tty is completely calming experience, there are no ads, no pressure, only code and text.

I will keep working on it to improve the tty experience, and I will actively work on reducing my dependency on the modern web, e.g. use go doc more than google, build local search indexes etc. I hope to will some of it to my daily work life.

PS:

Those kernel patches are just for fun, don't take them seriously.

Outages and Good Vibes

borislav nikolov — Wed, 11 Aug 2021 15:30:07 +0000

Recently I made a post of how we moved from elixir to go because we needed more dev power, and elixir's strengths were not panacea to us (which made the community very upset), which was something I hate, but now I would like to talk now about something I love.

Outages.

I love outages.

When pagerduty calls, or someone posts 'hey something is wrong' on #tech-escalation, and my adrenaline starts pumping, I get so alive!

This is the list of things we do when outage starts.

1. Make a hangouts call

Before anyone knows what is going on, just make a call. Don't discuss on slack, don't wait. Make a call and ask for help, even if it is false alarm.

2. Pick a leader

It is super important to say who leads, it is almost always the person with most context, but sometimes they are busy doing reconnaissance, in that case it is usually me, or I pick one who I know can lead.

The call leader will resolve any deadlocks, make sure people are not duplicating work, as every second is essential, and also find support resources if needed.

3. Divide and Conquer

The leader starts assigning tasks, one person needs to identify the surface area of the bleeding, one has to preliminary rollback any service that is even remotely related, one has to start investigating the impact and communicate with Customer Success.

Always rollback first, think later. This is also why it is very important to have rollouts and rollbacks reasonably fast. The faster you can roll out, the faster you can roll back when you see something is wrong.

The leader also must call other people if needed, and also notify Ronen (our CEO) about the status.

4. Cleanup

After the issue is fixed, dedicate a small team to work with CS to do damage control, talk with the affected users and do as much as possible to mitigate the damage.

5. RFO

Write a small document identifying what happened and what we learned, and what we have to work on in order to be better.

Talk especially about:

a) Alerting and Monitoring

Did we find out about the issue ourselves or customer had to tell us? It is at upmost importance to be able to find about issues before they impact users, sometimes this means we missed something in our end-to-end test and it has to be fixed asap.

b) Rough timeline

We must write a rough timeline so we can investigate if we can speedup the process somehow, for example, how much time did it take between the first error message and the creation of the hangouts call, how long did it take to select the tasks and etc.

c) What was the impact

Just rough estimate about affected users

d) What was the fix, and how can we avoid having this class of issues.

Can we fix this with better linting? Can we tweak our process a bit?

The best part about an outage, is that it makes me be part of a team. Of course working with my team every day is also nice, but its the difference of camping with friends and camping with friends while being attacked by a grizzly bear. Outages are just exciting. The atmosphere so nice, how everyone has everyone's back. There is zero blame. Everybody trying their best to help.

Now during covid, I think outages is indispensable in bringing our remote team together.

Good vibes!

PS: pls if you are the kind of person who is looking for someone to point to when shit hits the fan, don't apply.

the voxel syndrome

borislav nikolov — Wed, 28 Jul 2021 21:46:09 +0000

For the first time in two years I met a man, and I felt like I am talking to a fellow man. A man that is not my self.

When two psyches interact, the poet said, the consequences are infinite, as each psyche is total.

And here we are, talking pictures, ghosts as Derrida would say, talking to each other, day after day, discussing the new feature we will ship.

But when two pictures talk, so much is lost.

This will be studied for decades to come, the ghosts talking phenomenon. I wonder what name they would give it, I kind of hope its 'the voxel syndrome' or something cool.

This is not a post about remote work or office work. It is a post about us, about my team, and my company.

The engineering community has pushed the 'I am more effective from home' thing for a while, and I absolutely support it, I am absolutely more effective from home. No distractions, no commute, no drama, no 20$ for coffee per day. I am confident we can deliver way more features like that, but it feels like I am alone in a multi player game.

I don't know if you ever had this experience, everything is there, and the world is working, you can do quests, and slay dragons, but its just not fun. I think there are people who enjoy that kind of work, just as there are people who enjoy playing solo games, and people who don't.

We are building a 0 to 1 product, it is difficult as fuck, its not "ship a bunch of features" thing or "move those tasks from in-progress to complete". Fuck the tasks. New ground we have to conquer.

People say "managers want people to go to the office because they want to control" and I say to those people, fuck you. Why are you working somewhere where the managers control you? Quit and stop bitching.

We need to cook together, we need to eat together, to share ideas and ship.

I hope we can go back to the office for 1 day a week, and I promise you, this day we will have a feast every week! And we will eat like Monkey D. Luffy after his adventures.

-b

Mutation is life / Boring Technology

borislav nikolov — Mon, 26 Jul 2021 14:12:11 +0000

TLDR:
We had horrible outage where rabbitmq node ran out of memory because I
forgot to unbind a queue (one of 50s or so), and we lost a whole bunch
of in-producer-memory state, this made our whole infrastructure
completely probabilistic and it took us 2 hours of blood sweat and
tears to recover the lost data. We should've just used postgres as a
queue.

In almost any system being built today we have users that perform
actions to mutate state. This is life, to create side effects, yes,
mutation is life.

I will illustrate it like this:

[ user(U) ]<------+
    |             |
    | action(A)   |
    v             |
[ receiver (R)]   ^ world
    |             | change
    |             |
    v             |
[ state (S) ] ----+

receiver:
   this is usually a backend endpoint

user:
   in our case is a chef

state:
   in our case: creating an order

You can move those pieces in any way, for example the action can be a
function of the state mutation, instead of the other way around, or
the receiver of the action can be the user itself and the user
directly mutates the state, etc. This however is what technical
implementation means, looks the same to the outside, but has very
different emergent properties.This is what this post is about,
emergent behavior and chaos.

Lets a possible technical implementation of 'user' creating an 'order'

user sends action to an endpoint /create-order
backend code pushes to 'order.new'
a) consumer of 'order.new'
- order is picked up transformed a bit and written to a database
- another message is sent to 'order.created' queue
a) consumer of 'order.created'
- create audit log of who/when/what
b) consumer 'order.created'
- send email to the interested parties
c) consumer 'order.created'
- builds an email and sends it to 'email.created'
- logs the email for archiving purposes
d) consumer 'order.created'
- create chat message with the order
- push to 'message.created'
e) consumer 'order.created'
- extract features for DS
- copy to salesforce etc
a) consumer of 'message.created'
- send push notification
a) consumer of 'email.created'
- sends the email and then pushes to a queue email.sent
a) consumer of 'email.sent'
- marks the email as sent

This is pretty much what we have now (maybe a bit more complicated but
not much), you can trivially add more reactive components listening to
specific topics, and you can fanout and etc just with rabbitmq.

It is super flexible and extendible, re-triable etc.

Of course it was not designed like that, but it grew over the years,
adding bits and pieces here and there, it is very easy to
unconsciously complicate it.

Everything was really good until I did some refactor and stopped
consuming one of the topics that was a clone of 'order.created', and I
forgot to unbind the queue, so it kept getting messages but nobody was
draining it, and a the RMQ node ran out of memory. Only 1 out of 3,
because we still use elixir for that process, we were relaying on
elixir's in-memory stability to keep a buffer of messages to resend if
need, which of course I killed when I restarted the cluster because I
wasn't sure what the fuck is going on.

That meant that 30% of all requests went to the abyss, the true abyss.
We had to stay to 5am to glue bits and pieces and to connect the state.

Caused the worse outage I have ever been firefighting, and once I was
involved in solving an outage that we were selling hotels for 1/100th
of the price, losing millions of euros.

Now lets discuss another implementation of the same thing:
'user' creating an 'order':

user sends action to an endpoint /create-order
backend code:

  begin
  insert the order
  insert the message (push_notification_sent_at = null)
  insert the email (sent_at = null, delivered_at = null)
  insert log
  commit

a) secondly cronjob

  select (for update) messages where push_notification_sent_at is null
  send the push notification
  update the message

b) secondly cronjob

  select (for update) email where sent_at is null
  send the email
  update the email table

Use the database as a queue, works totally fine, we can scale
postgress vertically forever. Why forever you ask? Because there are
~30000 restaurants in London, and we can geo-shard it, and there is
physical upper bound on amount of data in a region.

Fucking queues, it is so easy to overuse them, without knowing it
creeps up on you, and in the end you have infrastructure spaghetti.

Anyway We are migrating from queues to transactions and fuck it, I cant keep
it in my head, ultimately it all ends up in postgres anyway, just with
extra steps.

Fuck.

The morale of the story is:
If shit ends up in postgres anyway, and you can afford to directly
write to it (which is not always the case), just write to it.

Do boring technologoy, the way we wrote php3 shit 20 years ago, get
the state and write it in the database, even though mysql didn't have
transactions (it was helpfully accepting to BEGIN/COMMIT though haha),
it was ok.

PS:
There were 2 missed deliveries, but CS handled them like a
king. Sending uber to pick up the things form the supplier and sending
it to the chef and etc. It is much easier to firefight when you know
CustomerSuccess has your back.

PPS: I think outages are the best, everyone groups up and we solve the
problem, some panic, some adrenalin, some pressure, but in the end the
whole company becomes more of a team.

Work in the kitchen.

borislav nikolov — Wed, 21 Jul 2021 13:22:00 +0000

TLDR:
Use your product, and talk to your users, as a developer there is no
better way to work on the things that matter. Almost every other path
leads to institutional imperative[1] and deepest technical debt that
is almost impossible to pay.

There is no better way to code than to work in the kitchen.

Hi,

I joined REKKI 8-9 months ago, REKKI is an ordering app that looks
like a chat app, that chefs use to order from their suppliers (kind of
like inverse deliveroo). 'Whatsapp for chefs' is how Ronen described
it when we met (best fucking CEO I have ever seen, it is pointless to
talk here about his vision and etc., but if you get a chance to meet
him, don't miss it). My mom has a small grocery shop in Sofia, and I
know how much pain is for her to order from her suppliers, and also I
believe that the whole producer->supplier->restaurant->consumer market
is completely non transparent and everything can be improved.

So as I said, I joined 8-9 months ago, and then I really did not
understand what is the actual difference between whatsapp and REKKI,
so I asked Ronen if he can set me up to work in a restaurant for 2
weeks. "2 weeks? are you crazy, who will write code?" he said, so he
gave me 2 days.

My first 10 hour shift was few days after that.

My expectations were fairly low, besides me being in complete panic
and outside of my comfort zone, I thought I will wash the dishes and
observe the dynamics.

I wanted to not be 'minus' to the staff, so I pre-trained myself how
to use the industrial dishwashers watching youtube videos[2]. BTW
those things have the best UX ever, the colors they use to convey
information, the way everything works is just incredible.

PANIC! The day has arrived! I calmed myself down by watching more
dish-washing videos from the metro on my way to Restaurant X. I was
there at 14:00 (as agreed).

The restaurant is fairly big, and the kitchen is in the middle, so
everyone can see you cooking, and the dishwasher was in the back in
smaller kitchen. You can't imagine my smile when it was one of the
dishwashers I have pre-trained for.

The staff was very welcoming, and when I told them I trained for the
dishwasher they were happy for me to wash the dishes while they do
some veg-prep (that's kitchen slang for "vegetable preparation" like
chopping potatoes haha). Shortly after that they gave me a knife and
some carrots to peel,I am fairly good with the knife (I do
leatherworking[3] and woodworking as a hobby), but in no way I am
comparable with a chef, for them knifes are more like body parts than
anything else, still I didn't do bad, and when they saw I didn't draw
any blood they gave me more and more veg-prep work.

The restaurant opens at 18:00, and the guests arrive around 18:30, we
ate around 18:00 and then the guests started coming, I was in kitchen
(wearing apron and everything, also I have just as much tattoos as the
chefs, so on the outside there was no question if I am a chef or not).

Because of my veg-prep and dish-washing performance(I suspect) they
gave me actual dish to cook: mushroom ravioli.
I think it involved those things:

long mushroom stems (the ones I cut earlier)
some other mushrooms
truffle butter
chicken broth one of the tricks I learned, everything tastes better if you use chicken broth instead of water
some sauce I forgot how it is called
stirring until mushrooms get golden
some olive oil in the end

BTW, you can also check out my soup recipe
https://txt.black/~jack/soup.txt if you are interested in my cooking
abilities.

Turns out all people order food in the same time, I am telling you I
have been dealing with distributed system consensus for the last 6-7
years, and I never thought unorganized entities can act in this way
without behavior emerging from simple rule (like ducks following), its
CHAOS, I have to do so much mushroom ravioli.. PANIC...You know how
all chefs stir the pan by tossing the food in the air? Well I have to
use a spoon to stir; also all the guests can see me using a spoon..
Anyway I did ok, nobody returned their food! At 22:00 It is the first
time I can step outside for a breath of fresh air for 5 minutes.

Then things get easier, from 22:00 to 00:00 guests only drink, and we
started cleaning the kitchen around 23:30, my last metro leaves at
00:15, so if I don't leave at 00:00 I won't catch it, and of course
there are guests that won't leave, despite restaurant closing.

My legs hurt, from 14:00 to 00:00 nonstop, bloody hell this is some
brutal work. I couldn't take notes, or do anything, just chop chop
chop, stir stir stir.

BTW If the people of RestaurantX happen to read this post, thank you
so much for letting me cook! You rock!

Few weeks after that I got my second shift, in a small michelin star
restaurant. Things went in quite similar way, but I was more
confident, I did some dishes, a lot of veg-prep, which they thought
was boring, but I kind of liked it, there is not that many times in my
life when I know exactly what to do.

Again from 14:00 to 00:00, and again people did not leave when
restaurant is closing, fuck people just leave! Chefs are tired!

I will do a small divergence here to tell you about the tech stack
we had when I joined:

Elixir was chosen, because whatsapp uses erlang, and we need
presence and "who is typing" feature as any serious chat app.

In order to build a chat app, elixir makes a lot of sense, hot code
reloading can keep the sockets alive during deploys, you have native
presence with phoenix channels, etc. also super easy to build
channels and joining and leaving and everything.

So I came back to work, and let me tell you REKKI is not a Whatsapp
for chefs app. It is an ordering app that looks like a chat, but there
is no way that two chefs are in the app in the same time, if one is
ordering from suppliers, the others are cooking, or veg-prepping or
anything but not ordering.
Oh and I forgot to mention: the internet in the kitchen is total shit,
constantly switching wifi and 3g, and in some parts you have neither.

Phoenix channels(abstraction on top of websocket) are full duplex, but
serial, which means you basically can't block on your endpoints
because no new requests can come through the channel, so the payload
was sent to rabbitmq and then something else executed it. This lead to
incredibly interconnected and complex (especially with retries) system
in order to write a record in a database, but it was a requirement if
we want "someone is typing", except we don't!
And guess how good the websockets perform with the kitchen's internet.
Heartbeats, reconnects, flushing queues, pending actions etc etc, all
had to be taken care of.
This is true technical debt, you can only solve by doubling down on
the problem you already have, and of course, you make it work, and you
improve it, but you can only make incremental improvements.

You see if you work in the kitchen you see what your technology is
doing, and what your users do.

Now we are moving from elixir to go, we moved from channels to
endpoints, pulled up the queue actions in the endpoints so the chef
gets immediate feedback if there is any kind of error with their
request. I have to say I love the app now, how snappy and clean it is.
We moved away from elixir because we can't get devs and also every one
can read and write go (it has like ten reserved words..), and also
empowered frontend devs to become fullstack devs and make their
endpoints. (also it is super annoying to write business logic in a
functional language)

There is no better way to code than to work in the kitchen.

--
Fri 7 Feb 19:38:07 CET 2020
Borislav Nikolov (https://github.com/jackdoe)

References:

[1] https://www.berkshirehathaway.com/letters/1989.html
BERKSHIRE HATHAWAY 1989 shareholders letter

[2] https://www.youtube.com/results?search_query=dishwashing+training

[3] https://txt.black/~jack/jar-to-mug.txt

Interface Dispatch

borislav nikolov — Wed, 21 Jul 2021 13:07:11 +0000

TLDR:

Always measure before you optimize, I introduced super shitty bug to https://github.com/rekki/go-query, because I thought the bottleneck is in the binary search implementation, and gained nothing; calling interface methods in hot for loops adds ~10-15 instructions per call.

Hi,

I want to illustrate why intuition in the field of performance is
often misleading, and just like propaganda, the more you think you you are immune to the illusion, the more effect it has on you.

Let me tell you when I introduced terrible unreproducible and contextual bug in a library we use everywhere.

Cost of simple things

First, we will discuss the cost of simple things, like counting from 1
a million.

package main

import "testing"

var sum = 0

func BenchmarkNoop(b *testing.B) {
        for i := 0; i < b.N; i++ {
                sum++
        }
}

// go test -bench=.
// BenchmarkNoop-8         873429528                1.36 ns/op

the for loop needs to do one comparison and one jump

for i:= 0; i < 1000000; i++ {
}

is something like

0: inc i
1: cmp i, 1000000
2: jlt 10
3: ... code
4: jmp 0
10: .. rest of program

you can also think about it implemented with goto:

i = 0
check:
if i > 1000000
   goto done

for loop code
...
goto check

done:
rest of program

(it is also very interesting to investigate the modern branch
predictors)

In order to mitigate the loop overhead sometimes it is possible to
unroll the loop, which will do something like:

i++
code
i++
code
i++
code

Of course, not all instructions are created equal, today it is far
from intuitive what exactly is happening (especially now when people
don't own the computers they run their code on), but for me a good
intuition is:

bitwise, sum, local jump is fast
division by 2 is natural, so it is fast (x >> 1). division is a search https://youtu.be/o4-CwDo2zpg?t=354
same with multiplication anyting but 2 is slow (x << 1)
almost all other math is slow
locality is king
hot method calling adds ~10 instructions cost for i:=0; i<n; i++ { f() }
atomic operations are ~25 instructions

My CPU is ~3ghz, which means it can do 3 * 10^9 things per second, so
we have the for loop in the benchmark so 1.36 per iteration sounds
about right, considering about 4 things per iteration.
(https://en.wikipedia.org/wiki/Arithmetic_logic_unit)
Now that we are so far abstracted from the machine, it is incredibly
difficult to have good intuition about how much things cost, so I
usually multiply my guess by 5 or so (to be cloud native, haha).

Anyway, let's move to the cost of calling a function.

Function calling

Calling functions is quite simple. We need to prepare its parameters
and then call it. This involves putting their values on the call stack
(offset by SP) and then invoking CALL. The function itself uses those
parameters and then invokes RET.

// tell the go compiler to not inline the function by doing go:noinline
// semantic comment
//go:noinline
func add(a, b int) (int, int) {
        return a + b, 1000
}
// go build main.go && go tool objdump -s main.add main
MOVQ $0x0, 0x18(SP)
MOVQ $0x0, 0x20(SP)
MOVQ 0x8(SP), AX
ADDQ 0x10(SP), AX
MOVQ AX, 0x18(SP)
MOVQ $0x3e8, 0x20(SP)
RET

func main() {
        a, b := add(100, 200)
        add(a, b)
}

// go build main.go && go tool objdump -s main.main main
MOVQ $0x64, 0(SP)
MOVQ $0xc8, 0x8(SP)
CALL main.add(SB)
MOVQ 0x10(SP), AX
MOVQ AX, 0x38(SP)
MOVQ 0x18(SP), AX
MOVQ AX, 0x30(SP)
MOVQ 0x38(SP), AX
MOVQ AX, 0x28(SP)
MOVQ 0x30(SP), AX
MOVQ AX, 0x20(SP)
MOVQ 0x28(SP), AX
MOVQ AX, 0(SP)
MOVQ 0x20(SP), AX
MOVQ AX, 0x8(SP)
CALL main.add(SB)

So calling a function with two parameters, that returns 2 values does
at least 6 things. (prepare the call stack, call, prepare the return
stack, return)

Inlining

Because of the calling overhead, all compiled languages try to inline the functions so it won't have to do do the CALL,RET and stack preparation work.

First let's discuss how the CPU executes code. In the Von Neumann
architecture(which is pretty much all modern computers), code and data are in the same memory, so it is only in the eye of the beholder if something is code or data, pretty much the same way as it is up to you if something is the character 'a' or the integer 97, when 'a' is encoded as 01100001.

https://en.wikipedia.org/wiki/Von_Neumann_architecture
https://en.wikipedia.org/wiki/Harvard_architecture (alternative
architecture that has separate memory for code and data)

Let's create super simple 8 bit CPU with two 1 byte general purpose
registers, R0 and R1:

1 MOV addr, r0   copy addr in r0
2 MOV addr, r1   copy addr in r1
3 MOV r0, addr   copy r0 in addr
4 MOV r1, addr   copy r1 in addr
5 MOV r0, $value store $value in r0
6 MOV r1, $value store $value in r1
7 ADD  adds r0 and r1 and stores value in r0
8 JMP  addr jump to given address
9 HAL  stop

(from Richard Buckland's 4 bit CPU)
https://www.youtube.com/watch?v=gTeDX4yAdyU

in our example, we will set the CPU to execute from memory address 0,
and an example program that counts forever would look like:

0000: MOV r0, $0
0001: MOV r1, $1
0002: ADD
0003: MOV r0, 10
0004: JMP 1

now let's look at the memory layout byte by byte

addr:0  00000101 // mov r0, $0
addr:1  00000000 
addr:2  00000110 // mov r1, $1
addr:3  00000001 
addr:4  00000111 // add
addr:5  00000011 // mov r0, 10
addr:6  00001010
addr:7  00001000 // jmp 1
addr:8  00000001
addr:9  00000000 // nothing
addr:10 00000000 // result of mov r0, 10

As you can see, there is no difference between address 0 and address

The only thing stopping the cpu to execute address 8,9,10 is the JMP on address 7, that makes it go to address 0

The way the cpu actually executes it is by using fetch-decode-execute
cycle (https://en.wikipedia.org/wiki/Instruction_cycle)

Fetch Stage: The next instruction is fetched from the memory address
that is currently stored in the program counter and stored into the
instruction register. At the end of the fetch operation, the PC points
to the next instruction that will be read at the next cycle.

Decode Stage: During this stage, the encoded instruction presented in
the instruction register is interpreted by the decoder.

Execute Stage: The control unit of the CPU passes the decoded
information as a sequence of control signals to the relevant function
units of the CPU to perform the actions required by the instruction,
such as reading values from registers, passing them to the ALU to
perform mathematical or logic functions on them, and writing the
result back to a register. If the ALU is involved, it sends a
condition signal back to the CU. The result generated by the operation
is stored in the main memory or sent to an output device. Based on the
feedback from the ALU, the PC may be updated to a different address
from which the next instruction will be fetched.

Repeat Cycle

To access ram is in the order of 100ns, so if we have to waste 100ns
for every instruction the whole thing will be horrible, to mitigate
this CPUs have hierarchy of caches (tlb[21], l1,l2,l3).

L1 cache reference                           0.5 ns
Executing Instruction                        1   ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns 14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns 20x  L2 cache,
                                                    200x L1 cache

http://norvig.com/21-days.html#answers
https://gist.github.com/jboner/2841832
http://www.cim.mcgill.ca/~langer/273/18-notes.pdf
https://people.freebsd.org/~lstewart/articles/cpumemory.pdf

The l1 cache is usually split into an instruction cache and data
cache, and it is in the order of ~32kb per core (how this cache
hierarchy works in the modern multi-core CPUs is a topic on its own,
but lets think as if we have one core only).

There are other mitigations to how slow the fetch-execute cycle is,
such as instruction pipelining, vectorization, out of order execution
etc (you can check them out on the bottom of the instruction cycle
wiki page https://en.wikipedia.org/wiki/Instruction_cycle)

https://en.wikipedia.org/wiki/Instruction_pipelining
https://en.wikipedia.org/wiki/Hazard_(computer_architecture)
https://en.wikipedia.org/wiki/Out-of-order_execution
https://en.wikipedia.org/wiki/Speculative_execution

Modern CPUs are incredibly complicated systems, and the model I use is
that in most case (unless proven otherwise) "it does one thing at a
time" works ok to do back-of-the-napkin calculations. (e.g., some
modern hashing algorithms that abuse vectorization can hash incredible
amounts of data)

So, inlining is a dance.

I will refer to:
http://www.cs.technion.ac.il/users/yechiel/c++-faq/inline-and-perf.html
[9.3] Do inline functions improve performance?
Yes and no. Sometimes. Maybe.

There are no simple answers. Inline functions might make the code
faster, they might make it slower. They might make the executable
larger, they might make it smaller. They might cause thrashing,
they might prevent thrashing. And they might be, and often are,
totally irrelevant to speed.

inline functions might make it faster:
  As shown above, procedural integration might remove a bunch of
  unnecessary instructions, which might make things run faster.

inline functions might make it slower: 
  Too much inlining might cause code bloat, which might cause
  "thrashing" on demand-paged virtual-memory systems. In other
  words, if the executable size is too big, the system might spend
  most of its time going out to disk to fetch the next chunk of
  code.

inline functions might make it larger:
  This is the notion of code bloat, as described above. For
  example, if a system has 100 inline functions each of which
  expands to 100 bytes of executable code and is called in 100
  places, that's an increase of 1MB. Is that 1MB going to cause
  problems? Who knows, but it is possible that that last 1MB could
  cause the system to "thrash," and that could slow things down.

inline functions might make it smaller:
  The compiler often generates more code to push/pop
  registers/parameters than it would by inline-expanding the
  function's body. This happens with very small functions, and it
  also happens with large functions when the optimizer is able to
  remove a lot of redundant code through procedural integration â€”
  that is, when the optimizer is able to make the large function
  small.

inline functions might cause thrashing:
  Inlining might increase the size of the binary executable, and
  that might cause thrashing.

inline functions might prevent thrashing
  The working set size (number of pages that need to be in memory
  at once) might go down even if the executable size goes up. When
  f() calls g(), the code is often on two distinct pages; when the
  compiler procedurally integrates the code of g() into f(), the
  code is often on the same page.

inline functions might increase the number of cache misses: 
  Inlining might cause an inner loop to span across multiple lines
  of the memory cache, and that might cause thrashing of the
  memory-cache.

inline functions might decrease the number of cache misses:
  Inlining usually improves locality of reference within the
  binary code, which might decrease the number of cache lines
  needed to store the code of an inner loop. This ultimately could
  cause a CPU-bound application to run faster.

inline functions might be irrelevant to speed: 
  Most systems are not CPU-bound. Most systems are I/O-bound,
  database-bound or network-bound, meaning the bottleneck in the
  system's overall performance is the file system, the database or
  the network. Unless your "CPU meter" is pegged at 100%, inline
  functions probably won't make your system faster. (Even in
  CPU-bound systems, inline will help only when used within the
  bottleneck itself, and the bottleneck is typically in only a
  small percentage of the code.)

There are no simple answers: You have to play with it to see what
is best. Do not settle for simplistic answers like, "Never use
inline functions" or "Always use inline functions" or "Use inline
functions if and only if the function is less than N lines of
code." These one-size-fits-all rules may be easy to write down,
but they will produce sub-optimal results.

Interfaces

Lets say we have a common case where a struct gets an io.Writer
interface to write into:

type Database struct {
        writer io.Writer
}
func (d *Database) SetWriter(f io.Writer) error {
        d.writer = f
}
....
func (d *Database) writeBlob(b []byte) error {
        checksum := hash(b)
        _, err := d.writer.Write(checksum)
        if err != nil {
                return err
        }
        _, err := d.writer.Write(b)
        return err
}
....

Now because d.writer can change at runtime, the compiler can never
know what the other end of d.writer is, so it can never inline it even
if it wants to (you can imagine the actual os.File.Write is just doing
the write syscall)

Another issue is that the thing on the other end of d.writer could be
a pointer, so it has to be checked because if it is and it is nil (0),
we must panic accordingly.

The way this works is pretty much:

if x == nil
   goto panic
... code

return x

panic:
help build a stacktrace

(pseudo code)

START:
  CMPQ 0x10(CX), SP
  JBE CALL_PANIC

  MOVQ 0x40(SP), AX
  TESTQ AX, AX
  JLE PANIC
  ... work work ...
TRACE:
  MOVQ CX, 0x50(SP)
  MOVQ 0x28(SP), BP
  ADDQ $0x30, SP
  RET
PANIC:
  XORL CX, CX // cx = 0
  JMP TRACE
CALL_PANIC:
  CALL runtime.morestack_noctxt(SB)
  JMP START

That means on every function call to writeBlob has to check if
d.writer is nil, because it can be and there is no way for the
compiler to know at compile time, and then prepare the stack and call
it.

How Slow is Dynamic Dispatch exactly?

More concrete example:

package main

type Operation interface {
        Apply() int
}

type Number struct {
        n int
}

func (x Number) Apply() int {
        return x.n
}

type Add struct {
        Operations []Operation
}

func (x Add) Apply() int {
        r := 0
        for _, v := range x.Operations {
                r += v.Apply()
        }
        return r
}

type Sub struct {
        Operations []Operation
}

func (x Sub) Apply() int {
        r := 0
        for _, v := range x.Operations {
                r -= v.Apply()
        }
        return r
}

type AddCustom struct {
        Operations []Number
}

func (x AddCustom) Apply() int {
        r := 0
        for _, v := range x.Operations {
                r += v.Apply()
        }
        return r
}

func main() {
        n := 0
        op := Add{Operations: []Operation{Number{n: 5}, Number{n: 6}}}
        n += op.Apply()

        opc := AddCustom{Operations: []Number{Number{n: 5}, Number{n: 6}}}
        n += opc.Apply()
}

Lets look at main.Add.Apply and main.AddCustom.Apply

// go build main.go && go tool objdump main
TEXT main.Add.Apply(SB) main.go
MOVQ FS:0xfffffff8, CX
CMPQ 0x10(CX), SP
JBE 0x4526c7
SUBQ $0x30, SP
MOVQ BP, 0x28(SP)
LEAQ 0x28(SP), BP
MOVQ 0x40(SP), AX
TESTQ AX, AX
JLE 0x4526c3
MOVQ 0x38(SP), CX
XORL DX, DX
XORL BX, BX
JMP 0x452678
MOVQ 0x20(SP), SI
ADDQ $0x10, SI
MOVQ AX, DX
MOVQ CX, BX
MOVQ SI, CX
MOVQ CX, 0x20(SP)
MOVQ DX, 0x18(SP)
MOVQ BX, 0x10(SP)
MOVQ 0(CX), AX
MOVQ 0x8(CX), SI
MOVQ 0x18(AX), AX
MOVQ SI, 0(SP)
CALL AX
MOVQ 0x18(SP), AX
INCQ AX
MOVQ 0x10(SP), CX
ADDQ 0x8(SP), CX
MOVQ 0x40(SP), DX
CMPQ DX, AX
JL 0x452666
MOVQ CX, 0x50(SP)
MOVQ 0x28(SP), BP
ADDQ $0x30, SP
RET
XORL CX, CX
JMP 0x4526b4
CALL runtime.morestack_noctxt(SB)
JMP main.Add.Apply(SB)

TEXT main.AddCustom.Apply(SB) main.go
MOVQ 0x8(SP), AX
MOVQ 0x10(SP), CX
XORL DX, DX
XORL BX, BX
JMP 0x4526fa
MOVQ 0(AX)(DX*8), SI
INCQ DX
ADDQ SI, BX
CMPQ CX, DX
JL 0x4526f0
MOVQ BX, 0x20(SP)
RET

Benchmark:

goos: linux

goarch: amd64

BenchmarkInterface-8    171836106                6.81 ns/op

BenchmarkInline-8       424364508                2.70 ns/op

BenchmarkNoop-8         898746903                1.36 ns/op

PASS

ok      command-line-arguments  4.673s

Optimization

You know the golden rule: always measure first.

When I wrote https://github.com/rekki/go-query I thought the slowest
part was the binary search, I thought: well..the loop cant be
unrolled, and the branchy algorithm is too difficult on the
branch-predictor, so it is probably the slowest, I will just optimize
it, and I did.

Total waste of time, the overhead of the interface dispatch is 70% of
the time, not only I wasted my time, but I introduced a super shitty
bug that was not caught by 90% coverage and etc, luckily it did not
end up in production (by sheer luck!).

Anyway, this is a reminder to myself: always measure.

Oh BTW, of course, this whole thing almost never hits you, ~10 extra
instructions per call never matter. (almost)

How to measure though? Considering what is executed in the benchmark
is very likely not the thing being executed in production, so for
example, I could have a benchmark that is super fast and nice, but
when it runs in prod, the thing gets inlined and starts page
thrashing.

There is a whole spectrum of tools to use from statistics powered
micro benchmark tools to flamegraph[19] visualizations, and the whole
topic is a post on its own, but I believe anything you do to measure,
will be better than intuition.

Luckily go's profiling tools are incredible[20] and you can get good
insights very quickly.

BTW, I did have a profile for go-query before I went into optimizing,
and I saw that the cost is in the Next() call, but because I did not
get deeper into it and I decided to start with optimizing the binary
search, instead of looking carefully into the data. As the poet says:
"everywhere you look, you see what you are looking for".

--
Fri 7 Feb 18:08:42 CET 2020,
Borislav Nikolov (https://github.com/jackdoe)

References:
[1] https://github.com/rekki/go-query
Low level full text search library.

[2] https://youtu.be/o4-CwDo2zpg?t=354
Fastware - Andrei Alexandrescu

[3] https://www.youtube.com/watch?v=Qq_WaiwzOtI
CppCon 2014: Andrei Alexandrescu "Optimization Tips - Mo' Hustle
Mo' Problems"

[4] https://www.youtube.com/watch?v=FJJTYQYB1JQ
CppCon 2019: Andrei Alexandrescu â€œSpeed Is Found In The Minds of
People"

[5] https://en.wikipedia.org/wiki/Arithmetic_logic_unit
[6] https://en.wikipedia.org/wiki/Von_Neumann_architecture
[7] https://en.wikipedia.org/wiki/Harvard_architecture
[8] https://en.wikipedia.org/wiki/Instruction_cycle
[9] https://en.wikipedia.org/wiki/Instruction_pipelining
[10] https://en.wikipedia.org/wiki/Hazard_(computer_architecture)
[11] https://en.wikipedia.org/wiki/Out-of-order_execution
[12] https://en.wikipedia.org/wiki/Speculative_execution

[13] https://www.youtube.com/watch?v=gTeDX4yAdyU
Lecture 3: Machine Code - Richard Buckland UNSW

[14] http://norvig.com/21-days.html#answers
Basic numbers for back of the napkin calculations

[15] https://gist.github.com/jboner/2841832
Basic numbers for back of the napkin calculations

[16] http://www.cim.mcgill.ca/~langer/273/18-notes.pdf
TLB misses

[17] https://people.freebsd.org/~lstewart/articles/cpumemory.pdf
What Every Programmer Should Know About Memory

[18] http://www.cs.technion.ac.il/users/yechiel/c++-faq/inline-and-perf.html
[9.3] Do inline functions improve performance?

[19] http://www.brendangregg.com/flamegraphs.html
Flamegraphs

[20] https://blog.golang.org/profiling-go-programs
Profiling Go Programs

[21] https://en.wikipedia.org/wiki/Translation_lookaside_buffer
Translation lookaside buffer

[22] https://godbolt.org
Compiler Explorer is an interactive online compiler which shows the
assembly output of compiled C++, Rust, Go (and many more) code.