Forem: Tamir Bahar

More Memory Profiling (in Python)

Tamir Bahar — Fri, 02 Jul 2021 15:10:54 +0000

TL;DR

Graph memory-usage over time, correlate with logs, profit.

Overconfidence

Recently, I had to reduce the memory consumption of a Python process that became entirely unreasonable. Now, a while back I wrote about finding memory leaks in Python. I was pleased with myself and sure that with the knowledge I gained then, I can surely get this done!

And oh, was I wrong...

Harsh Reality

You see, both pympler and tracemalloc are wonderful tools. But like all tools, they have limitations. When you have a long-running (days) process with many (hundreds of millions) objects, the memory and performance costs of your tools add up quite significantly. Waiting for pympler to query all objects takes forever, and following references is completely impractical; viewing tracemalloc statistics is nice, but doesn't help you narrow things down enough.

So, after 2 weeks of zero-to-minimal improvements (though I was sure I'm on the right track) I decided to try a different approach to understanding the memory usage of my code.

To The Rescue

Enter memlog.py.

memlog is a simple, naive tool. It tracks the overall memory usage on a machine, and logs it (with a timestamp) to a CSV. That's it. While the recorded data may include significant noise, running your code (& memlog) inside a container can reduce it significantly. Also, background memory noise tends to be insignificant when your process hogging all of your memory...

So, I ran my process (with logs), ran memlog, and plotted a memory-over-time graph:

And oh, oh no.

Insight

Looking at the graph, we can divide it into 3 parts:

A near-instant rise at the beginning. This is by far the bulk of the memory-usage increase;
A slow, gradual increase over the entire time-scale;
A near-instant drop in memory-usage.

Those parts are basically:

Loading the data-set and various initialization;
The bulk of the processing;
Program termination.

And for the past 2 weeks I've been busy reducing the memory-usage of... the second part. Being absolutely sure it's the most significant.

So yeah, that hurt. But only for a short time. For you see, with this newfound knowledge I could safely focus on the first few minutes of execution and disregard the rest for the time being.

True. I'll have to test the whole thing once I'm make any significant changes. Memory-usage might spike at a later point. Memory-optimization may cause performance degradation. But unless I reduce that uptick at the beginning I won't get any significant improvements.

Profit

A week later, we managed to reduce memory-usage by 30% while reducing overall processing time by a similar percentage. We had to:

Add a de-duplicating weakref based cache;
Add a pre-processing step;
Make our code more cache-friendly by sorting our data;
Remove a massively over-engineered control mechanism.

But it was all made possible by focusing on the right part. Had I not plotted that memory graph, I could've easily spent another 2 weeks without any significant progress.

Old & Wise

So whatever you do, I highly suggest you graph your data. No need to be smart about it. Log it, graph it, correlate to your logs.

GitLab CI Tricks

Tamir Bahar — Wed, 10 Mar 2021 09:02:10 +0000

In writing our CI setup at Vdoo, we came across some interesting challenges. Having solved them and used the solutions for quite a while, we decided it is best to share, and hopefully save others some time and effort solving similar problems.

Of course, there are alternative solutions to the challenges we have dealt with, and some solutions are probably superior to ours. We are happy to hear about such solutions and to improve our own.

As for the code presented in this post - it was extracted from our CI and cleaned up a bit. As such, it is missing some necessary boilerplate. It will not work as-is, and some work will be required to adapt it to your CI. That said, it should clearly lay out the solutions. (You can think of it as slide-ware.)

LFS-Check

All transitions can be bumpy. For us, the transition from storing binary files as regular git blobs to storing them using LFS was one such bumpy transition.

We made sure to include all the LFS-relevant files & patterns in a .gitattributes file, but ensuring everyone (including people who only occasionally work on the relevant repo) properly setup their environments for LFS took some time. In the mean-time, we kept getting files that should be in LFS committed and pushed as regular files in Merge-Requests.

To circumvent that, we set up a simple check at the start of our CI process to ensure all relevant files are indeed stored in LFS.

lfs-check:
  only:
    refs:
      - merge_requests
  script:
    - git lfs install
    - git add --renormalize -u
    - |
      if ! git diff --cached --name-only --exit-code ; then
        echo
        echo
        echo "=============================="
        echo "#  Please renormalize files  #"
        echo "=============================="
        echo
        echo "git add -u --renormalize"
        echo "git commit --amend"
        exit 1
      fi

When people pushed the files the wrong way - the CI would fail with an informative error and instructions.

Required Commit

Every so often a change is made to the code, rendering the code before the change unworkable or irrelevant. This can happen for many reasons. Here are some examples:

A bug was fixed in the CI. This is all too common when forgetting to properly lock your dependencies (including recursive ones!)
A very time-consuming update was made (re-training an ML model, anyone?)
The change is significant and will make rebasing a pain
A significant bug was fixed, making tests on the previous versions mostly irrelevant

Once you introduce such a change to your code, you want people to know about it, and you want to stop wasting cycles on it.

To achieve this goal, we created a required-commit mechanism in our CI. For the CI to work, the required-commit must be an ancestor of the current commit. If it isn't - the CI fails with a descriptive error & instructions for fixing the issue.

Once we have a new required-commit, we inform all developers in a dedicated Slack channel and update the CI to match. This ensures that even if a developer misses the notification on Slack, the CI will let them know what needs to be done.

The solution consists of a simple CI job:

required-commit:
  only:
    refs:
      - merge_requests
  script:
    - apt-get update
    - |
      if ! git merge-base --is-ancestor ${REQUIRED_COMMIT:-HEAD} HEAD ; then
        echo
        echo
        echo "============================="
        echo "#      Rebase Required      #"
        echo "============================="
        echo
        echo "Your base commit is out of date."
        echo "Please update to ${REQUIRED_COMMIT} or later."
        echo "The easiest fix is to rebase-onto or merge-from origin/main."
        echo
        echo
        exit 1
      fi

And a custom variable defined in the CI settings (see Create a custom variable in the UI):

Conditionally Building Job Docker Images

Some of our code is deployed via Docker images. As such - we want our CI to build and test those images. Some tests require running a Docker container and communicating with it, but some tests (especially unit- and integration-tests) are easier to run inside the said containers. To accommodate the latter, we use our Docker images as the base images for the CI test jobs.

This is easy enough to do in the CI. In our case, however, building the Docker images takes a very long time. In trying to reduce this time, we split our build into two parts. The first - a long compilation phase, building some rarely-changing code; the second - installation of our fast-changing Python code & all relevant dependencies.

Noticing the split between the fast-changing and rarely-changing parts of our build, we decided to split it in half, only building the first part when there's an actual change to it.

To do that, however, we have to conditionally build the Docker image for the first half, and in the second half use either the preexisting first half or the newly built one.

The Solution - Build, Proxy, Promote

Our solution uses a model consisting of multiple CI jobs handling different parts.

Build jobs - responsible for building docker images. Either conditionally (for the first part) or consistently (for the second part).
Proxy jobs - responsible for handling the conditional nature of the build jobs, providing the next job with the relevant tag for the Docker images - either :latest or the current commit.
Promote jobs - responsible for tagging the newly built images with :latest and pushing them. They run last.

For our use-case, we used the following setup:

Conditional Build job to build the rarely-changing code
Proxy job to yield the relevant tags
Build job to build & install the fast-changing code
Test job, to test the newly build code
Promote job, pushing the newly build images as :latest if the tests passed

To implement it, we created the following .yml configuration, representing the build-proxy-promote model, and used include:file to bring it into our .gitlab-ci.yml.

# This file allows creating a prebuild-proxy-(build)-promote workflow with ease.  
#  
# The idea is that that in the prebuild step we build less-frequently-changed  
# docker images than in the build step. This allows us to significantly speed  
# up CI times.  
#  
# Setting Up  
# ==========  
#  
# A basic setup consists of the following:  
#  
# 1\. Prebuild (extends .bpp:build)  - build docker images if relevant files changed  
# 2\. Proxy (extends .bpp:proxy) - allows the rest of the CI to know whether Prebuild created new images or not  
# 3\. Build [Optional] (extends .bpp:build) - builds extra, more-frequently-changing images.  
#                                             This is not a conditional step!  
# 4\. Use (custom step) - here we actually use the images we created!  
# 5\. Promote (extends .bpp:promote) - if required, pushes the newly built images to the project's repository.  
#  
# These 5 steps should be in 5 different, consecutive stages for things to work.  
# The prebuild step, being conditional, should not have any other step requiring it.  
# All other steps (that need the prebuilt images) should require the proxy step instead,  
# and use the ${PROXY_TAG} to as a label to the relevant docker images.


.bpp:build:
  variables:
    # The names of all the docker images we want to pull from our registry
    TO_PULL: ""
    # The tag to use for pulling the images. This will usually be ${PROXY_TAG}
    PULL_TAG: ""
    # The name of the image and path of the dockerfile for building docker images.
    # The root path for the dockers will be the root of the project
    # Format the variable as follows:
    #
    #     >-
    #       "some_name the/relevant/path/Dockerfile"
    #       "some_other_name another/relevant/path/Dockerfile"
    #
    # Note that the quotes are significant!
    TO_BUILD: ""
    # The names of the images we want to push.
    # They will all be pushed with the ${CI_COMMIT_SHA} tag.
    TO_PUSH: ""
  script:
    - docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY_IMAGE}
    - export DOCKER_BUILDKIT=1 # This cannot be in the `variables` field since users overwrite it.
    - |
      for IMAGE_NAME in ${TO_PULL}
      do
          echo "***********************************"
          echo "Pulling ${IMAGE_NAME}"
          echo "-----------------------------------"
          echo "docker pull ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${PULL_TAG}"
          echo "DOCKER_BUILDKIT=1 docker tag \
                ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${PULL_TAG} \
                ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:latest"
          echo "***********************************"
          docker pull ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${PULL_TAG}
          docker tag \
              ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${PULL_TAG} \
              ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:latest
      done
    - |
      eval "ARRAY=($TO_BUILD)"
      for ITEM in "${ARRAY[@]}"
      do
          MY_NAME=${ITEM% *}
          MY_PATH=${ITEM#* }
          echo "***********************************"
          echo "Building ${MY_NAME} from ${MY_PATH}"
          echo "-----------------------------------"
          echo "DOCKER_BUILDKIT=1 docker build \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              -t ${CI_REGISTRY_IMAGE}/${MY_NAME} \
              -t ${CI_REGISTRY_IMAGE}/${MY_NAME}:${CI_COMMIT_SHA} \
              -f ${MY_PATH} \
              --label "commit_sha=${CI_COMMIT_SHA}" \
              ."
          echo "***********************************"
          docker build \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              -t ${CI_REGISTRY_IMAGE}/${MY_NAME} \
              -t ${CI_REGISTRY_IMAGE}/${MY_NAME}:${CI_COMMIT_SHA} \
              -f ${MY_PATH} \
              --label "commit_sha=${CI_COMMIT_SHA}" \
              .
      done
    - |
      for IMAGE_NAME in $TO_PUSH
      do
          echo "***********************************"
          echo "Pushing ${IMAGE_NAME}"
          echo "-----------------------------------"
          echo "DOCKER_BUILDKIT=1 docker push ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${CI_COMMIT_SHA}"
          echo "***********************************"
          docker push ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${CI_COMMIT_SHA}
      done


.bpp:proxy:
  variables:
    # The names of jobs we want to proxy - if any of them succeeded, we proxy.
    BUILD_JOBS: ""
  script:
    - PROXY_TAG=latest
    - apt-get -qq update
    - apt-get -qq install jq
    # Get the successful jobs for the current pipeline
    - >-
      curl
      --header "PRIVATE-TOKEN:${GITLAB_TOKEN}"
      "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/pipelines/${CI_PIPELINE_ID}/jobs?scope[]=success"
      > jobs.json
    # Compare the job names from the pipeline with the provided job names
    - EXECUTED=$(comm -12 <(jq -r '.[].name' jobs.json | sort) <(echo ${BUILD_JOBS} | tr ' ' '\n' | sort))
    # If a build job was executed, we need to set the proxy tag to the current commit sha.
    - |
      if [ ! -z "$EXECUTED" ]
      then
          PROXY_TAG=${CI_COMMIT_SHA}
      fi
    - echo "PROXY_TAG=${PROXY_TAG}" >> deploy.env
    # Print out the proxy tag - for debug purposes
    - echo "PROXY_TAG=${PROXY_TAG}"
  artifacts:
    # To get the proxy tag, you need to get the artifacts from this job.
    # The proxy tag will be ${PROXY_TAG}
    reports:
      dotenv: deploy.env


.bpp:promote:
  variables:
    # The names of the images you wish to promote.
    # They need to have been built & pushed by a previous build step.
    TO_PROMOTE: ""
  script:
    - |
      if [ "${PROXY_TAG}" = "latest" ]; then
          echo "Nothing to promote."
      else
          echo "Promoting docker image."
          docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY_IMAGE}

          for IMAGE in ${TO_PROMOTE} ; do
              docker pull ${CI_REGISTRY_IMAGE}/${IMAGE}:${CI_COMMIT_SHA}
              docker tag ${CI_REGISTRY_IMAGE}/${IMAGE}:${CI_COMMIT_SHA} ${CI_REGISTRY_IMAGE}/${IMAGE}:latest
              docker push ${CI_REGISTRY_IMAGE}/${IMAGE}:latest
          done
      fi

Finding a memory-leak in my Python code

Tamir Bahar — Sat, 30 Jan 2021 15:33:26 +0000

Last week I had to fix a memory leak in a Python program for the first time. A long running process started eating too much RAM (only ~20GB to much) and the friendly OOM Killer had to step in and terminate this. Since this kept happening, I had to go ahead and fix the issue.

Step 1 - Reproduction

As with every bug, before you can reliably fix it, you must reproduce it.

Now, while I had a reliable reproduction (after all, the process had regular dates with the OOM Killer), 3 days isn't the best cycle time when you wanna solve a bug. So into the code we go.

The main idea is to start with the main loop, and try to narrow down the code that is must run for the leak to manifest. The process involves some educated guesses (where are the likely memory and allocation hogs in your process? What parts are likely to leak? Do you have any code that requires cleanup?), waiting, frustration, and tools.

tracemalloc

While each developer and codebase have their own unique guesses and frustrations, good tooling applies more widely. For this part, I used Python's tracemalloc module.

Among other things, tracemalloc allows tracking memory usage between 2 points in your code in a very low-overhead manner.

tracemalloc.start() # Start the memory trace

code_suspected_of_leak()

current, peak = tracemalloc.get_traced_memory() # Get memory stats

After running this code, peak will hold the peak-memory-usage during the trace period, and current will hold the difference from the start of the trace to the current state. You should expect current to be non-zero. But if it goes too high - your code is probably leaking.

By placing such traces around suspect pieces of our code, we can find which parts are leaking. Just remember - only do this with functions that are expected to retain no state. If a function mutates an external object, or is a member function, it is very to exhibit changes in memory usage.

Step 2 - Triage

Once we have a reproduction (that hopefully takes a relatively short amount of time), we want to find the leaking code. We can try and keep narrowing our measured code down until we find the relevant line, but the deeper we go, the harder it is to separate the leak from normal execution.

So at this point, we'd like to look into the allocated memory, and see which objects are there when they shouldn't be.

pympler

For inspecting the objects in a Python process, I recommend using pympler.

Pympler is a development tool to measure, monitor and analyze the memory behavior of Python objects in a running Python application.

We're going to use it to do 2 things.

Inspecting Allocated Objects

First, we're going to use pympler to show us which objects were allocated during our repro & are still allocated.

from pympler import tracker, muppy, refbrowser
from functools import lru_cache

# Naive code to trigger a leak
class Value:
    def __init__(self, value):
        self._value = value

    def __repr__(self):
        return f"Value({self._value})"


@lru_cache(maxsize=100)
def get_value(value):
    return Value(value)


def code_suspected_of_leak():
    for x in range(10):
        print(get_value(x))

# Measuring code
def main():
    tr = tracker.SummaryTracker()

    code_suspected_of_leak()

    tr.print_diff()

Once we run this, we get a nice table showing us a summary of objects created and destroyed:

                  types |   # objects |   total size
======================= | =========== | ============
                   list |        4892 |    500.59 KB
                    str |        4887 |    341.45 KB
                    int |        1053 |     28.79 KB
                   dict |          13 |      1.90 KB
         __main__.Value |          10 |    640     B
  function (store_info) |           1 |    144     B
                   cell |           2 |    112     B
                weakref |           1 |     88     B
                  tuple |          -8 |   -680     B

As you can see - there are quite a few primitive objects generated, and also some __main__.Value objects. In my experience, primitives are harder to track, as they lack meaning in the code. Your own types, however, are usually only used in certain parts of the codebase, making them easier to make sense of.

Now that we see that we have 10 new Value objects, it is time to figure out who's holding them in memory.

def output_function(o):
    return str(type(o))

all_objects = muppy.get_objects()
root = muppy.filter(all_objects, Value)[0]
cb = refbrowser.ConsoleBrowser(root, maxdepth=2, str_func=output_function)
cb.print_tree()

This'll print the following:

<class '__main__.Value'>-+-<class 'list'>
                         +-<class 'functools._lru_cache_wrapper'>-+-<class 'list'>
                                                                  +-<class 'dict'>

Giving away the issue - the lru_cache is keeping our Value objects. Just as designed...

I know this looks like a bit of a contrived example, but the lru_cache keeping objects in memory was exactly the issue I had. It was just buried under far more code.

Step 3 - Solution

Currently, I use the ugliest solution I can imagine - functions decorated with lru_cache have a cache_clear() method, and I'm calling that at specific places in my code. It's ugly, but it works.

A cleaner solution would require dedicated caches & better cleanup mechanisms. You can read a relevant discussion here.

TIL Python attribute lookup order is tricky

Tamir Bahar — Wed, 03 Jun 2020 18:59:53 +0000

This post is brought to you in the spirit of converting tweetstorms to blogposts.
to the tweetstorm

Surprise! 🎁

In Python, if property access raises AttributeError, and the class implemented getattr, it will get called with the property name.
This results in some very cryptic errors.

If you run the following code (repl):

class Thing:
    name = "Thing"


class NameProvider:
    def __init__(self, name):
        self.name = name

    def get_name(self):
        return self.nam


class ThingWrapper:
    def __init__(self, thing, name_provider):
        self.thing = thing
        self.name_provider = name_provider

    @property
    def name(self):
        return self.name_provider.get_name()

    def __getattr__(self, name):
        return getattr(self.thing, name)


def main():
    thing = Thing()
    name_provider = NameProvider(name="Not Thing")

    thing_wrapper = ThingWrapper(thing, name_provider)

    print(thing.name)
    print(thing_wrapper.name)


if __name__ == '__main__':
    main()

You'll get a surprising result:

Thing
Thing

You might have expected "Not Thing" as the second line, or maybe an exception to be raised from NameProvider.get_name() due to the typo there (self.nam instead of self.name). But instead, we got the name attribute from our Thing instance.

Analysis 🔎

If you've every used __getattr__() you know that it is called when the named attribute was not found using other lookup mechanisms. That said, it might not be clear to you that this includes properties raising AttributeError exceptions. It definitely wasn't clear to me.

That is, it was unclear to me despite being clearly stated in the documentation for getattr()

object.__getattr__(self, name)
Called when the default attribute access fails with an AttributeError (either __getattribute__() raises an AttributeError because name is not an instance attribute or an attribute in the class tree for self; or __get__() of a name property raises AttributeError). This method should either return the (computed) attribute value or raise an AttributeError exception.

Beside being surprising, there are 2 main issues here:

Any code down the stack from the property can effectively change attribute lookup for the class by throwing an AttributeError. In the above example - a typo in NameProvider caused an attribute to be taken from Thing instead, against the programmer's obvious intention.
The exception is silenced. There is no way for the programmer to catch the exception outside the property getter. This makes the errors very hard to track down. This also means that whenever you add __getattr__() to a class, you're silencing all AttributeError exceptions that were previously thrown from properties.

Like anything in Python, you can hack around the issue. In this case - with fancy decorators!

Solution? 🐍

Consider the following code (repl):

class ExceptionCatcher:
    def __init__(self, f):
        self.f = f
        self.exception = None

    def __call__(self, *args, **kwargs):
        try:
            return self.f(*args, **kwargs)
        except Exception as e:
            self.exception = e
            raise
        else:
            self.exception = None


def store_exception(f):
    return ExceptionCatcher(f)


def load_exception(f):
    def _raise_property_exception(instance, name):
        try:
            class_attr = getattr(instance.__class__, name)
            if not isinstance(class_attr, property):
                return
            exception = class_attr.fget.exception
        except AttributeError:
            return

        if exception:
            raise exception

    def _wrapper(*args, **kwargs):
        _raise_property_exception(*args, **kwargs)
        return f(*args, **kwargs)

    return _wrapper


class ThingWrapper:
    def __init__(self, thing, name_provider):
        self.thing = thing
        self.name_provider = name_provider

    @property
    @store_exception
    def name(self):
        return self.name_provider.get_name()

    @load_exception
    def __getattr__(self, name):
        return getattr(self.thing, name)

If you run this version, you'll get the following exception:

AttributeError: 'NameProvider' object has no attribute 'nam'

This matches our expectations far better.

This result is achieved in two steps. First, we store all the exceptions thrown from name() so that we can throw them again if needed. Then, before calling __getattr__(), we check if we got there due to a property raising an exception. If we did - we just re-raise that exception.

The rest is implementation details, and I probably missed something there (you might notice that I corrected a bug when converting the tweets to this post - in the previous version, I forget to reset the exception storage after successful property retrieval).

While this solution works, and may be useful for detecting similar bugs, I would probably avoid using it in production code. Instead, I'd be happy to have some standard Python construct to provide this functionality.

Snake Eyes: Extension Methods

Tamir Bahar — Thu, 05 Mar 2020 22:24:49 +0000

Today we set out to implement a feature I saw and liked in Kotlin - Extension Methods.

You can follow along with working code samples here, or get the code here

Extension methods are a nice piece of syntactic-sugar that allow you to define free-functions and call them like instance methods. In Kotlin, it looks something like this:

fun Square.draw() {
    drawSquare(this)
}

// ...

val square = getSquare()
square.draw()

Now, since they are free, static functions, they follow the same rules. They are not part of the class, nor have access to private members. And they can only be called in a scope where they are visible. Adding them in your code does not affect other code. Additionally, true member functions, if they exist, take precedence over extension methods (this is especially important with generic extension methods).

In our code today, we'll try to mimic the features of extension methods as closely as possible. We'll use the following syntax:

@extend(Square)
def draw(square):
    draw_square(square)

For extension methods, and the following implementation of Square in our code throughout:

from dataclasses import dataclass

@dataclass
class Square:
    length: int

Monkey Patching 🙈

Python is a very dynamic language. Among other things, it allows us to change the attributes of (non-builtin) types at run-time. This means that we can extend our Square class by adding a draw method to it at run-time.

Square.draw = draw_square

We're now free to call square.draw(). Before we discuss the draw-backs, let's implement it with the syntax we defined:

def monkey_extend(cls):
    def _decorator(f):
        setattr(cls, f.__name__, f)
    return _decorator

@monkey_extend(Square)
def draw(square):
    draw_square(square)

Let's go over this. monkey_extend is a decorator with arguments. This is a common pattern where we use a decorator factory (monkey_extend) to create a new decorator (_decorator) as a closure, giving it access to the parameters passed to the factory (cls). Then, in the core of the decorator, we use setattr to do our monkey-patching.

While this works, it has several issues:

Scope - once set, it can be used with any Square in any scope
Precedence - it will override any existing Square.draw

Dealing with precedence is easy (using hasattr to check for existing .draw) so we'll focus on the scoping first.

Dynamic Attribute Lookup ✨

The first thing we know is that we need our new attribute to be there in some cases, and be gone in others - we need dynamic resolution. To do that, we'll use __getattr__. In Python classes, __getattr__ is used in attribute lookup as a last resort, called when the other ways of looking up attributes came up empty. We'll write our __getattr__ along the following lines:

def my_getattr(obj, name):
    if not has_extension(obj, name):
        raise AttributeError()
    if not is_in_scope(name):
        raise AttributeError()
    return our_extension

The first check, has_extension, is basically checking whether the name we got matches the name of our extension method. Nothing to elaborate yet. Scoping, once again, remains the trickier part.

import functools
import inspect
from collections import ChainMap

def scoped_extend(cls):
    def _decorator(f):
        def _getattr(obj, name):
            # (2)
            if name != f.__name__:
                raise AttributeError()

            # (3)
            frame = inspect.stack()[1].frame
            scope = ChainMap(frame.f_locals, frame.f_globals)
            if scope.get(f.__name__) == f:
                raise AttributeError()

            # (4)
            return functools.partial(f, obj)

        # (1)
        cls.__getattr__ = _getattr
        return f

    return _decorator

This is a bit much, so we'll go over it in detail.

As a basis, we used the same decorator-with-parameters pattern here. We have scoped_extend take the class we want to extend, then return _decorator to get the job done. But instead of setting the attribute we want to extend, we monkey-patch cls's __getattr__ to our implementation (See (1)). This will override any existing implementation of __getattr__, but we'll get to that later. For now, we'll focus on our implementation of __getattr__.

In (2) we implemented has_extnesion - we simply compare the name we got to the name of our extension method. Then, in (3), comes some Python magic. Python allows us to inspect the running program, to see where we were called from and what variables were in scope in that code. To do that, we use the inspect module. We use inspect.stack() to get the call-stack for the current execution, then access the second frame ([1]) to get our caller. This will be where getattr(obj, name) is invoked or obj.name is used. We use .frame to get the execution frame, and .f_locals and f_globals to get the local and global variables available in that scope. They are equivalent to calling globals() or locals() in the relevant frame.

With the scope at hand, we perform a lookup to see whether the extension method we defined is in that scope. To make sure we have our extension method, we get it by name, then ensure that it is truly our method.

Finally, in (4), when we know our method should be active, we bind it to the instance of the extended class and return it.

Better Scoping

While our scope retrieval code works, it's better to put it in a function rather than use it inline:

def _is_in_scope(name, value):
    frame = inspect.stack()[2].frame
    return ChainMap(frame.f_locals, frame.f_globals).get(name) == value

But, oh, we have to increment the stack index to 2 since we're deeper in the callstack. This is risky. Instead, we'll use the following trick to get the frame:

def _get_first_external_stack_frame():
    for frameinfo in inspect.stack():
        if frameinfo.filename == __file__:
            continue
        return frameinfo.frame

def _is_in_scope(name, value):
    frame = _get_first_external_stack_frame()
    return ChainMap(frame.f_locals, frame.f_globals).get(name) == value

Instead of counting the frames in our code, changing them with every change - we'll use the module system. We know that all of our scaffolding is in the same module, but the usage is not. This allows us to easily traverse the stack until we find code that does not belong in our module. That is our calling code.

Since you're probably wondering - yes. You need to change _get_first_external_stack_frame() if you want to put it in a different module. Implementing it is left as an exercise to the reader.

Preserving `getattr`

As mentioned before, our current implementation overrides any existing __getattr__ function for the class. Lucky for us, fixing it is easy:

def no_override_extend(cls):
    def _decorator(f):
        def _default(_obj, _name):
            raise AttributeError()

        original_getattr = getattr(cls, '__getattr__', _default)

        def _getattr(obj, name):
            with suppress(AttributeError):
                return original_getattr(obj, name)

            if name != f.__name__:
                raise AttributeError()

            if not _is_in_scope(f):
                raise AttributeError()

            return functools.partial(f, obj)

        cls.__getattr__ = _getattr
        return f

    return _decorator

In (1) we get the original __getattr__ method, to be stored for later usage. We use the _default function to avoid an extra if later. In (2) we use the saved __getattr__, making sure that we only proceed to our code if it raised an AttributeError exception.

Interlude 🐍

With no_override_extend we have our first "to-spec" implementation of extension methods. We have both scoping and precedence down. It is time to celebrate and rest. But our quest is not done yet.

While our code works well for a proof-of-concept, there are still significant usability issues with it. Since the extension methods we create have nice and clean names, it is likely that we'll want to use those names for other things. Unfortunately, once we do that, we'll override the existing extension methods and they will no longer work:

@extend(Square)
def draw(square):
    draw_square(square)

def draw():
    print("Drawing is awesome!")

# ...

square.draw()  # This will fail, as `draw` has been replaced in this scope.

Indirection 🔀

The Fundemental Theorem of Software Engineering (FTSE) says that any problem can be solved by adding another level of indirection. Let's see how this applies to our problem.

As mentioned in the interlude, our main issue is that of naming. Our extension method is bound to a name, and that name can be overriden in the scope that defines it. If that happens, we lose our extension method. To solve that, we'll add another level of indirection - a scope that can safely hold our extension methods and protect them from being overriden. If you read our previous post you might recall that classes are wonderful for scopes. So we'll use a class.

Our new syntax will look like this:

@extension
class ExtensionMethods(Square):
    def draw(self):
        draw_square(self)

While we're still using a decorator, you may notice that it takes no parameters. Instead, we use the extended type as the base type for our extension class. This allows us to write the extensions like any other subclass, with standard Python syntax, and then use the decorator to install the extensions in it.

Since we've already gone over the principles behind the construction of the decorator, let's jump straight to the code and focus on the differences from the previous version:

def extension(scope_cls):
    def _default(_obj, _name):
        raise AttributeError()

    # (1)
    cls = scope_cls.__base__
    original_getattr = getattr(cls, '__getattr__', _default)

    def _getattr(obj, name):
        with suppress(AttributeError):
            return original_getattr(obj, name)

        # (2)
        if not hasattr(scope_cls, name):
            raise AttributeError()

        # (3)
        if not _is_in_scope(scope_cls):
            raise AttributeError()

        # (4)
        f = getattr(scope_cls, name)

        return functools.partial(f, obj)

    cls.__getattr__ = _getattr

    return scope_cls

First, you can see that there is no nested decorator - only the main one. And, as we mentioned before, we use inheritance to indicate which type we're extending. So in (1) we access the base-class of our extension class to get the class we're extending. Then, in (2) we check whether the requested attribute exists in our extension class. As you can see, the changes are pretty simple and straight-forward. In (3) we make the most important change - we check for the extension class in the scope, not the extension methods. This is the core of this change! And lastly, in (4), we get the required attribute from out extension class.

And with that, we're done.

Final Words

I hope you enjoyed this article. Regardless of that, I hope you never use it in production code.

Snake Eyes: Scopes and IIFE

Tamir Bahar — Fri, 27 Dec 2019 12:09:43 +0000

One of my pet peeves is taking concepts from other languages and "translating" them to Python. Not because it makes good code, but because it's a challenge and it makes me happy.

This time, I've gone after two simple concepts - nested code blocks and IIFE. Both serve similar purposes, and both are missing from Python.

In C++, blocks are often used to limits the lifetime of objects and keep them out of our way when we're done with them. In Python, lifetime is usually less of a concern (as we replace RAII and destructors with context-managers), but having variable names out of our way is desirable.

IIFE offers us a bit more in terms of functionality, as it both creates a scope for our operations, and allows us to get a value from that scope. This is useful both for simpler flow control, and for easily initializing const-qualified variables.

Python does not have any of those constructs. There is no way to create a nested code-block in Python (adding another level of indentation would just have it complain about unexpected-indent), and while lambdas exist, they only allow for a single expression, making them mostly irrelevant for IIFE. On the other hand, Python offers us two wonderful constructs that can be used virtually everywhere - classes and functions.

Classes & Nested Blocks 🧱🧱🧱

In Python, both classes and functions can be nested. You can define a class inside a class, a function in a function, a class in a function or a function in a class. It is all the same. What's more - you can have flow-control in both function (well, obviously) and class bodies (meta-programming much?). Additionally, both nested functions and nested classes create new variable scopes, keeping their insides inside, and are also closures, capable of capturing values from their enclosing scopes. As such, they are the perfect tools for our language-bending shenanigans.

First, nested code blocks. I offer you the following solution:

def f(x):
    print('Classes are great for creating blocks.')
    class _:
        y = x * 2
        print(f'y = {y}')

    print('y is not defined here.')
    y

f(21)

By defining a class, we create a new scope. Inside it, we can do whatever we want, knowing that the code will get execute inline and in order, and the results will not leak into the enclosing scope.

That said - there are some caveats. First, the class remains in scope, and so do all the variables defined in it. They cannot be garbage collected until the function terminates. You can verify it yourself by trying to access _.y in the above example. To remedy that, we need to get rid of the class, or at least its contents. There are many ways to achieve it:

# Replace the class with a bool
@bool
class _:
    x = 1
    print(x)


# Replace the class with None
def empty(*args): return None


@empty
class _:
    x = 1
    print(x)


# Use a metaclass to delete all the variables inside the class
class BlockMeta(type):
    def __new__(cls, name, bases, dict_):
        return super().__new__(cls, name, bases, {})


class Block(metaclass=BlockMeta):
    pass


class _(Block):
    x = 1
    print(x)

I am personally torn between the meta-class approach, as it is explicit and clear, and the @bool approach, as it requires to additional boilerplate.

The second issue with classes as blocks is that while they can be nested freely, a nested class cannot access the variables of its nesting class, rendering block-nesting moot. I do not have a solution for that at present.

Functions & IIFE 🐍🐍🐍

With a solution for nested blocks in hand, it is time to get proper IIFE in Python. For that, we'll naturally be using function. Along with those, we'll use a function's best friend - the decorator!

def iife(f):
    return f()

def describe_number(n):
    @iife
    def message():
        if n < 0:
            return f'{n} is smaller than 0'
        elif n > 0:
            return f'{n} is larger than 0'
        return f'{n} is 0'

    print(message)

describe_number(-1)
describe_number(0)
describe_number(1)

Using the decorator, we immediately call the function, binding the function's name to the return value instead of the function itself. A function returning None (or without a return statement) will just bind the name to None.

While this looks a bit more messy, it can also double as a solution for nested blocks. And unlike the class solution - it can be freely nested.

def iife(f):
    return f()


def block(f):
    f()


def f(x):
    print('Functions are great for creating blocks.')

    @block
    def _():
        my_x = x + 1
        @iife
        def y():
            return my_x * 2

        @block
        def _():
            print(f'y = {y}')

    print('y is not defined here.')
    y


f(20)

That's it for today. I hope you had some fun.

Why do websites ask me where I'm from?

Tamir Bahar — Wed, 17 Oct 2018 10:17:06 +0000

Whenever I need to enter my address in a web-form, I always need to choose the country. It never defaults to the country I'm browsing from.
This always seemed weird to me as the advertisements and localized content always know where I'm from.

Why then don't websites use the same location & tracking info to make my job easier when filling forms?

Recognizing Ducks

Tamir Bahar — Sat, 13 Oct 2018 16:00:21 +0000

After multiple attempts at finding a funny narrative that holds for the entire article and failing miserably, I decided to go with the technical parts alone. Enough of my colleagues found it interesting, so I guess it will hold up without the jokes.

Python gives us multiple ways to check that the objects we pass to functions are of the types we expect. Each method has its advantages and disadvantages.

Just not caring

The first option is naturally to not care about types - just write your code, and hope for the best.
This is a viable method, and is often employed. It is especially fitting in short snippets or scripts you don't expect to maintain much. It just works with no overhead whatsoever.

def poke(duck):
    duck.quack()
    duck.walk()

Inheritance

Another option, as common in OOP languages, is to use inheritance. We can define an Anas class, and expect all of its derivatives to be sufficiently duck-like.

class Anas:
    def quack(self): pass
    def walk(self): pass

class Duck(Anas):
    def quack(self): print('Quack!')
    def walk(self): print('Walks like duck.')

class Mallard(Anas):
    def quack(self): print('Quack!')
    def walk(self): print('Walks like duck.')

def poke(duck):
    assert isinstance(duck, Anas)

Interfaces

While inheritance kinda gets the job done, a robotic duck is definitely not of the genus Anas, while it does implement all the characteristics we care about. So instead of hierarchical inheritance, we can use interfaces.

from abc import ABC, abstractmethod

class IDuck(ABC):
    @abstractmethod
    def quack(self): pass

    @abstractmethod
    def walk(self): pass

class Duck(IDuck):
    def quack(self): print('Quack!')
    def walk(self): print('Walks like duck.')

class RoboticDuck(IDuck):
    def quack(self): print('Quack!')
    def walk(self): print('Walks like duck.')

def poke(duck):
    assert isinstance(duck, IDuck)

Great. And if we don't control the types, we can always write adapters.

The Duck Test

But this is Python. We can do better.

As we know, Python uses duck-typing. So we should be able to use the Duck Test for types. In our example, every object implementing quack() and walk() is a duck. That's easy enough to check.

def is_a_duck(duck):
    for attr in ('quack', 'walk'):
        if not hasattr(duck, attr):
            return False
    return True

def poke(duck):
    assert is_a_duck(duck)
    duck.quack()
    duck.walk()

This works. But we list the isinstance(...) call. We can surely do better.

Metaclasses & Subclass Hooks

Metaclasses are wonderful constructs. As their name may suggest, they take part in the construction of classes. They even allow us to set hooks into basic Python mechanisms, like isinstance(...), using __subclasshook__.

from abc import ABC

def is_a_duck(duck):
    for attr in ('quack', 'walk'):
        if not hasattr(duck, attr):
            return False
    return True

class DuckChecker(ABC):
    @classmethod
    def __subclasshook__(cls, C):
        if cls is not DuckChecker:
            return NotImplemented
        return is_a_duck(C)

def poke(duck):
    assert isinstance(duck, DuckChecker)
    duck.quack()
    duck.walk()

And we're back in business. That said, is_a_duck is still a stringly-typed mess, and gonna be very painful to maintain.

Wouldn't it be nice if we could just use our IDuck interface to check for duck-ness?

Abstract Methods, Again!

Lucky for us - we can!

Among other things, the ABC parent class enumerates all @abstractmethods and stores them in the __abstractmethods__ member variable. This means that we can easily enumerate them in our subclass hook and check for them.

from abc import ABC, abstractmethod

class IDuck(ABC):
    @abstractmethod
    def quack(self): pass

    @abstractmethod
    def walk(self): pass

    @classmethod
    def __subclasshook__(cls, C):
        if cls is not IDuck:
            return NotImplemented
        for attr in cls.__abstractmethods__:
            if not hasattr(C, attr):
                return False
        return True

class Duck:
    def quack(self): print('Quack!')
    def walk(self): print('Walks like a duck.')

def poke(duck):
    assert isinstance(duck, IDuck)
    duck.quack()
    duck.walk()

poke(Duck())  # Quack!
              # Walks like a duck.

Awesome. Next step - separating the interface from the checking logic.

Protocols

Reading through Python documentation and nomenclature, you might have seen the term "protocol" here and there. It is Python's way to call duck-typed interfaces. So you could say we just created a "protocol checker". Now, we can separate it into a base-class.

from abc import ABC, abstractmethod

class Protocol(ABC):
    _is_protocol = True
    @classmethod
    def __subclasshook__(cls, C):
        if not cls._is_protocol:
            return NotImplemented
        for attr in cls.__abstractmethods__:
            if not hasattr(C, attr):
                return False
        return True

class IDuck(Protocol):
    @abstractmethod
    def quack(self): pass

    @abstractmethod
    def walk(self): pass

And that's it. That little _is_protocol flag is there for good reason. Usually, we'd check protocol-ness using isinstance(...). In this case, however, we're hooking into that mechanism and that would lead to infinite recursion.

We can now use our Protocol base-class freely to create new protocols as we need them, with friendly interface-like syntax. We're almost done.

This Dog is a Duck

In some cases, the protocol checks may not be what we want. The obvious reasons coming to mind are:

We can't really check the desired semantics using the protocol trick.
We want to wreck havoc.

For those cases (well, mostly for the first one) the ABC base class provides another trick. Instead of defining __subclasshook__ to check the interface, we can simple register classes as valid, "virtual subclasses".

from abc import ABC

class IDuck(ABC): pass

class Duck:
    def quack(self): print('Quack!')
    def walk(self): print('Walk like a duck.')

IDuck.register(Duck)

def poke(duck):
    assert isinstance(duck, IDuck)
    duck.quack()
    duck.walk()

Remember that this method puts all the pressure on the programmer. Writing IDuck.register(Dog) is the equivalent of saying "I vouch for this dog to be a duck". It might pass inspection, but won't necessarily yield the desired results.

Summary

In this article we covered multiple ways of checking the "duck-ness" of objects. From belonging to the Anas genus, to just placing a sticker on their head saying "Duck!". Some of those methods are more useful or applicable than others, but I still think it worthwhile to be familiar with all of them. Additionally, there are many topics not covered here, like static type checking.

Working Across Time-Zones

Tamir Bahar — Sat, 09 Jun 2018 16:34:16 +0000

Internationalization is a difficult problem is software-engineering. Usually that statement would refer to the technical aspects of providing a good user experience for your customers. Today, however, I am referring to the social aspects.

Just as it has become common-place to have your users spread across the globe, so it has with developers. It is not uncommon to work with teams in another country, or even to have a specific team-member working from a remote location. Being able to cooperate with developers across the globe is a great enabler. But as the case is with - well - everything, it has some obvious - and some less obvious - pitfalls to avoid.

Where (When?) Are My Colleagues?

First and foremost - know the time-zones your colleagues reside in.

You already know that some of your teammates tend to arrive a bit early or a bit late. You know it, and you adjust to it. You won't make a colleague who starts 2 hours before you stay for a meeting when you know they should be picking up their kids from kindergarten, right?

Well, all you need to do now is adjust to having teammates arriving at work 10 hours before you.

Time For Your 12AM Daily Standup

Yes, that's AM. The one that's confusingly set at midnight.

Now imagine that you're in a UTC-07:00 time-zone (Pacific Daylight Time) and the rest of your team is in a UTC+03:00 time-zone (Israel Daylight Time). Your daily-standup is at 10AM, which sounds reasonable. But that's in Israel. Sadly, this translates to 12AM for you. But you can accommodate that, can't you?

Luckily, in the real-world company HQs tend to be in the USA, not Israel. So you're safe. You can hold your meetings at any time you want, and they'll have to adjust. You can have them wake up early, or stay up late, or just work very awkward shifts. They'll accommodate. But do you really want to make your teammates do that?

Get It Done By EOD Today

No.

It's not that I don't want to, it simply isn't possible.

If you have a large enough time-difference (say, 6 hours difference over a 9 hours work-day) your day starts when your colleagues' day ends (or later!)

No matter how hard they work, they cannot get it done by EOD if EOD has already passed. You can talk about getting things done "by tomorrow", but not EOD. It causes un-needed tension as the need to explain this comes up over and over during planning meetings.

Planned Downtime: 9PM-6AM

Great. You've just killed of a day's work for your distant colleagues. 9PM San-Francisco is 7AM in Tel-Aviv. Making it a 7AM-4PM downtime for them.

I'm not saying you can't do that, but you should try and keep that in mind.

And Then, There's Israel

And on the seventh day God ended His work which He had done, and He rested on the seventh day from all His work which He had done.

Then God blessed the seventh day and sanctified it, because in it He rested from all His work which God had created and made.

Genesis 2:2-3

And so in Israel we rest on the 7th day, and add the 6th in for good measure.

We work Sunday through Thursday, and rest on Friday and Saturday. And this is annoying.

It is annoying when we need support on Sunday and have to wait.

It is annoying when we travel, and are never sure if we should report Friday as overtime, or Sunday as a day off.

And it is annoying to be expected to work on Friday evening just because it's a working-day for you. You wouldn't want to work Saturday nights, would you? I didn't think so.

Also, you should expect email send on Fridays and late Thursdays to be ignored until Sunday.

In Conclusion

This has been a bit of a rant, and a bit of advice as well. I'm not sure which dominated the tone.

This might seem like a minor issue, but it can cause a great deal of pain when ignored.

But in the end, the rules are simple. All you need to do is be aware, and be considerate. You can do it for the people you physically see everyday, and you can do it for your remote colleagues.

Try and set your long-distance meeting at times suitable for both ends. And when that isn't possible - make sure you make an effort as well. Waking up an hour earlier every now and then goes a long way.

Adventures in Rust

Tamir Bahar — Fri, 27 Apr 2018 17:21:57 +0000

In the few years since Rust came out, I've frequently found myself explaining what an amazing language Rust is. Laying out the ways in which it is going to change the systems-programming world, and what a pleasure Rust is to code in.

A few months back, I finally got around to trying it out. It was horrid. The darned borrow-checker just wouldn't let me do anything. I couldn't get anything meaningful to compile. Despite years of programming in various languages I felt completely incompetent to handle it. I resolved to drop Rust and focus on other things.

Despite my initial disappointment and discouragement, I still could not get Rust out of my head. Its premise is too promising to put down after one go. Besides, I cannot be beaten by a programming language! And yet, it can wait. It is too much work for now. Some day, maybe in a few years, I'll write Rust.

Using Rust, however, is a completely different experience. Rust tools are absolutely amazing. As a Windows user through-and-through I got used to open-source tools (especially command line tools) not being supported on Windows. (No, cygwin is not a valid solution.) I don't blame the devs - they work on Linux. Even if they have the time to spend on the Windows port, they don't necessarily have a Windows machine to test it on. And yet - I am used to being disappointed. That is why, when I first heard of rg(an amazing grep replacement) and fd(an amazing find replacement) I knew that they would not work on Windows. But, being my optimistic self - I checked. And a good thing I did that.

To install Rust tools, the easiest way is to install the Rust toolset and compile them. A daunting task in every other language, yet a breeze in Rust.

Head over to rustup.rs and install Rust (A single command-line on Linux, a single executable on Windows)
cargo install ripgrep fd-find
That's it. Really. Now use the tools.

This was when I realized how amazing Rust really is. Even if you ignore the language completely - it's tooling and package management is unparalleled. Having published Python packages in the past, I was amazed at the simple publishing and installation mechanisms. Having used C and C++ I was simply amazed at a systems-programming with package management. So while still somewhat scared of the borrow-checker, I decided that my next CLI tool will be written in Rust. The easy-as-pie deployment bought be over completely.

Some months after that, I finally found myself in need of a new CLI tool. I was faced with a continuous log I wanted to filter for duplicates. sort -u sorts, so it cannot work on streams. Of course, there is probably some sed or awk magic I can use, but I want something simple. Besides, a tool that filters out duplicates seems like the perfect beginner project for getting hands-on with Rust. So I went ahead and creates uq. After finishing it, I published it on crates.io. cargo install uq on a second machine, and it worked. Both Windows and Linux. A friend tried it, and it simply worked! I never had such a pleasant deployment experience. It is practically easier to install from source then send a compiled binary! And it works cross-platform out of the box.

A short while later I wanted to group some log entries by a regex. I looked around and could not find a simple way to do it. So, once again, I turned to Rust. Some borrow-checker-wrestling later and the first version of groupby was complete. Yay!

A short time later I had one of the best open-source experiences I've ever had. Someone started using groupby, looked at my terrible code, and posted this issue:

Little code suggestions #1

vorner posted on Mar 15, 2018

Hello

I find this little program will be useful for many things I do (I usually do something like that with combination of sed, sort, …). I also looked into the code. Do I guess right that you're still learning Rust? Could I provide few little tips?

I glimpsed at least one unwrap that can be triggered by the user input (giving a too large group ID), which will result in ugly error message instead of nice useful one.
Do you choose BTreeMap/BTreeSet for some specific reason? Is it the order of elements? If not, HashMap and HashSet are likely to be faster.
Both variants (unique vs all) look very similar and differ only in the inner data type and the method used to push/insert. I think this could be done with just one piece of code that is generic over the type, and adding your own trait that implements the adding for either. Would you like me to show you such code?

View on GitHub

Having someone more experienced in Rust come in and help me improve my very naïve code was great. And it was my first time getting a "this is great, may I help you?" comment and not a "this is great, I want this as well" one.

For the time being, I keep spending more time wrestling the borrow-checker than writing actual code in Rust. But I am (almost) sure it is due to lack of experience. On the plus-side, I'm becoming better at detecting lifetime issues in other languages as well.

So, for anyone who hasn't done it yet, I highly recommend using Rust-based tools. Just for the amazing experience of things working (and compiling!) out of the box. Later, if you choose to actually code in it, be sure to brace yourself for a somewhat bumpy ride. Friends tell me that after a time Rust becomes easier, speeding up their development. I'm not there yet, but I'm working on it.

Types of Loops

Tamir Bahar — Sat, 27 Jan 2018 14:00:09 +0000

Recently I've been helping & tutoring some true code beginners. Not someone new to a language, but completely new to programming.

I've done a lot of training in the past. Both beginner and advanced training, both programming and reverse-engineering. But as green as my previous students have been, they have always had some prior knowledge, some experience with code. In at least one programming language.

Usually the training is about teaching language features, special tricks, best practices, and getting the trainees familiar with the new patterns. The trainees send out probes, looking for familiar things, and I just fill them in at the right time. They know what they are looking for, or can be easily guided towards the right thing. When people are completely green, they don't.

This is a very new experience for me, and it got me thinking a lot about programming and the ways we approach code. The patterns we seek to find or form. The amazing number of things that we do without even thinking as experience programmers. Each of those, no matter how simple, needs to be broken up and explained to new-comers. They have no previous knowledge to build upon for this.

From my current experience, it seems the understanding the meaning of syntax, and understanding forward-flowing programs is easy enough. Conditionals are a non-issue. The first road-block comes with loops. Especially writing loops. Where do I put the return statement? Where do I define my variables?
Trying to explain those things, and give simple rules, I came to some useful realizations of useful patterns, and some painful truths about our use of jargon.

Let's go ahead and see the patterns.

Find Loops

public static int indexOf(String[] haystack, String needle) {
    for (int i = 0; i < haystack.length; ++i) {
        if (needle.equals(haystack[i])) {
            return i;
        }
    }

    return -1;
}

In those loops we iterate over the array, looking for an item that fulfills our condition. Once we find it, we immediately return that value. There is no need to declare any variables.

Count Loops

public static int countOf(String[] haystack, String needle) {
    int count = 0;
    for (int i = 0; i < haystack.length; ++i) {
        if (needle.equals(haystack[i])) {
            count++;
        }
    }

    return count;
}

In those loops we iterate over the array, looking for items that fulfill our condition. Whenever we find one, we increment the count. Once we exhaust the iteration, we return the counter. The counter and return statement are outside the loop.

Action Loops

public static void printAll(String[] haystack) {
    for (int i = 0; i < haystack.length; ++i) {
        System.out.println(haystack[i]);
    }
}

In those loops we iterate over the array, and perform an action on each and every item. There are no variables and no return statements.

Now, those simple loops can do quite a lot, and can be expanded and composed to do more. And I find that they help beginners. But did you spot my error? I used the word "iterate".

Vocabulary

While the meaning of "iterate" is clear to existing programmers, and looking at the loops you can easily tell that we are iterating or looping over haystack, it is not clear for beginners. Moreover, the words themselves sound weird. This is critical, and becomes more pronounced as you try and loop in slightly more advanced ways

int i = 0;
for (Node current = myList.head; current != null; current = current.getNext(), ++i) {
    // ...
}

Here, we are looping (or iterating) over myList, but we don't change anything about it, or even access it directly. We do change i (which is no longer our counter) and current which is a node. This makes the code and language quite dissonant. "We iterate over myList while maintaining an index" is a true statement, but not an immediate translation from the code. The language forces to go in a very roundabout manner. This is true for many languages.
But now, consider slightly more modern syntax:

for i, node in enumerate(myList):
    # ...

for i, node := range myList {
    // ...
}

for (i, node) in myList.enumerate() {
    // ...
}

In all of those, the situation is far clearer. We can see that we're looping over myList, and it is clear that we have both a node and an index.
While this difference might be minor for experienced programmers, it is a world of difference for newcomers.

Learning to code is not just learning a programming language. Not just learning to think in a specific way. It is learning your own language again. You know English? Well, now you need to learn programming-English. You know Hebrew? Learn programming-Hebrew. We keep changing the meaning of existing words, and expect people to follow and understand them. It is hard. The least we can do is try and minimize the difference between the code we read (programming languages - Java, C, C++, Python, Go, Rust...) and the code we speak (well, I guess English is a programming language as well?).

Write'em Pull Requests

Tamir Bahar — Sun, 19 Nov 2017 16:16:21 +0000

This micro-post is provided as a service to some of my friends, who find it too difficult and time consuming to submit pull requests.

In the past, contributing to open-source was sometimes difficult. You had to contact the maintainers, email them patches, and all sorts of nasty things.
Then, circa 2008, GitHub came along. With GitHub you clone the repo locally, make your changes, push to your own fork, and create a PR. Simple enough.
If you wish to avoid the CLI, though, GitHub allows you to go a step further.

Say you are using my GraphGrabber plugin for IDA Pro, and find that it fails in IDA 7.0. The Qt layout is different now, and line 51 needs to be changed

diff --git a/graphgrabber.py b/graphgrabber.py                            
index 8c301b7..5d8191c 100644                                             
--- a/graphgrabber.py                                                     
+++ b/graphgrabber.py                                                     
@@ -48,7 +48,12 @@ def graph_zoom_fit():                                  


 def grab_graph():                                                        
-    widget = sark.qt.get_widget('IDA View-A').children()[0].children()[0]
+    widget = sark.qt.get_widget('IDA View-A').children()[0]              
+    try:                                                                 
+        widget = widget.children()[0]                                    
+    except IndexError:                                                   
+        pass                                                             
+                                                                         
     width = widget.width()                                               
     height = widget.height()

All you have to do now, is click the Fork this project and edit the file button

Edit the file in-place

And propose the change

That's it. You're Done!

Now, sure, there are better ways to do this. Details on reproduction of the error would be nice. And when it is not a bugfix, some further discussion and decisions may be required. But this is a start. And once something is started, it is often easier to follow through then without starting it.