Forem: topjer

My colleague did overtime for two weeks straight, here is what I told him

topjer — Mon, 11 Aug 2025 19:47:26 +0000

My vacation stand-in told me, he had to do overtime for two weeks straight in order to handle the workload and today I want to share with you how I reacted. But let's start at the beginning.

How did it come to this?

It all happened in the project team I am currently leading.

My vacation plans for this year were overlapping with the other senior developer in that project by two whole weeks. Which is bad! Because we both would usually be the stand in for the other. The only way this would be feasible is when we both found someone willing to keep the ball rolling while we are gone.

A not-yet senior developer in the team, we both regard highly, came to mind and since he plans to join the senior ranks, we thought it would be great opportunity for him to see what it could be like.

The idea was that we would prepare tasks in advance to be distributed to the other team members. This way, he could focus on supporting them.

Unfortunately, it did not work out as planned. Even shortly before our vacations, it was not clear whether we could get the most important requirements to a workable state. This is due to general difficulties in the project, but this is an entirely different story.

After all, we managed to formulate a plan and have some tasks ready for the upcoming two weeks. So, we took our well-deserved leave, even though we were feeling some remorse.

How did it go

We rightfully placed our trust in him. The team might not have progressed slower than usual, but at least no one was twiddling their thumbs during the two weeks - which was the scenario we were afraid off the most.

Once I acclimated myself back at work, I managed two have a one on one chat with the brave stand-in, so here is what he told me.

In general, he did enjoy the new challenges greatly. He never had to coordinate a team before, but doing so felt very rewarding. His first priority was to support the rest of the team whenever possible. Mostly in the form of requirement clarification and code reviews.

This would come at a cost though.

All his time was spent while making sure everyone else is able to work. Yet, he did not want to disappoint us, so after a day full of meetings and support, he would do overtime in the evening, where he worked on his own tasks.

This was going on for the entirety of the two weeks.

My reaction

My first reaction was to thank him for this dedication and effort, I was thankful for him to go the extra mile.

But, I was not happy, because it seemed unnecessary to me, so I told him he does not have to do that the next time.

If he had not gone this extra mile and told me that his entire day would have been spend with supporting his team members, leaving no time to work on anything himself, then I would have understood that.

Together with him, we would have discussed strategies on how to move his task forward, so we can make the target date.

Now, you might be wondering why I make a big fuzz out of some overtime. A bit of crunch is naturally part of the job, right?

To understand my aversion against overtime we have to look at ...

The bigger picture

The first thing you need to understand is, that the project we are talking about is somewhat ... problematic.

It started out chaotic, with no clear target picture and a lack of strong leadership. Which was the reason why I joined it and I was able to create some structure and got us moving forward. With the initial hurdle cleared, it soon became clear that it is going to be a tight one.

With the months passing, the timeline becomes only more unrealistic because we are slowly unveiling what is actually expected from us.

This situation creates massive pressure on our developers. Since they are still inexperienced, they do not know how to handle it. Which often means that they to compensate for the delay.

But, individual contributors should never feel responsible for the state of a delayed project.

It is not their fault and it is not in their power to fix it. The reasons for a delayed project rarely lie in a lack of dedication of the team. It is more likely due to bad planning, too many parallel tasks, communication issues between stakeholder etc.

Instead what they should do, is to make this transparent. They should voice their concerns with a dead line that is too tight. Then we can search for ways to solve the issue as a team.

Closing thoughts

Going back to the initial story, it hopefully is clearer that my reaction was the way it was, because it was the latest example of a developer doing more work than necessary. Something that is rarely seen nor appreciated.

Have you experienced similar situations in the past? If so, I would really like to know how you have handled it.

Want to be informed every time I publish something new? Then subscribe to my substack

Introduction to monkey patching with pytest

topjer — Mon, 16 Jun 2025 19:14:10 +0000

Not too long ago, I had a pair-programming session with a colleague where we understood how monkey-patching works. A technique both of us had not really used before.

In this article I want to share our findings. But I will not simply present you the final solution. Instead we will retrace the steps and incrementally improve our code. Hopefully, this will not only teach you how to use monkey-patching but also how to "work with" error messages.

The Problem

Assume we want to test a function like this:

# utils.py
from pathlib import Path

def read_file(name: str) -> int:
    complete_name = name + "_foo"
    name_path = Path(complete_name)
    with name_path.open('r') as file:
        content = file.read()

    return len(content)

The elephant in the room

The mentioned function comes with a big problem and that is, that it opens the file itself and thus performs interaction with the file system.
This makes testing much harder than it needs to be. Because we want to test the logic around it and not that Path.open is working properly. Yet in order to do so, we have to deal with it.

Now, one could go ahead and have the function actually read a file, but this leads to follow up problems like:

Will the file be cleaned up after the test?
Will it also be cleaned up if something fails?
What if there is already a file with the needed name?

As mentioned before, we do not actually want to test Path.open so why not take it out of the equation completely?

About monkeypatching

Definition of monkeypatching according to wikipedia

In computer programming, monkey patching is a technique used to dynamically update the behavior of a piece of code at run-time.

We will see how to use monkeypatching to replace functionality with mock objects that make testing easier.

First step

Let us assume the following folder structure

mp_test/
    utils.py
    testing.py

where utils.py contains the code from above. As a first step, lets try to do this:

# testing.py
from mp_test.utils import read_file

def test_read_file():
    value = read_file('test_file')

Running this test will give us the expected error

FAILED testing.py::test_read_file - FileNotFoundError: [Errno 2] No such file or directory: 'test_file_foo'

Based on the documentation on the monkeypatch fixture lets try something like this:

# testing.py
from mp_test.utils import read_file
import pathlib

def test_read_file(monkeypatch):
    monkeypatch.setattr(pathlib, "Path", lambda x: print(x))
    value = read_file('test_file')

What did happen here?

First of all we are using the monkeypatch fixture and use it to replace how Path is working in pathlib. For starters, I just want it to print the name of the path that was given to it.

But there is a problem, when we execute this code, the same error like before does occur.

# pytest output
FAILED testing.py::test_read_file - FileNotFoundError: [Errno 2] No such file or directory: 'test_file_foo'

So what went wrong here? Did we do something wrong? Is monkey-patching just a scheme which does not really work?

Turns out, the problem is what (or better where) we have patched. To understand that, we should take a look at this part of the unittest.mock documentation which is linked in the pytest documentation:

patch() works by (temporarily) changing the object that a name points to with another one. There can be many names pointing to any individual object, so for patching to work you must ensure that you patch the name used by the system under test.

The basic principle is that you patch where an object is looked up, which is not necessarily the same place as where it is defined.

If we look again at the content of utils.py we see that Path is also imported there. So in order for the patch to work, you have to also patch it there.

# testing.py
from mp_test.utils import read_file
import mp_test

def test_read_file(monkeypatch):
    monkeypatch.setattr(mp_test.utils, "Path", lambda x: print(x))
    value = read_file('test_file')

This code leads to some progress ... a different error message.

# pytest output
--------------------- Captured stdout call ------------------
test_file_foo
========== short test summary info ================
FAILED testing.py::test_read_file - AttributeError: 'NoneType' object has no attribute 'open'

This error is also to be expected because we replaced Path with the print function. That is the reason why we can see the file path in the output.
A consequence is that name_path is set to None because that is the return value of print. This leads to the above error, when the code tries to call the open function of name_path.

Now that we know how to replace functions, we can go to the next step.

Cooking up mock objects

Our goal is to create a MockPath class which replaces the real Path and this mock should just return the path name when the open method is invoked.
So let's try something like this:

# testing.py
import mp_test
from mp_test.utils import read_file

class MockPath:
    def __init__(self, path: str) -> None:
        self.path = path

    def open(self, type) -> str:
        return self.path

def test_read_file(monkeypatch):
    monkeypatch.setattr(mp_test.utils, "Path", MockPath)
    value = read_file('test_file')

This is a step in the right direction but unfortunately it gives us a - somewhat - cryptic error:

FAILED mp_test/testing.py::test_read_file - TypeError: 'str' object does not support the context manager protocol

But a closer look at utils.py reveals the problem.

The output of Path.open is used as a context manager which has to provide the read method.

This means that we need another mock which has the methods __enter__ and __exit__ in order to define a context and a method read which returns the filename. So we will end up with something like this:

# testing.py
import mp_test
from mp_test.utils import read_file

class MockReader:
    def __init__(self, content: str) -> None:
        self.content = content

    def read(self) -> str:
        return self.content

    def __enter__(self):
        return self

    def __exit__(self, *args):
        pass

class MockPath:
    def __init__(self, path: str) -> None:
        self.path = path

    def open(self, type) -> MockReader:
        return MockReader(self.path)

def test_read_file(monkeypatch):
    monkeypatch.setattr(mp_test.utils, "Path", MockPath)
    value = read_file('test_file')
    assert value == 13

We have added the MockReader class which can be initialized with some content and when calling the open method, it returns said content. Upon entering, it returns itself and it does nothing upon exit.

With these mocks in place, our test actually succeeds.

Finishing touches

There is actually - at least - one problem with the test setup. Can you spot it?

The way it is approached now, we are not checking whether Path is invoked with the correct path. So we might end up in a situation where not the right file is loaded.

In order to test that, we need a way to see with which arguments MockPath has been initialized. One possible way to do that would be to use class variables which could look like this:

# testing.py
class MockPath:
    paths: list[str] = []

    def __init__(self, path: str) -> None:
        self.path = path
        self.paths.append(path)

    def open(self, type) -> MockReader:
        return MockReader(self.path)

def test_read_file(monkeypatch):
    monkeypatch.setattr(mp_test.utils, "Path", MockPath)
    value = read_file('test_file')
    assert value == 13
    assert 'test_file_foo' == MockPath.paths[0]

The shortcomings of the example

Let me be honest with you. Instead of writing unit tests like this, I would not have written create_file like this to begin with. Instead, I took it, because it is an easy to understand example.

Usually, I try to avoid having functions that mix logic - determining the name, processing the content - with the code that gets data from an interface. Instead, I like putting all logic into isolated functions that take "primitive" Python data types as input. This makes testing really easy.

But there is a middle ground between those two extremes and that would be using abstraction.

For example, one could do the following thing in utils.py

# utils.py
from pathlib import Path

class Reader:
    def __init__(self, path, mode):
        self.path = path
        self.mode = mode

    def __enter__(self):
        self.stream = Path(self.path).open(self.mode)
        return self.stream

    def __exit__(self):
        self.stream.close()

class ReaderContext:
    def __init__(self, path: str):
        self.path = Path(path)

    def open(self, mode):
        return Reader(self.path, mode)


def read_file(name: str, context):
    complete_name = name + "_foo"
    ctx = context(complete_name)
    with ctx.open('r') as file:
        content = file.read()

    return len(content)

Some things have changed here. Let's take a closer look.

The biggest change is that we now have two classes Reader and ReaderContext.

The naming is questionable, I know, please ignore that for now and focus on what is happening. The ReaderContext is just a wrapper around the Readerclass which is again a wrapper around Path.open. Also note that the signature of the read_file function has changed. A context is now passed in which is called.

All in all, this should seem familiar to you because it is very similar to the setup of our Mock objects.
These objects already where an abstraction of the functionality that we need to execute read_file. In the code above we are only wrapping the Path.open method in this abstraction.

Now we can also simplify our test, because we no longer need to mock anything. Instead we can pass our mock classes directly to read_file:

# testing.py
def test_read_file():
    value = read_file('test_file', MockPath)
    assert value == 13
    assert 'test_file_foo' == MockPath.paths[0]

One last comment, the code could probably be simplified to the point where only one class is needed instead of two, but I intentionally mirrored what was already there in the test. So it is easier to understand. Feel free to simplify the code yourself as an exercise.

Next steps

There is one benefit that the more complex implementation brings with it and that is the fact that we are now in a position where we could write other Reader and ReaderContext classes that can handle different data sources.

It would be cleaner to define abstract base classes for both and derive every other class from those. But this is outside of the scope of what I wanted to achieve with this article, so I will leave that for another time.

Conclusion

I hope that you found this short introduction to monkey-patching in your tests helpful. The focus was to touch upon different concepts without going too deep on any on them.

Feedback is always appreciated and let me know whether you would like to read more about certain topics in the future.

Level up your pip install skills

topjer — Mon, 12 Aug 2024 07:08:15 +0000

In this article we are looking at different ways to install code into your virtual environment with pip.

These will increase in complexity, but fret not, I am there for you every step of the way. pats your back

Enough talk! Let's start with something easy.

Installing a local repository

Assume the following situation: You just checked out a repository and want to install the requirements.

This can easily be done by using the following command ... of course after you have created a virtual environment:

$ python -m venv (name of virtual environment)
$ source (name of virtual environment)/bin/activate
$ pip install .

If you are wondering about the install command and the absence of a requirements.txt, I have bad news for you. It is 2024 and you should no longer use a requirements.txt.

This is of course only my own opinion, but all repositories I am working with have a pyproject.toml and I would strongly recommend to use one in every of your projects as well. Viable exception might be sandbox projects and small scripts.

The "why" would be misplaced here, but allow me to give you a sneak peek. Not only does it allow you to define your requirements themselves. You can also define optional dependencies the user can install if needed.

This is especially useful for development tools, that you do not want in the productive application, like testing libraries or formatters.

But this is only the beginning of the list of features. They are also a place for metadata and allow for custom entry points to your application.

Here the install command again:

$ pip install .

Make sure that you are in the folder where the pyproject.toml can be found.
Here a pro-tip, use the following command if you intend to work on the repository you are installing

$ pip install -e .

This performs an editable install also known as "Development Mode" which allows you to iteratively test your code changes without the need to reinstall your project.

What does that mean?

Did it ever happen to you that you changed code in a module you are importing from, but the changes do not seem to take effect?

Try editable installs!

Interlude: Installing from a branch

Quick question: What do you do, if you want to install the version of a specific branch instead of the default branch?

The answer is obvious

$ git checkout (branch name)

and repeat the steps above! Right?

You fool, you just activated my trap card!

See, ever since version 2.23 there is a new kid in town which allows for more intuitive switching of branches and its name is git switch.

git checkout has been deprecated ever since.

So do not expose yourself in front of you colleagues by using out-dated tools. Instead, casually drop a git switch next time you are sharing your screen to let everyone know that you mean business.

Install from a private repository

Brace yourself!

Everything up to this point was mere child's play. Now it is time for some big boy pip usage.

See, everyone can install packages available in a package repository, but only knowing how to install from there will mean that all the gold in private repositories will remain inaccessible to you.

It is also helpful to test your own code before turning it into a package.

If you ever find yourself in such a situation, use this command:

$ pip install git+ssh://git@(your provider)/(owner)/(repo name).git

Here an example without the placeholders, that might make it easier to understand.

$ pip install git+ssh://git@github.com/pandas-dev/pandas.git

Fun fact: everything after the '://' is almost identical to the ssh command generated by git. But notice, instead of the colon used to separate 'github.com' and the owner 'pandas-dev' a slash has to be replaced.

What if you want to install from a branch ... or any other reference for that matter?

Easy! Just add a @(ref) at the end of the command. So it can look something like

$ pip install git+ssh://git@github.com/pandas-dev/pandas.git@1.5.x (branch)
$ pip install git+ssh://git@github.com/pandas-dev/pandas.git@v2.2.2 (tag)

Private Repos and the pyproject.toml

But what if it is not enough to install packages from the command line? What if also your build pipeline should install from a private repository?

Hopefully, you agree that adding individual pip install statements to your pipeline is out of the question.

So instead, let me show you what to add to the dependencies section of the pyproject.toml. You will see, that it is very similar to the previous command:

"pandas@git+ssh://git@github.com/pandas-dev/pandas.git@1.5.x",

With this added, run again pip install -e ..

Congrats! You have just installed an outdated version of pandas into your environment. You might want to repeat that with the actual package you need.

Now make it fast

Since you stuck with me until this point, I will throw in a bonus tool recommendation.

These past couple of months I have used uv, which is a drop-in replacement for pip (among other often used tools in the python eco system) written in Rust.

The biggest selling point is that it significantly speeds up the creation of virtual environments and the installation of packages. Especially if you are recreating virtual environments, because it uses caching. We are talking about being 10 times faster ... or even 100 times faster if the cache is warm.

The list of benefits is much longer than that, but also this is a thing for another article. So, give it a try for now and thank me later.

Conclusion

Let's wrap this thing up.

These were all the ways I have used during my work when it comes to the interplay between git and pip. There might be other ways to install stuff, but this should cover 99% of use-cases.

Have I forgotten your favorite command line trick? Then share it in the comments.

I hope you learned something new with this article and if you are interested in more technical articles about Software Development, then consider a follow.

Why you should care about Poetry

topjer — Mon, 26 Feb 2024 21:52:18 +0000

Let me be frank. Python dependency management with pip kinda is a pain.

I never found it too outstanding, but once I have experienced dependency management with ... (make sure the inner voice reading you this text takes on a loving tone for the next word) Cargo ... (back to normal) something had to change.

Before we talk about this change, let's first review what had to change.

Forgetting dependencies

The following situation happens way to often to me.

I check out a repository from the company github, navigate to the file containing dependencies, let pip do its magic. Only to face an error message indicating a missing dependency once I try to run the code.

How the hell could that happen?

But all is not yet lost. I see what is missing, right? The solution is to simply pip install the missing library, right? Right?

Wrong!

See, what I just installed was the latest version of said dependency. But the code was written based on an older version of said dependency that saw several API breaking changes in the meantime.

Driven by anger and frustration, I open up the commit history of the repository to see who pushed the last changes ... only to find that it was me...

Activating the virtual environment

Attentive readers might have noticed that some steps were missing in the last section.

Of course, you should create a virtual environment in which you install all requirements. Please, do not install everything globally, as that would be the direct path into dependency hell.

In the off-chance that you are unfamiliar with virtual environments, they are - as the name would imply - a dedicated Python environment, that neither interferes with your global environment nor with other virtual environments.

They are super useful and should be a trusted tool for all Python developers.

But their isolating super powers are worth nothing if you forget to actually activate it. Something that can quickly happen if you are in a hurry and especially if your shell does not indicate activated environments. Trust me, happened to all of us!

How Poetry alleviates these issues

For installing dependencies, you can use the command poetry add {name of dependency}

Like pip, this installs the dependency. But unlike pip, it automatically installs it into a dedicated virtual environment for the current project. The goodness does not stop there. On top of it, it also adds the dependency to your pyproject.toml.

You can even constraint the allowed versions of the dependency with powerful tools.

This can be achieved with
poetry add {name of dependency}@{constraint}

In order to enable full repeatability of builds, it creates and maintains a lock file that contains the exact version numbers of your current configuration.

The section up to now already contains a hint at how it solve the second problem: By fully automating the handling of virtual environments. No longer do you need to activate it. You don't even have to remember to create it. Poetry does all of that for you.

Poetry keeps track which environment was used for which project and uses it whenever necessary.

The best thing about it, you can even have multiple environments for the same project. Gone are the days where you have to ask yourself: "Was env1 the environment with Python 3.9 or was it env2?"

Thus, in the end, all you need to do is type poetry run python {file name}.

There is also no need to create an environment for a repository you just checked out. Simply use poetry install and all the needed requirements are being installed into an environment that is also being created for you.

Conclusion

I do not want to drag this one out any further as this article was intended to be an appetizer.

By now you are armed with some basic commands and hopefully some curiosity to try it out for yourself. So head on over to the installation page and you can be the proud owner of your very own copy of poetry.

I hope you learned something in this article! Feedback is always appreciated and see you in the next one!

Don't make these mistakes in your CV

topjer — Mon, 21 Nov 2022 11:33:21 +0000

Ah yes, another year is slowly coming to an end. This means that project plans are nearing fruition/failure. Maybe people start to feel the call for change, thinking to themselves that 2023 will be the year when they leave the hell hole they wander daily.

Or maybe their company is bought by a eccentric billionaire who decides to lay of half of his software engineering team based on the lines of code written in the last year - sick metric by the way!

Whatever the reason might be, we get a ton of applications at my company. Since I am one of the most senior member of my team I get the honor of screening all those CVs together with my boss. A process that turned out to be surprisingly painful.

So I thought I share some of that with you. Mostly to entertain. But who knows? Maybe there will even be an insight in here.

The Context

Allow me to give you some context.

My Company is based in Germany. So some things I complain about might not be relevant in other countries like the US.

Also, the position we are trying to fill was the one of a Senior Python Developer. So, we wanted to quickly identify people that can show relevant experience and invite them to an interview.

As it turns out, this is not that easy.

Brevity Is The Soul of Wit

Let me start by saying this: I don't like reading applications! It keeps me from doing my actual work.

There is clearly the need for it. Our workload is increasing and we can always use some fresh meat. Yet, I get mildly irritated when I have to read a CV for more than 5 minutes.

Again, all I need is the faint impressions that you have done some Python in the last 3 years. BAM Your are in the next round!

A CV is not there to get you hired. It is there to get you invited to an interview.

Yet, what I was getting were applications that meticulously wrote about the projects they have been involved in. Pages over pages. In the worst cases it was not even apparent what their part has been in all of that.

Some I topped that, because it was not already enough to read. they Also added an evaluation from their current employer. Two pages of prose about what a great employee they are written in font size 8. At least this is what I suppose it said because I did not bother reading it.

You want to seem professional? Then respect the time of whoever is going to read you application.

The Other End of the Spectrum

There were also CVs that were drastically different.

One guy, fresh from university applied to the position and sent a one page CV that only contained the information that he just recently finished his Bachelor Thesis.

The infuriating thing here is not that he applied to a Senior Positions with hardly any work experience. This could have been due to the fact that our Recruiter actually was the first to approach. No, it was infuriating because there was nothing the read from this CV.

"If the CVs are long it is bad but if they are short it is just as bad. Seems like this guy is never happy" might be what you are now thinking. So allow me to elaborate what I was looking for in a CV.

Your Past Experiences

If you want to apply for developer position using language X then make it clear why you are the right one for it based on your past experiences. Highlight the problems you solved with X, which libraries you have used etc.

Also, be more detailed in recent jobs and become more coarse with jobs that are longer in the past. It is not really convincing when the last relevant job experience was 15 years ago.

This is even applicable if you are fresh from university. You might not have any work experience but there hopefully are some coding projects you can name where you have used X.

Now that we have established what I would like to see, let's continue with some things I did not like to see.

Tell Me Something About You

Here in Germany it is common practice to end you CV with some words about yourself.

By that I literally mean that a couple of activities are listed, people do in their free time. I assume the idea here is to give an impression of the person behind the CV.

Yet it seems rather futile to compress something as complex as a humans nature into 5 nouns.

Well, this CV does not contain any relevant work experience but the fact that she likes to perform traditional polish dance is intriguing. We should definitely give her a chance.

And please, for gods sake. If you are thinking about adding 'Watching Netflix' to it ...

Don't!

Really, I mean it.

Which scenario are you imagining where this would make you more attractive as an employee?

the screen gets blurry and the next thing we see is the Head of a software engineering team lost in deep contemplation

"I'll be damned. Before me sits the best team of software engineers the world has ever seen. Each one of them has their own area of expertise which they complement with a broad overview other topics.

They are living and breathing software development. Each waking hour is spent either writing, reading or thinking about code.

Heck, some of them even learned lucid dreaming so they can keep solving engineering problems while they dream.

But not a single one of them can give me a recommendation what to watch on Netflix. I am getting sick and tired of rewatching The IT Crowd."

He is torn from his thoughts by the sound of trumpets echoing from above. Confused as to what could be the origin of these otherworldly melodies, he approaches the window.

What he sees is that the clouds are parting. His eyes squint as he adjusts to the heavenly glow. Finally he can make out a shape. It is the hand of the creator of all things reaching out for him.

To an outside observer this scene would be reminiscent to "The creation of Adam".

But unlike in the picture, the hand is holding something. Something divine. Something to end his monotonous binges.

It is your CV.

Snap, back to reality.

If this is the scenario you are hoping for, then good luck with that.

I Really Want to Work at {insert name here}

Not sure whether this is a thing outside of Germany. But here we usually get a letter alongside the CV that allows the applicant to give a more detailed insight into reasoning for the application. (This is changing somewhat, yet often times we still get them)

Most letters I have seen share two qualities.

On the one hand, they are a joy to read. They eloquently explain the applicants motivation. Why your company is the ideal workplace and how thrilled they would be to get a chance to work alongside you.

Sometimes these letters are so poignant that I am filled with a deep gratitude. Because I am actually working for this wonderful company.

If only it weren't for the second quality: Being written so generically that it could fit any company out there.

I do understand that it is unrealistic to write a unique letter for each company you are applying to. Still, if you decide to add a cover letter, then make at least one paragraph unique to the company. Refer to something found in the job description for example.

Yet, even better - although it will put a tear to the eyes of all those cover letter ghost writers out there - do not bother with it.

The Worst is yet to come

You know what I really despise about reading CVs?

It is when people are judging their skill set themselves. There even seem to be websites out there that create these overviews for you.

Just choose the technologies you are familiar with and then indicate your skill level on a scale from 0 to whatever.

Aside from the fact that they sometimes are so horribly formatted (font and color scheme) that I dread just looking at them. They are also absolute BS.

There are probably enough studies out there indicating that we are not capable of objectively judging our skills. With a clear tendency towards overestimating the own proficiency.

Thus these overviews are only a groundless claim. Worthless without backup.

If, on the other hand, I see that they have been working as a Python developer for the past five years. Explicitly naming - some of - the projects they have been working on, together with the libraries / technologies used. It becomes a more reliable basis to judge the skill.

Bottom line: Ditch these overviews. They are just noise (and make my eyes bleed).

Conclusion

In case you had as little patience with this article as I was having with some CVs and skipped right to the end, I want to repeat the main point.

Past experience is key.

List all your past experiences that are relevant for the position you want to apply to. Be succinct and precise and cut out all the noise.

The interviews are the place where you can shine with your unique personality (and your Netflix recommendations).

Let me know in the comment section how you approach the topic of a CV or maybe you are in a similar position and have read some CVs in the past. Then feel free to share your opinion as well.

I saved money, time (and the environment) with Python

topjer — Tue, 01 Nov 2022 21:24:45 +0000

Introduction

Admittedly, the title of this article is ambitious. So allow me to explain.

Not too long ago, I have started a subscription for some workout program. My plan was to watch the program on my phone while being in the gym.

It soon turned out that this program was merely a collection of YouTube videos. The ones you can only access when you have the link.
Paying 50+ Euro for that every month, by the way.

The scrooge in me came up with an idea.

"Why not download these videos and cancel the subscription?"

So, I downloaded an App to my phone and started the dirty work. But here comes the catch: Since I did not want to spend money for this App, I went with a free version.

This came with its own set of problems. It forced me to watch a ton of adds. Adds when I downloaded a video. Adds when I wanted to watch a video.

Now, the programmer in me came up with an idea.

"Should be possible to download these videos with some Python code."

And so it began.

Let's talk about morale

I won't hide the fact that I feel a tiny bit of guilt for downloading the videos.

My justification is that I think that the product provided does not work well as a subscription model.

It just feels wrong to pay 50 bucks every month to access an existing library of YouTube videos with a handful of new additions every month. YouTube videos which you could probably still access without the subscription if you store the IDs somewhere.

In a desperate attempt to divert the blame, I would even say that it is their own fault for making a product that you can simply download.
It's like leaving your valuables lying around in your front yard.

This diversion works only as long as no one asks why I did not just search a different product if this one is not worth the money for me. To that I can only answer:

Because I like to eat my cake and keep it!

With all that being said and nothing gained, dearest reader, just forget about all of this and get distracted by the shiny things in the rest of this article.

Back to the title

Let's address the points in the title individually.

It should be obvious how I have saved money. I no longer had to pay for the subscription.

This way I would miss out on the new content, but I could come back after a few months, download the new stuff and quit again.

Because, I really know no shame.

Also, I have a limited amount of mobile data. Since I do not have to upgrade that in order to watch videos I save again money. Admittedly a bit far fetched, I know.

How was time saved?

Well, since I was watching these videos on my phone, I would have to endure Ads. Either in YouTube or in the App used for downloading. Not anymore, Sir!

Time lost with buffering issues will also be saved on top of that.

But how in the hell did I save the environment?

My thought process on this is simple: There are videos I like which I henceforth watch multiple times. If I were to
stream that video multiple times, servers would have to provide that to me and the transmission infrastructure would have to transport that to me.

This requires electricity and leaves a carbon foot print. If it is avoided, the environment is saved (a tiny bit).

With a high likelihood my thought process is too simplistic and studies are a bit vague about the actual impact streaming has on the environment.

So ... did I mention the environment in the title in order to get more attention? The answer to that is a clear ... maybe. But since we are already here, let's talk about ...

The code

Three major problems had to be solved:

How can I access the website of the product?
How can I extract the video IDs from said website?
How can I download these videos?

To sum it up quickly:

The first problem was solved with the requests package and providing the active session credentials as cookies and headers.
The second problem was solved with the BeautifulSoup4 package to perform some - crude - html parsing.
The last problem was solved with pytube which makes downloading YouTube videos a breeze.

In hopes that you are not some kind of coding savant that already connected all the dots, you will find the details in the following sections.

How To Get In

This is not my first rodeo, I have scraped websites before. This usually did the trick for me:

import requests

awesome = requests.get('https://topjer.net')

Firing the piece of code usually resulted in a return code of 200 - huge success - and full access to the code of the website.

Yet the website required authentication. Instead of a shiny 200, I got a gloomy 403. So I did what every developer would do ... I went over to stackoverflow.

Io and behold, I found this.

The solution I ended up using was to navigate to the network tab in the developer tools of my browser after logging in and extracting the request as a cURL. This can be copied to https://curlconverter.com/ and it will create the python code needed to access the site.

Now the code looks something like this, with COOKIES and HEADERS taken straight from the before mentioned website.

response = requests.get(CURRENT_URL + '/index', cookies=COOKIES, headers=HEADERS)

These are the solution I like. They work first try and don't require me to understand what the hell is even going on.

Am I concerned because I have entered potentially sensitive information into some random website?

No, you dummy! Because the Privacy Note on the website promises that everything happens in my browser and no information is sent anywhere.

Why would I be distrustful if a site promises me my data is safe?

The extraction

On to the next step. What we have right now is the pure html code, so full of pointy '>' and '<' that I am afraid I might hurt myself if I stare at it for too long.

What to do? Like in real life: if you take something dangerous and cook it for some time, it becomes less dangerous ... and maybe delicious in the process.

Terrible analogy, but all for the sake of transitioning to the Python library I have been using: BeautifulSoup.

According to the website:

Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.

My highly sophisticated workflow looks like this:

Load the website in my browser.
Right click the object I am interested in and choose 'Inspect'.
With the element highlighted in the html code of the website, I try to identify parents with hopefully unique tags to make my way down to the element.
Rinse and repeat if necessary.

Here is the structure I had to navigate in my case:

The starting page had several drop down menus on it. The third one contained entries that would lead me to the site with the videos I wanted.

So here is what I had to do:

Find all drop down menus and get the content of the third one.
Extract all links from said drop down menu.
Loop over the list of links and in every instance find the embedded YouTube video and extract the ID.

Lets have a look at some code snippets. Note that this is not the actual code I have used. It merely is intended to show the functionalities I have been working with.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html_mess, "html.parser")
dropdown_menus = soup.find('div', class_='col-xs-12').findAll('li', class_='dropdown')

movement_menu = dropdown_menus[2]

entries = movement_menu.findAll(lambda tag: tag.name == 'li' and tag.has_key('data-post_id'))

for entry in entries:
    video_source_id_raw = entry.contents[0]['data-video_source_id']

Also, since this is not an in depth guide about BeautifulSoup, I will just briefly explain what the purpose of my code is. I claim by no means that it is the most efficient way of doing it.

The first step is to parse the html string, which in my case comes from 'response.content'.

In order to find the tags I am looking for I mainly use the 'find' and 'findAll' functions. Mind blowing, I know.
If only the first tag that matches a description is needed, use 'find'. If all instances are of interest, use 'findAll' (or 'find_all').

The first argument you must provide is the type of the tag, i.e. the first thing that comes after the '<' and then whichever combination of attributes is fitting.

In my case it was sufficient to look for the class attribute of the tag.

But it is also possible to provide a function that returns True or False. In my case I use lambda functions which check for the tag to be 'li' and whether the tag has an attribute 'data-post_id'.

For me personally, it was easier to work with lambda functions and I also assume that they give a greater flexibility. But I might as well be wrong with that statement...

Note that the result of the find function is again a Soup object, so multiple consecutive searches can be executed.

In case you want to go through the list of direct children of the tag, i.e. all top-level tags within said tag, use the 'contents' attribute.

If you want to access the value of a given attribute, then you can do that as if the soup object were a dictionary, as I do it in the last line.

And with this, I want to conclude the section about how I extracted the video IDs. If you want to know more about html parsing with BeautifulSoup, then check out the tutorial, which is excellent. Or let me know in the comment section and I might go more 'in depth' on it.

Securing the Goods

After all this heavy lifting is done, we can finally get to the interesting part. Downloading the videos themselves. A task that is pretty easy due to the hard work of other people.

There is this package called 'Pytube' which takes care of everything. Here is all the code you need:

from pytube import YouTube

video = YouTube('https://www.youtube.com/watch?v=' + video_id)
stream = video.streams.filter(file_extension='mp4', resolution='720p', progressive=True).first()
title = video.title.replace(' ', '_') + '.mp4'
file_path = stream.download(filename=title, output_path='./downloads/')

First, you 'connect' to the video by supplying its url.

Then you can get the list of all available streams via 'video.stream'.

This list can be filtered by varying attributes like the file extension or resolution.

One comment to the attribute 'progressive=True'. If you check a video, you will notice that most have 'progressive=False'. Which means that for most streams audio and video are separated. In that case you have to download video and audio separately and combine them afterwards.

Because I am lazy, I hope for a stream that has 'progressive=True'.

Then I choose the first result I get, remove those pesky spaces from the title and proceed to downloading the video.

Sadly, by default, you do not get much feedback on the current progress of the download. The code simply halts execution until the download is done.
It seems like you could provide a callback function upon progress of the download and thus get some insight on how much is left to download. But I have not looked into that much.

Added bonus: Download music

Since we are here, I might as well mention that slight modifications of the above code will allow you to only download the audio track of a video. Ideal if you want to download some music.

video = YouTube('https://www.youtube.com/watch?v=' + video_id)
title = video.title.replace(' ', '_') + '.webm'
file_path = video.streams.filter(mime_type="audio/webm").last().download(filename=title)

All you need to do is change the filter. In my examples it was most reliable to check for the 'mime_type' and since mp3 was not always available, I used 'webm'.

Conclusion

Now you should have all the pieces to glue together your own tool to extract youtube video ids from a website and then download them.

I'd love to hear your feedback in the comment section below.

Arbitrage - this is not a Game

topjer — Wed, 22 Jun 2022 15:28:31 +0000

This article will describe how I created and implemented a trading model for a game I have recently started playing. Even though it is "just a game" and therefore "not real". It is still a great sandbox to play around with, to explore concepts and to improve coding skills.

I will begin with describing the market conditions and drawing conclusions from then. The next step will be to dissect the first model I have created and lastly I will describe all the flaws and how to improve.

It All Started With a Game

Some months ago I have started to play Final Fantasy 14, an massively multiplayer online role playing game (MMO) set in the popular Final Fantasy universe.
As is usually the case for MMOs, this game has an auction house where you can buy and sell in-game items for in-game currency.

Whilst this is probably only a side feature for most people, it can be a great source of virtual income if played right.

The Market

The Facts

Let us first gather the hard facts of the market as they set the boundaries to the model we do need.

The in-game currency is called 'Gil'.
The markets are called 'Market Boards'.
The game has so called 'Data Centers' which are, well, data centers in their respective region.

Every data center has multiple worlds and you can travel without much effort between those worlds. (It only takes a couple of seconds to load and you do not have to pay anything.)

The market boards between the worlds of a data center are not connected. That means that prices can evolve differently between the worlds depending on the supply and demand generated by their player base.
A character can only sell items on the market board of their home world, i.e. the world they have been created on but can buy from every world of their data center.
If you want to sell an item you must place a sell order, specifying the price per item and the number of items you want to sell as a stack. Stack size is limited to 99.
Buy orders are not available! If you want to buy an item you have to choose among the existing sell orders.
Partially buying a sell order is not possible. You must buy the whole position.
Selling incurs a fee of 5%. As a matter of fact it could be reduced to 3% but for the sake of simplicity we will work with 5%, as this will not change the general approach.
Buying an item only comes with a fee of 5%, if you do that on a world that is not your home world.
It is not possible to directly access market data, but there are websites that have users collect and upload price data. For example Universalis, which handily provides an API we can use to query order book and sales data. (As a side note, there actually seems to be an API for the market data in the game but it is not publicly accessible.)
Trading happens 24/7.

The Implications

Now it is time to draw a couple conclusions from the facts above. Again, let's do this in the form of bullet points.

Because the market boards of different worlds are not connected, cross worlds arbitrage opportunities can arise.
Ideally you would want to match buy orders of your home world with sell orders from a foreign world and execute immediately. But since you can only create a sell order on your home world it might happen that the price drops rapidly, leaving you at a loss. This means there is a certain risk associated to the chosen approach, so technically it might not be fully correct to talk about 'arbitrage'. Yet I will continue to do so, as it makes it sound way cooler.
Not being able to partially execute an order is also bothersome. On the one hand you might not be able to afford a lucrative position. On the other hand, it forces us to think about good stack size when selling ourselves.
Because we cannot access up to date date, it can happen that arbitrage opportunities we find are no longer available.

The Model v0.1

Here is the high level work flow of what my first model was doing.

Query a list of items that have last been updated on a given world. There is an API endpoint for that, provided by Universalis.
For each item I would determine the minimum price on each world of my data center by again querying the API from Universalis.
Compute the difference between those prices from other worlds to the price on my home world and remember all servers where the price difference is positive, i.e. where I can make money.
Query the list of past transactions (another call to the Universalis API) and filter for the last 14 days. Compute the total number of traded items per day. Afterwards, compute the max, min and median over the available days.
As a measure for how profitable an item could be, I computed the 'median win' which is the computed price difference times the median number of traded items.
In the end I would report those items that have
- a profitable price difference
- trading activity on 85% of days in the past 14 days.

The Shortcomings

Every journey has to start with a first step. The same is true in this case.
It will probably not surprise you, that this model did not work too well. But it allowed for exploration of basic concepts and fundamental implementation techniques.

Enough talk! Here are the issues that quickly arose alongside with some possible improvements for the future.

The 'median win' was incredibly misleading. It would usually overestimate the potential win, obviously due to its poor design.
- It could be that only a single position would be priced much cheaper than the rest. So the same price difference could not be achieved with the other positions.
- Alternatively, it could happen that the number of favorable positions is less than the median.
- It would be much better to compare the actual listings of other worlds with the minimum price on my home world. That way I could see the profit to be made.
Turns out that picking the items 'at random' is not the best approach. What a Shocker! That way it is hard to find potential arbitrage in general and even if something is found, it might be in a category that has little demand.
- It would be much more promising to focus on categories with higher demand and thus higher trading activity. The margins might be smaller in this segment but being able to quickly close a position reduces risk. The longer a position is open, the higher the chance that the prices fall.
- Going by category requires to prepare the mapping of items to categories. This can be achieved as this data is freely available.
I was ignoring the fees entirely.
- Luckily, the impact of this is not too bad. It just slightly overestimates profit margins. On top of that, it is easy to fix. Simply add the fees to the formula for the price differences. So easy that it makes you wonder why it was not implemented in the first place.
My script had poor error handling that would result in it crashing for items that are iliquid.
- Mostly annoying and nothing that couldn't be fixed quickly.
Working with absolute prices differences. Expensive items would produces higher price differences but would also carry a greater risk / require a greater initial investment. A price difference of 5000 sounds like much but when the item costs 7 mio Gil, then it might not be worth the effort.
- This can be fixed by putting the win in relation to the required investment.
It was hard to decide on stack size for my sell orders. I had an idea of the daily volume but not which stack size I should choose.
- This is a tricky one. I have noticed tendencies for some items, e.g. people would pay more for a smaller stack when they need less of those items.
- One possible approach could be to look at the histogram of past stack sizes but I have the feeling that there is much more to uncover here.
The last one is less of a shortcoming but contains a lot of potential: There seem to be fluctuations in the price of items depending on the weekday. Which can be attested to the fact that more people play on the weekends. Yet the impact of that is not easy to predict. Maybe this means that the demand is increased because everyone needs a certain item or -conversely- the supply increases because more is being produced.
- The big issue here is data availability, as Universalis does not seem to keep a complete sale history. Thus it would be necessary to create a price time series yourself and then analyze the price change with regards to the week day.

Implementation

In the end, I want to briefly touch upon how I implemented this script without going into too much detail.
Everything is implemented in Python and is run under Windows Subsystems for Linux on my Windows gaming PC. This makes me chuckle a bit every time because I assume that I am the only one with a bash window on his second screen while playing.

The API requests are done with the 'requests' package and the most frequently needed calls are encapsulated in functions.

Data wrangling is done with 'Pandas' which gave me quite a lot of head ache because of how unfamiliar I am with the package. Also because I had clear ideas how to achieve what I want as a set of SQL statements.
This was especially annoying in conjunction with dates and date time operations.

Conclusion

The project so far had its ups and downs.

The implementation was rather frustrating at times. Especially because I knew clearly how to solve the problems with other tools. As if that was not enough, the outcome turned out to be less usable in the end then I had hoped for.

But how do they say: 'It is about the journey and not the destination'. It was really fun to think about ways to find arbitrage and the first version of the model made it really easy to come up with improvements.

The issues with the implementation were also very educational and now I have a first impression of pandas. I would probably still have to google everything, if I were to repeat the implementation. But maybe I would be a bit quicker in the process.

I definitely want to continue with this project as it holds a ton of potential for further learnings even outside of the things mentioned here and I will definitely take you along on the journey.

Do you have other ideas how to improve my model or did you realize arbitrage possibilities somewhere else? Let me know in the comments.

What is docker and why should you care?

topjer — Sat, 14 May 2022 17:12:58 +0000

At my work it manifests itself that our future lies with docker containers. As a matter of fact even more confusing terms are being used. They talk about ‘CRC’ aka ‘Code Ready Containers’, Kubernetes clusters and other things that make my head spin so violently that I have to hold onto my chair, else I would topple over.

In case it is not clear already, I am not in the slightest familiar with these technologies. But this is no reason to fret! It is my believe that the IT sector means constant learning and so I welcome this step towards ‘modern’ tools. (In an industry that is evolving as quickly as it does one should always be careful when using the word modern.)
Especially, since my company has a tendency to lack behind – significantly – when it comes to technology.

So I started to look into Docker containers and first tried to answer the question:

“What are they and why should I care”.

Let’s have a look at the bigger picture by looking at where we are coming from.

The simple beginnings

In the past things were much simpler. Every server had exactly one operating system installed. Let it be a version of Windows (Server) or a flavor of Linux. Everything would be controlled by one OS.

This simplicity would come with a lack of flexibility. What if you have Linux installed but need Windows for an application or vice versa? What if you need more computational power or less? Would you want to invest in new hardware and have to maintain it in that case?

The issues do not stop there. On application level there are further issues when you have just one big server. What if you have conflicting dependencies between the software you use? Maybe your WordPress and your data base cannot come to terms about drivers they need.

Divide and conquer

And with that, the stage opens for virtual machines. They use something called a hypervisor to divide a computers resource for multiple guest operating systems. They come in two different flavors. Either they function as a lightweight operating system and run directly on the hardware or they run on a host operating system.

All of a sudden, you are able to quickly react to changing needs. You can create a new virtual machine on your existing hardware that runs Windows Server or Linux. If one machine needs more power, you can simply assign more resources (if available), no need to get new hardware.

In a sense, Virtual Machines are virtualizing hardware.

One step back and a giant leap forward

Docker now goes one step further … or maybe back. You no longer need a hypervisor.
Instead a single operating system on your hardware suffices. Because what docker is doing is to virtualize the operating system instead of the hardware.

The docker engine runs on the host operating system and hosts itself so called docker “containers”. Which could be described as isolated micro computers. They can have their own CPU, OS, Memory and Network.

Why care?

Their strongest selling point is that they are lightweight. Once you have the Docker engine installed you for example pull the docker image for Ubuntu – which is currently only 77mb in size – and you have it running within seconds.

Installing Ubuntu on a virtual machine takes magnitudes longer.

But how is that even possible?

The secret is that Linux based docker containers share their Kernel with the host operating system. This brings one big drawback with it and that is that a Linux OS can only host Linux docker container whereas a Windows host can only run Windows docker container. (Not taking into account that you could use Windows Subsystems for Linux).

The biggest advantage is yet to come. Technically I have already mentioned it, but I want to emphasize it …strongly! They are isolated and therefore highly portable.

A scene like the following happens every day around the globe and is with a high probability also familiar to you:

Developer: Finally I am done with this feature. All tests are green. Time to give it up for review.

other Developer / Tester tries to run the code and it fails

Developer: Don’t know the problem. IT RUNS ON MY MACHINE.

Who knows how much money / time is lost because of differing runtime environments. May it be the environment variable that you have changed months ago or this one dependency that you forgot to document. Maybe you run a different version of insert about anything or who knows what might be the reason.

This will be a thing of the past. Because everything you need for a script to run must be part of a docker image. When it runs on your machine, it will run on every other machine. Because all dependencies and all those little tweaks must be part of the docker image.

Quick break to explain terminology: an image is the abstract definition of the environment and
every instance of this image is called a docker container. Cool? Cool!

You wanna know what else is neat? It’s this thing about isolation. You can have different parts of your server run in their own docker container and have those containers communicate with one another. Each having their needed dependencies in the version they need.

One component needs an update either for itself or a dependency? No problem. Your other components will not be affected. They might not even notice anything being different.

Conclusion

It’s time to wrap this up. This article was only intended to be a brief introduction into the topic of docker and as such can only be seen as a starting point.

Most information stem from this NetworkChuck video.

Check it out if you want to see some Docker containers in action.

In case you jumped right to the end in order to find out why you should care. Here are the 3 main reasons I see:

Docker containers are

lightweight
isolated
highly portable

6 crypto investment mistakes I made so you don't have to

topjer — Sat, 07 Aug 2021 15:44:32 +0000

The crypto world is exciting. Filled to the brim with technological possibilities unimaginable mere 10 years ago. As a result it becomes harder and harder not to dip your toe into these waters if you have even a remote interest in technology. This is at least how it was for me.

In the spirit of full disclosure I want to say that I find the whole conglomerate of DeFi, CeFi, Dex, blockchain and whatnot more than confusing. It happens rather regularly that I read an article about crypto which I simply do not understand.
The incredible opportunities come at a price of great complexity and this great complexity leads to one thing especially in the beginning ...

Mistakes.

Thus I want to give you an overview over the mistakes that I have done or just barely gotten away from.

Intransparency of value

If you delve deeper the matter of crypto and are not just in it to hold some Bitcoin then you might encounter a situation that is similar to the following:

Good, my 50 Euro have arrived at Binance. First I will buy some BNB. *buys BNB*
Now I want to transfer that to my wallet. *accepts fees and transfers it*
Sweet, next step: I buy some sweet sweet CAKE over at PancakeSwap. 'buys CAKE'
Wait, you can by lottery tickets with CAKE? Gimme five! *buys five tickets*
I always wanted to hold billions of coins. Time to buy some OSM *buys a ridiculous amount of OSM*

After this shopping spree you end up with a bunch of numbers in your wallet. My personally, I rather quickly forget the value these numbers actually hold.

Is a transaction fee of 0.01 BNB much?
How much did I really lose when I spend 5 CAKE on lottery tickets and not win anything?

I especially noticed that I was sometimes reckless with spending crypto because it was so intangible. Lottery tickets for 0.3 CAKE? Count me in! An NFT for 400 KSM? Interesting! (Not an actual investment of mine)

So my tip here would be the following: Whenever you want to spent some crypto, first ask yourself how much money that actually is. Once you know that, ask yourself whether you really want to go through with the transaction. This is especially true for transactions where you loose your coins, e.g. because you have to pay fees compared to transactions where you gain an equal amount of another coin.

Underestimate transaction fees

I want to stress the last point because I fell for that trap too often in the short time that I buy crypto. The worst thing is that you do not even immediately notice that you have fallen for it.
This is closely connected to the Intransparency of value I have mentioned before. Even though Meta Mask shows you the maximum gas fees, it can be hard to understand what it means.

This totally leaves aside the fact that it is easy to confuse a 0.0008 with 0.008 and thus you do not even notice that you spent a 10 times higher fee for a transaction.

Time for a story. It was the first time I traded crypto outside the safe haven that is a broker. I was ready for the wide world that is crypto and started with PancakeSwap.

For some time everything was smooth sailing. I could provide liquidity, I could invest in Syrup pools. Life was filled with syrup-like goodness.

But dark clouds were forming on the horizon. PancakeSwap has a mechanic where you can claim a bounty which activates the compounding function for the Auto pool. For this interaction you of course have to pay a small fee but simply wait for a bounty that is higher than the gas fees you have to pay. Problem solved
Nothing is as tempting as the promise of free money so I tried to claim the bounty … and failed.

As a side note, it can be really hard or virtually impossible to understand why a transaction failed. Especially in the beginning. In hindsight I think the problem was that Meta Mask was calculating the gas fees incorrectly or the bounty has already been claimed.

So I did the most natural thing to do in this situation. I kept on trying over and over again. Every time I got the same outcome.
At one point I decided to stop. Not because I came too my senses but I made a cruel discovery. For each failed transaction I had to pay gas fees!

Here an explanation why you have to pay for failed transactions on the ethereum network. I assume
that similar arguments also apply to other networks:

The Ethereum network requires gas to execute transactions. When you send tokens, interact with a contract, send ETH, or do anything else on the blockchain, you must pay for that computation. That payment is calculated in gas, and gas is always paid in ETH.

You are paying for the computation, regardless of whether your transaction succeeds or fails. Even if it fails, the miners must validate and execute your transaction, which takes computational power. You must pay for that computation, just like you would pay for a successful transaction.

https://metamask.zendesk.com/hc/en-us/articles/360045439051-Why-did-I-pay-gas-fees-for-a-failed-transaction-

In the end I had 6 failed transactions. The most expensive one costed me 0.69$. This is pretty high considering that I was trying to claim a bounty of 0.25$.

So always make sure you know how much you are paying for gas and if the transaction is not time sensitive then try to postpone it to another time with potentially better gas prices.

Unaware of "The others"

The internet is full of stories of people who made a ton of money. One smart investment. Getting into this one crypto that no one saw coming early. BOOM! Big returns. Big numbers. Incredible riches.

If it is not the story of people "having made it" then it is the promise that there is this one particular coin that will blow up soon. It is sooooo good it is only a question of time until it goes to the moon.
Side note: People advertising a coin are usually heavily invested in that coin themselves and would therefore profit of other people buying it as well.

In your head you are already thinking about where the perfect place for the new TV is that you will buy once you have become crypto rich as well.
But before you drift too deeply into this crypto dream I want to tell you something about the survivorship bias .

Let us start with with its definition, it is

the logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility.

https://en.wikipedia.org/wiki/Survivorship_bias

People that lost most of their investment because they put it into "the next big thing" will probably not brag about it on the internet. Maybe because they are ashamed.

"People will think I am dumb and not as smart as that other guy" might be a thought that is creeping up in their head.
Or they are simply too angry and want nothing to have to do with this whole crypto thing.
Lastly, you probably do not see these posts because they do not sell as well. They are not getting upvoted and
people who run a coin will hardly advertise the post of someone who lost everything with their coin.

So for starters I would advise you to keep in mind that for everyone who made money with a coin there
must be someone who lost money with that coin. This is of course somewhat of an oversimplification but the core holds true.

We can also address this from another angle. If you have already some investment experience then ask yourself how many of your losses you have talked about on the internet. I would assume that the answer is at least 'Not each one' and probably more along the lines of 'None'.

A more realistic view would now be to assume this to be true for everyone you come across. Everyone has some investment skeletons in the closet and rich people usually have big closets ...

Get rich quick mentality

This is an extension of the last point. So lets say that you are aware that people lose money. Not everyone
can be a winner. Still you could be one if you just approached it right.

How could you not want big numbers and prestige for yourself. So you start looking at dirt cheap coins with
zero utility and think: "Maybe this is it. Maybe if I invest just a tiny bit then it would not be too bad if the coins goes to zero. Yet it has potential to blow up really big."

Or you become part of a "pump and dump" community. The people are very nice and they guarantee success. They have done that many times before and everyone in there swears they have been making money.

For those of you who do not know what those communities do. They coordinate buying a certain coin. This results in an increase in price. Maybe they also spread the news like "XYZ is totally blowing up. You must get in now!"
Either way, other people will start noticing the increase in price and will buy as well. That is the moment when the community starts selling the coin to these new buyers for a higher price. Making a profit in the process (and letting those new buyers plunge into their doom).

In case you are wondering: Yes, this is market manipulation and would have severe consequences in other markets like the stock market for example. But good thing crypto is mostly unregulated. Am I right?
Just hope that the community you are in does not have an inner circle. An elitist group that has bought the coin before and then is dumping their coins onto you. Making you the poor fool plunging into his doom.

I think the point I am trying to make is that a get-rich-quick-mentality makes you vulnerable to scams. Stay away from riskier investments. Even if people claim there is zero risk involved understand this: you get a higher return on your investment the more risk you are willing to take. A high yielding investment without risk does not exist. Here I would like to ignore arbitrage situation for the moment which would be return at zero risk but that is a very special situation.

Here is my recommendation: In the beginning buy one of those established coins and hold it for a while. Get a feeling for the ups and downs of the crypto market. You can invest in shit coins. But only amounts that you can stomach loosing. Also do not spend all your time researching the next coin that is blowing up. Instead learn more about crypto. Because this world is actually quite fascinating once you look past making quick profit.

It always pays to take a second look

Let me make that crystal clear: Always make sure you are sending your crypto to the correct adress!

Picture this: you want to send a noteworthy amount of crypto from one wallet to the other. You have done that before. No big deal!
You click send in one wallet and then you wait. But your crypto does not arrive in the other wallet. Nervousness is rising. You decide to check the transaction which luckily you can because a block chain essentially is a vault made out of glass. Comparing the target address from the transaction with you intended target address you realize that one digit is false. Do you know what that means?

You crypto is gone! With a high probability will this wallet not belong to anyone. Even if it would belong to someone it would be nigh impossible to contact that person.

I am very grateful that this situation did not yet happen to me. Once I almost thought that I lost my money when trying to transfer SHIB from Binance to my wallet. In the end it turned out that I did not add the correct currency to Meta Mask and that was why it did not show my transferred crypto.

The take away point: If you intend to send a bigger amount of money or if you do your first transaction to a wallet it might be a good a idea to test it with a small transaction first. Alternatively it could be possible to simulate the transaction on the testnet if that is available for you crypto of choice.

Nothing but a speck of dust

It is wise to start with small amounts when investing into crypto. This helps to get a feeling for how the markets move.

But it also comes with the own set of problems. The following two points are more of a word-of-warning, things to be aware about and not concrete tips.

The transaction is too small compared to the fees

For me, everything started with buying 10 Euros worth of Ethereum. Having this money in my Coinbase wallet was exhilarating. It felt like I was holding a piece of future in my hand.
That it until I wanted to move that ETH into my MetaMask wallet and saw that I would have to pay 8 Euros of fees. Guess what! Those ETH stayed with Coinbase for a long time.

Your holding amount is to small to do what you want to do

It happened to me way to often that I wanted to do a certain transaction but could not do it because it required bigger amounts then I was holding.

Let me give you an example: I have a some money invested in Drip where I get 1% per day on my staked amount. Instead of reinvesting my interest, I claimed it accidentally. When I wanted to reinvest it I couldn't because only investments of more than 1 drip are allowed. So now I have 0.16 Drip lying around and I have no idea what to do with it.

Conclusion

There you have it. These are 6 beginner pitfalls that you hopefully can avoid now that you are aware of them. None of those hurt me too much but it either resulted in burning some money or loosing some hair because of the stress they caused me.

Feel free to share your beginner mistakes with me in the comments and it case that it was not clear already because I was talking about "mistakes", nothing in this article should be seen as investment advice.

My greatest Prank ... that went a little too far

topjer — Thu, 03 Jun 2021 21:01:09 +0000

At work I try my best to be professional. Or at least to make others think that I am.

Yet, I was once given the opportunity to pull a masterpiece of a prank on one of my colleagues. Needless to say that I took it.

The Setup

It was a normal day at work. I was inspecting the task manager on one of our servers and to
my surprise I noticed that I was actually an administrator on said server.

Actually it was not too surprising. It was the development server. I am a developer. Getting total freedom to test things is kinda useful and only makes sense. But let us continue

Curiosity took over and I started exploring this new world of possibilities. First, I noticed
that I was able to disconnect coworkers from the server. This is definitely the basis for some good prankery. But I was ultimately drawn towards a function that was a bit more subtle.

It was possible to send a message to any user logged in to the server.

I assume this can be used to inform users on a server directly about maintenance work
you have to perform. Totally serious stuff.

But even the most upright tool can be turned into something devilish in the hands of
a fiendish mind.

The functionality also played right into my hands. Not only was I able to freely choose
the message I send. Also the header of the message could be modified. Perfect for erasing
traces.

The Prank

The target was quickly acquired. A new colleague with a great interest in Bitcoin. He was talking about that 90% of the time.

It might sound like bullying to you. Picking on the new colleague. But rest assured that I really liked him personally. So it was more like a prank between friends.

My message was custom tailored to him with a pinch of ludicrousness known from infamous
email scams and topped off with a touch of impending danger.

But enough talk. Lay your eyes on the masterpiece I created.

The message was prepared. All that was left to do was to send it. Which I did. Expecting
a good laugh between colleagues.

Little did I know that it would have a far bigger impact.

The Fallout

I might be a genius trickster but my timing was abysmal. Because I sent my message immediately before went out for lunch. That way I was not able to stop the avalanche that was about to start moving.

My colleague, understandably bewildered, send an email to all developers. 'All' meaning me and another one.
Unfortunately he also took our boss into the loop. Our boss who has a slight tendency for overreaction.

Fearing the infection of chinese-russian malware he informed the server admins. A complete rollback of the server was in discussion.

Meanwhile, I was still on lunch break. Chuckling at the thought that I was playing a small prank.

Luckily my boss got a hold of me before the server rollback plans were turned into reality. He asked me if I have any idea what was going on here. It was easy to play clueless since the conversation took place over teams.

Unaware of the extend of the consequences of my actions I broke into laughter at some point.

Needless to say, my boss did not find it funny.

I got away with a lecture. The admins, exasperated, could be appeased. But I think they still
hold it against us.

Next time I checked, we developers were no longer admins on the server. Even though
that could be due to totally unrelated reasons. That's at least what I would like to believe.

What about my colleague? After he stomached the shock, we were able to laugh about it. We
even now do that from time to time when the memory comes up again.

Final Thoughts

Would I play a similar prank again if another chance arises?

Probably not, I have "grown" since then and try to be more of a senior at work.

Do I regret my actions?

Not a bit because it was an excellent prank and is the source for some good laughs.

What about you? Have you done something similar to a coworker? Feel free to share your
story in the comments.

Brave - The Browser that pays Crypto

topjer — Thu, 06 May 2021 07:03:34 +0000

Recently I read an article about upcoming innovative block chain technologies. What stood out the most to me was the mentioning of a new browser called 'Brave'.

It has a strong focus on security of your data. But what is truly remarkable is the fact that you can be rewarded BAT when using the browser. This stands for 'Basic Attention Token' and is a cryptocurrency, i.e. it can also be exchanged for real money.

Yet there is so much more to this browser.

Brave takes care of your security

You have to make sure that the lock is shown in the address bar of your browser whenever you visit a website. Everyone knows that, right? It indicates the use of https. This ensures the authenticity of the visited website and the protection of the privacy because the exchanged data is encrypted.
Brave will upgrade your connection to https whenever possible!

But it gets even better. It will block all trackers from websites. Online trackers are scripts designed to gather information about how you move through your web.

The following situation is probably familiar to you: You want to buy something and do some research.
Next thing you know is that you see advertisements for this product everywhere.
Perfect example of a "successful" tracker.

If you require more security you can also open a private tab with Tor. This will not only conceal your browsing history. It will also hide your IP address to the sites you are visiting.

Brave is faster

Brave keeps even more unwanted things out of your surfing experience. It will block any advertisements you encounter. This naturally leads to the side being loaded faster while also saving you bandwidth (which it likes to remind you of).

Number of trackers & ads blocked, bandwidth and time saved.

Still, the claim of higher speeds should be taken with a grain of salt. It might be true that Brave is faster than for example a normal Firefox browser. Yet, you can install extensions on you Firefox browser like the add blocker which does something similar and thus will also produce similar speed improvements.

The benefits of Brave here are that you do not have to look for extensions that fulfill your need for security and add-removal. It will come right out of the box. Also, you can rest assured that it will be maintained properly.

Brave and Ads

Let's talk about what I like most about this browser. No, it is not the fact that you can earn money with it. Well, only kind of.
It is the fact that they are "taking out the middle man". What do I mean by that?

My understanding of the current modus operandi of advertisement in the net is the following:
You are the product that is being sold. Websites get money for showing you advertisements. Your surfing behavior is analyzed and the gathered data is being used to show you custom-tailored ads. Whoever gathers this data can make a ton of money with it.

(I am sure that this an oversimplification. Hey, I never claimed to be an expert on this. So if anyone has more information how advertisement on the web really works, let me know.)

Brave is different. All ads you would normally encounter. Blocked! All trackers that would analyze your behavior. Blocked!

In some sense there is an advertisement void which Brave knows cleverly to fill. It gives you the option to be shown commercials. This happens in a very low key and non intrusive manner. No more "Your PC is infected. Get help now!" banners.

You have control over the number of ads you are being shown. If you want, you can even
turn of that feature entirely and browse completely ad free.

It should be mentioned that, in order to provide you custom tailored ads, Brave itself does analyze your browsing behavior

So how is that better than what others are doing?
It is better because this happens locally on your device. The information is - allegedly - not shared with anyone. At least not in a form where it could be connected to your devices or user identity.

Let's hear what Brave has to say about that:

We do not have access to identifiable user data. The anonymized aggregated ad campaign related data we do collect is used for accounting and reporting, but this data cannot be mapped back to devices or user identities of any kind.

In summary: you are only shown a handful of "high quality" ads at your own volition. On the other side advertisers will get better results because their ads are precisely targeted and they do not have to
give a huge chunk of their profit to middlemen.

Let's talk about BAT

So, what kind of ads to you get you ask? Either an ad is displayed when you open a new tab. Which you might not even notice. Or a box pops up which you might as well could ignore. Whenever this happens you are rewarded with a fraction of a BAT. To let you partake in the advertisement revenue.

An example of an ad

At first I was somewhat confused.

If I earn BAT by watching ads, does that not mean that evermore BAT is produced and you will have inflation?

But I was missing the point. The BAT that is earned is not produced. Instead it has been "bought" by the
advertiser and will be given to you as a reward for your attention.

In total the amount of BAT is limited to 1.5 billion of which 1 billion are in circulation, see here.

Before we continue, a big disclaimer. My understanding of cryptos is limited. I am fascinated by the topic but did not yet dare to dive in.

This brings me to the next point I like about Brave.

It lowers the barrier of entry to the topic crypto currencies. I did not mind changing my main browser (especially since I could easily import all settings and bookmarks without issues) and now I earn crypto while doing something I am very familiar with. Browsing the web.

The specifics of how to trade or cash in BATs is still nebulous to me. I intend to give you an update on that
once I more experience with that.

But there is another use for the BAT you amass.

You can support your favorite content creators with BAT

Brave also thought about all those content creators that might miss out on advertisement revenues. You can support them directly with your BAT. Either in the form of a monthly or a one time payment.

Support a verified website. Either with a monthly amount or a one time donation

You even have the option to let Brave decide whom to give a portion of your BAT based on your browsing
history. It determines the fraction of time you spent on every website and splits up an amount of your
choosing when it is payday.

Because you might want to give more BAT to content creators than you can earn in a month it is also possible to load additional funds into you wallet.

Again, you have fine control over the specifics here and can also choose to ignore this feature entirely.
I personally prefer to choose myself whom to give how much.

The drawback here is that we are still in the early adoption phase of this technology and most content
creators are not yet verified. If a creator or website did not verify itself with Brave then your payment will remain in your wallet until they do.

Brave is open source

Your heared right. The source code for Brave is completely open for public scrutiny. you want to, you can check out the entire source code over at Github.
This gives the statement that gathered data stays on your machine more weight. Even though it is more of a head thing because I am not going to check the code to verify it.

Brave is not without controversy

So far this browser sounds too good to be true. Thus I decided to find out whether there are critiques out
there and what they do have to say.
Turns out that there is actually some controversy surrounding Brave. I have found two things that happened in recent history.

"Hijacking" of referral links

When users visited certain cryptocurrency websites the browser would autocomplete the referral link of Brave without asking.
Some people called that hijacking. But I think that this is not entirely accurate. Hijacking for me would be when you use a referral code which is than replaced by the one of Brave. That was not the case.
They were only taking commissions that would be lost otherwise. Which probably does not even hurt the users but the platforms in question. Still I can understand why people got upset.

After the public outcry this feature was turned off by default. You can activate it if you are cool with it.

Intransparency of tips

Another controversial topic was tipping content creators that are not yet verified by Brave. Previously the BAT would go into some kind of omnibus account waiting for the creator to register. To some users this seemed like the money was taken by Brave in hopes that it will never be claimed.

The process was fundamentally changed once this came out. If you are tipping a content creator that is not yet verified then your tip will remain in your wallet until he does.

Bottom line

I am not entirely sure what to think about these two incidents. In case of the intransparency of tips I'd like to give them the benefit of the doubt.
The "hijacking" thing weighs a bit heavier. It was by no means behind the back because the user could clearly see the auto-completion. Yet I cannot deny that I would expect to be asked whether I am fine with that.
It shows me that I should use Brave with some vigilance.

Final Thoughts

My attitude towards Brave - as you might have guessed - is mostly positive. The two mentioned incidents leave a bit of bitter taste in my mouth. But not to such it an extent that it is off putting.
I find their approach to advertisement fascinating and their security focus reassuring.

For now I think of Brave as an experiment. It is the first application of the block chain that I can relate to.

In the future I will keep you updated once I have delved deeper into the crypto trading portion and the support of content creators.

Renting a server - This is what I learned

topjer — Wed, 28 Apr 2021 20:24:02 +0000

Maybe I should create a website?

This thought came up in my mind irregularly in the past. Usually when I thought about which creative outlet I should use. One option, the creation of videos, I did pick up but it did not feel like the right means for many things that I feel I want to say.

Blogging came to mind next since it focuses on the content and less about the delivery. It is content distilled down to the essence, so to speak. Yet, whenever I gave that option some thought my perfectionism countered with many questions, like "Which platform to use for blogging?" or "Use a paid variant or a free for the beginning?" and the most important question: "Which domain should I register?".

Feeling overwhelmed by this barrage of questions I usually decided to stop thinking about it.

This week something was different. I stumbled over a video of NetworkChuck where he was telling his viewers that they need to create their on website ... like right now. Seeing this video was the proverbial straw that broke the camels back.
In the past weeks I have become more "productive" in my free time. Finally, I started to do some programming projects that had been on my mind for quite some time. The more I programmed the more I wished to write about it. So NetworkChuck's message came at the perfect time.

The birth of a new site

Are you aware how easy it is to create a new site? The term child's play comes to mind. It takes less than 5 minutes and can even be for free.

To me it was immediately clear that I would choose a paid version. The prime reason being a simple one ... control. Another reason being that a paid site looks more professional. "topjer.host-your-site-for-free.com" simply does not have the same ring to it as "topjer.net" does. Lastly, the lack of ads also helps to give the site a clean look.

I briefly swayed from NetworkChuck's recommendation to check out other web hosting options only to quickly return to the one he mentions, hostinger.com.
It seemed like all hoster would basically give you the same service with minor differences. I feared that going into the details this early in my "career" would make it crash before it could even lift off.

Here is now the first thing I learned in the process:

Prices for web hosting are somewhat non-transparent.

Hostinger advertised a web server for 2,39€ a month. You even get a free domain on top. Sounds too good to be true? Well, it kinda isn't. See, you get the price of 2,39€ a month only if you agree to rent the server for 48 months. And afterwards you have to pay 4,99€ a month.
If you don't want to be bound to Hostinger and want to be able to cancel the subscription on a monthly basis you have to pay 7,99€.

Remember the free domain I was also talking about? Well, it is free for a year and than you have to start paying. Depending on the suffix you choose the price can vary greatly. I went with a '.net' which will be 12€ annually starting from the second year. Initially, I had my eye on '.tech' until I found out that this would cost a whopping 45€ a year.
If you break it down to a month this is still less than 4€ but since I had no idea how good blogging works for me, I went for the cheaper alternative.

As a side note I want to mention that I had to pay up front. At least I did not see the option to pay monthly. So expect to pay a bigger amount initially when getting your own site.

In all fairness, I want to say that Hostinger is by no means "the black sheep" in the herd of web hoster. On the contrary, it seems like most providers of web hosting services use similar tactics.
In the limited selection of hoster I checked out, Hostinger seemed to be one of the less non-transparent.

Did anyone say child's play?

After I have stomached the final price and decided on the domain - a process that took surprisingly long for the fact that I went with such a simple address in the end - I was finally able to start my blog.

As the foundation of my site I wanted to use WordPress which apparently is used by the majority of websites in the world wide web. We are talking about 60% of the software built sites.
Surely it will be easier this way than to code the site by hand. Well, I guess this is true. Especially since I know very little about front end development. Next to nothing to be honest. But it is by no means "easy".

Since WordPress is powerful tool I was faced with a plethora of options right from the start. Use WordPress to customize the outline of site. Elementor can be used for parts of the content. Interested in analytics? Simply make a Google analytics account and start with OptinMonster. Forms. Plugins. Search Engine optimization.

Right from the get go I made a grave mistake. I decided to create my side according to a template. This filled the side with some example content. My pride forbids me to tell you how long it took me to remove said content. Too long! Don't get me started on the endeavor of adding my socials to the side. There were many ways to present them on the site. Each working a bit differently.

So here is my next lesson

Tool assisted website creation is complex. So keep it simple in the beginning!

A door opened

There is one aspect which I completely neglected until now.

The initial goal was to find an outlet, to give me a medium to communicate my thoughts. To document my journey. Yet, I have attained so much more.

Having a whole web server for yourself creates possibilities that every developer can appreciate. The plan I choose did not only give me a hosted WordPress. No! It gave me a whole web server to play around with.

I can simply ssh onto the server and snoop around. Create cronjobs. Run my own scripts on it 24/7. Create databases and connect my scripts to it.
Do you remember me talking about professionalism in the beginning? What is more professional than having a mail address like contact@topjer.net?

Of course, compared to dedicated solutions the server I now have is lacking and probably laughable. But do not forget: I got all of that on top! I did not know that I would need all of that. Now that I have it, I am excited to play around with it. I can still switch to a more powerful dedicated server should I outgrow my current solution. After all it has gotten nearly as easy to get your own virtual machine in the cloud nowadays.

With renting this server I have embarked on a journey. The surprise about the prices and the almost overwhelming number of options aside the start of this journey has been mostly positive. Most importantly it has made me excited for what is about to come...