Forem: Marcelloh

Solve code complexity (in Go) Part 3

Marcelloh — Wed, 22 Nov 2023 09:09:13 +0000

Introduction

Please read the other two posts first to get an idea of what this is about.
Part 1

Because of my blog post, I was asked by several people to run my linter on their software. I sometimes did this on projects that didn't ask me to do it. I wanted to lint a lot of different Go-projects, just to check if my linter would report it in the way I expected it.

Reactions

I remember that when I was just a starting developer (around 40 years ago), when someone would say something about my code, it felt like they were saying something about the most beautiful thing I ever created, so I felt hurt.

Of course, there were a couple of those reactions back, from "Mind your own business!" to "It works, so why bother?".

You can imagine why people would give such answers, and I wish them all the best.

Overall, the reactions were positive, even when my linter reported a lot of "painpoints" in the best thing they ever created.

A Go-project was linted of one of the organisers of the Golang Amsterdam Meetup, and I asked him if he was interested in the results. He was, and he reacted quite positively. That led to having a 45-minute presentation at the very next meetup.

Golang Amsterdam Meetup

As the first presenter, it was nice to tell about my experiences so far. People got an idea about how the linter would look at complexity. I showed the top 3 Go projects that had the highest complexity per function.

And when it was time for questions, I really felt that the audience had understood the idea of my presentation.

Of course, there was the expected one:
"Can you lint your linter to see what the maximum complexity is?".
As I said during the presentation, I like my software to be simple (following the KISS principle) and maintainable, so it was no surprise that the linter has a maximum complexity of 9.

At the end of the Golang Amsterdam Meetup, I was given a Go mug, which is a nice thing to have.

Numbers

In the presentation, some numbers were shown:

81 not-so-random projects:

popular ones
idiomatic ones
newcomers

The linter analysed 47.601 files and 284.401 functions.
That’s 5.427.312 lines of code inside functions

Findings:
31.029 complex functions with 2.164.305 lines of code.

about 11 % of all functions are complex
about 40 % of lines belong to complex function

Facts / wisdom

In the end, there was a slide with some things to think about when you write software.

K.I.S.S (Keep It Stupid Simple).
Complex code can hide bugs easily.
Don’t be clever; be smart (or wise).
Code tends to get more complex over time.
To write code is easy, to write understandable, maintainable code is hard. (Marcelloh)
Treat your code the way you want others’ code to treat you.
Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius, and a lot of courage, to move in the opposite direction. (Albert Einstein)

The end?

Does this end here? Is this my own little pet-program, that I can use when looking at code?

The questions the people asked, had some really interesting ones between them, so I feel the urge to have some extra options inside.

One simple conclusion is that people don't all look at Go-code the same way, which is fine with me. If I can "capture" all those differences into a configuration, then the linter would behave the way they need it to.

The future

Can the linter be tested in a business environment? It will be helpful for a company if they want to write maintainable software that will survive a couple of years.
It will save them so much time and will probably squash some bugs along the way during refactoring that I think it should be worth something.

So that's what I'll be investigating.

The linter needs some reviews, and at the end of the presentation, I asked if there were 3 to 5 companies willing to have me look at their software (in their office) for free in exchange for a review.

I already have one company that will invite me.

If you read this, I can do the same for you, for free, for now, as long as I have the time and if you are willing to write a review about the outcome.

Solve code complexity (in Go) Part 2

Marcelloh — Thu, 02 Nov 2023 11:56:57 +0000

Introduction

So I wrote the previous post and looked at a lot of software, because my linter pointed me at some interesting parts of software made by others. The linter looks at code the way a developer would look at it and judges the code according to that "knowledge". I have to say that I totally agree with the findings of my linter. Some code is so hard to read and, therefore, to understand.

Testing my linter in the real world

I did some testing of Go projects that got my attention. Downloaded those and let my linter point me to it's "wisdom". I sometimes took the liberty of passing that information to the project it reported about.

Some projects really understood what the "flaw" in their software was, and how they could improve from this knowledge.

Other people ask me to explain more about the simplification and how I would do it.

And there were also some who just found it annoying. My software works, so what?

This last group can't be saved, but for the others, please read on.

Advantage

Using this linter as soon as you finish writing a part of the code would be ideal, because it will give you some hints about the code complexity. Since the software was just made, you can easily refactor it to make it more simple and maintainable. This would be the real advantage.

What can you do?

There are some tricks that you can use to make software simpler.

If there is an if-statement with a lot of code inside, check if you can change it into an early-return.

If there is a loop with lots of code inside, isolate that code into another function.

If there is a switch-statement with cases that have a lot of code inside, isolate that code into another function.

Take a close look at nested if-statments, and nested loops. See if you can rewrite this in another way or isolate more into separate functions.

Did you know that your IDE might have a refactor option to easily isolate selected code into a new function? (I know Visual Studio Code has a way to extract code into a new function.)

Still testing

I made some changes in my linter because my opinion of what complex is, can be totally different from what others think. I now have a configuration in which you can set your own complexity boundries. To me, this is more than perfect, because perhaps you can read spaghetti-code and your colleagues can too.

I am still unemployed and while looking for a job, I can do some linting for others.

For now, before I make the tool available, it means that you can drop me a message (on here or on LinkedIn: see my profile) with the url of your git repository and I will get you your report.

Part 3

Solve code complexity (in Go) Part 1

Marcelloh — Thu, 19 Oct 2023 08:37:12 +0000

Introduction

People who know me, know I'm a fan of the KISS principle (Keep It Stupid Simple). To me, this means if you write maintainable software that is so easy to understand, that even "the new kid on the block" can do some changes on it, without breaking it (I'll hope). Or your future self is probably grateful for the simplicity of the stuff you have made in the past.

Genius at work

Sometimes you're just busy implementing the latest idea into your code, or the companies code for that matter. As long as everything works according to specifications, you're good.

If you're lucky, you have that critic of a co-worker, who can point you to some nasty stuff you did. But to have that colleague, or to become your own critic, can take time.

Perhaps I can be of help?

Cognitive code complexity

Ever heard of cognitive code complexity? I've read about it a couple of years ago and I was intrigued.

Cognitive code complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and understand.

I decided to implement it in my own linter and followed the idea of one of the studies.

How it works

Imagine that each if-statement within a function adds up to the complexity of that function.

 complexity = 1
  + 1 (found 'if' at line: 159, complexity = 2)

We can look at more recognisable parts that will make the code more complex:

AssignStmt
BlockStmt
BranchStmt
CaseClause
DeferStmt
ForStmt
GoStmt
IfStmt
RangeStmt
SwitchStmt
TypeSwitchStmt

Well, it isn't that simple! Because, you will encounter nested statements. I actually add the depth of a statement to the complexity. (You can see in the following example)

mytube.go:57:1 - isPartialMatch has complexity: 4
  complexity = 1
  + 1 (found 'range' at line: 61, complexity = 2)
    + 2 (found 'range' at line: 62, complexity = 4)

He's clever!

This is a bit how most of the studies go about cognitive code complexity and I blindly followed their findings and advices.

When I finished my implementation, I proudly showed it to my son. He must have been about 18 years old at the time, studying to become a frontend developer.

He'd asked some questions about the inner-working when he popped "the question" which made all (or most) studies look bad.

I don't recall the exact question anymore, because that was about 5 years ago, but it was something along the lines of "Would you agree that an if-statement with a little content is less difficult than an if-statement with more content?"

I'm so proud of him. He proved to be as clever as his dad :-)
Perhaps even more clever, because I didn't think of that before, when I was a blind follower of people with a degree.

Challenge accepted

Ok, so my linter had to change to make it a bit more along the lines of "the question". It wasn't that hard to do and in a couple of free hours, I fixed it. It did run flawlessly and for the projects I'd tried it upon, I could tell in no-time where the pain-points were.

Looking for a job

Since I'm looking for a job and have some spare time at my disposal, I began to think about cognitive code complexity again.

I decided to make a separate linter with the same functionality, so people can enjoy this one day. It would be awesome, if you can pinpoint the functions that needs refactoring, all by yourself. You could make "the new kid on the block" or your future self happy! (Strike 1)

You'll even make your boss or company happy, because the maintenance costs will drop if the software will follow the KISS principle more. (Strike 2)

The chance that "KISS" software will survive a couple of years is much higher than "clever, but complex" software. (Strike 3)

Next steps

I tested it on some projects I like and gave them some stuff to think about. One project lowered the complexity from the highest being 27 to now a 12. (I consider 12 something "the new kid" can understand.)

It's ready enough to test it out in the open, but not ready enough for others to use it.

For now, it means that you can drop me a message (on here or on LinkedIn: see my profile) with the url of your git repository and I will get you your report.

But... if I get a lot of projects to lint, I might not help you that fast. Also I do some analyses on the linters findings, if it works as it should.

If you reached this line, thanks for reading!

Part 2

Library / Package selection

Marcelloh — Tue, 06 Jun 2023 10:47:00 +0000

Introduction

When you are making software as a team, or on your own, you know you can't make everything yourself. There are plenty of people working hard on something you might be able to use.

Sometimes the number of solutions for 1 challenge you're facing, are available in large amounts, sometimes there's only one. How do you know which solution to select? (The apples and pears dilemma.)

The scorecard

It is always a bit of a puzzle, so I tend to make a scorecard to give me a better idea. Points (1-10) are given to particular data. Depending on what you think is important for a solution, you can use a weighted scoring system.
You totalise the outcome and now you can compare.

Even if you only have 1 solution, if you do this every time, it gives you a rough idea how useful the solution is for you, and if another one appears on the horizon, it's easy to compare later on.

My scoring list

Popularity
Age
Stability/regular updates/clear releases
Roadmap
Documentation
- examples present?
Does it seem trustworthy?
Other publications present?
Open issues
- look at age too
- average time to react/solve
- are beginner questions anwered quickly?
Latest version
- checkout and find out testcoverage
- do your regular linting
- review code
Questions/answers on fora (stackoverflow/reddit/...)
- are there a lot of unanswered questions?
Micro-version
- use it in a small program in how you would need it
- is it easy to use?

Adapter, when in doubt

Often I see myself looking at several potential solutions which are strong candidates. Or I see a newcomer around the corner, which is tempting to try out. Or you find out that the solution isn't that trustworthy anymore.

Most of the time, I find myself making use of the Adapter pattern, which means that my software doesn't need to change if I decide to change a solution. The advantage is that if I choose a solution now, but later on there's a better one, fast one, simpler one, you name it, I can always switch without to much of a hassle.

FastDB, a speedy persistent KV store

Marcelloh — Wed, 22 Feb 2023 12:22:06 +0000

Introduction

I've been using several KV (key-value) store solutions, but was never satisfied with the speed. We all know that the database-part of any application is most of the times the weakest link.

So what if I could make this faster as the fastest one I was using? Would the application that I am working on, benefit a lot? The answer is yes, it does. So now my application runs faster, smoother than before.

Semi-technical stuff

The fastest key-value store is just a simple map, I think. But that was not enough, because I wanted to be able to use buckets. (Buckets are a kind of a "box" in which you store key-values of the same kind.)

So I made the map like this:
map[string]map[int][]byte

I tried the basics of this against the usual solution and it was faster. But I still had to make a solution where the data would be stored on disk. I saw some AOF (append only file) solution somewhere, but wasn't satisfied with the speed, so I've tweaked that a bit more.

So now, when you open the file, you tell it to synchronise the data to disk every X milliseconds. (I would suggest 100.)

Since this is only a KV-store, there is no backed-in mechanism for sorting the output. Perhaps I will make one later, but for now there is a solution which is quite fast.

Example (pseudo code)

Here you can see how to open a database and store data in a bucket "texts".

store, _ := fastdb.Open("your.db", 100)

record := &someRecord{ID: 1, UUID: "UUIDtext", Text: "a text"}
recordData, _ := json.Marshal(record)

_ = store.Set("texts", record.ID, recordData)

Here you can see how to get data.

memData, ok := store.Get("texts", 1)
memRecord := &someRecord{}
_ = json.Unmarshal(memData, &memRecord)

Github link

https://github.com/marcelloh/fastdb

Errors with stack trace in Go

Marcelloh — Thu, 12 Jan 2023 08:55:51 +0000

Introduction

I was always unhappy when it comes to collecting information about the source location of an error. Also the fact that it was sometimes hard to understand which route the program has made to get to that point. That's why I came up with a simple solution: I needed a stack trace.

I've used this for a couple of years now, but then I saw someone complaining that Go didn't offer stack traces. For me that was the trigger to write this, because there is a simple way. Please read until the end :-)

Code information

There is this package which is the solution:
import "github.com/pkg/errors"

To be able to make use of that, each time when you return an error, just wrap it like this:
return errors.Wrap(err, "")
this is functionality from that package

In your main program, where all the error are going to, you just show that trace:
trace.ShowError(err)
this is my own function

Go Playground example

I've made this example, so you can play with it to see how it works.

First:
run it, to see no stack trace

output

Program started
Program finished

Second:
remove the // from line 47
so the line will be active, you'll see the stack trace.
(remark: line 22 is calculated of course from the line of the file, and not the line in the playground editor)

output

Program started
2009/11/10 23:00:00 an error happened
play.groundstart.initialise
    tmpsandbox2363014146startrun.go:22
play.groundstart.Run
    tmpsandbox2363014146startrun.go:9
main.main
    tmpsandbox2363014146prog.go:16
Program finished

Third:
remove the // from line 36
so the line will be active, you'll see the stack trace.

output

Program started
2009/11/10 23:00:00 an error happened
play.groundstart.Run
    tmpsandbox503374963startrun.go:11
main.main
    tmpsandbox503374963prog.go:16
Program finished

Here the link:
https://go.dev/play/p/HOzvDTRo39D

Kiss IT

Marcelloh — Wed, 27 Jul 2022 10:09:20 +0000

While I roam the planet for a nice remote job, I amuse myself in the way that some companies try to lure people into their world. Some companies really do their very best to tell their story and feed you with the same enthusiasm as they experience themselves. It reads like a micro-novella and is exiting to fantasise how it would be to work there.

I must admit that I'm a huge Stephen King fan, so I like to read horror stories. Did you know that those exist as a micro-novella in the job-vacancy world as well?

What strikes me the most, is the fact that they probably don't know what they really mean to say, which I hope is actually the case in most of the time. But if they rally meant it, they should realise that there is a minor change that people will fall for the trap.

I know you are curious, so let me enlighten you. It's just a small sentence from a job-vacancy that does it.
"Do you like to work with complex programs?"

Let me break this down. If programs are complex, it probably means that it was build with the help of a mad genius, which means that the source code is hard to understand, difficult to test and that there is (just a guess) not that much technical documentation.

I like to work with complex problems, but solve that with simple solutions. Well maintainable, well tested and with good technical documentation.

Did I mention already I'm a huge fan of KISS too? And in this case, I'm not talking about the rock band, but about Keeping It Smart & Simple.

Why are people still create complex software (and do they need help in maintaining it)? I sometimes feel like a mad genius myself, but I always know that I have a responsibility towards the audience to make it smart & simple.

Don't fall for the trap of complexity. Do your very best to write the future!

Your code stinks

Marcelloh — Mon, 25 Apr 2022 07:16:56 +0000

How to deal with bad comments about your brilliant work

This writing is about how I deal with these kinds of situations. As it is my personal view on the subject, feel free to comment with how you would deal with it. And in the end, there might be a strategy for you to be used on occasions like this.

This is the situation

After a certain period of time a colleague and I finally finished our best work ever. We did every possible thing, because we know the drill.

There's tests with an almost 100% code coverage, there's documentation how to use it, and this time we made a video presentation about it to explain to everybody how this would change the world of today.

Before that

Of course there are some colleagues, who know what we are doing and sometimes we show a bit of the unfinished work.
We get some feedback and try to improve the software as much as we can, because the colleagues made some valid points.
They want to know how it is going, and all the interest gives us the idea that we're on the right track for a place in the company's hall of fame. We made jokes about it that we might even receive a tiny little golden statue for it.

The state it's in

So we've spend a lot of time making this solution. Because of changing requirements, it isn't completely ready. It is however useable and it's just a matter of 1 month which will make it feature complete. And if we just can get it out in the field by now, we can prove it works for the simpler cases already. There are some arguments against it, but since this it a product that will save so much time, we can make a shadow of a real solution. We know the real deal will take about 2 weeks and with ours it's just a matter of hours.

The soap bubble

And then somewhere on the middle of a beautiful sunny day the message comes in.
STOP THAT
From high up the tree, followed by an explanation. But your brain somehow is shut off a bit. Some of the information still arrives in your braincells, but some information takes a wrong turn making you feel sick or sad or even angry.
It's a K.O. (knock out) by words and you feel misunderstood.
As a real introvert, my external being shuts off and reacts to basic behaviour.
This is what happened to me and describes the very first moments after the explosion of the soap bubble.

What did they say

Does it really matter what they've said? No, it doesn't.
In our case it was about the investments (or time) spent on it. You might run into an open source project that does about the same, but isn't "home made", so they think that is cheaper and maintained by others.
Or sometimes it's just that they're not ready for things like this.

How I deal with that

I remember those occasions where this happened to me. Yes! More that once I was misunderstood. People with an IQ like me have that all the time, so I am used to this.
In the very beginning this really hits me like the unexpected return of the stick I threw, which turned out to be a boomerang.
But then I started to smile, because I like to smile a lot.
(And I know, what my brain tells me.)
It's not that I am looking for some hidden camera to see if I was tricked. It's about knowing what I did.

What do I like?

As a principle software engineer, but perhaps just as a developer, I like to learn. Learn about possibilities, learn about how to solve a more than complex puzzle, learn about estimating the impossible. Did I mention growth? I like to grow from the "expedition into the middle-earth" and take the knowledge with me into the future.
I also like to get paid for doing my job, which of course was the case.

If I'd asked any developer about their ideal job, this outcome would probably be the same:
Learning, gaining knowledge and get paid.
so... wait a minute (this is my brain talking to me)
You did your ideal job during this "expedition" and yet you feel bad about it?
In a couple of years you will think about this again and remember this "bad moment" as it was just a moment in your entire life. You've grown so much since then.
Come on, get a grip of yourself and be happy.
Find your next expedition and ....
Smile !!!