Forem: Adrian B.G.

A/B tests for developers

Adrian B.G. — Fri, 04 Jan 2019 13:16:07 +0000

This article is only the first part of the main story from my blog: A/B tests developers manual.

Best case scenario: Your [product owner,boss,producer] found out about A/B tests and you are here to learn more about how to implement them.

Worst case scenario: Your product is already a mess because of A/B tests, and you want to clean it up.

Either way, I’m writing this article so you do not have to repeat our mistakes.

In the last 5 years I worked mostly in the gaming industry. I had to implement hundreds of A/B tests and I learned it’s a powerful 💪🏾 tool. In the same time I learned that if you do not pay enough attention, your code transforms in a spaghetti 🍝restaurant.

I wish there would be a single simple 🎯way to implement A/B tests without making a mess in your code,but I don’t know any. By definition your code needs to have multiple versions of the same behavior.

Intro ⚓

Skip this block if you are already familiar with A/B testing.

A/B test is also called multivariate testing, A/B/C/D testing, split testing or bucket testing. It is an iterative process of experimentation, that helps you find out what is better for your product. More formal definitions: here and here.

Your product (game, app, website, shop …) can grow in 2 ways:

a person says “feature X will improve the Y KPI by 30%”, and you implement this feature. We, the mortal humans cannot predict the future, we can only guess it.
a person says “this X feature is the best”, other says “Y feature is the best”. You implement X,Y,Z variants and measure exactly which one is better.It may be none, one or more of them. You keep the best versions and improve them with further split tests.

The tests are done on smaller, but representative samples of users. This way you can test multiple versions in parallel and mitigate the effects. You do not know how it will affect the user behavior, so you want to minimize a possible damage.

You start by distributing your users into buckets, each bucket will provide a different user experience and you collect the data and analyze the impact of each version. You choose the best bucket and roll-out it to all the users. The process is more complex than this, but this was the main idea.

Basic example: you want to find out the price point of a new product you can make a test. 50% of the users are left out of the test, with $5 price. The remaining 50% are split equally into 5 A/B test versions ($5 control group, $10 version 1, $15 version 2, $20 version 3, $25 version 4).

Professional commitment ✍

I, the developer, swear not to be biased. We cannot allow any personal/technical difference or issue to affect the A/B test result (user behaviors) for example:

loading— make sure all the resources are loaded from the same source (CDN/hosting), so the network times are similar for the users
size— make sure the file sizes are the same for all the versions, 1 button has 1mb background and the rest are 50kb

…you get the picture. All the users must start from the same premise. If a technical irregularity appears please let the team know and repeat the test.

We, the developers are in charge of the implementation and technical details, we must guarantee the technical unbiased and act as a firewall.

"I, the developer, swear to collect the right data from the users and do not mess with the tracking." Easy to say, hard to do. The main idea is that the A/B test result is based on the KPI events, by observing users behavior during the split test. Data anomalies can mean “bugs” or a clear “winner”.

Usually when something is too good to be real, it is a bug.

A. Good to have 🛠

It’s easier to work with a better codebase. If your code is already a mess, the A/B tests will multiply it.

Modules If the code is already split to modules (modules, files, classes) most likely you will only need to modify 1 portion of your code, if not … then you must like spaghetti.

Parameters and configs. Everything in an A/B test is reduced to a parameter value, usually a string value. If your business logic has already the tested parameter as a variable or config value its very easy to implement the test.

No magic values If the parameter you are testing is hard coded in multiple places let this be your learning, do not use magic values (magic numbers are the most common mistake).

...

To read the full story you can continue on my website: A/B tests developers manual.

Remember to share and subscribe if you learned something new, Thanks!

Software Engineer Productivity: Workflow

Adrian B.G. — Fri, 21 Dec 2018 23:09:56 +0000

Over the years I managed to reduce my daily coding time under 5h out of 8h and maintain the productivity using the following techniques. And no, I cannot scale horizontally, if I work more I don’t deliver more quality code.

This article was originally published on my blog: https://coder.today/6-workflow-tips-for-a-software-engineer-productivity-ba35d9f7e267

Productivity tips is a series of general purpose articles for people that code. The tips should apply to any environment,framework,language or platform.

This story belongs to a multi-stream series:

Productivity: coding — concrete coding tips & tools
Productivity: workflow - this article

Hourly brakes 🦋

If you are a smoker skip this, you probably are already doing it.

A few minutes brakes for each hour are good for my health (by walking) and increased my productivity. If you have a rest area with games or other interactive stuff go there, or just walk around the office. I am a bad example 🚭, I am the biggest “passive smoker” from the office because I often make these breaks in the smoking area.

If you are applying the pomodoro 🍎 technique you are good already, just that my timer is 50m instead of 20m, at least for my current workflow.

”One of the best programming skills you can have is knowing when to walk away for awhile.” — Oscar Godson

Developers type a lot but, but we need to think more and code less.

“The Zone” is a risky addiction 🤠

The hourly break has another advantage, it keep us from doing “the zone” ⏰ overtime, that will most likely hurt the code. When you are deep inside the problem is hard to see the edge.

I think that “the zone” is our natural environment. You will find a developer there most often writing #happycode.

But there is a threshold 🚸, you can focus “too much” on the problem/code, as a result you will code “in a vacuum”, “in a bubble”. When you exit the bubble you realize you forgot stuff because your entire attention focus had a very small scope:

“I forgot we have that functionality in X module, I just wrote a new one”
“I forgot that I should have talk to X for this task” (edge cases or implementation details)
“I skipped the Unit Tests, I will do them tomorrow, they slowed me down”
“I used something that is not supported on X platform, I forgot”

Basically you are prune to do all the mistakes that usually appear when you are in a hurry, pressed by time or when you are not paying attention.

Most of the times, “the zone” give us the “productivity illusion” 💨.

I am 100% sure you will write more code, but what about it’s quality? To really see if the code was better you have to measure the time spent writing the original iteration + time spent to fix it + time spent to debug it + the effects it’s bugs, during the entire code production lifespan. In the “zone overtime“ you have a quick first iteration (“finish fast”), but the later stages duration are heavily increased.

“The zone overtime” also appears when the developer uses his brain as a Heap 🗒, by trying to understand a piece of code (usually a rotten code to replicate a bug). This can be mitigated in more ways, one is: refactor the code first and use a proper debugger.

Among the years I found a few scenarios when “the zone overtime” is good for business:

When the entire project fits in your head — more exactly when you are the only developer or the project is small.
When you are doing a prototype or the first version of a library.
When you are working on a single but complex algorithm.

Other than these cases, I didn’t found any good scenario for a developer to spend more than 40–70 min “in the zone”, without making at least 1 mistake that invalidates part of the code resulted.

PS: this is a generalization, I’m sure there are exceptions.

Distractions 👏🏽

Observe yourself, remember what caused the most interruptions or search for patterns. Try to fix these, yourself if you can or talk about them in your periodical team retrospective meetings.

It can be the staging server crashing ,the constant social media checking or the noise of the coffee machine. The distraction can take many forms.

Easy to say than done: don’t login on social networks 📪.

If your colleagues keep disturbing you just print & stick this to a nearby wall:

Jason Heeris

@detly

This is why you shouldn't interrupt a programmer: twitpic.com/dj27dh

09:20 AM - 28 Oct 2013

2462 1031

Learn to detach the debugger — your brain 😨

If you are stuck more than 5–15 min on a single issue run away. You are in a maze and you do not realize it 🤔. You will hurt yourself and most likely the code. By wasting time you are hurting the team/product/employer too.

Step away 🏃🏿, think more or not at all, talk to others, detach the debugger and return later with a broader perspective.

Use your brain as a queue 👩🏾‍💻

How to handle incoming requests, while you are working:

If busy {add the new task on the TODO list}
If they contacted you in person {recommend them to use an async communication system like chat/email next time}.
Reconsider the priorities when the current task is over

For this technique to work you need to divide your daily tasks in smaller chunks and do hourly breaks.

Notifications 👋

Good notifications: can be used to eliminate a distraction, example: instead of checking the team chat every 10s you can setup @mention notification alerts.

Bad notifications: you received a newsletter.

Silent as many notifications as possible. Do whatever it works for you to save more time. Tip: Smartphones have DND intervals, so they remain silent during the nights and work times.

Mobile 👩‍💻

Be mobile, use a laptop. Never mind, of course that you already using a laptop, erase this.

Thanks! 🤝

Please send me your feedback so I can improve the following posts.

Other resources 📚

The Perils of Future-Coding

Idiomatic Go

What Is 'The Zone' Anyway?

What are the 5 tips of a productive developer?

The Pomodoro Method for Increased Productivity, Explained in 37 Seconds 🎥

I was a (social) Game Developer for 5 years, Ask Me Anything!

Adrian B.G. — Fri, 30 Nov 2018 20:44:50 +0000

I recently "ended" my professional career as a Game developer (as in will no longer work for game studios). Being a full stack web developer I ended up doing Social games, On Facebook platform, web and mobile (Unity 3D), mostly card games.

I am also a Unity(3D) Certified Programmer, I develop games as a hobby, I have a passion for Game design, I was a Gamer (MMO addicted, clan leader) and I love any kind of game, I even designed a few small Board Games in my free time. I played with many related gadgets at work (as a dev): Sphero, Leap Motion, VR headsets.

Before being a game dev I worked 5 years in e-commerce and now as a Cloud Engineer.

TTL: I will try to answer all questions for ~2 days (1-2 Dec)

From monolith to cloud: Auto Increment to UUID

Adrian B.G. — Tue, 20 Nov 2018 19:20:25 +0000

This article was originally published on my website https://coder.today/software-engineer-from-monolith-to-cloud-auto-increment-to-uuid-a62f92f387c4

From monolith to cloud series 🌩

A series of articles designed to help developers switch from a monolith to a cloud mindset. The web is full of very good tutorials and examples on Why and How to make the switch, so I decided to focus on the small details.

This article focuses on the developers who worked only with numeric auto increment primary keys and need/want to switch to UUID’s.

ID int NOT NULL AUTO_INCREMENT 🔢

Entries in a relational database like MySql/SQL/Oracle are usually identified by an incremental, unique (to table) number int(2232). The server collects the parameters, sends an INSERT(...) statement and the database generates a new ID (the next incremental value) and returns it.

When you begin to scale you may end up with a bottleneck, your MySql master instance, because that is the only entity from your system that can generate a unique identifier.

You already know the benefits of an auto increment PK’s , here is a list of its limits:

need to have access (through a pipeline/API/server/connection) to the master instance
you depend on 1 instance from 1 server from 1 data-center (latency, availability)
all the write operations are done in a single location (most of the cases), this leads to a hardware limitation of generating new ID’s
easy to spoof ID’s (bonus: you can easily find out the number of customers)
MAX_INT — it’s a long shot, but still …worth mentioning

All of these issues can be mitigated to a degree (ex MySql sharding). To fix all of them you can use UUID’s.

UUID 🍱

An universally unique identifier (UUID) is a 128-bit number used to identify information in computer systems. The term globally unique identifier (GUID) is also used. The size of the UUID can differ on implementations.

UUID can be used in relationship databases and viceversa, auto incrementing in NoSql.

In its canonical textual representation it’s a 32 hexadecimal (base 16) digits, displayed in five groups separated by 4 hyphens: 123e4567-e89b-12d3-a456-426655440000.

Some bits represent the UUID version (the algorithm used to generate it), others the variant. Starting from a UUID format you can even add your own logic (bits representing some aspects of your business logic).

To generate the random bytes of the UUID more factors are used to ensure a better entropy like the timestamp and the clock sequence. For more technical details you can read the Official Protocol paper of the UUID. Beware of the implementation you use, not all the libraries respect the standard. I have found some implementations that just use the predictable pseudo-random function found in every language to generate a number.

Basically the UUID/GUID is a random ID, the values are not sequential and anyone can create a new ID.

There are a few advantages over a linear incremental value

it’s easier to shard
it’s easier to merge/replicate. There is no universal order.
more flexibility — you can generate UUID’s outside of the database, delegate to servers or clients, environment/platform independent, but you may lose some data integrity
you can even allow offline register (and sync when available), but you will never have a full DB snapshot (because of the out of sync clients)
scaling —UUID has a large…r limit of ID’s than an INT
you know the ID before the insert, it can simplify the logic/flow
the UUID can have your own format, you can split it in 4 numbers and each one of them represent something else, for example if you group the users from 20 websites, the first number can represent the application.

The new mindset 🤕

You will have to get used to a more difficult debugging process, UUID’s are impossible to remember. The trick of memorizing the first or last characters will probably not work.

The change is hard (any change) for the human brain, you will try to fight it, most likely using cheap reasons: an INT is prettier, occupies less storage space. Embrace the change for a greater good, is part of the software evolution.

New factors in town based on async: time and location. An UUID may exists somewhere, but is not synced YET or it’s in other shard. There is no universal sync view anymore. Your production is in chaos now, distribution can mess up your karma, don’t feel frustrated, is just another way of doing stuff.

A few bad things can result of distribution: duplicate or lost data. This means extra coding and extra meetings to explain why and how to the product owners.

I do not think that UUID are universal better or worst than incremental ID’s, they just serve different purposes. But …

I will recommend using UUID’s for any new apps, the current state of software needs demand iXXXt (scaling apps, multiple type of clients & platforms, offline apps…).

Before you go, I recommend reading some more:

Software engineer — from monolith to cloud: think small How does the switch to containers and microservices affect our mindset and workflow
Pinterest has a fleet of MySql shards and uses local auto increment ID, in the end they replicate a NoSql (generate UUID’s and store json). Same as Twitter and Facebook they used the wrong technologies from the start and tried to patchup things as their product got bigger.
The caveats and issues with UUID in production: UUID or GUID as Primary Keys? Be Careful!
General comparison between the most used NoSQL storage solutions:
Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Couchbase vs Hypertable vs…
Generating Globally Unique Identifiers for Use with MongoDB
ObjectId - Operational Segregation in MongoDB
When are you truly forced to use UUID as part of the design?
There are specific scenarios when do you need incrementing ID’s, like a queue: Auto Incrementing Keys in NoSql

Thanks!

Please (like)👏 and subscribe if you learned something new. Send me your feedback so I can improve the following posts.

Advanced JavaScript panels

Adrian B.G. — Mon, 12 Nov 2018 22:29:38 +0000

Previously published on my blog https://coder.today/advanced-javascript-2017-talks-fcc3717a9f3f. It contains some of the best (in my oppinion ofc) talks from 2017 JS conferences about internals and how things works. Even if they are "old" they are not outdated, hopefully you will learn many new things about your favorite language!

The success of JavaScript filled up the internet with low-entry level resources, introductions and tutorials. I do not complain but as result it is hard to find advanced topics about JavaScript.

I watch a lot of YouTube videos so I decided to share with you the most recent videos I found on advanced topics of JavaScript. Let’s dive in:

V8, Advanced JavaScript, & the Next Performance Frontier (Google I/O ‘17)

This video will give a very good insight into the browser’s JavaScript engine, how it works and what are the optimization techniques. In the 2nd part he will go into the details and explain the new V8 engine, why & how it was made, and how it will improve our NodeJS and Client apps. Performance vs fast startup & low memory vs max optimization.

Compiling for the Web with WebAssembly (Google I/O ‘17)

There are many panels about WebAssembly, but this one is more relevant nowadays and I think it does a better job explaining how & why we will use Native libraries in the JavaScript apps. You will also find out the limitations of working with WebAssembly.

Arindam Paul — JavaScript VM internals, EventLoop, Async and ScopeChains

If you skip the first 5 minutes and the hard to understand English accent you will learn a lot about JavaScript run time, memory model, scope chains and closure.

Node’s Event Loop From the Inside Out by Sam Roberts, IBM

This is a very technical glance into the NodeJS event loop. Find out what makes JavaScript “tick” on the server side.

Denis Radin: Applying NASA coding standards to JavaScript | JSConf EU 2017

Will you fly an airplane that is run by JavaScript? You will learn about some rules a scientific facility learned & applied while making a very stable & safe system using JavaScript. They work with NASA & appreciate clean, fast & stable code.

Advanced Async and Concurrency Patterns in JavaScript

It is only an introduction in the Concurrency topic, the full story last for many hours, but this is a good summary. Kyle Simpson speaks his minds about Promises, Generators, async & other cool new stuff from JS6.

Anjana Vakil: Immutable data structures for functional JS | JSConf EU 2017

Immutability isn’t a new topic in JavaScript, immutable.js is here for a long time. But I think most of the junior-mid JavaScript developers do not use this paradigm because they do not understand it, and what you do not understand you are afraid. Also I think she is a good speaker and a good panel. You will learn some techniques used in functional programming, and they will improve your code quality!

Thanks! 🤝

What can you do with JavaScript? 🤖Resistance is futile.

Defensive programming and reliability. Analysis of post mortem NASA software.

Software Engineer Productivity: coding

Adrian B.G. — Tue, 30 Oct 2018 10:16:11 +0000

This article was posted on my blog https://coder.today/software-engineer-productivity-coding-75fa21d3804. This is a revised version of it.

Productivity tips is a series of general purpose articles for people that code. The tips should apply to any environment, framework, language or platform.

This story belongs to a multi-stream series:

Productivity: coding — this article
Productivity: workflow — behavior & environment tips

I am lazy but I care about the most precious currency we have: time ⏰. That being said I want to share some tips that improved my efficiency.

TL;TR 🕮

automate everything
learn & customize your tools
store your snippets & other tips

Automate commands 🤖

Observe yourself, find out what commands you type multiple times each day and shorten them, group them if needed. If you used only a git visual client you should learn the underlying commands.

almost everything. I usually deliver the last command from an important chain manually (ex test&&build&&copy&&upload then manual deploy).

Using aliases you will do fewer human mistakes, and deliver more. Eliminate the boring steps from your daily routines.

The downside of custom aliases are: inflexibility and cognitive load. Meaning the aliases are only in one environment (ex your work laptop), and you have to remember them.

I recommend also to git auto complete or frameworks like bash-it, they are usually developed with some logic (in command naming) and they are easier to remember. Search for bash helpers built for your language, framework, platform.

Not having to type the current git branch name saved me over 1M characters typed and using aliases I avoided many possible mistakes.

Even if you do not use aliases, practies like using HEAD instead of master it will make you less prune to human errors, example get use to type git pull origin HEAD instead of git pull origin master.

Even if you are on Windows you can use Bash, I use Git bash from Git for Windows.

Bash tip: learn to use bash history, especially CTRL+R.

Customize your tools 🌂

You spend a lot of your time working with the IDE, bash and other tools. Customize your tools based on your needs and feelings. You should 💙 using them, otherwise your job will suck. If you hate something change it now (like a missing keyboard shortcut, display colors), go, on, fix it and return, I’ll wait.

I like to my bash to be cheerful, colored, and all my editors have black themes.

Customize your environment

Make your environment work for you, not against you. The most basic example is to add in your global git .gitignore file your IDE files. You will save time, by doing it only once per machine, instead for each project.

#~/.gitignore
.idea/
.vs/

Observer yourself, what other system settings were problematic in the past, firewall rules? python versions? Unity3D local cache? Solve them once it for all.

Learn your tools 🏧

As you learn new development techniques, new frameworks, paradigms and design patterns you should learn the tools that were designed for you.

Text editors are for editors that write text, IDE’s are for professional developers. Why? because we care about our time, we are professional, we are getting paid for our time, we want to deliver as much as we can in this time. IDE’s are heavier, because they have more builtin features designed for us. PS: a text editor with tens of plugins like compiler/git/linter integration is an IDE.

Improve your skills by stepping out of your comfort zone.

Upgrade your IDE and learn the new features,most likely the will fit your needs. Search for a new IDE once in a while, you may be surprised.

A big productivity gain you’ll get by using more keyboard shortcuts. The “expand selection” for example saved me a lot of time along the years. Other examples are live templates and multi-cursors.

One live template I often use is debugging messages like if IsDebugBuild {print("msg", var)} or try/catch expressions.

You may use some regex replaces now and then, but sometimes multi-cursors are simpler and more efficient.

Good enough scenarios for multi-cursors: hard coded data, Enums, templates and everything that is boring mostly.

The latest feature I saw for an IDE is a machine learning helper (Codota). It knows about all open-source code, design patterns and mistakes made by other devs.

Automate your dev env 🎰

Last time I got a new job it took a few full days to make a full working dev setup for a big project. It doesn’t have to be like this, there are tools designed for these kinds of automation. Boxstarter (I haven’t used it yet) can build your windows dev env in a manner of minutes, from a snippet of code.

I started to use Docker for these kind of things. Using simple CLI instructions you can install most of the tools/services/software you need. No need for compilation, search for packages or binaries.

Example: a persistent mongo-db instance with a specific version and a visual administration tool.

Do not be a 🔨

“To a man with a hammer, everything looks like a nail.”

If you need a system script do not use JavaScript for it, learn a bit of bash, ask a coworker or stack overflow how. If you need a small browser animation learn CSS3. You get the point. It will take you a few extra minutes but it will open a huge gate 🚪 in your future.

Most of the engineers have this problem. I used to make everything in PHP, CS graduates get stuck in Java or CPP. It’s human nature, the designers do the same mistake with Photoshop, the QA usually with a bug tracking software like Jira.
The idea is to expand your tool set, baby steps, 1 small script at a time.

* on save 💾

Automate everything you can at “save file” action, the most common thing I activate is “auto-format on save”.

Auto-format enables me to write quick & dirty code, I do not waste time on indentation, commas, semi colons or other beautification elements, that’s the IDE’s job.

Auto-format also eliminates more human mistakes like actually forgetting to format and generate git conflicts or stupid commits afterwards. You and your coworkers will be more productive.

The IDE Is here to help you, let it do IT’s work

You can make your own mini pipeline on save, depending on your needs: compilation, error checking, unit tests, hot push code (for web dev, js &css), you name it. For more time consuming scripts I use manual triggered Tasks.

For example, in Go I activate “run tests & coverage on save”, having the console open at all time improved my efficiency. I can work faster without being afraid to refactor, each minute I know that the code is fine, or not.

TODO 🔖

Use more //TODO's in your code. Do not brake your current flow, do not exit “the zone” by solving an adjacent thing. If you hit a small bump dump a TODO and return later with a better, larger perspective.

Snippets 📚

Do not store snippets on chat, like a few of my friends did, you can use Gists or other similar services. Your own repo, a doc in google drive, you name it. The idea is to store snippets of code that prove useful and were hard to find/achieve. It will save you time, your future time.

Share 1-n

We can extrapolate the previous solutions to other problems. If a coworker has a problem, and you are going to help him, most likely there will be other coworkers that will need that answer, in the future. As a thumb of rule, use 1-n persistent communication channels, sharing is caring.

Keep a project minimal documentation, write articles/stories, even if they are just internal. Record your internal meetups,
the idea is to not solve problems with emails/private chats, share the knowledge so other can access it.

It will save you time because you will not have to explain/solve the same problems twice.

Automate your API calls 📝

If you work with API’s of any kind I recommend using a tool like Postman. If you keep your calls in one place, you can even automate them (in a continuous delivery pipe), share them with your coworkers (as config files) and validate the responses (with a json schema lib). I used to keep the tests in the same repo as the server API source code.

Thanks! 🤝

Please send me your feedback so I can improve the following posts.

Other resources

Productivity: workflow — behavior & environment tips

The Joel Test: 12 Steps to Better Code

Barking at the right tree (1000 followers)

Adrian B.G. — Wed, 17 Oct 2018 15:40:05 +0000

New milestone

Hello!

I've reached a new (biggest so far) milestone in my ... publishing career, sort to speak ... more exactly 1000 followers on dev.to

BG Adrian ⚛

@b3at

All these years I was "barking at the wrong trees"
Twitter - 10 years - 226 followers
Medium - 1 year - 234 followers
@ThePracticalDev - 1 year - 1020 followers 🕺
New articles .... here I come!

15:07 PM - 17 Oct 2018

0 2

Like I said on twitter, I was "writing" on the wrong platforms all this long.

I made some experiments and studies on different social media channels, from instagram to linkedin, and I could throw poo all day long, but you can just look at the top content to see for yourself.

Dev.to

Here you do not have to be a super star in order to have people read your content.

I will treat this event as a good sign. I will continue to write and help as many devs as I can. I have some big plans for next year, the people I am currently helping encourage me as well:

new blog stack (transit to Hugo, medium no longer fits my needs, 8s loading time ...)
finish my 12+ series of Pragmatic Docker tutorials and real examples for developers
start a Youtube channel (I have previous editing experience, I just need some hardware and to improve my English), mostly for Programming and Cloud basic tutorials. I want to cover all the missing topics from Bootcamps and courses.

Context: now I am helping junior developers on 1-1 sessions, and I realized that I can help 1-n people just by recording the material, because most of the problems they encounter are common.

finish 2 chapters of my book (long stretch, probably will not happen, but hey ... wishful thinking)

Thank you !!!

Fast priority queues in Golang: Hierarchical Queue

Adrian B.G. — Sat, 13 Oct 2018 14:49:49 +0000

This article was originally published on my website https://coder.today/fast-priority-queues-in-go-hierarchical-queue-86daf6a7553a

Writing O(1) high performance data structures is a multi-post series:

Hierarchical Queue (this article)
Hierarchical Heap
Ladder Queue (soon)

A priority queue is an abstract data structure, used to store values (any data) with a priority. You can insert data at any time with any priority, but you can only take out the value with the highest priority.

Magic

When to use priority queues ⁉

I confess, I didn’t use many of them in my apps, but there are a few problems perfect for it:

triage — any processing you have based on a score/importance
ordering — when your data has a specific order of processing/analysis, ex: process the DAU KPI before MAU
Event driven simulation — priorities can be timestamps
Graph searching
Load balancing — the priority is the inverse of the load

Baby steps

I started a long last relationship with Go. I didn’t want my training to be in vain so I began to look for high performance data structures to implement (that aren’t already). After a few searches I ended up with this image, snippet from a book

The 2 bottom lines attracted my interest, they are O(1) multi-structure priority queues. First of them is the Hierarchical Heap, which is based on a Hierarchical Queue, so let’s dive in.

Hierarchical Queue

I couldn’t find any open-source code or pseudocode for this structure, if you find any issue please let me know.

“When the priority values are limited to small integers (e.g. digital images often have 8-bit or 12-bit integers as pixel values when they come off the imaging sensor), it is possible to allocate a FIFO queue (bucket) for each possible priority value. An array contains a pointer to each of these buckets, and when enqueueing an element, the correct bucket can be directly indexed using the priority value. Both enqueue and dequeue are O(1) operations.”

Papers used for the algorithm and snippets:

It is a simple structure, very fast but it has a few limitations.

It has only a limited number of priority values (in my implementation is uin8(0–255)). Each value has it’s own queue, so increasing the number you end up with a memory overhead.
Once the highest priority queue is empty, it is removed and the next queue can begin to empty, and it cannot be created again. This means we must fill the queues before we start to dequeue, otherwise our performance will decrease , example: The dequeue process has only 1 priority left (255) and we enqueue other elements. All the new elements will be pushed in the 255 queue (because the 0–254 are empty and removed).

Go code go

Priorities can be uint8 and values interface{} .

I decided to remove the 2nd restriction, I think it made the structure too restrictive for most of the real world scenarios. I managed to keep a very similar performance (≤ 50ns per operation). I cached the highest priority that has values and start to dequeue from there. In some cases, the Dequeue process can be O(1 + k) , where K number of empty queues, but overall is amortized if the priorities are well balanced.

This means that the life of the structure is extended, it can be reused and higher priority values can be added even after the dequeue process started.

In the first iteration I used linked lists as queues (used golang list.List), and the average operation time was 150–200ns. I ended up using a faster structure (thanks to a suggestion of Egon Elbre). It reduced the operation time down to under 50ns (it’s a pretty fast damn list), see collection/deque:

karalabe / cookiejar

A contestant's algorithm toolbox

CookieJar - A contestant's toolbox

CookieJar is a small collection of common algorithms, data structures and library extensions that were deemed handy for computing competitions at one point or another.

This toolbox is a work in progress for the time being. It may be lacking, and it may change drastically between commits (although every effort is made not to). You're welcome to use it, but it's your head on the line :)

Installation

To get the package, execute:

go get gopkg.in/karalabe/cookiejar.v2

To import this package, add the following line to your code:

import "gopkg.in/karalabe/cookiejar.v2"

For more details, see the package documentation.

Algorithms:

Graph
- Breadth First Search
- Depth First Search

Data structures:

Extensions:

fmt
- Scan and Fscan for int, float64, string and lines
math
- Abs for int
- Min and Max for int, big.Int and big.Rat
- Sign for int…

View on GitHub

Because concurrency is important in Go, the structure has a mutex. It can be used manually or automatically (each function call blocks the mutex).

To be as generic as could be, the queues can store any data type interface{}.

Packing up

The Hierarchical Queue structure is part of “priorityqueue” package, it has 100% test coverage, examples, benchmarks and documentation.

bgadrian / data-structures

Abstract data structures Go packages, built with performance and concurrency in mind to learn Go.

Data structures in Go

I am writing a collection of packages for different data structures in GO.

Why? To learn Go, practice basic structures and learning to code fast concurrent algorithms.

All the packages have 100+% test coverage, benchmark tests and godocs. Tested with go 1.9.

!! Warning This library wasn't used in production (yet). !!

priorityqueue

A collection of performant, concurrent safe, complex abstract data structures used for priority queues.

Priority queue is an abstract data type which is like a regular queue or stack data structure, but where additionally each element has a "priority" associated with it. In a priority queue, an element with high priority is served before an element with low priority.

Hierarchical Queue description

An O(1)/O(1+K) priority queue (very fast) implementation for small integers, that uses an assembly of N simple queues. It is optimized for large amount of data BUT with small value priorities…

View on GitHub

The next data structure is the Hierarchical Heap, which is based on the Hierarchical Queue, removing it’s limitations for a small cost of performance.
Fast priority queues in Golang: Hierarchical Heap.
Hierarchical Heap is a very efficient Priority Queue O(log n/k) for large data sets.

Contribution

It (was) my first library written in Go, I never used these algorithm in production, so please correct me if I did something wrong.

Thanks! 🤝

16 free tools & services that any developer should use

Adrian B.G. — Tue, 25 Sep 2018 17:25:24 +0000

This article is a 2018 revised version of my original article https://coder.today/free-tools-for-any-developer-707761504f7f

Free tools & services for any developer

I want to thank all the service providers and developers that allowed their product to be used for free. What is the best way to repay them? advertising.

I will share the most useful free & open source tools that I use in my work & home environment. The tools can be used in any env/ language/ framework/ project. If you are a devops in a large corporation or a freelance developer this article should be a good start.

Probably you know about most of the tools already, hopefully today you’ll hear about a new one.

I will skip chat clients like Slack, services such as Stack Overflow, twitter, YouTube and other Google products. I use most of them, everything in my life starts in a Spreadsheet. Inbox & google docs are awesome for prototyping, ideas, formulas, project docs, working alone or shared across the team.

Class A — Productivity+ for devs, ops, QA, IT

Visual Studio Code — Code editor, closer to an IDE than the rest I would say. I used it only for the last few weeks, but it became #1 right away, for me it replaced Atom/Sublime/Notepad. Visual Studio works with any language and config files (json, xml, ini …). It replaced MonoDevelop for Unity3D, I do TDD in Go, code in Java, C# etc.

I was very skeptic at first, being a Microsoft product, but I threw everything I had and it handled it perfectly. It just runs fast and across platforms. I will still use my PHPStorm (webstorm) for large projects though (for now). It has a very good learning curve, one of it’s features is the KeyMap, I use my Intellij keyboard shortcuts in VSCode, how cool is that?

Trello — Task list, to do list, shopping list, to visit places list, stuff to do with the kids, notepad for ideas, task management for small projects/teams, bug tracker, feature planning, should I go on? Trello runs across all platforms and the free tier is good enough. Also it has many integrations & plugins.

GetShareX — Screenshot and Screecast of your dekstop/window/region. I use it for bug reports, screenshots, examples, record “how to …”, etc. It’s similar to Snagit/Jing but better (is open source). Records video or gif, has scriptable actions like auto upload to imgur after a screenshot and a lot of other features. For Linux I use a mix of Shutter and Ubuntu screenshot builtin tools.

Toggl — Track your time spent working. Time is a valuable currency, so I keep tracking how I spend it. Toggl has a powerful idle detection (the measured time will be more accurate), browser, desktop and mobile clients. Supports multiple projects and even generates reports (good for freelancers/remote workers). It’s all about calculating the resources invested in a project vs the gains.

CodeWars.com—algorithms. If TopCoder is too serious for you, and CodeCombat too childish, you will love Codewars. It is all about keeping your brain in shape. If you are stuck at work doing the same algorithms over and over or fixing legacy code all day you need some good problems that can be solved in a few minutes. Supports multiple levels of difficulty and most programming languages. It’s all about improving your programming skills.

SourceTree — git client. I usually use bash for git commands, but when I need an UI I use SourceTree. It’s cross platform and easy to understand, we used it to teach non-technical team members how to use git. If you are looking for something more simple & friendlier check GitHub for desktop.

Storage — I use Bitbucket for free unlimited private git repositories and github for open source ones and gists (code snippets). Dropbox/Google drive for other resources and backblazeto host images/videos.

Vimium - This tool will improve your browsing speed, searching in history/bookmarks, closing tabs, opening links you name it. It follows the Vim mantra, everything done with the keyboard. Works on Chrome and Firefox.

Mentions:

GitLab - I will soon move to Gitlab and most probably it will replace BitBucket (for me) because of their rich interchangeability and devops features.
InterviewBit.com If you want to practice for your coding technical interviews and system design I strongly recommend. It has the best quality of UI,UX,problems, solutions and tests.
UTime — chrome extension to convert UnixTimestamps/dates
Postman — Web API tester, it’s simple awesome. You can save&share templates and the paid tier supports JS automatic tests!
Bash for windows — from Git for Windows, MobaXterm (if you connect often to remote servers) or Native Win10 Bash
Fiddler(windows) a very rich proxy for web development. I use it to debug web apps (their requests). CharlesProxy is better and works for iOS too but it is not free.

Class M — Media

Gimp2 — It is more than enough for my graphical needs. I use it for photos editing, creating simple GFX for prototypes, resize/transform images and create collages (different sizes, transparency etc). It took me a long while to get used to it (coming from Fireworks), but was worth it. Tip: activate Single Window Mode.

Canva — When you need social media posters, banners, collages, headers or logo and you lack a designer, Canva Team will help you. It has a good free tier for small projects and it’s awesome. It has thousands of pre-made templates and photos that you can use freely in your commercial projects.

Piktochart— One word, infographics. Piktochart knows all about them. It doesn’t have such a great free tier, but it’s an awesome start.

SoundCloud —Say it, you cannot work w/o music. If you do not care about latest tracks with half naked women, and you just want to listen to some good old music SoundCloud is all you need. It has even technical podcasts, or science like StarTalk Radio.

Mentions:

Inkscape — Open source vectorial drawings, it has more features than you would think.

Thanks for reading! I hope that you found something useful around here.

Golang, it was love at first sight.

Adrian B.G. — Tue, 18 Sep 2018 13:50:03 +0000

The article was originally posted on my personal website https://coder.today/go-go-go-flash-bang-d66f4c42eb7c. Meanwhile I have developed a few libraries and small apps in Go, even contributed to Go Tour and I changed some of my opinions, but I will keep this article as it was, I like immutability.

“Go Go Go … flash bang”

When I hear “Go” I always have this flash…back.

Intro

Usually, when people have free time they do some R&R, but me, being weird and all I like to learn, experiment and play with new things. I have enough new hobbies (earlier this year I learned to pick locks & started to do mobile photography), so I decided to step up my lost programming mojo.

If {you are like me (web or mobile developer, avoided low-level languages and concurrency for any reason) keep reading,
else you already know the truth, skip to the end and share the article 😁.}

I was away from the Web development for a couple of years, doing Unity3D (C#) & mobile, so naturally I began to catch up with the latest web trends. I made some small projects using MeteorJS (node +hot code +~mongodb + other steroids), React and D3JS. By switching to nodeJS I ditched PHP, working in only 1 language, with shared code, I greatly improved my productivity.

If {you are a JavaScript developer check this hard-core 2017 playlist}

Go — the programming language

Reading about new architectural styles, scalability & cloud I found a pattern, most of the big companies (www backbone) switched in a way or another to Go.

When reading about it I found that we had a lot in common (me & Go), it’s a good start for any relationship.

We are both very opinionated
We like to work with JavaScript (building an API is easy or they can be complementary)
We deeply care about developer productivity
We like KISS. Simple is better (vs cpp)
1 way to do it (idioms, code formatting). Having more ways complicates the project, the developer’s brain, team work and compiler’s life.
We suck at RegEx (but the contributors are working on it)
We hate bugs (so we enforce coding rules)
We like to multi taskand work with big data.

If {you are a Gopher please help me learn faster and review my code:}
bgadrian/data-structures
*data-structures - Abstract data structures Go packages, built with performance and concurrency in mind to learn Go.

For me Golang has a few more advantages:

It is the perfect mix between low & high-level languages. I want to deliver fast, get things done, so taking out the garbage is not for me. The steep learning curve keeps me away from low-level.
It has integrated Unit Tests — I was looking for a reason to start practicing TDD.
Community — after more than 60 hours of conferences panels and reading forums I can say it is a nice community.
I want to be more involved in the open source community, this is my Go sign.
It has identity & a cool mascot
It creates a pleasant environment (good tools, compiler warnings, idioms etc).
WebAssembly Learning a compiled language will help to gain an advantage & have fun writing JavaScript and WebAssembly modules in the same time.

Most of the “anti” Go reasons I consider them to be “pro” reasons, here are a few examples :

missing features like Exceptions- results in better code, who wants this chaos on this hands? .
“When 2 sides argue more than 15 min they are both wrong.” In this case wars took more than a decade, so Go being opinionated is a good thing. No more spaces vs tabs, functions vs objects, for vs while … Developers write the same code, compiler can optimize it better.
overloading — usually creates too much similar code, and there is no polymorphism, so no much use for it when you code in Go style. Overloading makes “cool looking code”, but is harder to debug, maintain and interpret. I think interfaces are enough (1 function does 1 thing, that can be relate to 1 interface) with explicit conversions.

Actual c++ code

code is art/ has no soul — I think the beauty is not in the source formatting, rather at the system/architecture level, solving difficult problems with simple solutions.
most of the reasons can be easily ignored: they are too subjective, don’t understand the Go paradigms, their minds are stuck in OOP never land or they compare it with other languages like C/C++ (which is not an alternative for) or apply to problems outside of the scope of Go.

If {you want to be a better programmer by practicing katas check CodeWars}

Go learned from 30 yrs of development mistakes

There, I said it! It went back to the roots and returned like a Pheonix, better and with a fresh look.

I spent too many hours of my life debugging and fixing bugs. I remember a lot of fixes, mistakes and anti-patterns that I fixed along the years, and a lot of them are addressed in the compiler rules and coding style of Go.

Most programmers are humans, we have feelings and bad days, different experiences and backgrounds, different level of professionalism, technology/env evolves around the code. Bad code will always exists, but if the Language we speak (Go) can fix a lot of issues (even before they exists), shall it be!

If {you want to learn Go check out my first steps:}
First steps in learning Golang
*How am I learning Go as my new main language, what resources & techniques am I using?*coder.today

Done, rant over, thanks for reading!

Thanks to ashleymcnamara, her gopher panel inspired me to start contributing.

Golang jobs: where can you search for open positions

Adrian B.G. — Sat, 01 Sep 2018 22:15:42 +0000

Hello,

I am currently researching the market and I am gathering a list of bookmarks about Go. I thought it will be nice to share them with the community, and find out about other ones as well.

Spoiler alert, there are not many openings for Go (I estimate around few hundreds around the globe, but mostly in the US), but the numbers are increasing each month.

The order of resources is random, I included all the websites in which you can find at least 5-10 positions with a clear tag (usually Go doesn't work), and they are not local (only for one country).

Other problem when searching on general websites (like linkedin) is that you will find many positive-negative openings that wrote ".... development experience in one or more languages: Java, Go, ...", but the position is definitely not on Go.

Other issue is that the recruiters do not use proper tags for their openings and the tag #golang is useless.

Bottom line is that is very hard to scoop the Golang specific positions, the language name and the HR is making it more difficult, and dedicated websites are a blessing for the community.

Sidenote, here is the "official" list of the companies that uses Go, usually in a small extent.

Go(lang) dedicated websites

golangprojects.com
welovegolang.com
chat- the channel #jobs on Gophers slack community
golang.cafe
forum.golangbridge.org - jobs
golangjob.xyz

Tip: these websites are not updated so often, check the date and if the position is still available before reading all about it!

Others

stackoverflow jobs
angel list - startups, you have to be logged in
glassdoor
indeed
remoteok.io
goremote.io remote jobs for all languages
golang.works-hub ~AI based
fullstackjob.com/jobs/go--developer-jobs multi-language positions
cybercoders

If you know more resources please put them in comments, I will try to update the article each week or so with your URLs.

Thanks! See you in the #cloud!

Last update: 19 jul 2019

What books did you used as a display/laptop stand?

Adrian B.G. — Sun, 08 Jul 2018 13:25:12 +0000

I couldn't find a real picture with my old setup, but I've used 2 large C++ books, the Bjarne Stroustrup "bible" and one other that had 1000+ pages.

What books did you used? It's ok, you can come clean now, it will be our little secret.

Forem: Adrian B.G.

A/B tests for developers

Intro ⚓

Professional commitment ✍

A. Good to have 🛠

Software Engineer Productivity: Workflow

Hourly brakes 🦋

“The Zone” is a risky addiction 🤠

Distractions 👏🏽

Learn to detach the debugger — your brain 😨

Use your brain as a queue 👩🏾‍💻

Notifications 👋

Mobile 👩‍💻

Thanks! 🤝

Other resources 📚

I was a (social) Game Developer for 5 years, Ask Me Anything!

TTL: I will try to answer all questions for ~2 days (1-2 Dec)

From monolith to cloud: Auto Increment to UUID

From monolith to cloud series 🌩

ID int NOT NULL AUTO_INCREMENT 🔢

When you begin to scale you may end up with a bottleneck, your MySql master instance, because that is the only entity from your system that can generate a unique identifier.

UUID 🍱

UUID can be used in relationship databases and viceversa, auto incrementing in NoSql.

Basically the UUID/GUID is a random ID, the values are not sequential and anyone can create a new ID.

The new mindset 🤕

I will recommend using UUID’s for any new apps, the current state of software needs demand iXXXt (scaling apps, multiple type of clients & platforms, offline apps…).

Before you go, I recommend reading some more:

Thanks!

Advanced JavaScript panels

V8, Advanced JavaScript, & the Next Performance Frontier (Google I/O ‘17)

Compiling for the Web with WebAssembly (Google I/O ‘17)

Arindam Paul — JavaScript VM internals, EventLoop, Async and ScopeChains

Node’s Event Loop From the Inside Out by Sam Roberts, IBM

Denis Radin: Applying NASA coding standards to JavaScript | JSConf EU 2017

Advanced Async and Concurrency Patterns in JavaScript

Anjana Vakil: Immutable data structures for functional JS | JSConf EU 2017

Thanks! 🤝

What can you do with JavaScript? 🤖Resistance is futile.

Defensive programming and reliability. Analysis of post mortem NASA software.

Software Engineer Productivity: coding

TL;TR 🕮

Automate commands 🤖

Customize your tools 🌂

Customize your environment

Learn your tools 🏧

Automate your dev env 🎰

Do not be a 🔨

* on save 💾

TODO 🔖

Snippets 📚

Share 1-n

Automate your API calls 📝

Thanks! 🤝

Other resources

Barking at the right tree (1000 followers)

New milestone

Dev.to

Thank you !!!

Fast priority queues in Golang: Hierarchical Queue

When to use priority queues ⁉

Baby steps

Hierarchical Queue

Go code go

karalabe / cookiejar

A contestant's algorithm toolbox

CookieJar - A contestant's toolbox

Installation

Contents

Packing up

bgadrian / data-structures

Abstract data structures Go packages, built with performance and concurrency in mind to learn Go.

Data structures in Go

!! Warning This library wasn't used in production (yet). !!

priorityqueue

Hierarchical Queue description

Next

Contribution

Thanks! 🤝

16 free tools & services that any developer should use

Free tools & services for any developer