Forem: Marvin

CI/CD for Licensed Software You Don't Host

Marvin — Tue, 07 Apr 2026 06:43:41 +0000

We are three weeks away from shipping NavEngine v4, an echo from the previous piece on Business Driven development.

I say "shipping" loosely. There is no deployment script I run, no SSH session I open and no Kubernetes rollout I watch. The software lives on infrastructure I have never seen, behind firewalls I cannot reach, on machines whose specs I do not know. Shipping, in this context, means pushing an image to a registry and trusting that a process running inside a container on a customer's server will eventually notice and do something about it.

That gap - between what I push and what the customer runs - is what this piece is about.

The Assumption Collapse

Every CI/CD tool I have ever used was built on a premise so foundational that nobody thought to state it: you control the deployment target. You own the server. You have the keys. Deployment, in the conventional sense, is just automation wrapped around access you already have.

NavEngine is quite the opposite. It exists as a custom image - qcow2 - shipped onto the customer's infrastructure. The customer owns the machine. I do not have SSH access unless I go through DWService and even then, that is a support channel, not a deployment one. Yet somehow, we need to continuously deliver updates to machines we cannot reach, across connections we cannot guarantee, without breaking software that is actively in use.

So the question became: if you cannot push, how do you deliver?

Indulge The moment you decide to ship software you don't host, you have made a decision with consequences that will follow you for the life of the product. Not just operationally. Architecturally. Every assumption your codebase makes about the environment it runs in now belongs to someone else's infrastructure. That is not a deployment problem. It is a design problem that shows up at deployment time.

The Answer is Pull

Watchtower. It runs as a container alongside the rest of the stack, polls the Docker registry on a configured interval, and when it detects a new image digest on the tag it is watching, it pulls and restarts the relevant containers. No webhook, no push, no SSH. The installation phones home for updates and takes what it finds.

The key design decision here was the floating tag. Every customer's Butane config ships with core:stable. Not core:v3.0.39. Not a digest pin. stable. When Watchtower polls the registry and sees that stable now resolves to a different digest than what is currently running, it pulls. What "stable" points to is entirely under my control, from the registry side, without touching anything on the customer's machine.

This sounds obvious once you say it. It took longer than I would like to admit to get there.

Two Registries, Two floating tags, One gate

Here is the full pipeline as it actually runs.

Every push to main triggers a build on the dev registry. The image gets tagged with a version identifier and a floating tag - staging. A staging environment - running the same Compose stack, same Butane configuration, same structure as a customer installation - pulls from staging. This is where the image lives until I am satisfied it works.

When staging looks good, I create a release. The production registry builds from that release, tags the image with the version (v3.0.40) and overwrites the floating stable tag. Customer installations, on their next Watchtower poll interval, see a new digest on stable and update.

The critical detail: stable is never overwritten by a push to main. Only by a release. The staging gate is the only thing standing between a broken image and a customer's running installation. There is no automated rollout percentage, no canary fleet, no gradual traffic shift. The gate is a human decision, made after watching staging run and deciding it is ready.

For a solo-operated product at this stage, that is the right call. Complexity in release infrastructure that you do not need is just surface area for things to go wrong.

Indulge A staging environment that does not accurately reflect production is a very expensive placebo. It gives you confidence without giving you information. The hardest thing about shipping self-hosted software is that your staging environment runs on infrastructure you understand, with data you created, on a network you control. Your customer's environment is none of those things. No pipeline fully closes that gap. The best you can do is know exactly where your confidence ends.

What Happens When stable is broken

It will happen. An image that passes staging will break in a customer environment for a reason that staging did not surface - a schema migration that assumed a clean database, a dependency that behaves differently on older hardware, a configuration value that was present in staging and absent in the field.

The recovery flow is: fix on main, watch it pass staging, cut a new release. v3.0.41 overwrites stable. Watchtower picks it up on the next poll interval. The customer, who may or may not have noticed anything, is now running the fixed image.

The window between the broken image landing and the fix arriving is real. Depending on how fast the hotfix moves through staging and how long the Watchtower poll interval is, a customer could be running broken software for anywhere from minutes to hours. There is no remote kill switch. There is no way to reach in and restart a service. There is DWService if the situation is bad enough to warrant it, but that is a support escalation, not a deployment tool.

This is the honest cost of not controlling the deployment target. You accept a recovery latency that you cannot compress below a certain floor. The mitigation is not a cleverer pipeline. It is investing deeply in staging fidelity and in making sure the image fails loudly rather than silently - health checks that surface problems immediately, startup validation that refuses to run on bad configuration rather than running badly.

A system that fails loudly is a system that can be fixed. A system that degrades quietly is a system that erodes trust before anyone knows there is a problem.

Enterprise and standard: different cadences, same pattern

How do we manage customers that diverge from the main product line with an enterprise license?

NavEngine has two license tiers. Enterprise customers are on a separate release cadence from standard customers. The mechanism is straightforward: separate floating tags on the production registry. core-enterprise:stable and core-standard:stable. The Butane config shipped to each customer points at the appropriate tag. Enterprise releases can go out on a different schedule, carry different feature sets, and move more cautiously than standard releases.

What prevents a standard customer from pointing Watchtower at the enterprise tag? Mostly friction. The Compose file is baked into the Butane config at provisioning time. There is no SSH access to change it. A customer would need console access and the motivation to go looking - unlikely for most, impossible to rule out for all.

The proper answer is registry-level access control: pull tokens scoped to the tags each customer is entitled to, issued at license activation and revoked at expiry. This means the registry enforces entitlement, not just the application. An expired license means an expired pull token means no updates, enforced at the point of delivery rather than after the fact.

This is on the roadmap. For v4, the answer is friction and trust.

Indulge License enforcement in self-hosted software is a negotiation between what you can technically control and what you have to trust. You cannot fully control what runs on a machine you do not own. At some point, a sufficiently motivated customer can circumvent almost any enforcement mechanism you build. The goal is not to make circumvention impossible. It is to make compliance easier than circumvention, and to make the value proposition strong enough that the question rarely comes up.

The license server is not in the update path

One decision I am glad we made early: the licensing server and the Docker registry are separate infrastructure. They do not share a failure domain.

Watchtower polls the registry. The license server is called from within one of the running containers as part of normal application operation. If the registry is unreachable, the software keeps running. If the license server is unreachable, the backend falls back to its last known state - persisted to disk, not held in memory, so it survives a container restart. The check runs periodically. The grace window is generous enough that a license server outage does not immediately affect customers, but not so generous that expired licenses can run indefinitely.

This matters because the failure modes compound. A product update that requires a license check to proceed has just made the license server a dependency of your deployment pipeline. Any outage that hits your license infrastructure also hits your ability to ship updates to paying customers. Keeping these paths separate means they fail independently, and independent failures are recoverable in ways that cascading ones are not.

What the pipeline actually looks like

The immutable version tags are not just for auditing. They are the rollback reference. If v3.0.40 breaks everything, v3.0.39 still exists in the registry. I can retag it as stable manually and customers will roll back on the next poll. This has not been needed yet. It is there for the day it is.

Indulge Most CI/CD writing treats deployment as the end of the story. Ship it, watch the metrics, move on. Self-hosted software inverts this. Deployment is the beginning of a period during which software you cannot reach is running in an environment you cannot see, on behalf of a customer whose experience you will only hear about if something goes wrong. The pipeline is not a delivery mechanism. It is a trust mechanism. Every decision in it is a decision about how much you trust the image before it leaves your hands.

Three Weeks Out

NavEngine v4 is three weeks away. The pipeline is running. Staging has held. The tags are in place.

None of that answers the only question that matters: what happens when the software leaves you?

It will run on machines you have never touched, against data you have never seen, in environments that do not care about your assumptions. By the time it fails, if it fails, it will already be someone else’s problem - and still entirely yours. The customer notices before you do. That is the thing. I am shit scared.

I suppose that this is the essence of these notes - to document real systems in real time. This is the inversion self-hosted systems force on you: treating deployment not as the end of control but as the beginning of its absence.

So you design for that absence.

You design for recovery over prevention.

For visibility over certainty and trust over control.

Everything else is just what it takes to make that possible.

Necessity versus Opportunity

Marvin — Thu, 20 Apr 2023 06:57:34 +0000

Over the past few weeks, I have been exploring the world of entrepreneurship, attending various events where accelerators gather, joining breakfasts for entrepreneurial-minded individuals, and participating in debates on the subject. Throughout this journey, I have been focusing on funding opportunities, education, and mindset shifts that ultimately lead to the creation of successful businesses.

An opinion piece that caught my attention was one by Adam Molai entitled "Africa lacks the right kind of entrepreneurs," along with the paper "Jobs, Economic Growth and Capacity Development for Youth in Africa," both of which form the basis of my thoughts today.

Indulge

With the world at 8 billion persons as of 15th November 2022, Africa remains one of the poorest continents on the globe, despite the fact that it will account for 2 out of 5 working-age individuals by the end of the 21st century. The peculiar part of this statement is the quality and purposes with which we build our ventures within the African economic zones.

The crux? Was this business built out of a pure innovative spirit - to create what once was not - or was it a means to an end?

Returning to the population question, with a 245.0% increase in population within the working age group, we expect a rise in the number of graduates and professionals alike. All this while, the market is not primed to absorb this number of individuals. If anything, the tech lay-offs that have happened over the last couple of months have been a testament to what happens when projects fail and companies are bloated.

Was this business built out of a pure innovative spirit - to create what once was not - or was it a means to an end?

This raises the question, what does this mean for education?

Having been brought up in an African household and in relation to how we raise this generation of dreamers and builders alike, provided we exist in the space where entrepreneurship is treated as a side-hustle for those incapable of being absorbed into the job market, we cease to create foundations upon which the future can stand.

The next Bill Gates will not build an operating system. The next Larry Page or Sergey Brin wont make a search engine. And the next Mark Zuckerberg wont create a social network. If you are copying these guys, you arent learning from them. - Peter Thiel (Zero To One)

It is high time we create sustainable solutions that navigate from bringing in the everyday meal to those that impart beyond sustenance. To leave you something to nibble and ponder, here is an article from TechCabal about tech talent navigating its way back to Africa.

Thought:

Perhaps the danger of Artificial Intelligence is not in it taking away employment ( unless you champion for bullshit jobs) but that with the increased use of the same technology, AI relearns from itself and other old content rather than what is made a-new. The limit remains; Artificial Intelligence will only learn from what already exists.

To OpenSource and Beyond

Marvin — Wed, 13 Oct 2021 19:11:40 +0000

It is yet another October, or better yet, another period and time frame set aside for opensource contribution.

We have had an incredible year here at TheGreenCodes, and with all this , we still come back at some of the projects that might spark interest and might give in to some learning along the way. We mention a few whose source code is out for the naked eye to peruse, learn from and contribute to (Find a bug? Why not make a pull?).

TheAssessMe Project

https://github.com/MarvinKweyu/AssessMe

Built as a quiz application, The AssessMe Project has two kinds of users; a teacher and a student. The instructor, in our case, the teacher, creates multiple-choice questions form where the students enrolled in a certain topic can give their answers and have the results right on. No, wait times anymore! The instructor then gets the results in a CSV file neatly formatted together with the average score. I'm sure we can work more off of this. Keep an eye on the project. There's more to come here.

So what seems to be the issue? Simple. We intend to split the settings into those for production and development. That's about it!

We , as well, want to show a timer on our student portal. How else would users know the time left on the quiz?

FeedCreator

https://github.com/MarvinKweyu/FeedCreator

Created as a Django SSR, FeedCreator is a blog application serving to publish blog articles of any kind, with tags, RSS feed , comments and article filtering. A great entry point for Django eyy? We all know that tutorial all so well.

Here, however, we aim to improve upon and build it together. To work through the changes we deem fit for an application of its like.

We currently have two issues attached to the project.

ColorDetect

ColorDetect repo

We know this one all too well by now. So we shall just give an overview of the tasks at hand.

We want to not only get the colors present in a video , but also get the colors present at a specific time cap in the file parsed (Sneaky one here).
We also still need tests ! Have I mentioned it? Tests. Any test really. Find a feature you think has not been tested well enough or could be done better and make a task of that.

Tickle that curiosity of yours and see what you come up with as you work through the issues or walk through the code you find. Ask any questions you may in the project pages and share what you discover. The community , you and I included , are online.

Software Engineering Principles To Live By

Marvin — Tue, 15 Jun 2021 17:16:38 +0000

Back again, with another off the top shelf; Software principles to recite before bed - literally.

Introduction

Throughout the software engineering lifecycle, practices have come and gone. Those that have stuck and those that have withstood the test of time have been the building blocks of what we have today. They've pushed teams and developers alike to better codebases and practices. Here , we discuss a few of the most notable.

YAGNI (You Ain't Gonna Need it)

An echo off the last article, where we discussed the endless stream of libraries , and our constant want to know them all. Are you going to use it really? Another dead end , eyy? The libraries you are actually going to use are those that you stumble upon while looking for a specific need and problem to solve, not those that you blindly go online and scroll for.

SOLID

Not a state of matter. Coined by Robert C. Martin, SOLID is a software engineering acronym, broken down to:

S - Single-responsibility Principle

O - Open-closed Principle

L - Liskov Substitution Principle

I - Interface Segregation Principle

D - Dependency Inversion Principle

The goal of this is to make whatever that project you, as the engineer,is working on, be maintainable and extendable. Elaborate this.

Single-responsibility principle

A class or module should have one and only one function.

For instance, we have a class that renders a GUI application, for instance , a PyQt program. Lots of things happening within this. Our program, in this case, streams our favourite music streams from source ABC , a video player of sorts. What about creating just one class for this, seems simple enough, right?
Well no. Beyond that corner of 'the program worked! ' lies a wet mop willing and ready to slap you in the face.

As with any other program, we expect downtimes , a stream not found , an additional functionality to be added to make it pomp out. You get this gist. So break it down. Our single class cannot have all this all at once, make modules to work on error logging, make modules specific for the video player, a class-specific to listing our feed and another to show history. If it can be small, make it smaller. Just do not overdo it and clear out the whole point of it.

Open-closed Principle

Objects or entities should be open for extension but closed for modification.

A function, class or module can be extended, but not modified by an external entity. Within this function, method or module, include items that are mandatory for controlling the , but none of the optional methods which would limit the flexibility of the implementations.

All notification services show a , well, notification, but do all of them show an error notification? Break it off. Utilize polymorphism. All my error service needs to do is pass ABC to XYZ. The rest should be none of its concern. Pass it relevant info and let the rest be handled. this same principle would be extended for the success or information notification services.

Liskov Substitution Principle

Derived classes must be substitutable for their base classes.

Objects in a program should be replaceable with instances of their subtypes, without changing the correctness of the program.

Let's simplify this: what we mean is if S is a T subtype, then Type T objects can be replaced with Type S objects.

Bear with me; A school application.

We have a class Staff that houses all the staff of an institution. As well, we have two subclasses, that is, Instructor and Support staff, both of whom are still staff members.

As usual, we have our assumptions as to what properties and methods belong to what object. Following the same principle, we state the below:

Do not enforce stricter rules in the subclass

Using the example; we break the programming principle by stating , as a property that class Instructor, must not only be an instructor , but also be an instructor of a specific institution. Pause a little and reason this out.

We are modifying the parent class from within the child. This is similar to saying a dog species class, say Grayhound, modifies a parent class , Dog, to have color red. We just break everything else that depends on that single class Dog. We have, essentially, stated that all dogs are red, even though it is just this one species.

Interface Segregation Principle

Make fine grained interfaces that are client-specific.

We have a program, a class , Library, that helps manage our books. Creating an instance of this class is the same as interfacing with it.

Take , as another example , an e-commerce platform. We lean towards micro services at this point. We have a database that houses our stall , an admin panel for the store owner and the user navigation section, where our buyers get to see, and hopefully purchase, our products. As micro services, at the top level this becomes:

Admin panel micro service
Client micro service(we assume our buyers are the users)

Why should the buyer interface with the admin panel when they are not using it in the first place? Why would all that code be with them at that specific instance?

Dependency Inversion Principle

Depend upon abstractions, [not] concretions

The last of the solid principles, which, if you have been following those before, should fall right into place.
A higher class should always depend upon the abstraction of the class rather than the detail.
A good example of this is the implementation of abstract classes in Django models; the Abstractuser model.

In our application models, we would , upon migration, have models from classes that inherited from it, but have no table within the database, to represent the abstract model itself, because it's, well, abstract.
You may note , how:

High-level modules do not depend on low-level modules. Both should depend on the abstraction.
1. Abstractions should not depend on details. Details should depend on abstractions.

A high-level module in any program, is one that depends on others. We specify, I repeat, abstraction; an interface upon which we build.

DRY(Do not repeat yourself)

Oh, duplicate code , where have you been? You have a piece of code from A that is exactly familiar to the one in C. you have a div, (a little web development for a while), across multiple pages. Repeating the same CSS styles and methods across those multiple pages. Why not make it a component in itself? Then tag it and write one CSS file and so forth and change data based on props?

KISS(Keep It Stupid Simple)

Often than not, engineers find themselves getting lost in algorithms and data flow other than the value the application is going to bring to the table. We focus more on 'features' as opposed to what the user will actually want our program to do.

Conclusion

So go forth engineer, KISS it, keep it SOLID, keep it DRY and remember to install libraries you actually need. No bloatware! Even on your current device. Do you need that app or is it there for that one time you thought about it?

Yours,

TheGreenCodes

Vue unit testing: Tests must fail

Marvin — Mon, 03 May 2021 17:14:41 +0000

Kicking off from A guide to better predictable code, we create our project boilerplate; assuming you have the vue-cli already installed. Right on, our awesome-todo

vue create awesome-todo

We manually select our project setup. Using Vue-2, allowing the router, vuex as well as unit testing along with the defaults selected. We select jest when it comes to our unit testing solution as well and store the configuration in our package.json file. Are you ready? Good, let's get on to the next step.

For purposes of this guide, we'll use bootstrap; particularly, bootstrap-vue. Let's shorten those CSS classes.

npm install bootstrap-vue

In your main.js file, add the necessary configurations.

import { BootstrapVue, BootstrapVueIcons } from 'bootstrap-vue'

import 'bootstrap/dist/css/bootstrap.css'
import 'bootstrap-vue/dist/bootstrap-vue.css'

Vue.use(BootstrapVue)
Vue.use(BootstrapVueIcons)

Spot on!

On the home/ landing page of our application, we want a welcome message displayed. For our case;

' Welcome to TheGreenCodes awesome to-do list. '

First of all, though, we test. As I said, we think before we implement. We think of the feature we want to implement before actually implementing it. We then write tests for this feature, to avoid the addiction cheat trap where we say we'll write a test after then never actually get to it.

Create a file home.spec.js under the unit directory in the tests folder.

Notice how we name the file. Our test runner will look for javascript files with the spec keyword, through the project, under this directory. Now copy the following code snippet below.

import { shallowMount } from '@vue/test-utils'
import Home from '@/views/Home.vue'

// what componet are the tests referring to
describe('Home.vue', () => {

// what feature or spec are we targetting specifically
  it('Displays a welcome message', () => {
    const welcomeMessage = 'Welcome to TheGreenCodes awesome to-do list'
    const wrapper = shallowMount(Home)
    expect(wrapper.text()).toMatch(welcomeMessage)
  })
})

To run this as well as consecutive tests:

npm run test: unit

Our test fails; rather horribly. Reading the shell, you see:

  expect(received).toMatch(expected)

    Expected substring: "Welcome to TheGreenCodes awesome to-do list"
    Received string:    ""

We gave it a variable with the message to expect in the component Home. What we know for certain, however, is that we haven't even touched that component.; hence the failure. Head over the Home component under views, remove the HelloWorld import and use and add an h2 tag with our welcome message. Re-run this test and see the difference.

Before we get any further, we should explain what the elements in our Home test mean.

As we have made use of comments, we shall describe target areas:

    const wrapper = shallowMount(Home)

We create a variable , wrapper, that holds our component. How we do this you ask? We import shallowMount from Vue test utils. Just like the default component life-cycle hooks, our component is initialized, only this time, since we specified we wanted a shallow mount, any child component within this parent component is not included.

We then ask the question:
'Hey! Is there any mention of our title from within this component?' To which the suite complies with a yes or no depending on what we have. We expect this component to have text, not only text but that which matches our welcome message.

Behold! We have done the foundation building block of a test; tests must fail, tests must pass and the code must be refactored.

We break this statement down:

Tests must fail
Our feature has not been implemented yet, so why on earth would we expect a test to pass?

Test must pass

Hey Marv, I wrote down that cool little feature. What next? Simple; the test must pass. The test whose feature we just wrote should pass.

Code must be refactored

When the same piece of code is edited later, does the code still pass? Can this component or code be edited and still let the test pass gracefully?

Do we get the 'It broke everything else!' exclamation?

Tests must fail, tests must pass and the code must be refactored.

We could go further with this test and specify what element we wanted to have the title:


    const titleFound = wrapper.find('h2')
    expect(titleFound.text()).toMatch(welcomeMessage)

You get the same result. Ignore the warning for UI element registration, incase you are already using it within your code, for now, we'll fix this in a while.

Let's not just make our application, but let's make a great UI. Adjust your code to look as below:

Home component

<template>
  <div class="home">
    <h2>Welcome to TheGreenCodes awesome to-do list</h2>
    <div class="container">
      <b-card class="list-container">

      </b-card>
    </div>
  </div>
</template>

<script>
export default {
  name: 'Home',
  components: {
  }
</script>

<style scoped>
.list-container{
    max-width: 170%;
}
</style>

Refactor the routes: (This would mean another component under views with the name Completed)


 {
    path: '/completed',
    name: 'Completed',
    component: () => import(/* webpackChunkName: "about" */ '../views/Completed.vue')
  }

Our application entry component would also have a link to the completed. section, either as a replacement to the about page or an add-on

<router-link to="/completed">Completed</router-link>

In the end, we should have something similar to this:

We have a basic layout up, and a first taste of what testing involves. To dive or not to dive is the question I pose to you.

Hush for a moment and let it settle. We will proceed with our application in an upcoming article.

Be sure to check the code, if need be, from TheGreenCodes repo. Specifically, the project tag awesome-todo v0.1.0.

Stick around for a while as we delve more into the internals; and yes, we can continue this conversation on tech twitter, where Potato, oh, my bad, Larry and I, Marvin, talk everything between code and smelling flowers over the weekends.

Peace out.

A small laugh, in case Lewis, finds himself here again:

'Kuingia kimahuru'

Unit testing in Vue

Marvin — Tue, 27 Apr 2021 15:08:50 +0000

Hold up! I got a confession to make.

I pushed my code to the master branch without tests.

Until I stumbled upon TDD, I had never really understood the purpose or relevance of tests. I remember writing a whole project without tests and publishing it. Can you believe it? Was I out of my mind blind?

Thinking about this still gives me heartache. Did the project work? Of course, it did. Until it didn't and I had to spend 3 whole weeks staring into the screen wondering what on earth had gone wrong. I mean, I just added a mini-feature, it should work! Sound familiar?

This could be us ... but we are writing tests now.

We are going to split this up into the fundamental building blocks of all tests, which are; Tests must fail, tests must pass, and the code must be refactored. Let's dive in, shall we?

First of all, we have outlined the pain that comes from not testing, but not really as to the importance of testing in itself. For a view of this, we would summarize into the below off-the-batt pointers.

Giving the developer an understanding of the project requirements from the client's point of view.

In this sense, a developer who writes tests following the principles outlined earlier in this piece is made to think in terms of the project requirements earlier on. Which in short says; programming is more thinking than coding. Once you visualize what you want to achieve before doing it, it significantly reduces the chances of veering off the project in itself. You know this part of the page should display what, where this data comes from and expected results should the fetch not happen. You get the project's aerial view; writing, in the end, smaller modular code.

The confidence in shipping without fear of breaking other features of your code.

Small project? Sure, fine. Can this scale? How certain are you? What about that last little piece the project manager requested? Will it mean my service or call or method will not work? What about when the project grows larger and we change something? Will we go page by page checking whether each part still works and shows the expected message? Do we have this time?

Shorter feedback loop

Shorten the 'x y and x pieces need w z k' feedback loops. When you can tell directly from the onset that this piece of code will not work as it is given how the backend team has refactored the API, you shorten the time it takes for the QA team to notice it, the time it gets to get to you and the time it takes to figure out where in the code you need to fix. You have, at this point, already identified it.

These, are some of the reasons we write tests.

So what are some of the ways we can use to get this done? Where do we get started here? Glad you asked.

To kick us off, we highlight the tools and options.
PS: We are not installing the internet of node_modules in our project, you can breathe.

Jest

Created by Facebook, Jest is an out-of-the-box package that comes bundled with assertion. Plainly, it shows not only that the test fails, but also where it failed; whether variable Y was not equal to X, and so forth. This is necessary more so when you find a test comparing, for instance, an arithmetic sum 5 to a test data of 5, failing, only later to realize that you passed the string '5' instead of an integer.

Mocha

As an elder brother to its counterpart, Mocha works just the same way, but with a little bit more configuration. To be precise, you have to include an assertion library separately. Most commonly used as a partner her, would be Chai.

Whichever of the two most used packages you use is up to preference. It all depends on how customizable your tests need to be and or what you are more comfortable with.

The two mentioned are global across javascript frameworks. What about the specific framework of choice here in our case (Vuejs)? We could say, we have libraries that make it much easier to test in Vuejs. Specifically, we are talking about Vue test utils, which is the official unit testing utility library for Vue.js and the Vue testing library, an abstraction of the previous.

So, first, we decide whether we want to use Mocha + Chai or jest, then we go ahead with what works for us between Vue test utils and the vue testing library or perhaps both.

To engrave this knowledge on testing, we intend to build a simple web application; a ToDo list. With this, we can track items, check them off as done, see what was done, edit items, delete these items, and so forth. Along the way, we use different approaches of testing and ping what one approach has to offer vs the other. Every step of the way will be guided by a systematic approach, to give a clear outline of the intent beforehand. So pause here, for a while at least, as we get our tools ready for the next section of this series.

ColorDetect: Python Image processing algorithms

Marvin — Mon, 29 Mar 2021 10:33:14 +0000

It's been a while since we touched on ColorDetect. Having taken an overview of what exactly we could achieve in the past, between getting colors from both images and video and in different formats and counts. We went ahead and described some of the use-cases of such a package, just to mention but a few.

In this piece, we highlight the improvements we have so far made, along with some notable contributors since the piece rolled out. Most notable, Clifford, whose algorithms have helped push from v1.1 ... to v1.4. Hurray Cliff!
Let's get down with it.

In case you still haven't done so:

pip install ColorDetect

We'll take it from the new features and enhancements. Specifically, text customization and color segmentation (which, I realized, came in at just the right time)
In the use of ColorDetect, we faced two to three challenges.

We now have the ability to get the colors off media files, but we want this text in a customizable format. The ability to have our own font and or styling to this text. This comes in handy especially in instances where we have a dark image. We cannot write black text on this and still maintain readability now, can we?
Hey there, what does an RGB code 5.0, 211.0, 212.0 even mean? Can the algorithm give me a more friendly color to decipher?
Okay great, I can get the percentage colors and dominance off the files I have, but can I tell what this color code or percentage refers to on the image? Can I mark the image to make it visible without guessing?

These, among others, are what we got to tackle. Let's do a quick overview of each.

To start with, In our virtual environment, we create a file custom_styling.py, and write our colors and text differently, from the defacto configurations.

For this, we will use virginia lackinger's photo on Unsplash


from colordetect import ColorDetect

# parse the image.
flowers = ColorDetect('./images/flowers.jpg')

flowers.get_color_count(color_format="human_readable")

flowers.write_color_count(top_margin=40)

flowers.save_image(location='./images', file_name='processed-human-image.jpg')

The output you might ask?

A human-readable color display. We gave some space between the text and top section of the image, as well as implicitly clarify that we want a human-friendly color format for the output. We may go further into specification of the font color to write in by parsing font_color=(0,255, 0) (for green), for instance, to the method write_color_count. Along with this, comes the font_size, font_thickness, line_type and left_margin, among other configurable options.

Let's take a look at how we separate our colors from the image. With this, we refer to color segmentation. How do we tell which sections, no matter how small, from the given image, are color XYZ?

We parse in the color ranges we want to grab.

import cv2
from colordetect import ColorDetect

# parse the image.
flowers = ColorDetect('./images/flowers.jpg')

# provide a lower and upper range for our target colors
monochromatic, gray, segmented, mask = flowers.get_segmented_image(lower_bound=(20,50,50), upper_bound=(40,255,255))

cv2.imshow('Segmented', segmented)
cv2.imshow('monochromatic', monochromatic)
cv2.imshow('mask image', mask)
cv2.imshow('grey image', gray)
cv2.waitKey(0)

You get four results. For brevity, we show three, assuming you know what gray looks like in an image; more of black and white.

Segmented image

masked image

monochromatic

Just highlight the color I need. Turn the rest of the image into black white

Yatza! It works!

As of now, we have color codes of the images, in the format we want, and a return of images with our target color-highlighted. The same could be applied to videos, without the segmentation and one or two features, for now.

A call to action

We, however, know that much more could be achieved; and that is why we have a call to action. What features would you want to be implemented? What bug did you find?

For one, we need more tests. Do you feel an aspect of the codebase has not undergone sufficient testing? Let it be known. Get the ColorDetect repo,
take a look at the contribution guidelines and make a pull.

We can leave it here for a while. Do feel free to go through the documentation as we update it with one or two features down the road.

As for this articles code, we have it on TheGreenCode's page

Launching Django applications

Marvin — Sun, 21 Feb 2021 16:29:58 +0000

Deployment day, and it's Django. Where to start. We got a server we can SSH into, we got our code and endpoints - if any- ready to be consumed.

Right here, we are going to look into the best practices in terms of developing our backend applications, or SSRs(if we were using Django templating). Through the process, we elaborate on the whys of having a production-ready application as soon as we create our project as opposed to waiting on the completion of the whole development process.

Why now?

For one, a good product is never really finished. There is always that one UI change, that one bug or logic discrepancy, or better yet, user feedback prompting a feature addition - our application must be getting some traction. Look at this in this way, would we wait till we got all our user feedback before optimizing our application for production? What about how we serve our media files? Would we still be serving it directly all through? Would we have passwords to our databases hardcoded? What if I add a feature or adjust a UI on my local setup? Would it break the main production app right off the batt?

A pointer we are getting to is this: launch your application and make small incremental changes as you go. There is no 'Tadaaa! It works and will never break or be changed!' moment. If you were simply coding as though it was on your machine all through and waiting for the 'perfect' opportunity, you have to STOP. Stop it. Get your application and pipelines up as soon as you start the project. Be production-ready with 'hello world'.

Enough talk. Let us have something we can work with here. We create a virtual environment, install our dependencies, and have our boilerplate project.

GreenCodes ➤ mkdir launch-ready                                                                                                                                       
GreenCodes ➤ cd launch-ready                                                                                                                                          
launch-ready ➤ python -m venv .venv                                                                                                                                   
launch-ready ➤ source .venv/bin/activate                                                                                                                              
(.venv) launch-ready ➤

We've made the directory called launch-ready, our intended application name. We've also created and activated the virtual environment. Go ahead and install Django and create the project while at it.

pip install django
django-admin startproject LaunchReady .

Our directory should now look like this:

(.venv) launch-ready ➤ ls                                                                                                                                             
LaunchReady  manage.py
(.venv) launch-ready ➤ ls LaunchReady                                                                                                                                 
asgi.py  __init__.py  settings.py  urls.py  wsgi.py
(.venv) launch-ready ➤

Running python manage.py runserver will give the default launch page for Django.

Our application is up and running. To prepare for production, however, we have a couple of things to configure.

Create production and development settings.

The LaunchReady directory has one setting file. Within this directory, we create yet another directory named settings and move our settings file into it, renaming it in the process to base.py.

mkdir settings
mv -i settings.py settings/base.py

Create two additional files within the settings directory and name them dev.py - to contain development settings - and prod.py - to have production settings. Our directory ends up looking as below:

(.venv) settings ➤ ls                                                                                                                                                 
base.py  dev.py  prod.py

Locally, on our development machine, we will opt for our local installation of MySQL. Go ahead and create a database and user then head over back for the next step.

As with best practice in terms of version control, never store your passwords in code. Opt at all times, for the environment variables. To manage this, we create a .env file that will never make it past our local machine. Just to be sure, we add it to the .gitignore file.

echo .env >> .gitignore

We then, go ahead and install python-dotenv (to read the variables from the .evn file )and mysqlclient (a wrapper to interact with MySQL from Django).

pip install python-dotenv mysqlclient

Our development settings will look as below.

dev.py

from dotenv import load_dotenv

from LaunchReady.settings.base import *

load_dotenv()

# since it's running on my machine, show me the errors
DEBUG = True

SECRET_KEY = os.getenv("SECRET_KEY")


DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.mysql",
        "NAME": os.getenv("DATABASE_NAME"),
        "USER": os.getenv("DATABASE_USER"),
        "PASSWORD": os.getenv("DATABASE_PASSWORD"),
        "HOST": os.getenv("DATABASE_HOST")
    }
}

# show mail messages on the terminal
EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"

# run on every host.
ALLOWED_HOSTS = ["*"]

So we have our environment variables being read from within the development settings, but where do we set these items. Simple. We create a .env file next to our dev settings.

(.venv) settings ➤ ls  -a                                                                                                                                               
base.py  dev.py  prod.py .env

The file content?

SECRET_KEY = 'django-generated-secret-key'
DATABASE_NAME=database_name
DATABASE_USER=database_user
DATABASE_PASSWORD=database_password
DATABASE_HOST=localhost

Note: We can have multiple ways with the database and secret key configuration. It is not linear.

For our production application, we take the example off of Heroku. This works the same as with any other server we might get, with one or two tweaks.

We install dj-database-url - a python package that lets us read and perform actions on the database in the event it was on a separate platform or other from our main application.

pip install dj-database-url

This, we use in our production-ready settings as below:

prod.py

import dj_database_url
from LaunchReady.settings.base import *


ADMINS = (("Developer name", "Developer email"),)

# always set this to false in production
DEBUG = False

SECRET_KEY = os.environ["SECRET_KEY"]

ALLOWED_HOSTS = ["launch-ready-domain.com", "server-ip-address"]

DATABASES = {}
DATABASES["default"] = dj_database_url.config(conn_max_age=600, ssl_require=True)


# ToDo: get an email host provider
EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
EMAIL_HOST = "smtp.gmail.com"
EMAIL_HOST_USER = os.environ["EMAIL_HOST_USER"]
EMAIL_HOST_PASSWORD = os.environ["EMAIL_PASSWORD"]

EMAIL_PORT = 587
EMAIL_USE_TLS = True

As our DEBUG is set to False, we, instead of getting the not found error, want to get a notification via mail. This is especially so when one of our views returns an exception. In this case, we set the developer(s) responsible.

ADMINS = (("Developer name", "Developer email"),)

As we have our Linux server up. We create persistent environment variables, this time, read from the bash configuration files, setting them in either, /etc/environment (for system-wide environment configuration), or in ~/.bashrc- for per-user configuration in case we have multiple.

** /etc/environment**

SECRET_KEY=''
DATABASE_URL=''
# this is common with heroku and can be set via the console or dashboard interface
DJANGO_SETTINGS_MODULE=''
EMAIL_HOST_USER=''
EMAIL_PASSWORD=''

The only major difference from the previous development setting would be how we read our database configuration.

DATABASES = {}
DATABASES["default"] = dj_database_url.config(conn_max_age=600, ssl_require=True)

This would be how we configure our application to read from an external database from our application server location.

Let's try this run. Go to your shell and run your development server. Remember, we nested and split our settings. So to run anything while in development, we have to remind Django which settings we want to use.

python manage.py runserver --settings=LaunchReady.settings.dev

Our application default template is back up.
For production, we would have it run as below.

python manage.py runserver 0.0.0.0:8000 --settings=LaunchReady.settings.prod

Launch your browser and head to http://launch-ready-domain.com:8000. You should get the very same page as with your development environment.

What next?

Of course, there's more in terms of deployment server settings and configuration, especially in terms of serving static files. This, however, can be left for the next hour. We touch base with this rather lengthy piece as we fuel up and do this further in the next, and yes, the code can be accessed right off TheGreenCodes repository.

Till next time,

TheGreenCodes

Data structures: The linear non-primitives

Marvin — Wed, 16 Dec 2020 11:16:42 +0000

On our first set of data structures, we get into the definition and scope of non-primitive structures. Have a look at the previous read on The Power of Data structures in case you feel a little lost. Right off the batt, we define what it means to be a non-primitive set, and how this can be further broken down.

Let's get right into it. A non-primitive structure is what happens when you combine two or more primitives. What we mean is simple. A char and int , for instance, are primitives. The simplest representations of data.

Broadly speaking, linear non-primitives can be grouped further as below:

Linear non-primitives

Stacks
Queues
LInked lists
Arrays

Here, as stated before, data is sequential - one after another. Some of these structures might ring a bell, stackify, is that you?.

Arrays

These are what we call stores of homogenous (meaning similar) data. So one might have an array like this:

my_array = [1,2,3,4,5,6,7,8,9] ;

vowels = [' a', 'e', 'i' , 'o', 'u' ] ;

Note how my_array has only integers while vowels has char. In both instances, our array has a collection of primitives - one type of primitive per array.

Arrays are of a fixed size. Once declared, the computer knows the amount of memory reserved for items to be stored there. You might ask:

*"But what if the type of data I have changes in size greater than the array?" *

Well, you might need to consider a different type of structure to store this data.
Use arrays, for instance, to store months of the year. We will not be changing this count anytime.

Above declared, the sizes of the arrays are inferred. Meaning the types, since not defined per se, are 'figured out' based on the values and how we use them.

We might as well have this:

For python.


import array

count_down = array.array('i', [5,4,3,2,1])

# where *i* stands for int (an array of integers).

Types may go on as:

c for char
f for float
d for double
u for unicode and so forth.

Have a look at the documentation for more insight into this. AS well, take note that the array module has different functions to enable array manipulation.

Rust:


let count_down: [i32; 5] = [5,4,3,2,1];

// where *i32* is signed integers, and *5* in *[i32; 5]* is the size of the array. i.e have five integers

Important pointer:
Python lists and arrays are not the same!

Operations on an array will be based on the index of the array element, with indexing starting from 0. Hence, the first item, in either case, would be:

count_down[0]

The result from the above is 5 (the semi-colon has been left out for brevity but should be used based on the language you are using).

Let's take a look at another structure that looks similar, but is not the same - Linked lists.

Linked lists

Unlike arrays, linked lists store data in a non-contiguous form. This implies one piece of data is not placed side by side to the other as: [1,2,3,4,5], rather it is stored in form of nodes with pointers to the next as so:


[data1, pointer_to_data2] ...  [data1, pointer_to_data3] ... and so on

The last item in our linked list, since there is no pointer to the next, would have null.

As well, linked lists are dynamic- their size is not fixed at initialization.

Linked lists will be further broken to:

a) Singular linked lists (used in the elaboration of linked lists above)

b) Circular linked lists

c) Doubly linked lists

Circular linked lists

These have three items in a node: the previous data, the data, and finally, the next data in the sequence.

[null, data1, pointer_to_data2] ...  [pointer_to_data_1 ,data1, pointer_to_data3]

The first item (the head), has a null as the previous pointer while the last node has null on the next pointer.

Doubly linked lists

Similar to circular linked lists, these have two pointers as well. The only difference would be the last node, instead of a null, has a pointer back to the first item in the list. 1 + 1 = circular.

In most of what we implement through code, we have implementations of linked lists. You will hardly get yourself doing this manually, but understanding that some of the things we call 'mutable arrays ' are actually implementations or wrappers around more complex structures.

Example:
If navigating through my directory, my computer needs to know where I am, where I'm from, and possibly have the correct link/structure to the nested directory I'm navigating to. So to the top-level directory, I cannot go any higher, no previous node. Likewise to the last directory or file, there is no 'next' option.

Till now, we have come to understand the importance of data structures, the groups they have been placed into, and we have an idea of what some of the linear data structures involved. We could go on and on on the details of each, but that is a tale for another day. A breather for us both at this point. Go read something non-algorithm-like for a moment. We'll still meet here, same time, same drive.

The Power of Data Structures

Marvin — Wed, 18 Nov 2020 12:38:38 +0000

A trip down memory lane avid reader. Let's take a walk through the core of it all: data structures. What are they and why are they so important? A 'hello' to a reader that might have missed our talk on Memory management, where we delved into what happens to our code in variable assignment. Do take a look, even if it's a refresher you're looking for.

Alright then. What is a data structure?

It's a way of organizing data. Say we have a couple of friends coming over for our fortnightly game night( ah...the things on our to-do list). We need snacks, a number of people expected to be there, a list of activities to partake. I can go on and on. So how do we organize this in say a program we want to implement to fix the hustle of planning? How do we store the things to be done or the items to be purchased? It's all data in the end. We need structure. We need data structures. The how of getting our data and manipulating it from these structures are our algorithms.

Right back to our game, who will do what?

How do we decide what comes before what in terms of structuring this data? Does our simple program make it easy to retrieve this information? Is it fast enough? What about efficiency?

Whatever kind of application we are talking about, we need a data structure(s) that will allow traversing, insertion, searching, deletion( in case the last casserole was outright nasty), sorting, or merging of bits of data. Call it CRUD with extra steps.

As from the Memory management article, we got data, then collections of data. It's all data. While memory allocation might not be something we think of with javascript of even Python, it holds water when doing C or C++. So keep this in mind.

Broadly speaking, we group data structures, much like data types in itself, as:

a) Primitive

b) Non-primitive

Primitive structures would hold primitive data types; that is the chars, the int the float. Basically, data types that hold a single value.

Non-primitive structures, on the other hand, break down to linear (which holds data in a sequence) and of course non-linear. Hold your breath in case you lost it a moment there. We'll go into the details soon enough.

As of now, you have a general idea, a scope if you may, of what structures involve.
They are quite a number.

Over the course of the next couple of weeks, we'll be covering some of these structures. Unraveling them to bits and pieces. Finding what makes them tick as so and getting why many a time, developers new and old alike, have challenges addressing them. Stick a while, I cannot wait for our next piece in this!

ColorDetection - Python ColorDetect package

Marvin — Mon, 28 Sep 2020 07:04:11 +0000

Images. That's it. Images. As a point of practicality, take a fashion designer (as a forum member vividly described to me at one point). You are given an image or have an image at your disposal that simply tickles your curiosity and want to incorporate it in one of your new lines. Let's swerve a little into the genetics section. Given a petri dish image for instance, with pigmented bacteria or similar organisms, and you would like to find the abundance of that organism or organisms in this specific image. Get the gist?

That's why we have ColorDetect. Letting you knab those colors right from the image, or if you were a tad bit crazy, a video. For an overview, we'll kick it off by installation.

As all Pythonistas have it, create a virtual environment, and install.

pip install ColorDetect

For this demonstration, we'll borrow Greyson Joralemon's photo.

A program that gets the colors in this specific image.

from colordetect import ColorDetect

user_image = ColorDetect('media/random_balls.jpg')

colors = user_image.get_color_count()

print(colors)

ColorDetect will go, do its thing and return this:

$ python get_colors.py

{'[59.0, 70.0, 198.0]': 7.63, '[245.0, 155.0, 186.0]': 9.0, '[232.0, 22.0, 103.0]': 11.98, '[207.0, 143.0, 3.0]': 35.54, '[88.0, 70.0, 34.0]': 35.85}

A color description of the image, breaking down the relevant most abundant colors to say, hey, this image has 7.63 % of it occupied by this RGB color: '[59.0, 70.0, 198.0]'.

If we wanted the hex for this instead, we'd pass that as the parameter to get_color_count().

colors = user_image.get_color_count(color_format='hex')

#colors
# {'#3b46c6': 7.63, '#f59bba': 9.0, '#e81667': 11.99, '#cf8f03': 35.56, '#584622': 35.83}

We could, of course, look for more color than just five of the most dominant.

user_image.get_color_count(color_format='hex', color_count=8)

Depending on the size or ratio of the image, it may take a moment. A case in point, a low graphic image, say 2MP vs a high definition top of the line camera image. These two images, despite pointing to the same object or scene, have very different pixel ratios.

Let's go ahead, and instead of getting the color as a variable, write this color onto the image.


user_image.write_color_count()
# save the image after writing the color count to it
user_image.save_image('media', 'processed.jpg')

We save this image in the media directory with a name processed.jpg:

Perfecto!

we did have something about the crazy people with videos, didn't we?
Now, where is that video...oh, here it is. Our earth.mp4 file.

from colordetect import VideoColor


my_video = VideoColor('media/earth.mp4')

video_colors = my_video.get_video_frames( progress=True)

print(f"{video_colors}")

{'[137.0, 165.0, 182.0]': 0.92, '[71.0, 84.0, 95.0]': 2.16, '[24.0, 30.0, 50.0]': 11.17, '[7.0, 10.0, 26.0]': 17.59, '[0.0, 0.0, 0.0]': 68.83, '[143.0, 170.0, 187.0]': 0.84, '[76.0, 89.0, 101.0]': 2.09, '[26.0, 32.0, 52.0]': 11.18, '[8.0, 11.0, 28.0]': 16.69, '[135.0, 163.0, 181.0]': 0.95, '[76.0, 88.0, 98.0]': 2.05, '[8.0, 11.0, 27.0]': 15.43, '[127.0, 160.0, 179.0]': 0.94, '[71.0, 83.0, 94.0]': 2.38, '[7.0, 11.0, 27.0]': 15.72, '[124.0, 160.0, 181.0]': 0.9, '[69.0, 84.0, 96.0]': 2.26, '[26.0, 32.0, 53.0]': 13.12, '[125.0, 160.0, 182.0]': 0.89, '[68.0, 82.0, 95.0]': 2.27, '[132.0, 166.0, 186.0]': 0.79, '[71.0, 87.0, 100.0]': 2.1, '[25.0, 32.0, 52.0]': 14.18, '[132.0, 164.0, 183.0]': 0.89, '[70.0, 85.0, 97.0]': 2.08, '[132.0, 165.0, 183.0]': 0.9, '[73.0, 88.0, 99.0]': 2.06, '[26.0, 33.0, 53.0]': 12.11, '[8.0, 10.0, 27.0]': 16.76, '[134.0, 166.0, 184.0]': 0.88, '[132.0, 165.0, 185.0]': 0.86, '[74.0, 88.0, 100.0]': 2.0, '[26.0, 33.0, 52.0]': 10.65, '[7.0, 10.0, 27.0]': 16.93, '[124.0, 157.0, 178.0]': 0.99, '[68.0, 82.0, 93.0]': 2.14, '[25.0, 31.0, 50.0]': 10.66, '[124.0, 160.0, 182.0]': 0.88, '[67.0, 82.0, 94.0]': 2.19, '[25.0, 31.0, 49.0]': 10.68, '[124.0, 160.0, 183.0]': 0.85, '[67.0, 83.0, 95.0]': 2.0, '[25.0, 30.0, 49.0]': 11.04, '[123.0, 160.0, 182.0]': 0.87, '[24.0, 29.0, 47.0]': 11.15, '[23.0, 29.0, 47.0]': 10.6, '[6.0, 9.0, 26.0]': 19.34, '[67.0, 83.0, 97.0]': 2.0, '[24.0, 29.0, 48.0]': 9.31, '[125.0, 161.0, 184.0]': 0.85, '[67.0, 84.0, 97.0]': 1.98, '[127.0, 162.0, 183.0]': 0.87, '[67.0, 83.0, 96.0]': 1.96, '[23.0, 29.0, 46.0]': 8.58, '[5.0, 8.0, 25.0]': 17.77, '[125.0, 161.0, 183.0]': 0.88, '[68.0, 84.0, 98.0]': 1.9, '[24.0, 29.0, 46.0]': 6.95, '[67.0, 84.0, 99.0]': 1.89, '[133.0, 166.0, 186.0]': 0.81, '[67.0, 86.0, 99.0]': 1.85, '[23.0, 28.0, 45.0]': 6.83, '[5.0, 8.0, 24.0]': 22.22, '[135.0, 165.0, 186.0]': 0.85, '[69.0, 86.0, 100.0]': 1.79, '[22.0, 27.0, 43.0]': 7.22, '[5.0, 7.0, 24.0]': 22.48, '[73.0, 91.0, 105.0]': 1.69, '[129.0, 163.0, 185.0]': 0.85, '[69.0, 85.0, 98.0]': 1.9, '[21.0, 27.0, 44.0]': 7.25, '[4.0, 7.0, 24.0]': 21.7, '[68.0, 86.0, 101.0]': 1.9, '[22.0, 27.0, 45.0]': 7.91, '[126.0, 160.0, 181.0]': 0.94, '[66.0, 83.0, 96.0]': 1.91, '[22.0, 27.0, 46.0]': 9.19, '[129.0, 164.0, 185.0]': 0.84, '[69.0, 86.0, 99.0]': 1.96, '[21.0, 27.0, 46.0]': 10.65, '[133.0, 165.0, 185.0]': 0.85, '[23.0, 29.0, 48.0]': 10.61, '[7.0, 9.0, 26.0]': 17.7, '[135.0, 165.0, 185.0]': 0.85, '[73.0, 88.0, 100.0]': 1.96, '[24.0, 29.0, 50.0]': 11.34, '[139.0, 164.0, 177.0]': 0.92}

We may find the colors are too much for our use case. So let's shorten this:

print(my_video.color_sort(color_count=6))

{'[0.0, 0.0, 0.0]': 68.83, '[5.0, 7.0, 24.0]': 22.48, '[5.0, 8.0, 24.0]': 22.22, '[4.0, 7.0, 24.0]': 21.7, '[6.0, 9.0, 26.0]': 19.34, '[5.0, 8.0, 25.0]': 17.77}

This will return the top 6 most dominant colors from the whole video, having taken a frame for every second. Looks much better! Unless you want to use all the colors, that is.

I'll iterate on this. This all depends on the quality of the input media file, and length if it is a video in this case. take it this way, a video 5 minutes long, showing a wide variety of colors from all sorts of crayons vs a short video as just shown. Remember, the process is per frame of every second. I'm certain this will be addressed in a future release.

We can hold off here and let the steam cool off.
Do keep up to date with the package as more features and performance improvements come to light.

Yours,

TheGreencodes

Memory management - Deep and Shallow Copying

Marvin — Sat, 22 Aug 2020 16:58:00 +0000

Let's go back one moment. A little further down to our data structures. The dear heaps and stacks of them.Quite literally.

What happens when I assign a variable? What about when I pass it as a parameter?
How does the program know this is what I'm talking about?

Before we go to a higher level of abstraction, that is, looking at it like a note title taken on a piece of paper so that you can flip the page and just look at the variable value, we should look at how the machine 'thinks' about it.

Close your eyes for a moment and declare a variable:

my_variable = "random";

See, when you do this, you're adding to the list of things the program has to remember. It's like one on top of another of 'cart' lists. A stack.

As it stands, your program will store your string as below:

Where the first diagram with the pointer(location of the variable in memory), len(for the length of the string) and capacity(the amount of memory, in bytes, the machine gives to this variable for use) is a stack and the other is a heap. Whatever we do with this variable, be it a mutation, concatenation, making a substring or whatever there is, refers to the stack.
In the stack, the pointer holds the location of where the actual data is stored. So we are just given a reference to it.

Why do we do this? Why do we store it in different data structures?

Simple. It's about the speed.

Stacks are faster compared to heaps. So instead of moving around a whole chunk of data(the heap) while mutating it, just carry the reference to it. I mean, the program is already doing its tasks (whether heavy or not), so there is no need to add overhead here.

A point to remember, not all data is assigned as such. For static data types, that is, boolean, integers, floats, and chars, variables are added directly onto the stack. So we would have no heap to store a simple 456.98 because the sizes of these types are already known by the program except in the rare case it is user input!

The size of these types, more so numbers(integers and floats), are determined based on whether they can be negative (signed) or exclusively positive(unsigned). This should remind you of how you declare your variables in math. You would say that any number in your paper is positive unless stated otherwise, or as we call it here unless signed.

So this assignment would work with compound data types - the result of combining two or more static types*.

Example:

string (a combination of chars)
arrays
tuples ... and so forth, depending on how your language of choice calls it, for instance, dictionary vs javascript object.

Back to copying.

You want two variables to refer to the same thing and you want to edit one of the variables without affecting the other.
I suppose you could assume a simple re-assignment, right?

my_variable = "random"

my_other_variable = my_variable

Whether it's console.log(), print() or printf(), (choose your weapon and let's make the battle legendary!), anything that your muscle memory has right now. The two will show random in stdout/ output. The caveat?

Yes, you've duplicated the stack, not the heap. Being as it is, every language's goal at optimum performance. Keeping it as tidy and efficient as possible. No overhead. Pointing to the same memory space.

So what happens if I mutate one variable?

my_other_variable  + 'ramblings' 

print(f"My second variable: {my_other_variable}")
print(f"My variable: {my_variable}")

In both cases, the output is a string:

random

To get completely two different items with the same data, in that both can be mutated independently, you have to take a different approach; deep copy.

A warning

As far as memory is concerned, deep copying is memory consuming as it has to get the pointer and follow it to where the data is stored then duplicate this heap.

Depending on what language you are using, we have the inbuilt copy module in python, javascript and or copy for lower-level languages and so on and so forth (We cannot simply list all the ways to deep copy across the multiverse)

import copy

my_variable = "random"

my_other_variable = copy.deepcopy(my_variable)

Love javascript much?


let my_variable = "ramblings"


let my_second_variable = `${my_variable}`


my_second_variable = 'random ' + my_second_variable

console.log('My first variable',my_variable)
console.log('My second variable',my_second_variable)

There are, of course, other multiple ways of doing this. It is, after all, javascript.
A point to mark, especially with objects, [lodash](https://lodash.com/], dearest ramda or rfdc work perfectly. Custom method for your implementation? Go ahead, just not JSON.stringify().

The mad rustacean?

let my_variable = String::from("random");

let my_other_variable = my_variable.clone();

Having done this, you can manipulate your new variables in any way you want. Go to the moon if need be. Just need a couple of dollars more.

It is this same principle that governs the passing of variables across functions and objects. Passing a pointer to the original data and not the whole heap. Comprende? I sure hope so. So go forth and choose wisely.

Let's leave this piece at that, and chat in the comments if need be.

And yes, we can chat tech on twitter too. marvinus_j