Forem: Vladimir Uogov

Listen to a silence

Vladimir Uogov — Wed, 10 May 2023 17:50:37 +0000

Monitoring and observability start with collecting the various telemetry and other related data. Then, various providers spent years and millions of man-hours on design and development efforts to advance and polish the data processing and analysis capabilities. Database design and development produced by multiple vendors have been tuned to almost perfection, which leads to storing more and more data consumed by those systems. And various collectors and protocols have been designed for helping to collect data for that processing and analysis. And the more complex systems, applications, and integrations we bring to existence, the more telemetry data they produce. So, total observability seems directly tied to the ever-growing processing of larger quantities of data. At least cohorts of Architects, Sales Engineers, Technologists, and other specialists helping with Monitoring and Observability constantly convince us to think that is the case. And I do not say they are all wrong.
In most cases, you need data supporting certain discoveries and conclusions. But I assure you, if you look at your data, you only get a complete picture once you also begin to look at the gaps in the data. So, not only is the data your valuable resource that helps you to come to specific findings about the state of your environment, but “no data,” the gaps in the data are equally valuable for those assessments.

What is “no data ?” The gap in the data means that your environment needs to deliver the expected metrics to your observability platform but failed to do so for one reason or another. We observe a data gap if data is not provided from your source for some time. And the gap in the data, otherwise called “no data,” is data too. Gaps have a time when we discover a “no data” condition. Gaps have a period defined for how long we are not receiving data, and gaps have a key indicating which data we are not receiving. Clearly, “no data” is an event. Are all gaps in receiving specific data types could be treated as ones? The answer to the last question is: “Of course not.” Not all data types could be considered a source of “no data” events. For example, suppose the data itself is irregular versus regular metric collection. In that case, if it is a log or irregular event, you cannot use them as indicative sources of the gaps. But if desired, comparing the timestamp of when that non-periodic data was generated and when it was accepted could be an authoritative source of a “no data” event.

So, “no data” is data. But what “no data” could tell us? The “No data” condition could mean multiple things, but most commonly one of the two, either data could not be collected, leading to a “true gap.” Or data could not be delivered, leading to a “delivery gap.” What is the difference between those two? “True gap” means that the data is not collected. There is no data. It will not appear later on. "True gap" will remain as a gap for all eternity. “Delivery gap” usually means that due to some issue, collected telemetry data could not reach the intended destination in time. “Delivery gap” could not necessarily lead to a “true gap” because some solutions in the Monitoring and Observability domain cache collected data and repeatedly will try to deliver data until that possibility is restored.

Detection of the “true gap” is either straightforward or extremely difficult. How your Observability platform must detect and react to a “true gap” in the data. Suppose a source is generally detectable. For example, the source host is pingable, but some data is missing. This indicates that we have a situation that causes a “true gap.” The complexity of detecting the origins of “true gaps” is related to difficulties in seeing problems within integrations that collect the data from the source. Commonly, “true gaps” are caused by failures in the integration layer. So, you must indicate an issue and verify if the integration is functioning correctly. Collecting telemetry through some proxy and detecting a “true gap” in the telemetry could also be a collector proxy issue. Checking for proxy status and health and proxy restart could be a reasonable option. But as I mentioned, detecting the root causes of the “true gaps” could be challenging. Sometimes very challenging. Frequently, the root causes are hidden in third-party integration configurations and implementations. Establishing some form of monitoring over integrations shall provide a path to combat those challenges.

Detection of the “delivery gap” is usually much more straightforward than the “true gap.” As this is commonly tied to the problems on the network or collection proxies. Monitoring your network performance and availability is one of your key responsibilities. If you detect that part of your network is inaccessible, or your telemetry delivery proxy is not online, be ready for a “delivery gap.” Ensure your alert policies have dependencies and silence “delivery gap” detection if a known root cause is detected. When you create your observability solutions, make sure you pick the one supporting telemetry caching.

Detecting either a “true gap” in the data or a “delivery gap” and proper reaction to those conditions must be among the standard solutions you implement in your Observability platform. Your platform will function accurately only if you guarantee proper data delivery and adequately respond to and address all related issues.

A Zen of monitoring

Vladimir Uogov — Tue, 09 May 2023 14:49:38 +0000

This article is not a tutorial, but a philosophical reflection on the question that many professionals involved in creating or using monitoring systems ask: “Why are we doing what we are doing, and how are we doing that.”

Introduction.

What I will say in this blog post may sound strange or incomplete. But this is how this problem presented itself. I do not know the ultimate answer to “what is monitoring and how to do it properly.” I doubt any human being knows an answer to this question, which will cross all t's and dot all i's. But this post is somewhat of a silly attempt to clarify some points. And if some of my thoughts will be useful to you as well, I will be glad that I've spent time bringing all my arguments on this matter together. Of course, as I perceive it, the side effect of this article is to ask more questions and dive deeper into this fascinating issue. And now, without any further ado.

Humble History.

When speaking about IT monitoring and observability, many IT professionals make the same mistakes. They are to make an impression that the idea of monitoring and observability for the matter is something:
Created recently as a part of the IT revolution.
Exists separately from other monitoring forms (industrial and others).
Unique in its approaches and none of the experiences that's been gained by the engineers of the past.
None of those statements are true. Monitoring has been a part of human activities for centuries. Whenever there is some process, someone usually observes that process. Making sure that it is through. And IT monitoring and observability as it emerged as an IT topic initially, IT monitoring and observability were treated equally to any other form of monitoring and observability. And it shouldn't be treated any differently today. Because there is nothing new in the world regarding how human beings connect with their surroundings. We can create better tools, but we have yet to change the nether mechanics of an eye or how the human brain processes input data about that surroundings. And with this idea in mind, let us try to answer a straightforward question:

Why are we doing that?

There are many answers to that question in the IT crowd, but let's think, for a second:

Why, historically, were people observing and watching the fire?
Why were captains watching the weather, tides, and directions of the winds?
Why the train engineers were watching a boiler?
Why are pilots watching airplane controls?

I brought those few samples of human activities to make the point. The purpose of all those actions is to keep driving some process, of the fire, of the ship movements, of the engine safety, of the control of the airplane. So, every time we think of “monitoring” or “observability,” we do need to think, “This is all for the control of some process,” not for personal curiosity. Not to satisfy some external requirements without questioning “why.” Not only to establish some fact without bringing this fact into a proper context. So, the foremost task of any “monitoring” and “observability” is “Control.” Everything else is a secondary task. Even if we are involved in the monitoring and instrumentation of some scientific experiment, the primary task is to keep the process under control to a maximum extent and then gain scientific data. So, after answering the first and probably most important question, we must ask ourselves:

How are we doing that?

And at this point, there will be no shortage of various answers. We will hear about how we get and compute the data, thresholds and aggregations, statistical computation, and visualization. But let us step back and think now: “What is the matter of controlling? How can we be certain that the process is controllable? How shall we organize our observability, so each element in this effort matters ?”
And again, I am taking a step back from the beginning to propose a multiple-choice solution and try to dig to the root of the idea of “observability.” What are we observing while seeking control over something? Every time we build a fire, we watch that the fire shouldn't die or get out of control. The whole purpose of seamanship is to deliver humans and cargo across the waters. While doing that, you are taking care of unfavorable conditions preventing you from getting this delivery done. Whenever you control some process or mechanism, you are looking at what you are managing and doing everything to complete this task by removing the obstacles. So, the method of control is detecting and preventing barriers that stand in the way of some processes and may block this process from fulfilling the process's purpose. So, to keep something under control, we have to detect not the problems. For example, if the train boiler is disintegrated into smithereens, we can safely say that the current state of the boiler is beyond any control. But we shall rather find the traits that lead to the issues. But what are those traits? What are the indicators that the issue is about? In the dynamic environment, which is characteristic of any process we seek to gain control over, those traits through the collection and observation of patterns through various methods.
So, if we are beginning to look at the root of the problem, monitoring, and observability is the process of constant search and detection of patterns in the data. For the user, who is trying to keep some processes under control, the observability platform produces help in catching “fingerprints” of the data that may lead to concerns and the patterns that lead to the restoration of normality. While observing the repeatable patterns is the primary tool for controlling some processes, more than just patterns is needed. But why? What's wrong with just the patterns? Detecting potential issues by searching for the known (or hardly known) traits in telemetry will remove a significant burden from the observer. Still, neither the observer nor us knows everything. And as the second line of observation, we may set the relationship between known patterns and other events in the system. Because “all good” and “all bad” are related, one observation usually leads to another. Otherwise, you may need to learn how to detect something. What is the summary, and what can we do correctly or incorrectly? Let me recapture my thoughts and come up to the conclusion:

Monitoring and observability are for control. If you do not have an outcome of monitoring that implies control over your process, you are not monitoring. You are just busy.
The purpose of control is not to let the undesirable outcome progress beyond a point where you cannot control and contain the situation. If your monitoring informs you that the wrong thing already happened, or worse, you learned that you do not have nether monitoring or observability from your user or the morning newspapers.
The way of detecting the problems is by observing telemetry patterns in motion. The way of detecting normality is through observation of telemetry patterns.
Finding new patterns by observing the relationship between known patterns and behavioral patterns of other telemetry items. And now, let me counter-sample of how we can get the monitoring wrong:
We are considering monitoring and observability as an instrument for secondary-level issues, such as inventory control, capacity management, and planning.
We are “detecting problems” instead of the conditions that may lead to the problem.
We are overly obsessed with golden signals and thresholds for those golden signals. Furthermore, we are trying to automate a “momentary analysis” based on thresholds and simultaneously apply dynamic analysis without automation. And this means we measure points and create alerts based on entries, while seeking individual telemetry charts for mental pattern recognition.
We are overly obsessed with issues and ignore the detection of normality.
We are not looking at relationships between the behavior of different telemetry items.

This is all I can say about this very intricate engineering problem. While I am not claiming that I am 100% correct, I am instead claiming that thoughts that reflections are the result of years of observing monitoring and observability as IT and industrial discipline and how engineers perceive it.

"A Game of Life" to be used in the real life.

Vladimir Uogov — Tue, 08 Feb 2022 15:30:57 +0000

Introduction

Famous "Game of Life" been an inspiration for a generations of mathematicians, computer programming geeks, scientists and people alike. But what about put this algorithm to a practical use ? By someone who do not holding a postdoctoral in mathematics... I already proposed a slightly modified rules, to help to make this algorithm to become a practical instrument, and now, I am proposing an implementation in Golang, which will make that you will be able to use that algorithm in your practical applications, written in Go.

Architecture

All elements of the Go module github.com/vulogov/GoLife implemented as a structures with interface functions to those structures.

Implementation of the Cell

Each Cell is placed in the world and do have an X and Y coordinates in this world. Those coordinates could be obtained with interface functions X():int and Y():int. World do have a limited size and is convoluted. More about World later. In addition to the coordinates, each Cell have a name. Name is set with interface function SetName(string): and could be obtained with interface function Name():string. Name do not play any particular role in the game logic. You can set any name as string to a Cell. When world is created, all Cells in the World do have coordinates, but not a name. So, the name is an abstract, that is relevantly used outside of GoLife module.

Next and very important attribute is a status. This attribute holds value false if Cell is dead, or true if Cell is Live. The value of the status attribute could be obtained with an interface function Alive():bool. Attribute age and related function Age():int will give you access to the Age of the Cell. When Cell changes the status, Cell Age is reset to 0. Each step in the "Game of Life" increase Cell Age to 1.

Next group of attributes dedicated to the Cell immunity to an unfavorable condition. Attributes upImmunity:int and opImmunity:int are holding the current "under-population" and "over-population" immunities thresholds. If Cell is detected to be a subject to a under-population or over-population, it will not die immediately. Instead of that, with each step of the Game, value of ether opImmunity or upImmunity will be decreased to 1. If Cell is Live and one of those counters goes 0, Cell changes status to the Dead. Immunity assigned to the Cell during the process of bringing Cell to Life. Two random numbers are generated and there values are from 0 to maximum Immunity threshold, defined for the World. upImmGiven:int and opImmGiven:int are holding initial immunity given to the Cell when the Cell comes to Life. The purposes of those attributes are, if in the World we enable Cell recovery, then if Cell "health" been compromised by unfavorable condition, but did not become 0, which is threshold for Death, when Cell go back into a favorable condition, upImmunity:int and opImmunity:int will be increased to 1 with every step while Cell is in healthy condition, until those values will reached initial values defined in upImmGiven:int and opImmGiven:int.

The last pair of useful attribute/function is a degrading:bool and Degrading():bool*. If Cell is Live and in unfavorable condition, but immunity is not yet compromised, this function returns true, otherwise if Cell is Live and is in favorable condition, it returns false

Interface function Step():bool brings Cell to the next Step.

If Cell is Live and it's age reaches limit, which is set to the World, Cell will "die of old age" and become Dead.

Interface function String():string return a string representation of the Cell.

Implementation of the World.

World is a home for the matrix of the Cells. Each Cell is having X and Y coordinate in the world and Cell name plays no role in Game. World is convoluted or folded, "North" and "South" boundaries of the World are connected as well as "East" and "West". World defined as a structure with an interface functions.

First, World is having a name:string . Name of the World is reserved for a User to use and play no role in algorithm.

World do have a size with xMax:int and yMax:int. Interface functions X():int and Y():int are returning those values.

World do have an ageMax:int. This value is used to determine of what to do in case if Live Cell reaches that age. It will die. If Dead Cell reaches that Age, during the Step():bool, GoLife will call the function defined in procreateFunc:func () bool. If this function returns true, Cell will change from Dead to Live.

immunityMax:int bring an upper limit to a value assigned to a Cell immunity. Lower limit is 0.

age:int holds information on current age of the World. This value manifestoes in "now many times function Step() was called for the World"

stepFunc: func() is a reference to a function executed during each step.

NotificationCh: chan Cell is a channel for the Cell's. Every time when Cell changes status, GoLife sending it to this channel. Size of the channel defined by constant CHSIZE.

And now, few words about basic interface functions to the World:

Cell(int,int): (*Cell, error). This function returns a reference to a Cell defined by coordinates and error
GetCell(int,int): *Cell Unlike previous function, this one returns a reference to a Cell or nil.
Step(): This function calculates the next step for the World, by calculating Step():bool for each Cell in the World.
Print(): This debug function display printable representation of the World on STDOUT.

Two group of interface functions I would like to discuss separately. First group, represented by function Procreate(func ():bool): This function set the function to the world, that called in determination, shall the Dead Cell procreate Life. If function returns true, then Cell will change status from Dead to Live. Otherwise NOOP.

Second group, controls if the World, permits to the Cells recovery from unfavorable conditions. By default, it will not. If you call ImmunityRecovery():, then recovery will be permitted beginning next step. If you call NoImmunityRecovery():, then, it will not.

Use of the module

The use of the github.com/vulogov/GoLife module is very simple. First, you have to create a new World by calling package function GoLife.NewWorld(string,int,int,int,int): *World. This function will return a reference to the created World.

first parameter is a string with a name of the world
second and third, defines the x and y size of the World
fourth, defined a maximum Age
fifth, will set the maximum immunity threshold.

Example:

world := GoLife.NewWorld("world", 5, 5, 120, 10)

This command, will create World with 25 cells organized in 5x5. Maximum age will be 120, upper immunity threshold will be 10.

By default, life procreation is disabled. You have to define a function, which will control procreation. Let's define a function, which will generate a random number between 1 and 10 and permits procreation if this number is 0. Example:

world.Procreate(func() bool {
    if rand.Intn(10) == 0 {
      return true
    }
    return false
})

You can define this function ether inline, or import it from some other module.

Next, you better set some initial Cells to Live. Otherwise, you will be waiting for Procreation and if you do not set the Procreation function, your World will be desolate forever. Example:

world.ToLife(4,0)
world.ToLife(4,1)
world.ToLife(4,2)
world.ToLife(10,10)

Note, if you will try to operate with Cells outside of the World boundaries, NOOP will happens.

And now, when everything is set, let's loop our world through the steps and see the Life evolving. You have to call Step(): function for the World and read information about changed Cells from the World.NotificationCh: chan Cell attribute. Example:

for {
    world.Print()
    world.Step()
    if len(world.NotificationCh) > 0 {
      fmt.Printf("World is changing on step [%v]\n", 
             world.Age())
      for len(world.NotificationCh) > 0 {
        cell := <- world.NotificationCh
        fmt.Println(cell.String())
      }
    }
}

If everything is fine, you can enjoy watching the "Game of Life"

Conclusion

This very simple module, will permit you to bring a power of modified Life algorithm into your application. Bring your application to Life and spread the joy. Hope, use of github.com/vulogov/GoLife will be fun and enjoyable. If you feel, that you can improve the code, author accepts pull requests. Report bugs, and be good !

A "Life". With a twist.

Vladimir Uogov — Mon, 07 Feb 2022 22:35:28 +0000

A "Game of Life"

Many of you, heard about "Game of Life". This is quite old zero-player game, where you can set the initial condition for the game and then watch story to be happening in the front of your eyes. The idea of the "Game of Life" or "Life" is strikingly simple. There are endless grid, stretching left, right, up and down. Each cell on this grid could be "live" or "dead". If "live" cell have a lesser than 2 "live" neighbors across all 8 directions, it dies to "underpopulation". If "live" cell have more than 3 "live" neighbors, it dies of "overpopulation". If "dead" cell have exactly 3 "live" neighbors, it changes it's state to "live", so 3 "live" cells making a new "live" cell. Overall, this game is looks like this:

You are placing a pattern of cells on the gameplay and watch a sequences of life and death in this strange universe. While this game may look like a toy, it's been a very influential model, impacting many fields of science, such as "Automata theory", "Game theory", "Decision theory" and many others. And I can not understate the importance of this very simple concept for a Mathematics and consequently for the IT industry.

A twist in the "Life"

Like many math and computer enthusiasts before me, I was captivated with this very simple, but very powerful concept. But few things had bother me for a quite a while, and I did not know what to do with the questions I've had. Without an external help, I've tried to sink my teeth into this game and while failed, I've moved closer and closer to the answer to the questions I've been bothered with. Let me sum-up those questions:

Why the classic "Life" is bit off from the real life, from how the cells actually behave ?
Why so much dependency is fallen on initial positioning of the "live" cells on the grid. How we can end up this dependency and create an "immortal" cellular automata ?
How the behavior of the "Life" will be changed if the grid of the game will be non-infinite and convoluted ? North will continue on South, West on East.
What kind of practical applications could be done with this algorithm, applications understandable by IT professionals, not just by a bunch of professors and post-doc fellows ?

And after giving this problem some while for thoughts, I've came up with twisting the classic "Life" rules. I still do not have an answer to all the question I have, but by twisting the rules a little bit, I've resolved some of the issues, or as I see them as an "issue" with a classic "Game of Life".

In real life, every cell have an "immunity" a resistance to unfavorable conditions.
In a real life, cell do have a limit for an age. Even in a favorable condition, cell can not live forever.
In a real life, if the place for a cell been unoccupied for certain period of time, the "life" can self-procreate itself in the "dead" cell with some randomness.
Each new cell, when born to "life", do have a unique ability to resist to an unfavorable conditions.
If "live" cell been hit with unfavorable conditions and had enough immunity to live through, own immunity is "compromised" and will not be healed with time. So, cell will resist to an unfavorable conditions as long as initial immunity allows to continue to live.
We can define a separate immunity threshold for "overpopulation" and "underpopulation".
"Life" world is not infinite and it is convoluted.

Well, once you have an "immortal cellular automata", the practical applications will be readily available. But before we discuss practicality, let's look at how "the Game of Life" will behave if we will have a "live" cells with immunity and "Life" world with self-procreation of "life".

I only captured 128 "steps" of the Game for this movie, but my attempt to run the "Game of Life" with modified rules for a few days, gave me an understanding that we do have an immortal automata (well based on number of steps for a "dead node" to be a "dead" and randomness of pro-creation. I probably will need a time/help to build a math base under this statement.

A practical application for the modified algorithm.

First, when you place a limitation on the size of your convoluted world, you are bringing the data grid to a practicality. There are number of applications for this modified algorithm, and let me discuss one of them, known as a "Resource access limitation". In the real life, we always dealing with a limited resources and capacities of different kind. For example, you always do have a limited capacity of the servers delivering of some software update on the cluster. Due to a capacity limitation, servers can serve only limited number of requests simultaneously and access must be throttled, but eventually, all servers must be enabled to communicate with an update cluster.

Modified "Game of Life" could serve as a throttling mechanism for this request. All servers that are in the production cluster can be represented as a "cells" in the "Life" World. When cell become "live", it is enabled to communicate with an update servers and this enablement lasts until cell is "live". When cell is in "dead" state, server represented by this cell can not use that limited resource, till cell becomes "live" again. So, this is one of the ways on how to enable a fair access to a shared resource.

There are so many ways oh how you can utilize that algorithm, and this simple example is only a tip of the iceberg. Try to implement and experiment with this algorithm. Look at my experimental implementation here and share your results.