Forem: Paimon Sorornejad

#AskingForExperts: Where Do You Keep Your SRE Artifacts

Paimon Sorornejad — Wed, 17 Jun 2020 14:13:21 +0000

Hi All;

I have seen some fantastic articles here regarding SRE and some of the efforts that you have taken to start implementing SRE principals to your respective organizations.

At my company, our org is starting to build a vertical that is focused on reliability and availability by applying SRE principals. One thing we were wondering was, how have folks collaborated and organized some of their artifacts with resepect to SRE (i.e. FMEA docs, arch docs, etc).

My initial thoughts were things like Confluence, or Jira, or Sharepoint - but any "SRE" specific tools?

Thanks!

Crisis Response: How Rapid Risk Aversion Becomes Technical Debt

Paimon Sorornejad — Fri, 13 Mar 2020 01:11:45 +0000

Intro

The world is currently in a crisis. As the spread of virus makes its way through countries and continents, businesses are ramping up their crisis response teams. This response is not isolated to any one form of business, and ranges from corporate, industrial, agricultural and educational. Businesses with critical IT footprints are rapidly reacting to possibilities of large scale remote asset management, not just with bare-metal, but with people.

This rapid response is a necessity for companies to continue to serve their customers, protect their people, and preserve their revenue. Unfortunately, there is an unavoidable side effect of rapid response in IT, and that is technical debt.

What Is Technical Debt

Technical debt is a natural byproduct of business driven IT. As funded projects make their way through delivery, developers are often left with risk based decision making that help stay on course, both financially and temporally.

These decisions can be as simple as leaving out comments in code, to more complex situations where features are built rapidly to serve an immediate business need without a long term support strategy.

Many businesses are finding themselves in this exact dilemma as they are responding to the current COVID-19 crisis. As engineers rearrange production priorities, many are left to quickly gap fill for potential business impacts and increased observability. As financial impacts are being felt around the world, there is little time to waste to ensure that the business continues delivery while strengthening their response and recovery plan.

Do Now, Fix Later

Rapid response has a direct negative impact to well architected solutions, assuming otherwise is a denial that technical debt exists. This denial however is present in many organizations, and it is unfortunately a part of historical business culture that didn't embrace failures as learning opportunities. Doing now, and fixing later is more often than not required and essential to the sustained operation of a business. With that being said, we need to make sure that we stay honest that there is a "fix later" phase, which means we need to accept that our solution isn't perfect, and there is always room for increased conformity and improvement.

Embracing Technical Debt

For a business to succeed as a force in IT, the fact that technical debt is an influencer to innovation must be embraced. Engineers need to understand that there are different solutions to common problems, and problems with no immediate solutions need to be documented and shared.

Technical debt can range from physical challenges ("we do not have the data transfer bandwidth to meet our SLO"), technical challenges ("our automation is taking too long to complete"), as well as financial challenges ("my current infrastructure is expensive to scale out") but it is important to note that initially one doesn't outweigh the other.

As we encounter these debts, we need to encourage ourselves and our staff to document them in a common place for everyone to see. Teams that have the luxury of being located in the same physical location should use visual cues like a whiteboard, post its, or even pin ups. The goal is to ensure that these debts are always in sight and not forgotten.

Virtual teams are not exempt from this process, and should use any content management tool at their disposal. For example, the mechanism for tracking can range from a shared cloud to-do list, to tools like bug trackers or agile project management software. The purpose is to ensure that all members of the team have the ability to contribute, and encouraged to do so.

These debts should be simple and conform to a set of loosely defined rules. For example, treat it like a "tweet" from your favorite influencer. The most impactful messages are the ones that are short, and direct. We are not trying to propose a solution, we are simply stating the problem we encounter while doing our jobs, or a time when we had to break away from a defined pattern. Keeping it as simple as possible ensures that the practice is not a distraction to our daily jobs.

Paying Back Debts

As time goes on, there will be certain debts that are encountered multiple times, and possibly by multiple people. The most important part of debt tracking is recording them as they reoccur, for example with a check mark for physical lists, or some sort of virtual marker. This will help build a natural priority to approaching our debts, and which ones may have the largest return on investment. Remember, the reason some of these debts were created was due to financial, technical or timing constraints therefore not all of them can be solved, or will need to be solved. Some will be the most expensive financially to solve, but provide the least amount of ROI because they are not encountered often.

Now that debts are being tracked, they must be reviewed and reminded of. Depending on the size of the organization, its agility, and other possible factors, reviews can be done after several weeks, or even months. This should become part of the normal business practice so that the debts are embraced as a team, and members become less adverse to avoiding them.

Respond Rapidly, Resolve Rapidly

Technical debt is unavoidable, and its introduction to the business landscape is required to respond effectively not only to crisis, but shifting production priorities, employee turnaround, as well as accelerated delivery timelines. We must encourage our teams to embrace these debts, not shy away from them, and expose them as soon as possible.

As teams are responding to pandemic, rapid deployment of infrastructure, architecture, and solutions will naturally need to take the quickest path to production. There will be processes that will break for the sake of agility, and checks and balances that will be temporarily bypassed. Stay vigilant to your priorities, but don't lose sight of any shortcuts that were needed to be successful.

Enterprise Metadata Tags: Tagging Your Cattle Humanely

Paimon Sorornejad — Wed, 22 Jan 2020 02:13:08 +0000

... Royals, Nobles - Short, tall - Upper Class, Lower Class - Green, Brown - Democrat, Republican ...

For centuries, society has developed a number of labels, adjectives, and monikers to help align different groups of people to different categories. These categories have been used for good, bad, and indifference, but no matter its intent, the purpose has always remained the same:contextual organization. This organization method has spanned not only with respect to people, but throughout all ends of the world, from farmers tagging cattle, to land developers tagging zones, even down to the core of human anatomy with genome and DNA tagging. While the methodology of each may be different, the ultimate purpose is the same; organization.

This article will start with a very brief retrospective on data. We will explore the importance of adding context to our data through a set of labels more commonly referred to as tags, similar to how a farmer would tag his cattle.

How We Got Here

In as little as 30 years, technology has rapidly evolved, from mainframes to quantum computing, physical architecture to the almighty cloud, all of which leading to a phenomenon known as Big Data. With trillions of bytes of data being procured every day, we need to make sure we have a way to organize the data, a way to label, most importantly, a way to tag.

The concept of data tagging isn't new. Starting as early as the late 1970s the International Press Telecommunications Council (IPTC) defined a set of standards for descriptors that were added to images, more commonly known as metadata. Just like tags on a cow, metadata became the tag for which researchers could sift through digital data, building patterns, analyzing security, and determining anomalies.

Insights Driven by Experience

I have spent over 10 years in IT, while in professional context doesn't sound very long, within the technology era those 10 years spanned through multiple generations. My career has been spent with great companies, all sharing a common problem, an abundance of data, but no common tagging of the herd.

It would be unfair to say that there wasn't any organization of data. Each team or silo has their own mechanisms for data tagging, whether it was through an application or product, or stored within a database or even a cross enterprise collaboration tool. These tags were useful, helping build correlations between data and the business, but they were unfortunately stuck in these small silos, each with their own naming conventions, requirements, and terminology.

Most of my career has been spent in site reliability engineering (SRE), where we spend the majority of our work hours sifting through logs, metrics, and various other forms of observability data. The goal is to measure ways to help guarantee the health of the business and its critical services. At any given time, terabytes of data flow across the business landscape, in hopes that someone is listening, watching, and learning. Historically, that audience has been targeted towards the developers themselves. Who else to understand the data better than those who helped create it, or create its underlying vehicle? However, that same landscape has changed over time, the audience has become much more than that, the audience is the business, the SRE's, the data scientists, operations, even customer service.

Natural Evolution

As the data landscape has grown, so has another set of tools and practices: automation and orchestration. We can take data and entrust code and logic to perform pre-defined sets of tasks based on the look and feel of the data. The pattern here is our data is turning to insights, insights to actions, actions to business value. All of which live and die by one common core concept which is data, and its organization.

The journey to this data organization doesn't come easy, especially with companies who have been around for decades and surviving with existing practices. This journey needs to start as soon as it possibly can for the sake of the greater good of the business and its ~~cattle~~ data.

Ultimately we must treat our data like a farmer would when hand selecting raised cattle to introduce to their farm. Things a farmer would have to consider include: Where did they come from? Who do they belong to? What was their purpose? What environment were they bred in? Who can I call for some history if there is a problem? These questions are asked to gain a sense of understanding, history, as well as help us profile what we are introducing. The same questions should be asked when we look at data: Where did it come from? Who owns it? What is its purpose? What environment does it belong to? Who supports the data? These simple questions, and their answers, ultimately become tags which belong in an enterprise managed card catalog.

Interviews With The Handlers

Realistically, data doesn't magically appear within our streams. It is procured by something, be it a service, a platform, a product, or a person. Just like a farmer would expect to be handed the history of their cattle during purchase, we want to make sure that our data has the answers to our questions as it travels through our information pipeline. We must make sure that the source tools incorporate tagging concepts to the best of their ability, and we need to make sure that they follow standards.

Building The Farm

There isn't a one size fits all set of tags that can be used across an enterprise. This is important to understand because trying to convince the powers that be that there is will surely end up in a failed effort. Instead, we need to start to consider tagging as an architecture composed of pillars, all starting from a common base and each representing a strategic organizational function, including Security, Infrastructure, Application, Organization, and so on.

The base is the most critical component as it will be the piece that will support all pillars. We expect it to be the most controversial as we will be defining tags that are central to the business, expecting the most accuracy with respect to inclusion. In my experience, this is where you will encounter debates on what belongs, and what doesn't.

This process is important because in the event that the base set of tags isn't strong enough, the entire architecture will collapse on itself. The more accurate our base tags are, the faster we can answer the most common questions of the business's digital infrastructure, like those previously mentioned: where did it come from? Who owns it? What is its purpose? What environment does it belong to? Who supports the data?

Strength In The Hands of the Architects

As we stand up pillars in our ever growing farm, we must ensure that each pillar is given attention with respect to its purpose. While we can try to perform the constructing of each piece of our farm, we must be cognizant that we cannot all be masters of each domain. We must trust that while there is a core set of architects watching, caring for, and feeding our base, that there are a set of architects who can focus on each of our pillars. If you have a pillar for security, with tags specific to data types focused on things like data restrictions, you must ensure that you have a specialist as consult, one with a core understanding of the business needs with respect to security.

Similar to how a farm has farm-hands to help maintain the property, we will want to build a community of practice that can keep track of this entire data landscape. The purpose of this community will be to provide a platform for consumers to voice requirements, as well as gain understanding of our topology.

Consistency In The Brand

Consistency is important in all aspects of business and that holds true right down to the data. Typically driven by standards, this consistency ensures that the proper practices are followed which continue to align our original goal of organization. Just like a farmer's tags have the same look and feel, data tags are only as valuable as the standards behind them.

When developing your tagging strategy, it is best to consider what standards you will want to adopt across the enterprise. One of the most overlooked standards is the format of how tags are spelled, and in many cases, tag names can very well be case sensitive. Consider the following examples of how without standards, tag sprawl can disturb the overall goal:

nonproduction , Nonproduction, nonProduction, NonProduction, non-production

name, Name, NAME

Setting consistency rules up front will help to prevent sprawl like the ones mentioned above, and will help new adopters conform.

Time To Profit From Your New Farm

Hopefully by this point, you have been able to follow along as we covered topics of tagging, consistency, and data organization. Sticking true to the title of the article, we have taken our herd of data cattle, and humanely tagged them, finding them a home in a pillar based farm structure, where they will carry on life with a true identity, and not just defined by an array of bytes.

At this point, we have helped set the structure that will help data consumers (visitors to our farm if we are still continuing our analogy here), start to build metrics, automation, even cost allocations based on our tags. Our data now has a persona, and an identity, as well as a history. Our data has taken its step towards our goal: organization.