Forem: Damian Mazurek

Processing EventHub Captured Messages in Avro Files Using Databricks

Damian Mazurek — Sat, 13 May 2023 18:19:59 +0000

Step 1: Mount Azure Blob Storage

The first step is to mount the Azure Blob Storage, which contains our Avro files. Azure Blob Storage is a scalable and secure object storage platform. Databricks file system (DBFS) allows you to mount blob storage so that it can be accessed like a local file system. Below is the command to mount Blob Storage:

dbutils.fs.mount(
  source = "wasbs://{container}@{storage_name}.blob.core.windows.net",
  mount_point = "/mnt/iotdata",
  extra_configs = {"fs.azure.account.key.{storage_name}.blob.core.windows.net":"AccessKey"})

Replace {container}, {storage_name}, and AccessKey with your container name, storage account name, and access key respectively.

Step 2: Load all Avro Files

After mounting the storage, we can load Avro files using the spark.read.format('avro') function. Avro is a data serialization system that allows big data to be exchanged among programs written in different languages.

The number of * in the path represents the depth of subdirectories that you set in EventHub in the capture naming convention. Each * represents a directory level.

df=spark.read.format('avro').load("/mnt/iotdata/*/*/*/*/*/*/*/*/*.avro")
display(df)

This command reads all the Avro files from the mounted blob storage that are captured by EventHub native functionality and displays the DataFrame.

Step 3: Transform Binary Message Payload (Body) to String

The Avro files contain binary data, which needs to be converted into a string for further processing. You can convert the binary data to a string format using the cast function in PySpark.

body_df=df.withColumn("Body",df.Body.cast("string")).select("Body")
display(body_df)

Step 4: Map String to JSON Schema

After converting the data to string format, we can map it to a JSON schema using PySpark's StructType and from_json functions.

First, define the schema for the data:

from pyspark.sql.types import *
data_schema = StructType(
    [
        StructField("a",DoubleType(),True),
        StructField("b", DoubleType(), True)
    ]
)
json_schema = StructType(
    [
        StructField("factoryId", LongType(), True),
        StructField("timeStamp", TimestampType(), True),
        StructField("data", data_schema, True)
    ]
)

Then, convert the string data to JSON format according to the schema:

from pyspark.sql.functions import from_json, col
json_df = body_df.withColumn("Body", from_json(col("Body"), json_schema))
display(json_df)

Step 5: Map JSON to Table Format

After converting the data to JSON format, we can flatten the data and convert it into a table format.

final_df=json_df.select(
 col("Body.data.a"),
 col("Body.data.b"), 
 col("Body.factoryId"),
 col("Body.timestamp"))

Step 6: Save it using Spark SQL in Silver Layer

Finally, we save our processed data using Spark SQL. Here, we'll save it in a silver layer, which is a cleaned, processed representation of our raw data.

First, create a database for our processed data:

%sql
CREATE DATABASE iotdata
LOCATION "/mnt/iotdata"

Next, save the processed DataFrame as a table in the newly created database:

final_df.write.saveAsTable("iotdata.tablename")

And there you have it! You have successfully processed Avro files captured from Azure Event Hubs using Databricks. You've transformed binary data into a readable format, mapped it to a schema, converted it into a silver layer tabular format, and saved it using Spark SQL.

AI Journey — from chess to software development!

Damian Mazurek — Sat, 18 Mar 2023 10:57:37 +0000

I recently read a fascinating book about AI created by Garry Kasparov. In his book “Deep Thinking: Where Machine Intelligence Ends and Human Creativity Begins,” chess grandmaster reflects on his famous match against the Deep Blue computer in 1997 and the subsequent evolution of artificial intelligence.

Rule-based chess engines

The history of human versus artificial intelligence (AI) chess competitions stretches back decades, with each new match raising the stakes and pushing the boundaries of what was thought possible.
In the early days of AI, computers were programmed to follow a set of rules to play chess. These rule-based systems were limited by the instructions they were given and were no match for top human players.
Rules-based chess engines were computer programs that used a set of predefined rules to analyze and play chess. These rules were based on chess strategy and tactics. They evaluated the positions of the pieces on the board and made decisions about which moves to play.
One of the primary tasks of a rules-based chess engine was to search the game tree to find the best move for a given position. This approach involves considering all possible actions that could be made from the current situation and evaluating the resulting positions to determine which is the most advantageous. The engine used its predefined rules to choose how to value each position and select the move that leads to the best position.

AI will never win in chess against a human — this is too complex for a machine.

At this point, people did not believe that machines could beat human chess players. Chess had a long history and a rich tradition, and many people thought the game was too complex and subtle for a machine to master. Many people believed that chess required a level of creativity, strategic thinking, pattern recognition, and intuition that could only be found in humans and that computers would never be able to match the skills of the best human chess players. They were right. In the 80s, AI was not even close to chess’s top human performance level. But this was about to change in the next 20 years.

Deep Blue

As AI technology advanced, so too did its ability to play chess. In 1997, the Deep Blue computer developed by IBM faced off against chess grandmaster Garry Kasparov in a highly publicized six-game match.
Deep Blue was explicitly designed to play chess. It was programmed with a database of chess positions, moves, and algorithms for analyzing and choosing the best move in a given situation. It was able to analyze millions of positions per second and make decisions faster than any human player could.
Kasparov won the first game, but Deep Blue won the second, and the match ended in a draw. Deep Blue emerged as the champion in the final four games, winning three and drawing one. Overall, Deep Blue won the match with a score of 3.5–2.5.
The match between Kasparov and Deep Blue was a historic event that marked a significant milestone in the development of artificial intelligence and the potential for computers to outperform humans in specific tasks.

Modern-era chess engines

However, this was just the beginning of the AI chess revolution. In the years that followed, AI systems consistently improved and outperformed top human players. In 2005, an AI system called Hydra defeated world champion Vladimir Kramnik in a match.
In 2008 the first version of Stockfish was released. Since its inception, Stockfish has consistently ranked as one of the strongest chess engines in the world. It has won numerous titles and accolades, including the World Computer Chess Championship in 2014, 2015, 2016, and 2017 and the Top Chess Engine Championship in 2015, 2016, 2017, 2018, and 2019.
AlphaZero — reinforcement learning in chess.
As AI technology continued to advance, so did how it was used in chess. In 2017, an AI system called AlphaZero, developed by DeepMind, made headlines when it defeated the world’s best chess-playing computer program, Stockfish, in a 100-game match, winning 28 games and drawing 72.
AlphaZero was trained using a novel approach called reinforcement learning, in which it learned to play chess by playing against itself and receiving rewards for winning or drawing games.
This victory was significant because it demonstrated that the reinforcement learning approach could produce a chess-playing AI that could outperform traditional chess engines designed and tuned to play chess.
There was a crucial difference in how they played chess: Stockfish was known for its strong positional evaluation and ability to search deep into the game tree. In contrast, AlphaZero was known for its ability to come up with creative and unexpected moves.

From chess to other games

In 2019, DeepMind released MuZero, a variant of AlphaZero that was designed to be more general and flexible so that it can be applied to a broader range of tasks. Like AlphaZero, MuZero used reinforcement learning to learn how to play a game by playing against itself and receiving rewards for winning or drawing matches. However, unlike AlphaZero, MuZero did not require prior knowledge about the game it was learning to play, and it could learn to play a game from scratch based on raw pixel data.
MuZero was able to match AlphaZero’s performance in chess and shogi, and it even set a new world record by improving upon AlphaZero’s performance in Go. Additionally, MuZero surpassed state-of-the-art in mastering a suite of 57 Atari games known for their visually complex environments. These achievements demonstrate the flexibility and adaptability of MuZero, as it was able to learn and master a wide range of games with ease.

Why is Software Development new chess for AI?

The development of AI in chess is impressive. Over 30 years, we have come from engines based on sets of rules to mechanisms that can independently learn what the rules of the game are and how to play at a world-class level. Meanwhile, the new playground for AI was selected — software development.
Software development and chess can be viewed as complex problems that can be approached using artificial intelligence. Both involve making decisions based on incomplete information and adapting to changing circumstances. They require creativity, strategic thinking, pattern recognition, and adaptability. Both software development and chess also involve optimizing for specific goals or objectives. The objective of software development is to create efficient and reliable code that meets the user’s needs. In chess, the goal is to win the game by placing the opponent in a position where they cannot make a legal move. This makes software development an ideal ground to apply and develop AI systems.
An additional problem that AI can solve in this area is a problem with fewer skilled software engineers compared to demand. In this case, artificial intelligence can help with the following:

Automating repetitive tasks: AI can be used to automate tasks that are time-consuming and repetitive, such as testing and debugging code. This can free up developers to focus on more complex and creative tasks. Enhancing productivity: AI tools can help developers write code faster and more accurately by automatically providing suggestions and completing tasks.
Providing assistance: Some AI tools can help developers with tasks such as identifying and fixing bugs or suggesting alternative solutions.
Improving security: AI can detect and prevent security vulnerabilities in code, helping ensure software systems’ security and integrity.
Code review: AI can analyze code and suggest improvements, such as refactoring code more efficiently or adhering to coding standards. In the future, like in the chess example, AI can be more efficient in software development than humans. It could create high-quality code based only on user requirements and be able to define requirements for specific applications. Maybe in a couple of years, the user will only say: “Create me a social media platform,” and AI will:
Generate all use cases.
Propose ideas for innovative functionalities.
Create a user interface.
Create an architecture for the whole system.
Create specific code for services and tests for this code.
Create CI/CD pipelines.

At this moment, many people will say that this is impossible. I will agree with them, but as AI chess history teaches us, this can be a reality in the future (15–25 years from now).

Main players in AI-based software development

How the current state of AI looks like in software development? Here we have many different tools, but I will concentrate on two solutions that can dominate this market. The first is Codex from OpenAI, and the second is AlphaCode from DeepMind.
OpenAI is a research organization that aims to promote and advance the development of artificial intelligence in a way that is safe, transparent, and beneficial to humanity. It conducts research in machine learning, natural language processing, and robotics and develops and releases AI tools and technologies for use by the broader research community and industry. OpenAI strongly cooperates with Microsoft (MS invested $1 billion in OpenAI).
DeepMind is a leading artificial intelligence company founded in 2010 and is now a subsidiary of Alphabet Inc. The company uses machine learning and other advanced techniques to build intelligent systems that perform various tasks, such as language translation, image recognition, and game playing. DeepMind is known for its work in natural language processing, computer vision, and robotics and has made significant contributions to the field of AI through its research and development efforts. The company has also developed several practical applications for its AI technologies, including a system that is used to help reduce energy consumption in data centers.

Open AI Codex

OpenAI Codex is a natural language processing to code system developed by OpenAI based on GPT-3 language model. It is designed to allow users to write code using simple, natural language instructions rather than traditional programming languages.
The Codex system uses advanced machine learning algorithms to understand and interpret human language and can generate code in various programming languages in response to user prompts. This makes it easier for people with limited experience with coding to create software and other applications by simply expressing their ideas in plain English.
As soon as Codex was released through the OpenAI API in August 2021, it quickly gained attention from developers around the world. It was named a game-changer, making it easier than ever before to write code. And it wasn’t just individual developers who were excited about Codex — major companies were also taking notice.
One of these companies was GitHub, the popular platform for developers to collaborate on code. They saw the potential in Codex and decided to incorporate it into a new tool they were developing called GitHub Copilot. Codex became the principal building block of this tool, and it has since become an essential part of the GitHub ecosystem.
GitHub’s recent evaluation showed that, on average, users accepted 26% of the completions suggested by GitHub Copilot. They also found that, on average, more than 27% of developers’ code files were generated using GitHub Copilot, and for specific languages such as Python, that number was almost 40%.
👉 Link to the full description of copilot: https://github.com/features/copilot

AlphaCode from DeepMind

AlphaCode is a system for code generation developed by DeepMind.
The model was designed to address the main challenges of competitive programming, such as: searching in the vast space of possible programs, limited access to example tasks for training, and a restricted number of submissions per problem. It achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform, making it the first artificial intelligence system to perform competitively in programming competitions.
Researchers from DeepMind used a large dataset of code from GitHub to train their models to do a better job at writing code. This dataset was enormous, at 715 GB, and they used it to teach their models how to predict the next word in the code. To do this, they split the code files into two parts and used the first part as input for their models. Then, they trained the models to try to write the second part of the code. This helped their models learn how to write code as humans do, so that when they gave them a specific task to do, they would be better at it.
AlphaCode can solve programming problems by generating and analyzing millions of diverse programs using specially trained transformer-based networks. It then filters and clusters these programs, submitting a maximum of only 10 for evaluation.
👉 Full research paper on AlphaCode: https://www.science.org/stoken/author-tokens/ST-905/full

The AI future in software development

While AI has the potential to automate many tasks and make the software development process more efficient, it is unlikely to replace the creativity, problem-solving completely, and critical thinking skills of human developers in the next five to ten years.
In order for AI to completely replace software developers, it would need to be able to perform all of the tasks currently carried out by developers, including:

Understanding and interpreting complex requirements and specifications
Designing and implementing software solutions
Debugging and testing code
Collaborating with other developers and stakeholders
Continuously learning and adapting to new technologies and methods While AI has made significant progress in some of these areas, it is unlikely to be able to perform all of these tasks at the same level as human developers soon. Overall, AI will likely augment and assist software developers rather than replace them, helping them be more productive and efficient while maintaining the creativity and problem-solving skills essential to the software development process in this time scope. In the long term, 10–25 years from now, this technology will surpass human abilities in this area, as it did in chess in the last 20 years. The AI era has begun…

The five things we can learn about the data adoption process from the fishing company!

Damian Mazurek — Thu, 19 Jan 2023 08:01:49 +0000

I have been visiting some of my clients for the last four weeks. One of them was a fishing company that transformed its business using data.

They’ve come a long way to do this, but the results are stunning! Today I want to share with you a couple of things we can learn from them.

Let’s start with a quick introduction. Our main character is one of the largest fishing companies. They are operating in four different countries on three continents. A couple of years ago, they were not an IT company and didn’t collect any data. Today they are a data-driven organization, which gives them a massive advantage over their competition! Based on data, they can optimize their revenue, directly update their fish harvesting processes, decrease their environmental footprint, and even cure fish diseases by changing some environmental variables — and this is only the beginning of their data-based abilities 😉

So what can we learn from them?

Lesson 1: Try to understand why data is essential.

First, let’s start with why we need data in our organization. There are many reasons for that. We need them to:

Make informed decisions — not based on your gut feeling, but on the actual data
To track and improve the performance of specific processes — to optimize something, we need to measure it
To ensure compliance
To make data-driven decisions
To automate and create predictive systems (like legendary predictive maintenance applications 😉)
In the case of our client, many decisions people made based on their gut feelings and previous experience, but no one could tell the exact reason for specific feeding patterns or harvesting time. Moreover, these decisions were specific to different locations. Because of this, it took a lot of work to improve something or to standardize it. These challenges led to one conclusion — we needed to collect data in our organization and become a data-driven company!

Lesson 2: Start collecting your data as soon as possible!
Working with data is tricky — at the beginning, you need to know which of them you will need in the future and store them. I always recommend collecting all possible data at the start — sure, some of them will be junk — but thanks to that, you are building your organization’s data awareness culture. At some point, you will need to curate those data and create data governance standards, but at the beginning, start collecting them.

To do this, you need to figure out to which data you already have access and what other metrics, logs, or data you need to start collecting. Sometimes it will require new IoT systems that can measure new parameters. In other cases, you could stream data from existing applications or build simple ETL processes to collect them. At this stage, concentrate on one thing — how to collect data.

Lesson 3: Create first insights based on stored data.

At this point, you should have some data stored. Try to prepare simple reports showing their utility to others in the organization. It could be elementary reports with some KPIs or metrics that will offer some basic but essential information for them. Start with small steps and ask your colleagues what information will be most beneficial for them, and then make them more visible using BI tools.

In this specific case, together with the client, we created simple PowerBI reports for business owners, c-level, and people that work directly in their facilities.

In this step, our primary goal is to deliver first insights that can help others and build fundaments for a data-driven decision process.

Lesson 4: Add more advanced analytic mechanisms and create your data architecture.

At this point, your organization already collects data and is aware of the benefits of using them. It is the time to use their full potential. To do this, you need to do two things. The first one is to create your data architecture and data governance standards. If you are using the public cloud, you should develop your data landing zone and data management zone (I will explain what they are in another article).

The second one is the process of adopting advanced analytic solutions. To do this, start asking questions and defining fundamental problems that can be solved using data. After that, go to the technology part 😉 Try to use some data science tools and methodology to create models that will be able to answer those questions. Create more advanced reports with dynamic customization based on specific parameters. Make simple predictive systems that can tell users which actions they should take to have specific results. Explore and try to adopt the advantage of data fully.

Lesson 5: Create insights personalization and embrace data-driven decision process.

You now have many reports, systems, and insights for your co-workers. They can see every single thing using data that your organization collected. But wait, for some reason, they are not using them! The problem lies in complexity and data chaos. Too much data have the same value as no data. There is a way to overcome this. Here you will need to use some AI mechanisms and create personalized data feeds for our end users. Imagine that every day a user will receive an e-mail that they can read in five minutes with their morning coffee. In this mail, they will have every vital insight, data summary, and recommended actions for today. It will prime their brain to make data-driven decisions and increase user experience. If they are interested in specific information, they will find it themselves using one of the reports. If the system detects something important throughout the day, it will notify the user.

To do this, you need to create automation based on AI mechanisms that will react to particular conditions and inform the user about them. That will be able to create a simple data summary every morning with selected data for the user.

Only then — when the user receives filtered data with suggestions and essential insights the data-driven decision process will work on daily basics in your organization.

Those are five lessons learned in this specific case. I think they will help you to adopt data fully in your organization. Please share any questions, suggestions, or other ideas in the comments!

Going to the cloud? Start with Landing Zone!

Damian Mazurek — Sat, 14 Jan 2023 16:48:56 +0000

Many companies are considering the public cloud as a solution to many problems. Scalability, security, global replication, increased development process speed, or cost reduction are the most popular reasons for choosing the cloud. After a year, or maybe two, comes disappointment. The public cloud is more expensive, application performance is weaker, and security doesn’t exist. There are many unknown resources, and no one knows their true purpose. It is complete chaos. Many smart people in the organization are starting to wonder: what has gone wrong? 🤔

The answer is simple — they didn’t start from Landing Zone and Cloud Governance standards! Ok, but what is Landing Zone? Why is it so important? Let’s try to find out!

Landing Zone

In aviation, the landing zone is a predetermined area where aircraft can land and take off. The landing zone must be large enough to accommodate the plane and have a clear approach and departure path.

The same concept is in the public cloud — here, Landing Zone is a place where you can deploy your applications to a dedicated space in a fully controlled process. There are many similarities between those two areas. In aviation, you need to know which plane will be landing, talk with the pilot, separate air traffic, and have security standards that must be followed. Cloud computing is the same. You need to be able to identify your resources based on tagging, separate network traffic for them, continuously monitor your application, take some automated actions if something goes wrong, and enforce security policies.

In cloud computing, the landing zone is a complex environment that needs to address several aspects:

Resource governance
Networking
Identity
Security
Monitoring
Infrastructure
Cost management
Automation and CI/CD processes
Data management and architecture (we can create sub-zones called data landing zone and data management zone).

Every one of these aspects has a couple of different elements. For example, in resource governance, we need to:

plan resource hierarchy,
naming convention
standardize tagging
build policies (that will force or block some behaviors)

In networking, we must figure out how to:

create hybrid connectivity,
allocate IP addresses,
manage DNS,
create PaaS solutions networking
create network architecture
plan secure internet communication

In identity, we have to:

plan accounts management
create or use specific roles
manage hybrid identity if we have on-prem identity providers
create privileged identity management mechanisms (or use cloud-based existing ones)
manage service accounts for our resources and applications

Working on infrastructure, we must:

specify resources allowed by our organization
choose geo-localizations for resources
plan disaster recovery mechanisms
design HA configuration for resources
build scaling mechanisms

There are many more different aspects that we need to address to create a fully operational cloud environment. Most of these topics we need to consider at the beginning of our cloud journey, especially when talking about big organizations that already have some on-prem systems.

Landing Zone example for MS Azure. Source: https://learn.microsoft.com/pl-pl/azure/cloud-adoption-framework/ready/landing-zone/

Do we need it at the beginning?

Let’s imagine that in our organization, somebody has an idea to start working with the cloud. This person is searching the web for information on cloud computing, doing online courses, and maybe visiting meetups for technological inspiration and knowledge. Then there is a decision — let’s try to do something in the public cloud. At this point, you do not need a landing zone. You need to experiment with the cloud and check if it will fit your requirements. You can use a cloud web console, try to create some resources, and maybe make a simple PoC. You need to get familiar with the public cloud concept and all benefits and risks of this model.

So when to start thinking about a landing zone? When you want to put your first production system inside the cloud. It is the point when you need to build a robust cloud environment, and a landing zone is an excellent way to do this.

How to build a landing zone?

Building a suitable landing zone could be a challenge. There are materials from every cloud provider that describe the design process and many open-source codes with LZ implementation in Terraform.

The problem is that they are very general.

Let’s get back to aviation. Every airport is different, and because of it, landing zones for each one will be different and fully customized for this specific airport. The same situation is in cloud computing. Every organization has a different culture, technological stack, and people, so why should they have the same landing zone? To take full advantage of cloud computing, you must have a landing zone customized for your organization. For that, it is good to use an external consultant — who already has experience in this area. They will help you and will be your guide in the cloud adoption process. This person specialized in this task and worked with many different companies. The consultant has seen which approaches work in other companies and will be able to point out the most significant things you need to take care of in this process.

How should the process of creating a landing zone look?
I know that every consultant can have a different approach, but here I want to share with you one that, based on my experience, is the most efficient. It consists of three phases: knowledge sharing, gathering insights, creating architecture of LZ, and the last one — implementation.

Build fundamental knowledge about cloud computing in your organization.

It is always the first step in our cloud journey. To create a fully operational landing zone, you need committed people who understand the public cloud concept. They need to know what cloud services exist and how they can use them. Here it is essential to build the T-shape skills model.

The vertical bar in the “T” represents a person’s depth of knowledge and expertise in a specific field. This could be a particular subject area, like networks, data engineering, software development, etc. The horizontal bar in the “T” represents general knowledge about the cloud computing model. Understanding how PaaS, SaaS, and IaaS services work in the cloud and when you should use them. Be familiar with cloud cost models, monitoring and observability, and access control.

Many companies at the beginning try to skip this step, but if you do it, there will be consequences in the future. Cloud is complex, and mistakes at the beginning can be costly in the future. You need to have a team with good cloud skills.

Start from workshops where you will be able to gather all requirements and create the architecture of your landing zone.
In this stage, when our people understand specific concepts of the cloud, we can start gathering insights and requirements. I always do this in the form of workshops.

For every topic, we schedule 2–4 hours of workshops.

Workshop list:

Resource governance
Networking
Identity
Security
Monitoring
Infrastructure
Cost management
Automation and CI/CD processes
Data management and architecture (we can create sub-zones called data landing zone and data management zone).
At this point, we create high-level architecture for each aspect and work on standardizations.

Every workshop requires specific specialists from your organization to share their insights on particular topics.

Create documentation with cloud governance standards based on workshop results.

After the workshops, you need to document the landing zone architecture and cloud governance standards. You must go from high-level to low-level architecture for our landing zone at this stage. This is the stage when you need to create also standardization and governance policies for your cloud environment.

Implement your landing zone in the IaC approach.

The last stage is the simplest one — you need to implement everything in the IaC approach. To do this, you need to use a landing zone low-level architecture and cloud governance standards as inputs. Based on the cloud you are using, you can choose one of a couple of IaC technologies.

Infrastructure as code (IaC) is a practice that involves managing infrastructure using code and configuration files rather than manually configuring resources through a web-based interface or using a manual process. This process allows your organization to automate infrastructure resource provisioning, configuration, and management, making it easier to deploy and manage complex environments.

Before the implementation phase, you should decide which tool you will use. I will list a couple of them that, in my opinion, are worthy candidates:

Terraform: Terraform is a tool that enables users to define and manage infrastructure as code using a declarative configuration language called HashiCorp Configuration Language (HCL). It supports many infrastructure providers and can manage infrastructure across public clouds (AWS, GCP, MS Azure), on-premises, and hybrid environments. Currently, it is the most popular tool for IaC, so there are many publications, videos, and online courses about it. You can also easily find cloud engineers that are very skilled in it.
Pulumi: is a cloud-native infrastructure as code (IaC) platform that enables users to define, deploy, and manage infrastructure using familiar programming languages, such as JavaScript, TypeScript, Python, and Go. It allows users to create and manage resources across many clouds and on-premises platforms, including Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), and Kubernetes. With Pulumi, users can define infrastructure as code using a high-level, declarative syntax, which is then translated into the appropriate API calls for the target platform. This enables users to define and manage their infrastructure in a version-controlled, reusable, and maintainable way, making it easier to deploy and manage complex environments.
Kubernetes Operators: Kubernetes operators are custom controllers that extend the Kubernetes API to create, configure, and manage instances of complex, stateful applications in a Kubernetes cluster. Operators use the Kubernetes API to watch for and respond to events in the cluster, such as the creation of a new resource or the scaling of an application, and can be used to automate the management of complex workloads. While Kubernetes operators were originally designed to manage applications and resources within a single Kubernetes cluster, it is possible to use operators to control resources in public cloud environments, such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). To use Kubernetes operators to control public cloud resources, you will need to install and configure the operator in your Kubernetes cluster. This typically involves creating a custom resource definition (CRD) for the operator and deploying the operator’s code and any necessary dependencies to the cluster. Once the operator is installed and configured, you can use it to create, configure, and manage public cloud resources by creating and modifying instances of the operator’s custom resource. For example, you might create a custom resource to define an Amazon EC2 instance and use the operator to create and manage that instance in your cluster. When most of your workloads will be in the cloud-native model, the approach of K8 operators to IaC may be the best solution for you.
Azure Bicep is a domain-specific language (DSL) for defining Azure infrastructure as code (IaC) in a declarative way. It allows you to write reusable, maintainable, and testable infrastructure deployment templates in a simple and intuitive way. Bicep is designed to be easy to learn and use, especially for those familiar with Azure and familiar with JSON or YAML syntax. It provides a more concise and user-friendly syntax for defining Azure resources, as well as a set of built-in functions and directives for extending and customizing your infrastructure deployment templates. With Bicep, you can define all the resources and their dependencies needed to deploy an Azure solution, and then deploy the solution using Azure Resource Manager (ARM) templates. Bicep compiles your Bicep files into ARM templates, which are then deployed to Azure using Azure CLI or Azure PowerShell.
AWS Cloud Formation is a service that helps you automate the process of creating and managing AWS resources. It enables you to use templates to define and deploy your infrastructure in a predictable and repeatable way. CloudFormation templates are written in JSON or YAML and use a declarative syntax to specify the AWS resources and their properties that you want to create. These templates can be used to create, update, and delete stacks of AWS resources, such as Amazon EC2 instances, Amazon RDS databases, and Amazon S3 buckets. CloudFormation provides a set of built-in resource types that you can use in your templates, as well as the ability to create custom resource types using AWS Lambda functions. You can also use CloudFormation to automate the provisioning and management of other AWS services, such as Amazon ECS, Amazon EKS, and Amazon DynamoDB.

After choosing a specific tool, you can start implementing your Landing Zone based on low-level architecture and cloud governance standards.

If you decide to use K8s operators, you must create a seed code in Terraform to create your first management cluster and basic infrastructure for network architecture and hybrid connectivity.

You should invest in the landing zone!

Landing Zone is a crucial part of the cloud adoption journey. Yes, it can be complex, especially at the beginning, but you must build it. You can always use external consultants as your guides in this process if it is too complicated for your organization. A good landing zone will help you better control your cloud resources, have a higher level of security, and lower your cloud infrastructure costs. It is an investment that you should take.