Forem: Sean Overton

4 more lessons from a senior engineer that you may not know yet

Sean Overton — Sun, 06 Apr 2025 10:24:21 +0000

Many articles aim to define what a senior engineer is and how to become one. That is not this article. The motive of this article is simply a reflection for couple lessons I have relearnt over and over again in my software engineering career as I have matured and become more senior. This article will not be the secret recipe for becoming senior because other resources online cover that topic but instead focuses on one lesson that has personally stood out to me that many engineers still need to learn.

1. Engineering is almost never binary

That is, the 'correct' way of engineering something may be the 'wrong' way of designing the same thing given different circumstances. To not set yourself up for embarassment, before you judge the 'correctness' of a design especially when you are early in your career you should question, even though I think this could be 'better' (in my opinion) from x design principle I learnt at university or insert-some-other-trending-technology, ask yourself this question: 'why would some senior engineer design it this way?'

Before you start saying 'x is wrong and y is how we can fix it.' reconsider why it would be engineered this way.
Even better, read any design docs or ask the original developers if you can. In my career so far I have found that more often than not I am incorrect in my initial naive reaction of 'x is bad, and if we simply apply y it will fix it'. I am an advocate to give the original engineers the benefit of the doubt.

Although be wary of the other side too, although I am advocating to give existing solutions the benefit of the doubt, they could still be objectively 'bad' but be open to more context. One example to be wary of is previous engineers personal agendas or learning interests for specific technologies at the time. If x engineer just wanted to learn about WASM so they wrote the startups entire frontend with it. This wasn't necessarilly a good decision for the business but again just understanding even this agenda can still help influence and improve future decisions.

A particular context factor to remember is:

2. Consider the 'maturity' of the product

The maturity of the product; was/does it currently make money? was/does it have users? was/is it profitable? was/does it have 100 000 Daily-active-users (DAU) or 100 DAU. These factors all significantly influence the engineering decisions in the product. These external factors could be anything surrounding the product and should be an important reminder to engineers of why they do need to care about having an understanding of the business and product.

3. Learn the product and business

As software engineers, the bottom line is that we are meant to be deliverying value to the business. This is important to remember and connect the dots to revenue. But more than just connecting to revenue, learning the current business context is most important for making correct engineering decisions for the business right now and for the future. It is also imprortant to remember that these context factors change over time.

This means, an engineering solution that was quick and dirty 2 years ago when the project's goal was to get it's first 100 users and may be proving to be an extreme heartache for engineering now that the product has 100 000 DAU is a perfectly normal and justifiable experience. This is a common phenomena that all engineering teams around the world will experience. The poor engineering decisions of our past becomes what we call technical debt.

Understanding at a high level the previous stages and growth of the business and product allows engineers to understand and appreciate the context of where products have come from but more importantly the why behind the engineering decisions and the why behind technical debt. You don't need to know everything but learning the fundamentals and a brief history of the product from a business stand point will do you wonders.

4. Technical debt is normal

Startups may not have automated testing or an automated deployment pipeline or any proper design system or whatever the next 'best practice' is to introduce. As previously mentioned, it's never black and white, some things are good in some cases but it depends on the circumstance. Just because you learn some new perfect design pattern, doesn't mean it will be the perfect solution for all situations. Naive engineers have a tendency to learn something and think this will now work for everything, but in engineering it's always compromise of weighing up pros and cons.

A warning to check your ego

This will depend on company to company and team to team, but at some point in your career you will join a certain project and be blown away by the tech debt. Seemingly senior engineers working on huge successful corporations or unicorn startups will have mountains of tech debt and spaghetti code. You may even naively think to yourself despite your lack of experience: "woah, I could do better than this?!", "How did they get away with this?" etc. etc. the thoughts continue. But I write this as a warning.

It's generally naive to criticise the tech debt as soon as you see it. It may be justifiable to the business and you can't blame the engineer for poor engineering. Context changes, just because the design is obviously innapropriate now doesn't mean it was not the seemingly best solution when it was first introduced.

On the flip side of understanding the current state of a project. You have to also be mindful of the new decisions you introduce. Just because you learnt about microservices architecture and how scalable it is at university, do we need to implement this for our new early stage startup that has no users yet? Definitely not. This is over optimising and over engineering. So we start with a monolith. 5 years later another straigh-out-of-uni graduate joins, sees all the issues they are having scaling the monolith and thinks out loud: 'the engineers that designed this were so naive. They should've built it with microservices from the start' but this would be added complexity and overheads for the startup stage and could've killed the company before it made any money. The company at the early stage just needed simplicity so they could focus on tangible features for the customers.

I recently attended the 'Programmable' conference in Sydney, Australia where a distinguished tech leader (and current CTO) did a fantastic talk on addressing technical debt ('Legacy code'). His talk again reminded me of how engineering decisions are never black and white and that technical debt is normal. One particular quote stood out to me:

If you don't have bad legacy, you have over engineered
Jeremy Burton

This line elegantly captures majority of this article.

TLDR; in summary

Tech debt is healthy and normal. Too much will lead to an unsustainable product but too little is a sign of over engineering and could kill the company. 'Good' engineering decisions are never binary. The maturity and context of the product has significant influence on what 'good' engineering looks like at that time. The product context changes over time so the engineering also has to evolve. This should be considered when looking at existing projects or working on new features to get the most appropriate outcome for the business.

Thanks for reading!

Hey, I am Sean Overton, a passionate software engineer and technical writer who strives on deliverying real user value in products.

Please share with any colleagues that are interested in becoming more well-rounded engineers.

3 quick tips for your upcoming system design interview

Sean Overton — Sat, 23 Nov 2024 22:01:20 +0000

I recently read "System Design interview volume 1" by Alex Xu from the ByteByteGo youtube channel and wanted to take some time to write a quick review and share some tips on system design interviews.

The book was a fantastic read with an easy to follow methodology to approach system design interviews, many commmon system design examples and walkthrough solutions and addresses the essential building technologies that you can pull together in a system. It is quite a comprehensive read and I would recommend for both people who are preparing for a system design interview BUT also for engineers looking to improve their system design practices in general.

Here are some extra tips for your upcoming interviews:

1. Know your cloud technologies.
This is not necessary but I think is the practical application of system design. By applying and actually building your systems and knowing how to build them practically allows you to gain a more intimate understanding of the black boxes you might mention in a system design interview. This will be obvious in interviews when you discuss details and are capable of deep diving into the blackboxes that are traditionally brought up in system design interviews.

2. Practice common back-of-the-envelope calculations.
Know your commmon calculations. Honestly for myself before reading the System Design Interview book I did not realise how important this might be and also the specific important areas where calculations would be necessary. I believe their is still a bit of an art to deciding where to focus your attention during an interview but asking your interviewer and working with them is one of the best approaches you can take.

3. Practice the common questions.
Obviously you can not prepare for every scenario and interviewers will always be creating new problems but I have heard and seen countless times that they will ask common problems. It does not hurt to be prepared for these designs as it is common practice for the problems to be updated with a 'twist' to make it slightly more unique. Both of the "System Design Interview" books (vol. 1 and vol. 2) provide plenty of the common problems to practice and learn from so I highly recommend them.

Thanks for reading!

This has been a very quick review of the System Design Interview books by Alex Xu as well as some tips for your upcoming system design interview.

Thanks again for reading and feel to leave any thoughts in the comments!

Follow me for more book reviews and quick tips I pick up along the way as I continue my book-per-week reading journey.

For my full reading list: https://github.com/SeanOverton/books-i-have-read/blob/main/books.yml

A practical introduction to GitOps

Sean Overton — Tue, 05 Dec 2023 06:04:47 +0000

Here is the code base used in this tutorial:
https://github.com/SeanOverton/example-gitops-serverless-infra

What is GitOps?

GitOps is a methodology using Git as the source of truth for both infrastructure and application code. It's declarative, ensuring that desired system states are specified in version-controlled files. Changes trigger automatic deployments through CI/CD, providing easy rollbacks and robust version history. Popular in cloud-native environments, GitOps simplifies managing and deploying complex systems.

Why GitOps?

GitOps is embraced for its efficiency and reliability in managing and deploying software systems. By making Git the single source of truth, it ensures a declarative approach to configuration, allowing teams to define and track the desired state of both infrastructure and applications. This methodology automates deployments through continuous integration and deployment (CI/CD), streamlining processes and reducing manual errors. With version-controlled changes, GitOps provides a robust audit trail and simplifies rollback procedures. Particularly advantageous in dynamic, cloud-native environments, GitOps enhances collaboration, observability, and overall system resilience, making it a preferred methodology for modern software development and operations.

Q. Who is the GitOps Engineers customer?

A. Other developers!

It is fundamental to GitOps engineering to understand that your 'customer' is other engineers and your 'product' is the repo being provided. This means that the customers and product should be treated similarly to any other outbound facing product. For example, active communication and feedback should be established with the customer.

What we will cover in this tutorial

In this tutorial, we will explore the concept of GitOps with an example demonstrating how to build a self-service code repository (using terraform and AWS) to enable developers to quickly deploy REST API's with minimum understanding required of the underlying architecture so developers can focus on the application code itself. The patterns introduced in this example are common to GitOps engineering and help communicate the fundamentals of what GitOps actually is.

Who this tutorial is for?

Any aspiring engineers looking to automate more, anyone interested in getting hands on with IaC and anyone interested in understanding the power of GitOps.
It is expected that readers have some knowledge required as a pre-requisite such as basics with GitHub, Git and AWS (including an AWS account).

Building a Self-Service Code Repository:

A self-service code repository allows development teams to manage, share, and collaborate on code efficiently. In the following steps we will build an example self-service repository that allows a developer to quickly deploy an API utilising AWS serverless computation: lambdas.

Enough talk, let's code!

The code is available here:
https://github.com/SeanOverton/example-gitops-serverless-infra

What does the repo do?

build AWS lambda functions (by default runs python)
build an AWS rest API gateway, with associated endpoints for each lambda created
optionally add cognito authorizer to endpoints that require auth

Setting up:

Fork repo: https://github.com/SeanOverton/example-gitops-eng-serverless-infra/tree/main
Sign in to the AWS console and create a user with Access Key for programmatic access. https://docs.aws.amazon.com/keyspaces/latest/devguide/access.credentials.html
Note: for a production environment it is not recommended to have use access credentials with admin permissions (but for this tutorial, feel free to attach an admin policy). Roles with the concept of least privilege should be applied. *
Add these credentials to GitHub secrets with keys:

AWS_ACCESS_KEY
AWS_SECRET_ACCESS_KEY

Create an S3 bucket via the AWS console that is only accessible by your previously created user (which you stored the credentials). This bucket will be used to host the tfstate file. This is a dependency of using terraform as seen in this architecture. Your bucket name will need to be unique and will have to be added into the terraform config at: ./main.tf

terraform {
  backend "s3" {
    bucket = "bucket-name"
    key    = "terraform.tfstate"
    region = "ap-southeast-2"
  }

Configure AWS CLI credentials locally and run terraform init to initialise terraform backend state file in the S3 bucket.

How a developer interacts with the repo

Infrastructure config

Once the project is set up from the above steps. A developer would only have to update the ./config.tfvars file to build infrastructure for the new endpoint. The changes only require understanding of a few minor parameters as shown below.

But first create a branch locally: git branch feature/gitops-is-fun

Then make the below changes.

lambda_functions = {
  + helloworld = {
    + function_name   = "helloworld"
    + auth_required   = false
    + endpoint_method = "GET"
  + },
}

cognito_user_arns = []

stage_name = "default"

The example config code above will create the endpoint and AWS cloud infastructure to support an endpoint that can receive HTTP GET requests. The application code will be added next.

The endpoint application code

A new folder and file can now be created to store the endpoints application code: ./lambdas/helloworld/lambda_function.py
Warning: the folder name 'helloworld' matches the function_name from the previous config!

This file will be what runs at the above configured GET request endpoint. Add the following helloworld response to this file:

import json

def respond(err, res=None):
    return {
        'statusCode': '400' if err else '200',
        'body': err.message if err else json.dumps(res),
        'headers': {
            'Content-Type': 'application/json',
            'Access-Control-Allow-Origin' : '*',
        },
    }

def lambda_handler(event, context):
    return respond(None, "Helloworld")

Now the hardwork is done! Watch the platform's magic work!

Merging in

Push. git push
Pull request. On GitHub, create a pull-request from your recent changes.
This will trigger a plan workflow that ends with something similar to:

...
Plan: x to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tf-plan

To perform exactly these actions, run the following command to apply:
    terraform apply "tf-plan"

This is terraform highlighting what changes will be made to your current infrastructure. This is a dry run when you can detect if everything will work as expected before actually deploying the infrastructure.

Merge. After reviewing the expected changes. Merge the PR into the main branch.
Now check the actions tab to find your deployment url in the latest workflow run. (This was defined as a terraform output)

...
Apply complete! Resources: x added, 0 changed, 0 destroyed.

Outputs:
deployment_url = "https://<deployment_url>/<stage>"

Test!

Visit: https://<deployment_url>/helloworld.
You should receive a helloworld response. If not, review the steps and checkout the workflow outputs.

Understanding the repo itself

The repo has a couple of stages that need to be understood.

1. Terraform Modules (IaC) and Config
The first stage is the terraform config. This is the tfvars that was edited ./config.tfvars. The .tfvars file type is a file type used for passing user defined values for input variables into terraform code similar to parameters being passed into a function. Note the function_name parameter is important here and will be used for the API endpoint name ie. https://unique-aws-path/<function_name> as well as required for the folder structure defining the application code: ./lambdas/<function_name>/lambda_function.py

2. The GitHub action (automated workflows for continuous deployment)
The GitHub action (defined at .github/workflows/terraform-workflow.yml) is the automated workflow that automates the terraform dry-run and deployment.

2.1 The terraform plan is triggered when a pull-request is raised which causes a dry-run to run and would allow a team to effectively review the expected changes before actually applying them.

2.2 The terraform apply (which only runs when changes occur in the 'main' branch) will use the above created plan and deploy it.

When used with GitHub branch protection with minimum required reviews this then allows for a safe but efficient workflow for teams to review the expected changes, approve and deploy the API endpoints quickly.

3. The application code
This is referenced in the terraform modules to be deployed at the AWS lambda function and was defined in ./lambdas/helloworld/lambda_function.pyin the example above. It should be noted, that any name could replace the helloworld in this case and as long as the tfvars config is also updated with corresponding information the new API endpoint code will be deployed.

Conclusion

This has been a hands-on introduction demonstrating an example of what Gitops engineering may look like!

Do you want to instantly improve your API design?
Check out this article: https://dev.to/s3an_0vert0n/10-quick-tips-to-instantly-improve-api-design-59mc

Follow me for more book reviews and quick tips I pick up along the way as I continue my book-per-week reading journey.

For my full reading list: https://github.com/SeanOverton/books-i-have-read/blob/main/books.yml

A quick Functional Programming recap

Sean Overton — Mon, 02 Oct 2023 06:17:25 +0000

I recently read Grokking Simplicity: Taming complex software with Functional Thinking by Eric Normand to formalise my understanding of the foundations of functional programming as I realised I had only ever been introduced to this paradigm anecdotally from colleagues. An extremely practical book that breaks the Functional Programming (FP) stigma that it is a traditionally very theoretical subject. The book avoids the maths and presents digestible, actionable tips and examples laying the foundations. Read on to understand what, why and how it is applied.

What is FP?

"Functional programming provides techniques for organizing side effects so they don't occur just anywhere"

So what are side effects?

Some examples:

Implicit arguments such as global variables
If a variable is passed by reference and it is written to inside the function
Anything that has an external affect such as sending an email or interacting with an external API
A function is defined as impure if it has any side effects. These functions are also sometimes referred to as actions.

So what are pure functions?

Opposite of impure functions XD
Inputs are all explicit arguments. That is they are passed in as actual function arguments. As opposed to an implicit argument such as global variable, which is still an input to a function BUT not passed in as an argument.
Outputs are all returned values (no implicit outputs)
No globals (implicit arg)
Data/pure functions are 'idempotent' (idempotent meaning it doesn't matter how many times it is run/accessed it returns the same result)

Comparison by example:

var someGlobalData = [];

function impureExample(){
someGlobalData.doSomething(); // this global var is both an implicit argument and implicit output
}

function pureExample(explicitArg){
// the function argument here is explicitly an argument
explicit_arg.doSomething()
return true; // explicit output
}

As you can see in the above example, if you were to call the impure function, it can be difficult to tell what side effects will occur, while a pure function is a lot more predictable as you supply the data and can expect a returned value. Hopefully that helps clarify the differences as well as starts showing why Functional programmers tend towards pure functions where possible.

Now, why is this beneficial?

Functional programming works well in distributed systems where Idempotency is important. Idempotency is important when concurrent read/writing and/or actions could be occuring.

Pure functions are also easier to test and more predictable.

Note: this does not mean FP should be applied in all cases (OOP still has its use cases (; ).

Functional programming in practice:

The following is a few key tips to help apply FP:

Use pure functions where possible (it's impossible to eliminate because they are where actual actions occur but aim to reduce and control where they occur).
Impure functions create other impure functions
eg.

// this function is also impure because it calls another impure function. The impurity spreads to all the calling functions.
function send_email_wrapper(){
    send_email() // this function is impure because it sends an email ie. it has a 'side-effect' and effects an external system
}

Keep impure functions small: This can be achieved by pulling all of the data and whatever pure functions you can to be external of the impure function
seperate data from functions so they evolve independently and are decoupled
Make pure functions generic
Use immutable data to avoid side effects. Immutable data is data that can't be changed. So how do you make a system that is not completely static? Make copies, and you will have to change the copies. Read on.

Defensive copying:

This is the idea that by copying the input data inside a function and returning a copy of the data, no side effects should escape from a function.

Summarised as:

Copy data as it enters the code
Copy data as it leaves the code

Dealing with async functions:

A general rule for helping to make async functions as 'pure' as possible is to 'return' a callback function (by calling it) rather than returning a value.

Thanks for reading!

This has been a very brief summary highlighting the foundations of Functional Programming as explained in Grokking Simplicity: Taming complex software with Functional Thinking by Eric Normand. A fantastic read for anyone looking for a practical read on functional programming principles.

Thanks again for reading and feel to leave any thoughts in the comments!

Follow me for more book reviews and quick tips I pick up along the way as I continue my book-per-week reading journey.

For my full reading list: https://github.com/SeanOverton/books-i-have-read/blob/main/books.yml

8 tips from the Engineers' Survival Guide

Sean Overton — Sun, 17 Sep 2023 08:26:01 +0000

This week I read Engineers' Survival Guide by Merih Taze and I can't recommend this book enough for all software-engineers. A fantastic read packed with insightful anecdotes and tips for advancing and improving your career. The following is my top 8 tips from this fantastic book.

1. Visibility is everything.

The author Merih Taze highlights in this chapter the importance of being 'known' at your workplace and how much it will positively impact you career. When it comes to promotions or finding the right person for a new opportunity people should recognise who you are because of your visible contributions across the company making it easy for them to accept and recommend you. Some simple tips he shares to improve your visibility include:

Share status updates to your team but also the wider community if it is relevant.
Engage on public posts.
Be active on design documents.
Comment on code reviews.
Avoid answering DM's and request to share them on the wider community dev channels to answer more people that might have similar questions.

2. The importance of tools and patterns.

Always consider if their is a tool that can do your task better, faster and with fewer mistakes. Stay active in learning and discovering new tools. For example see if their is a CLI tool that can help parse this document and extract what you need. etc.

3. Side projects and knowing your surroundings.

This chapter outlined why it is good practice to network with other teams and learn what other projects are going on around the organisation. The benefits of this include:

Becoming more senior as you are the goto person for checking if ideas are worth pursuing.
Connect and help other teams via design discussions. Growth opportunities.
Opens new career opportunities. You may be able to join projects you are more passionate about.

4. Importance of networking.

This is similar to some of the previous points. Networking with colleagues, which may come across as daunting for many engineers can be greatly beneficial for your career if used correctly. Never think of networking as I need to network to gain something but just introduce into your day-to-day life. Get involved in whatever networking opportunities present themselve.

5. Do not redesign a working system.

This might be a tough one for engineers to digest. And their may be exceptions to this rule but the costs vs benefits should be seriously considered before undertaking a redesign. Merih Taze emphasizes that so much business logic will be tied up in that spaghetti code where various bugs and exceptions have been caught to get to this stable system that it can be a huge amount of work to redesign or reimplement a working system. He suggests careful consideration needs to be made as to why and for what benefits the changes are necessary.

6. Knowing popular frameworks and how they work.

This chapter was one that particularly resonated with myself. A general high level knowledge of popular frameworks will enable you in wider community design discussions and even just understanding what different teams across the company are up to and general chat with other engineers, helping you build relationships. For me personally, I could quickly develop a respectful relationship with senior engineers as a junior engineer who was aware of the wider variety of frameworks for both frontend and backend engineering in a suite of languages.

7. Use debuggers. Learn them and master them.

Debuggers are a seemingly illusive tool that many junior engineers surprisingly manage to avoid proper contact during the early stages of their career. My personal anecdotal data would suggest universities struggle or complete miss the opportunity to teach this fundamental skill of software engineering. Merih Taze suggests that learning and becoming a master at using a debugger in your respective IDE is a vital skill for progressing as an engineer. If you find yourself typing print(some_variable) several times in your code while debugging, you need to start learning and using a proper debugger and reaping it's rewards.

8. Align, align, and align again.

This one, I feel like is one of the most powerful comments the Engineers' survival guide makes. It suggests that as an engineer and employee you should align your goals with your team, management, division and company. This should be revisited often as change is common and ensures as an individual you are making sure you are impacting and adding value in the area's that the company needs it most.

Thanks for reading!

That summarises my top 8 favourite tips from the Engineers' survival guide by Merih Taze. I definitely recommend getting a copy for yourself.

Thanks again for reading and feel to leave any thoughts in the comments!

Follow me for more book reviews and quick tips I pick up along the way as I continue my book-per-week reading journey.

For my full reading list: https://github.com/SeanOverton/books-i-have-read/blob/main/books.yml

10 quick tips to instantly improve API design

Sean Overton — Sun, 17 Sep 2023 05:14:13 +0000

After recently reading "REST API design rulebook" by Mark Masse, I am writing to share a couple of quick take-aways for designing web API's.

Here are the 10 quick tips. Happy reading!

1. A singular noun should be used for document names.

eg. https://example-domain.com/blogs/my-blog

2. A plural noun should be used for collection/store names.

eg. https://example-domain.com/blogs

3. A verb or verb phrase should be used for controller names.

Controllers are endpoints that have a non-standard function outside of HTTP standard method types such as GET/POST/UPDATE/DELETE.
eg. https://example-domain.com/send-email

4. CRUD function names should not be used in URIs.

eg. Code smell:
https://example-domain.com/update-post/1234
or https://example-domain.com/delete-post/1234

Instead this should be the same endpoint and the correct http methods should be used:
PUT https://example-domain.com/posts/1234
DELETE https://example-domain.com/posts/1234

5. Query parameters are for either filtering or pagination:

eg. GET /posts?author=sean-overton

These should support a 'partial' or filtered JSON response where only required fields are returned.

6. Use appropriate HTTP methods for their expected functions.

Post = Create object /posts
Get = Retrieval /posts or /posts/1234
Put = Update Object in place /posts/1234
Delete = Delete's object in place eg. /posts/1234

Don't tunnel other request methods via GET/POST. This is a common code smell observed in junior API developers code.
eg. don't delete an entity with a GET request. Use the delete HTTP method.

7. Use response status codes appropriately. (so they can be handled appropriately in the client!)

100: Transfer protocol information
200: Okay/success responses
300: Redirects
400: Bad requests (client side error in request)
500: Server side errors (server side error occurs)

8. Use Http Headers with appropriate metadata defined.

eg. Content-Type: application/json (a required header!)

Note: These are occasionally abstracted away in frameworks with defaults but it is still recommended to test these for yourself and override when appropriate.

9. Use Cache-Control, Expires, and Date response headers.

This is to encourage caching which in turn should help improve performance. These fields can define time to live (TTL) for respective objects before they are required to be refected.

10. Errors should have a consistent form (JSON/XML schema):

eg.

{
"id" : 1234,
"message": "Some error occured"
}

This is so error's can be consistently handled in the frontend or client service.

Thanks for reading!

With the development of automatic code generation tools such OpenAPI generator (https://openapi-generator.tech/), the importance of API design rather than implementation is emphasized.

Follow me for more book reviews and quick tips I pick up along the way as I continue my book-per-week reading journey.

For my full reading list: https://github.com/SeanOverton/books-i-have-read/blob/main/books.yml