Forem: muthandir

On Communication.

muthandir — Tue, 05 Apr 2022 13:58:35 +0000

“Ineffective communication is the primary contributor to project failure one third of the time and had a negative impact on project success more than half the time.” - PMI, Project Management Institute.

Communication in a projects happens in different ways (sync: phone calls, meetings or async: email, chat (I think chat is async but that's debatable (not technically, but hypothetically - wow, that's the 3rd nested bracket))). But I believe communication is more than just participating in a conversation over a medium. Imagine a continuous deployment flow where committed source codes are deployed to QA servers every 30 minutes. If a simple Slack integration generates a notification with commit messages every time a deployment is successfully completed, then a QA engineer knows exactly when and what to test at any given time. Now remove the QA engineer and imagine an automated test suite receiving a similar message and immediately starting test execution. This makes me think that a well-established communication framework creates an information network and bridges the gap between stakeholders. Note that the nodes (or endpoints) in such a communication network can be humans and/or machines.

Event Driven Architecture

muthandir — Tue, 29 Mar 2022 13:57:59 +0000

When we create a software, we try to satisfy most (if not all) requirements that are critical to the business. But we all know there will always be new requirements (sometimes a very surprising one) that will make us scratch our head in order to implement. It doesn’t really matter whether we have a big monolith or microservices, we don’t want to clog up our application servers with actions that are not crucial to the core business activity. For example, if you click a button to book a flight, there are basic things the system must do at the transaction time. To name a few:

Charge the credit card (requires integration with a financial institution)
Make the actual booking (requires integration with a travel fulfiller)
Create transactional records (for reporting, invoicing etc.)
Send an email to the customer

I will keep the list short for the sake of simplicity, but you can guess how complicated it may get. From a business perspective, you will have to decide what a booking means in the most granular term for your business or what makes it incomplete. There is no right or wrong answer here but there is a common ground to which you will come closer. Most people would say #1 is crucial because this is the way business makes money. #2 is crucial as this is how the customer will board the airplane. #3 is crucial because this is the backbone of your system for bookkeeping, issue management etc. You may argue that #4 is also important but hey, maybe the email could drop in customer’s inbox a few minutes later.

Ok, so how will we implement it? I’ll break down below different parts of such a system in an event driven architecture using AWS services:

Message publisher(s): your applications will produce events (like the booking above) and publish the event messages. This is about notifying the system that something interesting did just happen. In our case we’ve made a custom NPM package that publishes simple AWS SNS messages for all (or filtered) http requests that our server replies. This way any incoming request (any user actions) gets the ability to produce an event in the system.
Message broker(s): The broker filters and routes the messages to the appropriate listeners (or subscribers). AWS SNS is one of the most straightforward answers in the Amazon world to build an application that reacts to high throughput and low latency events published by other applications. With SNS, you can route your messages to all subscribers (including Amazon SQS queues, AWS Lambda functions, HTTPS endpoints, and Amazon Kinesis Data Firehose) which would create a basic fan-out implementation. Alternatively, you can also do topic or attribute-based filtering for routing your messages to specific subscribers. I know that it sounds very tempting, however filtering your messages (using topics or attribute policies) can lead to very complicated rules that are hard to maintain in a real-world scenario. Plus, I don’t want to change a property in my infrastructure every time I need to make a change in event processing requirements. In most of the cases, I tend to do a fanout and inspect the messages in the workers using a NPM library that I built to filter-out the messages.
Recipient(s): For easy throttling and delivery reliability we added AWS SQS in-between our worker applications and SNS. In other words, SNS sends the event messages to SQS queues, and the worker apps listen to the SQS messages for event processing. This also helps with scaling because SQS is ultimately scalable and if you need to process more messages per second, all you need to do is to fire up another worker server and let it fetch messages from SQS.

In such a system, you may easily get lost trying to trace a transaction going back and forth between layers, so you will want to have a neat logging and tracing ability. You may find more information about logging in this post.

In the example above, there are still a few things you will want to do after the time of transaction:

Check the airline system (approximately) 5 minutes after the time of purchase (In some cases airlines systems create important information within first few minutes after you make a booking (like a free-upgrade, free lounge for corporate clients, last ticketing date and so on))
Send a reminder email to the customer prior the flight (generally 24h before)
Let’s add some more fun and imagine that this is a corporate purchase and the client wants bi-weekly invoices with the total amount of transactions that occurs in 2-week time frame.

These requirements go in the territory of deferred and batch executions, which I explain in the next post.

Trying to create an onboarding document for junior devs. What are the must-have items?

muthandir — Fri, 25 Mar 2022 20:30:16 +0000

Here are my 2 cents:

First of all I think the onboarding period is often too short. I observed that 3-4 weeks of preparations are of most help to new employees (with no or little professional experience)
Onboarding activities should be well documented.

Here are several real-world examples I created for a node.js backend developer that takes ± a month:

Development methodology
Programming Paradigms (Event loop, Promise chaining (for legacy codes), async-await, functional programming, ES6, exception handling)
Most frequently used libraries (mocha, sequelize, express, async etc.)
Version Control System (Git)
Tools (VS Code, ESLint, debugging)
In house libraries
Actual Code Structure
A task in the sprint (TBD later on)

How do you handle the data ownership of a multi-tenant product?

muthandir — Thu, 24 Mar 2022 18:23:57 +0000

Multi tenancy is very crucial for most of the SAAS products and I think, data ownership is one of the most important aspects of multi tenancy. So how do you deal with the data ownership?

(Note: by "data ownership", I mean:
-customer1 can only read customer1's data (and can never access customer2's data) or
-customer1 can only upsert data for customer1.

or in an enterprise situation,
-user1 of customer1 can only read customer1's data (and can never access customer2's data) or
-user1 of customer1 can only upsert data for customer1

Application Logging and Production Monitoring

muthandir — Wed, 23 Mar 2022 18:51:12 +0000

In my old days, I used to work in the corporate world as a developer, tech lead, architect etc. Back in those days I rarely worried about how we should do logging & monitoring. We always had tools, means and ways to get end 2 end visibility.

Later on, I co-founded a startup and my partner and I had to pick our tech stack. Me being a .net guy forever and him being a laravel pro, we went on with node.js 🙂 (For several reasons, but that is another story).

Back to logging, what we needed was the ability to save the entire lifetime of an incoming request. This means the request body/header info, service layer calls and respective responses, DB calls and so on. Additionally we wanted to use microservices back then (Again, another story with lots of pros and cons). So the entire lifetime also includes the communication between the microservices back and forth. So we needed a request id, and with it we could filter the logs and sort by time. Let me break it down to separate steps:

UI: We use a SPA on our front-end. The UI makes HTTPs calls to our API.

API Layer: Our business services in the APIs are instantiated using Factories which inject the dependencies. So in theory you could create a custom logger, enrich it with “request-id” and inject the logger to the business services for the use of developers, so they can log whenever they need so. But it feels like logging is not something we could leave up to our preferences. What we needed was an automated way to flush data. Additionally, the logs also reduce the readability and they could potentially cause bugs. (In theory, a business logic code should not be “polluted” with extra logging codes). To accomplish the task, our factories, instead of injecting the logger into the services, wrap the service functions with a self-logging capability (using an in-house logging library) which simply adds another layer of Javascript promise to capture the input parameters and resolve the response objects. This way, all input and return values are available in the in-house logging library for enriching (method name, function start/end time, server ip, microservice name, elapsed duration etc) and logging. We, as the developers, don’t have to worry about it and know that the system will capture everything that is needed in a well-formatted fashion.

Microservice Communication: We created another in-house library, a forked version of “Request Promise Native”. It helps our developers with injecting out of band request-id info so the target microservice can read and use it throughout the lifetime of its underlying services. This means, all our microservices have the capability to read the incoming request-ids and forward it to outgoing microservice calls.

Logger: A word of caution, please mask your messages and don’t log any sensitive data! I’ve seen logs with PII or credit card info in the past, please don’t do it. Your users depend on you and this is your responsibility! Anyways, there are tons of good logging libraries out there. We decided to use Winston because,
1-Winston is good
2-It has Graylog2 support, which brings us to our next item:

Log Repository: In the last 10 years or so, I don’t remember a single case when I had to check the server log files for monitoring/debugging purposes. It is just so impractical to walk through those files with a line of log after the other all coming from different requests. It simply won’t help and actually in one of the US banks that I used to work at, the Devops folks suggested that we could simply stop creating them. Of course, that doesn’t mean you could stop logging. ‘Au contraire!’, it is very important that you have a log repository where you can search, filter, export and manage your logs. So we reduced our options to the following tools:
-Splunk
-Graylog
We selected Graylog because we had experience administrating a Graylog server, it is an open-source tool (meaning much lower costs as it just needs a mid-sized server) and it does the job.

Your logs will show you lots of insights about your application and will potentially help you to uncover bugs. My team regularly walk-through the logs before each release to understand if we are about to introduce any new unexpected errors. With a tool like Graylog, you can create alerts for different scenarios (http response codes, app error codes etc) and this way you will know there is a problem even before the customer sees the error message. Your QA team can insert request-ids in the tickets so the developers can trace what exactly happened at the time of test. If you want to dive deeper, I remember using Splunk logs for fraudulent behavior mitigation through near-real-time and batch analysis. For whatever reason we use the logs, we want them, embrace them, love them:)