Forem: Sanchit Balchandani

Harnessing AWS Glue for High-Volume JSON Processing 🚀

Sanchit Balchandani — Thu, 25 May 2023 20:29:31 +0000

Introduction:

In today's data-driven world 🌐, managing and interpreting substantial amounts of data has become increasingly vital. We often encounter scenarios where JSON files flood our AWS S3 buckets, demanding efficient processing. In this blog, I will share a recent use case where I successfully used AWS Glue, Python, and Terraform(mainly to handle infra) to tackle the challenge of processing a deluge of small JSON files efficiently.

Background:

Data handling at scale can be an arduous task, especially when dealing with a continuous influx of JSON files. I was asked to process approximately 1 million small JSON files per hour which are landing into an s3 bucket! 🕰️ These files needed to be transformed into larger, more manageable XML files for further analysis by another application. To address this demanding task, I leveraged the power of AWS Glue and other technologies to create a robust and scalable data processing pipeline.

Tech Stack:

Terraform for creating AWS Resources 🏗️
Glue, S3, Systems Manager, Cloud Watch services from AWS 🛠️
Python for writing the Glue Scripts 🐍

Overcoming the Small Files Problem with Glue:

The first hurdle I encountered was the infamous "Small Files Problem" within AWS Glue. This challenge arises when the Spark driver in AWS Glue is overwhelmed with list() method calls to S3, leading to memory exhaustion and, unfortunately, job failures. The first iteration of my solution ran for 6 hours and, you guessed it, failed due to a memory error. 😅

To combat this issue, I devised a primary Glue job written in Python, leveraging PySpark and Boto3. This job merged multiple small JSON files into larger ones based on a configurable parameter, mitigating the Small Files Problem. It also optimized downstream processing by partitioning the merged files into date-based folders within the S3 bucket. I saved the files in the following format:

YYYYMMDD/HHMM

This approach helped me run this partition job multiple times per day, partitioning files based on specific hours. This made the main transformation job, running at midnight, more efficient as it dealt with fewer files.

Transforming and Optimizing Data Processing:

The secondary, or main, Glue job was designed to initialize a Spark session and fetch essential parameters like batch size and date from the AWS Systems Manager (SSM) Parameter Store. This job processed the JSON files in batches using Spark's inherent capabilities. I used the popular Python library xml.etree.ElementTree to convert JSON data into the desired XML format. To optimize storage and write performance, the transformed data was compressed using gzip before being written back to the S3 bucket.

Here is the high-level architecture

The Fruitful Outcome:

The result? A super-efficient pipeline that processed approximately 70k files in less than 2.5 minutes! 🚀 This feat required 10 Data Processing Units (DPUs) in AWS Glue. However, the solution is scalable and can accommodate additional DPUs to meet future requirements. With the current setup, it can easily process 1 million JSON files in around 35 mins which is quite fast and seems scalable as well.

Key Takeaway: The Power of Synergistic Tools and Strategies

The experience of using AWS Glue, Python, and Terraform to handle large amounts of JSON data highlights the significance of using the right combination of tools and strategies to address big data challenges successfully. As data keeps growing rapidly, AWS Glue offers a reliable platform for creating strong data processing pipelines. Python's flexibility allows for intricate data transformations, and Terraform helps optimize infrastructure efficiency. By leveraging these synergistic tools and strategies, you can effectively tackle the complexities of big data processing. 🏗️👏

P.S. - Thanks to generative AI for the edits ;) and making my English look better. The use case is real though :D

Building Strong Bonds with Your Engineering Team(s)

Sanchit Balchandani — Fri, 10 Feb 2023 19:21:20 +0000

While leading a decent-sized team for the last 2 years or so, I have learned the importance of building strong relationships with my team members. In my experience, managing a decently big team requires a lot of effort, patience, and the right approach. In this article, I would like to share some of the strategies I have used that have been effective for me and my team. These approaches are based on my own personal experience and I believe they can work well for others, although every leader has their own unique approach. I prefer staying connected with my team and these strategies have worked well for me. Please feel free to offer criticism and share your feedback(if any).

Respect the Skill

One of the most important things I have learned as a leader is to respect the skill, not particularly the seniority level. It doesn't matter if someone is junior to me, but if they have higher skills, they have my immense respect. This has helped to build trust and credibility with my team members, and they have become more confident in their abilities. When there are such cases, I don't shy to tell them upfront about this.

Hiring for Competence

When hiring for any position, I look for someone who is more skilled than my current team members. If I need to hire someone at my level, I'll make sure to find someone even more skilled than myself. This is because I think it's important to surround ourselves with people who are more knowledgeable and capable than we are. That way, we can learn from them and they can help us grow as leaders. "Skills and experience are important."

Trusting Your Team

One of the most important things a leader can do is to trust their team. When you trust your team members, you give them the confidence to perform at their best. Even if someone has failed once or twice, if you continue to trust them and vouch for them, it can push them to do even better. I've seen this happening in real-time and people do well even if they have some mental support from their leader. In today's rush, we kinda miss these things, but this sort of trust can go a long way in building a strong and motivated team.

Being Empathetic

Being an empathetic leader is crucial to building strong relationships with your team members. Work is not just about getting the job done, it's also about how you are being treated. As a leader, it's important to remain empathetic towards all of your associates and to encourage them to do the same. Not everyone might follow, but if you as a leader set the bar right, it becomes the culture.

Reward Often

Rewarding and recognizing your team members often can give them a lot of motivation. Whether it's a simple thank you or more formal recognition in email, this can help to boost morale and increase job satisfaction. I sometimes feel that I kind of miss out on appreciating people and in fact got this feedback from someone that I missed appreciating him for some good work done. So it's evident that your peers would love even the smallest of appreciation. So, don't hesitate to recognize people and also don't miss out on appreciating your low performers, they need it the most :)

Giving Ownership

When assigning tasks to your team members, it's important to give them a sense of ownership. This means giving them tasks that they can lead and make their own. By doing this, you are giving them the opportunity to showcase their skills and take ownership of their work. I always believe that once you identify the potential in a person and sense even a minimal potential to take ownership, make them feel responsible and give them full ownership of the task/program. At the start, you might need to just oversee but in most cases, they will do beautifully well and you'll feel awesome about their achievements.

Minimizing Interventions

it's important to minimize your interventions when your team members are leading processes. You should just oversee their work to make sure everything is running smoothly, but avoid micromanaging. This gives your team members the space and freedom they need to take their work to the next level. Otherwise, it's more or less the same you are executing on your own terms and your peers are just helping you run the show. So, be all ears to their ideas and let them implement them with some validations.

Succession Plan

As a leader, it's important to make sure you don't slow down your team's progress. This means sharing your responsibilities and making sure someone else knows what you do. This way, you can build a stronger team that trusts each other and works well together.

Quality Over Quantity

When it comes to team development, it's important to focus on quality rather than quantity. A team of let's say, N+ members could be a big achievement, but it's more important to have a team that is high-performing, motivated, and skilled. By focusing on quality building (like asking your team members to develop new skills), you are investing in the long-term success of your team and ensuring that they are equipped to meet new challenges that might come their way.

Open Communication

Finally, being open and transparent with your associates is crucial to building a strong and successful team. Whether it's about salary, the quality of work, or any other concerns, it's important to be open and to set up a transparent culture. This makes things easier for everyone, and it helps to build trust and credibility with your team members. Some folks, might not like it but it is okay you need to think from a perspective of a person who wants very transparent, open and detailed communication because that's how you'll make them feel connected and bring transparency.

In conclusion, managing a large team can be very challenging, but with the right approach, it can also be one of the most wonderful experiences of your career. These approaches make you more human and well-connected with your team, it creates a bond and I strongly feel such teams can achieve much more.

Elevating Your Serverless Development with AWS SAM

Sanchit Balchandani — Sun, 22 Jan 2023 17:34:33 +0000

IAC is essential for modern day development as it allows for more efficient and effective management of infrastructure. It helps to automate the provisioning and deployment of resources, and enables software engineers to manage their infrastructure as code in a version control system. This allows for better collaboration and tracking of changes, making it easier to rollback or update resources as needed.

I've been working on Terraform for a bit now, but never had a chance to explore much about SAM which can easily let you deploy AWS serverless resources using CloudFormation. So I thought of picking up on AWS SAM and explore why it is so good. So in this blog, I tried giving some context about AWS SAM and why it is useful and also sharing one sample use-case which I tried recently.

AWS Serverless Application Model a.k.a AWS SAM is an open-source tool that allows software engineers to define their serverless infrastructure as code, making it easy to manage, test, and deploy their applications. It also comes with a CLI which can be used to deploy & manage serverless applications on AWS.

One of the main benefits of using SAM is that it simplifies the process of creating and deploying serverless applications. With SAM, we can use a simple YAML or JSON template to define the resources that their application needs, such as Lambda functions, API Gateway, and DynamoDB tables. SAM then takes care of creating and deploying these resources to AWS, making the process much easier and more efficient than doing it manually.

Another benefit of using SAM is that it provides a consistent development experience across different languages and frameworks. SAM supports multiple languages, including Python, Node.js, and Go, and it can be used with different frameworks, such as Flask, Express, and Django. This means that we can use the same template and CLI commands to deploy the applications, regardless of the language.

To get started with SAM, we need to install the SAM CLI on the local machine. The SAM CLI can be installed using the pip package manager by running the following command:

pip install aws-sam-cli

Once the SAM CLI is installed, we can use it to create a new application template by running the following command:

sam init --runtime <runtime>

Where is the language and version that the application will be written in, such as nodejs12.x or python3.8. This command will create a new directory with a basic template that includes a Lambda function and an API Gateway.

To deploy the application to AWS, we can use the sam deploy command. This command will create or update the resources defined in the template, and it will also create an S3 bucket to store the application's artifacts. To deploy the application, we need to run the following command:

sam deploy --stack-name <stack-name> --capabilities CAPABILITY_IAM

Where <stack-name> is the name of the CloudFormation stack that will be created. The --capabilities CAPABILITY_IAM flag is required to create IAM resources, such as roles and policies, that are defined in the template.

Let's do a small demo now!

Below is an example of using SAM to deploy a simple Python application on AWS. The application consists of a Lambda function that handles a GET request to an API Gateway endpoint, which retrieves data from a DynamoDB table. Above commands while explaining SAM were just for reference, to build the demo please start using commands from here on -

First, we will create a new directory for our application and initialize it with the SAM CLI:

sam init --runtime python3.8

After running above command, it will ask you some questions, please choose 1 to select AWS Quick Start Templates and then subsequently choose 1 to select Hello World Example and when you are asked for app name, type it as demo-app-sam.

This will create a new directory with the basic template files, including template.yaml and hello_world directory.

Next, we will create a new directory demo and create a new file called app.py this file will contain our Lambda function code:

mkdir demo
touch demo/app.py

Then in our app.py file, we will import the necessary modules like json, including boto3 to interact with DynamoDB, and write a function named handler that will handle the GET request to the API Gateway endpoint, and retrieve data from the DynamoDB table:

import json
import boto3

def handler(event, context):
    dynamodb = boto3.resource("dynamodb")
    table = dynamodb.Table("demo-table")

    response = table.scan()
    return {
        "statusCode": 200,
        "body": json.dumps(response["Items"]),
        "headers": {
            'Content-Type': 'application/json',
        }
    }

Now we will update our template.yaml to include the resources needed for our application.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  demo-app-sam

  Sample SAM Template for demo-app-sam

Globals:
  Function:
    Timeout: 3
    MemorySize: 128
    Tracing: Active
  Api:
    TracingEnabled: true

Resources:
  DemoFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: demo/
      Handler: app.handler
      Runtime: python3.8
      Policies:
      - DynamoDBCrudPolicy:
          TableName: demo-table
      Architectures:
      - x86_64
      Events:
        DemoApi:
          Type: Api
          Properties:
            Path: /
            Method: get
  DemoTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: demo-table
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      ProvisionedThroughput:
        ReadCapacityUnits: 1
        WriteCapacityUnits: 1

This template defines a Lambda function named DemoFunction that is triggered by a GET request to the /endpoint of an API Gateway. It also defines a DynamoDB table named demo_table with a single id field.

Finally, to deploy our application, we will run the following command:

sam deploy --stack-name demo-app-sam --capabilities CAPABILITY_IAM --guided

This will create or update the resources defined in the template, including the Lambda function, API Gateway, and DynamoDB table. Once the deployment is complete, we can test our application by sending a GET request to the API Gateway endpoint, which should return data from the DynamoDB table.

In this example, we have demonstrated how to use SAM to deploy a Python application on AWS, which consist of a Lambda function that handles a GET request to an API Gateway endpoint, retrieves data from a DynamoDB table, and returns it to the client.

Now to summarize, AWS SAM is a powerful tool for building and deploying serverless applications on AWS. It provides a simple and consistent way to define serverless infrastructure as code, making it easy to manage, test, and deploy applications. The SAM CLI also provides a convenient way to create and deploy applications to AWS, making the process much more efficient and streamlined. With the help of examples and code, we can easily understand the process and start building their own serverless applications on AWS.

Feel free to reach out to me if you have any questions, I'd be happy to chat.

P.S. - Once you are done running and testing your demo, don't forget to delete the created resources from AWS ;)

Use below command to do that -

sam delete --stack-name demo-app-sam

How to migrate Python 2 projects to 3 & some useful tips

Sanchit Balchandani — Tue, 12 Jan 2021 12:08:22 +0000

Python 2.7 came to end of life in January 2020 and even before that lot of software migrations to Python 3 have been happening. However, I was a little late to the party and got to work on one such project in late 2020. So in this blog, I'll try and summarize what steps we followed and some of the tips which can be helpful for a fellow developer who may need to migrate a Python project in future.

Before we get started, let me also add a small disclaimer, that I'll not be touching upon the differences of Python 2 & 3, there is a lot of content available on the internet for the same so please feel free to check it out.

We'll be talking about the migration where we were not worried about Python2 compatibility because that was the case with us. However, I'll share some reference links for py2&3 compatible migrations as well. So let's get started and see what are the steps one can follow when migration to Python 3 is to be done. We migrated from Python 2.7 to 3.7 but the latest version is 3.9 so if you are confident enough to choose the latest Python version, you can do that as well.

Steps for migration

The first step would be to understand some fundamental difference between Python 2 & 3, that will help fix issues
Check the code coverage of the code and make sure that we have decent test coverage. The more the code coverage, the easier would be to migrate the code
Set up a virtual environment for both Python2.7 and Python 3.7+ versions

Virtual Env for Python2

python2.7 -m  pip install virtualenv
python2.7 -m virtualenv py27
source py27/bin/activate

Virtual Env for Python3

python3.7 -m virtualenv py3
source py3/bin/activate

Also, do create a separate branch for Python 3 codebase and maintain these two versions parallelly unless the full migration is completed
The next steps shall be to figure out the dependencies of the codebase compatible with Python 3, for that I'd recommend using the following tool
- canisepython3: This helps to find out which existing dependencies are compatible with python3
```
pip install caniusepython3
```
- pip-tools: When we are migrating and we may also worried about pinning the dependencies & sub-deps then pip-tools is very handy, do explore it
```
pip install pip-tools
```
Once dependencies are sorted then run the test cases and find out the number of failures or any major issues, keep a note of test case failures, that will help you estimate the work up to some level
After this, we need to move to the conversion of the code, so we already have some auto conversions tools available like 2to3
In our case we used a library called futurize which is also based on 2to3
```
pip install future
```
We can use below command to auto-convert the code to Python 3
```
futurize --stage1 mypackage/**/*.py
```
Once the code is auto-converted, run the test cases and check the number of failures and start fixing them
Once the test cases are fixed, we can proceed with our functional testing, regression testing and CI pipeline upgrades

Note - While fixing the functional test cases, it will be handy to have both Python 2 & Python 3 version of apps running in two diff virtual envs, because then we can go back n forth between two versions and see the behaviour change if we are stuck with fixing a bug

Few Gotchas

As it's mentioned in futurize library's documentation that its auto conversion output may not be fully accurate and it has some issues and bugs as well, so it is important for us to review the automated fixes before merging the code, so from our experience few things can be taken care:

Auto-conversion will end up adding list inbuilt function to all such cases where we might be using dict.iteritems() , after the conversion it becomes list(dict.items()) because iteritems returns a list in Python 2.7 and in Python 3 it returns <class 'dict_items'> . However, It is an unnecessary call because conversion to list takes time and you can still iterate through dict.items() the same way you do over the converted list
Futurize can also add a lot of unused imports to your codebase, we can get rid of them if we are not worried about backward compatibility, but be mindful of not removing all unused import and do check and analyze the module before doing it
If you might have used zip function in Python 2.7 and then tried converting it to dict then futurize might wrap zip with a list function call, you won't need that list() function call in most cases after migrating to Python 3

all_vals = dict(list(zip(array1, array2))) # Unnecessary list function call
all_vals = dict(zip(array1, array2)) # Correct

Relative imports are no longer supported in Python 3. Any code that used relative imports in Python 2 now has to be modified to support absolute imports
In Python2, CSV files should be opened in binary mode. but, in Python 3, the files are to be opened in the text mode

Common Error Fixes

So during the migration, we may end up fixing some very basic level errors which may get introduced because of futurize or because of the Python 2 & 3 differences. So I'm listing few of them which we faced and that might help you to quickly get rid of those -

The old division would not work in Python3, and we can fix such cases like this

from past.utils import old_div
x = y / 1000 # Would give unexpected result in Python 3 and existing test case may fail because of this
x = old_div(y, 1000) # This would result as Python 2

In all exception handling cases, we might end up getting this common error, wherein Python 2 we had an attribute called message for the exception class and python 3 doesn't have it

# Old case
except Exception as e:
    raise CustomError(message="message: {}".format(e.message))

# How to fix
except Exception as e:
    raise CustomError(message="message: {}".format(e))

For the same exception handling errors when we remove message attribute from the e variable than in some case we might face below error where we will do concat operation on e and some string, that will cause a failure

# Failure case
try:
    1/0
except Exception as e:
    print("custom_message"+e) # This would give an error
# How to fix
try:
    1/0
except Exception as e:
    print("custom_message"+str(e))

hasattr(dict, 'iteritems') would return False in Python 3, to fix please use this

hasattr(dict,'items’)

In Python 2 we used to use string.replace() as well in somecases to replace string within a string, that will not work in Python 3, hence fix it like this -

a = "asdfghj"
string.replace(a, "asd", "xcv") # Will not work
a.replace("asd", "xcv") # correct

cmp() function which used to be available in Python 2 is not available in Python 3, so you might need to implement something like this to fix the errors -

def cmp(a, b):
    return (a > b) - (a < b)

Conclusion

I hope this information was helpful for you and I also hope that this may expedite your migration time. Overall, I must also confess that even though migrations steps seems kind of pretty straight forward but the whole process can be tiring because of fixing some common errors again and again. If you are able to fix these small errors quickly then you can focus on the functional part of the migration which is most important. So which is why I tried to list some of the common issues I faced during the migration, hope it can be useful for someone. Thanks for reading and feel free to share any feedback in the comment section.

References:

https://wiki.openstack.org/wiki/Python3#Port_Python_2_code_to_Python_3
http://python3porting.com/
https://docs.python.org/dev/howto/pyporting.html
https://python-future.org/quickstart.html
https://python-future.org/quickstart.html#to-convert-existing-python-2-code (On python 2 compatibility)

Disclaimer: This is a personal [blog, post, statement, opinion]. The views and opinions expressed here are only those of the author and do not represent those of any organization or any individual with whom the author may be associated, professionally or personally.

Why do Software Developers need to be More Empathetic?

Sanchit Balchandani — Thu, 24 Dec 2020 00:09:39 +0000

We all know about "Empathy" and especially during these unprecedented times, we might have heard this term even more often. However, do we really practice and understand the real importance of Empathy in our life?

In my little experience and the number of people I've met during my career, most of us understand the importance of it but very few would practice it in real life because sometimes even if you know what's right or wrong, you may follow your subconscious mind and end up being unempathetic in that moment.

In this blog, I'll try and convince you why do software developers and in general, we need to be more thoughtful about being Empathetic.

Also, before we move forward let me clarify a quick point - "Sympathy is not Empathy!"

Sympathy is more about feeling sorry for someone. However, Empathy is about actually understanding someone else's pain/suffering. In case of being sympathetic, you'd probably feel for that person at that moment and move on, but empathy doesn’t work the same way. it opens the door to a range of long-lasting emotions and feelings.

Now let's come back to the importance of Empathy for a Software Developer.

Software Developer is a generic term or title and I assume that software developers play different roles in their organizations - Business Analysis, Coding, Testing, Leadership, DevOps, Maintenance etc. Each of these roles would involve talking to people and wherever you need to deal with people, empathy can play a key role. It can help you create better client relations, create better products, retain employees & help you build a great team. We can go into the depth of each of these roles and probably try to understand the importance but let's not go the orthodox way and instead, I'll add few pointers from my experience which may be limited to certain roles but in general applicable to most of the cases.

Working with a fellow associate - A lot of time when you go up the ladder or let's say if you are super skilful, and you are working with someone who may not be equally skilful; that shouldn't allow you think or develop some subconscious bias about that person - e.g. He/she is not good or he/she is taking too long to finish the task or he/she is giving me excuses. Because if you do that, your subconscious mind may not let you experience that person's actual state or condition. That's where Empathy plays a very important role. If you are a fellow developer or a lead and if you practice being empathetic, you can create a better work environment for that person and also help him/her grow eventually. Note - arrogance & rudeness are disproportional of Empathy.

While taking an Interview - When you take an interview, Empathy also can play a great role for the interviewer to identify the candidate's potential. With this, I don't mean that one shall become empathetic and hire a candidate which doesn't fulfil the criteria. However during the interview, If you can be kind enough, give decent time to the candidate even if he/she is not doing well can do wonders. It is generally the interviewers' responsibility to get the best out of the candidate and it will only happen when you'll show empathy towards the candidate. Sometimes if you find candidate very nervous, you may have to play dumb and that eventually can make him/her feel comfortable. If you are able to achieve that by being little empathetic then it certainly creates a great experience for both interviewer and the candidate. And who knows you'll end up hiring a great team player :D

During the customer Interactions - This is more of an obvious one because customer interactions are generally tricky and can be complicated at times for software developers. We(software devs) tend to think more from a software endpoint but if the customer is non-technical then it can be a nightmare as well. I've seen a lot of back n forths happening because of small communication gaps, it can be frustrating for both sides. The simple solution is when you go to these discussions try and be empathetic, this might enable you to think from a customer standpoint and when gradually both parties start to think in same directions, there are always possibilities of lesser conflicts and unneeded iterations.

If you are a leader - Empathy is a must-have quality in the modern-day industry leaders and many companies are focusing on these qualities as well while hiring them. If you are a leader & Empathetic -

You will be able to better understand the needs of those people who you work & deal with
You will experience the world in a bigger perspective as you perceive through not only your perspective but of those around you as well
Also, You will learn how to motivate the people around you and help them grow
You will be able to more effectively convince others of your point of view
Moreover, you will be a better leader, and most importantly, a better friend.

How to practice it? - If you are reading till this point, I hope it is evident and clear that empathy is not only a feeling but a very much needed skill in today's software developer's profile :) and any skill can be learned and practised.

So being mindful about your daily interactions, be it with anyone in your office or personal life can help you learn and practice this. I generally prefer to re-think some of my conversations and If I feel I could have been better then I try not to repeat.

Conclusion - I hope that I was able to convince you that Empathy is critically very important. Considering that title of the blog is about the importance of empathy for a software developer, so I'd like to conclude mentioning that companies are trying to build practices, engineering excellence cultures, adopting a lot of software best practice to grow exponentially but not greatly focusing making Empathy a must-have skill. Also, these days leaders are trying to adopt the "trust culture" within engineering teams. However, I strongly feel trust is the output of Empathy :)

So looking at all the points I mentioned in the blog above, I feel times have changed and Empathy should be given a lot of value as a skillset and there shall be efforts to train engineers about it.

With that, I'd like to end this blog here, hope you liked it. Do reach out to me if you have any questions or suggestions or inputs.

I'm available at Twitter and Linkedin.