Forem: Rutika Khaire

My First Data Governance Project: From Confusion to Clarity (Lessons I Learned the Hard Way)

Rutika Khaire — Mon, 30 Mar 2026 15:52:14 +0000

When I was asked to work on Data Governance for the first time, I had no idea where to start.

Honestly, I was completely blank.

Like most beginners, I did what we all do:
I watched YouTube videos, read blogs, and tried to understand frameworks. Everything sounded important-and complicated.

So I decided to create a Data Governance plan.

With the help of AI tools, I quickly built something that looked impressive-structured, detailed, and “complete.”

But during the review, I faced a simple problem:

👉 I couldn’t explain what I had written.

That moment changed everything.

Lesson 1: If You Don’t Understand It, You Can’t Implement It

Using AI gave me speed-but not understanding.

I realized that Data Governance is not about creating fancy documents. It’s about clarity.

Lesson 2: Don’t Start with Complexity

My initial mistake was trying to build a “perfect” governance model from day one.

After a few reviews, I learned:

Keep things simple
Use clear language
Focus on what actually matters

Lesson 3: Start Small

Instead of complex frameworks, I shifted to:

Simple governance plans
Basic role definitions
Minimal, clear processes

And suddenly, everything became easier to explain—and implement.

Lesson 4: Always Ask ‘Why?’

One question helped me more than anything else:

👉 Why do we need Data Governance?

This helped me focus on:

Data quality
Consistency
Better decision-making
Reducing confusion

If something didn’t answer “why,” I removed it.

Lesson 5: Data Governance is About People, Not Just Documents

While preparing for stakeholder discussions, I realized:

People don’t care about frameworks-they care about impact.

So instead of explaining governance models, I focused on:

What problems it solves
How it helps the business
Why it matters to them

Lesson 6: Tell a Story, Not Just a Plan

When I prepared my demo, I didn’t just present documents.

I explained:

The current challenges
What could go wrong without governance
A simple way forward

That made all the difference.

Final Thought

My first Data Governance experience wasn’t perfect—but it taught me something important:

👉 Start simple. Understand deeply. Then build.

[Boost]

Rutika Khaire — Wed, 24 Dec 2025 06:25:02 +0000

Behind the Scenes of Data Ingestion: How Small Issues Cause Big Headaches

Rutika Khaire ・ Dec 24 '25

#dataingestion #medallion #dataengineering #datafactory

Behind the Scenes of Data Ingestion: How Small Issues Cause Big Headaches

Rutika Khaire — Wed, 24 Dec 2025 06:24:25 +0000

Data ingestion is often treated as a solved problem-until it breaks. What looks like a simple pipeline moving data from source to destination can quietly introduce inconsistencies, missing records, or silent failures that ripple across analytics and reporting systems.

In this blog, we’ll go behind the scenes of a real-world data ingestion architecture, explore common issues that arise, uncover their root causes, and share best practices to build more resilient pipelines.

Why Data Ingestion Matters

Data ingestion is the foundation of every data platform. When ingestion goes wrong:

Reports show incorrect numbers
Business decisions are based on stale or incomplete data
Engineers spend hours firefighting instead of building features

The most dangerous problems aren’t always the obvious failures—they’re the subtle ones that go unnoticed for weeks

Our Ingestion Architecture at a Glance

Azure Data Factory (ADF) for orchestration
SQL Server Views as the source layer
Medallion Architecture
Bronze: Raw ingested data
Silver: Cleaned and transformed data
Gold: Business-ready datasets

Watermark-based incremental loads to process only changed data

This architecture is scalable and efficient—but only when implemented carefully

Issue #1: Missing Deletes – “Why Is This Customer Still Active?”

Real-World Example

A customer account is deleted in the CRM system due to GDPR requirements.
However, the sales dashboard still shows the customer as active weeks later.

Impact

Compliance risk
Incorrect KPIs
Loss of trust from business users

What Actually Happened

Source System        Target Table
Customer Deleted  →  No Delete Captured

Fix (Realistic Approach)
Source (CDC Enabled) │ ├── Insert ├── Update └── Delete ──► Target Table

Enable CDC or soft-delete flags
Add delete-handling logic during MERGE
Periodically reconcile record counts

Issue #2: Race Conditions – Incremental Loads Miss Late-Arriving Updates

Real-World Example
Incremental loads often rely on a watermark column such as LastModifiedDate.

Example:

Last successful watermark = 10:00 AM
A record is updated at 9:55 AM
That update arrives late due to upstream delays

Because the timestamp is older than the watermark, the incremental query skips it entirely.

Why This Is Dangerous

No pipeline failure
No alert
Data is permanently missed unless a full reload happens

This is a silent data loss scenario.

Common Root Causes

Eventual consistency in source systems
Batch updates applied late
Reliance on timestamps without buffering

How to fix it

Use a lookback window (e.g., watermark − 5 or 10 minutes)
Prefer CDC or version-based sequencing
Reprocess recent partitions regularly

Issue #3: Over-Aggressive Filtering – “Where Did My Data Go?”

Real-World Example

A filter is added to exclude test users:

WHERE username NOT LIKE '%test%'

Suddenly, legitimate users like contest_winner disappear from reports.

Hidden Damage

No pipeline failure
No alert
Business notices weeks later

Better Filtering Strategy
Incoming Data │ ├── Valid Users ──► Continue └── Test Users ──► Logged + Reviewed
How to fix it

Use exact match lists (e.g., TestUserList)
Log filtered records
Validate filters with production samples

Best Practices

Resilient Ingestion Design
Detect Change │ Validate Data │ Apply Idempotent Load │ Monitor + Reconcile
Operational Guardrails

Row-count reconciliation
Data freshness checks
Alerting on anomalies—not just failures

Final Thoughts

Most data ingestion failures don’t crash pipelines—they quietly corrupt trust. Missed deletes, late-arriving updates, and poorly managed watermarks often go unnoticed until business users start asking uncomfortable questions.

The takeaway is simple: build ingestion pipelines for real-world behavior, not ideal scenarios. Expect delays, partial failures, and messy data. Validate early, reconcile often, and treat control logic like watermarks as first-class citizens.

In data engineering, it’s rarely the big failures that hurt the most—it’s the small ones you didn’t see coming.

My First Open Source Contribution

Rutika Khaire — Mon, 06 Oct 2025 04:55:05 +0000

For the longest time, I believed contributing to open source was something only “seasoned developers” could do. I pictured huge, complicated codebases and intimidating review processes. But recently, I made my very first open source contribution—and it turned out to be one of the most rewarding learning experiences I’ve ever had.

In this post, I’ll walk you through my journey: how I got started, what I contributed, what I learned, and why you should try it too.

How To Find the Right Project

The first challenge was figuring out where to contribute. GitHub is full of amazing repositories, but it’s easy to feel overwhelmed. I started by looking for beginner-friendly issues using labels like good first issue and help wanted.

Eventually, I came across a project called first-contributions.

Understanding the Workflow

Before writing any code, carefully read the project’s README and CONTRIBUTING.md file. These documents are gold for beginners—they explain how to set up the project, the coding style, and the process for submitting changes.

Making My First Contribution

My first contribution wasn’t something massive—it was improving a README section.

Here’s the process I followed:

Forked the repo
Cloned it to my local machine
Created a new branch for my fix
Made the changes
Committed and pushed the code
Opened a Pull Request (PR)

What I Learned

Here are my biggest takeaways from this first step into open source:

Open source isn’t as scary as it looks—maintainers usually want you to succeed.
Even small contributions make a difference. Don’t underestimate fixing a typo or improving documentation.
Reading project guidelines before jumping in saves a lot of time.
Collaboration and communication matter as much as writing code.

If you’re thinking about contributing to open source, here’s my advice:

Start small. Documentation fixes or simple bugs are great entry points.
Look for “good first issue” labels. They exist specifically to help beginners get started.
Don’t be afraid of mistakes. PR reviews are part of the learning process.

If you’ve made your first contribution, I’d love to hear your story too!

How to Scale Your Application to Handle Peak Loads and Increase Throughput

Rutika Khaire — Wed, 18 Dec 2024 12:22:05 +0000

Introduction: The Party Planner’s Dilemma

Imagine you're hosting a party. You plan for 50 guests, but suddenly 200 people show up. You need more food, more chairs, and a way to keep everyone happy without chaos. This is exactly what systems face during peak loads. How to keep everything running smoothly?

1. Scaling Up: Adding Chairs to the Party

Scaling your system during peak loads is like finding extra chairs when more guests arrive. Scaling ensures that the application has enough resources to handle increased demand.

Here’s how to do it with Azure:
Vertical Scaling (Adding More Power):
Adding resources to existing servers, like upgrading a chair to a sofa.
Example: If your Azure App Service instance needs more resources, scale up to a higher pricing tier.

How to Do It?

Navigate to your App Service.
Select Scale Up (App Service Plan) and choose a higher tier (e.g., Standard to Premium).

Horizontal Scaling (Adding More Machines):
Adding more servers, like borrowing chairs from your neighbors.
Example: During a flash sale, e-commerce platforms add more servers to handle the load.

How to Do It?

Go to your App Service.
Select Scale Out (App Service Plan) and configure autoscaling rules based on metrics like CPU usage or request count.

Azure-Specific Tip: Use Azure Monitor Autoscale to define rules like “Add one instance if CPU usage exceeds 70% for 5 minutes.” Refer this for more detailed information.

2. Thresholds: Knowing Your Party’s Limit

Every system has a breaking point—its threshold. It’s like knowing your living room can only hold 30 people before things get cramped. Thresholds are the breaking points of a system. Knowing them helps avoid disasters.

Identify Thresholds in Azure:
Example: Your SQL Database might have a DTU (Database Transaction Unit) limit of 1000.

How to Do It in Azure:

Use Azure Monitor to track resource metrics like DTU usage, CPU percentage, and memory utilization.
Set up Alerts in Azure Monitor for critical thresholds. For instance, send an email when database utilization exceeds 85%.

Set Alarms:
Example: Monitor your Azure Function’s execution time or failures.

How to Do It in Azure:

Navigate to Azure Monitor → Alerts → New Alert Rule.
Configure conditions like “Trigger an alert if CPU utilization > 80% for 5 minutes.”

3. Handling Thresholds: Avoiding the Crash

What happens when you hit a threshold? Use these strategies with Azure:

Graceful Degradation:
Example: Serve cached product pages if your backend API is overloaded.

How to Do It in Azure:

Use Azure Front Door to cache and serve static content from edge locations.
Configure Azure CDN to offload traffic from your backend during peak times.

Queueing Systems:
Example: Process orders in batches during peak traffic.

How to Do It in Azure:

Use Azure Service Bus or Azure Queue Storage to enqueue requests for asynchronous processing.
Implement a Logic App or Azure Function to process messages from the queue.

4. Throughput: Keeping the Line Moving

Throughput is the speed at which you can serve your guests. In tech terms, it’s how many requests or tasks your system can handle per second.
Here’s how to improve it with Azure:

Optimize Database Performance:
Example: If your Azure SQL Database is slow, optimize queries and scale up.

How to Do It in Azure:

Use the Query Performance Insight feature in Azure SQL to identify slow queries.
Scale your database using the DTU-based or vCore-based model.

Use Load Balancers:
Example: Distribute traffic evenly across multiple virtual machines.

How to Do It in Azure:

Set up Azure Load Balancer for Layer 4 (TCP/UDP) traffic.
Use Application Gateway for Layer 7 (HTTP/HTTPS) traffic, and enable autoscaling.

Implement Caching:
Example: Reduce repeated database calls for product details.

How to Do It in Azure:

Use Azure Cache for Redis to store frequently accessed data like product details or user sessions.

Compress Responses:
Example: Compress API responses to reduce latency.

How to Do It in Azure:

In Azure App Service, enable compression under Configuration → General Settings.

5. Put It All Together: The Ultimate Party Plan

Example Scenario:
You’re running a ticket-booking platform for a concert, and traffic surges at ticket release.

Scaling:

Use Azure App Service autoscaling to handle increased HTTP requests.
Scale your Azure SQL Database to a higher tier (e.g., Standard to Premium).

Thresholds:

Monitor DTU utilization for Azure SQL Database using Azure Monitor.
Set alerts for API response times exceeding 500ms.

Throughput:

Cache event details using Azure Cache for Redis.
Use Azure Front Door to route traffic globally and serve cached content faster.
Implement message queues with Azure Service Bus to process ticket purchases asynchronously.

Conclusion: Keep the Party Going!

Azure provides an array of tools to manage peak loads, thresholds, and throughput efficiently. With features like autoscaling, Azure Monitor, Service Bus, and Front Door, you can ensure your system remains reliable and performant—even under heavy demand.

By implementing these best practices, your applications will stay resilient, agile, and ready to handle any traffic surge.

This blog is designed for novices to scaling or those with intermediate skills who want to strengthen their understanding of handling peak loads, thresholds, and throughput. Using relatable analogies and real-world scenarios, I have tried to break down these concepts into easy-to-grasp sections.

Reference Links
Autoscaling

Boost Productivity and Data Accuracy: Essential Tactics for Software Integration

Rutika Khaire — Thu, 06 Jun 2024 13:49:22 +0000

Understanding Software Integration

Nowadays, integration has become a fundamental aspect of software development. Every organization strives to remain competitive in today's fast-paced world. Integration enables different systems to communicate seamlessly and share information efficiently.

The most effective software integration strategies involve thorough planning, clear communication, and collaborative efforts among teams.

Why do we need Software Integration

Enhanced Workflow Efficiency: Automating the flow of information between systems speeds up business processes and minimizes delays.
Unified Data Management: Integrated systems ensure that data is consistent and up-to-date across all platforms, reducing the risk of errors caused by data discrepancies.
Real-time Data Access: Changes made in one system are instantly reflected across all integrated systems, ensuring all users have access to the latest information.
Automation of Tasks: By automating routine and complex processes, integration frees up employees to focus on higher-value activities.
Reduced Operational Costs: Integration reduces the need for maintaining multiple systems and data silos, lowering IT and operational expenses.

Key Integration Tactics

What it is: Key Integration Tactics are strategies and approaches used to effectively connect different software applications and data sources. They ensure smooth information flow and collaboration between these systems.

Why they matter: Without proper tactics, integration projects can become complex, error-prone, and ultimately fail to deliver the desired benefits.

Different methods of Integration

API Integration: This allows different software systems to communicate with each other via APIs.
Middleware: There are various middleware that can be employed to facilitate communication between systems.
Webhooks: This helps in real-time data exchange.
Microservices: This breaks down applications into smaller, interconnected services.

Planning and Choosing the Right Tactics

Data mapping and standardization
This is one of the crucial tactics to ensure smooth and accurate communication between different software applications. There are several benefits that include data accuracy, efficiency, maintainability and interoperability.

Reconciliation fields
Though reconciliation fields are not a universally defined concept in software integration, but they play a crucial role in ensuring data consistency between integrated systems.

What are reconciliation fields?

Reconciliation fields are specific data points used to compare and identify discrepancies between data sets in integrated systems. They act as a common ground for both systems to verify if the information they hold aligns.

Few advantages of using reconciliation fields include pinpointing the source of the error and taking corrective actions by analyzing discrepancies in such fields. Reliable data from reconciled systems leads to better insights and informed decisions.

Error Handling and Security
Implementation of robust error handling and security measures is another crucial tact to ensure smooth integration. It is a process of anticipating, detecting, and recovering from unexpected issues that may arise during data exchange between integrated systems.

Few techniques to implement it include:

Logging errors: This is one of the very first go-to option in order to tackle issues and understand the root causes behind the problems.
Retry logics: You can implement logics that would re attempt data transfers after some delay.
Alerts: There are different ways to define alert systems to notify respective teams about issues to take necessary and appropriate actions.

Testing and Monitoring
Thorough testing and ongoing monitoring are the most essentials for ensuring the smooth operation and long-term success of your software integration project.

Testing in simple words means ensuring that the integration meets the requirements and the data exchange functions accurately. There are various types of testing like below that really help in ensuring that systems meets all expectations.

Unit testing
Integration testing
Regression testing
Smoke testing
Functional and Non-Functional testing
System testing
Load testing
Negative testing

Monitoring on the other hand involves continuously observing the performance and the health of the integrated system after it is deployed. There are different metrics available that help in tracking the exchange of data and identify failed transactions.

Future Trends in Software Integration
Software integration is constantly evolving, driven by advancements in technology and the growing need for seamless data exchange across an ever-expanding application ecosystem. Here's a glimpse into some key trends shaping the future of software integration:

The Internet of Things (IoT): Integrating with IoT devices is going to become crucial in order to manage and analyze data collected from sensors and intelligent devices.

Artificial intelligence (AI) and Machine Learning (ML): AI and ML can be used to automate tasks within the integration process, like data mapping and error handling. Additionally, AI-powered integration platforms can learn and adapt over time, optimizing performance.

Low-Code/No-Code Integration Tools: There is a rise in tools that empower business users with limited or no coding experience to build basic integrations.

These trends highlight a future where software integration becomes more accessible, automated, and secure.

Looking Ahead
By adopting the key tactics explored in this blog, you can approach software integration projects with confidence. Remember, successful integration goes beyond just connecting systems. It's about establishing a well-defined strategy, utilizing the right tools and methods, and prioritizing ongoing maintenance and monitoring.

Building Real-time Notifications with Azure Web PubSub and Azure Functions

Rutika Khaire — Fri, 01 Dec 2023 04:31:18 +0000

Hello people!!! Today I am writing this blog to share my experience of using the Azure Web Pub-Sub service within Azure Function to send real time messages to my React application.

Many a times, we have a requirement to do some processing asynchronously behind the scenes so that we don't block user's activities on the site.

I came across a similar situation where I was working on a chatbot application. This application has a feature that allows a user to upload a file and then train the file on a specific model and then once the file is trained successfully, the user should be able to query the trained data and get responses from it. Very much similar to what we do in chatgpt.

So the requirement was as below:

User uploads a file in the chat box
Clicks the send icon
File is uploaded to Azure storage
User receives a success message saying - "File uploaded successfully"

Now, while the file is getting trained asynchronously, the user should be able to continue his/her conversation with rest of the available data.

So, then when should the user query for the data that he/she just uploaded as a file??? The answer to this is whenever the file is processed successfully behind the scenes, there should be some indication to the user saying that your file is now trained successfully and is ready to serve requests.

Here comes the concept of a PubSub notification. When the file is processed or trained, the PubSub service will publish a message. That is what Pub stands for. And this published message should be subscribed by some service/resource to receive it which is termed as Sub. So PubSub stands for Publish and Subscribe.

How to Subscribe to a message

You can use WebSocket connection from your frontend application to stay connected to your publisher service so that you can receive real time messages.

In my case, the publisher is Azure function and the subscriber is React application.

Create an Azure Web PubSub service

Go to Azure portal
Type Web PubSub Service in the search bar
Click on the Create button
Select Web PubSub and then add the necessary information and create the service

Below is how the service will look like when created successfully

Go to the Keys section and see that there are connection strings available. The primary connection string is to be used while establishing connection. So keep a note of it.

Azure Function

An Azure function is an event driven, serverless compute where you can write less code to save costs and maintain the infrastructure. Read more here

So, now my azure function is responsible to read the file from Azure storage and then pass the file to an API endpoint to train it.

As, the function is event driven, I have to add two functions in my function app. One is the Blob Storage Trigger which executes whenever a new file is uploaded to the specific storage. And the other is the HTTP Trigger which helps in keeping the Websocket http connection open from the React app to the Azure function.

You have to create a function app in Azure and then add the functions to that function app.

Steps to create the function app:

Go to Azure portal
Type Function App in the search bar
Click Create
Add the necessary and required information to create the function app
After the app is created, go to Configuration and add a setting for the Web PubSub connection string as below. Here you would need the primary connection string that was created when PubSub service was created.

Below is my code structure for Azure function:

The AiTrain function is a blob storage trigger function and Negotiate is the HTTP trigger function. I am using typescript.

Negotiate function

This function is responsible to return the connection object to the subscriber. So, the index.ts file in this function has below code.

module.exports = function (context, req, connection) { context.res = { body: connection }; context.done(); };

And in the function.json file, it has below settings:

{
  "bindings": [
    {
      "authLevel": "anonymous",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": [
        "get",
        "post"
      ]
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    },
    {
      "type": "webPubSubConnection",
      "name": "connection",
      "hub": "notification",
      "direction": "in"
    }
  ],
  "scriptFile": "../dist/Negotiate/index.js"
}

AiTrain function

This function is responsible to train the uploaded file and publish a message after success.

import { AzureFunction, Context } from '@azure/functions';

const blobTrigger: AzureFunction = async function (
    context: Context,
    myBlob: string
): Promise<void> {
    try {
        context.log('TypeScript Blob trigger function');

        const blobURI = decodeURIComponent(context.bindingData.uri);

        // file processing logic


        //Azure web pubsub actions
        const actions = {
            actionName: 'sendToAll',
            data: `File processing complete!`,
            dataType: 'text',
        };

        context.bindings.actions = actions;

        // No asynchronous operations, so we can return a resolved Promise
        return Promise.resolve();
        }
    } catch (error: unknown) {
        console.error(error);
    }
};

export default blobTrigger;

The function.json file has below code:

{
  "bindings": [
    {
      "name": "myBlob",
      "type": "blobTrigger",
      "direction": "in",
      "path": "files/{name}",
      "connection": "dev_STORAGE"
    },
    {
      "type": "webPubSub",
      "name": "actions",
      "hub": "notification",
      "direction": "out"
    }
  ],
  "scriptFile": "../dist/AiTrain/index.js"
}

This completes the first part of publishing a message from Azure function.

Subscribe to published messages

In order to subscribe the messages from the React application, all you need to do is establish a Websocket connection in your code.

Below is my react code that establishes a Websocket connection to the azure function.

useEffect(() => {
        const connect = async () => {
        const res = await fetch(`${API_URI}/api/negotiate`);
        const { url, accessToken } = await res.json();
        const ws = new WebSocket(url);
        ws.onopen = () => console.log('connected');
        ws.onmessage = (event:any) => {
            console.log(JSON.stringify(event.data));
            //Code Logic
        };

        };
        connect();
    }, []);

We have to call the negotiate endpoint to keep the React app connected to the Azure function.

And that's all. You can see the console messages to see the event object that has the data sent from the Azure function. You can choose any component to display this data as a notification to the user.

Conclusion

In conclusion, Azure Web PubSub when coupled with Azure Functions and a React application opens up exciting possibilities for real-time, scalable, and interactive web experiences. We've explored how seamlessly these components integrate to create a dynamic communication channel between the server and client, enabling instant updates and collaborative features.

Keep experimenting, pushing the boundaries, and discovering new ways to leverage Azure Web PubSub. The possibilities are vast.

I hope this blog has inspired you to dive deeper into the realm of real-time communication and discover the endless possibilities that await. Happy coding!

AWS vs. Azure: Comparison

Rutika Khaire — Fri, 29 Sep 2023 14:26:03 +0000

Introduction

Amazon Web Services (AWS) and Microsoft Azure are the two giants when it comes to cloud computing. The most critical business decision is to choose the cloud platform.

Not only cloud providers, but AWS and Azure have come a long way to provide technological services to revolutionize the way organizations operate and innovate.

In this comparison, I will share my understandings of the vast ecosystems they offer, the pricing models that govern their services, the global infrastructure, and the security measures they employ to protect sensitive data.

Ecosystems

Below are the Compute Services offered

AWS comprises of AWS Elastic Beanstalk, Amazon EC2 (Elastic Compute Cloud), AWS Lambda, Amazon ECS (Elastic Container Service), and more.

Azure comprises of Azure App Service, Azure Functions, Azure Kubernetes Service (AKS), Virtual Machines (VMs), and more

Then if we talk about storage then below are the Storage Services offered

AWS offers Amazon S3 (Simple Storage Service), Amazon EBS (Elastic Block Store), Amazon RDS (Relational Database Service), Amazon DynamoDB, and Amazon Redshift

Azure offers Azure Blob Storage, Azure File Storage, Azure Table Storage, Azure Disk Storage, and Azure Data Lake Storage

Let's see the Database Services

AWS offers Amazon RDS, Amazon Aurora, Amazon DynamoDB, Amazon Redshift, and Amazon Neptune

Azure offers Azure SQL Database, Azure Database for PostgreSQL, MySQL, and MariaDB, Azure Cosmos DB, and more

Pricing

Both AWS and Azure provide a free tier that you can use to try your hands on to explore their service offerings. Both also offer pricing calculators that you can use to make cost estimations as per your needs.

There is also pay as you go pricing that charges you based on the amount of time you use the resources. It can be per minute or per hour basis.

If you want to use the resources for a longer duration like for a year or more then both AWS and Azure offer special discounted rates which are termed as Reserved Instances

Global Infrastructure

Azure global infrastructure is made up of two key components—physical infrastructure and connective network components. The physical component is comprised of 200+ physical datacenters, arranged into regions, and linked by one of the largest interconnected networks on the planet.

With the connectivity of the global Azure network, each of the Azure datacenters provides high availability, low latency, scalability, and the latest advancements in cloud infrastructure—all running on the Azure platform.

The AWS Cloud has across 102 Availability Zones within 32 geographic regions around the world, with announced plans for 12 more Availability Zones and 4 more AWS Regions.

Security Measures to protect sensitive data

To manage access to resources, AWS offers IAM roles, users and group and in Azure, you can use Azure Active Directory

If you want to secure configurable data then AWS offers Secrets Manager and Azure provides Azure Key Vault

Conclusion

This is a brief comparison that I have shared based on my experience of working on Azure and AWS. There is still lot out there in Azure and AWS that you can explore by visiting their official documentation.

Thanks for reading!

AI-Powered Coding: A Look at GitHub CoPilot's Capabilities

Rutika Khaire — Thu, 13 Apr 2023 12:05:42 +0000

Let me give you an insight of a powerful tool called the GitHub CoPilot that has revolutionized the way we write code. And congratulations to all the developers as we have an AI powered assistant that will assist us in writing fast and efficient code.

The term "CoPilot" generally refers to a person who assists the primary pilot in controlling an aircraft. Similarly, GitHub CoPilot is the term that conveys the idea of an AI powered assistant for a developer. Like a co-pilot who helps the pilot in controlling an aircraft, GitHub CoPilot helps developers in controlling the code they write.

GitHub CoPilot's Capabilities

Increased Productivity

Isn't the goal of a developer to maximize productivity? Why not utilize a potent tool to enhance your productivity?

GitHub CoPilot provides suggestions and autocompletions thus automating repetitive tasks. While it won't write the entire code according to your exact specifications, it can still be a valuable time-saver, particularly when it comes to catching unintended syntactical errors. By preventing you from spending your productive hours fixing minor mistakes that you may have inadvertently made, it allows you to focus on more meaningful tasks.

Understanding Natural Language

One of the most impressive capabilities that I found is that it is able to understand plain English language and generate code for you.

For example, if you want to write a code to filter non matching data from two JSON objects then you can simply type the text in form of a comment and it will give the code.

//Filter out non matching personIDs from teamMembers and assignmentExists const filteredTeamMembers = teamMembers.filter((e:any)=> !assignmentExists.find((n:any)=>n.dataValues.PersonID === e.dataValues.PersonID));

Helps in reducing errors

It is able to recognize errors and correct those, thus saving lot of time.

Better quality of code

As it is able to give proper code suggestions, you can focus on improving the code quality. In my case, it gave me async await code suggestions which I feel is much better than a promise based code.

Conclusion

So these are few of the capabilities that this tool offers. While there are still a few limitations as well but it is still powerful enough.

You can explore this tool by creating a free account for 2 months after which it will charge you. If you are using Visual Studio Code editor, then simply install the extension and start coding.

For more info visit link.

The easiest way to use HubSpot Search API

Rutika Khaire — Fri, 24 Mar 2023 13:47:24 +0000

Whenever working on integrations, we basically use the APIs to retrieve the information. Some APIs are pretty straightforward to understand, so just by looking at the endpoint we can make out how to use it but sometimes, you need to dig in a bit to understand the usage.

I came across this situation in my application where I wanted to retrieve all the contacts from HubSpot based on a particular property value. For eg. If there is a contact property named Company_Affiliation and I want to retrieve all the contacts with a company affiliation value of say Fictional Company then how should I use the Search API?

The Endpoint

POST
/crm/v3/objects/contacts/search

The Input

const inputHubSpotSearchObject=
    {
        "objectName":"contacts",
        "limitValue":"100",
        "afterValue":"0",
        "filters":[
            {
                "value":req.body.company_name,
                "propertyName": "company_affiliation",
                "operator": "EQ"
            }

        ]
    }

The API call

const response = await axios.post(`${process.env.API_URL}/api/hubspot/search`,inputHubSpotSearchObject,
                          {
                          headers: {
                              "Content-Type": "application/json",
                              Authorization: `Bearer ${accessToken}`,
                          }});

The Response

Conclusion

I hope you found this article helpful. For more information visit link

Happy Coding!!!

Understanding the basics of Express Middleware

Rutika Khaire — Wed, 28 Dec 2022 09:23:24 +0000

In this article, we will understand what Express middleware is and how to use it in a NodeJS application.

As the name in itself is self explanatory, Express works as a middleware in an application. Typically an application is accessed from a client system to retrieve some information from a server. So, here there is a Request that is sent by the client to the server and there is a Response sent by the server back to the requesting client.

So, Express Middleware consists of functions that are executed after a Request is sent by client and before the Response is sent by the server. Something, that we want to execute in the middle of the Request and Response.

We can have multiple middlewares that can also modify the request object and then pass it to the next function in the sequence.

The base of Express is the Application object which is typically used as app. This app object exposes different methods and here we will take a look at the listen, use, HTTP methods (get, put, post etc.).

Create a server with app object

Whenever a website is created, it has a link / URL that we use to access the website. And as the website stays on a server, there is a client that comes into picture that calls the website's URL. So, now as the website is being called, there must be someone who is listening to the calls right? In this case, the server is the listener. And that is the purpose of the listen method of app. There is a port number where these calls from a client are getting listened by the server.

The syntax for creating a server using the app object is as below:

const express = require('express')

const app = express()
const port = 3000

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Now the above code will make the server up and running at the port 3000. When we run the application we can see the message given in the console.log as below.

Now, we need to define some endpoints (path/URI) in the code that will allow the users to call different pages of the website. For, eg. a website can have a Home Page, a Contact Page, an About Page and so on. Ideally, the URL would be something like https://somedomain/Home, https://somedomain/About, https://somedomain/Contact and so on. The part of the URL like /Home, /About, /Contact etc. is what is termed as the endpoint that we should be adding in our code using the app object. Here, the HTTP methods (get,put,post etc.) come into picture. Whenever we try to access the website's page, we get some information that is displayed in the browser. So, for the above mentioned pages, we should use the get method of app.

This is termed as Routing. Below is the definition of Routing from express official documentation.

Routing refers to determining how an application responds to a client request to a particular endpoint, which is a URI (or path) and a specific HTTP request method (GET, POST, and so on).

The basic structure of a route is as following:

app.METHOD(PATH, HANDLER)

Where:

app is an instance of express.
METHOD is an HTTP request method, in lowercase.
PATH is a path on the server.
HANDLER is the function executed when the route is matched.

Let's see how we can use the HTTP methods

Following example shows the get method:

app.get('/Home', (req, res) => {
  res.send('Welcome to the Home Page!')
})

The code snippet would be as below:

It can be accessed in the browser using URL http://localhost:3000/home and we can see the following message:

An important point to note here is that if we try to access the link as http://localhost:3000, we will get an error that says Cannot GET /

What does this error mean? It means that we have not specified the base path which is "/" in our code and we are still trying to access it in the browser.

Let us add the code for base path and see how it works. Following code snippet shows a new get method for base path.

app.get('/', (req, res) => {
    res.send('This is the base call!')
  })

The complete code till now looks like below:

In the browser now if we access the URL as http://localhost:3000/, we see the below output.

The purpose of app.use method

Now, let's understand what is the app.use method used for. From the definition app.use ->

Mounts the specified middleware function or functions at the specified path: the middleware function is executed when the base of the requested path matches path.

So, in a simple language, if we want to execute some logic before the a specific path is matched and the respective handler is executed, we can configure it using the app.use method.

For eg. app.use('/apple', ...) will match “/apple”, “/apple/images”, “/apple/images/news”, and so on. So, if we call "apple/images", it will first execute handler of "/apple" and then the handler of "/apple/images"

If we configure a middleware without a path then it gets executed for every request to the app. For eg. the following function will be executed for every request.

app.use(function (req, res, next) {
  console.log('Time: %d', Date.now())
  next()
})

Middleware functions are executed sequentially, therefore the order of middleware inclusion is important.

The sequential execution of functions is achieved using the next() function. As used in the above code snippet, we are using the next() call to ensure that the next function in the sequence is executed.

Following code illustrates this:

NOTE: You have to call the URL http://localhost:3000/ in the browser to see the output

Let's see what happens if we remove the next() call from the function.

See, the log that we were seeing earlier after the Time was displayed (This is the base call!) does not appear now. This simply means that the control does not pass to the next function in sequence. So, it is important to decide the sequence of executing the functions and add those appropriately.

I hope you have enjoyed this article and hope that it helped you in understanding the basics of Express. For more information you can definitely go through the official express documentation here.

Sequelize ORM with NodeJS

Rutika Khaire — Fri, 30 Sep 2022 16:34:23 +0000

Introduction

Sequelize is a promise based ORM (Object Relational Mapper) for NodeJS. We can use multiple databases with Sequelize like Oracle, Postgres, MySQL, MariaDB, SQLite and SQL Server, and more.

What does ORM actually mean?
An Object Relational Mapper represents the database records as objects. It lets you create and manipulate data from a database using an object oriented paradigm.

So, using Sequelize, you can perform DML operations like SELECT, INSERT, UPDATE, DELETE etc. using class methods. You can also define relationships on your database tables using class methods like hasOne(), belongsTo() and hasMany(), etc.

So, now let's get started

Create a NodeJS application

Create a new folder at your desired location and initialize this as your node.js app using below command

npm init

Keep pressing the enter key after adding the required information and your node.js app is ready.

Now install all the required dependencies using below command

npm install express mysql2 cors sequelize - save

The package.json file will look like below after you have all the dependencies successfully installed.

The next step is to create a new express web server. Add a filename.js file at the root of your folder and add below code.

const cors = require("cors");
const express = require("express");

const app = express();

var corsOptions = {
  origin: "http://localhost:8081"
};

app.use(cors(corsOptions));

// parse requests of content-type - application/json
app.use(express.json());

// parse requests of content-type - application/x-www-form-urlencoded
app.use(express.urlencoded({ extended: true }));

// simple route
app.get("/", (req, res) => {
  res.json({ message: "Welcome to NodeJs App!!!" });
});

// set port, listen for requests
const PORT = process.env.PORT || 8080;
app.listen(PORT, () => {
  console.log(`Server is up and running on port ${PORT}.`);
});

Use the following command to execute the server

node filename.js

You will get the following message

Now, if you go to the browser and type the URL -> [http://localhost:8080/] you can see the application is up and running

Create a database

You can go to MSSQL Server and create a new database. In my case, I created it on Microsoft Azure. The creation of tables can be done with the help of Sequelize.

Next step is to put all the database configurations in a file. So, I have created a config.js file as below.

module.exports = { HOST: "localhost", USER: "root", PASSWORD: "", DB: "student_db", dialect: "mysql", pool: {//pool configuration max: 5,//maximum number of connection in pool min: 0,//minimum number of connection in pool acquire: 30000,//maximum time in ms that pool will try to get connection before throwing error idle: 10000//maximum time in ms, that a connection can be idle before being released } };

Initialize Sequelize

Create a new folder called models in the root directory and add a new file called index.js. Add below code there.

const dbConfig = require(“../config/config.js:);

const Sequelize = require(“Sequelize”);
const Sequelize = new Sequelize(dbCofig.DB, dbConfig.USER,
dbConfig.PASSWORD, {
    host: dbConfig.HOST,
    dialect: dbConfig.dialect,
    operationsAliases: false,
    pool: {
    max: dbConfig.pool.max,
    min: dbConfig.pool.min,
    acquire: dbConfig.pool.acquire,
    idle: dbConfig.pool.idle
    }
};
const db = {};

db.Sequelize = Sequelize;
db.sequelize = sequelize;

db.student= require(“./student.js”) (sequelize, Sequelize);

module.exports = db;
The user should not forget to summon the sync() method in the server.js.
const app = express();
app.use(....);

const db = require(“./models”);
db.sequelize.sync();

When you need to drop the existing tables and the database is required to be resynchronized, enter the force: true code like the below:

db.sequelize.sync({force: true}).then(() => {

console.log(“Drop and resync db.”);

});

We now need to create a new model named student.js

module.exports = (sequelize, Sequelize) => {
    const Student = sequelize.define("student", {
      name: {
        type: Sequelize.STRING
      },
      admission:{
        type:Sequelize.INTEGER
      },
      class: {
        type: Sequelize.INTEGER
      },
      city: {
        type: Sequelize.STRING
      }
    });

    return Student;
  };

Creating Controller

Below is the code for a controller.

const db = require(“../models”);
// models path depends on your structure
const Student= db.student;

exports.create = (req, res) => {
// Validating the request
if (!req.body.title) {
res.status(400).send ({
message: “Content can be placed here!”
});
return;
}

// Creating a Student
const student = {
name: req.body.name,
admission: req.body.admission,
class: req.body.class,
city: req.body.city
};

// Saving the Student in the database
Student .create(student). then(data => {
res.send(data);
}) .catch(err => {
res.status(500).send ({
Message:
err.message || “Some errors will occur when creating a student”
});
});
};

Retrieving Data

You can use below code to retrieve data.

exports.findAll = (req, res) => {

  const name = req.query.name;
  var condition = name ? { name: { [Op.like]: `%${name}%` } } : null;

  Student.findAll({ where: condition })
    .then(data => {
      res.send(data);
    })
    .catch(err => {
      res.status(500).send({
        message:
          err.message || "Some error occurred while retrieving data."
      });
    });

};

Now that we are done adding the controller and model, we need to have a route defined in our application that will execute the controller. Let's go ahead and create a route.

Defining Route

You can create a new folder called routes and add a new route.js file in it. Add below code in that file.

module.exports = app => {
    const students = require("../controllers/student.js");

    var router = require("express").Router();

    // add new student
    router.post("/", students.create);

    // view all students
    router.get("/", students.findAll);
 };

Now include the route in the student_server.js file using below code

require("./routes/routes.js")(app);

You can test the API by calling the routes in postman.

Thank you for reading.