Forem: Siddharth Bhamare

End-to-End CI/CD Setup for .NET Microservices on AWS EKS Using Azure DevOps

Siddharth Bhamare — Fri, 31 Oct 2025 12:27:45 +0000

Create the ecosystem connections between Azure DevOps, AWS, and your repo first, and understand how every piece (Dockerfile, YAML, Kubernetes manifests, etc.) in terms of cloud infrastructure. Understand CI/CD, Cloud compute service EKS, and API Gateway for your microservice (NGINX)

Overall Cloud Architecture (Visual Overview)

🧭 PART 1 — The Big Picture Flow (Visual Overview)

Here’s the end-to-end relationship of files, pipelines, and services:

                ┌──────────────────────────────────────────────────┐
                │                 Azure DevOps                     │
                │──────────────────────────────────────────────────│
                │                                                  │
                │ 1️⃣ Build Pipeline (azure-pipelines.yml)          │
Developer Push  │    ↓                                             │
     Code  ────▶│  (uses Dockerfile + dotnet test + code coverage) │
                │    ↓                                             │
                │  Builds Docker image & pushes to AWS ECR         │
                │    ↓                                             │
                │  Publishes artifacts (manifests/templates)       │
                │                                                  │
                │ 2️⃣ Release Pipeline / Deploy Stage               │
                │    ↓                                             │
                │  (takes templates → replaces variables → applies)│
                │    ↓                                             │
                │  kubectl apply to AWS EKS (Kubernetes cluster)   │
                └──────────────────────────────────────────────────┘
                                     │
                                     │  Deploys Container Image
                                     ▼
                 ┌───────────────────────────────────────────┐
                 │                AWS EKS Cluster             │
                 │───────────────────────────────────────────│
                 │ Deployment.yaml → creates Pods             │
                 │ Service.yaml → creates ClusterIP/LoadBalancer │
                 │ ConfigMap/Secrets → inject env vars         │
                 │ Ingress.yaml (NGINX) → exposes API          │
                 └───────────────────────────────────────────┘
                                     │
                                     ▼
                        🌐 Accessible API via Ingress
                          Logs → Application Insights

🧩 PART 2 — How Files Connect (Step-by-Step Flow)

Let’s link each file and configuration together in order:

Step	File / Component	Purpose	Consumes / Produces
1	Dockerfile	Defines how to build your .NET app into a container.	Consumed by build pipeline (`docker build`)
2	azure-pipelines.yml	Your CI/CD automation definition. It: - Builds app - Runs tests + coverage - Builds Docker image - Pushes image to AWS ECR - Deploys manifests to AWS EKS	Consumes: Dockerfile, test project, manifests. Produces: Docker image in ECR, deployment in EKS
3	configmap.template.yaml / secrets.template.yaml	Define environment variables (non-sensitive in ConfigMap, sensitive in Secret).	Consumed by Deploy stage. Gets substituted and applied to Kubernetes.
4	deployment.yaml.template	Defines how many Pods to run, which image to use, and what ConfigMap/Secret to pull.	Consumed by Deploy stage. Gets image tag replaced and applied to EKS.
5	service.yaml	Exposes your app internally (ClusterIP / LoadBalancer).	Linked to Deployment via label selectors.
6	ingress.yaml	Exposes app to the public through NGINX Ingress Controller.	Linked to Service. Controls routing and domain mapping.
7	App Insights config (via environment variable)	Captures logs from the app and sends to Azure.	Linked via Secret/ConfigMap and Serilog in code.

⚙️ PART 3 — What to Set Up First (Foundation Setup Order)

Here’s the correct first-time setup order for Azure DevOps ↔ AWS ↔ Repo.

🧩 Step 1: Setup AWS Resources

Resource	Action	Why
ECR (Elastic Container Registry)	Create a repo (e.g., `myservice`).	Stores your Docker images built by Azure DevOps.
EKS (Elastic Kubernetes Service)	Create a cluster (via `eksctl` or AWS Console).	This will host your deployed containers.
IAM User/Role for Azure DevOps	Create an IAM user with permissions for ECR (push/pull) and EKS (read/write). Generate Access Key + Secret Key.	Used by Azure DevOps to authenticate with AWS.

✅ Tip: Attach AWS managed policies:

AmazonECRFullAccess
AmazonEKSClusterPolicy
AmazonEKSWorkerNodePolicy
AmazonEKS_CNI_Policy
AmazonEC2ContainerRegistryPowerUser

🧩 Step 2: Setup Azure DevOps Project

Go to dev.azure.com → create Project (e.g., MyServiceProject).
Create a Service Connection for AWS:

Project Settings → Service connections → New → AWS.
Enter Access Key, Secret Key from Step 1, Region.
Name it e.g., AWS-Prod-Connection.
1. Create Pipeline:
Go to Pipelines → New → Import azure-pipelines.yml from your repo root.
1. Create Environment Variables / Pipeline Variables:
APP_ENV, DB_CONN, FEATURE_FLAG_X, etc.
1. Create Variable Groups (optional) for environment-specific settings.

🧩 Step 3: Setup Repo and Link to Azure DevOps

Push your repo (code + pipeline + k8s templates) to Azure DevOps Repos or GitHub.
Ensure your pipeline YAML file is at repo root.
In Azure DevOps → Pipeline → Create pipeline → Choose “Existing Azure Pipelines YAML file”.

🧩 Step 4: Setup Kubernetes Connectivity

Inside Azure DevOps pipeline, we run commands like:

aws eks update-kubeconfig --region <region> --name <EKS_CLUSTER_NAME>

🔑 This works only if your AWS Service Connection user has EKS cluster permissions.

You can test connectivity manually on your local machine:

aws eks update-kubeconfig --region <region> --name <EKS_CLUSTER_NAME>
kubectl get nodes

If it works locally, it’ll work inside the pipeline.

🧩 Step 5: Setup NGINX Ingress Controller

Once the EKS cluster is ready:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace

It’ll deploy the NGINX ingress controller.
Then your ingress.yaml connects external traffic to your app’s service.yaml.

🧩 Step 6: Setup Application Insights (Azure Portal)

In Azure Portal → Create a Log Analytics workspace + Application Insights.
Copy its Connection String.
Add that value as a Kubernetes Secret or pipeline variable (to inject during deployment):

   APPINSIGHTS_CONNECTIONSTRING="InstrumentationKey=..."

Your .NET app automatically starts sending logs once deployed.

🧩 Step 7: Run the Full Flow

✅ Push code to main → Build pipeline triggers →
✅ Docker image built + pushed to ECR →
✅ Deploy stage runs →
✅ kubectl apply updates EKS →
✅ NGINX Ingress exposes service →
✅ Application logs flow to AppInsights.

🔁 Understanding Template Variables

Let’s clarify what to replace in your .template files:

Placeholder	Replace With	Comes From
`__IMAGE__`	`<AWS_ACCOUNT_ID>.dkr.ecr.<region>.amazonaws.com/myservice:<Build.BuildId>`	Build pipeline
`__APP_ENV__`	`Production` / `Staging`	Pipeline variable
`__FEATURE_FLAG_X__`	`true/false`	Pipeline variable
`__DB_CONN__`	PostgreSQL connection string (from AWS RDS / Secrets Manager)	Pipeline variable / Secret
`__APPINSIGHTS_CONNECTIONSTRING__`	Value from Azure Application Insights	Secret
`__EKS_CLUSTER_NAME__`	Your EKS cluster name	AWS Console

🧩 BONUS — Suggested First-Time Order of Execution

Step	Action	Where
1	Create AWS ECR repo	AWS Console
2	Create AWS EKS cluster	AWS Console / eksctl
3	Setup IAM User with ECR/EKS permissions	AWS IAM
4	Setup Azure DevOps Project + AWS Service Connection	Azure DevOps
5	Push repo (code + Dockerfile + pipeline YAML + k8s templates)	Git
6	Create pipeline (YAML) and run once manually	Azure DevOps
7	Verify ECR image pushed	AWS Console
8	Verify EKS deployment created via `kubectl get pods`	Local or DevOps log
9	Install NGINX ingress, create ingress.yaml	AWS EKS
10	Add Application Insights connection string & verify logs	Azure Portal

Day 4 - 🔐 Securing API with Keycloak

Siddharth Bhamare — Fri, 05 Sep 2025 13:23:02 +0000

If you already have a .NET API running, you can integrate Keycloak to provide authentication and role-based authorization without rewriting your backend. This guide covers setup, token handling, role-based policies, and common questions.

What is Keycloak?

Keycloak is an open-source identity and access management (IAM) solution. It helps developers secure applications and services with minimal effort. Keycloak provides:

Single Sign-On (SSO): Users log in once to access multiple apps.
User Management: Create, update, and delete users via admin console or API.
Role-Based Access Control (RBAC): Assign roles to users to restrict access to specific resources.
Support for Multiple Protocols: OpenID Connect, OAuth2, SAML.
Integration with Multiple Platforms: Works with Java, .NET, Node.js, and more.
Keycloak removes the need to implement authentication and authorization logic in your application, making it ideal for both development and production environments.

Why Keycloak?

Even if you are using IdentityServer (Duende), Keycloak provides:

Ready-to-use IAM server with UI and SSO.
Role management and RBAC to secure endpoints easily.
Open-source and production-ready with clustering and Kubernetes support.
Supports multiple protocols for future integrations (mobile, frontend, microservices).

When to choose Keycloak:

Centralized authentication across multiple services.
Admin UI to manage users, roles, and clients.
Plan to implement SSO for frontend apps or mobile apps.

Pull Request Example

Git Hub Pull Requests demonstrates:
JWT authentication in an existing API
Role-based authorization (admin and vendor)

- Minimal changes to existing controllers

Setting Up Keycloak (Docker)
The easiest way to try Keycloak locally is via Docker.
Official Guide: Keycloak Docker Getting Started

docker run -d \
  --name keycloak \
  -p 8080:8080 \
  -e KEYCLOAK_ADMIN=admin \
  -e KEYCLOAK_ADMIN_PASSWORD=admin \
  quay.io/keycloak/keycloak:25.0.6 start-dev

Access the admin console: http://localhost:8080

Username: admin
Password: admin

Keycloak Configuration

Create a Realm: demo-realm
Create a Client for Your API:
Client ID: demo-api

Protocol: OpenID Connect
Access Type: confidential
Copy the Client Secret

Create Roles:

admin
vendor

Create Users and Assign Roles:

Example: admin-user → assign admin
Example: vendor-user → assign vendor

Securing Your Existing .NET API
Assuming your API is already created, you only need to add authentication and authorization.

Microsoft.AspNetCore.Authentication.JwtBearer

Update Program.cs

var builder = WebApplication.CreateBuilder(args);

// Authentication
builder.Services.AddAuthentication("Bearer")
    .AddJwtBearer("Bearer", options =>
    {
        options.Authority = "http://localhost:8080/realms/demo-realm";
        options.TokenValidationParameters = new Microsoft.IdentityModel.Tokens.TokenValidationParameters
        {
            ValidateAudience = true,
            ValidAudience = "demo-api"
        };
    });

// Authorization policies
builder.Services.AddAuthorization(options =>
{
    options.AddPolicy("AdminOnly", policy => policy.RequireRole("admin"));
    options.AddPolicy("VendorOnly", policy => policy.RequireRole("vendor"));
    options.AddPolicy("AdminOrVendor", policy => policy.RequireRole("admin", "vendor"));
});

var app = builder.Build();

app.UseAuthentication();
app.UseAuthorization();
app.MapControllers(); // Existing API controllers
app.Run();

Applying Role-Based Access in Controllers

[Authorize(Roles = "admin")]
[Route("api/[controller]")]
public class AdminController : ControllerBase
{
    [HttpGet("dashboard")]
    public IActionResult GetAdminData() => Ok("Hello Admin 👑");
}

[Authorize(Roles = "vendor")]
[Route("api/[controller]")]
public class VendorController : ControllerBase
{
    [HttpGet("dashboard")]
    public IActionResult GetVendorData() => Ok("Hello Vendor 🛍️");
}

Getting an Access Token
Using cURL [use powershell] - Update your url, username, password and client id

curl.exe --insecure -X POST "http://localhost:8080/realms/demo-realm/protocol/openid-connect/token" `
>> -H "Content-Type: application/x-www-form-urlencoded" `
>> -d "grant_type=password&client_id=demo-api&client_secret=<client-secret>&username=yourusername&password=yourpassword"

Authority vs ValidIssuer

Authority: Base URL of Keycloak (http://localhost:8080/realms/demo-realm). Middleware fetches metadata automatically.
ValidIssuer: Exact iss claim from token. Usually not required unless running behind reverse proxies.

Best Practice: Use Authority + ValidAudience. Only override ValidIssuer if needed.

Common Doubts/Questions

- Why Keycloak over IdentityServer?
Keycloak is ready-to-use with UI, SSO, and role management; IdentityServer requires more custom coding.

- Can multiple APIs share one realm?
Yes, each API can have its own client in Keycloak.

- How are roles assigned?
Realm roles = global, Client roles = API-specific, users can have multiple roles.

- Is Keycloak production-ready?
Yes, supports clustering, load-balancing, and Kubernetes deployment.

- Can I do client-to-client authentication?
Yes, using the client credentials grant type.

Resources:

Keycloak Docker Guide: https://www.keycloak.org/getting-started/getting-started-docker

GitHub PR: SmartKart-Catalog #4 https://github.com/SiddharthBhamare/SmartKart-Catalog/pull/4

🚀 Day 3: Architectures Explained with My Implementation

Siddharth Bhamare — Sun, 24 Aug 2025 15:23:59 +0000

(DDD + Clean + Onion + Microservices)

Many times in interviews, questions come like:
👉 “What’s the difference between DDD, Clean, and Onion Architecture?”
👉 “How do you apply them in a real project?”
👉 “Why Microservices over Monolith?”

Today, I’ll simplify these with my Catalog Service example (Products, Categories, Brands).

1️⃣ Domain-Driven Design (DDD)

What it is:

Focuses on the Domain Model (business rules, not just CRUD).
Uses Entities, Value Objects, Repositories, Aggregates, Aggregate Roots.
Communication happens via Ubiquitous Language (same words as business).

Layers in DDD:

Domain Layer (Inner) → Entities (Product, Category), Interfaces (IProductRepository).
Application Layer (Middle) → Services (ProductService orchestrating use cases).
Infrastructure Layer (Outer) → EF Core, DB, external APIs.

My Example:

Product is an Aggregate Root.
IProductRepository ensures access through the domain contract.
EfRepository<Product> (Infra) implements IProductRepository.
Application Layer (ProductService) coordinates actions like “Add Product to Category.”

👉 Interview Tip: DDD ensures business-first design, avoids anemic models.

2️⃣ Clean Architecture

What it is:

Robert C. Martin (Uncle Bob) → emphasis on independence of frameworks, UI, DB.
Dependency Rule: All dependencies point inward, towards the Domain.

Layers (from inner to outer):

Entities (Domain) → Business rules.
Use Cases (Application) → Application-specific business logic.
Interface Adapters → Controllers, Presenters, DTOs.
Frameworks & Drivers (Outer) → EF Core, ASP.NET Core, DB, external systems.

My Example:

Entities: Product, Category, Brand.
Use Cases: ProductService, CategoryService.
Interface Adapters: ASP.NET Core Controllers (ProductController).
Frameworks/Drivers: CatalogDbContext, EF Core repositories.

👉 Interview Tip: Clean Architecture is technology-agnostic → I can switch from EF Core to Dapper without touching Domain/Application.

3️⃣ Onion Architecture

What it is:

Similar to Clean, but visually represented as onion rings.
Inner Core = Domain, outer layers depend on inner.

Layers (from core out):

Core (Domain) → Entities (Product, Category), Interfaces (IRepository<T>).
Application Layer → Services (ProductService).
Infrastructure Layer → Repositories (EfRepository<Product>).
Presentation Layer → API Controllers.

My Example:

Inner Onion Ring = Product + IProductRepository.
Middle = ProductService.
Outer = EfRepository<Product> + ProductController.

👉 Interview Tip: Onion vs Clean = philosophy is same (dependency direction). Onion is more visual metaphor, Clean is more principle-driven.

4️⃣ Microservices

What it is:

Break monolith into independent services around business capabilities.
Each service has its own DB + bounded context.

My Example:

Catalog Service = Products, Categories, Brands.
Future Order Service = Orders, Payments.
Communication → REST / gRPC / Messaging (RabbitMQ, Kafka).
Independent DBs → CatalogDB, OrderDB.

👉 Interview Tip: Microservices solve scaling + team autonomy, but add complexity (distributed systems).

🎯 How They All Connect in My Project

DDD → defines my Domain & Repositories.
Clean Architecture → ensures dependency flow is inward.
Onion Architecture → shows layered structure visually.
Microservices → breaks Catalog, Order, Auth into separate deployable units.

✅ So when interviewers ask “Which architecture are you following?” → My answer:

“I use a combination of DDD for domain modeling, Onion/Clean for dependency flow, and Microservices for scalability and independence. For example, in my Catalog Service, Product is an Aggregate Root in the Domain Layer, EfRepository<Product> sits in Infrastructure, ProductService in Application, and ASP.NET Controllers in the Presentation Layer. This separation ensures testability, maintainability, and future scalability.”

📌 GitHub repo https://github.com/SiddharthBhamare/SmartKart-Catalog.git

Stay tuned! 🚀

dotnet #architecture #ddd #cleanarchitecture #microservices #onionarchitecture #softwareengineering

🔐 SmartKart Microservices Series Day 2

Siddharth Bhamare — Sat, 26 Jul 2025 13:13:04 +0000

🧭 Day 2: Kicking off Auth Service – Exploring Keycloak vs. Duende IdentityServer

In our journey to build a robust, secure, and scalable e-commerce platform (SmartKart 🛒) using .NET Core and Microservices, we are now diving into the Authentication and Authorization layer.

Why Start With AuthService?
Authentication is a cross-cutting concern and a foundational piece for secure APIs. Starting here ensures all downstream services follow a consistent security model.

🔍 What is Keycloak (in simple words)?
Keycloak is an open-source identity and access management tool.

Think of it as a central place where:

Users register and log in 🔑
Roles and permissions are managed 🛡️
Tokens (JWT) are issued for secure API access 🧾

And the best part? It already has all the features built-in — you don’t have to code login pages, password management, token handling, etc., yourself!

✅ Why We Chose Keycloak Over Other Options
As a .NET Core engineer, I evaluated a few options like:

ASP.NET Identity + JWT (custom)
Duende IdentityServer (formerly IdentityServer4)
Keycloak

Here's why I picked Keycloak:

✔️ 1. Fully Open Source
No license needed for commercial use. Duende requires a paid license for most real-world projects.

✔️ 2. Feature-Rich, Out of the Box
Login UI, forgot password, role mapping, token issuance — all ready without writing extra code.

✔️ 3. Centralized User Management
You get a user-friendly admin panel to:

Add/edit users
Assign roles
Configure clients/apps

✔️ 4. Standards-Based Protocols
Supports OAuth2.0, OpenID Connect, and even SAML — works well with .NET Core's JWT middleware.

✔️ 5. Easy to Integrate with .NET Core
Though written in Java, it’s protocol-based, so integration with .NET Core is seamless using:

JWT Bearer Authentication
OpenID Connect client libraries

✔️ 6. Scalable & Cloud-Friendly
Supports Docker, Kubernetes, and clustering — ideal for microservices.

📌 Summary Comparison Table :

Feature	Keycloak ✅	Duende IdentityServer ❌
Open Source (free to use)	✔️ Yes	❌ No (requires paid license)
Admin UI for users/roles	✔️ Built-in	❌ Needs custom development
Login/Register/Forgot UI	✔️ Provided	❌ Build yourself
Protocol Support (OAuth2/OIDC)	✔️ Yes	✔️ Yes
Easy .NET Core Integration	✔️ Yes (via JWT/OIDC)	✔️ Native
Multi-tenant support	✔️ Realms	❌ Manual effort

🚀 SmartKart Microservices Series – Day 1: Architecture & Flow

Siddharth Bhamare — Fri, 25 Jul 2025 17:02:35 +0000

🔷 Welcome to Day 1 of my Full-Stack Learning Challenge!

Over the next few weeks, I’ll be building SmartKart, a complete E-Commerce system using cutting-edge technologies and principles. This series will not only help me learn but also share back insights with the community 💡

🧠 Why SmartKart?
As a backend engineer, I wanted a project that showcases:

Microservices using .NET 8, Clean Architecture, and DDD
API Gateway, Service Discovery, and Message Brokering with RabbitMQ
Caching with Redis
Full-text search using Elasticsearch
Secure Auth with JWT, OAuth2, and OpenID Connect
ORM with EF Core, performance boosts with Dapper
Future-ready: scalable frontend in React/React Native + Docker CI/CD

🏗️ Architecture Overview

🔹API Gateway
Acts as the single entry point, handling routing, authentication, rate limiting, etc.

🔹 Auth Service
Implements IdentityServer/OpenID for user login, registration, and token issuance.

🔹 Product, Order, Cart, and Payment Services
Each built as isolated microservices with their own DB, following Clean Architecture and DDD.

🔹Event Bus (RabbitMQ)
Enables asynchronous communication between services for tasks like inventory update, order placed, etc.

🔹 Redis
Used for caching product data and session tokens to boost performance.

🔹 Elasticsearch
For blazing-fast search experience in products and orders.

🔹 Service Registry (Consul/equivalent)
Enables service discovery for scalable deployment.

🔹 Database
Each service uses its own DB (PostgreSQL or SQL Server), connected via EF Core, and selectively using Dapper where performance is critical.

🔹 Dockerized Setup
All services will run locally in Docker containers to ensure production-like behavior without cloud dependency.

🗺️ Request Flow (Example)
1️⃣ A user logs in → Auth Service returns a JWT
2️⃣ API Gateway validates JWT and routes request to Product Service
3️⃣ Product Service fetches data (from DB or Redis), returns to Gateway
4️⃣ User adds product to cart → Cart Service updates DB and emits event to RabbitMQ
5️⃣ Order Service consumes the event and updates the order pipeline
6️⃣ All updates are searchable via Elasticsearch!