Forem: Ahmed Adel

Amazon Nova 2: AWS's Reasoning Models That Actually Get Production Needs!

Ahmed Adel — Sun, 28 Dec 2025 15:13:11 +0000

🚀 Amazon Nova 2: AWS's Reasoning Models That Actually Get Production Needs!

Hey devs! 😎 Ever built an AI agent that overthinks pizza orders but chokes on complex workflows? Amazon Nova 2 fixes that with adaptive reasoning—think more on hard problems, less on easy ones. Launched Dec 2025, these Bedrock models bring 1M-token context, multimodal magic (text+images+video+speech), and thinking budgets you control.

🔥 Why Nova 2 Feels Like Cheating

Nova 2 Lite & Pro solve the classic AI tradeoff: speed vs intelligence.

Key superpowers:

🎛️ Thinking Budgets: Low/Medium/High—dial reasoning depth to match task complexity
🛠️ Built-in Tools: Web grounding, code interpreter, GitHub MCP server integration
🌐 1M Token Context: Long-range planning without forgetting
🔒 Safety First: Content moderation, Bedrock Guardrails, IP indemnity

"Nova 2 Lite outperforms Nova Premiere on agentic workflows at 7x lower cost and 5x faster"

📊 The Full Nova 2 Family Breakdown

Model	Status	Best For	Context	Modalities	Price-Perf Edge	🌟 Standout
Nova 2 Lite	GA ✅	Code gen, search, daily tasks	1M tokens	Text+Image+Video	7x cheaper than Premiere	Adaptive budgets
Nova 2 Pro	Preview 🔓	Agentic coding, complex planning	1M tokens	Text+Image+Video+Speech	SOTA math/tool use	Multi-step orchestration
Nova Sonic	GA ✅	Real-time voice agents	300k tokens	Speech (5+ langs)	Human-like convos	Low-latency dialog
Multimodal Embeddings	GA ✅	Agentic RAG/search	Unified space	Text+Image+Video+Audio	Single model for all	Cross-modal retrieval
Nova Act	Research Preview 🔬	Browser automation	N/A	Web UI	Clicks+forms+search	Autonomous web agents

💼 Real-World Wins That'll Blow Your Mind

1. LangChain Repo Update (Live Demo!) 🎥

Nova 2 Pro agent analyzed GitHub issues, created branches, read existing code, implemented changes, tested, and submitted a production-ready PR—all autonomously using GitHub MCP server integration.

Workflow breakdown:
Analyze issue → Plan solution

Create working branch

Analyze code structure

Implement Nova 2 support

Test + Create PR ✅

2. Production Case Studies 🏢

Company	Use Case	Nova Model	Impact
Fortinet	Support chatbot	Nova Micro	Efficient doc access, reduced inference costs
Infosys	Document extraction	Nova Pro	Text+images → structured data extraction
Loka	Video surveillance	Nova Pro	Analyzed millions of visual events
Accenture	Personalization	Nova + Agents	Content personalization at scale
Monks	Brand compliance	Nova models	Real-time decisions across 20 docs

🧠 Thinking Budgets: The Secret Sauce

Simple tasks ⚡ Reasoning OFF
User: "Summarize these 5 URLs"
Nova 2: Parallel web search → Instant summary

Complex agents 🧠 Reasoning HIGH
User: "Update LangChain for Nova 2"
Nova 2: Plan → Multiple tool calls → Execute → Test → PR
___

Pro Tip: Chat apps = reasoning OFF. Multi-step agents = reasoning ON with appropriate budget.

🔧 Developer Quickstart

Access via Amazon Bedrock API

curl -X POST https://bedrock-runtime.us-east-1.amazonaws.com/models/nova-2-lite
-d '{
"prompt": "Plan LangChain Nova 2 integration",
"thinking_budget": "high",
"tools": ["web_search", "code_interpreter", "github_mcp"]
}'

Key parameters:

Thinking budget: "off|low|medium|high".
Tools: Built-in web grounding, code interpreter.
Framework: Strands (agent workflows) + Agent Core (runtime)

🌍 Language & Fine-tuning Power

Capability	Details
Languages	200+ supported, optimized for 15 (English, Arabic, Chinese, Hindi, etc.)
Fine-tuning	Text + vision fine-tuning on proprietary data
Customization	Brand-specific outputs, domain adaptation

💰 Cost Breakdown (Why You'll Love It)

Task Type	Old Approach	Nova 2 Approach	Savings
Simple Q&A	Expensive overthinking	Reasoning OFF	7x cheaper
Agent workflows	Slow + costly	Adaptive budgets	5x faster
Multimodal RAG	Multiple specialized models	Single embeddings model	Simplified stack

🚀 Get Started NOW

Free Tier: aws.amazon.com/bedrock/free
Documentation: Nova 2 User Guide
Console: Bedrock → Nova models → Test playground
Frameworks: Strands + Agent Core for production agents

🎯 Who Should Build With Nova 2?

DevOps 👨‍💻: CI/CD automation, infrastructure agents
Fullstack 🌐: Chat + RAG + tool-calling apps
Data teams 📊: Multimodal search/extraction
ML engineers 🔬: Production fine-tuning at scale

Nova 2 delivers production-ready intelligence that scales without breaking the bank. Who's spinning up their first agent today? Drop your builds below! 👇🚀

Architecting Large-Scale Migrations with Fannie Mae and the NRO (AWS re:Invent 2025 – WPS201)

Ahmed Adel — Sat, 13 Dec 2025 13:48:37 +0000

Session: WPS201 - AWS re:Invent 2025

This comprehensive session explores how to migrate complex enterprise and federal workloads to AWS without disrupting critical missions or compromising security controls. The presentation combines core migration patterns, AWS Well-Architected guidance, and real-world customer journeys from the National Reconnaissance Office (NRO) and Fannie Mae.

Why large migrations are hard

The session opens by revisiting familiar migration drivers: lowering compute costs and increasing innovation velocity as organizations move from on-premises environments to rehosted, replatformed, and cloud-native architectures.

The 7 Rs of Migration

The speaker frames migration strategy in terms of the "7 Rs," emphasizing that large-scale programs almost always use a mix of these approaches rather than a single pattern:

Strategy	Description	Best For
Rehost	"Lift and shift" to cloud	Quick migration, minimal changes
Relocate	Move to different infrastructure	Hypervisor-level migrations
Repurchase	Move to SaaS solutions	Replacing custom applications
Retain	Keep on-premises	Compliance or latency requirements
Retire	Eliminate unused applications	Reducing technical debt
Replatform	Minor cloud optimizations	Moderate benefits with low risk
Refactor	Redesign for cloud-native	Maximum cloud benefits

⚠️ Key Message: Tooling alone is not enough, especially when dealing with diverse legacy systems, complex databases, and tight budgets. Organizations with limited cloud experience must invest early in automation, governance, and repeatable mechanisms for account management.

Foundational patterns: MRA, phases, and landing zones

The session recommends using an AWS Migration Readiness Assessment (MRA) as an on-ramp for any large program. The MRA process helps:

Inventory applications and discover dependencies
Determine appropriate migration strategies per workload
Avoid one-off, unstructured decisions

Migration Phases and Tools

Phase	Focus	Key AWS Tools
Assess	Discovery & planning	Application Discovery Service, Migration Readiness Assessment
Mobilize	Setup & preparation	Control Tower, Landing Zone, Account vending
Migrate/Modernize	Execution & optimization	Migration tools, Well-Architected reviews

In parallel, organizations should establish an AWS Control Tower-style landing zone or "account vending machine" to provision new accounts quickly and consistently, with built-in guardrails for security, compliance, and governance.

Applying the AWS Well-Architected Framework

Throughout the session, the AWS Well-Architected Framework acts as the backbone for design and review. The six pillars are positioned as a way to reason about trade-offs for each workload, not as a checklist to satisfy after migration.

AWS Well-Architected Framework Pillars

Pillar	Focus Area	Key Considerations
Operational Excellence	Running & monitoring systems	Automation, procedures, continuous improvement
Security	Protecting information & systems	Identity, permissions, data protection
Reliability	System availability & recovery	Fault tolerance, backup, disaster recovery
Performance Efficiency	Using resources effectively	Right-sizing, monitoring, technology selection
Cost Optimization	Delivering value at lowest cost	Resource optimization, pricing models
Sustainability	Minimizing environmental impact	Efficiency improvements, renewable energy

💡 Practical Tip: Ask business stakeholders which pillars matter most for each application. Some workloads may emphasize reliability and operational excellence, while others prioritize cost or performance. These decisions should drive architecture choices, capacity planning, and operational runbooks.

Organizational enabler: a Cloud Center of Excellence

A recurring theme is the importance of a Cloud Center of Excellence (CCoE) for sustained modernization. The CCoE is described as a cross-functional team spanning:

🏗️ Cloud architecture
🖥️ Infrastructure
🔒 Security
⚙️ Operations
💻 Software engineering

CCoE Responsibilities

Rather than a purely formal committee, the CCoE serves as a pragmatic mechanism for scaling expertise:

✅ Standardizing landing zones and network patterns
🔐 Defining IAM controls and security policies
💰 Establishing cost management practices
📚 Codifying best practices and lessons learned
🚀 Enabling innovation across business units

Both Fannie Mae and the NRO leveraged CCoEs to significantly increase their "innovation velocity" after completing initial migrations.

Fannie Mae: reimagining financial forecasting

🏦 Fannie Mae Case Study: Financial Forecasting Transformation

The Fannie Mae story focuses on transforming a decades-old financial forecasting process into a modern, scalable platform on AWS. As a cornerstone of the U.S. housing finance system, Fannie Mae:

🏠 Purchases home loans from lenders
📦 Packages them into mortgage-backed securities
💼 Sells them to investors

This makes accurate, timely forecasting of portfolio performance a mission-critical capability.

Legacy System Challenges

Challenge	Impact
Spreadsheet-heavy processes	Manual errors, limited scalability
Fragmented systems	Data silos, integration complexity
End-of-life infrastructure	Performance bottlenecks, maintenance costs
Spread-thin expertise	Knowledge silos, single points of failure
Heavy customizations	Slow change cycles, brittle systems

These limitations severely constrained the organization's ability to provide forward-looking insights in a dynamic macroeconomic environment.

Ambitious goals: from 80 days to 10 days

🎯 Program Objectives: Transforming at Scale

Launched in September 2023, the Forecast Transformation Program set ambitious goals:

Metric	Before	Target	Improvement
Forecasting lifecycle	~80 days	~10 days	87.5% reduction
Stress-test execution	Sequential	Parallel	Concurrent processing
System consolidation	~80 systems	1 platform	98.75% reduction
Calculations consolidated	~2,000	Unified	Single source of truth
Loan records capacity	Limited	1.5B per run	Massive scale

Additional Requirements

⚡ Auto-scaling compute based on demand
🔒 Strict regulatory compliance (security, auditability, data quality)
🎭 Holistic approach - tackling people, process, technology, and governance simultaneously

Target architecture: S3 backbone, EMR, and Step Functions

🏗️ Solution Architecture

Fannie Mae selected AWS for its comprehensive storage and database services and ability to meet stringent security and compliance requirements.

Component	AWS Service	Purpose	Key Benefits
Data Backbone	Amazon S3	Unified data domain	Input data, model outputs, calculation results, analytics data
Compute Engine	Amazon EMR	Big data processing	Billions of loan records, multiple clusters, auto-scaling
Orchestration	AWS Step Functions	End-to-end workflow	Data ingestion, models, calculations, system integration
Metadata & Config	Aurora + DynamoDB	Configuration management	Scenarios, model parameters, calculation rules
User Interface	Angular on Fargate	Business user experience	Scenario definition, input specification, execution triggers
Analytics	SageMaker + Tableau	Reporting & analysis	Regulatory reporting, internal analytics

💡 Smart Design Choice: YAML Business Logic

A particularly clever design encodes business calculations in YAML, which:

✅ Decouples business logic from application code
✅ Enables business-driven changes without code redeployment
✅ Reduces development cycle time for rule updates

Execution flow: from user input to analytics

🔄 Execution Flow: From Input to Analytics

graph TD
    A[Business User Input] --> B[Data Ingestion]
    B --> C[S3 Data Backbone]
    C --> D[Model Execution]
    D --> E[EMR Calculations]
    E --> F[Output Processing]
    F --> G[Analytics & Reporting]

Step-by-Step Process

👤 User Input: Business users select scenario types and enter assumptions via UI
📥 Data Ingestion: Platform pulls data from multiple systems of record
🗄️ Data Storage: All data written to S3 data backbone
🤖 Model Execution: Invokes existing and new platform-specific models
⚙️ Rule Application: EMR applies thousands of YAML-defined business rules
📤 Output Distribution: Results routed to downstream systems via APIs and SNS
📊 Analytics Preparation: Data prepared for regulatory and management tools

Real-world engineering challenges

🏗️ Engineering Challenges at Scale

Building this platform was not straightforward - here's what the team faced:

Team & Process Challenges

Challenge	Scale	Solution Approach
Team coordination	100+ engineers	Standardized guardrails, coding standards, integration patterns
System integration	76 upstream/downstream systems	Careful API design, minimal disruption approach
Program duration	Multi-year timeline	Consistent governance, regular architecture reviews

Technical & Operational Challenges

💾 Data Volume: Each scenario generates ~500 TB of data
- Required early archival strategy design
- Drove significant storage cost considerations
⚡ Performance Optimization: Multiple EMR clusters processing massive workloads
- Minimized cross-AZ traffic for latency reduction
- Controlled data transfer costs
📈 Scaling Limits: Constantly hitting AWS service quotas
- Continuous monitoring and quota increases
- Proactive capacity planning

Cost, performance, and reliability: applied Well-Architected

💰 Cost Optimization: Business Requirement, Not Nice-to-Have

Cost optimization was treated as a business requirement, with continuous architecture refinement using the Well-Architected Framework.

Optimization Strategies

Strategy	Implementation	Benefit
S3 Partitioning	Smart data organization	Reduced query costs, improved performance
Redshift Spectrum	Analytics over S3 data	Cost-efficient analytics without data movement
Pre-partitioning	S3 optimization	Support for tens of millions TPS
Real-time monitoring	Cost visibility framework	Proactive cost management
Pre-execution estimates	UI cost preview	Business user cost awareness

EMR Fleet Strategy

Mixed instance types and architectures for optimal cost-performance:

🔧 Intel processors for compatibility
⚡ AMD processors for cost-performance balance
🚀 AWS Graviton for energy efficiency and cost savings
🎯 Workload-based selection for optimal resource utilization

Measurable outcomes

📈 Measurable Outcomes: Transformational Results

The transformation delivered significant and measurable improvements across all key metrics:

Performance & Cost Results

Metric	Before	After	Improvement
Infrastructure costs	Baseline	61% of baseline	39% reduction
Execution time	Baseline	30% of baseline	70% reduction
Manual processes	Baseline	40% of baseline	60% reduction
System execution	~20 days	~3 days	85% reduction
Loan records/scenario	~1.4M	~36M	25x increase

Operational Efficiency

📊 Scenario capacity: 20+ scenarios per month at massive scale
👁️ Observability: Dramatically improved monitoring and governance
👥 Team efficiency: From 15 squads (build) → 2 squads (support)
🎯 Operational complexity: Significant reduction in day-to-day management

These results represent a complete transformation from a legacy, constraint-heavy system to a modern, scalable, cloud-native platform.

Lessons you can apply

🎓 Key Lessons for Your Organization

The session concludes with actionable lessons that extend beyond this single use case:

Core Principles

Principle	Description	Why It Matters
Marathon mindset	Many sprints, not a big bang	Sustainable progress, manageable risk
Celebrate incremental wins	Acknowledge progress milestones	Maintain team motivation, stakeholder support
People over technology	Focus on collaboration & governance	Technology is an enabler, people determine success
Business-first approach	Start with problems, not solutions	Ensures technology serves business objectives

Non-Negotiable Foundations

🗄️ Data Quality & Governance

Critical for migration success
Essential for advanced analytics and AI enablement
Must be established from day one, not retrofitted

🤝 Agile Stakeholder Engagement

Iterative collaboration with business stakeholders
Continuous refinement with cloud providers
Architecture evolution based on real-world feedback

💡 Final Takeaway: Successful large-scale migrations require a balanced approach combining technical excellence, organizational change management, and continuous stakeholder engagement. Technology amplifies good processes and governance - it doesn't replace them.

Linear and Canary releases in AWS ECS

Ahmed Adel — Sat, 08 Nov 2025 20:34:14 +0000

Ahmed Adel

Nov 7 '25

💡 Next-Level Deployments in AWS ECS: Step-by-Step Guide to Linear & Canary Releases 🚀

Comments

2 min read

💡 Next-Level Deployments in AWS ECS: Step-by-Step Guide to Linear & Canary Releases 🚀

Ahmed Adel — Fri, 07 Nov 2025 12:43:32 +0000

AWS ECS just dropped one of its most practical features for modern DevOps: built-in Linear and Canary deployments! 😲 No more stitching together custom scripts or relying on external tools for smooth traffic-shifting and safer releases.

Whether you’re all-in on microservices or just want to reduce the risk of production rollouts, this guide’s for you! Let’s get our hands dirty with a real ECS service so you can confidently say, “I did it!” ✌️

Why Linear & Canary Deployments? 🧐

Safer Updates: Gradually test new versions with real traffic.
Instant Rollbacks: Detect issues and revert 🚨 fast.
Automation: Support from AWS Console, CLI, CloudFormation, CDK, Terraform.
Works with ALB & Service Connect: Play nice with modern networking.

Hands-on: Update Your ECS Service with a Canary or Linear Deployment

🎨 We'll use plenty of screenshots and CLI output. You can grab icons or illustrations from AWS Console or use royalty-free image sites for ECS diagrams.

🛠️ Prerequisites

Existing ECS service using ALB or ECS Service Connect.
New Docker image version ready in ECR.
Permissions to update services.

1. Open the ECS Console & Select Your Service

Go to your ECS Cluster
Pick the Service to update

2. Hit “Deploy new revision” and Configure Deployment Settings

You’ll now see new options: Linear and Canary deployments! 🎉

3. Configure Deployment Params

Linear: Set the step percentage (e.g. 20%). ECS will shift 20% ➡️ 40% ➡️ 60%... of live traffic to the new revision in intervals you specify.
Canary: Route just a slice (e.g. 10%) of traffic to the new revision. Let it bake, then move all over if healthy.
Add CloudWatch alarms to trigger auto-rollbacks if things go wrong.

CLI Example for Linear

aws ecs update-service
--cluster my-cluster
--service my-service
--deployment-controller type=ECS
--deployment-configuration
deploymentType=Linear,stepPercentage=20,stepBakeTimeInMinutes=10

4. (Optional) Add Deployment Hooks 🪝

Pause for tests/validation between steps! Use AWS CLI or SDK to configure.

5. Watch the Magic Happen 🧙‍♂️

You’ll see tasks update, traffic shift, and alarms ready to catch issues. Instant rollback if trouble is detected.

Real-World Usage: When Should You Use What?

Strategy	Best For	Example
Linear	Gradual releases, user testing	Move 20% of traffic each step
Canary	Quick prod validation	Test 10%, then release to 100%

🎯 Pro-tips

Use deployment hooks for integration/health checks.
Monitor everything with CloudWatch.
Works great with both HTTP and gRPC APIs.
Combine with ECS Service Connect for simple, secure service-to-service comms.

🌟 Wrap-up

This feature is fresh, highly practical, and ready for you! Demos or code in this post won't be found in most tutorials (yet). Give it a try and tell your infra team you’re on the cutting edge! 🤓

📝 Try it, share your results, and let me know if you want an in-depth look at deployment hooks or ECS Service Connect next!

(Images: AWS product pages & console screenshots. Get creative with icons and emoji for your audience!)

A Brief Guide to AWS SageMaker Services

Ahmed Adel — Sat, 05 Apr 2025 07:54:58 +0000

A Practical Tour of Key Amazon SageMaker Services

Amazon SageMaker is a managed platform that helps teams build, train, and run machine learning (ML) workloads without stitching together separate tools. Instead of maintaining custom infrastructure for every stage of the ML lifecycle, you can rely on SageMaker components for data prep, training, deployment, monitoring, and governance. This article walks through several important SageMaker capabilities and how they fit into a modern ML workflow.

1. SageMaker Automatic Model Tuning

SageMaker Automatic Model Tuning runs many training jobs with different hyperparameter settings and searches for the configuration that yields the best objective metric. It uses Bayesian optimization under the hood to decide which combination of hyperparameters to try next, so you do not have to manually run dozens of experiments yourself.

Key benefits:

Cuts down on manual hyperparameter search effort.
Often improves model quality by efficiently exploring the search space.
Works with built‑in, marketplace, and custom algorithms.

2. SageMaker Deployment and Inference

SageMaker provides managed infrastructure to host trained models and expose them as endpoints or batch jobs, so you can serve predictions at scale without managing servers directly. It supports real‑time, batch, asynchronous, and serverless inference options, along with traffic‑splitting for A/B testing and safe rollouts.
Common deployment patterns:

Real‑time endpoints for low‑latency prediction APIs.
Batch transform for large, offline scoring jobs.
Asynchronous inference for long‑running requests.
Serverless inference for spiky or low‑throughput workloads.

3. SageMaker Studio

SageMaker Studio is a web‑based integrated development environment that centralizes many ML activities in a single visual interface.
Within Studio, you can launch notebooks, run experiments, debug models, and monitor performance without constantly switching tools.

Main capabilities:

Managed notebooks for exploration and prototyping.
Experiment tracking to organize and compare runs.
Debugging tools to inspect training behavior.
Built‑in monitoring views for deployed models.

4. SageMaker Data Wrangler

Data Wrangler is designed to shorten the “data cleaning and feature engineering” phase by providing a visual workspace for data preparation. You can connect to data sources, explore data distributions, apply transformations, and then export the prepared dataset into downstream SageMaker workflows.

What you can do:

Apply hundreds of built‑in data transformations and feature ops.
Visualize data quality and distributions as you iterate.
Engineer features and send them into SageMaker Pipelines.

5. SageMaker Clarify

SageMaker Clarify provides bias and explainability tooling for your ML models, helping you understand both how they behave and how they might impact different user groups. It analyzes training data and predictions to surface fairness metrics and generate feature‑importance explanations.

Key features:

Bias detection metrics on datasets and model outputs.
Model explainability reports for stakeholders.
Feature importance for both global and local explanations.
Support for compliance and responsible‑AI requirements.

6. SageMaker Ground Truth

Ground Truth helps you build labeled datasets by managing annotation workflows and integrating with human labelers. You can use Amazon’s own workforce, third‑party vendors, or your internal team, depending on the data sensitivity and scale requirements.

Labeling options:

Public workforce through Amazon Mechanical Turk.
Vendor workforces managed by AWS partners.
Private labeling teams within your organization.

7. SageMaker Model Cards

Model Cards in SageMaker provide a structured template for documenting important details about each model, which is critical for governance and audits. They centralize context like intended use, training data characteristics, and evaluation metrics in a standardized format.

Typical sections include:

High‑level model description and owners.
Expected use cases and known limitations.
Training data sources and configuration.
Evaluation methodology and performance results.

8. SageMaker Model Dashboard

SageMaker Model Dashboard offers a unified view across models that are running in production so teams can track health and risk in one place. It aggregates information from monitoring tools to help you see which models might require retraining or investigation.

Core capabilities:

Tracking model performance indicators over time.
Detecting data and prediction drift patterns.
Configuring alerts and notifications for issues.

9. SageMaker Model Monitor

Model Monitor continuously evaluates the quality of models deployed in SageMaker by comparing live traffic to a defined baseline. When it detects drift or quality issues, it can surface alerts so you can decide whether to retrain, roll back, or adjust the model.

Things it can monitor:

Input data quality and schema consistency.
Prediction quality and performance metrics.
Bias drift over time on key attributes.
Changes in feature attribution patterns.

10. SageMaker Model Registry

The Model Registry is a catalog for your trained models, enabling controlled promotion from experimentation to production. It tracks versions, metadata, and approval states so that ML and MLOps teams can coordinate deployments more safely.

Key capabilities:

Versioning of model artifacts and metadata.
Approval workflows for moving models between stages.
Lineage tracking across training, evaluation, and deployment.

11. SageMaker Pipelines

SageMaker Pipelines is a purpose‑built workflow engine for ML that lets you define and automate multi‑step pipelines, from preprocessing to training to deployment. Pipelines make ML workflows reproducible, repeatable, and easier to integrate with CI/CD systems.

Advantages:

Ensures reproducible runs with versioned definitions.
Encourages reusability of standardized pipeline steps.
Integrates with DevOps tooling for automated releases.

12. SageMaker Role Manager

Role Manager simplifies the process of configuring permissions for SageMaker resources by offering templates and guided workflows. Instead of manually writing complex IAM policies, you can start from predefined roles and then adjust them as needed.

Highlights:

Curated roles for common SageMaker usage patterns.
Fine‑grained permission controls for security teams.
Tight integration with AWS Identity and Access Management (IAM).

13. SageMaker JumpStart

JumpStart provides a gallery of pre‑built solutions, pre‑trained models, and example notebooks that you can deploy or customize with just a few steps. It is particularly useful when you want to quickly prototype a use case like classification, forecasting, or computer vision without building everything from scratch.

What JumpStart offers:

Ready‑to‑use pre‑trained models for common tasks.
End‑to‑end solution templates with infrastructure code.
Example notebooks you can open directly in Studio.

14. SageMaker Canvas

SageMaker Canvas targets business users who want to generate predictions without writing code or managing infrastructure. It provides a visual interface where users can connect data, build ML models with AutoML, and explore predictions interactively.

Key traits:

Drag‑and‑drop interface for non‑technical users.
Automatic model building and selection behind the scenes.
Focus on business‑friendly workflows and collaboration.

15. SageMaker MLflow Integration

SageMaker integrates with the open‑source MLflow ecosystem so that teams already using MLflow for experiment tracking and model management can tie those workflows into SageMaker services. This integration lets you log runs, track artifacts, and manage model versions while still deploying on SageMaker.

You can use it to:

Track experiments and metrics in MLflow while training on SageMaker.
Register models in an MLflow registry and connect them to SageMaker deployments.
Store training artifacts in a centralized and queryable location.

Wrapping up

AWS SageMaker brings together many specialized services that collectively cover data preparation, experimentation, deployment, monitoring, and governance for ML workloads. By adopting the right mix of these tools, teams can shorten development cycles and operate ML systems more reliably at scale.

For deeper dives into specific features, see the official SageMaker documentation.

References

AWS SageMaker Documentation.
AWS Machine Learning Blog.
AWS re:Invent sessions and technical talks.

🚀 EBS Volumes for EC2: Should You Use Multiple Small Volumes or One Large Volume? 💡

Ahmed Adel — Tue, 07 Jan 2025 13:30:44 +0000

🚀 Attaching Persistent EBS Volumes to EC2: Multiple Small Volumes vs. One Large Volume 🚀

When working with Containers in Amazon EC2, you might need to attach persistent EBS (Elastic Block Store) volumes to your containers. But what if you have multiple containers, and each one needs its own storage? Should you create multiple smaller EBS volumes or one large volume? Let’s break it down! 💡

🤔 The Dilemma: Multiple Small Volumes vs. One Large Volume

Imagine you have 5 containers, and each needs 20 GB of persistent storage. You have two options:

Create 5 separate EBS volumes (20 GB each).
Create 1 large EBS volume (100 GB) and share it among the containers.

Which one should you choose? Let’s explore the cost, performance, and management implications of each approach. 🧐

💰 Cost: Is There a Difference?

The cost of EBS volumes depends on the size, type, and usage duration. Here’s the breakdown:

EBS Pricing Model: You pay for the provisioned storage size (per GB per month), regardless of how much data you actually store.
Cost Comparison:
- 5 x 20 GB Volumes: If each 20 GB volume costs $X per month, then 5 volumes would cost 5 × $X.
- 1 x 100 GB Volume: A single 100 GB volume would cost 1 × $X.

In most cases, the total cost for 5 x 20 GB volumes is the same as 1 x 100 GB volume. However, if you use advanced features like provisioned IOPS, the cost may vary slightly.

⚡ Performance: Isolation vs. Shared Resources

Multiple Smaller Volumes

Each EBS volume has its own IOPS (Input/Output Operations Per Second) and throughput limits.
If your containers have independent I/O workloads, multiple smaller volumes provide better performance isolation.
One container’s I/O won’t affect another.

Single Larger Volume

A single 100 GB volume has higher aggregate IOPS and throughput compared to a 20 GB volume.
However, if multiple containers share the same volume, their I/O workloads may compete for resources, leading to potential bottlenecks.

🛠️ Management: Complexity vs. Simplicity

Multiple Smaller Volumes

You’ll need to attach and manage each volume separately to the EC2 instance.
Each container will need its own mount point on the EC2 instance.
This approach is more complex but provides better isolation for each container.

Single Larger Volume

Easier to manage since you only need to attach and mount one volume.
You can create subdirectories on the volume for each container and mount them separately in the ECS task definition.
However, this approach requires careful planning to avoid I/O contention between containers.

🏆 Recommendations

Use Multiple Smaller Volumes (5 x 20 GB) if:

Each container has independent I/O workloads and requires performance isolation.
You want to avoid I/O contention between containers.
You are okay with the additional management overhead.

Use a Single Larger Volume (1 x 100 GB) if:

The containers have low I/O workloads and can share the same volume without contention.
You want to simplify management and reduce the number of volumes to monitor.
You can use subdirectories to logically separate data for each container.

🛠️ Example Setup

Option 1: Multiple Smaller Volumes

Create 5 EBS volumes (20 GB each).
Attach all 5 volumes to the EC2 instance.
Mount each volume to a separate directory (e.g., /mnt/volume1, /mnt/volume2, etc.).
In the ECS task definition, map each container to its respective volume mount point.

Option 2: Single Larger Volume

Create 1 EBS volume (100 GB).
Attach the volume to the EC2 instance.
Mount the volume to a directory (e.g., /mnt/shared-volume).
Create subdirectories for each container (e.g., /mnt/shared-volume/container1, /mnt/shared-volume/container2, etc.).
In the ECS task definition, map each container to its respective subdirectory.

💡 Final Thoughts

Cost: The total cost for 5 x 20 GB volumes is likely the same as 1 x 100 GB volume.
Performance: Multiple smaller volumes provide better performance isolation.
Management: A single larger volume is easier to manage but may lead to I/O contention.

Choose the approach that best fits your performance requirements and management preferences. If you need isolation and independent performance, go with multiple smaller volumes. If you prefer simplicity and shared storage, use a single larger volume.

Happy deploying! 🚀

What’s your take? Do you prefer multiple smaller volumes or one large volume? Let’s discuss in the comments! 👇

AWS re:Invent 2024 - Zero to production serverless in 8 weeks

Ahmed Adel — Sun, 22 Dec 2024 17:33:54 +0000

🚀 Building an ADHD Prescription Integration Under Tight Deadlines Using Serverless Architecture 🌐💊

In early 2024, we embarked on an exciting but challenging project at Chemist4U, the UK's third-largest online pharmacy dispensing an impressive 175,000–200,000 items monthly. Our mission? To build a secure, scalable integration for a clinic specializing in ADHD diagnostics and treatment. This new system would replace the clinic’s existing pharmacy partner, which struggled to scale alongside their growing demand.

With tight deadlines and high stakes, this project was a race against time to ensure patients could continue receiving critical medications without any interruption. Here's how we made it happen! 💪

🎯 The Challenge: What Did We Need to Solve?

We were tasked with building a comprehensive solution that tackled:

Secure Data Integration: A safe, reliable way to transfer sensitive patient and prescription data.
Regulatory Compliance: ADHD medications are highly controlled, requiring checks on patient history and strict adherence to UK regulations.
Seamless Payment Handling: Private prescriptions needed secure payment processing, while NHS prescriptions required accurate invoicing for reimbursements.
Customer Service Enablement: Equipping the support team with tools to answer patient inquiries effectively.
Warehouse Integration: Ensuring prescriptions were processed smoothly by our ERP and warehouse systems.

Deadline: Eight weeks ⏳

🔧 Why Serverless?

Given the complexity and urgency, serverless architecture was the perfect choice. Here's why:

⚙️ Built-in High Availability: No need to manage uptime—it’s baked in.
📉 Low Administration Overhead: Skip the headache of managing EC2 instances or Kubernetes.
🚀 Rapid Development: Tools like AWS CDK allowed us to prototype and deploy features faster.
📈 Auto Scaling: Effortlessly handle peaks and troughs in prescription volume.

Curious about serverless? Check out AWS’s guide to serverless architecture. 🌟

💡 The Approach: Prototyping First

We skipped the proof of concept stage and dove straight into building a working prototype. Prototyping allowed us to test our ideas in real-world conditions while leaving room for rapid iterations.

🛠 Tools and Technologies

AWS CDK: For Infrastructure as Code (IaC).
Vitest: Unit testing from Day 1 ensured reliability.
Biome: Automated linting and formatting saved valuable time.
Hexagonal Architecture: Enabled modular development—essential for teamwork!
Serverless Framework: Combined with AWS Lambda, API Gateway, and DynamoDB for backend functionality.
React with Vite: For the front-end interface.

🎉 Day 1 Wins!

By leveraging pre-built boilerplate code and automation tools, we launched:

A basic authenticated website using AWS Cognito.
A stubbed API with validation powered by API Gateway.
CI/CD pipelines for automated deployments.
Fully isolated developer environments for efficient parallel work.

🏗️ Building the Core Integration

The centerpiece of the system was the integration with the clinic’s existing platform. Here's how we approached it:

1️⃣ Defining Interfaces Early

To avoid delays, we deployed a stubbed API with validation using Swagger. This allowed the clinic’s team to proceed with development in parallel.

2️⃣ Validation Handling

Initially, we used API Gateway’s built-in validation. However, the generic error messages were a bottleneck during testing. Switching to Zod gave us:

Customizable validation messages.
Consistent logic reuse in TypeScript.
Cleaner, more actionable error handling.

⚡ Infrastructure Setup with AWS CDK

Our serverless architecture consisted of:

API Gateway: Managed RESTful API endpoints.
Lambda Functions: Executed core business logic.
DynamoDB: Stored prescription and patient data securely.
S3 Buckets: Hosted the React front end and stored documents.

Want to learn CDK? Check out this CDK documentation. 📚

🌈 Rapid Iterations with Serverless

Serverless enabled us to:

Quickly deploy isolated development environments.
Make reproducible changes using IaC.
Maintain stability with automated tests.

🛠 Lessons Learned

💡 Prototyping is Key: Starting with a working prototype minimized rework.

💡 Better Error Handling Matters: Switching to Zod drastically improved testing efficiency.

💡 Serverless Speeds Things Up: By offloading infrastructure management, we could focus on delivering business value.

🎯 The Result

In just 8 weeks, we delivered a fully functional system that:

Secured sensitive patient data 🔒.
Automated workflows adhering to strict regulations ✅.
Integrated seamlessly with the clinic and warehouse systems ⚡.
Scaled to meet the growing demand for ADHD prescriptions 📈.

📺 Original Video

Want to dive deeper into our process? Watch the original video on YouTube:

What do you think of our approach? Have you worked on similar tight-deadline projects? Let us know in the comments! 👇

Hands-On: Deploy a Web App on an Nginx Server Using AWS App Runner

Ahmed Adel — Tue, 17 Dec 2024 19:02:25 +0000

Overview

In this tutorial, you will learn how to deploy a containerized application on an Nginx server using AWS App Runner. This hands-on guide is perfect for beginners and covers everything from building a container image to deploying it via App Runner.

AWS App Runner is a fully managed service that enables you to deploy containerized web applications and APIs at scale, with minimal effort. Whether you're starting with source code or a container image, App Runner takes care of the heavy lifting: automatic builds, deployments, load balancing, and traffic scaling.

What You Will Accomplish

By the end of this tutorial, you will:

Create a container image for your web app.
Push the image to Amazon Elastic Container Registry (ECR).
Deploy the image using AWS App Runner.
Clean up resources to avoid additional charges.

Prerequisites

Before diving into the steps, ensure you have:

An AWS Account:
- Ensure the account has administrator-level access. Accounts created within the past 24 hours may not have access to required services.
AWS Command Line Interface (CLI) installed and configured:
- Get started with AWS CLI.
Docker Engine installed and running:
- Follow the Docker installation guide.
Visual Studio Code (or your preferred code editor):
- Download Visual Studio Code.

Time and Cost

Time: ~20 minutes
Cost: Less than $0.16 (if completed within two hours and resources are deleted at the end).

Step 1: Create a Container Image

1.1 Set Up Your Project

Open a terminal and run the following commands to create a new directory for your project:

mkdir nginx-web-app
cd nginx-web-app

1.2 Create the Web Page

Open the nginx-web-app folder in Visual Studio Code (or your code editor). Inside this folder:

Create a new file named index.html and paste the following code:

<!DOCTYPE html>
<html>
<head>
    <title>Sample Web App</title>
    <style>
        html { color-scheme: light; }
        body { width: 35em; margin: 0 auto; font-family: Amazon Ember, Verdana, Arial, sans-serif; }
    </style>
</head>
<body>
    <h1>Welcome to AWS App Runner!</h1>
    <p>If you see this page, the Nginx web server is successfully installed and working.</p>
    <p><em>Thank you for using AWS App Runner!</em></p>
</body>
</html>


![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xpb5yllbmewq0etrlp3m.png)

1.3 Create a Dockerfile

Create another file named Dockerfile in the same folder and add the following content:

FROM --platform=linux/amd64 nginx:latest
WORKDIR /usr/share/nginx/html
COPY index.html index.html

1.4 Build the Container Image

Run the following command in the terminal to build the Docker image:

docker build -t nginx-web-app .

Step 2: Push Container Image to Amazon ECR

2.1 Create a Repository

Click Create Repository.

Enter nginx-web-app as the repository name and leave the defaults. Click Create Repository.

2.2 Push the Image to ECR

Select the newly created repository and click View push commands.

Follow the push commands provided, which typically include:

Authenticating Docker with ECR:

 aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account_id>.dkr.ecr.<region>.amazonaws.com

Tagging the image:

docker tag nginx-web-app:latest <account_id>.dkr.ecr.<region>.amazonaws.com/nginx-web-app

Pushing the image:

docker push <account_id>.dkr.ecr.<region>.amazonaws.com/nginx-web-app

Step 3: Create an AWS App Runner Service

Open the AWS App Runner Console.

Click Create an App Runner service.

In the Source and Deployment section:
- Select Amazon ECR as the source.
- Click Browse and select the nginx-web-app repository.

In Deployment Settings, select Create new service role and click Next.

On the Configure Service page:
- Enter nginx-web-app-service as the Service Name.
- Set the Port to 80.
- Click Next.

Review the configuration on the next page and click Create & Deploy.

Wait for the service to deploy. Once the status changes to Running, click the default domain name to view your web app.

Once the status updates to Running, choose the default domain name URL to view the web app.

The Welcome page and confirmation message should look like the image on the right.

Step 4: Clean Up Resources

To avoid additional charges, delete the resources you created:

In the AWS App Runner Console:
- Navigate to your service.
- Click Actions > Delete.

https://d1.awsstatic.com/Getting%20Started/tutorials/deploy-web-app-ngnix-app-runner/web-app-nginx-apprunner-4.2.d748703fb97dd7dace326ca9dfc144d89bcfcd86.png

In the Amazon ECR Console:
- Select the nginx-web-app repository.
- Click Delete.

Conclusion

Congratulations! You have successfully deployed a containerized Nginx web application using AWS App Runner. This guide introduced you to building container images, using Amazon ECR, and the power of AWS App Runner for seamless deployment.

Feel free to experiment further and explore advanced features like custom domains or continuous deployment pipelines. For more resources, check out the AWS App Runner documentation.

Happy coding! 🎉

[Boost]

Ahmed Adel — Wed, 04 Dec 2024 03:33:57 +0000

Discover Amazon S3 Metadata: A New Way to Explore Your Storage at AWS re:Invent 2024

Ahmed Adel for AWS Community Builders ・ Dec 4 '24

#aws #s3 #cloud #data

Discover Amazon S3 Metadata: A New Way to Explore Your Storage at AWS re:Invent 2024

Ahmed Adel — Wed, 04 Dec 2024 03:33:19 +0000

Amazon Web Services (AWS) has revolutionized data storage and management once again with a groundbreaking feature introduced at AWS re:Invent 2024: Amazon S3 Metadata. This new addition to Amazon Simple Storage Service (S3) simplifies the way we interact with and analyze the metadata of our S3 objects, empowering businesses to streamline workflows and enhance data insights.

Here’s everything you need to know about this powerful new feature.

The Challenge of Scale

Organizations leveraging Amazon S3 often deal with massive datasets — billions or even trillions of objects in a single bucket. Identifying specific objects based on characteristics like size, tags, or patterns in their keys is no easy task. Historically, businesses had to build custom systems to manage and query metadata, which could be complex, hard to scale, and prone to falling out of sync with the actual data.

What is Amazon S3 Metadata?

Amazon S3 Metadata introduces automated metadata capture for objects stored in S3 buckets. This metadata is stored in Apache Iceberg tables, enabling compatibility with tools like:

Amazon Athena
Amazon Redshift
Amazon QuickSight
Apache Spark

With these tools, you can perform scalable queries on metadata to find objects of interest efficiently, whether for analytics, data processing, or AI training.

Rich Metadata Elements

The metadata schema includes over 20 elements, such as:

Bucket Name and Object Key
Creation/Modification Time
Storage Class
Encryption Details
Object Tags
User Metadata

Additionally, the feature supports storing application-specific metadata in separate tables for advanced queries.

How Does It Work?

1. Enable Metadata Capture

To get started, designate a bucket and table to store your metadata. Metadata updates are automatically recorded whenever objects are created, modified, or deleted. Each update includes:

Record Type: CREATE, UPDATE, or DELETE
Sequence Number: Tracks historical records
Timestamps: Capture modification times

2. Query Metadata Effortlessly

Using Iceberg-compatible tools, query metadata to retrieve insights like:

Objects uploaded within a specific timeframe
Objects matching a particular tag or key pattern
Size-based filters for optimizing storage costs

S3 Default Metadata:

By default, S3 Metadata provides three types of metadata:

1- System-defined metadata, such as an object's creation time and storage class

2- Custom metadata, such as tags and user-defined metadata that was included during object upload

3- Event metadata, such as when an object is updated or deleted, and the AWS account that made the request.

For details about what data is stored in metadata tables, see S3 Metadata tables schema.

How Metadata Tables Work

Amazon S3 takes the reins when it comes to managing metadata tables, ensuring their accuracy and performance. Here’s what makes them stand out:

Read-Only for Integrity: Metadata tables are fully managed by Amazon S3 and are read-only to all IAM principals. This guarantees they always reflect the exact state of your bucket. You can delete your metadata tables if needed, but you can't directly modify them.
Automatic Maintenance: Amazon S3 periodically performs maintenance activities, such as file compaction and removal of unreferenced files. These automated processes help:
- 🔧 Optimize query performance.
- 💰 Minimize storage costs for metadata tables.
No Effort Required: This maintenance happens automatically—no need for opt-ins or manual configurations. However, if customization is required, you have the flexibility to configure these activities.

Hands-On Example

Here’s how you can enable and query metadata in a few simple steps:

Step 1: Create a Table Bucket

aws s3tables create-table-bucket --name my-metadata-bucket --region us-west-1

Step 2: Configure Metadata Capture

Prepare a JSON configuration file:

{
  "S3TablesDestination": {
    "TableBucketArn": "arn:aws:s3tables:us-west-1:123456789012:bucket/my-metadata-bucket",
    "TableName": "my_s3_metadata_table"
  }
}

and Attach this configuration to your data bucket:

aws s3api create-bucket-metadata-table-configuration \
  --bucket my-data-bucket \
  --metadata-table-configuration file://config.json \
  --region us-west-1

Step 3: Query Metadata

Using Apache Spark

spark-submit \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.6.0 \
  --conf "spark.sql.catalog.mytablebucket=org.apache.iceberg.spark.SparkCatalog" \
  --conf "spark.sql.catalog.mytablebucket.warehouse=s3://my-metadata-bucket" \
  query.py

Why It Matters

With Amazon S3 Metadata, AWS eliminates the complexity of custom metadata systems. Now, you can:
Enhance data discoverability for analytics and AI workloads.
Maintain a scalable and synchronized view of your S3 objects.
Simplify compliance and auditing with enriched metadata tracking.

Further Resources

📺 Watch the Video Overview: Amazon S3 Metadata at AWS re:Invent
🌐 Explore the Official Feature Page: Amazon S3 Metadata
📝 Read the AWS Blog Post by Jeff Barr: Introducing Queryable Object Metadata for Amazon S3 Buckets

Follow me on:
Linkedin.

Low Level Design - Logging Framework

Ahmed Adel — Fri, 17 May 2024 15:30:39 +0000

🛠️ Practicing Low-Level Design: Building a Logging Framework 📝

Looking to enhance your low-level design skills using design patterns? In this tutorial, we'll embark on a fascinating journey of building a logging framework from the ground up.
By leveraging the power of three key design patterns Singleton, Chain of Responsibility, and Observer we'll not only create a functional logging system but also gain valuable insights into the art of low-level design. So, roll up your sleeves, fire up your favorite IDE, and let's dive into this hands-on coding adventure! 💻

Let's start with gathering the requirements ...

1- 📌 Multiple Sync: The framework should support logging in multiple places, such as console, log file, database, and distributed queue.
2- 📌 Multiple Categories: The framework should support logging into multiple categories, such as info, debug, and error.
3- 📌 Configurability: The category and logging location should be configurable.

Here is the result code we are going to implement today

Ahmed-Adel3 / Low-Level-Design

Low-Level-Design

Welcome to the Low-level Design Examples repository! Here, you'll find a collection of practical examples demonstrating various low-level design concepts and patterns. These examples are aimed at helping you understand and implement robust, efficient, and maintainable software solutions.

Contribute

We welcome contributions to this repository! If you have any low-level design examples or improvements to existing examples, feel free to open a pull request. Let's collaborate to create a valuable resource for the software development community.

If you find these examples helpful, please consider starring this repository to show your support. Thank you for visiting!

View on GitHub

So our main components will be:

📚 Logger Class: The main class exposed to the application for writing logs.
📚 Categories: Info, Debug, and Error.
📚 Target: Console, File, and Database.

Logger class:

For this class, we will start with a creational design pattern to create the logger.
Here, we will choose Singleton design pattern to create just one instance from the logger all over our application

// Singleton pattern ensures that a class has only one instance and provides a global point of access to it.
internal class Logger
{
    // Private static instance of the Logger class.
    private static Logger _logger;

    // Private constructor to prevent instantiation of the Logger class from outside.
    private Logger()
    {
        // Check if an instance of Logger already exists, throw an exception if so.
        if (_logger != null)
            throw new InvalidOperationException("Object already created");
    }

    // Public static method to provide access to the single instance of the Logger class.
    public static Logger GetLogger()
    {
        // Check if _logger is null (not yet initialized).
        if (_logger == null)
        {
            // Use locking to ensure thread safety in multi-threaded environments.
            lock (typeof(Logger))
            {
                // Double-check if _logger is still null (another thread might have initialized it while waiting for the lock).
                if (_logger == null)
                {
                    // Create a new instance of Logger and assign it to _logger.
                    _logger = new Logger();
                }
            }
        }
        // Return the single instance of Logger.
        return _logger;
    }
}

So this implementation:

🚫 Prevent Multiple Instances: The private constructor prevents the creation of multiple instances of the Logger class, so it's the class's responsibility to create the object and return the same instance when requested using the GetLogger function.

Now let's move to the second section, categorization of the logs ...

For that we will use Chain of responsibility design pattern
Chain of Responsibility is a behavioral design pattern that lets you pass requests along a chain of handlers. Upon receiving a request, each handler decides either to process the request or to pass it to the next handler in the chain.

Let's now create a simple Enum to carry the values that will identify the log level ...

    internal enum LoggerLevel
    {
        Info = 1,
        Error = 2,
        Debug = 3
    }

Now, let's implement the Chain of Responsibility pattern for categories (Info, Debug, and Error) using the AbstractLogger class and its concrete subclasses.

namespace LoggingSystem
{
    // AbstractLogger is an abstract base class for other logging classes.
    internal abstract class AbstractLogger
    {
        // Levels is a protected field that holds the logging level.
        protected LoggerLevel Levels;

        // _nextLevelLogger is a private field that holds the next logger in the chain of responsibility.
        private AbstractLogger _nextLevelLogger;

        // SetNextLevelLogger is a public method that sets the next logger in the chain of responsibility.
        public void SetNextLevelLogger(AbstractLogger nextLevelLogger)
        {
            _nextLevelLogger = nextLevelLogger;
        }

        // LogMessage is a public method that logs a message.
        public void LogMessage(LoggerLevel level, string msg, LoggerTarget loggerTarget)
        {
            // If the current logger's level matches the provided level, it displays the message.
            if (Levels == level)
            {
                Display(msg, loggerTarget);
            }

            // Regardless of whether it displayed the message, it passes the message to the next logger in the chain (if there is one).
            if (_nextLevelLogger != null)
            {
                _nextLevelLogger.LogMessage(level, msg, loggerTarget);
            }
        }

        // Display is a protected abstract method that displays a message.
        // This method has no implementation in this class and must be implemented in any non-abstract subclass.
        protected abstract void Display(string msg, LoggerTarget loggerTarget);
    }
}

AbstractLogger Class:

🧩 Chain of Responsibility: AbstractLogger implements the Chain of Responsibility pattern, allowing different loggers to handle log messages based on their logging levels.

🔄 Next Logger: It maintains a private field _nextLevelLogger to hold the reference to the next logger in the chain, enabling the passing of log messages to the next logger if needed.

🔒 Levels Field: The Levels field holds the logging level for the current logger, determining which messages to handle.

📝 LogMessage Method: The LogMessage method checks if the provided logging level matches the logger's level. If it does, it displays the message; otherwise, it passes the message to the next logger in the chain.

🚧 Display Method: The Display method is a protected abstract method that must be implemented by subclasses to handle the actual display of log messages.

🔗 SetNextLevelLogger Method: This method allows setting the next logger in the chain, establishing the order in which loggers handle messages.

Now let's create the concrete classes for displaying the message...

For Info:

    internal class InfoLogger : AbstractLogger
    {
        public InfoLogger(LoggerLevel levels)
        {
            this.Levels = levels;
        }

        protected override void Display(string msg, LoggerTarget loggerTarget)
        {
            //temporary for now ...
            Console.WriteLine("INFO: " + msg);
        }
    }

For Error:

    internal class ErrorLogger : AbstractLogger
    {
        public ErrorLogger(LoggerLevel levels)
        {
            this.Levels = levels;
        }

        protected override void Display(string msg, LoggerTarget loggerTarget)
        {
            //temporary for now ...
            Console.WriteLine("ERROR: " + msg);
        }
    }

For Debug:

    internal class DebugLogger : AbstractLogger
    {
        public DebugLogger(LoggerLevel levels)
        {
            this.Levels = levels;
        }

        protected override void Display(string msg, LoggerTarget loggerTarget)
        {
            //temporary for now ...
            Console.WriteLine("DEBUG: " + msg);
        }
    }

Now, let's edit or logger class ...

we need to use the abstract logger to create loggers for various levels.
So we will instantiate a new instance of the abstract logger, let's name it _chainOfLogger in the GetLogger function.

internal class Logger
{
    private static Logger _logger;
    private static AbstractLogger _chainOfLogger;
    private Logger()
    {
        if (_logger != null)
            throw new InvalidOperationException("Object already created");
    }
    public static Logger GetLogger()
    {
        if (_logger == null)
        {
            lock (typeof(Logger))
            {
                if (_logger == null)
                {
                    _logger = new Logger();
                    _chainOfLogger = CreateChainOfLogger();
                    //IS THIS REALLY WHERE THE CreateChainOfLogger FUNCTION BELONGS??
                }
            }
        }
        return _logger;
    }
}

- We should make a slight enhancement here ... CreateChainOfLogger doesn't actually belong to the logger class, it is not it's responsibility to build the chain...

So we will create another class called LogManager to handle it ...

internal class LogManager
{
public static AbstractLogger DoChaining()
{
    // Create an InfoLogger with the Info level.
    AbstractLogger infoLogger = new InfoLogger(LoggerLevel.Info);

    // Create an ErrorLogger with the Error level.
    AbstractLogger errorLogger = new ErrorLogger(LoggerLevel.Error);

    // Create a DebugLogger with the Debug level.
    AbstractLogger debugLogger = new DebugLogger(LoggerLevel.Debug);

    // Set the next logger for the InfoLogger to be the ErrorLogger.
    infoLogger.SetNextLevelLogger(errorLogger);

    // Set the next logger for the ErrorLogger to be the DebugLogger.
    errorLogger.SetNextLevelLogger(debugLogger);

    // Return the first logger in the chain (InfoLogger).
    return infoLogger;
}

}

🔗 Logger Chain Setup: The DoChaining method sets up a chain of responsibility for loggers, where each logger handles messages based on its logging level.

🔄 Chain Order: It creates three loggers (InfoLogger, ErrorLogger, and DebugLogger) with increasing logging levels (Info < Error < Debug) and sets the next logger in the chain accordingly.

🧩 Chain Completion: The InfoLogger is set to pass messages to the ErrorLogger, which in turn passes messages to the DebugLogger. This completes the chain, ensuring that log messages are handled appropriately based on their levels.

Then we will use this function to create the chain inside the Logger class.
After that, we will implement a function to create the log itself, it a general function accepting the message and the level, it will also be a private function, it will only be called from inside the logger class by three functions for each log level...
So the Logger class will be like below ...

internal class Logger
    {
        private static Logger _logger;
        private static AbstractLogger _chainOfLogger;

        private Logger()
        {
            if (_logger != null)
                throw new InvalidOperationException("Object already created");
        }

        public static Logger GetLogger()
        {
            if (_logger == null)
            {
                lock (typeof(Logger))
                {
                    if (_logger == null)
                    {
                        _logger = new Logger();
                        _chainOfLogger = LogManager.DoChaining();
                    }
                }
            }
            return _logger;
        }

        public void Info(string message)
        {
            CreateLog(LoggerLevel.Info, message);
        }

        public void Error(string message)
        {
            CreateLog(LoggerLevel.Error, message);
        }

        public void Debug(string message)
        {
            CreateLog(LoggerLevel.Debug, message);
        }

        // The CreateLog method is a private method that creates a log with the provided level and message.
        private void CreateLog(LoggerLevel level, string message)
        {
            // It calls the LogMessage method on the chain of loggers with the provided level, message, and logger target.
            _chainOfLogger.LogMessage(level, message, _loggerTarget);
        }
    }

Let's test our application till now, let's create a simple console app to test it ...

using LoggingSystem;

Logger logger = Logger.GetLogger();

logger.Info("This is info message");
Console.WriteLine("___");
logger.Error("This is Error message");
Console.WriteLine("___");
logger.Debug("This is Debug message");

and after running it, we will have the result as below

As you can see:
1- Info message prints only log
2- Error message prints info and error
3- Debug message prints info, error, and debug.

Now let's move to the last part of our implementation, which is the log target, here we will be using the Observer design pattern

_Observer _ is a behavioral design pattern that lets you define a subscription mechanism to notify multiple objects about any events that happen to the object they’re observing.

So, we will have two parts here ...
1- The target, which will be the changing part, where the log target is changing.
2- The constant part, which is the observer.

So Lt's create the 2 classes, first we will write an interface for all Log observers (console, file, database, ..etc).
This class will have only the log function, to force all its implementations to implement it.

    internal interface ILogObserver
    {
        void Log (string message);
    }

Now let's assume we will have console, database and file loggers ...
And let's ignore the implementation of the logging mechanism inside each of the logging targets for now
Console Logger

    internal class ConsoleLogger : ILogObserver
    {
        public void Log(string message)
        {
            Console.WriteLine("Writing to Console " + message);
        }
    }

File Logger

    internal class FileLogger : ILogObserver
    {
        public void Log(string message)
        {
            Console.WriteLine("Writing to File " + message);
        }
    }

Database Logger

    internal class DatabaseLogger : ILogObserver
    {
        public void Log(string message)
        {
            Console.WriteLine("Writing to Database" + message);
        }
    }

Now let's implement the observer class which will contain:

1- A list of observers.
2- A function to add a new observer.
3- A second method to remove an observer.
4- A third method to notify all observers.

    internal class LoggerTarget
    {
        private readonly Dictionary<LoggerLevel, List<ILogObserver>> _logObservers = new Dictionary<LoggerLevel, List<ILogObserver>>();

        public void AddObserver(LoggerLevel level, ILogObserver logObserver)
        {
            if (!_logObservers.ContainsKey(level))
            {
                _logObservers[level] = new List<ILogObserver>();
            }
            _logObservers[level].Add(logObserver);
        }

        public void RemoveObserver(ILogObserver logObserver)
        {
            foreach (var logObservers in _logObservers.Values)
            {
                logObservers.Remove(logObserver);
            }
        }

        public void NotifyAllObservers(LoggerLevel level, string message)
        {
            if (_logObservers.ContainsKey(level))
            {
                foreach (var logObserver in _logObservers[level])
                {
                    logObserver.Log(message);
                }
            }
        }
    }

Now, where will we notify our observers ?🤔

The Logger class of course...
Same as what we did with the chain of logger, we will instantiate Log target in the logger class in the GetLogger function using the Logger manager again.
Also the LoggerTarget needs to be passed to the LogMessage and Display function.

internal class Logger
{
    private static Logger _logger;
    private static AbstractLogger _chainOfLogger;
    private static LoggerTarget _loggerTarget;

    private Logger()
    {
        if (_logger != null)
            throw new InvalidOperationException("Object already created");
    }

    public static Logger GetLogger()
    {
        if (_logger == null)
        {
            lock (typeof(Logger))
            {
                if (_logger == null)
                {
                    _logger = new Logger();
                    _chainOfLogger = LogManager.DoChaining();
                    _loggerTarget = LogManager.AddObservers();
                }
            }
        }
        return _logger;
    }

    public void Info(string message)
    {
        CreateLog(LoggerLevel.Info, message);
    }

    public void Error(string message)
    {
        CreateLog(LoggerLevel.Error, message);
    }

    public void Debug(string message)
    {
        CreateLog(LoggerLevel.Debug, message);
    }

    private void CreateLog(LoggerLevel level, string message)
    {
        _chainOfLogger.LogMessage(level, message, _loggerTarget);
    }
}

And the LoggerManager will implement the new AddObservers function as below

internal class LogManager
{
    public static AbstractLogger DoChaining()
    {
        AbstractLogger infoLogger = new InfoLogger(LoggerLevel.Info);
        AbstractLogger errorLogger = new ErrorLogger(LoggerLevel.Error);
        AbstractLogger debugLogger = new DebugLogger(LoggerLevel.Debug);

        infoLogger.SetNextLevelLogger(errorLogger);
        errorLogger.SetNextLevelLogger(debugLogger);

        return infoLogger;
    }

        // AddObservers is a static method that sets up observers for different logging levels.
        public static LoggerTarget AddObservers()
        {
            // Create a new LoggerTarget. This is the target that the observers will be observing.
            LoggerTarget loggerTarget = new LoggerTarget();

            // Create a new ConsoleLogger. This logger will log messages to the console.
            ConsoleLogger consoleLogger = new ConsoleLogger();

            // Add the ConsoleLogger as an observer for Info level logs.
            loggerTarget.AddObserver(LoggerLevel.Info, consoleLogger);

            // Add the ConsoleLogger as an observer for Error level logs.
            loggerTarget.AddObserver(LoggerLevel.Error, consoleLogger);

            // Add the ConsoleLogger as an observer for Debug level logs.
            loggerTarget.AddObserver(LoggerLevel.Debug, consoleLogger);

            // Create a new FileLogger. This logger will log messages to a file.
            FileLogger fileLogger = new FileLogger();

            // Add the FileLogger as an observer for Error level logs.
            loggerTarget.AddObserver(LoggerLevel.Error, fileLogger);

            // Return the LoggerTarget with the observers added.
            return loggerTarget;
        }
}

👁️ Observer Setup: The AddObservers method sets up observers for different logging levels using the Observer design pattern. Observers watch for changes in the target (the LoggerTarget) and react accordingly.

📚 LoggerTarget Creation: It creates a new LoggerTarget, which serves as the target that observers will be observing. The LoggerTarget manages the list of observers and notifies them of any changes.

🖥️ ConsoleLogger Setup: The method creates a ConsoleLogger, which logs messages to the console. It then adds this ConsoleLogger as an observer for Info, Error, and Debug level logs.

📁 FileLogger Setup: Additionally, the method creates a FileLogger, which logs messages to a file. It adds this FileLogger as an observer specifically for Error level logs.

🔍 Observer Registration: Observers are registered with the LoggerTarget using the AddObserver method, specifying the logging level they are interested in and the corresponding logger to handle messages at that level.

📝 Return: Finally, the method returns the LoggerTarget with all the observers added, ready to observe and react to changes in logging levels.

Notice that in the Logger class, LoggerTarget is now sent to the LogMessage function, it will be passed to the display function for each type of logger (Debug, info, or error) to notify its observers.

So, let's edit the Abstract Logger too, to be like below

internal abstract class AbstractLogger
{
    protected LoggerLevel Levels;

    private AbstractLogger _nextLevelLogger;

    public void SetNextLevelLogger(AbstractLogger nextLevelLogger)
    {
        _nextLevelLogger = nextLevelLogger;
    }

    public void LogMessage(LoggerLevel level, string msg, LoggerTarget loggerTarget)
    {
        if (Levels <= level)
        {
            Display(msg, loggerTarget);
        }

        if (_nextLevelLogger != null)
        {
            _nextLevelLogger.LogMessage(level, msg, loggerTarget);
        }
    }

    protected abstract void Display(string msg, LoggerTarget loggerTarget);
}

Now let's edit all implementations of the Display function..
Info Logger

internal class InfoLogger : AbstractLogger
{
    public InfoLogger(LoggerLevel levels)
    {
        this.Levels = levels;
    }

    protected override void Display(string msg, LoggerTarget loggerTarget)
    {
        //replace console log with notify observer 
        loggerTarget.NotifyAllObservers(LoggerLevel.Info, "INFO: " + msg);
    }
}

Error Logger

internal class ErrorLogger : AbstractLogger
{
    public ErrorLogger(LoggerLevel levels)
    {
        this.Levels = levels;
    }

    protected override void Display(string msg, LoggerTarget loggerTarget)
    {
        //replace console log with notify observer
        loggerTarget.NotifyAllObservers(LoggerLevel.Error, "ERROR: " + msg);
    }
}

Debug Logger

    internal class DebugLogger : AbstractLogger
    {
        public DebugLogger(LoggerLevel level)
        {
            this.Levels = level;
        }

        protected override void Display (string msg, LoggerTarget loggerTarget)
        {
             //replace console log with notify observer
             loggerTarget.NotifyAllObservers(LoggerLevel.Debug, "DEBUG: " + msg);
        }
    }

let's run the solution now and check the result ...

And it's working as expected, because according to our configuration, we are

printing info message to console
printing error message in console and file (and I will be printed in error and info levels).
printing debug message in console and file (and I will be printed in error,info and debug levels).

and we can easily change this behavior...

Let's assume we only need to print in the same level as instructed, not the level and the levels below it, then in the LogMessage function, we can easily edit it like below

        public void LogMessage(LoggerLevel level, string msg, LoggerTarget loggerTarget)
        {
            //old: if (Levels <= level)
            if (Levels == level)
            {
                Display(msg, loggerTarget);
            }
         //The rest of the code of Abstract logger

Now the result will be

So as instructed in the LoggerManager class :

1- Console Logger will have info, debug and error.
2- File logger will have only errors.

Of course this LoggerManager in a real logger system will read from a config file to give you the flexibility to change logging levels and targets.

Wrapping Up: Embracing the Power of Design Patterns in Logging

In this tutorial, we've explored the intricate world of low-level design by building a logging framework from scratch. We've delved into the Singleton pattern, ensuring our logging class has only one instance, and the Chain of Responsibility pattern, allowing different loggers to handle log messages based on their levels. We've also embraced the Observer pattern, setting up observers to react to changes in logging levels.

By understanding and implementing these design patterns, we've not only created a functional logging system but also gained valuable insights into the art of designing flexible, extensible, and maintainable software solutions. We hope this tutorial has inspired you to apply these principles in your own projects and continue to explore the vast landscape of software design patterns. Happy coding!

Follow me on:
Linkedin.

Mountpoint for Amazon S3 is now generally available and ready for production workloads 🚀

Ahmed Adel — Wed, 09 Aug 2023 20:25:50 +0000

AWS Mountpoint: A High-Performance File Client for Amazon S3

Today, AWS announces the general availability of Mountpoint for Amazon S3.

➥ AWS Mountpoint is an open source file client that makes it easy for Linux-based applications to connect directly to Amazon Simple Storage Service (Amazon S3) buckets and access objects using file APIs. It is ideal for large-scale analytics applications that read and generate large amounts of S3 data in parallel, but don't require the ability to write to the middle of existing objects.

➥ It automatically translates local file system API calls to REST API calls on S3 objects. Mountpoint for Amazon S3 is optimized for high-throughput performance. It builds on the AWS Common Runtime (CRT) library, which is purpose-built for high performance and low-resource usage to make efficient use of your compute resources.

It is a powerful tool that can be used to accelerate a variety of workloads, including:

✏️ Data lakes.
✏️ Machine learning training.
✏️ Image rendering.
✏️ Autonomous vehicle simulation.
✏️ Extract, transform, and load (ETL) processes.

➽ Here are some of the key benefits of AWS Mountpoint:

✅ High throughput:

AWS Mountpoint can achieve up to 100 GB/s of throughput on a single EC2 instance, and can scale to support even higher throughput requirements.

✅ Low latency:

AWS Mountpoint has very low latency, making it ideal for latency-sensitive applications.

✅ Elasticity:

AWS Mountpoint is fully elastic, so you can scale up or down your throughput requirements as needed.

✅ Cost-effectiveness:

AWS Mountpoint is a cost-effective way to access your data in Amazon S3. You only pay for the storage and bandwidth that you use.

Watch Devabrat Kumar - Senior Technical product leader at AWS talks about it :

Getting started:

➥ Run these two commands to install Mountpoint for Amazon S3 on your Amazon Linux EC2 instance (for Graviton instances, replace x86_64 with arm64 in the URL):

wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
sudo yum install -y ./mount-s3.rpm

On Ubuntu, use these commands instead (for Graviton instances, replace x86_64 with arm64 in the URL):

wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
sudo apt-get install -y ./mount-s3.deb

See Installing Mountpoint for Amazon S3 for detailed instructions and other installation options (including Docker or building from source).

➥ Once you've got Mountpoint for Amazon S3 installed, you can mount your Amazon S3 bucket. You'll need valid AWS credentials to access your bucket; for example, Mountpoint will automatically use credentials from an IAM role associated with your EC2 instance, or you can use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

➥ To mount your bucket, run this command, replacing DOC-EXAMPLE-BUCKET with the name of your bucket and /path/to/mount with the directory you want to mount the bucket to:

mount-s3 DOC-EXAMPLE-BUCKET /path/to/mount

Now you can work with your bucket contents as if they were a local file system:

ls /path/to/mount
echo "Hello World!" > /path/to/mount/Data.txt
cat /path/to/mount/Data.txt

When you're finished accessing your bucket, you can unmount it (you might need sudo):

umount /path/to/mount

See Configuring Mountpoint for Amazon S3 for more details on how to configure and use Mountpoint, including options for providing AWS credentials.

To learn more about AWS Mountpoint, please visit the following resources:

🌐 AWS Mountpoint documentation
🌐 AWS Mountpoint GitHub repository
🌐 Mountpoint for Amazon S3
by Jeff Barr - Vice President & Chief Evangelist of Amazon Web Services
🌐 Mountpoint service page

I hope this article was helpful! If you have any questions, please feel free to leave a comment below.

Follow me on:
Linkedin.