Forem: AWS MENA Community

Why we use AWS cloud services

mustafa mahdi saleh — Wed, 30 Mar 2022 08:41:15 +0000

Why we use AWS cloud

لماذا نستخدم AWS

this article is in Arabic and for whom beginners in cloud

مثال عن مشكلة و حل

كان لدينا مشكلة في دائرتنا تتلخص المشكلة بعطل بورد احد أجهزة الخوادم الرئيسية الحديثة المشترات قبل 5 سنوات , وعدم مقدرة كوادرنا على صيانة العطل وعدم وجود جهاز خادم احتياطي ,
وعند مراجعة مركز الصيانة الرئيسي الواقع في محافظة أخرى تبعد 100 كم اخبرونا بعد عدم وجود قطع غيار للمنتج المذكور لأنه يعتبر قديم من ناحية السوق وهناك الكثير من النماذج صنعت بعده !
والحل يكون بشراء جهاز خادم اخر , علما ان العمل كان متوقف ل 4 اشهر تقريبا بسبب تشخيص الصيانة و عدم وجود سيولة مالية للدائرة لشراء جهاز خادم جديد في حينها ,

وتم الانتظار الى ان تم توفير مبلغ لشراء سيرفر مقارب لمواصفات جهاز السيرفر العاطل و توقف العمل وحصل تأخير كبير .

والحل الاسلم والاسرع والارخص الذي لم نعرف عنه في وقتها هم الخادم السحابي cloud server

فمشاكل وجود مراكز بيانات هي :-

توفير كهرباء مستقرة و مستمرة .
توفير تبريد ممتاز و خصوصا في فصل الصيف .
توفير تخصيص مالي كافي لشراء أجهزة خوادم و راك و مجهز قدرة وأجهزة احتياطية و موزع شبكة و شبكة انترانت وغيرها .
توفير شبكة داخلية للحواسيب .
توفير غرفة مناسبة لمراكز البيانات .
في الكثير من الأيام نقوم بإطفاء الخوادم يدويا مع انتهاء الدوام الرسمي بسبب عدم الحاجة او عدم وجود الطاقة الكهربائية وهذا الشي غير صحيح لأنه يقلل من العمر الافتراضي للخادم.

بينما في الحوسبة السحابية تكون الأمور مرنة جدا

تكون الأمور مرنة جدا بالإمكان انشاء خوادم وبالمواصفات المطلوبة وقت قياسي .
بالإمكان إيقاف تشغيل الخوادم و جدولتها بسهولة شديدة وعبر وقت قصير جدا مما يؤدي الى تقليل في التكلفة فقط تدفع مقابل ما تستخدمه وبالوقت المحدد .
بالإمكان تطوير مواصفات الخوادم (كزيادة او تقليل السعة الخزنية او زيادة المعالج او الذاكرة العشوائية او تغيير أنواعها ) .
في حالة وجود عطل في خادم سحابي بالإمكان تغييره عن طريق انشاء خادم غيره بسهولة وسرعة .
بالإمكان الاستفادة من قواعد البيانات التي توفرها شركات الحوسبة السحابية.
في حالة الخوف على البيانات من الانترنت بالإمكان وضعها في خادم محلي وربطها بال خادم السحابي مع وجود التطبيق على الخادم السحابي .
فقط نحتاج ان نوصل المستخدمين بخدمة انترنت جيدة .
وبالإمكان الاتصال بالخادم من أي مكان مع ضمان تحديد ال IP و عمل مفتاح و كلمة سر للمستخدمين.
وهناك مجالات كثيرة أخرى أيضا كال الذكاء الصناعي و تعلم الالة و انترنت الأشياء والتخزين وغيرها الكثير .

تقوم شركة AWS AMAZON WEB SERVICES وهي شركة رائدة في مجال الحوسبة السحابية بتوفير اشتراك مجاني لمدة سنة واحدة لاستخدام اغلب إمكانيات الحوسبة السحابية

وهناك الكثير من الشهادات التي توفرها الشركة للمستويات الابتدائية والمتقدمة مع الكثير من التدريبات المجانية والكثير من الدعم في اغلب مجالات الحوسبة

english version

https://dev.to/aws-builders/why-we-use-aws-cloud-2hij

AWS DevOps Monitoring Dashboard | AWS Whitepaper Summary

Dorra ELBoukari — Mon, 20 Dec 2021 14:47:51 +0000

Unlike the other AWS Whitepapers for which I wrote summaries , this content builds on an AWS DevOps Monitoring dashboard Architecture Diagram published on August 09 ,2021. We will go through all the details and we will shed the light on several facts,so I can give you ,dear reader, a complete grasp of the architecture under discussion.

I. Architecture Description:

The suggested approach is built to set up a DevOps reporting tool on the AWS cloud infrastructure by using AWS native tools. It automates the process of ingesting, analyzing, and visualizing continuous integration/continuous delivery (CI/CD) metrics.

I.1 Use Case of the discussed architecture:

If you are engaged in building sophisticated applications on an AWS infrastructure using AWS native DevOps tools ,you must be going through lots of deployments .Thus, you need to consider a solution that helps you to visualize, track and analyze your deployments through their DevOps lifecycle .

I.2 AWS CI/CD pipeline

On the left side of the architecture , you can remark the box that presents Customer AWS CI/CD Pipeline . It is crystal clear that the customer has implemented a full Devops CI/CD pipeline with AWS dedicated DevOps Tools.

I.3 AWS DevOps Tools

AWS DevOps Tools comprise a collection of services that are designed to work together so we can safely store and manage our application's source code, as well as automatically create,test and deploy applications to AWS or on-premises environment. In other words AWS DevOps Tools work in harmony to cover the entire software development lifecycle starting from code reviews to deployment and monitoring.

1. AWS CodePipeline:

Is commonly viewed as the equivalent of Jenkins (a CI/CD tool that can integrate with many cloud providers) but Code Pipeline is specific to AWS.It is a fully managed, PAYGO(pay-as-you-go) , continuous delivery solution that automates your release pipeline's build, test, and deploy phases for a fast and dependable application.

2. AWS CodeCommit:

As it name implies ( From Cambridge Dictionary Commit (verb) :To actively put information in your memory or write it down) , Code Commit is a managed source control service that hosts private Git repositories where contributors write down their code instantly . It is designed to be safe , highly scalable and to enable teams to collaborate on code in a secure manner. The contributions are encrypted in transit and at rest. There is no need to worry about scalability as well as the management of the source control system .

3. AWS CodeBuild:

Is an AWS service designed to build your software packages that are ready to deploy. It is a fully managed and a scalable continuous integration service that compiles source code, runs tests, and then produces software packages. CodeBuild processes multiple builds concurrently, so your builds are not left waiting in a queue.

4. AWS CodeDeploy:

Is a fully managed and scalable deployment service used to automate software deployments while eliminating the need for error-prone manual operations. Deployment can be applied to a variety of AWS compute services (Amazon EC2, AWS Fargate, AWS Lambda,on-premises servers)

II. Walking Through The DevOps Monitoring Dashboard Architecture

In this paragraph ,we will use the same architecture provided by AWS along with the same reference numbers. But since it looks crowded,we will " divide to rule ". In other terms, we will break the architecture down into smaller sections for a clearer understanding and a better visualization .
To have a Monitoring dashboard in general, we need to go through a process that contain four main steps :

1. Tracking Event Sources

2. Gathering Data

3. Analyzing Gathered Information

4. Visualizing Analysis Results

Our architecture, obeys to the same philosophy of the Four-Step process . This is the reason why we will divide it into two main sections :

Section1: Tracking and Gathering Data (Figure a and Figure b )

Section2: Analyzing data and Visualizing Analysis Results ( Figure c )

The Figure below explains the process and mentions which AWS Services contribute to the success of the required task at each step.

PS: You can deploy this solution using the available AWS CloudFormation template (Infrastructure As A Code)

II.1 Section 1 : Tracking and Gathering Data

In this paragraph, we mainly focus on the sections responsible for information gathering.
While tracking an application deployed with AWS DevOps tools, we have two main parts to shed the light on:

1. CODE BUILD :
which announces if the build SUCCEEDED or FAILED. (This will be explained in SECTION 1.0).

2. DEPLOYMENT :

After having a successful code build, we have to know whether deployment is SUCCESSFUL or NOT. In case if failure ,we need to see the logs to figure out the cause of this issue.(This is detailed in SECTION 1.1)

Through those two major parts , we will need an appropriate AWS service that will alert us about success or failure events, and the data related to them.In fact, once a contributor (from the development team) initiates an activity in AWS CI/CD Pipeline,his deeds need to be detected so we can visualize it on our DevOps Monitoring Dashboard. The convenient service for this task is AWS CloudWatch.

What is AWS CloudWatch ?
It is a monitoring and observability AWS service . It is also a metric repository that will provide data and actionable insights in form of events . For more understanding here is an AWS CloudWatch use case:
You want to be notified with an SMS (perform an action) once the CPU Utilization of an instance exceeds 60% (Condition on metric is fullfilled). In this scenario , you need to use AWS CloudWatch to watch for the metric CPU utilization until it exceeds the threshold 60%. Once this happens ,it is called an event that will initiate an action .The event will trigger AWS SNS Service which alerts you immediately with an SMS.

AWS CloudWatch is used to:
-Continously Stream important Data, in real-time, through Amazon Kinesis Data Firehose to Amazon S3 buckets. (Discussed in Section 1.0)
-Send data ,in near real-time,to Amazon EventBridge, through Amazon Kinesis Data Firehose and finally to Amazon S3 buckets.(Discussed in Section 1.1)

II.1.0 Section 1.0

As we have already mentioned, AWS CloudWatch continuously Streams data events related to code source compilations, tests, and software packaging produced by AWS CodeBuild to Amazon Kinesis Data Firehose -View Step 1 and Step 2 in the Figure (a)-. The delivery is in near real-time and with low latency.
At this Point ,Amazon Kinesis Data Firehose will perform an ETL operation (Extract, Transform, Load) where a Lambda function undergoes the Extraction and the Transformation .
In fact, each time AWS CloudFormation detects data in real-time ,The Lambda function is activated for few seconds to extracts the relevant data for each metric and to transform it into the convenient format.
Finally, Amazon Kinesis Data Firehose loads the data in real-time to the Amazon S3 data lake for downstream processing. View Step 4 in the Figure (a).

II.1.1 Section 1.1

In this second portion of our architecture, we can simply see how to detect any action performed by the Developement Team on AWS CodeCommit, AWS CodeDeploy and AWS CodePipeline.View Figure (b).
Once a developer commits a code to AWS CodeCommit and deploys his application with AWS CodeDeploy,the actions on the predefined AWS Event Sources are detected by Amazon CloudWatch as events and then transfered to Amazon EventBridge (Step 1).
Amazon CloudWatch alarms also monitor the status of an Amazon CloudWatch synthetics canary service (Step 3). Canaries are customizable scripts that monitor endpoints and APIs on a regular basis. Even if you don't have any user activity on your applications, they actually conduct the same behaviors as a customer to assist you check your customer experience. You can detect problems before your consumers do by utilizing canaries.
PS: Step 2 and 4 are the same as the previous section.

What is Amazon EventBridge ?
It is a serverless event bus service that is used to capture events from an Amazon CloudWatch alarm .We can assume that Amazon EventBridge is the successor of AWS CloudWatch .In fact, it was formely called Amazon CloudWatch Events because it uses the same CloudWatch Events API . It is similar to Amazon CloudWatch but with additional features.

III.1 Section 2: Analyzing data and Visualizing Analysis Results

In the second paragraph , we will go the final steps in the architecture. At this point , we have all the information gathered in an Amazon S3 bucket ,in the appropriate format for analysis (thanks to AWS Lambda Function).Now ,we have two more steps to go: analysis and visualization.

1. Analysis:

A detailed examination of the gathered information is performed by AWS Athena. It is a serverless AWS tool that enables performing interactive queries and data analysis on the big sets of data stored in Amazon S3 while using standard SQL. Analysis results will be mostly delivered within few seconds.

2. Visualization:

To extract easy-to-understand insights, nothing is more adequat that AWS QuickSight. It provides an interactive and visual dashboard while ensuring scalability. In this perspective, AWS QuickSight is our window to deeper DevOps insights. Furthermore, thanks to Amazon QuickSight Q, Management team members can ask questions about DevOps data in natural language ( plain English) , and receive accurate responses with relevent visualizations which make insights clearer.
Aws QuickSight is an important asset for Management team members. In fact ,it describes DevOps insight in a very simple manner . Every one in management team should be able to understant the insights , even if he lacks data science and DevOps experience.

Serverless مقدمة عن

Mohammed Ismaeel — Tue, 07 Dec 2021 13:03:07 +0000

Serverless ما هو ال

مجموعة من الخدمات السحابية التي تتيحُ لك بناء وتشغيل التطبيقات والخدمات دون الحاجة إلى التفكير بالسيرفير

الحوسبة بدون خوادم او سيرفرات تتيح للمطور بناء وتشغيل التطبيقات والخدمات دون الحاجة الى التفكير بالخوادم. ان خدمات السيرفيرلس لا تتطلب منك تجهيز او ادارة او توسيع البنية التحتية.

يمكن استخدام خدمات السيرفيرلس تقريبا في جميع المجالات كاتطوير المواقع او تطوير تطبيقات الموبايل او في الذكاء الصناعي

تعتبر خدمات السيرفرلس اقتصاديه جدا فانت كمظور لا تدفع للخدمه اذا كانت في الوضع الخامل انت تدفع فقط في حالة استعمال الخدمة وهذا شي رائع بالنسبة للشركات الصغيرة او الناشئة او في حال كنت تريد ان تجرب شي جديد

من الجدير بالذكر ان جميع التحديثات الامنية وتوفر الخدمة على مدار الساعة هي مسؤولية مزود الخدمة. ان تطوير او بناء خدمات بتقنية السيرفرلس تساعد الموطورين على التركيز على المنتج النهائي وهذا يؤدي الى سرعة الوصول الى السوق والمستخدم

Serverless services ما هي

تنقسم الى ثلاثة اقسام كما مبين ادناه

1. Compute

AWS Lambda
AWS Fargate

2. Integration

EventBridge
Step Function
SQS
SNS
API Gateway
AppSync

3. Data Store

S3
DynamoDB
RDS Proxy
Aurora Serverless

Serverless ما هي افضل استخدمات ال

معالجة الملفات
معالجة البيانات
ألشبكات
انترنت الاشياء
تطبيقات الويب والمحمول
تطبيقات قائمة على الأحداث Event Driven Application

فيديو تعليمي

youtube 1
youtube 2

Amazon Machine Learning| ML Key Concepts

‪Kareem Negm‬‏ — Wed, 24 Nov 2021 14:56:34 +0000

Amazon Machine Learning Key Concepts

Data sources

Term	Definition
Attribute	A unique, named property within an observation. In tabular-formatted data such as spreadsheets or CSV files
Datasource Name	A unique name for a dataset
Input Data	Collective name for all the observations that are referred to by a datasource.
Location	Amazon ML can use data that is stored within Amazon S3 buckets, Amazon Redshift databases, or MySQL databases in Amazon Relational Database Service (RDS)
Observation	A single data point that is part of a datasource
Schema	The information needed to interpret the input data, including attribute names and their assigned data types, and names of special attributes.
Statistics	Summary statistics for each attribute in the input data
Status	Indicates the current state of the datasource, such as In Progress, Completed, or Failed.
Target Attribute	the target attribute is the attribute whose value will be predicted by a trained ML model

ML Models

Term	Definition
Regression	ML model to predict a numeric value
Multiclass	ML model to predict values that belong to a limited, pre-defined set of permissible values.
Binary	ML model to predict values that can only have one of two state
Model Size	ML models capture and store patterns. The more patterns a ML model stores, the bigger it will be. ML model size is described in Mbytes.
Number of Passes	he number of times that you let Amazon ML use the same data records is called the number of passes.
Regularization	Regularization is a machine learning technique that you can use to obtain higher-quality models

Evaluations

Term	Definition
Model Insights	Amazon ML provides you with a metric to evaluate the predictive performance of your model.
Precision	the number of positive class predictions that actually belong to the positive class.
Recall	the number of positive class predictions made out of all positive examples in the dataset.
AUC	Area Under the ROC Curve (AUC) measures the ability of a binary ML model to predict a higher score for positive examples as compared to negative examples
Accuracy	Accuracy measures the percentage of correct predictions.
F1-score	The macro-averaged F1-score is used to evaluate the predictive performance of multiclass ML models.
RMSE	The Root Mean Square Error (RMSE) is a metric used to evaluate the predictive performance of regression ML models.
Cut-off	The cut-off is the threshold that you use to determine whether a predicted value is correct or not.

Batch Predictions

Term	Definition
Output Location	The results of a batch prediction are stored in an S3 bucket output location.
Manifest File	This file relates each input data file with its associated batch prediction results. It is stored in the S3 bucket output location.

Real-time Predictions

Real-time predictions are for applications with a low latency requirement, such as interactive web, mobile, or desktop applications.

Term	Definition
Real-time Prediction API	The Real-time Prediction API accepts a single input observation in the request payload and returns the prediction in the response.
Real-time Prediction Endpoint	To use an ML model with the real-time prediction API, you need to create a real-time prediction endpoint. Once created, the endpoint contains the URL that you can use to request real-time predictions.

AWS WhitePaper Summary

Serverless Architectures with AWS Lambda Summary | AWS Whitepaper Summary

Dorra ELBoukari — Mon, 15 Nov 2021 20:17:33 +0000

AWS Lambda is a serverless computing service launched in 2014 .It brought to existence a new architecture paradigm that doesn't rely on servers.AWS Lambda has also enabled a faster development speed and experimentation comparing to server-based architectures.
This paper summerizes the AWS Whitepaper entitled 'Serverless Architectures with AWS Lambda' released in 2017 to shed the light on Lambda serverless compute concepts .We focus mainly on the compute layer of the serverless applications where the code is executed, as well as the AWS developer tools and services used for best practices.

What Is Serverless?

Serverless literally means: "without need to provision or manage any servers". A serverless platform is responsible for:
1.Server Management:
Provisioning, installing and patching software,OS patching,etc.
2.Flexible scaling:
Applications are scaled automatically or by adjusting the capacity through the units of consumption(throughput,memory,etc)
3.High Availability:
Serverless applications have built-in availability and fault tolerance.
4.No idle capacity:
There is no charge when your code isn’t running

Here is a list of AWS different services that can used in serverless application:

AWS Lambda: Compute
Amazon API Gateway: APIs
S3 (Amazon Simple Storage Service): Storage
Amazon DynamoDB : Databases
Amazon SNS (Simple Notification Service) and Amazon SQS (Simple Queue Service): Interprocess messaging
AWS Step Functions and Amazon CloudWatch Events: Orchestration
Amazon Kinesis: Analytics

PS: Here is an additional list of AWS Serverless services that were not mentioned in the original document.

-AWS Redshift Spectrum: Analytics and interactive querying on S3
-Amazon Athena: interactive querying on S3 with read-on-schema technology
-Amazon Aurora Serverless: Databases
-AWS Fargate: Containerization
-Amazon QuickSight: Analytics and Visualization
-Amazon Cognito: Authentication ,authorization and user management
-AWS KMS: Key Management
-AWS Glue: ETL tool (Extract Transform and Load)
-Amazon EventBridge: Builds an event-driven architecture.
-Amazon AppSync: Create,publish and monitor secure GraphQL APIs and Subscriptions

AWS Lambda -the Basics

Lambda is a high-scale, provision-free serverless compute service . It is a (FaaS) Function-as-a-Service which scales precisely with the size of the workload .It runs more copies of the function in parallel in case of multiple simultaneous events to scale your code with high availability.AWS Lambda enables building reactive events as it works when triggered by an event .This reduces server's idle time and wasted capacity.
Each Lambda function contains:
-The Function Code that you want to execute
-The Function Configuration that defines how your code is executed.
-(Optional) The Event Sources that detect events and invoke you function as they occur. For example, an API Gateway(Event source) invokes a Lambda function when an API method created with API Gateway receives an HTTPS request.
You don’t need to write any code to integrate an event source with your Lambda function ,to manage infrastructure or scaling.Once you configure an event source for your function, your code is invoked when the event occurs.
Also ,it’s a natural fit to build microservices using Lambda functions thanks to the inherent decoupling that is enforced in serverless applications through integrating Lambda functions and event sources.

AWS Lambda- Diving Deeper

This section provides a further explanation of AWS Lambda's components mentioned above .

Lambda Function Code

It is the code you will run with AWS Lambda.AWS Lambda natively supports Java, Go, PowerShell, Node. js, C#, Python,PHP,SmallTAlk and Ruby code.It also supports libraries,artifacts and compiled native binaries.
AWS SAM Local: A set of tools used to compile and test the components you plan to run inside of Lambda within a matching environment .

The Function Code Package:

The Function Code Package contains all of the assets you want to have available locally upon execution of your code (additional files, classes, and libraries to be imported, binaries to be executed, or configuration files that your code might reference upon invocation.).
While creating the Lambda function (through the AWS Management Console or CreateFunction API) and even while publishing an updated code to existing Lambda functions (through UpdateFunctionCode API),you can upload the code package directly or you can refer to the S3 bucket and object key where the package is uploaded.
At minimum,the Function Code Package includes the code function to be executed when your function is invoked.

The Handler:

The handler is the code method (Java,C#) or the function (Node.js,Python) that is in your function code that processes events.It is the first thing that executes when a Lambda function is invoked.

The Event Object:

The Event Object is one of the parameters provided to the handler function.It includes all of the data and metadata that Lambda function needs.the event object differs in structure and contents, depending on which event source created it.

The Context Object:

The context object allows your function code to interact with the Lambda execution environment. It's contents and structure varies depending on the language runtime used by Lambda function.But at minimum it will contain:

AWS RequestId :Used to track specific invocations of a Lambda function(important for error reporting)
Remaining time :The amount of time in milliseconds that remain before your function timeout occurs (maximum 300 seconds)
Logging :The information about which CloudWatch Logs stream your log statements will be sent to.

Writing Code for AWS Lambda—Statelessness and Reuse

While writing your code for Lambda ,it’s important to understand that your code cannot make assumptions that state will be preserved from one invocation to the next.
However, each time a function container is created and invoked, it remains active and available for subsequent
invocations for at least a few minutes before it is terminated. We can define :
-Warm container:When subsequent
invocations occur on a container that has already been active and invoked at least once before
-Cold start:When an invocation occurs for a Lambda function that requires your function code package to be created and invoked for the first time

Fig(1):Invocations of warm function containers and cold function containers

Lambda Function Event Sources

Event sources are the triggers that invoke your code on AWS Lambda function through the Invoke API.You don't have to write, scale, or maintain any of the software that integrates the triggers with your Lambda function.

Invocation Patterns:

There are two models for invoking a Lambda function:
-Push Model: Lambda function is invoked every time a particular event occurs within another AWS service.
-Pull Model:Lambda polls a data source and invokes your function with it.
A a Lambda function can be executed synchronously or asynchronously through the InvocationType parameter . This parameter has three possible values. RequestResponse to execute Synchronously, Event to execute Asynchronously and DryRun to test that the invocation is permitted for the caller, but don’t
execute the function.

Lambda Function Configuration

This section is about the various configuration options that define how your code is executed within Lambda.

Function Memory

Function Memory Helps to define the resources allocated to your executing Lambda function by increasing/decreasing function resources (memory/RAM).You can optimize the price and performance of Lambda function by selecting the appropriate memory allocation.

Versions and Aliases :

Versioning is possible for AWS Lambda functions.Each and every Lambda function has a default version built in: $LATEST that addresses the most recent uploaded code. Each version has its own Amazon Resource Name (ARN).
PS: When calling the Invoke API or creating an event source for your Lambda function, you can also specify a specific version of the Lambda function .Otherwise, $LATEST is invoked by default.
Each Lambda function container is specific to a particular version of your function. A different set of containers is installed and managed for each function version.

Invoking your Lambda functions by their version numbers is useful during testing activities. However, this is not recommended for real application traffic. This this requires updating all of the triggers and clients invoking your Lambda function with Lambda alias to point at a new function version each time you wanted to update your code .Lambda aliases can be used to represent your Lambda function version (live/prod/active),to enable blue/green deployment pattern ,and for debuging when an alias is integrated with a testing stack for example

last paragraph page 15

IAM Role

Lambda Function Permissions

Network Configuration

Environment Variables

Dead Letter Queues

Timeout

Serverless Best Practices

Serverless Architecture Best Practices

Security Best Practices

Reliability Best Practices

Performance Efficiency Best Practices

Operational Excellence Best Practices

Cost Optimization Best Practices

Serverless Development Best Practices

Infrastructure as Code – the AWS Serverless Application Model (AWS SAM)

Local Testing – AWS SAM Local

Coding and Code Management Best Practices

Testing

Continuous Delivery

Sample Serverless Architectures

Conclusion

Hybrid Machine Learning | AWS Whitepaper Summary

Marwa Talaat — Sat, 13 Nov 2021 10:09:32 +0000

This article aims to discuss the outline, known considerations, design patterns, and solutions that customers should know when considering integrations between local compute and the AWS cloud across the machine learning lifecycle.

AWS purpose hybrid ML patterns as an intermediate step in their cloud and ML journey. The patterns involve a minimum of two compute environments, typically local compute resources such as personal laptops or corporate data centers, and the cloud.

This article is intended for individuals who already have a baseline understanding of machine learning, in addition to Amazon SageMaker.

Stage One: Basics

Developing technology that applies machine learning is challenging since it depends on data. Datasets vary from bytes to petabytes, objects to file systems, text to vision, tables to logs. Software frameworks supporting machine learning models evolve rapidly, undergoing potentially major changes multiple times a year, if not quarter or month. Nowadays, data science projects require different skill levels in team, from business stakeholders, quality and availability of datasets and models, and customer adoption

Companies who adopt a cloud-native approach realize its value of compute capacity with the needs of their business, technical resources to focus on building features, rather than taking on the burden of managing and maintaining their own underlying infrastructure. But for those companies born before the cloud and even for newer companies founded more recently, potentially those that made an informed decision to build on-premises, how can they realize the value of newly launched cloud services when the early requirements that were once infeasible on the cloud are now within reach?

For customers who want to integrate the cloud with existing on-premises ML, AWS propose a series of tenets to guide our discussion: -

Seamless management experience: Customers need end-to-end ML across multiple compute environments, limiting the burden of administrative while successfully operate complex tasks.
Tunable latency: The customers enjoy using applications that respond within the timelines of their moment-to-moment expectations, and designers of these applications understand the criticality of time-bound SLA’s. Engineers want to work with ML models that can respond to an app’s request within in milliseconds, regardless of the hosted. While not every customer requires response times at low latency levels, but faster is better.
Fast time-to-value: Customers expect solutions to be easy to use, with simple interfaces, and not requiring significant amounts of platform-specific engineering to execute a task.
Flexible: Customers need compute paradigms that provide the flexibility their business demands. ML applications may need to serve real-time responses to billions of users worldwide. Service providers should anticipate for deploying all environments.
Low Cost: Customers want transparency in their cost structure, they need to see a clear economic advantage to computing in the cloud relative to developing locally. Service providers need to anticipate this and compete on cost with respect to local compute options
End in the cloud: If there was any doubt that cloud computing is the way of the future, the global pandemic of 2020 put that doubt to rest. AWS also call out the final state of that design, helping customers understand which cloud technologies to leverage in the long runleverage in the long run.

There are two very different approaches to hosting – one type of pattern trains in the cloud with the intention of hosting the model itself on-premises, while another hosts the model in the cloud to applications deployed on-premises. Finally, a key pillar in applying these patterns is security.

What is hybrid?

At AWS, we look at “hybrid” capabilities as those that touch the cloud in some capacity, while also touching local compute resources. Local compute such as laptops hosting Jupyter notebooks and Python scripts, HDFS clusters storing terabytes of data, AWS Outposts stored on-premises, or AWS Outposts stored on-premises.

The hybrid architectures are having a minimum of two compute environments, what we will call “primary” and “secondary” environments. The primary environment as where the workload begins, and secondary environment is where the workload ends.

Depending on your case for instance if you are packaging up a model locally to deploy to the cloud, you might call your local laptop “primary” and your cloud environment “secondary.”. However, if you are training on cloud and want to deploy locally, you might call your cloud environment “primary” and your local environment “secondary.”

What hybrid is not?

There are some container-specific tools that provide a “run anywhere” experience, such as EKS and ECS. In those contexts, we will lean into prescriptive guidance for building, training, and deploying machine learning models with these services.

Stage Two: Hybrid patterns for development

Development refers to the phase in machine learning when customers are iteratively building models. This may or may not include exploratory data analysis, deep learning model development and compilation, software package installation and management, Jupyter kernels, visualization, Docker image building, and Python-driven data manipulation.

There are two major options for hybrid development that customer can apply one or both. Laptop and desktop personal computers. Self-managed local servers utilizing specialized GPUs, colocations, self-managed racks, or corporate data centers.

Develop on personal computers, to train and host in the cloud

Customers can use local development environments, such as PyCharm or Jupyter installations on their laptops or personal computer, and then connect to the cloud via AWS Identity and Access Management (IAM) permissions and interface with AWS service APIs through the AWS CLI or an AWS SDK (ex boto3). Having connected to the cloud, customers can execute training jobs and/or deploy resources.

Develop on local servers, to train and host in the cloud

Stage Three: Training

Hybrid patterns for training

Hybrid pattern for training comes down to one of two paths. Either you train locally and deploy on the cloud. Or the data sitting on local resources and want to select from that to move into the cloud to train.

Training locally, to deploy in the cloud

During enterprise migrations, training locally may be advantageous as a first step to develop a model. There are two key actions here. First, if you are training locally then you will need to acquire the compute capacity to train a model and think about size of dataset, and models that you want to use. When you are training on-premises, you need to plan for that well in advance and acquire the compute resources ahead of time.

After your model is trained, there are two common approaches for packaging and hosting it in the cloud. One Simple path is Docker, using a Docker file you can build your own custom image that hosts your inference script, model artifact, and packages. Register this image in the Elastic Container Registry (ECR) and point to it from your SageMaker estimator.

Another option is using pre-built containers within the SageMaker Python SDK. Bring your inference script and custom packages, upload your model artifact to Amazon S3, and import an estimator for your framework of choice.

In the following diagram, we outline how to do this from your laptop. The pattern is similar for doing the same from an enterprise data center with servers, as outlined above.

How to monitor your model in the cloud?

A key feature for hosting is model monitor, or the ability to detect data, bias, feature, and model quality drift. It is ability to capture data hitting your real-time endpoint, and programmatically compare this to your training data.

Enabling model monitor is easy in SageMaker. Upload your training data to an Amazon S3 bucket and use our pre-built image to learn the upper and lower bounds on your training data. This job uses Amazon Deequ to perform “unit testing for data,” and you will receive a JSON file with the upper and lower statistically recommended bounds for each feature. You can modify these thresholds. After confirming your thresholds, schedule monitoring jobs in your production environment. The jobs run automatically, comparing your captured inference requests in Amazon S3 with your thresholds.

CloudWatch will alert you when your inference data is outside of your pre-determined thresholds, and you can use those alerts to trigger a re-training pipeline.

How to handle retraining / retuning?

SageMaker makes train and tuning jobs easy to manage, because all you need to bring is your training script and dataset. Ensure your new dataset is loaded into an Amazon S3 bucket, or other supported data source.

Once you have defined a training estimator, it is trivial to extend this to support hyperparameter tuning. Define your tuning job configuration using tuning best practices and execute. Having defined a tuning job, you can automate this in a variety of ways. While AWS Lambda may seem compelling upfront, to use the SageMaker Python SDK (and not boto3) with Lambda, sadly you need to create an executable layer to upload within your function.

You may consider SageMaker Pipelines, an MLOps framework that uses your SageMaker Python SDK job constructs as argument and creates a step-driven framework to execute your entire pipeline.

How to serve thousands of models in the cloud at low cost?

You may consider Multi-model endpoints give you the ability to serve thousands of models from a single endpoint, invoking the name of the model when calling predict.
Create the multi-model endpoint, pointing to Amazon S3, and load your model artifacts into the bucket. Invoke the endpoint from your client application, (eg. with AWS Lambda), and dynamically select the right model in your application. It allows to host up to 5 containers on a single SageMaker endpoint, invoking the endpoint with the name of the model you want to use.

Storing data locally, to train and deploy in the cloud

when and how do I move my data to the cloud?

Schedule data transfer jobs with AWS DataSync

It is a data transfer service that simplifies, automates, and accelerates moving data between on-premises storage systems and AWS storage services, as well as between AWS storage services.

It can be easily moved petabytes of data from your local on-premises servers up to the AWS cloud.
It can be deployed can easily move petabytes of data from your local on-premises servers up to the AWS cloud.
It can be connected to your local NFS resources and deploy directly into Amazon S3 buckets or EFS volumes, or both.

Migrating from Local HDFS

As customers explore migrating data stored in local HDFS clusters, typically they find themselves somewhere between two extremes.

On the other, you might wholly embrace HDFS as your center and move towards hosting it within a managed service, Amazon Elastic Map Reduce (EMR).

Best practices

• Use Amazon S3 intelligent tiering for objects over 128 KB
• Use multiple AWS accounts, and connect them with Organizations
• Set billing alerts
• Enable SSO with your current Active Directory provider
• Turn on Studio!

Develop in the cloud while connecting to data hosted on-premises

Customers who see the value of outsourcing management and upkeep of their enterprise ML development platforms, i.e. through using managed services like Amazon SageMaker, can still connect in to their on-premises data store at the beginning and middle phases of their enterprise migration.

Data Wrangler & Snowflake

Data Wrangler enables customers to browse and access data stores across Amazon S3, Amazon 3rd Athena, Amazon Redshift, and party data warehouses like Snowflake. This hybrid ML patten provides customers the ability to develop in the cloud while accessing data stored on premises, as organizations develop their migration plans.

Train in the cloud, to deploy ML models on-premises

You can download whatever type of model artifact you need, but if you are deploying on-premises, you need to develop and host your own local web server.

This scenario builds on your previous experience developing and training in the cloud, with the key difference of exporting your model artifact to deploy locally. AWS recommends using dev and/or test endpoints in the cloud to give your teams the maximum potential to develop the best models they can.

If you are using a managed deep learning container, also known as “script mode,” for training and tuning, but you still want to deploy that model locally, plan on building your own image with your preferred software version, scanning, maintaining, and patching this over time. If you are using your own image, you will need to own updating that image as the software version, such as TensorFlow. Note that the best practice is to decouple hosting your ML model from hosting your application.

As models grow and shrink in size, hitting potentially billions of parameters and hundreds of gigs in byte size, or shrinking down to hundreds of parameters and staying under a few MB in size, you want the elasticity of the cloud to seamlessly map the state-of-the-art model to an efficient hardware choice.

Stage Four: Deployment

Monitor ML models deployed on-premises with SageMaker Edge Manager

Customers can train ML models in the cloud, deploy these on-premises, and monitor and update them in the cloud using SageMaker Edge Manager. SageMaker Edge Manage makes it easy for customers to manage ML models deployed on Windows, Linux, or ARM-based compute environments.

While customers do still need to provision, manage, procure, and physically secure the local compute environments in this pattern, Edge Manage simplifies the monitoring and updating of these models by bringing the control plane up to the cloud. However, you can bring your own monitoring algorithm to the service and trigger retraining pipelines as necessary, using the service the redeploy that model back down to the local device. This is particularly common for technology companies developing models for personal computers, such as laptops and desktops.

Hybrid patterns for deployment

Hybrid ML patterns around deployment can be interesting and complex. Choosing the “best” local deployment option has a lot of variety. You want to think about where your customers sit geographically, then you want to get your solution as close to them as you can. You want to balance speed with cost, cutting-edge solutions with ease of deployment and managing.

In this section will discuss the architecture for hosting an ML model via SageMaker in an AWS region, serving responses to requests from applications hosted on-premises. After that we’ll look at additional patterns for hosting ML models via Lambda at the Edge, Outposts, Local Zones, and Wavelength.

Serve models in the cloud to applications hosted on-premises

The most common use case for a hybrid pattern like this is enterprise migrations. You might have a data science team with tens of models, if not more than one hundred, ready to deploy via the cloud, while your application team is still refactoring their code to host on cloud-native services.

Host ML Models with Lambda at Edge to applications on-premises

This pattern takes advantage of a key capability of the AWS global network – the content delivery network known as Amazon CloudFront. Deploying content to CloudFront is easy, customers can package up code via AWS Lambda and set it to trigger from their CloudFront distribution.

What’s elegant about this approach is that CloudFront manages which of the 230+ points of presence will execute your function. Once you’ve set your Lambda function to trigger off CloudFront, you’re telling the service to replicate that function across all available regions and points of presence. This can take up to 8 minutes to replicate and become available.

This is a huge value-add for global companies looking at improving their digital customer experience worldwide.

AWS Local Zones

Local Zones are a way to extend your cloud resources to physical locations that are geographically closer to your customers. You can deploy ML models via ECS or EKS to serve inference with ultra- low latency near your customers, using AWS Local Zones.

AWS Wavelength

Wavelength is ideal when you are solving applications around mobile 5G devices, either anticipating network drop-offs or serving uses real-time model inference results. Wavelength provides ultra-low latency to 5G devices, and you can deploy ML models to this service via ECS or EKS. Wavelength embeds storage and compute inside the telecom providers, which is the actual 5G network.

Training with a 3rd party SaaS provider to host in the cloud

There are a lot of great SaaS providers for ML out there in the market today, like H20, DataRobot, Databricks, SAS, and others. 3rd Hosting a model in Amazon SageMaker that was trained from a party SAAS provider is easy. Ensure your provider allows export of proprietary software frameworks, such as with jars, bundles, images, etc. Follow steps to create a Docker file usingthat software framework, port into the Elastic Container Registry, and host on SageMaker.

Keep in mind that providers will have different ways of handling software, in particular images and image versions.

Control plane patterns for hybrid ML

AWS uses the concept of a “control plane,” or set of features dedicated to operations and management, while keeping this distinct from the “data plane,” or the datasets, containers, software, and compute environments.

Customers’ need it for operationalizing ML workloads are as varied and diverse as the businesses they exist within. Today it is not feasible for a single workflow orchestration tool to solve every problem, so most customers standardize on one workflow paradigm while keeping options open for others that may better solve given use cases. One such common control plane is Kubeflow in conjunction with EKS Anywhere.

SageMaker offers a native approach for workflow orchestration, known as SageMaker Pipelines.. It is ideal for advanced SageMaker users, especially those who are already onboarded to the IDE SageMaker Studio. The Studio also offers a UI to visual workflows built with SageMaker Pipelines. Apache Airflow is also a compelling option for ML workflow orchestration.

Orchestrate Hybrid ML Workloads with Kubeflow and EKS Anywhere

In this example we’re demonstrating training within local on-premises resources and orchestrating it using Kubeflow.

Stage Five: Other Services

Auxiliary services for hybrid ML patterns

AWS Outposts

Outposts is a keyway to enable hybrid experiences within your own data center. Order AWS Outposts, and Amazon will ship, install, and manage these resources for you. You can connect into these resources however you prefer and manage them from the cloud.

Outposts helps solve cases where customers want to build applications in countries where there is not currently an AWS Region, or for regulations that have strict data residency requirements, like online gambling and sports betting.

AWS Inferentia

A compelling reason to consider deploying your ML models in the cloud is the ease of accessing custom hardware for ML inferencing, specifically AWS Inferentia. You can use SageMaker’s managed deep learning containers to train your ML models, compile them for Inferentia with Neo, host on the cloud, and develop retrain and tune pipeline as usual. Using AWS Inferentia, Alexa was able to reduce their cost of hosting by 25%.

AWS Direct Connect

Ability to establish a private connection between your on-premises resources and your data center. Remember to establish a redundant link, as wires do go south!

Amazon ECS / EKS Anywhere

Both Amazon ECS and Amazon EKS feature “Anywhere” capabilities. This means that you can use the cloud as your control plane, to define, manage, and monitor your deployed applications, while executing tasks both in the Region and on-premises. The customers can use ECS Anywhere to deploy their models both in the cloud and on-premises at the same point in time!

The Final Stage: Use Cases

Hybrid ML Use Cases

Enterprise Migrations

One of the single most common use cases for hybrid patterns is enterprise migrations. For some of the largest and oldest organizations on the planet, without a doubt there is going to be a difference in ability and availability in moving towards the cloud across their teams.

Manufacturing

Applications within agriculture, industrial, and manufacturing are ripe opportunities for hybrid ML. After companies have invested tens, and sometimes hundreds, of thousands of dollars in advanced machinery, it is simply a matter of prudence to develop and monitor ML models to predict the health of that machinery.

Gaming

Customers who build gaming applications may see the value in adopting advanced ML services like Amazon SageMaker to raise the bar on their ML-applications but struggle to realize this if their entire platform was build and is currently hosted on premises. The AWS global delivery network to minimize end-user latency.

Mobile application development

With the introduction of AWS Wavelength, customers can deploy ML models directly inside of the 5G network. To solve applications such as anticipated network drop-off or hosting ML models in the cloud for real-time inferencing with 5G customers, you can use ML models hosted on ECS to deploy and monitor models onto AWS Wavelength. This becomes a hybrid pattern when customers develop and train in a secondary environment, wherever that may be, with the intention to deploy onto AWS Wave.

AI-enhanced media and content creation

Customers can host these billion-plus parameter models via ECS on AWS Local Zones, responding to application requests coming from on-premises data centers, to provide world-class experiences to content creators.

Depending on where customers develop and retrain their models, using Local Zones with SOTA models may or may not be a true hybrid pattern, but used effectively it can enhance content generator’s productivity and ability to create.

Autonomous Vehicles

Customers who develop autonomous machinery, vehicles, or robots in multiple capacity by default require hybrid solutions. This is because while training can happen anywhere, inference must necessarily happen at the edge.

Amazon SageMaker Neo is a compelling option here. Key questions for hybrid ML AV architectures include monitoring at the edge, retraining and retuning pipelines, in addition to efficient and automatic data labelling.

Conclusion

In this document, we explored hybrid ML patterns across the entire ML lifecycle. We looked at developing locally, while training and deploying in the cloud. We discussed patterns for training locally to deploy on the cloud, and even to host ML models in the cloud to serve applications on-premises.

At the end of the day, we want to support customer success in all shapes and forms. We firmly believe that most workloads will end in the cloud in the long run, but because the complexity, magnitude, and length of enterprise migrations may be daunting for some of the oldest organizations in the world, we propose these hybrid ML patterns as an intermediate step on customer’s cloud journey.

References:

If you are interested in learning how to migrate from local HDFS clusters to Amazon EMR, please see this migration guide: https://d1.awsstatic.com/whitepapers/amazon_emr_migration_guide.pdf
Original AWS Whitepaper

Overview of AWS: Quantum, Robotics, Satellite, Security, Identity, Compliance, and Storage Services |AWS WhitePaper Summary

‪Kareem Negm‬‏ — Mon, 08 Nov 2021 12:08:04 +0000

Quantum Technologies

Amazon Braket is a fully managed quantum computing service that helps researchers and developers get started with the technology to accelerate research and discovery. Amazon Braket provides a development environment for you to explore and build quantum algorithms, test them on quantum circuit simulators, and run them on different quantum hardware technologies.

Robotics

AWS RoboMaker is a service that makes it easy to develop, test, and deploy intelligent robotics applications at scale. RoboMaker extends the most widely used open-source robotics software framework, Robot Operating System (ROS), with connectivity to cloud services. This includes AWS machine learning services, monitoring services, and analytics services that enable a robot to stream data, navigate, communicate, comprehend, and learn. RoboMaker provides a robotics development environment for application development, a robotics simulation service to accelerate application testing, and a robotics fleet management service for remote application deployment, update, and management.

Satellite

AWS Ground Station is a fully managed service that lets you control satellite communications, downlink and process satellite data, and scale your satellite operations quickly, easily and cost-effectively without having to worry about building or managing your own ground station infrastructure.

Security, Identity, and Compliance

Amazon Cognito lets you add user sign-up, sign-in, and access control to your web and mobile apps quickly and easily. With Amazon Cognito, you also have the option to authenticate users through social identity providers such as Facebook, Twitter, or Amazon, with SAML identity solutions, or by using your own identity system.

Amazon Detective makes it easy to analyze, investigate, and quickly identify the root cause of potential security issues or suspicious activities. Amazon Detective automatically collects log data from your AWS resources and uses machine learning, statistical analysis, and graph theory to build a linked set of data that enables you to easily conduct faster and more efficient security investigations.

Amazon GuardDuty is a threat detection service that continuously monitors for malicious or unauthorized behavior to help you protect your AWS accounts and workloads. It monitors for activity such as unusual API calls or potentially unauthorized deployments that indicate a possible account compromise. GuardDuty also detects potentially compromised instances or reconnaissance by attackers.

Amazon Inspector is an automated security assessment service that helps improve the security and compliance of applications deployed on AWS. Amazon Inspector automatically assesses applications for exposure, vulnerabilities, and deviations from best practices.

Amazon Macie is a security service that uses machine learning to automatically discover, classify, and protect sensitive data in AWS. Amazon Macie recognizes sensitive data such as personally identifiable information (PII) or intellectual property, and provides you with dashboards and alerts that give visibility into how this data is being accessed or moved.

AWS Artifact is your go-to, central resource for compliance-related information that matters to you. It provides on-demand access to AWS’ security and compliance reports and select online agreements.

AWS Audit Manager helps you continuously audit your AWS usage to simplify how you assess risk and compliance with regulations and industry standards.

AWS Certificate Manager is a service that lets you easily provision, manage, and deploy Secure Sockets Layer/Transport Layer Security (SSL/TLS) certiﬁcates for use with AWS services and your internal connected resources. SSL/TLS certiﬁcates are used to secure network communications and establish the identity of websites over the Internet as well as resources on private networks.

The AWS CloudHSM is a cloud-based hardware security module (HSM) that enables you to easily generate and use your own encryption keys on the AWS Cloud.

AWS Directory Service for Microsoft Active Directory, also known as AWS Managed Microsoft AD, enables your directory-aware workloads and AWS resources to use managed Active Directory in the AWS Cloud.

AWS Firewall Manager is a security management service that makes it easier to centrally configure and manage AWS WAF rules across your accounts and applications. Using Firewall Manager, you can easily roll out AWS WAF rules for your Application Load Balancers and Amazon CloudFront distributions across accounts in AWS Organizations.

AWS Identity and Access Management (IAM) enables you to securely control access to AWS services and resources for your users. Using IAM, you can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources. IAM allows you to do the following:

Manage IAM users and their access
Manage IAM roles and their permissions
Manage federated users and their permissions

AWS Key Management Service (KMS) makes it easy for you to create and manage keys and control the use of encryption across a wide range of AWS services and in your applications.

AWS Network Firewall is a managed service that makes it easy to deploy essential network protections for all of your Amazon Virtual Private Clouds (VPCs). The service can be setup with just a few clicks and scales automatically with your network traffic, so you don't have to worry about deploying and managing any infrastructure.

AWS Resource Access Manager (RAM) helps you securely share your resources across AWS accounts, within your organization or organizational units (OUs) in AWS Organizations, and with IAM roles and IAM users for supported resource types.

AWS Secrets Manager helps you protect secrets needed to access your applications, services, and IT resources. The service enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle.

AWS Security Hub gives you a comprehensive view of your high-priority security alerts and compliance status across AWS accounts. There are a range of powerful security tools at your disposal, from firewalls and endpoint protection to vulnerability and compliance scanners.

AWS Single Sign-On (SSO) is a cloud SSO service that makes it easy to centrally manage SSO access to multiple AWS accounts and business applications. Your users simply sign in to a user portal with credentials they configure in AWS SSO or using their existing corporate credentials to access all their assigned accounts and applications from one place.

AWS WAF is a web application ﬁrewall that helps protect your web applications from common web exploits that could aﬀect application availability, compromise security, or consume excessive resources.

Storage

Amazon Elastic Block Store (Amazon EBS) provides persistent block storage volumes for use with Amazon EC2 instances in the AWS Cloud. Each Amazon EBS volume is automatically replicated within its Availability Zone to protect you from component failure, offering high availability and durability.

Amazon Elastic File System (Amazon EFS) provides a simple, scalable, elastic file system for Linux-based workloads for use with AWS Cloud services and on-premises resources. You can access your file systems across AZs and AWS Regions and share files between thousands of Amazon EC2 instances and on-premises servers via AWS Direct Connect or AWS VPN.

Amazon FSx for Lustre is a fully managed file system that is optimized for compute-intensive workloads, such as high performance computing, machine learning, and media data processing workflows.

Amazon FSx for WindowsFile Server provides a fully managed native Microsoft Windows file system so you can easily move your Windows-based applications that require file storage to AWS.

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics.

Amazon S3 Glacier is a secure, durable, and extremely low-cost storage service for data archiving and long-term backup. It is designed to deliver 99.999999999% durability, and provides comprehensive security and compliance capabilities that can help meet even the most stringent regulatory requirements.

AWS Backup enables you to centralize and automate data protection across AWS services. AWS Backup offers a cost-effective, fully managed, policy-based service that further simplifies data protection at scale. AWS Backup also helps you support your regulatory compliance or business policies for data protection.

The Storage Gateway is a hybrid storage service that enables your on-premises applications to seamlessly use AWS cloud storage. You can use the service for backup and archiving, disaster recovery, cloud data processing, storage tiering, and migration.

Thank you for following this series of articles and I hope it will be useful to you
If you want to read about the original Whitepaper , you can find it here

Overview of AWS : Media , Migration and Transfer, Networking and Content Delivery Services| AWS WhitePaper Summary

‪Kareem Negm‬‏ — Mon, 08 Nov 2021 11:59:13 +0000

Media Services

Amazon Elastic Transcoder is media transcoding in the cloud. It is designed to be a highly scalable, easy-to-use, and cost-eﬀective way for developers and businesses to convert (or transcode) media ﬁles from their source format into versions that will play back on devices like smartphones, tablets, and PCs.

Amazon Interactive Video Service (Amazon IVS) is a managed live streaming solution that is quick and easy to set up, and ideal for creating interactive video experiences. You can easily customize and enhance the audience experience through the Amazon IVS player SDK and timed metadata APIs, allowing you to build a more valuable relationship with your viewers on your own websites and applications.

Amazon Nimble Studio empowers creative studios to produce visual effects, animation, and interactive content entirely in the cloud, from storyboard sketch to final deliverable.

AWS Elemental Appliances and Software solutions bring advanced video processing and delivery technologies into your data center, co-location space, or on-premises facility.
AWS Elemental Live, Server and Conductor come in two variants: ready-to-deploy appliances, or AWS-licensed software that you install on your own hardware. AWS Elemental Link is a compact hardware device that sends live video to the cloud for encoding and delivery to viewers.

AWS Elemental MediaConnect is a high-quality transport service for live video.

AWS Elemental MediaConvert is a file-based video transcoding service with broadcast-grade features. With AWS Elemental MediaConvert, you can focus on delivering compelling media experiences without having to worry about the complexity of building and operating your own video processing infrastructure.

AWS Elemental MediaLive is a broadcast-grade live video processing service. It lets you create high-quality video streams for delivery to broadcast televisions and internet-connected multiscreen devices, like connected TVs, tablets, smart phones, and set-top boxes.

AWS Elemental MediaPackage reliably prepares and protects your video for delivery over the Internet.

AWS Elemental MediaStore is an AWS storage service optimized for media. It gives you the performance, consistency, and low latency required to deliver live streaming video content.

AWS Elemental MediaTailor lets video providers insert individually targeted advertising into their video streams without sacrificing broadcast-level quality-of-service.

Migration and Transfer

AWS Application Migration Service (AWS MGN) allows you to quickly realize the benefits of migrating applications to the cloud without changes and with minimal downtime.
by launching non-disruptive tests before migrating, you can be confident that your most critical applications such as SAP, Oracle, and SQL Server will work seamlessly on AWS.

AWS Migration Hub provides a single location to track the progress of application migrations across multiple AWS and partner solutions. This allows you to quickly get progress updates across all of your migrations, easily identify and troubleshoot any issues, and reduce the overall time and effort spent on your migration projects.

AWS Application Discovery Service helps enterprise customers plan migration projects by gathering information about their on-premises data centers.

AWS Database Migration Service helps you migrate databases to AWS easily and securely. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database.

AWS Server Migration Service (SMS) is an agentless service which makes it easier and faster for you to migrate thousands of on-premises workloads to AWS. AWS SMS allows you to automate, schedule, and track incremental replications of live server volumes, making it easier for you to coordinate large-scale server migrations.

The AWS Snow Family helps customers that need to run operations in austere, non-data center environments, and in locations where there's lack of consistent network connectivity. The Snow Family comprises AWS Snowcone, AWS Snowball, and AWS Snowmobile and offers a number of physical devices and capacity points, most with built-in computing capabilities.

AWS Snowcone is the smallest member of the AWS Snow Family of edge computing edge storage, and data transfer devices, weighing in at 4.5 pounds (2.1 kg) with 8 terabytes of usable storage.

AWS Snowball is an edge computing, data migration, and edge storage device that comes in two options. Snowball Edge Storage Optimized devices provide both block storage and Amazon S3-compatible object storage, and 40 vCPUs.

AWS Snowmobile is an exabyte-scale data transfer service used to move extremely large amounts of data to AWS. You can transfer up to 100 PB per Snowmobile, a 45-foot long ruggedized shipping container, pulled by a semi-trailer truck.

AWS DataSync is a data transfer service that makes it easy for you to automate moving data between on-premises storage and Amazon S3 or Amazon Elastic File System (Amazon EFS).

AWS Transfer Family provides fully managed support for file transfers directly into and out of Amazon S3 or Amazon EFS. Getting started with the AWS Transfer Family is easy; there is no infrastructure to buy and set up.

Networking and Content Delivery

Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale.

Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment.

Amazon Route 53 is a highly available and scalable cloud Domain Name System (DNS) web service.
Amazon Route 53 effectively connects user requests to infrastructure running in AWS—such as EC2 instances, Elastic Load Balancing load balancers, or Amazon S3 buckets—and can also be used to route users to infrastructure outside of AWS.

Amazon Virtual Private Cloud (Amazon VPC) lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define.

AWS App Mesh makes it easy to monitor and control microservices running on AWS. App Mesh standardizes how your microservices communicate, giving you end-to-end visibility and helping to ensure high-availability for your applications.

AWS Cloud Map is a cloud resource discovery service. With Cloud Map, you can define custom names for your application resources, and it maintains the updated location of these dynamically changing resources. This increases your application availability because your web service always discovers the most up-to-date locations of its resources.

AWS Direct Connect makes it easy to establish a dedicated network connection from your premises to AWS. Using AWS Direct Connect, you can establish private connectivity between AWS and your data center, office, or co-location environment, which in many cases can reduce your network costs, increase bandwidth throughput, and provide a more consistent network experience than Internet-based connections.

AWS Global Accelerator is a networking service that improves the availability and performance of the applications that you offer to your global users.
AWS Global Accelerator makes it easier to manage your global applications by providing static IP addresses that act as a fixed entry point to your application hosted on AWS which eliminates the complexity of managing specific IP addresses for different AWS Regions and Availability Zones. AWS Global Accelerator is easy to set up, configure and manage.

AWS PrivateLink simplifies the security of data shared with cloud-based applications by eliminating the exposure of data to the public Internet. AWS PrivateLink provides private connectivity between VPCs, AWS services, and on-premises applications, securely on the Amazon network.

AWS Transit Gateway is a service that enables customers to connect their Amazon Virtual Private Clouds (VPCs) and their on-premises networks to a single gateway. As you grow the number of workloads running on AWS, you need to be able to scale your networks across multiple accounts and Amazon VPCs to keep up with the growth.

AWS Virtual Private Network solutions establish secure connections between your on-premises networks, remote offices, client devices, and the AWS global network. AWS VPN is comprised of two services: AWS Site-to-Site VPN and AWS Client VPN. Each service provides a highly-available, managed, and elastic cloud VPN solution to protect your network traffic.

Elastic Load Balancing (ELB) automatically distributes incoming application traffic across multiple targets, such as Amazon EC2 instances, containers, and IP addresses.

Overview of AWS : Management and Governance | AWS WhitePaper Summary

‪Kareem Negm‬‏ — Mon, 08 Nov 2021 11:52:42 +0000

Management and Governance

Amazon CloudWatch is a monitoring and management service built for developers, system operators, site reliability engineers (SRE), and IT managers. You can use CloudWatch to set high resolution alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues, and discover insights to optimize your applications, and ensure they are running smoothly.

AWS Auto Scaling monitors your applications and automatically adjusts capacity to maintain steady, predictable performance at the lowest possible cost. With AWS Auto Scaling, your applications always have the right resources at the right time.

AWS Chatbot is an interactive agent that makes it easy to monitor and interact with your AWS resources in your Slack channels and Amazon Chime chat rooms. With AWS Chatbot you can receive alerts, run commands to return diagnostic information, invoke AWS Lambda functions, and create AWS support cases.

AWS Compute Optimizer recommends optimal AWS resources for your workloads to reduce costs and improve performance by using machine learning to analyze historical utilization metrics.

AWS Control Tower automates the set-up of a baseline environment or landing zone, that is a secure, well-architected multi-account AWS environment. Control Tower automates the set-up of their landing zone and configures AWS management and security services based on established best practices in a secure, compliant, multi-account environment.

AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.

AWS CloudTrail is a web service that records AWS API calls for your account and delivers log files to you. The recorded information includes the identity of the API caller, the time of the API call, the source IP address of the API caller, the request parameters, and the response

AWS Config is a fully managed service that provides you with an AWS resource inventory, configuration history, and configuration change notifications to enable security and governance. The Config Rules feature enables you to create rules that automatically check the configuration of AWS resources recorded by AWS Config.

AWS Launch Wizard offers a guided way of sizing, configuring, and deploying AWS resources for third-party applications, such as Microsoft SQL Server Always On and HANA based SAP systems, without the need to manually identify and provision individual AWS resources. To start, you input your application requirements, including performance, number of nodes, and connectivity on the service console.

AWS Organizations helps you centrally manage and govern your environment as you grow and scale your AWS resources. Using AWS Organizations, you can programmatically create new AWS accounts and allocate resources, group accounts to organize your workflows, apply policies to accounts or groups for governance, and simplify billing by using a single payment method for all of your accounts.

AWS OpsWorks is a configuration management service that provides managed instances of Chef and Puppet. Chef and Puppet are automation platforms that allow you to use code to automate the configurations of your servers.

AWS Proton is the first fully managed delivery service for container and serverless applications. Platform engineering teams can use AWS Proton to connect and coordinate all the different tools needed for infrastructure provisioning, code deployments, monitoring, and updates.

AWS Service Catalog allows organizations to create and manage catalogs of IT services that are approved for use on AWS. These IT services can include everything from virtual machine images, servers, software, and databases to complete multi-tier application architectures.

AWS Systems Manager gives you visibility and control of your infrastructure on AWS. Systems Manager provides a unified user interface so you can view operational data from multiple AWS services and allows you to automate operational tasks across your AWS resources.

AWS Systems Manager contains the following tools:
Resource groups: Lets you create a logical group of resources associated with a particular workload such as different layers of an application stack, or production versus development environments.
Insights Dashboard: Displays operational data that the AWS Systems Manager automatically aggregates for each resource group.

Run Command: Provides a simple way of automating common administrative tasks like remotely executing shell scripts or PowerShell commands, installing software updates, or making changes to the configuration of OS, software, EC2 and instances and servers in your on-premises data center.
State Manager: Helps you define and maintain consistent OS configurations such as firewall settings and anti-malware definitions to comply with your policies.
Inventory: Helps you collect and query configuration and inventory information about your instances and the software installed on them.
Maintenance Window: Lets you define a recurring window of time to run administrative and maintenance tasks across your instances.
Patch Manager: Helps you select and deploy operating system and software patches automatically across large groups of instances.
Automation: Simplifies common maintenance and deployment tasks, such as updating Amazon Machine Images (AMIs).
Parameter Store: Provides an encrypted location to store important administrative information such as passwords and database strings.
Distributor: Helps you securely distribute and install software packages, such as software agents. Systems Manager Distributor allows you to centrally store and systematically distribute software packages while you maintain control over versioning.
Session Manager: Provides a browser-based interactive shell and CLI for managing Windows and Linux EC2 instances, without the need to open inbound ports, manage SSH keys, or use bastion hosts.

AWS Trusted Advisor is an online resource to help you reduce cost, increase performance, and improve security by optimizing your AWS environment.

AWS Personal Health Dashboard provides alerts and remediation guidance when AWS is experiencing events that might affect you. While the Service Health Dashboard displays the general status of AWS services, Personal Health Dashboard gives you a personalized view into the performance and availability of the AWS services underlying your AWS resources.

AWS Managed Services provides ongoing management of your AWS infrastructure so you can focus on your applications.

The AWS Console Mobile Application lets customers view and manages a select set of resources to support incident response while on the go. Customers can view ongoing issues and follow through to the relevant CloudWatch alarm screen for a detailed view with graphs and configuration options. In addition, customers can check on the status of specific AWS services, view detailed resource screens, and perform select actions.

AWS License Manager makes it easier to manage licenses in AWS and on-premises servers from software vendors such as Microsoft, SAP, Oracle, and IBM. Administrators gain control and visibility of all their licenses with the AWS License Manager dashboard and reduce the risk of non-compliance, misreporting, and additional costs due to licensing overages.

The AWS Well-Architected Tool helps you review the state of your workloads and compares them to the latest AWS architectural best practices.

Overview of AWS : Machine learning Services| AWS WhitePaper Summary

‪Kareem Negm‬‏ — Wed, 03 Nov 2021 18:21:32 +0000

Machine Learning services

Amazon Augmented AI (Amazon A2I) is a machine learning service that makes it easy to build the workflows required for human review.

Amazon CodeGuru is a developer tool that provides intelligent recommendations to improve code quality and identify an application’s most expensive lines of code.

CodeGuru Reviewer uses machine learning and automated reasoning to identify critical issues, security vulnerabilities, and hard-to-find bugs during application development and provides recommendations to improve code quality.

CodeGuru Profiler helps developers find an application’s most expensive lines of code by helping them understand the runtime behavior of their applications, identify and remove code inefficiencies, improve performance, and significantly decrease compute costs.

Amazon Comprehend is a natural language processing (NLP) service that uses machine learning to find insights and relationships in a text. No machine learning experience is required.
Amazon Comprehend uses machine learning to help you uncover the insights and relationships in your unstructured data. The service identifies the language of the text; extracts key phrases, places, people, brands, or events; understands how positive or negative the text is; analyzes text using tokenization and parts of speech; and automatically organizes a collection of text files by topic.

Amazon DevOps Guru is a Machine Learning (ML) powered service that makes it easy to improve an application’s operational performance and availability. DevOps Guru detects behaviors that deviate from normal operating patterns so you can identify operational issues long before they impact your customers.

Amazon Elastic Inference allows you to attach low-cost GPU-powered acceleration to Amazon EC2 and Amazon SageMaker instances to reduce the cost of running deep learning inference by up to 75%. Amazon Elastic Inference supports TensorFlow, Apache MXNet, PyTorch, and ONNX models.

Amazon Forecast is a fully managed service that uses machine learning to deliver highly accurate forecasts.
Amazon Forecast is a fully managed service, so there are no servers to provision, and no machine learning models to build, train, or deploy. You pay only for what you use, and there are no minimum fees and no upfront commitments.

Amazon Fraud Detector is a fully managed service that uses machine learning (ML) and more than 20 years of fraud detection expertise from Amazon, to identify potentially fraudulent activity so customers can catch more online fraud faster.

Amazon HealthLake is a HIPAA-eligible service that healthcare providers, health insurance companies, and pharmaceutical companies can use to store, transform, query, and analyze large-scale health data.

Amazon Kendra is an intelligent search service powered by machine learning. Kendra reimagines enterprise search for your websites and applications so your employees and customers can easily find the content they are looking for, even when it’s scattered across multiple locations and content repositories within your organization.

Amazon Lex is a service for building conversational interfaces into any application using voice and text. Lex provides the advanced deep learning functionalities of automatic speech recognition (ASR) for converting speech to text, and natural language understanding (NLU) to recognize the intent of the text, to enable you to build applications with highly engaging user experiences and lifelike conversational interactions.

Amazon Lookout for Equipment analyzes the data from the sensors on your equipment (e.g. pressure in a generator, flow rate of a compressor, revolutions per minute of fans), to automatically train a machine learning model based on just your data, for your equipment – with no ML expertise required.

Amazon Lookout for Metrics uses machine learning (ML) to automatically detect and diagnose anomalies in business and operational data, such as a sudden dip in sales revenue or customer acquisition rates.

Amazon Lookout for Vision is a machine learning (ML) service that spots defects and anomalies in visual representations using computer vision (CV). With Amazon Lookout for Vision, manufacturing companies can increase quality and reduce operational costs by quickly identifying differences in images of objects at scale.

Amazon Monitron is an end-to-end system that uses machine learning (ML) to detect abnormal behavior in industrial machinery, enabling you to implement predictive maintenance and reduce unplanned downtime.

Amazon Personalize is a machine learning service that makes it easy for developers to create individualized recommendations for customers using their applications.
Amazon Personalize is like having your own Amazon.com machine learning personalization team at your disposal, 24 hours a day.

Amazon Polly is a service that turns text into lifelike speech. Polly lets you create applications that talk, enabling you to build entirely new categories of speech-enabled products. Polly is an Amazon artificial intelligence (AI) service that uses advanced deep learning technologies to synthesize speech that sounds like a human voice.

Amazon Rekognition makes it easy to add image and video analysis to your applications using proven, highly scalable, deep learning technology that requires no machine learning expertise to use. With Amazon Rekognition, you can identify objects, people, text, scenes, and activities in images and videos, as well as detect any inappropriate content.

Amazon SageMaker is a fully-managed service that enables developers and data scientists to quickly and easily build, train, and deploy machine learning models at any scale. SageMaker removes all the barriers that typically slow down developers who want to use machine learning.
SageMaker removes the complexity that holds back developer success with each of these steps. SageMaker includes modules that can be used together or independently to build, train, and deploy your machine learning models.

Amazon SageMaker Ground Truth helps you build highly accurate training datasets for machine learning quickly. SageMaker Ground Truth offers easy access to public and private human labelers and provides them with built-in workflows and interfaces for common labeling tasks.

Amazon Textract is a service that automatically extracts text and data from scanned documents. Amazon Textract goes beyond simple optical character recognition (OCR) to also identify the contents of fields in forms and information stored in tables.

Amazon Transcribe is an automatic speech recognition (ASR) service that makes it easy for developers to add speech-to-text capability to their applications. Using the Amazon Transcribe API, you can analyze audio files stored in Amazon S3 and have the service return a text file of the transcribed speech. You can also send a live audio stream to Amazon Transcribe and receive a stream of transcripts in real time.

Amazon Translate is a neural machine translation service that delivers fast, high-quality, and affordable language translation.

Apache MXNet on AWS is a fast and scalable training and inference framework with an easy-to-use, concise API for machine learning.

The AWS Deep Learning AMIs provide machine learning practitioners and researchers with the infrastructure and tools to accelerate deep learning in the cloud, at any scale.

AWS DeepComposer is the world’s first musical keyboard powered by machine learning to enable developers of all skill levels to learn Generative AI while creating original music outputs.

AWS DeepLens helps put deep learning in the hands of developers, literally, with a fully programmable video camera, tutorials, code, and pre-trained models designed to expand deep learning skills.

AWS DeepRacer is a 1/18th scale race car which gives you an interesting and fun way to get started with reinforcement learning (RL). RL is an advanced machine learning (ML) technique which takes a very different approach to training models than other machine learning methods.

AWS Inferentia is a machine learning inference chip designed to deliver high performance at low cost. AWS Inferentia will support the TensorFlow, Apache MXNet, and PyTorch deep learning frameworks, as well as models that use the ONNX format.

TensorFlow on AWS enables developers to quickly and easily get started with deep learning in the cloud. The framework has broad support in the industry and has become a popular choice for deep learning research and application development, particularly in areas such as computer vision, natural language understanding and speech translation.

Overview of AWS : IOT , End User Computing , Front-End Web & Mobile , Game Tech Services| AWS WhitePaper Summary

‪Kareem Negm‬‏ — Tue, 02 Nov 2021 12:44:44 +0000

Internet of Things (IoT)

AWS IoT 1-Click is a service that enables simple devices to trigger AWS Lambda functions that can execute an action. AWS IoT 1-Click supported devices enable you to easily perform actions such as notifying technical support, tracking assets, and replenishing goods or services.

AWS IoT Analytics is a fully-managed service that makes it easy to run and operationalize sophisticated analytics on massive volumes of IoT data without having to worry about the cost and complexity typically required to build an IoT analytics platform.
IoT data is highly unstructured which makes it difficult to analyze with traditional analytics and business intelligence tools that are designed to process structured data. IoT data comes from devices that often record fairly noisy processes (such as temperature, motion, or sound).

The AWS IoT Button is a programmable button based on the Amazon Dash Button hardware. This simple Wi-Fi device is easy to conﬁgure, and it’s designed for developers to get started with AWS IoT Core, AWS Lambda, Amazon DynamoDB, Amazon SNS, and many other Amazon Web Services without writing device-speciﬁc code.
You can code the button's logic in the cloud to conﬁgure button clicks to count or track items, call or alert someone, start or stop something, order services, or even provide feedback.

AWS IoT Core is a managed cloud service that lets connected devices easily and securely interact with cloud applications and other devices. AWS IoT Core can support billions of devices and trillions of messages and can process and route those messages to AWS endpoints and to other devices reliably and securely.

AWS IoT Device Defender is a fully managed service that helps you secure your fleet of IoT devices. AWS IoT Device Defender continuously audits your IoT configurations to make sure that they aren’t deviating from security best practices.
AWS IoT Device Defender also lets you continuously monitor security metrics from devices and AWS IoT Core for deviations from what you have defined as appropriate behavior for each device.

AWS IoT Device Management makes it easy to securely onboard, organize, monitor, and remotely manage IoT devices at scale. With AWS IoT Device Management, you can register your connected devices individually or in bulk, and easily manage permissions so that devices remain secure.

AWS IoT Events is a fully managed IoT service that makes it easy to detect and respond to events from IoT sensors and applications. Events are patterns of data identifying more complicated circumstances than expected, such as changes in equipment when a belt is stuck or connected motion detectors using movement signals to activate lights and security cameras.

AWS IoT Greengrass seamlessly extends AWS to devices so they can act locally on the data they generate, while still using the cloud for management, analytics, and durable storage. With AWS IoT Greengrass, connected devices can run AWS Lambda functions, execute predictions based on machine learning models, keep device data in sync, and communicate with other devices securely – even when not connected to the Internet.

AWS IoT SiteWise is a managed service that makes it easy to collect, store, organize and monitor data from industrial equipment at scale to help you make better, data-driven decisions.

AWS IoT Things Graph is a service that makes it easy to visually connect different devices and web services to build IoT applications.

The AWS Partner Device Catalog helps you find devices and hardware to help you explore, build, and go to market with your IoT solutions. Search for and find hardware that works with AWS, including development kits and embedded systems to build new devices, as well as off-the-shelf-devices such as gateways, edge servers, sensors, and cameras for immediate IoT project integration.

FreeRTOS is an operating system for microcontrollers that makes small, low-power edge devices easy to program, deploy, secure, connect, and manage.

End User Computing

Amazon AppStream 2.0 is a fully managed application streaming service. You centrally manage your desktop applications on AppStream 2.0 and securely deliver them to any computer. You can easily scale to any number of users across the globe without acquiring, provisioning, and operating hardware or infrastructure.

Amazon WorkSpaces is a fully managed, secure cloud desktop service. You can use WorkSpaces to provision either Windows or Linux desktops in just a few minutes and quickly scale to provide thousands of desktops to workers across the globe

Amazon WorkLink is a fully managed service that lets you provide your employees with secure, easy access to your internal corporate websites and web apps using their mobile phones.
With Amazon WorkLink, employees can access internal web content as easily as they access any public website, without the hassle of connecting to their corporate network. When a user accesses an internal website, the page is first rendered in a browser running in a secure container in AWS.

Front-End Web & Mobile Services

Amazon Location Service makes it easy for developers to add location functionality to applications without compromising data security and user privacy.

Amazon Pinpoint makes it easy to send targeted messages to your customers through multiple engagement channels. Examples of targeted campaigns are promotional alerts and customer retention campaigns, and transactional messages are messages such as order confirmations and password reset messages.

AWS Amplify makes it easy to create, configure, and implement scalable mobile applications powered by AWS. Amplify seamlessly provisions and manages your mobile backend and provides a simple framework to easily integrate your backend with your iOS, Android, Web, and React Native frontends. Amplify also automates the application release process of both your frontend and backend allowing you to deliver features faster.

AWS Device Farm is an app testing service that lets you test and interact with your Android, iOS, and web apps on many devices at once, or reproduce issues on a device in real time. View video, screenshots, logs, and performance data to pinpoint and fix issues before shipping your app.

AWS AppSync is a serverless back-end for mobile, web, and enterprise applications.
AWS AppSync makes it easy to build data driven mobile and web applications by handling securely all the application data management tasks like online and offline data access, data synchronization, and data manipulation across multiple data sources.

Game Tech

Amazon GameLift is a managed service for deploying, operating and scaling dedicated game servers for session-based multiplayer games. Amazon GameLift makes it easy to manage server infrastructure, scale capacity to lower latency and cost, match players into available game sessions, and defend from distributed denial-of-service (DDoS) attacks.

Amazon Lumberyard is a free, cross-platform, 3D game engine for you to create the highest-quality games, connect your games to the vast compute and storage of the AWS Cloud, and engage fans on Twitch.

Overview of AWS : Containers, Database, Developer Tools Services| AWS WhitePaper Summary

‪Kareem Negm‬‏ — Sun, 31 Oct 2021 22:50:14 +0000

Containers

Amazon Elastic Container Registry (ECR) is a fully-managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images.

Amazon Elastic Container Service (Amazon ECS) is a highly scalable, high-performance container orchestration service that supports Docker containers and allows you to easily run and scale containerized applications on AWS.

Amazon Elastic Kubernetes Service (Amazon EKS) makes it easy to deploy, manage, and scale containerized applications using Kubernetes on AWS.

AWS App2Container (A2C) is a command-line tool for modernizing .NET and Java applications into containerized applications. A2C analyzes and builds an inventory of all applications running in virtual machines, on-premises, or in the cloud.

Red Hat OpenShift Service on AWS (ROSA) provides an integrated experience to use OpenShift. If you are already familiar with OpenShift, you can accelerate your application development process by leveraging familiar OpenShift APIs and tools for deployments on AWS.

Database

Amazon Aurora is a MySQL and PostgreSQL compatible relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-eﬀectiveness of open source databases.
Amazon Aurora is up to five times faster than standard MySQL databases and three times faster than standard PostgreSQL databases. It provides the security, availability, and reliability of commercial databases at 1/10th the cost.

Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. It's a fully managed, multi-region, multi-master database with built-in security, backup and restores, and in-memory caching for internet-scale applications.

Amazon ElastiCache is a web service that makes it easy to deploy, operate, and scale an in-memory cache in the cloud. The service improves the performance of web applications by allowing you to retrieve information from fast, managed, in-memory caches, instead of relying entirely on slower disk-based databases.

Amazon ElastiCache supports two open-source in-memory caching engines:

Redis - a fast, open-source, in-memory key-value data store for use as a database, cache, message broker, and queue. Amazon ElastiCache for Redis is a Redis-compatible in-memory service that delivers the ease-of-use and power of Redis along with the availability, reliability, and performance suitable for the most demanding applications.
Memcached - a widely adopted memory object caching system. ElastiCache for Memcached is protocol compliant with Memcached, so popular tools that you use today with existing Memcached environments will work seamlessly with the service.

Amazon Keyspaces (for Apache Cassandra) is a scalable, highly available, and managed Apache Cassandra–compatible database service. With Amazon Keyspaces, you can run your Cassandra workloads on AWS using the same Cassandra application code and developer tools that you use today.

Amazon Neptune is a fast, reliable, fully-managed graph database service that makes it easy to build and run applications that work with highly connected datasets. The core of Amazon Neptune is a purpose-built, high-performance graph database engine optimized for storing billions of relationships and querying the graph with milliseconds latency.

Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while automating time-consuming administration tasks such as hardware provisioning, database setup, patching and backups.

Amazon Relational Database Service (Amazon RDS) on VMware lets you deploy managed databases in on-premises VMware environments using the Amazon RDS technology enjoyed by hundreds of thousands of AWS customers.

Amazon Quantum Ledger Database (QLDB) is a fully managed ledger database that provides a transparent, immutable, and cryptographically verifiable transaction log ‎owned by a central trusted authority. Amazon QLDB tracks each and every application data change and maintains a complete and verifiable history of changes over time.

Amazon Timestream is a fast, scalable, fully managed time-series database service for IoT and operational applications that makes it easy to store and analyze trillions of events per day at 1/10th the cost of relational databases.

Amazon DocumentDB (with MongoDB compatibility) is a fast, scalable, highly available, and fully managed document database service that supports MongoDB workloads.

Developer Tools

Amazon Corretto is a no-cost, multiplatform, production-ready distribution of the Open Java Development Kit (OpenJDK). Corretto comes with long-term support that will include performance enhancements and security fixes.

AWS Cloud9 is a cloud-based integrated development environment (IDE) that lets you write, run, and debug your code with just a browser. It includes a code editor, debugger, and terminal.

AWS CloudShell is a browser-based shell that makes it easy to securely manage, explore, and interact with your AWS resources. CloudShell is pre-authenticated with your console credentials. Common Development and operations tools are pre-installed, so no local installation or configuration is required.

AWS CodeArtifact is a fully managed artifact repository service that makes it easy for organizations of any size to securely store, publish, and share software packages used in their software development process.

AWS CodeBuild is a fully managed build service that compiles source code, runs tests, and produces software packages that are ready to deploy. With CodeBuild, you don’t need to provision, manage, and scale your own build servers.

AWS CodeCommit is a fully managed source control service that makes it easy for companies to host secure and highly scalable private Git repositories.

AWS CodeDeploy is a service that automates code deployments to any instance, including EC2 instances and instances running on premises.

AWS CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates.

AWS CodeStar enables you to quickly develop, build, and deploy applications on AWS. AWS CodeStar provides a unified user interface, enabling you to easily manage your software development activities in one place.

AWS Fault Injection Simulator is a fully managed service for running fault injection experiments on AWS that makes it easier to improve an application’s performance, observability, and resiliency.

Fault Injection Simulator simplifies the process of setting up and running controlled fault injection experiments across a range of AWS services so teams can build confidence in their application behavior. With Fault Injection Simulator, teams can quickly set up experiments using pre-built templates that generate the desired disruptions.

AWS X-Ray helps developers analyze and debug distributed applications in production or under development, such as those built using a microservices architecture.

Forem: AWS MENA Community

Why we use AWS cloud services

Why we use AWS cloud

this article is in Arabic and for whom beginners in cloud

english version

AWS DevOps Monitoring Dashboard | AWS Whitepaper Summary

I. Architecture Description:

I.1 Use Case of the discussed architecture:

I.2 AWS CI/CD pipeline

I.3 AWS DevOps Tools

II. Walking Through The DevOps Monitoring Dashboard Architecture

II.1 Section 1 : Tracking and Gathering Data

II.1.0 Section 1.0

II.1.1 Section 1.1

III.1 Section 2: Analyzing data and Visualizing Analysis Results

1. Analysis:

2. Visualization:

Serverless مقدمة عن

Serverless ما هو ال

Serverless services ما هي

1. Compute

2. Integration

3. Data Store

Serverless ما هي افضل استخدمات ال

فيديو تعليمي

روابط ذات صلة

Amazon Machine Learning| ML Key Concepts

Amazon Machine Learning Key Concepts

ML Models

Evaluations

Batch Predictions

Real-time Predictions

Serverless Architectures with AWS Lambda Summary | AWS Whitepaper Summary

What Is Serverless?

AWS Lambda -the Basics

AWS Lambda- Diving Deeper

Lambda Function Code

The Function Code Package:

The Handler:

The Event Object:

The Context Object:

Writing Code for AWS Lambda—Statelessness and Reuse

Lambda Function Event Sources

Invocation Patterns:

Lambda Function Configuration

Function Memory

Versions and Aliases :

last paragraph page 15

IAM Role

Lambda Function Permissions

Network Configuration

Environment Variables

Dead Letter Queues

Timeout

Serverless Best Practices

Serverless Architecture Best Practices

Security Best Practices

Reliability Best Practices

Performance Efficiency Best Practices

Operational Excellence Best Practices

Cost Optimization Best Practices

Serverless Development Best Practices

Infrastructure as Code – the AWS Serverless Application Model (AWS SAM)

Local Testing – AWS SAM Local

Coding and Code Management Best Practices

Testing

Continuous Delivery

Sample Serverless Architectures

Conclusion

Hybrid Machine Learning | AWS Whitepaper Summary

Stage One: Basics

What is hybrid?

What hybrid is not?

Stage Two: Hybrid patterns for development

Develop on personal computers, to train and host in the cloud

Develop on local servers, to train and host in the cloud

Stage Three: Training

Hybrid patterns for training

Training locally, to deploy in the cloud

How to monitor your model in the cloud?

Hybrid ML Use Cases