Forem: philbasford

AWS re:invent 2022 wish list

philbasford — Wed, 23 Nov 2022 21:59:56 +0000

Just before it is wheels up and hopping over the Atlantic Ocean to the US for AWS re:invent. I wanted to do a quick blog containing my wish list. So here it is:

More features in SageMaker Serverless

SageMaker Serverless went GA in April 2022, having been announced at last years re:invent. However, it has the following limitations:

GPUs, AWS marketplace model packages, private Docker registries, Multi-Model Endpoints, VPC configuration, network isolation, data capture, multiple production variants, Model Monitor, and inference pipelines.

These for me are pretty much showstoppers for any situation I wish to use SageMaker Serverless, so I revert back to Lambda and do things the long way. OK off course I know GPU is still an issue with Lambda but please AWS can have these features for SageMaker Serverless.

Managed MLFlow

I am seeing growing demand for MLflow (https://mlflow.org/) and I am seeing a lot of people looking at Databricks as commercial offering for MLflow. Alternatively, some popele are implementing something like Managing your Machine Learning lifecycle with MLflow. Therefore, I think this was on my wish list last year, but I really hope AWS announce a Managed MLFlow Service. I know version 2.X is too new but at least 1.X would be great start.

Public CodeCommit

I used CodeCommit every day and it offers basic GIT functionality for private repositories. However, I wish to share anything with the public then I need to use my GIT Hub account as a second remote and remember to push too both. It would be awesome for AWS to offer public repositories that I can push my open-source projects too.

3rd party integration in StepFunctions

This year I have got into Apache Airflow and I wrote a blog Running Thousands of Models a month with Apache AirFlow. For me the main reason you would choose Apache Airflow over StepFunctions is the type of integrations you require. If you’re mainly using a lots of Open Source or Commercial offerings, either SaaS or PaaS, then your best using Apache Airflow. However, if you’re going mainly AWS Cloud Native then your use StepFunctions. Maybe with the odd Lambda for any non-AWS integrations. It would be awesome for AWS offer away, like for the boto3 SDK, to offer a way to use 3rd Party SDKs

StepFunctions Local Dev

I know I am not the only that fines this tedious but testing step functions locally with local integrations is hard work. It also not easily to integration test a complete serverless app. If you really do have time, you can build something with testcontainers and docker compose to spin up various AWS lock offerings:

Please AWS I would love a complete local serverless development environment for the main serverless services.

Cross account param store replication

Lastly my final wish is replication of parameters in Parameter store between accounts. I
have recently built some MLOPS pipelines where the S3 buckets and Dynamo DB tables
exist in one account and they are used by other accounts. I want the name of the buckets and tables, that are created dynamically by CloudFormation to be shared via Parameter store with other accounts. This would mean other Cloud Formation in the other accounts could just reference the Parameter

I hope you like the list and you want to meet at re:invent, contact me via PeerTalk

re:Invent 2022 Sessions

philbasford — Tue, 11 Oct 2022 09:39:55 +0000

This year I will be heading back to Las Vegas for re:Invent 2022. My third re:Invent and for first time post Pandemic. I have looked over the session catalog and here my picks for 2022.

Community

The following are some sessions that follow AWS Ambassadors, Heros, and Community Builders are presenting at:

Matt Houghton: Migrating 600 databases to AWS and making them better, faster, and cheaper

COM311
This dev chat covers the entire journey of a real-life large-scale database migration from Oracle on premises to PostgreSQL on AWS. Learn about the process and unique details, issues, and tips learned from zero-downtime database migration, change data capture, and schema conversion. Find out how the migration not only made the databases cheaper by eliminating licensing costs, but also sped up development and helped the application run faster. Finally, find out about the missing features the migration team needed for PostgreSQL and how they open-sourced them.

Date: Thursday, December 1
Time: 11:00 AM - 11:30 AM
Session level: 300 - Advanced
Session type: Dev Chat
Location: Hall B, Expo, Developer Lounge, Booth #2818,

Rolf Koski: Building a sustainable practice for tomorrow

PEX201
Do you want to have a positive impact on environmental sustainability when you advance your practices? Attend this session to learn how to reimagine your sustainability posture, innovate new lean solutions, and optimize your cloud workloads with a positive impact on the environment. Learn about user patterns, software designs, and AWS service considerations that organizations of any size can apply to their workloads to transform businesses into a sustainable future. This session is intended for AWS Partners.

Date: Monday, November 28
Time: 4:45 PM - 5:45 PM
Session level: 200 - Intermediate
Session type: Breakout Session
Location: Level 1, Encore Ballroom 2, Encore

Jon topper: MKT306: Build or transform to an AWS Well-Architected SaaS using AWS Control Tower

MKT306
Building AWS Well-Architected infrastructure is fundamental to building SaaS products that meet demanding compliance requirements around security, performance, and reliability. With AWS Marketplace professional services offerings, you can achieve compliance certification quickly and reduce per-tenant onboarding time. Join this chalk talk to learn best practices and approaches from experts.

Date: Monday, November 28
Time: 12:15 PM - 1:15 PM
Session level: 300 - Advanced
Session type: Chalk Talk
Location: Level 1, Montrachet 1, Wynn

Jimmy Dahlqvist: An IoT-enabled smoker for great BBQ

COM203
Maintaining a good temperature is important if you want to make really good BBQ. In this dev chat, take a look at how to create an IoT-enabled BBQ smoker using Raspberry Pi and AWS IoT Greengrass. Dive deep into the AWS services you need to create a serverless, event-driven system, using services like AWS IoT Core, AWS Lambda, and Amazon EventBridge that can be used to monitor and notify you when a temperature falls outside of a specified range.

Date: Tuesday, November 29
Time: 1:00 PM - 1:30 PM
Session level: 200 - Intermediate
Session type: Dev Chat
Location: Venetian

Martijn van Dongen: Terraform providers using AWS CloudFormation custom resources

BOA316
Terraform, AWS CloudFormation, and AWS CDK support a broad set of AWS services. Often, users want to automate more and even create resources that are not supported out of the box. In this session, explore how to build CloudFormation custom resources using your chosen programming language or Docker container. This method has been successfully used with CloudFormation and CDK, and it also works well with Terraform. Using Terraform modules, you only have to add a few lines of code to your projects to use a custom provider. Explore how this solution is designed to be secure, simple, future-proof, and reliable.

Date: Tuesday, November 29
Time: 12:30 PM - 1:30 PM
Session level: 300 - Advanced
Session type: Breakout Session

AI & ML

New approaches to implementing quantum algorithms

QTC403
While it is becoming easier to implement popular quantum algorithms on hardware through services like Amazon Braket, it can be challenging to generalize these algorithms, prepare them to run at scale, and code them in a way that is both efficient and hardware-aware. Many organizations have begun investigating quantum computing but have discovered that moving forward with developing algorithms that are production-ready can seem like an insurmountable challenge. In this chalk talk, join AWS and Classiq, an AWS Partner and quantum software company, to walk through a customer journey for a financial use case based on Amazon Braket.

Date: Monday, November 28
Time: 12:15 PM - 1:15 PM
Session level: 400 - Expert
Session Type: Chalk Talk
Location: MGM Grand

Extract AI-driven customer insights using Post-Call Analytics

AIM402
Companies are transforming existing contact centers by adding AI/ML to deliver actionable insights and improve automation with existing telephony systems. Join this workshop to learn how to use the AWS Contact Center Intelligence (CCI) Post-Call Analytics solution to derive AI-driven insights from virtually all customer conversations. Explore how you can automatically generate insights like sentiment, call issues, agent actions, and outcomes, as well as detect entities. See how you can increase the accuracy of the transcriptions using the custom vocabulary features of Amazon Transcribe. Finally, move on to building analytics dashboards on Amazon QuickSight to report the metrics that are important to your business. You must bring your laptop to participate.

Date: Monday, November 28
Time: 4:00 PM - 6:00 PMa
Session level: 400 - Expert
Session Type: WorkshoÅp
Location: Mandalay Bay

Train and deploy large language models on Amazon SageMaker

AIM405
Join this chalk talk to dive deep into training and hosting large language models for high accuracy and low cost using Hugging Face Transformers on Amazon SageMaker. Also, hear how Mantium achieved high-throughput training and low-latency inference for large transformer models using SageMaker distributed training libraries and DeepSpeed on SageMaker.

Date: Tuesday, November 29
Time: 11:45 AM - 12:45 PM
Session level: 400 - Expert
Session Type: Chalk Talk
Location: Caesars Forum

From cutting-edge ML research to product with the Amazon Research Awards

AIM408
The Amazon Research Awards (ARA) oﬀers unrestricted funds and AWS promotional credits to support research at academic institutions and nonprofit organizations in areas that align with the mission to advance customer-obsessed science. Join this session to learn about current state-of-the-art research from ARA awardees and how you can take advantage of these cutting-edge ML innovations using AWS services.

Date: Tuesday, November 29
Time: 12:30 PM - 1:30 PM
Session level: 400 - Expert
Session Type: Breakout Session
Location: Mandalay Bay

Choosing the right ML instance for training and inference on AWS

AIM407
As a data scientist, choosing the right compute instance for your workload on AWS can be challenging. On AWS, you can choose from CPUs, GPUs, AWS Trainium, and Intel Habana Gaudi to accelerate training. For inference, you can choose from CPUs, GPUs, and AWS Inferentia. This chalk talk guides you through how to choose the right compute instance type on AWS for your deep learning projects. Explore the available options, such as the most performant instance for training, the best instance for prototyping, and the most cost-effective instance for inference deployments. Learn how to use Amazon SageMaker Inference Recommender to help you make the right decision for your inference workloads.

Date: Wednesday, November 30
Time: 11:30 AM - 12:30 PM
Session level: 400 - Expert
Session Type: Chalk Talk

Maximize margins with AI/ML dynamic pricing models

TRV401
Many airlines want to adapt pricing in real time to match market conditions, making pricing models more complex. In this chalk talk, learn how you can use machine learning to drive increased margins with the appropriate risk mitigation and customer experience controls. Using Amazon SageMaker, Amazon Athena, AWS Lambda, and Amazon Kinesis, walk through data feature engineering considerations, example model approaches for demand forecasting, and margin adjustment techniques using search and booking data.

Date: Wednesday, November 30
Time: 12:15 PM - 1:15 PM
Session level: 400 - Expert
Session Type: Chalk Talk
Location: Mandalay Bay

Prepare data and model features for ML with ease, speed, and accuracy

AIM326
Join this chalk talk to learn how to prepare data for ML in minutes using Amazon SageMaker. SageMaker offers tools to simplify data preparation so that you can label, prepare, and understand your data and engineer model features. Walk through a complete data preparation workflow, and learn how to extract data from multiple data sources and transform it using the prebuilt visualization templates in SageMaker Data Wrangler. Then, discover how to store, update, retrieve, and share model features using SageMaker Feature Store for usage in training and inference.

Date: Friday, December 2
Time: 10:45 AM - 11:45 AM
Session level: 300 - Advanced
Session Type: Chalk Talk
Location: Caesars Forum

Explainable attention-based NLP using perturbation methods

BOA401
Explainable AI has gained a lot of attention from both legislators and the scientific community. There are many advantages of being able to explain the reasoning behind the decisions a model makes, top among them are fairness, accountability, and causality. More and more, explainability is used to improve both human and machine decision-making in a mutually reinforcing loop. This session specifically focuses on post hoc local explainability for transformer-based NLP. This session first details the general and scientific methods for explaining BERT and then explores challenges for a real-world implementation.

Date: Monday, November 28
Time: 12:15 PM - 1:15 PM
Session level: 400 - Expert
Session Type: Breakout Session
Location: MGM Grand

Build human-like customer experiences with conversational AI

AIM403
Today’s contact centers provide omnichannel services so that customers can interact by chat or phone. With shorter hold times, chat has helped solve some of the challenges associated with traditional voice-based contact centers, but it is still constrained by the need for live agents. While interactive voice response (IVR) systems help with some automation, their deterministic behavior frustrates many customers. Contact centers can solve for this using conversational AI. In this workshop, learn how you can use powerful conversational AI with Amazon Lex, Amazon Kendra, and the Amazon Chime SDK to build fully automated human-like chat experiences and alleviate challenges with IVR systems. You must bring your laptop to participate.

Date: Tuesday, November 29
Time: 11:45 AM - 1:45 PM
Session level: 400 - Expert
Session Type: Workshop
Location: Mandalay Bay

Exploring applications of QNNs and Monte Carlo simulations

QTC402
This chalk talk provides a close look at several cutting-edge applications for future quantum computers. First, learn how AWS Partner QC Ware and healthcare industry leader Roche are using Amazon Braket to build more interpretable and well-behaved quantum neural networks (QNNs) for medical image recognition. Then, discover how QC Ware and financial services organizations are building solutions with Amazon Braket to speed up financial Monte Carlo simulations.

Date: Tuesday, November 29
Time: 4:15 PM - 5:15 PM
Session level: 400 - Expert
Session Type: Chalk Talk
Location: Caesars Forum

Scaling ML training and inference workloads on Amazon EC2 and PyTorch

CMP401
Attend this chalk talk with Meta PyTorch and AWS to learn about distributed training, accelerated inference, bin-packing models, and scaling with PyTorch on AWS. In addition, hear about relevant large-companies that use AWS infrastructure and services for distributed training and to serve thousands of models with low latency and cost.

Date: Wednesday, November 30
Time: 1:45 PM - 2:45 PM
Session level: 400 - Expert
Session Type: Chalk Talk
Location: Mandalay Bay

Train and host foundation models with PyTorch on AWS

AIM404
Foundation models with hundreds of billions of parameters activate new applications for machine learning. Models such as OPT-175B, BLOOM, Jurassic, GPT-3, and DALL-E demonstrate exciting new use cases for text and image generation, but training and serving these models for inference poses new challenges. In this session, learn how to train and serve foundation models at scale using PyTorch on AWS. Discover how to reduce training time and cost by optimizing compute, network communication, input/output, checkpointing, and offloading from GPUs to CPUs. Also learn how to pick the right model parallelization strategies and best practices for training and serving these models.

Date: Wednesday, November 30
Time: 7:00 PM - 8:00 PM
Session level: 400 - Expert
Session Type: Breakout Session
Location: Venetian

Elastic Fabric Adapter advanced topics for AI/ML and HPC

CMP407
Elastic Fabric Adapter (EFA), built on the Scalable Reliable Datagram (SRD) protocol, is the foundation for scaling message passing interface (MPI) and NVIDIA Collective Communications Library–based high performance computing (HPC) and machine learning (ML) codes at AWS. This builders’ session addresses application scaling challenges and is led by an AWS EFA team principal engineer. It includes a roundtable-style discussion, so come prepared to share your thoughts and questions. You must bring your laptop to participate.

Date: Wednesday, November 30
Time: 7:45 PM - 8:45 PM
Session level: 400 - Expert
Session Type: Builders' Session
Location: Wynn

Using AutoML to develop deep learning solutions automatically

BOA402
Many companies, startups and enterprises alike, do not have the teams and processes needed to create their own research teams. The promise of AutoML is that it allows companies and scientists to create deep learning solutions without possessing a deep knowledge of deep learning. This workshop focuses on describing AutoML methods and walks you through a practical example of creating a solution on a public dataset using AutoGluon, an open-source AutoML framework developed by AWS. Finally, look into Amazon SageMaker Autopilot, which is developed using AutoGluon. You must bring your laptop to participate.

Date: Thursday, December 1
Time: 11:00 AM - 1:00 PM
Session level: 400 - Expert
Session Type: Workshop
Location: Mandalay Bay

Deploy deep learning models with hyperscale performance on SageMaker

AIM401
In this workshop, explore the hardware and software optimizations available for deep learning model deployment, when and how to use them, and the impact on the model performance and cost. Learn how to run thousands of deep learning models using Amazon SageMaker multi-model endpoints and reduce model serving costs. Learn how to use the AWS Neuron SDK to compile and deploy models on AWS Inferentia chips. Lastly, dive deep into model compilation and optimization techniques using TensorRT and NVIDIA Triton Inference Server features to achieve hyperscale inference on Amazon SageMaker. You must bring your laptop to participate.

Date: Friday, December 2
Time: 8:30 AM - 10:30 AM
Session level: 400 - Expert
Session Type: Workshop
Location: Venetian

architecture

Achieving extreme resiliency through recovery-oriented architectures

ARC401
This chalk talk describes application partitioning strategies, failure mitigations, and trade-offs for designing extremely resilient recovery-oriented architectures. Developers can achieve extreme levels of resilience by building redundant application replicas and designing routing policies that can detect and remove impaired application replicas from service: these are called recovery-oriented architectures. Using recovery-oriented architectures, service owners can make changes one replica at a time and can immediately remove a replica from service when it experiences an impairment. When application replicas are aligned to AWS fault domains, including Availability Zones or AWS Regions, infrastructure impairments can be contained to an individual application replica and applications recove...
See more

Date: Monday, November 28
Time: 5:30 PM - 6:30 PM
Session level: 400 - Expert
Session Type: Chalk Talk
Location: Wynn

Proactive auto scaling for optimal cost and availability

CMP412
Amazon EC2 Auto Scaling groups allow you to extract the elasticity benefits of the AWS Cloud. In this workshop, learn how to make the most of the latest innovations from Amazon EC2 Auto Scaling to further improve your web application availability at lower costs. Specifically, dive into using a combination of predictive scaling, dynamic scaling, and warm pool features to automatically launch and terminate capacity with changing demands throughout a day. With more responsive and proactive scaling, you run only the required number of instances at any time of the day, reducing the cost of overprovisioned Amazon EC2 instances. You must bring your laptop to participate.

Date: Wednesday, November 30
Time: 5:30 PM - 7:30 PM
Session level: 400 - Expert
Session Type: Workshop
Location: Venetian

Concepts to take your Kubernetes operations and scale to the next level

CON402
In this workshop, learn about Kubernetes best practices and dive into the critical concepts, tools, and configurations necessary to ensure application health. Gain hands-on experience securing, scaling, and observing Kubernetes to identify, debug, and improve overall performance. You must bring your laptop to participate.

Date: Thursday, December 1
Time: 2:00 PM - 4:00 PM
Session level: 400 - Expert
Session Type: Workshop
Location: MGM Grand

Advanced VPC design and new Amazon VPC capabilities

NET302

This is must see if you studying for a the SA Pro Exam

Amazon VPC gives you more control over your AWS virtual networking environment. Given this ability, have you ever wondered how new Amazon VPC features might affect how you design your AWS networking infrastructure or change your current architecture? In this session, learn about the new design and capabilities of Amazon VPC and how you might use them.

Date: Friday, December 2
Time: 8:30 AM - 9:30 AM
Session level: 300 - Advanced
Session Type: Breakout Session
Location: Level 1, Forum 121, Caesars Forum

A day in the life of a billion requests

SEC404
Every day, sites around the world authenticate their callers. That is, they verify cryptographically that the requests are actually coming from who they claim to come from. In this session, learn about unique AWS requirements for scale and security that have led to some interesting and innovative solutions to this need. How did solutions evolve as AWS scaled multiple orders of magnitude and spread into many AWS Regions around the globe? Hear about some of the recent enhancements that have been launched to support new AWS features, and walk through some of the mechanisms that help ensure that AWS systems operate with minimal privileges.

Date: Wednesday, November 30
Time: 10:45 AM - 11:45 AM
Session level: 400 - Expert
Session Type: Breakout Session
Location: Venetian

AWS Nitro: Enhancing security and future-proofing your instances

CMP404
AWS has completely reimagined the virtualization infrastructure with the AWS Nitro System, which provides a rich collection of building blocks that offload many traditional virtualization functions to dedicated hardware and software to deliver high performance, high availability, and high security while reducing virtualization overhead. In this chalk talk, dive deep into the Nitro System and discover how AWS extended the Nitro System’s modern hardware and software so organizations can run their workloads well beyond the typical lifetime of the underlying hardware.

Date: Wednesday, November 30
Time: 10:45 AM - 11:45 AM
Session level: 400 - Expert
Session Type: Chalk Talk
Location: Grand

Enhancing OS and application security on AWS Graviton–based instances

CMP409
AWS Graviton processors feature key capabilities that enable you to run cloud-native applications securely and at scale. AWS Graviton3 processors feature always-on memory encryption, dedicated caches for every vCPU, and support for pointer authentication. In this chalk talk, dive deep into how developers can build secure applications using pointer authentication.

Date: Wednesday, November 30
Time: 2:30 PM - 3:30 PM
Session level: 400 - Expert
Session Type: Chalk Talk
Location: Caesars Forum

Enabling multi-party analysis of sensitive data using AWS Nitro Enclaves

CMP403
In this chalk talk, learn how to utilize AWS Nitro Enclaves, take advantage of its unique capabilities, and integrate with services such as AWS KMS. Use these options to enable collaboration and multi-party computation of sensitive datasets in a secure manner, without revealing the dataset of any party to another.

Date: Wednesday, November 30
Time: 3:15 PM - 4:15 PM
Session level: 400 - Expert
Session Type: Chalk Talk
Location: Caesars Forum

Building modern data architectures on AWS

ARC313
The modern data architecture is an evolution from data warehouse and data lake–based solutions, and it allows you to query data across your data warehouse, data lake, and purpose-built analytics services to gain faster and deeper insights that would not be possible otherwise. In this session, learn how to design, create, and operate a modern data architecture on AWS by using AWS purpose-built data services. Also learn about modern data architecture concepts, why customers are adapting to it, modern data architecture pillars, reference architectures, and best practices while building modern data architectures on AWS for optimal performance and cost effectiveness.

Date: Friday, December 2
Time: 9:15 AM - 10:15 AM
Session level: 300 - Advanced
Session Type: Breakout Session
Location: Caesars Forum

Canary deployment with metrics-driven rollback ability for Amazon ECS
CON409
In this builders’ session, learn about the importance of deployment strategies and how to implement a CI/CD strategy using AWS CodeSuite services for performing canary deployments on Amazon ECS, with AWS App Mesh for communication between the microservices. The session also covers what can be achieved with AWS CodeDeploy and the differences in our reference architecture, which does not require ELBs for internal communication. You must bring your laptop to participate.

Date: Thursday, December 1
Time: 3:30 PM - 4:30 PM
Session level: 400 - Expert
Session Type: Builders' Session
Location: Caesars Forum

Serverless

Best practices for advanced serverless developers

SVS401
Are you an experienced serverless developer? Do you want a helpful guide for unleashing the full power of serverless architectures for your production workloads? Are you wondering whether to choose a stream or an API as your event source, or whether to have one function or many? This session provides architectural best practices, optimizations, and useful cheat codes that you can use to build secure, high-scale, and high-performance serverless applications and uses real customer scenarios to illustrate the benefits.
Reservations available soon

Date: Tuesday, November 29
Time: 12:30 PM - 1:30 PM
Session level: 400 - Expert
Session Type: Breakout Session
Location: Mandalay Bay

Automated testing for serverless applications

COM401
During this dev chat, learn briefly about the theory behind testing serverless applications in the cloud, using real AWS services instead of locally emulated resources. The chat covers common and advanced topics, such as how and when to write integration, and when end-to-end tests; how to test asynchronous workflows; and how to configure your serverless project so that tests can be fully automated. Bring your questions for answers using suitable examples, including code samples and demos.

Date: Wednesday, November 30
Time: 12:30 PM - 1:00 PM
Session level: 400 - Expert
Session Type: Dev Chat
Location: Venetian

Analytics

How Poshmark accelerates growth via real-time analytics & personalization

ANT342
In this session, learn how Poshmark is focusing on achieving top-line growth through personalization and enhanced user experience. The initial approach of using batch processing for personalization and security did not meet expectations for customer experience improvement. To enhance customer experience, Poshmark designed real-time personalization using real-time event capture with Amazon MSK and real-time data enrichment with Amazon Kinesis Data Analytics. Amazon SageMaker and Kinesis Data Firehose supported real-time personalization by saving copies of the personalizations to Amazon S3. This solution improved customer experience, reduced security risks, and allowed end users to more confidently interact with the Poshmark app.

Date: Friday, December 2
Time: 8:30 AM - 9:30 AM
Session level: 300 - Advanced
Session Type: Breakout Session
Location: Caesars Forum

Build a web-scale application with purpose-built databases & analytics

DAT401
In this workshop, learn to build modern web applications at scale using purpose-built databases. Discover how to apply development patterns using Amazon DynamoDB, Amazon ElastiCache, Amazon Neptune, and Amazon OpenSearch Service to build a fully functional and scalable bookstore ecommerce application and dive deep into best practices along the way. Basic familiarity with AWS concepts and services such as IAM, VPC, networking, and storage services is recommended. You must bring your laptop to participate.

Date: Monday, November 28
Time: 10:00 AM - 12:00 PM
Session level: 400 - Expert
Session Type: Workshop
Location: Caesars Forum

Event detection with Amazon MSK and Amazon Kinesis Data Analytics

ANT403
In this workshop, take on the role of a technology manager for a Las Vegas casino. Your assignment is to create a stream processing application that identifies customers entering your casino who have previously gambled with large sums of money and sends you a text message when big spenders sit down at a gaming table. To do this, use Amazon MSK to capture events, Amazon Kinesis Data Analytics Studio to detect events of interest, and AWS Lambda with Amazon SNS to send you a message for any events. You must bring your laptop to participate.

Date: Monday, November 28
Time: 10:00 AM - 12:00 PM
Session level: 400 - Expert
Session Type: Workshop
Location: Wynn

Deliver a connected travel experience with distributed data mesh

TRV402
As more travel and hospitality businesses migrate to the cloud, there’s an opportunity to integrate global datasets to offer travelers and guests a seamless, connected experience across the businesses they interact with on a single journey. Learn how your organization can interact with trusted third-party entities while protecting customer data and adhering to privacy regulations by synchronously and asynchronously serving your data as a product in a distributed, decentralized, and multi-cloud data mesh.

Date: Monday, November 28
Time: 1:45 PM - 2:45 PM
Session level: 400 - Expert
Session Type: Chalk Talk
Location: Caesars Forum

Human vs. machine: Amazon Redshift ML inferences

ANT402
In this builders’ session, use Amazon Redshift machine learning (ML) capabilities to draw inferences for a speed race between humans and machines. A training dataset from previous races is provided for you to build and train an ML model. Use simple SQL to build the ML model and then draw inferences using the inference dataset provided. You must bring your laptop to participate.

Date: Tuesday, November 29
Time: 11:00 AM - 12:00 PM
Session level: 400 - Expert
Session Type: Builders' Session
Location: Caesars Forum

Other

How Amazon uses AWS IoT to improve sustainability across its buildings
IOT204
Sustainability has quickly become a top operational priority for companies managing large real estate portfolios. A key challenge companies face is deciding how to make their efforts more data-driven, starting at the edge. This means rethinking the IoT layer for a new generation of cloud-based insights that can scale to meet the needs of companies that manage millions of square feet of physical space. In this session, see how Amazon uses AWS IoT services in its physical locations to scale sustainability efforts and remove blockers that impede data-driven decision-making.

Date: Friday, December 2
Time: 11:30 AM - 12:30 PM
Session level: 200 - Intermediate
Session Type: Breakout Session
Location: Caesars Forum

Take these open-source tools on your AWS adventure
BOA202
You’ve set out on a grand adventure to learn, build and expand on AWS. Like any good adventure, it has its challenges. Time to gear up! Grab your best tools and gear to help you on your way. In this session, have a look at open-source tools that can help make your AWS adventure easier. See something for security and permissions, something for cost management, and a few more things for building in the cloud—tools like Infracost, IAMLive, and more.

Date: Friday, December 2
Time: 11:30 AM - 12:30 PM
Session level: 200 - Intermediate
Session Type: Breakout Session
Location: Venetian

See you there

For this going to reinvent I hopes to see you their and DM on twitter (@philipbasford) and lets meet up

Fish Cam : Live water temperature monitoring

philbasford — Sat, 02 Apr 2022 10:08:11 +0000

In this blog I will be looking into how to implement live monitoring of water temperature inside the Fish tank where my daughters two goldfish, Goldie and Star, spend their days swimming in. The tank is located next to the desk that I am writing this blog from.

Further background:

My "Office" (aka my daughters bedroom) is located at the front of our house. Our house is north facing, therefore in the morning the sun is on the back of the house and in the afternoon it is on the front where my "Office" is. My "Office has a window that I look out from when I am working from home. In late afternoon the sun especially shines straight in this window (you may see this on some of the YouTube videos that I have done).

The main thing here is the safety of the Fish. They ** must not** be put in direct sunshine, this would heat the water too much and also increase the algae due to the sunlight. So an extra safeguard I typically close the curtains when the sun is very bright or strong. However the ambient temperature of the room still increases and this means the water temperature of the tank also increases. Therefore it would be good to know the water temperature as Goldfish require a range of between 20°C and 23°C.

Hardware:

To measure water temperature you need to invest in some additional components to extend your RaspberryPI. Both a submersible temperature Sensor like the DS18B20 (’https://www.amazon.co.uk/dp/B08HHWD1K9) and a breadboard is essential for this. The DS18B20 is ideal as it has an integrated digital interface where you can read straight from it and you do not need to fiddle with an analog converter.

To setup the hardware you need to do the following:

Shutdown your Pi and unplug it
Connect your breadboard to the GPIO on the Pi
Connect the sensor to the breadboard.
Start up your Pi

Here is a wiring diagram that helps, thanks to Circuit Basics:

Some tips:

A pull-up resistor of at least 4.7 Ohm is required. This basically allows the sensor's reading to be between a value of 0 and 5 volts and that the digital interface contained within the housing can understand it. The PI then reads this as a temperature from the digital interface. I had to use a resistor from my breadboard kit and this website will help you workout what Ohm is your resistor is https://www.calculator.net/resistor-calculator.html. If you need use a couple of resistors then make sure you wire them in series.
Notice the sensor above comes with a connector at the end. For my first attempt to wire this up, I cut off the connector, then I stripped the wires and tried push them into my breadboard. This was a mistake and did not work. Luckily my breadboard came with a single row pin header. This allowed me to put the row pin header into the board and connector on to the row pin header. Much easier but make sure you place the single row pin header horizontally not vertically.

My final breadboard wiring looked like this:

Turning on and setting up the sensor:

You will need to turn on the digital(one-wire) interface on the PI (GPIO 4). This can be done using the raspi-config.

Select the Interfaces Menu:

Select one-wire:

Turn on one-wire:

Then restart your Pi

Testing

Using cat then we can see if we can read the temperature value straight from the digital interface(one wire). One-wire mounts devices at /sys/bus/w1/devices/. This folder should contain another folder with the name of the UUID of this sensor. This is in case you have more than one and each one would have its own folder. With in the sensor's folder then there is a file called 'temperature' and it contents is the °C value from the sensor.

Here is an example:

ssh pi                       
Linux raspberrypi 5.10.17-v7+ #1414 SMP Fri Apr 30 13:18:35 BST 2021 armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Apr  2 10:53:44 2022 from 192.168.x.x
pi@raspberrypi:~ $ ls /sys/bus/w1/devices/
28-00000001a7df  w1_bus_master1
pi@raspberrypi:~ $ cat /sys/bus/w1/devices/28-00000001a7df/temperature 
20625
pi@raspberrypi:~ $

So that is 20.625c.

Lambda:

In order to get the temperature from the one wire interface into AWS we need to write some custom code. To do this I used a lambda written in Python. The lambda reads the temperature from the interface like any other normal file. The value is then parsed and turned into a JSON object. Then using the publish option in the Greengrass SDK I send the message via an MQTT topic into IoT core. Here is some very important concerns/notes when doing this:

The lambda needs to be a long running lambda. This basically means it is not a lambda at all. It is in fact a Python app that is deployed via GreenGrass and started when Greengrass starts. It needs a run loop, poll the temperature and publish. Listening to sigterm and sigint
It needs to run as a container with read access to /sys on the host.
The Greengrass v2 and v1 way off sending MQTT messages and how a deployment is done is completely different. My code was originally v1 so I had to deploy the LegacySubscriptionRouter with the correct configuration to make it compatible
I had to do a OTA upgrade of nucleus so that my other components were deployable

Here is the main function executed within the run loop:

def process(iot_client: Client):

    # conh
    sensor_id = os.getenv('TEMP_ID', '28-00000001e2d1')
    path = os.getenv('TEMP_PATH', '/sys/bus/w1/devices/')

    temperature = gettemp(sensor_id, path) / float(1000)
    msg = {
        'sensor_id': sensor_id,
        'temperature': temperature
    }

    send_message(iot_client, msg)

    return msg

This is how to read the temperature:

def gettemp(sensor_id, path):
    try:
        mytemp = ''
        filename = 'w1_slave'
        f = open(path + sensor_id + '/' + filename, 'r')
        line = f.readline()  # read 1st line
        crc = line.rsplit(' ', 1)
        crc = crc[1].replace('\n', '')
        if crc == 'YES':
            line = f.readline()  # read 2nd line
            mytemp = line.rsplit('t=', 1)
        else:
            mytemp = 99999
        f.close()

        return int(mytemp[1])

    except Exception as e:
        LOGGING.exception(e)
        return 99999

This is how to send the message via MQTT (note this is V1 style):

def send_message(iot_client: greengrasssdk.IoTDataPlane.Client, message: dict, topic=TOPIC):

    iot_client.publish(
        topic=topic,
        payload=json.dumps(message).encode("utf-8")
    )

Now if you want to look at code in more detail it is located [https://github.com/philbasford/fishcam] and also note I deployed this using AWS SAM.

IOT Component:

First select the Fish Cam Lambda and use the lambda name and version as the initial component details. However you may need to update the version later if you need to reconfigure.

As mention before one of the most important things is that the lambda is pinned/long running, so that it starts up and down with Greengrass. Also remove any event source:

Make sure you use a container but with access to /sys:

Create IOT Deployment:

Create a new deployment:

Add the Lambda component you created, then the legacy subscription router and nucleus public components.

Now go into the legacy subscription router configuration:

Add the following(change the function name to match yours):

{
    "subscriptions": {
        "Greengrass_FishCam_to_cloud": {
            "id": "Greengrass_FishCam_to_cloud",
            "source": "component:cfFishCam-FishCamFunction-R7vEIfJai8vl",
            "subject": "fishcam/temperature",
            "target": "cloud"
        }
    }
}

Then submit it:

Leave the Advance Settings as is:

Then deploy:

Now if it deployed ok, then within the test area of IOT Greengrass section of the AWS Console you should see the messages coming in:

IOT Rule & CloudWatch

To get the messages into CloudWatch you need to create a new rule in the Act area of IOT Greengrass section of the AWS Console:

Use a the following SQL to make sure you process all the messages sent to the topic:

The add action and select CloudWatch Metric:

Then use the following settings with substitutions for the values contain in messages:

Your need a IAM Role and here are the permissions needed:

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "cloudwatch:PutMetricData",
        "Resource": [
            "*"
        ]
    }
}

Conclusion:

Measuring temperature of the water in the fish tank is really easy, I was able to do this in few hours once I had purchased the additional sensor. It is a good example of how to chain a lot AWS services together to make something useful at low cost.

Here is the final CloudWatch chart for the temperature for a a day, notice the as the sun comes up and goes down the temperature changes:

Thank you and I hope you found this interesting. Next time I will be looking at how to use machine learning to spot fish!

re:invent 2021 Sessions

philbasford — Thu, 04 Nov 2021 22:51:00 +0000

It is back and it is in person, it is re:invent! Now comes the biggest challenge, working out where to go and which sessions to catch. Therefore I have put this re:invent session guide/recommendations list together so that if your are into ML Ops, ML Architecture, Edge Computing or Data Analytics then it can act as a starting point. However please leave sometime in your schedule because as AWS announce new stuff they will add more sessions.

Hang on, what if you are not going, then your are like me. I have to stay home for some personal reasons. However you can still catch the keynotes and leadership sessions on the live stream. Do not forget the big three keynotes; Adam Selipsky (taking over from Andy), Werner Vogels (always dev centric), Swami Sivasubramanian (all ML). Later you can also use this guide to watch the Breakout Session once AWS have made them available on demand.

Sessions

Session ID	Title	Type	LINK
AIM320	Implementing MLOps practices with Amazon SageMaker	Breakout Session	Invite
AIM407	Train deep learning models at scale with Amazon SageMaker	Breakout Session	Invite
AIM405	Right-sizing Amazon SageMaker compute instances for ML model training and inference	Chalk Talk	Invite
AIM413	Detect machine learning model drift in production	Chalk Talk	Invite
DOP211	Building scalable machine learning pipelines	Chalk Talk
ARC323	Designing Well-Architected machine learning workloads	Chalk Talk
KYN003	Swami Sivasubramanian Keynote	Keynote
AIM401	Create, automate, and manage end-to-end ML workflows at scale	Workshop	Invite
IOT306	Building a people counter with anomaly detection using AWS IoT and ML	Workshop

Must sees

Out of these 10, my highlights and must see list, that I can not wait to catch is:

Implementing MLOps practices with Amazon SageMaker : ML Ops is not all about pipelines, however they are one of the key technology enablers. This session is looking at Amazon SageMaker pipelines and how to rapidly deploy your models
Detect machine learning model drift in production : One of my personal passions I speak about is the 360 view of a model. This is the business impact, the ML performance (accuracy, etc), System Performance, Inference Observability, and Cost. This session will look in detail at ML performance and detecting drift
Designing Well-Architected machine learning workloads : AWS has just released an update to the Well Architected ML Len and this session will cover all the best practice about how to lower your costs, keep your security rock tight, cope with failures, scale to millions of predictions and keep a watchful eye.

Fun!

So what about the parties and the fun! If your in Vegas checkout:

Midnight Madness
re:Play
Amazon’s World Famous Chicken Wing Eating Contest

If you are staying home:

Checkout your local user group. I know some are doing local watch parties! thats a nice way to still feel part of it.
Another fun thing also I know is happening is ComSum (Community Summit UK) www.comsum.co.uk will be back in action. ComSum will be giving you a byte size digest of the days big news and expert punditry on everything happening in Vegas. This will include myself returning as one of their ML and Data Pundits. More details soon!

Lastly, Don't worry I will be at re:invent next year, so watch out Vegas!

Fish Cam : Lights, Camera, Swim!

philbasford — Wed, 20 Oct 2021 12:12:35 +0000

So far in this series we have looked at the general background and how to get your Raspberry PI setup. Now it is time to have a little more fun with what you can do with a Web Cam, a Raspberry PI and AWS. For me, my initial fun was creating a live stream of two Goldfish, Goldie and Star, swimming around their tank. I wanted to capture live footage stream it real-time to anywhere in the world (well where ever my daughter is on my iPhone!)

Initial Web Cam Setup

Firstly plug your web cam into a USB port on your PI. Then via the desktop or remote access, run raspi-config and enable the camera.

I did this via SSH:

ssh pi

sudo raspi-config

This brings up the main menu and please select the Inference Options:

Then the Camera option and then enable it:

Your then your see a message confirming it is enabled (hopefully):

Then your need to reboot:

Once rebooted, then ssh (or whatever you use) to go back into your PI and test it out. To test I used the standard fswebcam util to capture a frame:

sudo apt install fswebcam

fswebcam -r 1280x720 image.jpg

Then open up another connection, copy the file to your local machine and open the image:

scp pi:image.jpg .
image.jpg                                                                                             100%  401KB   5.9MB/s   00:00 
open image.jpg

You should see something or even your self in the image:

If you want more details then please see the official docs

Capturing Video

To stream live video then you need to push the output from the Web Cam to a Kinesis Video Stream. To do this AWS have a C producer lib/SDK that you need to compile on your PI, configure and then run. This uses the GStreamer Framework and here is link to producer project located on GIT Hub:

https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp/releases

Create Kinesis Video Stream

In order to use the producer library first you need to create a stream. You can do this in the console or via the CLI (plus many more ways). I decided to use the CLI s it is simple enough:

aws kinesisvideo create-stream --stream-name FishCam --region us-west-2

If this is successful then the output contain a ARN for the stream and we will use the ARN in the next section. Therefore please take a note of it.

Create IAM User

Next we will need create a IAM user for the library to use to authenticated against AWS. Now to be honest I would prefer to use an assumed role here and we might revisit this later in other articles in the series when using GreenGrass. For now however please create an IAM user and generate an Access Key and Secret Key, but please make sure you lock it down with something like the attach policy using the ARN (we noted down) as the resource:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kinesisvideo:DescribeStream",
                "kinesisvideo:Get*",
                "kinesisvideo:PutMedia"
            ],
            "Resource": [
                "arn:aws:kinesisvideo:us-west-2:123456789011:stream/FishCam/1634380180064"
            ]
        }
    ]
}

Install Prepresquities

The Producer Library is C, so we have to compile and build the library. This requires some prepresquities and you can installvia apt:

sudo apt install pkg-config cmake m4 git

Now to use GStreamer we need to install some more:

sudo apt-get install libssl-dev libcurl4-openssl-dev liblog4cplus-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev gstreamer1.0-plugins-base-apps gstreamer1.0-plugins-bad gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-tools

Clone And Build

Now we can compile and build the Producer Library, this can take sometime so I recommend doing it via a stable SSH or VNC connection.

First clone the project from GIT:

git clone https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp.git
mkdir -p amazon-kinesis-video-streams-producer-sdk-cpp/build
cd amazon-kinesis-video-streams-producer-sdk-cpp/build

Then run CMake and Make. The GStreamer plugin is optional however it is essential for our streaming. So make sure you pass it as an option to CMake as per below:

cmake -DBUILD_GSTREAMER_PLUGIN=TRUE ..
make

Testing it out

So let’s test it out and see if we can get the stream running. First lets check the step-up by running the following:

cd ..
export GST_PLUGIN_PATH=`pwd`/build
export LD_LIBRARY_PATH=`pwd`/open-source/local/lib
gst-inspect-1.0 kvssink

This should result in an output like:

Factory Details:
  Rank                     primary + 10 (266)
  Long-name                KVS Sink
  Klass                    Sink/Video/Network
  Description              GStreamer AWS KVS plugin
  Author                   AWS KVS <kinesis-video-support@amazon.com>

Plugin Details:
  Name                     kvssink
  Description              GStreamer AWS KVS plugin
  Filename                 /home/pi/amazon-kinesis-video-streams-producer-sdk-cpp/build/libgstkvssink.so
  Version                  1.0
  License                  Proprietary
  Source module            kvssinkpackage
  Binary package           GStreamer
  Origin URL               http://gstreamer.net/

However if you see something like the statement below it means something failed and your need to look into the make logs to kind out why. Mostly it is the GStreamer dependancies not being present so check they installed correctly.

No such element or plugin 'kvssink'

Start The Stream

To start the streaming you need to take the IAM users credentials, the Stream name plus AWS Region name and replace the place holders in this command and then run it on your PI:

$ gst-launch-1.0 autovideosrc device=/dev/video0 ! videoconvert ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! 
                x264enc bframes=0 key-int-max=45 bitrate=500 ! video/x-h264,stream-format=avc,alignment=au,profile=baseline ! 
                kvssink stream-name=FishCam storage-size=512 access-key="YOUR_ACCESS_KEY" secret-key="YOUR_SECRET_ACCESS_KEY" aws-region="YOUR_AWS_REGION"

This should result in log that looks like below that tell you bytes streamed:

2021-10-15 22:50:04 [1237316704] DEBUG - fragmentAckReceivedHandler invoked
2021-10-15 22:50:04 [1237316704] DEBUG - postReadCallback(): Wrote 1024 bytes to Kinesis Video. Upload stream handle: 1
2021-10-15 22:50:04 [1237316704] DEBUG - postWriteCallback(): Curl post body write function for stream with handle: PICam and upload handle: 1 returned: {"EventType":"PERSISTED","FragmentTimecode":1634334601102,"FragmentNumber":"91343852333348846620194705670290539091726766250"}

2021-10-15 22:50:04 [1237316704] DEBUG - fragmentAckReceivedHandler invoked
2021-10-15 22:50:04 [1237316704] DEBUG - postReadCallback(): Wrote 303 bytes to Kinesis Video. Upload stream handle: 1
2021-10-15 22:50:04 [1237316704] DEBUG - postReadCallback(): Wrote 296 bytes to Kinesis Video. Upload stream handle: 1
2021-10-15 22:50:04 [1237316704] DEBUG - postReadCallback(): Wrote 696 bytes to Kinesis Video. Upload stream handle: 1
2021-10-15 22:50:04 [1237316704] DEBUG - postReadCallback(): Wrote 427 bytes to Kinesis Video. Upload stream handle: 1
2021-10-15 22:50:04 [1237316704] DEBUG - postReadCallback(): Wrote 871 bytes to Kinesis Video. Upload stream handle: 1
2021-10-15 22:50:04 [1237316704] DEBUG - postReadCallback(): Wrote 551 bytes to Kinesis Video. Upload stream handle: 1

If you see something like below it means your not replaced the placeholders correctly (watch the quoting):

2021-10-15 22:03:50 [1389352032] DEBUG - describeStreamCurlHandler(): DescribeStream API response: 
2021-10-15 22:03:50 [1389352032] INFO - describeStreamResultEvent(): Describe stream result event.
2021-10-15 22:03:50 [1389352032] WARN - curlCompleteSync(): curl perform failed for url https://kinesisvideo.YOUR_AWS_REGION.amazonaws.com/describeStream with result Couldn't resolve host name: Could not resolve host: kinesisvideo.YOUR_AWS_REGION.amazonaws.com
2021-10-15 22:03:50 [1380959328] WARN - curlCompleteSync(): HTTP Error 0 : Response: (null)
Request URL: https://kinesisvideo.YOUR_AWS_REGION.amazonaws.com/describeStream
Request Headers:
    Authorization: AWS4-HMAC-SHA256 Credential=YOUR_ACCESS_KEY/20211015/YOUR_AWS_REGION/kinesisvideo/aws4_request, SignedHeaders=host;user-agent;x-amz-date, Signature=40201726f3815617c82ecdc62d5b9476936d19f81ce6cdda43fb3a05605bfc6c
    content-length: 37
    content-type: application/json
    host: kinesisvideo.YOUR_AWS_REGION.amazonaws.com
    user-agent: AWS
2021-10-15 22:03:50 [1380959328] DEBUG - describeStreamCurlHandler(): DescribeStream API response: 
2021-10-15 22:03:50 [1380959328] INFO - describeStreamResultEvent(): Describe stream result event.
2021-10-15 22:03:50 [1380959328] WARN - curlCompleteSync(): curl perform failed for url https://kinesisvideo.YOUR_AWS_REGION.amazonaws.com/describeStream with result Couldn't resolve host name: Could not resolve host: kinesisvideo.YOUR_AWS_REGION.amazonaws.com
2021-10-15 22:03:51 [1389352032] WARN - curlCompleteSync(): HTTP Error 0 : Response: (null)
Request URL: https://kinesisvideo.YOUR_AWS_REGION.amazonaws.com/describeStream
Request Headers:
    Authorization: AWS4-HMAC-SHA256 Credential=YOUR_ACCESS_KEY/20211015/YOUR_AWS_REGION/kinesisvideo/aws4_request, SignedHeaders=host;user-agent;x-amz-date, Signature=40201726f3815617c82ecdc62d5b9476936d19f81ce6cdda43fb3a05605bfc6c
    content-length: 37
    content-type: application/json
    host: kinesisvideo.YOUR_AWS_REGION.amazonaws.com
    user-agent: AWS

Handy Bash Script

To make things easier I then created the following script to start up the streaming when I needed. This saves me having to remember all the steps:

cd ~/amazon-kinesis-video-streams-producer-sdk-cpp
export GST_PLUGIN_PATH=`pwd`/build
export LD_LIBRARY_PATH=`pwd`/open-source/local/lib

gst-launch-1.0 autovideosrc device=/dev/video0 ! videoconvert ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! x264enc bframes=0 key-int-max=45 bitrate=500 ! video/x-h264,stream-format=avc,alignment=au,profile=baseline !  kvssink stream-name="PICam" storage-size=512 access-key="XXXXX" secret-key="YYYYY" aws-region=us-west-2

More details using the library can be found here :

https://docs.aws.amazon.com/rekognition/latest/dg/streaming-using-gstreamer-plugin.html

Hopefully you got it working and let's leave it running whilst we work on how to view the stream.

Viewing the Live Streaming Video

That all good! but hay we still cannot see Goldie and Star swimming around that lovely tank we got them. Also let's remember the key requirement is seeing them doing their thing from wherever my daughter is on holiday using our tablets or phones. Therefore I decided to use HLS as most modern devices support it and you can uses it within a HTML5 webpage. Luckily Kinesis Video Streams supports HLS and they have a ready made HTML5 page, see here.

To make this work we need to reuse IAM user and hardcode the same credentials into the page. This is not good, so I did a look at Cognito and I did setup a role etc. However it just returned a 403 all the time and it took me a few hours of digging into to find out why. Here the shocker! Authentication via Cognito does not support all aws services see Access Policies in this doc. Plus I have no idea why Kinesis Video Streams is not on that list, especially given Kinesis Data Streams are. So it means we are stuck with the IAM user and hardcoded credentials :-(.

HTML5 client

This resulted in a HTML 5 like the following:

<script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/aws-sdk/2.278.1/aws-sdk.min.js"></script>
<link rel="stylesheet" href="https://vjs.zencdn.net/6.6.3/video-js.css">
<script src="https://vjs.zencdn.net/6.6.3/video.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/videojs-contrib-hls/5.14.1/videojs-contrib-hls.js"></script>

<video id="video"  controls autoplay></video>
<script>

  var protocol = 'HLS';
  var streamName = 'FishCam';

  // Step 1: Configure SDK Clients
  var options = {
    accessKeyId: 'AAAAAAAAA',
    secretAccessKey: 'BBBBBBBBBB',
    region: 'us-west-2'
  }
  var kinesisVideo = new AWS.KinesisVideo(options);
  var kinesisVideoArchivedContent = new AWS.KinesisVideoArchivedMedia(options);

  // Step 2: Get a data endpoint for the stream
  console.log('Fetching data endpoint');
  console.log(AWS.config.credentials);

  kinesisVideo.getDataEndpoint({
    StreamName: streamName,
    APIName: "GET_HLS_STREAMING_SESSION_URL"
  }, 
  function (err, response) {

    if (err) { return console.error(err); }
    console.log('Data endpoint: ' + response.DataEndpoint);
    kinesisVideoArchivedContent.endpoint = new AWS.Endpoint(response.DataEndpoint);

    // Step 3: Get a Streaming Session URL
    var consoleInfo = 'Fetching ' + protocol + ' Streaming Session URL';
    console.log(consoleInfo);

    kinesisVideoArchivedContent.getHLSStreamingSessionURL({
      StreamName: streamName,
      PlaybackMode: 'LIVE',
      DiscontinuityMode: 'ALWAYS',
      MaxMediaPlaylistFragmentResults: '1000',
      Expires: '30000',
      HLSFragmentSelector: {
        FragmentSelectorType: 'SERVER_TIMESTAMP'
      }
    }, function (err, response) {

        if (err) { return console.error(err); }
        console.log('HLS Streaming Session URL: ' + response.HLSStreamingSessionURL);
        var video = document.getElementById('video');
        if (Hls.isSupported()) {
          var hls = new Hls();
          hls.loadSource(response.HLSStreamingSessionURL);
          hls.attachMedia(video);
          hls.on(Hls.Events.MANIFEST_PARSED, function () {

            video.play(function (err) {
              if (err) { return console.error(err); }
            })
          });
        }
        // hls.js is not supported on platforms that do not have Media Source Extensions (MSE) enabled.
        // When the browser has built-in HLS support (check using `canPlayType`), we can provide an HLS manifest (i.e. .m3u8 URL) directly to the video element through the `src` property.
        // This is using the built-in support of the plain video element, without using hls.js.
        // Note: it would be more normal to wait on the 'canplay' event below however on Safari (where you are most likely to find built-in HLS support) the video.src URL must be on the user-driven
        // white-list before a 'canplay' event will be emitted; the last video event that can be reliably listened-for when the URL is not on the white-list is 'loadedmetadata'.
        else if (video.canPlayType('application/vnd.apple.mpegurl')) {
          video.src = response.HLSStreamingSessionURL;
          video.addEventListener('loadedmetadata', function () {
            video.play();
          });
        }
        else if(playerName === 'VideoJS') {
            var playerElement = $('#videojs');
            playerElement.show();
            var player = videojs('videojs');
            console.log('Created VideoJS Player');
            player.src({
              src: response.HLSStreamingSessionURL,
              type: 'application/x-mpegURL'
            });
            console.log('Set player source');
            player.play();
            console.log('Starting playback');
        }
    })
  })
</script>

Hosting the Webpage

We need somewhere to host our HTML5 client and to be honest I did not want to spend too much money in doing so. Therefore I created a new S3 bucket with relatively meaningful name and then enabled static hosting:

I then named the the page index.html and uploaded it to s3 so that is was served as root page of the web address.

Lastly I took the static hosting URL and placed it my browser and on my phone and tested it out. It can take a few seconds to load but if successfull you should see your fish swimming around or whatever your subject is

Conclusion

This was great fun and works surpassingly well. My daughter was able to view her fish when she has been on holiday in Cornwall and Scotland. I do also have a Philips Hue light bulb in the room so if it is too dark then I can turn the light on as well. I hope you enjoyed this and please get touch if you have any feedback.

Come back soon for more fishy fun!

Fish Cam : Hooking up fish to the cloud

philbasford — Mon, 20 Sep 2021 16:14:32 +0000

Hopefully reading my first blog, Fish Cam : Background, Architecture and Hardware, inspired you to start your own exciting ‘pet’ project or maybe you're just playing around. Either way fantastic, so hopefully your got your PI and USB web cam handy and connected up. Don’t worry about connecting the breadboard to the GPIO, I will handle that in another blog. In this blog we will cover installing an OS, settling up AWS IoT Greengrass and connecting it to AWS Systems Manager.

Basics:

One of best features of my PI that I love is that hard disk is a SD disc. This means that you have to flash it with OS image of your liking. However you can buy a few SD discs for under £10 and this means you can keep swapping them. Allowing you to try out few different options plus setups. neat!

There are many options for an OS on a PI, but they need for an ARM variant of Linux 4.4 kernel for Greengrass (more details on requirements here) [https://docs.aws.amazon.com/greengrass/v2/developerguide/setting-up.html#greengrass-v2-requirements]. Raspberry PI OS, formally Raspbian, is the prime choice so that’s what I used for Fish Cam. Now you have choice on edition; easy, medium and hard:

Raspberry Pi OS with desktop and recommended software: Easy, has lots installed and if you only got one SD plus your are a first timer then it will allow to explore all things PI. However it is very bloated if you just want to do a IoT and/or ML project.
Raspberry Pi OS with desktop: Medium, this edition has everything you need plus a desktop. Some of you might like to use that as I did starting out. However your need some more kit; a monitor, a keyboard and a mouse. It also effects your remote access options (more on that later)
Raspberry Pi OS Lite : Hard, This is for more experienced Linux people as basically you only have a command line and no desktop. I love the command line and all the setup of Greengrass and Systems Manager is done via the command line anyway.

To be honest I said LITE was hard but actually it someways it is the quickest and most straightforward. Therefore I used this edition for Fish Cam. To get your PI up and running then need four things (most in the start kit I recommended in Background, Architecture and Hardware)

The SD card that came with your PI or bought separately.
A desktop or laptop, I have MacBook Pro with only USB C. I will cover the Mac steps in this blog but same roughly works for other platforms
A micro micro SD reader.
Your SD reader might go into a SD port or usb straight but if not an adapter to convert (mine is micro SD to USB 2 so I used a usb 2 -> usb 3 adapter)

Once that’s all plug together and plug into your Mac, then the volume appears Devices on the finder sidebar. If your reusing an SD then your need to format (do not lose anything important in doing so).

Install Raspberry Pi OS

To flash your SD your need to download and run the Raspberry Pi Imager. Selecting the edition you would like and then writing it to the SD.

If your using easy or medium (a desktop edition) then you can do all the network and remote access configuration from the desktop, so eject your SD, put it in your PI, boot your PI and please follow :

https://www.raspberrypi.org/documentation/computers/configuration.html#configuring-networking-2

If your going down the hard edition route your need to create a blank file called SSH with no content and a file called wpa_supplicant.conf at root of your SD (this is the boot partition) containing your WiFi details (see also then guide above):

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
country=<Insert 2 letter ISO 3166-1 country code here, I used GB>
update_config=1

network={
 ssid="<Name of your wireless LAN>"
 psk="<Password for your wireless LAN>"
}

Now sometimes you power management can turn off your network connection when idle. Therefore killing your remote access and not allowing you to bring it out of the idle state. So your need to disable this with:

sudo iwconfig wlan0 power off

Note you may need to persist this for reboots and you can find someways of doing this here.

Now you're ready, so eject your SD, put the SD in your PI and boot it.

Accessing my PI

Now basically there are few ways of connecting your PI:

SSH (hard)

The key is to know your IP so you can connect and for those who took ether of the easy or medium options then this shown to you in the desktop editions in wizard when you first boot. However for the hard option (command line only edition) you should able then scan your network and find your new IP address (I used Fing on my iPhone). Then once you know the IP address of your PI then you will be able connect via SSH. I connected from my MacBook into the PI using the PI user and default password. If this is your first connection, the please change your password and I recommend setting up a ssh key. This will give you slick and secure access, more details here

VNC (easy or medium)

If you went for one of the desktop editions then you can also use VNC for remote access and for this I would recommend the creating a cloud account and registering your PI. This means you can then connect from a VNC Viewer on any device. Including from your iPhone on holiday to restart something when the Web Cam stopped working. See here for more details

Python

I am using Python 3.8 to develop some of the features of Fish Cam within Greengrass and all my sensors work with Python Libs. Therefore I installed it from source (up to you what python version to use):

wget https://www.python.org/ftp/python/3.8.12/Python-3.8.12.tgz -o python.tgz
tar -xzvf python.tgz 
cd Python-3.8.12/
./configure --enable-optimizations
make
sudo make install

Greengrass

Greengrass is AWS technology that connects devices located at the “edge”; devices in remote locations and that could be low powered up to AWS. This allows you to do lots of things like; collecting sensor data and change the state of device, i.e. powering it on or toggling a switch. For Fish Cam will mainly use it for measuring water temperature, PH and Total Dissolvable Solids (more on how to do this in the future).

AWS has two versions of Greengrass, V1 and V2. For Fish Cam it really did not mater which version I used as both support the features we need. However if your using Greengrass for anything else you may wish to check here. Also if your using Greengrass on certified hardware then you need also beware of what major and minor versions are supported.

Anyway for Fish Cam I went with v2 and followed the AWS guides which are very good. To start with I followed Steps 9, 10 and 11:

https://docs.aws.amazon.com/greengrass/v1/developerguide/setup-filter.rpi.html

I then completed “Set up your environment” Step 4 and all of “Install the AWS IoT Greengrass Core software”:

https://docs.aws.amazon.com/greengrass/v2/developerguide/getting-started.html

My setup command was:

sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE \
  -jar ./GreengrassCore/lib/Greengrass.jar \
  --aws-region eu-west-1 \
  --thing-name FishCamv2Core \
  --thing-group-name FishCamv2CoreGroup \
  --thing-policy-name GreengrassV2IoTThingPolicy \
  --tes-role-name GreengrassV2TokenExchangeRole \
  --tes-role-alias-name GreengrassCoreTokenExchangeRoleAlias \
  --component-default-user ggc_user:ggc_group \
  --provision true \
  --setup-system-service true

Systems Manager

AWS Systems Manager is not really a service but a suite of services that are aim at managing the OS, remote management above the infrastructure and remote connectivity of your "instances". Here is a AWS guide in how setup Systems Manager up on your PI and the following picture shows the activation screen I used for activating Systems Manager:

Fleet Manager

New at re:invent 2020, Fleet Manager allows you to interact and manage one premise servers or remote devices from AWS. By installing the Systems Manager agent on my PI it was automatically enrolled into Fleet Manager. Therefore I decided to take a look around it and what it could do for my PI.

This is the first thing to look at is the main dashboard screen, where you can see an overview of your fleet and then drill into each device:

One of the features of Fleet Manager is that you can browse the files system off the device, create/delete directories, and upload/download files:

Another feature is the ability to monitor system performance from the AWS Console:

The last main feature of Fleet Manager is ability to manage OS User and Groups from the AWS Console:

Fleet Manager has two tiers; standard and advance. Advance has the following additional features:

You need use Session Manager to interactively access on-premises instances.
You want to use Patch Manager to patch Microsoft applications hosted on on-premises instances (not needed for Fish Cam).
Need Register more than 1,000 on-premises instances (not needed for Fish Cam).

However the advance tier costs a small amount per hour per instance more, see https://aws.amazon.com/systems-manager/pricing. This can be enabled the settings area in Fleet Manager

Session Manager

Session Manager provides the ability to connect into instances both in AWS and remotely. To connect to remote instances like my PI then your need to enable encryption and advance tier or your get an error like below.

AWS provide some details here and here my guide to it.

KMS

In order connect securely to the instance your first need to create a KMS key and I recommend setting up alias to that easy to reference it and allow for it to rotated, I called mine /fishcam/instances:

KMS Resource Policy

I applied the following key policy to allow the instances role to use the key and for CloudWatch Logs to use it to encrypt the terminal logs:

{
    "Version": "2012-10-17",
    "Id": "key-consolepolicy-3",
    "Statement": [
        {
            "Sid": "Allow access for Key Administrators",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
 "arn:aws:iam::1234567890:role/SandboxLocalAdmin",
               "arn:aws:iam::1234567890:role/FullAdminRole"
                ]
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::1234567890:role/service-role/AmazonEC2RunCommandRoleForManagedInstances"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant",
                "kms:Describe*"
            ],
            "Resource": "*",
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.eu-west-1.amazonaws.com"
            },
            "Action": [
                "kms:Encrypt*",
                "kms:Decrypt*",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:Describe*"
            ],
            "Resource": "*"
        }
    ]
}

IAM role

I also had to add the following policy to the AmazonEC2RunCommandRoleForManagedInstances role (you may wish to restrict them further for production system):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "kms:*",
                "logs:*"
            ],
            "Resource": "*"
        }
    ]
}

Log Group

Once the KMS key is place then you can create the Log Group with the KMS encryption enabled:

Enable Encryption

Then Session Manger can be configured to use the KMS and the Log Group:

Session Manager Shell

Once this is done then Session Manger will be able to connect:

Run Command

Another feature of Systems Manager is the ability to execute Run Books to patch, install or do common tasks using the RunCommand.

Conclusion

I hoping this blog has showed you how to get your PI up and running. Also all the different ways that you can connect to it and basically now you have your own DeepLens clone to play with and do some fun stuff.

If you want to know how to live stream then here is the next blog in the series https://dev.to/aws-builders/fish-cam-lights-camera-swim-5efe

Fish Cam : Background, Architecture and Hardware

philbasford — Mon, 06 Sep 2021 22:16:03 +0000

Fish Cam is a ‘pet’ project of mine, it comes from wanting to allow my daughter to see her two goldfish, Goldie and Star in their thank from wherever she is on holiday. In doing so it provided me with a way to play with some of latest technology and AWS services including IoT, Real-Time Data Ingestion and ML @ the Edge.

Fish Background:

So let’s start with what not do to or assume:

Goldfish are easy pets for kids to look after
It’s very romantic to win them at a funfair
Think that a small round bowl or by buying a tank from the fun fair that it will do.

All of this is basically wrong and any respectable pet store will tell you that. I blame cartoons for these falsities. However Goldie and Star where funfair fish that my daughter won about 3 years ago.

In addition I was worried about us becoming Goldfish owners. This was based on myself having previous experience with goldfish whilst growing up. My sister had some and I remembered that they seemed to die very often. So I was very sceptical that they would survive and I kind of knew we had kit them out properly. So we took decision and down to pet store we went to buy a proper tank, filter and pump.

It transpired from the advice given by the store and online research that this experience of fish dying is very common. The reason most goldfish die is due to two basic things:

The tank is too small and this stumps their growth
The water quality is poor and they suffer disease or suffering, water quality is again related to tank size and regular cleaning

Two very hasty ways two die! Both the owners fault. The reason for this is that goldfish are very “messy” or produce a lot of 💩. Yep it also comes down to the amount of 💩 and how often it is cleaned away. Goldfish need about 35-60 litres (or more) of clean water each, thats a considerable size tank for two fish and it will set you back around £200 in the UK, then your need to clean it roughy every 2-3 weeks. Hence they are not really a low maintenance or child friendly pets. Your pet store will advise all this and in most cases won’t sale you goldfish without an adequate tank and filter.

So what we did was to buy an initial £80 tank (around 45 litres) to get started (that’s still a lot bigger than those round bowls or fairgrounds tanks). Also after a year, once the fish had grown plus we knew what we where doing, then we upgraded to a 65 litre tank (see below).

So my aims of doing the project was also to help monitor the fish and water quality. The fish however gave us their own strong indication that they are happy and healthy fish, as we got eggs this year. Luckily our fish are both female.

Technical Background:

From a young age I have always been into technology and also an inventor of sorts. I am very lucky as lots of my current work is Data, Analytics and ML. However my specific interest in this project and producing a Fish Cam predates this.

In 2017 I worked for another company at the time, leading a cloud migration project and I was lucky enough to go to my first re:invent. It was brilliant and inspiring, lots of what I do today is direct related to those 5 days. For AWS, 2017 was also a big year! And one thing that caught my eye was the DeepLens.

The DeepLens (as shown in the picture above) is a small device with a camera that allows you to run a computer vision model. You can also connect them to the AWS Cloud as an IoT devices running Linux.

Now at re:invent 2017, according to Andy Jassy, you could go home with one if you did a hands on workshop or if you attended any session of ML track you would get a voucher to buy one when they went GA. So I did a session on SageMaker (also a new 2017 service) and waited for my voucher. It never turned up. Even after asking around my AWS contacts I did not get one and further more it transpired that Amazon would not be able to ship them to UK 😭

That was the end of that! Well until 2019, when my wife and daughters gave me a Raspberry PI 3 for my birthday. I had an old HD webcam in my box of cables, so I wondered if I could build my own DeepLens. So this project is how todo it.

Hardware:

To build your on IoT play device / DeepLens clone you can use a lot of different devices but here is what I used:

Raspberry PI 3 or 4 (including case, sd card 32gb or more, flash module, usb power) : Something like Raspberry pi 3 b+ Starter Kit with 32GB Micro SD Card https://smile.amazon.co.uk/dp/B09BFF1PSM/ref=cm_sw_r_cp_api_glt_fabc_BE0VD7TGQBBBRCT7J2HY?_encoding=UTF8&psc=1
A standard USB HD webcam This was the one I had Logitech C920 HD Pro Webcam, Full HD 1080p/30fps https://smile.amazon.co.uk/dp/B006A2Q81M/ref=cm_sw_r_cp_api_glt_fabc_6YSZ5772CNJG7VGSYFXP?_encoding=UTF8&psc=1

Now for future needs you may want to buy a breadboard and some jumpers etc. This will allow you to connect sensors to the PIs GPIO controls and make up circuits. Here is what I purchased (but others will also do)

ELEGOO Upgraded Electronics Fun Kit w/Power Supply Module, Jumper Wire, Precision Potentiometer, 830 tie-points Breadboard for Arduino UNO R3, MEGA, Raspberry Pi, STM32, Datesheet Available https://smile.amazon.co.uk/dp/B01LZRV539/ref=cm_sw_r_cp_api_glt_fabc_MYA29BNRZP9GT0ZJ353W?_encoding=UTF8&psc=1

Lastly:

One hair band: My daughters spare hair band to hold the lid on the case

Features and Architecture

Once I had connected up the components I then built the following solution with them and AWS:

The features are:

Real Time Video Streaming: One of the key needs from my daughter was that she wanted to able to see her fish swimming their tank from where ever she is on her tablet or my iPhone. For this I used a Kinesis Video Steam, A C++ client to capture the video from the USB camera and send it to the stream, then a client written in HTML 5 Client host in S3, connecting to the stream using HTTP live streaming.
Real Time Water Quality: For this I connected a Temperature Sensor DS18B20 to the GPIO interface on my Raspberry PI. I then created an Edge Lambda to read the raw data from the device interface. The lambda then created a JSON message and sent it via IOT Greengrass and IOT Core to a MQTT topic. Then I created an IOT rule for the topic that uses the Cloud Watch Put Metric connector to place the temperature into Cloud Watch. Finally I created a CloudWatch Dashboard that showed the temperature over whatever time period I wanted up to the last minute.

Real Time Fish Detection: The ability to detect the fish swimming around a tank in real time. Hay not really a requirement from my daughter, but I was just playing to see how good Rekognition was. So to do this I connected the existing Kinesis Video Steam to Recognition (Real Time) by a subscription that then push its results into a Kinesis Data Steam that I a created. I then created a Kinesis Firehose Delivery Steam pulling data from the Kinesis Data Steam and to save the results every minute to S3.
Edge Device and Management: A prerequisite for Real Time Water Quality Dashboards was that I needed to set up my Raspberry PI as a Thing! to do this I had to install Java 11, the IoT GreenGrass V2 software, and run the configuration on my PI and connect it to IOT Core.
Remote hands and patching: Lastly I needed the ability to login to the Raspberry PI from a remote location and see what was happening. To be honest VNC cloud works really well for this. Again as this was for fun and for exploring AWS I decided take some advice and also to give the combination of Session Manager in Systems Manager + Fleet Manager (a new toy) ago, therefore I installed the SSM agent on my PI and activated it. This allowed me to access the terminal on the device from AWS and also patch it remotely using Documents and the RunCommand in Systems Manager.

Conclusion

This was a great "pet" project that allowed my daughter to see her fish from anywhere. It also allowed me to explore lots of the capabilities of AWS and linking them together. I am now looking into how to improve the Real Time Water Quality feature to better predict when filters need changing and water cleaning. I also want it to alert me if Goldie and Star are getting too hot (above 26c) or cold (below 20c)

..More blogs to follow that will deep dive into how to do some of the features. However first you need to install your OS and get access therefore see my next blog here

….Finally you can see a demo here https://www.twitch.tv/videos/1135918428

AWS Advanced Networking

philbasford — Sat, 17 Jul 2021 16:20:46 +0000

I recently completed the AWS Advanced Networking Specialism which is focused on core networking principles and the implementation of them on AWS. Here are my notes and also my mega tip for all AWS exams.

Preparation

For my preparation and training I used and did the following:

Took the ACloudGuru course
Done the AWS exam readiness training
I have 6 years AWS experience and have designed and implemented lots of VPC etc
Completed my SA pro (I would say that if you do this your at least 60% on the path to this specialism and the security one)
Completed my DevOps pro, helped a lot on automation
I had the complete luxury about 5 years ago of actually helping to setup Direct Connect
The APN ambassador programme has run some additional training and sessions over the last month and year.
Reading the Direct Connect and VPN docs in detail
10 years ago I built a network using VLANs, VPNs etc in our office at the time

“There is no compression algorithm for experience” but training helps.

Main Subjects

Your going to have to know for this certification:

VPCs And Subnets
Security Groups and NCLs
Route 53
Routing, BGP and how AWS does it
Direct Connect
VPN
CloudFormation (infrastructure elements)

Odd Ball Questions

Now I always prepare for getting questions I don’t know or even getting some wrong, we are all human and you need to make sure you complete the exam. So here is how I deal with them. If I have a complete blank (the ones I just don’t know), then I take a guess, don’t waste time or worry about it. Any hard or unsure ones that I know will take time I have a quick stab at them, “Flag for review”, and then move on. If I have any time left later I will have a better crack at it.

Also be advised that some questions could be trial questions (you get about 5 per exam that are new questions your not marked on). But you won’t know if they are trail ones or not. So I am not saying they where or are but I did get some questions a little outside of the normal. They where:

Transit GW : Most wisdom is that this is too new for the certificate. It is not as it is over 6 months old.
Global Accelerator: something I had not covered but knew just enough
ipam and how to integrate a 3rd party or build your own.

Mega Tip

In exam centres where you take your exam you can ask for and use a wipeboard and a pen for working outs and notes. For online proctoring and if you use Pearson VUE online then there is online whiteboard feature, a little basic like MS paint but good enough. To use this whiteboard option, then it is located on the top bar next to “chat” with moderator. All handy.

So given I can’t do subnet maths in my head and I got hit with two or three questions in first few minutes. Therefore I decided to crack open the online whiteboard and typed out the following table:

YYYY.YYYY.YYYY.XXXX
32	1	Remember the first 4, plus boardcast cannot be used
31	2	Remember the first 4, plus boardcast cannot be used
30	4	Also don’t forget VPCs and Subnets can be only /28. However for security groups, routes and NCL you can use them for non VPC addresses
29	8
28	16
27	32
26	64
25	128
24	256
YYYY.YYYY.XXXX.ZZZZ
23	512
22	1024
21	2048
20	4096
19	8192
18	16384
17	32768
16	65536	Remember /16 is the max you can have for VPC or Subnet. However you can use ones up to /8 for security groups, NCL or routes for non VPC addresses

For rest of the exam I was able to answer questions requiring subnet maths quickly and my head hurt a lot less. You can do the same tip
for a confusion metric for the ML specialism.

Enjoy and if doing this certification good luck

re:invent 2020 Sessions

philbasford — Sat, 21 Nov 2020 15:05:06 +0000

The following are some inital re:invent 2020 Sessions that I recommend into Machine Learning and Data Analytics. I am aiming to few some these and give a wrap up as apart of my www.comsum.co.uk punditry.

Machine Learning:

How to use fully managed Jupyter notebooks in Amazon SageMaker
Implementing MLOps practices with Amazon SageMaker
Reinventing medical imaging with machine learning on AWS
From POC to production: Strategies for achieving machine learning at scale
Secure and compliant machine learning for regulated industries
How to use machine learning and the Data Cloud for advanced analytics (sponsored by Snowflake)
Train and tune ML models to the highest accuracy using Amazon SageMaker
Fast distributed training and near-linear scaling with PyTorch on AWS
Train large models with billions of parameters in TensorFlow 2.0
Choose the right machine learning algorithm in Amazon SageMaker
Interpretability and explainability in machine learning Privacy-preserving machine learning
Build quality ML models easily & quickly with Amazon SageMaker Autopilot
Scaling MLOps on Kubernetes with Amazon SageMaker Operators
Machine learning inference with Amazon EC2 Inf1 instances
Architectural best practices for machine learning applications
Accelerate machine learning projects with pretrained models
Detect machine learning (ML) model drift in production

Data Analytics

How BMW Group uses AWS serverless analytics for a data-driven ecosystem
How FINRA operates PB-scale analytics on data lakes with Amazon Athena
Break down data silos: Build a serverless data lake on Amazon S3
Serverless data preparation with AWS Glue
Run big data analytics faster at lower cost with Amazon EMR
Improving analytics productivity for overwhelmed data teams (sponsored by Matillion)
5 best practices for migrating large analytics systems to AWS (sponsored by Teradata)

Serverless and Containers

AWS Fargate: Are serverless containers right for you?
Getting started building your first serverless web application
Serverless everything: Replatforming for speed and ownership (sponsored by Datadog)
Scalable serverless event-driven architectures with SNS, SQS & Lambda
AXA: Rearchitecting with serverless to accelerate innovation
Power modern serverless applications with GraphQL and AWS AppSync
Best practices for securing your serverless applications
CI/CD for serverless applications
Becoming proficient with serverless application observability
Decoupling serverless workloads with Amazon EventBridge
Best practices for security governance in serverless applications

Using CodeArtifact from SAM

philbasford — Tue, 10 Nov 2020 22:39:18 +0000

Following my blog (https://www.inawisdom.com/amazon/codeartifact-storing-your-dependencies/) I have worked out how you can use private dependancies in CodeArtifact with SAM. The following is a quick guide.

Firstly use the get-authorization-token to get a secure token for CodeArtifact using your AWS CLI and store it in a env var called CODEARTIFACT_AUTH_TOKEN:

export CODEARTIFACT_AUTH_TOKEN=$(aws codeartifact get-authorization-token --domain my-domain --domain-owner 112333322 --query authorizationToken --output text --profile my-profile --region eu-west-1 )

Then create a Pipfile with the CodeArtifact URL but note the use of the CODEARTIFACT_AUTH_TOKEN env var:

[[source]]
name = "pypi"
url = "https://aws:$CODEARTIFACT_AUTH_TOKEN@my-domain-12333222.d.codeartifact.eu-west-1.amazonaws.com/pypi/PrivatePyPi/simple/"
verify_ssl = true

[dev-packages]
pylint = "*"
#awscli = "==1.16.292"
#aws-sam-cli = "==0.40.0"
flake8 = "*"
flake8-print = "==3.1.4"
flake8-logging-format = "==0.6.0"
flake8-builtins = "==1.4.2"
flake8-eradicate = "==0.3.0"
flake8-comprehensions = "==3.2.2"
flake8-breakpoint = "==1.1.0"
flake8-docstrings = "*"
flake8-rst-docstrings = "*"
flake8-blind-except = "*"
pep8-naming = "==0.9.1"
cfn-lint = "==0.27.4"
pytest = "*"
pytest-cov = "==2.8.1"
moto = "*"
bandit = "*"
safety = "*"
twine = "*"

[packages]
pcb_common = "*"

[requires]
python_version = "3.8"

Then use pipenv to create a requirements file:

pipenv lock -r > src/requirements.txt

Then use the requirements.txt to pull the dependancies:

sam build  CommonLayer --template template.yaml --use-container --base-dir .

The result is the ability to pull your private library. Simple as that!