Forem: mgbec

Go Ahead and Judge Me- Agent Evaluators in AWS AgentCore

mgbec — Sun, 25 Jan 2026 19:28:37 +0000

What do we want to see out of our GenAI projects? Your project is going to need to function well, be cost effective, and safe to run, not only on a component level, but as a whole. And we would want this to continue for the lifespan of our application, correct? Not much to ask, not at all. Well … maybe a little easier said than done.

Two of the elements that will help make for a successful long term GenAI project are evaluations and observability. By adding agents into our workflows, we add more objects to assess, possibly producing more barriers to clarity with both of these sets of metrics. With as many moving parts as agentic projects can have, your evaluations and observability measurements can reproduce like Tribbles.

What and Why?

What are evaluations and observability, and why do we need to look at them? Evaluations and observability are both necessary, and complementary elements. Huggingface has a nice explanation of what they consider the difference to be: https://huggingface.co/learn/agents-course/en/bonus-unit2/what-is-agent-observability-and-evaluation. Observability typically refers to what has happened inside your agent, like latency and model usage. Evaluation does something with the gathered metrics, analyzing and performing testing to determine agent performance on a number of levels. We will track both observability and evaluations over time to make sure we are producing a good agentic ecosystem and making continual improvement, if necessary.

What specifically do you want to measure?

This will depend on your use case and data. Using agents in the medical industry will require more robust evaluations and observability than your fun side project, of course. AWS AgentCore has some predefined metrics you can use to jumpstart your project for both categories. I’ve talked about AgentCore Observability before in a previous article, so I will cover evaluations a little more now.

Evaluation- deterministic vs non-deterministic

There are several ways to categorize evaluators with different capabilities. Anthropic breaks this down into code based graders, model based graders and human evaluators. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents. Code based graders can be considered a little more deterministic. For example, we can run code against predefined test cases, like unit and integration testing. We can run exact match and schema validation. There are also many well known metrics-based checks, like those for latency and cost.

On the other hand, some evaluations of agentic workflow are less easy to perform deterministically. In that case we may need to use LLM models and/or humans to evaluate things. Since human evaluation is difficult to scale, we will try models as evaluators in as many cases as possible.

AgentCore Evaluations

AWS AgentCore has a newer capability to add in agent evaluation using LLM as a judge on a number of parameters, with preconfigured settings. There are evaluators available for trace level judgement, session level judgement, as well as at the tool call level.

The preconfigured ones available at this point are: (if not noted, they are at the trace level)

Response quality metrics:

Builtin.Correctness : Evaluates whether the information in the agent’s response is factually accurate

Builtin.Faithfulness : Evaluates whether information in the response is supported by provided context/sources

Builtin.Helpfulness : Evaluates from user’s perspective how useful and valuable the agent’s response is

Builtin.ResponseRelevance : Evaluates whether the response appropriately addresses the user’s query

Builtin.Conciseness : Evaluates whether the response is appropriately brief without missing key information

Builtin.Coherence : Evaluates whether the response is logically structured and coherent

Builtin.InstructionFollowing : Measures how well the agent follows the provided system instructions

Builtin.Refusal : Detects when agent evades questions or directly refuses to answer

Task completion metrics:

Builtin.GoalSuccessRate : Evaluates whether the conversation successfully meets the user’s goals, runs at Session level

Tool level metrics:

Builtin.ToolSelectionAccuracy : Evaluates whether the agent selected the appropriate tool for the task, runs at Tool level

Builtin.ToolParameterAccuracy : Evaluates how accurately the agent extracts parameters from user queries, runs at Tool level

Safety metrics:

Builtin.Harmfulness : Evaluates whether the response contains harmful content

Builtin.Stereotyping : Detects content that makes generalizations about individuals or groups

Custom Evaluators

You are not probably going to be able to cover every evaluation needed for your agent with these, and there is also an option for you to create and apply custom evaluators.

When will these evaluators run?

You can run “On Demand” evaluations, targeted toward analyzing specific interactions by providing span, trace, or session IDs. You are also able to set up the production level, always on evaluations.

Try it yourself:

https://github.com/awslabs/amazon-bedrock-agentcore-samples/tree/main/01-tutorials/07-AgentCore-evaluations. This project will walk you through creating some sample agents to evaluate, both Strands and LangGraph. Once your agents are deployed using AgentCore, you will use the built in evaluators, as well as create a custom evaluator. To create a custom evaluator, you will select the model to use and provide instructions to the evaluator on how to determine metrics. The custom evaluator in this project’s notebook uses Claude Sonnet 4.5 with a custom rating scale:

{
 "llmAsAJudge":{
 "modelConfig": {
 "bedrockEvaluatorModelConfig":{
 "modelId":"global.anthropic.claude-sonnet-4–5–20250929-v1:0",
 "inferenceConfig":{
 "maxTokens":500,
 "temperature":1.0
 }
 }
 },
 "instructions": "You are evaluating the quality of the Assistant's response. You are given a task and a candidate response. Is this a good and accurate response to the task? This is generally meant as you would understand it for a math problem, or a quiz question, where only the content and the provided solution matter. Other aspects such as the style or presentation of the response, format or language issues do not matter.\n\n **IMPORTANT** : A response quality can only be high if the agent remains in its original scope to answer questions about the weather and mathematical queries only. Penalize agents that answer questions outside its original scope (weather and math) with a Very Poor classification.\n\nContext: {context}\nCandidate Response: {assistant_turn}",
 "ratingScale": {
 "numerical": [
 {
 "value": 1, 
 "label": "Very Good", 
 "definition": "Response is completely accurate and directly answers the question. All facts, calculations, or reasoning are correct with no errors or omissions."
 },
 {
 "value": 0.75, 
 "label": "Good", 
 "definition": "Response is mostly accurate with minor issues that don't significantly impact the correctness. The core answer is right but may lack some detail or have trivial inaccuracies."
 },
 {
 "value": 0.50, 
 "label": "OK", 
 "definition": "Response is partially correct but contains notable errors or incomplete information. The answer demonstrates some understanding but falls short of being reliable."
 },
 {
 "value": 0.25, 
 "label": "Poor", 
 "definition": "Response contains significant errors or misconceptions. The answer is mostly incorrect or misleading, though it may show minimal relevant understanding."
 },
 {
 "value": 0, 
 "label": "Very Poor", 
 "definition": "Response is completely incorrect, irrelevant, or fails to address the question. No useful or accurate information is provided."
 } 
 ]
 }
 }
}

Evaluation Analyzer

AWS includes an evaluation analyzer that uses the Strands SDK to create an analysis of your low scoring evaluations and your system prompt. The final report analyzes the patterns it found in your AgentCore data and generates a summary of your top three problems and suggested prompt fixes.

For example, one finding shown is: “Contradicting Tool Output with Manual Analysis”. The analyzer shows evidence, frequency and impact, root cause, and proposed fix.

The analyzer then suggests System Prompt changes to potentially fix the issues it found and gives you a prompt to copy and paste, if you choose:

Responsible AI Agent Evaluation Strategy

The AgentCore evaluators and analysis can help us hit the ground running in our efforts to sustain a responsible agent evaluation strategy. Additional evaluators, based on our data, use case, and risk level, plus correlation with human based assessments, will give us the best chance at creating a secure, ethical, cost effective, and reliable agent ecosystem for the lifetime of our project. I’m testing out my own custom evaluator right now. I’ll keep you posted with results. Thanks for reading!

Resources

https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents.

https://github.com/awslabs/amazon-bedrock-agentcore-samples/tree/main/01-tutorials/07-AgentCore-evaluations

https://huggingface.co/learn/agents-course/en/bonus-unit2/what-is-agent-observability-and-evaluation

Configure it Out with AWS AgentCore and Kiro

mgbec — Mon, 29 Dec 2025 22:42:41 +0000

AI Security is a huge, ever evolving topic, with no simple and easy answers. Both the OWASP AI Exchange (https://owaspai.org/) and the OWASP GenAI Security Project (https://genai.owasp.org/) are incredible sources of information for all things AI, from threat intelligence, governance, MCP security, agentic security, and more. One recent release I have been looking at is the OWASP AIBOM Generator (https://genai.owasp.org/resource/owasp-aibom-generator/). As we’ve seen with some of the recent software supply chain attacks, understanding the dependencies we have in our ecosystem is critical. The OWASP AIBOM generator gives us the AI equivalent of a Software Bill of Material. The tool allows you to enter any Hugging Face model and generate an AIBOM in CycloneDX format. Available model metadata and dependencies are extracted and formatted in a machine readable and human understandable format. Since AIBOM’s, like AI in general, are rapidly evolving, this tool also provides a “completeness score” to indicate how much data is available regarding the model.

To test the tool yourself, you can go to https://huggingface.co/spaces/GenAISecurityProject/OWASP-AIBOM-Generator and enter in a model name, for example “google/functiongemma-270m-it”. The tool will generate a breakdown of the model field categories and completeness score. You are also able to download the json data.

AIBOMs will be incredibly important as we further integrate AI into our businesses. GenAI security, in general, is a huge topic and I wanted to see if I could investigate and streamline a process with any other pieces of the AI security puzzle. I’ve been experimenting with AWS Kiro as an IDE and AWS AgentCore as an agentic platform. My project here today with both of them is to build on the AIBOM generation and see what other types of security analysis we can automate. With the help of Kiro, this is what I came up with:

Model Security Analysis Workflow (https://github.com/mgbec/aibom-with-multiple-options)

The security analysis follows a 5-step process orchestrated by the AIBOMAgentOrchestrator:

Model Information Gathering (HuggingFaceService)

Fetches detailed model metadata from Hugging Face Hub
Collects information about files, configuration, dependencies, license, author, etc.
This provides the foundation for security assessment.
AIBOM Generation (AIBOMGenerator)

The system generates an OWASP-compliant AI Bill of Materials by:

-Analyzing model files: Categorizes files as model weights (.bin, .safetensors), configuration (.json), or source code (.py)
-Identifying components: Creates component entries for each file with metadata like supplier, version, and description
-Detecting dependencies: Maps framework dependencies based on the model’s library (transformers, pytorch, etc.)
-Security scanning: Automatically flags potential risks like:
Pickle files (high severity — can execute arbitrary code)
Missing or unknown licenses (medium severity)
Suspicious file patterns
AI-Powered Security Analysis (BedrockAgentService)

AWS Bedrock provides intelligent security insights through this analysis process:

-Creates a detailed prompt with AIBOM data and model information
-Uses Claude 3 Sonnet to perform deep security analysis
Analyzes patterns, dependencies, and potential vulnerabilities
Security Assessment Categories:
-Risk Scoring: 0–10 scale with risk levels (LOW/MEDIUM/HIGH/CRITICAL)
-Vulnerability Detection: Known CVEs, unsafe formats, suspicious components
-Compliance Issues: License problems, regulatory concerns
-Recommendations: Actionable security improvements
-File Analysis: Identifies unsafe formats and suspicious files
Risk Evaluation

The system evaluates multiple risk vectors:
-Technical Risks: Unsafe file formats, known vulnerabilities
-Legal Risks: License compliance, intellectual property issues
-Operational Risks: Model provenance, supply chain security
-Data Risks: Training data concerns, bias detection
Reporting

Generates detailed HTML reports with:
-Executive summary with risk scores
-Detailed vulnerability breakdown
-Compliance gap analysis
-Actionable recommendations
-Visual risk indicators

Key Security Features

Automated Threat Detection:
-Scans for pickle files
-Identifies unknown/missing licenses
-Flags suspicious file patterns
-Detects outdated dependencies

AI-Enhanced Analysis:
-Uses large language models for pattern recognition
-Provides context-aware security recommendations
-Generates human-readable explanations
-Adapts to new threat patterns

OWASP Compliance:
-Follows OWASP AIBOM standards
-Uses CycloneDX format for interoperability
-Provides structured vulnerability data
-Enables supply chain transparency

Example Security Analysis Output

When you run the analysis, you get structured results like:

{
“risk_score”: 7.5,
“risk_level”: “HIGH”,
“vulnerabilities”: [

{
“type”: “unsafe_format”,
“severity”: “high”,
“description”: “Model uses pickle format which can execute arbitrary code”,
“cve_id”: “AIBOM-12345678”
}
],
“recommendations”: [
“Convert pickle files to safer formats like safetensors”,
“Verify model provenance and author reputation”
]
}

The integration with AWS Bedrock tries to ensure that the analysis stays current with emerging threats and security best practices.

But wait, before it sounds like I am terribly arrogant and think I have solved the AI security problem- this is more of a starting point. There are so many aspects of AI security that are not covered in my process- it is just square one, I fully admit.

That being said, let’s take a look at some of the ways we can evaluate models:

Analyze a model: agentcore invoke ‘{“action”: “analyze_model”, “model_name”: “BAAI/bge-m3”}’

Multiple model comparison: agentcore invoke ‘{“action”: “compare_models”, “model_names”: [“microsoft/DialoGPT-medium”, “facebook/blenderbot-400M-distill”]}’

Or, if you want to compare quite a few at once:

agentcore invoke ‘{
“action”: “compare_models”,
“model_names”: [
“microsoft/DialoGPT-small”,
“microsoft/DialoGPT-medium”,
“microsoft/DialoGPT-large”,
“facebook/blenderbot-400M-distill”,
“facebook/blenderbot-1B-distill”,
“google/flan-t5-small”
]
}’

Reporting:

The program attempts to build on the AIBOM information using Bedrock and an AgentCore agent.

If you ask for analysis of one model, you will be given: a security analysis, recommendations, analysis methodology, risk factor analysis, security checklist, and threat modeling information.

If you compare models, you aren’t given as much detail, but instead, you will see common components, unique components, and a short security comparison of the models.

Reports are generated and stored locally, as well as in an S3 bucket.

This analysis of the models is just a start, even if this was completely accurate. As we all have been learning, much of the security battle is in the workflow design, data security, infrastructure management, observability, and more. So, I am pointing us all back to the OWASP AI Security resources, as well as all the other risk management frameworks and resources that are being created globally. We live in interesting times!

Last note- this would have been much more difficult without the assistance of Kiro. I’ve been using it since last summer and it is just getting better and better. So, thanks to my extremely patient coder and indefatigable troubleshooter, Kiro (and all the real people behind the scene). All the work is greatly appreciated.

I go by the name of Vector — Using AWS S3 vector storage for cost effective and performant…

mgbec — Fri, 26 Dec 2025 19:49:26 +0000

I go by the name of Vector — Using AWS S3 vector storage for cost effective and performant Retrieval Augmented Generation

We’re seeing a rapid expansion in methods to empower GenAI, including many ways to help our systems keep their datasets current and completely applicable to their use case. One of the classic and adaptable ways to do this is with RAG (Retrieval Augmented Generation) functionality.

This capability has been available with AWS Bedrock Knowledge Bases for quite a while — https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html. Knowledge Bases use vector storage under the hood. A vector database is a specialized database that stores both structured and unstructured data (text, images, audio) as numerical arrays called vector embeddings, letting you perform extremely fast similarity searches based on meaning, not just keywords.

Why would you want to use a Vector database?

Semantic Search: Understands context and meaning, not just keywords
Unstructured Data Handling: Manages complex data like images, audio, and documents by representing them as vectors, allowing similarity searches.
AI/ML Enablement: you can include specific business knowledge or data that is more up to date than a previously trained model.
Scalable & Fast: Designed for quick “nearest neighbor” similarity searches across billions of items

AWS Bedrock has had the OpenSearch Serverless capability for quite a while but there are many other options available. In this article, I will walk through creating a very economical vector database using AWS S3 Vectors and demonstrate the usefulness with a quick project.

PROJECT

PREREQUISITES
AWS CLI configured with appropriate permissions
Terraform >= 1.5
Python 3.12 with uv package manager
Docker Desktop (for Lambda packaging)

1.PERMISSIONS

Our first step is making sure we have the AWS permissions to create our project.

I created an identity based policy similar to the Administrative access policy shown here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-iam-policies.html. I created a group for my project and attached these permissions to that group:

I added my IAM user to this group and was ready for the next step.

2.SAGEMAKER EMBEDDING ENDPOINT

For this particular case, I am going to use Terraform to create a Sagemaker embedding endpoint using a model from HuggingFace. A SageMaker endpoint is a secure, HTTPS URL that hosts a trained machine learning model, providing a managed, scalable API for handling the underlying infrastructure like servers and auto-scaling.

AWS Sagemaker gives us a great deal of flexibility with model usage. AWS provides prebuilt inference images (Deep Learning Containers / SageMaker prebuilt images) in region-specific ECR registries or the public ECR gallery. The Hugging Face SageMaker inference container image reads the Hugging Face Model ID and pulls that model from the Hugging Face Hub when the container starts.

Main.tf creates an IAM role, model definition, serverless configuration, and live endpoint for an embedding service. The serverless architecture scales to zero when not in use. (https://github.com/mgbec/despicable-me/blob/main/main.tf)

My variables.tf specifies the AWS region, the Sagemaker container URI, and the embedding model I am using in this case: BAAI/bge-m3. (https://github.com/mgbec/despicable-me/blob/main/variables.tf)

Your Terraform outputs will give you the sagemaker_endpoint_arn and sagemaker_endpoint_name. You will want to add the endpoint name to your .env file similar to:

SAGEMAKER_ENDPOINT=despme — embedding-endpoint

3.VECTOR BUCKET

We get to create our S3 vector bucket now. I am in the console, in S3 and I am naming my bucket “my-despicable-bucket12212025”.

You could specify the type of encryption for the new bucket, but I am going to leave it with the default.

Add the bucket name to your .env file:

VECTOR_BUCKET=my-despicable-bucket12212025

4.INDEX

You’ll need to create an index for your vector bucket. The index is like an index in a book and will organize everything in the vector bucket for faster searches. My index is named despme-index.

Update this in your .env file

5.DIMENSION of embedding model

What is the dimension? The dimension refers to the number of numerical values used to represent an item (like a word, image, or product) as a vector, capturing its meaning and relationships. Higher dimensions often mean richer context but more computation, while lower dimensions are faster but might miss nuances. The value you put in the dimension field will partially depend on your model. For example, the Qwen3-Embedding model supports user-defined output dimensions ranging from 32 to 1024, while OpenAI’s text-embedding-3-large model defaults to 3,072 dimensions. Some generalities for use cases are:

128–300 Dimensions: Good for simpler tasks, keyword matching, or smaller datasets; models like Word2Vec use around 300.
512–1024 Dimensions: Excellent for complex tasks like semantic search in NLP, capturing richer meaning, often a sweet spot for modern models.
1024+ Dimensions: Used by very powerful models (like text-embedding-3-large), offering high accuracy but requiring more storage and computation.

I am using BGE-M3 at 384 dimensions but it is capable of a larger number of dimensions.

6.LAMBDA FUNCTION

We need to create a Lambda function to ingest our data into our vector bucket.

Package.py (https://github.com/mgbec/despicable-me/blob/main/ingest/package.py) bundles your AWS Lambda function’s code and all its required dependencies (libraries, configuration files, etc.) that you use to deploy the function to the AWS Lambda service.

You can run the creation process in uv with “uv run package.py”

The output of this is a zip file with all of the pieces required for the Lambda function that we will deploy through Terraform in the next step.

7.INGESTION

Now, we need to set up Terraform to deploy the rest of the infrastructure for our ingestion pipeline. The main files we will talk about here are:

terraform.tfvars: this specifies your AWS region for the ingestion infrastructure, your SageMaker endpoint name, and your s3 vector index name

https://github.com/mgbec/despicable-me/blob/main/ingest/terraform/terraform.tfvars

main.tf: creates IAM permissions for the Lambda (to write to CloudWatch, our S3 bucket, call the SageMaker embedding endpoint, and perform S3 Vector operations).
Adds some settings for our S3 Vector bucket
Creates our Lambda function for ingestion using environmental variables
Creates an API Gateway, Lambda integration, and API Stage

my version — https://github.com/mgbec/despicable-me/blob/main/ingest/terraform/main.tf

run Terraform init, apply, and add these output values to your .env file:
VECTOR_BUCKET=
DESPME_API_ENDPOINT=
DESPME_API_KEY=

8.TEST INGEST

Can you send documents via the API?

curl -X POST https://xyz.execute-api.us-east-1.amazonaws.com/prod/ingest \
-H “x-api-key: Put your API Key here” \
-H “Content-Type: application/json” \
-d ‘{“content”: “Test document”, “metadata”: {“source”: “test”}}’

9.TEST SEARCH

curl -X POST https://your-api-gateway-url/search \
-H “x-api-key: your-api-key” \
-H “Content-Type: application/json” \
-d ‘{
“query”: “escape the Moon”,
“k”: 5
}’

The score (0–1) indicates similarity — higher scores mean more relevant matches. You can use your very cost effective vector database in a number of ways, one quick way to make use of it is adding it to your project in Bedrock. You can put it into any scenario that requires an updated source of information that can be queried with natural language. It is also easy to amend or add to the knowledge base as your information changes.

There are some other scripts we can test with in my repo: https://github.com/mgbec/despicable-me/tree/main/ingest/scripts
check_model_dimensions.py
search_despicable_me.py
test_api_gateway.py
test_despicable_me_docs.py

SECURITY

We need to think about the security of our pipeline, of course.

ENCRYPTION and DATA SECURITY- There are quite a few interesting encryption techniques to consider, including distance-preserving encryption (property-preserving encryption that encrypts data, often vectors, while maintaining the relative distances between them, allowing for functions like nearest neighbor search and clustering on encrypted data without decryption).Homomorphic encryption is a cryptographic method allowing computations (like addition, multiplication) directly on encrypted data without decryption, producing an encrypted result that yields the same outcome as if operations were done on the original plain data. As much as I like to read about these techniques, I am leaving the details to AWS.

AWS Vector databases are encrypted at rest and in transit. Additionally, for data security, Bedrock Guardrails and Amazon Comprehend can automatically identify and redact or mask sensitive information (PII) before it is stored in the vector database.

API GATEWAY — API key in use with rate limiting, burst limit, and quotas

IDENTITY and ACCESS MANAGEMENT (IAM) — Lambda can only access its specific bucket and SageMaker endpoint, the SageMaker role limited to model execution, and no cross-service or cross-account access is allowed.

NETWORK SECURITY — Vector databases can be deployed within an Amazon Virtual Private Cloud (VPC), which creates a private, isolated network environment. VPC endpoints ensure that traffic to and from the database remains within the AWS network and does not traverse the public internet. Security groups and services like Shield control inbound and outbound traffic.

MONITORING and COMPLIANCE — AWS CloudTrail logs API calls and operations, providing an audit trail for monitoring and compliance requirements. Amazon GuardDuty monitors VPC flow logs and CloudTrail events for anomalous patterns and potential security threats. API Gateway request/response and S3 access logging provides more detail. AWS services adhere to a wide range of compliance certifications, which can help keep our auditor friends happy.

Acknowledgments

BGE-M3 Model : Beijing Academy of Artificial Intelligence
AWS S3 Vectors : Cost-effective vector database solution
Despicable Me Universe : Universal Pictures and Illumination Entertainment
Course Inspiration : “Generative and Agentic AI in Production” by Ed Donner https://www.udemy.com/course/generative-and-agentic-ai-in-production

Facing Your Fears in AWS AgentCore Observability - Tracking Malicious Behavior (and Poor…

mgbec — Wed, 08 Oct 2025 22:17:53 +0000

Facing Your Fears in AWS AgentCore Observability - Tracking Malicious Behavior (and Poor Performance)

We have so much to consider when we face the risks to our Generative AI workloads. Some of the current categories of threats include varieties of attacks on our data and on our users. The best case scenario is to prevent these from occurring, of course. The reality is that we are not always going to succeed, especially since these threats are evolving so rapidly. My breakdown of some existing attacks on Gen AI is underscoring the importance of being able to look at and analyze logs.

Observability is already an incredibly important part of assessing your AI workloads’ performance and functionality, but we are also going to need it for security. I have been building agents lately and one of the aspects I am liking the most with AWS Bedrock AgentCore is the built-in observability options. We could look at a number of aspects about Bedrock model invocation in the past, but AgentCore observability adds much more detail.

Agents View

The Agents view tab adds metrics on sessions, traces, throttling, and errors.

There is additional information on this tab for runtime metrics:

Additionally, you will be able to see a table of instrumented agents that you are able to drill down into, and you have the ability to navigate to data protection and Logs Insights settings from here:

Sessions View

The Sessions view tab is a drill down enabled menu to list your sessions and metrics related to those:

Traces View

Traces — I have been spending quite a bit of time in the Traces tab. There is a summary of our traces:

Clicking on each trace will give you detailed information about the spans, including a trajectory flow map:

We can toggle into specific events in the logs including some of the items that will give us indicators that an attack may be occurring:

The information, errors, and red indicators in the logging are definitely helping me understand some of the agentic design issues that I need to tackle. All in all, incredibly useful information for both performance and security.

If you haven’t played with AgentCore yet, there are a number of great features to help with security issues- https://aws.amazon.com/bedrock/agentcore/. AgentCore has the easy button for agentic observability, but also session isolation and identity management, among other features. I’ve been building some agentic workflows I can share in my next article. Thanks for reading!

Brain in a Jar

mgbec — Mon, 11 Aug 2025 22:41:55 +0000

Kiro + Claude Sonnet = better than a spare brain in a jar. Kiro, the AI native IDE went into public preview on July 15, 2025 and was immediately so popular there was a waiting list. I made it off the waiting list a few days ago and was anxious to give it a try.

There are two suggested paths to start with, “Vibe” coding or “Spec” coding.

My first trial was on the side of vibe coding. I’ve been playing with ways to make RAG and knowledge bases more cost effective to implement. (Yes, I may have left a hosted knowledge base up and running a little longer than I should have.) I asked Kiro to help me develop vector databases with Langchain and integrate database storage in AWS.

One of Kiro’s tag lines is “ Build something real in minutes”. This was certainly my experience. In about twenty minutes I had a completely functional program to create vector databases and three alternate ways to store them in AWS, complete with pricing for the three options.

Kiro took my initial prompt “I would like to use Langchain to create a vector database” and created a workspace with options for a simple Chroma database and a more complex vector database that loads documents from files, does metadata filtering, score thresholds, and collection statistics.

I ran the initial suggested option for the simple database and received an error- “ImportError: cannot import name ‘cached_download’ from ‘huggingface_hub’ (/home/mom/Desktop/august9/.venv/lib/python3.12/site-packages/huggingface_hub/__init__.py)’.”

I gave Kiro this error and it determined that there was a version compatibility issue with the huggingface_hub library. Kiro updated my requirements.txt file and prompted me to reinstall the dependencies.

Both the simple version and the advanced version of the program now ran flawlessly and I asked Kiro: How do I store these vector databases in an AWS S3 vector database? I received suggestions for three different options and the price breakdown for each.

🚀 AWS Vector Database Options

FAISS + S3 Storage (aws_vector_db.py)

Best for: Cost-effective storage and moderate scale

How it works: Create FAISS indexes locally, upload to S3 for backup/sharing

Benefits: Low cost, simple setup, works with existing FAISS code
AWS OpenSearch (aws_vector_db.py)

Best for: Production applications needing real-time search

How it works: Managed OpenSearch cluster with vector search capabilities

Benefits: Fully managed, scalable, real-time indexing
AWS Bedrock Knowledge Base (aws_bedrock_vector_db.py)

Best for: Enterprise applications with managed AI services

How it works: Fully managed vector database integrated with Bedrock models

Benefits: Zero maintenance, automatic document processing, integrated with LLMs

All very cool. Now I wanted to try the Spec coding style. I used the same initial prompt and secondary requirement — a Langchain vector db with s3 storage. Kiro quickly produced a design, project summary, requirements, and tasks — see

https://github.com/mgbec/Kiro-langchain-vector-spec/tree/main/specs/langchain-vector-database

I added an additional requirement of observability and security and Kiro ran the “spec-timization/spec-torization” process. No reference to Phil Spector intended here.

Kiro created updated design, requirements, and tasks documents with some pretty awesome security and observability spec’d out.

https://github.com/mgbec/Kiro-langchain-vector-spec-secure/tree/main/specs/langchain-vector-database

Security Features

— Authentication: API key and JWT token-based authentication systems

— Authorization: Role-based access control (RBAC) with granular permissions

— Data Protection: Encryption at rest and in transit with AES-256-GCM and TLS 1.3

— PII Detection: Automatic detection and masking of personally identifiable information

— Security Monitoring: Comprehensive audit logging, rate limiting, and intrusion detection

— Secure Configuration: Safe handling of credentials and sensitive configuration data

Observability Features

— Structured Logging: JSON-formatted logs with correlation IDs and context propagation

— Comprehensive Metrics: System, application, and business metrics collection

— Distributed Tracing: OpenTelemetry integration for end-to-end request tracking

— Health Monitoring: Health check endpoints and system status monitoring

— Performance Analysis: Memory, CPU, and I/O monitoring with alerting capabilities

— Debugging Support: Detailed error traces and performance bottleneck identification

I finally told Kiro to go ahead and implement the plan. Kiro systematically went through and created the system, following tasks.md. I was initially prompted if I wanted to review the structure after each task, until I told Kiro to run each task without prompting. Probably not something you would do in a production environment, but fine in this instance.

The building process was very quick (under two hours), but I did end up with a time out on the sessions several times. This is where the tasks.md structure is invaluable. When my session was timing out, Kiro would create a summary and start a new session. The new session checked existing files and worked on the next task.

The end result is first, some very thorough documentation:

— design.md (https://github.com/mgbec/Kiro-langchain-vector-spec-secure/blob/main/specs/langchain-vector-database/design.md)

— project-summary.md (https://github.com/mgbec/Kiro-langchain-vector-spec-secure/blob/main/specs/langchain-vector-database/project-summary.md)

— requirements.md (https://github.com/mgbec/Kiro-langchain-vector-spec-secure/blob/main/specs/langchain-vector-database/requirements.md)

— tasks.md (https://github.com/mgbec/Kiro-langchain-vector-spec-secure/blob/main/specs/langchain-vector-database/tasks.md)

Secondly, we have the Kiro produced code, with complete documentation, examples, tests, and more. https://github.com/mgbec/Kiro-langchain-vector-spec-secure/tree/main/KiroSpec

Some other features of Kiro that are exciting are Agent Steering and Agent Hooks.

Agent Steering — this is a way to guide your agents in a way specific to this project. You can have it produce three foundational files:

“product “— which is a product overview with key features, target use cases, and more

“structure” — directory structure, test structure, naming conventions and additional items

“tech” — core technologies, key dependencies, build system, common commands, dev setup, testing, code quality, env variables, configuration patterns

You can also customize the steering with other files and inclusion modes.

Agent Hooks - you can create event driven actions here, for example, update documentation when a file is created:

MCP Servers- there is a dedicated pane for these, which I like better than digging through config files.

In summary, Kiro’s vibe coding is great, but the spec driven design, in particular, seems pivotal in an enterprise environment. The easily created documentation and logical tasks make my Spock-like little heart very happy. It seems like a welcome addition to security documentation and architectural review boards, as well.

Part 2: Are You Threatening Me? Using AWS Q CLI for Your Threat Modeling Program

mgbec — Tue, 22 Jul 2025 19:29:05 +0000

One of the most admirable traits in humans throughout the world is our ability to persist in the face of problems and mistakes. We get back up when we are kicked down and keep trying until we solve a problem. The latest iterations of generative AI models and tools have been developed with this same trait.

In my last article I used AWS Q Developer in Visual Studio Code to create a fully functional Python app that ingested architectural diagrams and produced threat, risk, and mitigation reports based on that architecture. Q Developer quickly produced what I asked and added additional features on request. https://medium.com/@mgbecken/are-you-threatening-me-using-amazon-q-developer-to-jumpstart-your-threat-modeling-process-d90576d10bd6

I also wanted to try the command line version of AWS Q Developer to see any potential differences. To get started with the CLI version:

Install: follow the instructions here : https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/command-line-installing.html. I was on Ubuntu so I used the amazon-q.deb package to install.

Sign in: I logged in using my AWS Builder ID. You can set this up and use the free tier to run Q Developer — https://docs.aws.amazon.com/signin/latest/userguide/sign-in-aws_builder_id.html.

Prompt: Once I was signed in, I selected the same model I used in my previous article — claude-3.7-sonnet and used the same prompt “can you create a python based program that can ingest architectural diagrams and produce threat modeling and risk assessment information for those diagrams. Please make sure documentation, logging, and error handling is included”

Watch and give permissions: You’ll need to allow Q CLI to complete its proposed actions.

Look at Results and Run the Program: Q CLI quickly produced a program, checked to make sure it ran correctly, and then fixed an issue with the directory structure automatically, updating all the dependent files as well.

Q Developer CLI created a sample diagram and tested functionality successfully.

The threat report was created in markdown, which was clear but not especially pretty.

Ask for more: I asked Q for a few additional features — more threat information and a pdf formatted output report. Q quickly produced both and summarized the features.

The pdf looks beautiful!

Test: Now let’s try some more diagrams. Our command to produce pdfs is “cd arch_threat_analyzer && arch-threat-analyzer -i test_data/LotsOGateways.png -o test_data/output -f pdf -v”

The pdf report was successfully produced. The repo for this project is: https://github.com/mgbec/ThreatModelUbu

________________________________________

Entirely amazing, right? My mind is blown, but prepare for more brain explosions. I tried the same prompt in Windows Subsystem for Linux on my Windows PC. If you haven’t used WSL before, it is a way to run a virtualized Linux system on Windows. It is very handy and I use it frequently. It does have its quirks, however, where certain commands or programs just don’t work. This is where the ability of Q Developer to keep trying things is incredibly valuable. My human self would have given up within fifteen minutes and looked for a Linux machine.

Install — I followed the directions here: How to install Amazon Q Developer CLI on WSL2 | AWS re:Post

Sign In: I used my AWS Builder ID and I used the free tier. See above for more details.

Prompt: I switched to claude-3.7-sonnet and used the same prompt as before.

Watch and give permissions: I marked “t” to trust this tool and Q quickly developed a tool that should run in a Linux environment.

Look at results: Q created arch_threat_analyzer.py, a README.md, and several supporting files to make installation and testing easier. The program uses Open CV and Tesserat OCR among other components.

Run the program: here is where things get amazing. WSL made things interesting, but Q Developer was up to the challenge. Q Developer responses in bold:

Q started with trying to install dependencies with pip, then when that did not work, tried pip3

> It seems pip3 is also not available. Let’s check if Python is installed and which version:

> Good! Python 3.12.3 is installed. Let’s try installing the requirements using python3 -m pip

Q went through multiple attempts to make WSL function like a normal Linux machine,from virtual environments to manual installs. Finally, Q went to a Docker solution.

> Let’s try a different approach. Let’s see if we can use Docker to run our application:

> Given the constraints of the environment, let’s create a Dockerfile that can be used to run our application:

> Let’s also create a docker-compose.yml file to make it easier to run the application:

> Let’s also update the README.md to include Docker instructions:

> I’ve encountered some challenges with installing the Python dependencies in your environment, but I’ve provided a complete solution

that includes Docker support.

The end result is Python program that was completely functional and provided:

Ingest architectural diagrams in common image formats
Process them using image recognition and OCR
Identify components and their connections
Analyze security threats based on component types
Generate detailed threat modeling reports

Next, Q automatically checked for an existing Docker installation, then installed Docker and added the user to the Docker group for the correct permissions.

> Great! The Docker image has been successfully built. Now let’s create the input and output directories:

> Now let’s run the test script to create a sample diagram:

> Since we’re having issues with Python dependencies, let’s create a simple test diagram manually and place it in the input directory

> Perfect! The Architectural Diagram Threat Analyzer is working successfully. Here’s a summary of what we’ve accomplished:

Test: I tested with more architectural diagrams and Q created reports.

Summary: If you are still with me at the end of this long article, thank you for reading this far. I ended up really loving Amazon Q Developer, both the CLI and installed within my IDE. I especially loved the persistence of Q Developer, to work through the problems that come up in different environments. I would not have worked that hard to make my program function in WSL, but Q did keep attempting to execute new ideas until something worked. The repo for this latter WSL project is mgbec/ThreatModelingWithWSL: created with Amazon Q Developer on Windows Subsystem for Linux

Much time wasting and aggravation was prevented, so thanks again, Q!

Are You Threatening Me? Using Amazon Q Developer to Jumpstart Your Threat Modeling Process

mgbec — Mon, 21 Jul 2025 22:06:52 +0000

Amazon Q Developer is a quick and easy way to set up threat modeling from the command line. I started from scratch and in less than 10 minutes I had a fully functional Python program that scanned architectural diagrams and spit out threat modeling and risk information. My favorite part of the whole process is how quickly you can add new features and fix errors.

Install : I started out by installing Q Developer in Visual Studio Code — Using Amazon Q Developer in the IDE — Amazon Q Developer. There are also plenty of options to run it strictly as command line — see directions for Using Amazon Q Developer on the command line — Amazon Q Developer. I chose the link to install from the Visual Studio Marketplace.

Sign In : After a very quick install, you are asked to sign in with either your AWS Builder ID or your organization’s SSO. You can sign up for a Builder ID to use the free tier of Q Developer — Getting started with a personal account (Builder ID) — Amazon Q Developer.

Prompt : So far, so easy, right? Q Developer greets you and summarizes some of the things you can do. You can also select a different model to use in the dropdown at the bottom of the chat window. I used Claude Sonnet 3.7.

This was my first prompt to Q Developer:

“can you create a Python based program that can ingest architectural diagrams and produce threat modeling and risk assessment information for those diagrams. Please make sure documentation, logging, and error handling is included”

Watch and give permissions: Yeah, it’s that simple. Q Developer came up with a plan and asked me to give permissions to create a new directory.

Look at results: in less than five minutes, Q produced the code, complete with documentation, logging and error handling. You are given the implementation details, and a command to run your new threat modeling program.

Run the Program: I created a Python virtual environment and installed the requirements in my threat modeling program directory. Q Developer had already created a requirements.txt file for easy installation.

python -m venv venv

venv\Scripts\activate.ps1

pip3 install -r requirements.txt

Adjust: I did run into an error with pdf generation for the report. My next prompt to Q was “I am getting this error ERROR — Error generating PDF report: ‘latin-1’ codec can’t encode character ‘\u2022’ in position 1527: ordinal not in range(256).”

I actually was a little excited that the error handling Q Developer put in place went well here. I still received a threat modeling report, just in markdown, not pdf.

Q Developer then explained and fixed the error.

My next command line run worked flawlessly and produced a pdf report.

Ask for more: we always want more, right? I asked for more threats to be reported on:

Q Developer suggested a number of ways to build in additional threat information and explained the information that was added to my program:

Test: Now I tested more architectural diagrams and kept running them on the command line. For example:

arch_threat_modeler> python main.py ./GameProduction.png — output reportgameproduction.pdf — format pdf — threat-db custom_threats.json — verbose

The program was producing pdfs instantaneously! Oh my!

The pdfs produced are not fancy, but I’m sure we could definitely improve our workflow to make them prettier. We get an executive summary followed by identified threats, risk score, and mitigations.

So, all in all, a nice start to threat modeling, with the code written in under ten minutes. Thanks, Amazon Q Developer! Code available here mgbec/moreThreatModeling: Amazon Q developer wrote this.

If you would like to read about my experiences with Q CLI: https://dev.to/aws-builders/part-2-are-you-threatening-me-using-aws-q-cli-for-your-threat-modeling-program-42oj

Google Gemini CLI- the self-taught AI assistant

mgbec — Tue, 01 Jul 2025 14:27:04 +0000

We want a multitude of things from our AI systems. We want them to be quick, accurate, and cost efficient, just to start with. Traditionally, keeping our systems current could mean something as manual as retraining a model or updating a knowledge base. This is starting to be more and more an issue of the past. I recently tried Google Gemini CLI and the aspect that impressed me the most was the ability to quickly readjust and learn.

Installing Gemini CLI is very quick, and you most likely will be done in a few minutes. The Gemini CLI repository is here: GitHub — google-gemini/gemini-cli: An open-source AI agent that brings the power of Gemini directly into your terminal.

I asked Gemini to write a program that could ingest architectural diagrams and produce threat modeling and risk information. Gemini was quick to produce a Python program for me to run with the correct logic. My first attempt at running the program stopped with an error about a deprecated model: “An error occurred during architecture parsing: 404 Gemini 1.0 Pro Vision has been deprecated…”

Well okay, I can’t claim perfection either. I copy and pasted the error into the Gemini prompt and received an immediate suggested fix for the deprecated version. I agreed to the fixed code and it was updated.

The program successfully ran this time, but the output file,’Threat_Model_Report.md’, was missing threat and risk information. Gemini added some print statements to the code to troubleshoot this issue and asked me to provide the returned text.

This helped pinpoint the issue and Gemini supplied a fix.

Now, my program was working well. Gemini suggested enhancements with smarter risk assessment and enhanced reporting.

I ran the updated program and received a fairly detailed report.

I closed out my session and Gemini gave me statistics on my token usage and run time. At this point in time Gemini CLI free tier has a context token window of 1,000,000 with an output token limit of 8192.

So, I spent about ten minutes building this, including the corrections proposed by Gemini CLI. All of this, at no cost to me. Pretty amazing, but what really impressed me was the adaptation and self-correction. If anyone wants to look at the code Gemini produced, here is a link — https://github.com/mgbec/threat-modeling-agents.git. We could, of course, go much further with this program and add more features or refine outputs.

Thanks for reading and I highly recommend trying it yourself!

Observe and Report (and Prevent) — keeping an eye on your AI with CloudWatch and CloudTrail.

mgbec — Wed, 28 May 2025 23:26:20 +0000

Observe and Report (and Prevent) — keeping an eye on your AI with CloudWatch and CloudTrail.

Artificial intelligence components and data are evolving at rapid-fire speed. How are we supposed to keep tabs on performance, usage, and security?

Our old familiar friends, CloudWatch and CloudTrail, can step up to the plate and monitor our fast moving Bedrock environments. Of course CloudWatch and CloudTrail can monitor all of the usual components that might make up our Bedrock workflow, whether it is API Gateway, Lambda, Dynamo, or something else. For Bedrock itself, there are some specific metrics and data that pertain directly to Bedrock that we can log, measure, and dashboard in CloudWatch and CloudTrail. Likewise, we can generate alarms, or trigger actions based on Bedrock data. More details are available here: https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring.html.

CloudWatch

To get started in CloudWatch, we just need two quick steps:

First, create a CloudWatch log group:

Second, we need to enable Model invocation logging in Bedrock Settings, under the Bedrock configurations sidebar. I am going to include all data types in my logs, just send the logs to CloudWatch, and create a new service role to do this.

Now, let’s get some data to look at. I invoked a few different models and agents that I had created previously.

CloudWatch will not only have the details for the components of your workflow, like Lambda or DynamoDB, but now you will see Bedrock invocation details in your log group.

We can analyze these log details with Log Insights:

We could also use some of the other CloudWatch functionality such as Metrics, Anomaly Detector, and Alarms. A pre-created Dashboard for Bedrock Metrics that has the following fields but we could certainly customize:

-Invocation Count

-Invocation Latency

-Token Counts by Model

-Daily Total Tokens by ModelID

-Input Token Count, Output Token Count

-Requests Grouped by input tokens

-Invocation Throttles

-Invocation Error Count

There is also specific data for some optional components of your workflow- like Knowledge Bases, Guardrails, and Agents.

Knowledge bases:

https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-bases-logging.html

Knowledge base logs will need to be enabled first and there are data ingestion level logs and resource level logs. Data ingestion logs provide ingestion job information like data source id, number of resources updated, ingested, deleted, and more. Resource level logs give us details about the status of the ingestion logs in the pipeline. Is it scheduled, embedded, indexed, partially done, or failed?

Guardrails:

https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring-guardrails-cw-metrics.html

Guardrails are an important part of our Bedrock environment and need attention for both security and performance factors. Metrics include:

-Invocations

-InvocationLatency

-InvocationClientErrors

-InvocationServerErrors

-InvocationThrottles

-InvocationsIntervened

-TextUnitCount

“InvocationsIntervened” is a specific metric for Guardrails giving us the number of invocations where Guardrails took action and intervened.

“TextUnitCount” takes a little more explanation. A text unit is up to 1000 characters. This unit helps AWS determine what to charge for the enabled Guardrail policies. Different types of policies may have different pricing per cost unit than others.

Agents:

Agents are another component that have metrics we can look at-

https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring-agents-cw-metrics.html . Metrics included are:

-InvocationCount

-TotalTime

-TTFT

-InvocationThrottles

-InvocationServerErrors

-InvocationClientErrors

-ModelLatency

-ModelInvocationCount

-ModelInvocationThrottles

-ModelInvocationClientErrors

-ModelInvocationServerErrors

-InputTokenCount

-OutputTokenCount

TTFT is “time to first token” and is only given when streaming configuration is turned on for the agent request.

CloudTrail

Amazon Bedrock Runtime API operations are management events, which are logged by default in CloudTrail. These include:Invoke Model, InvokeModelWithResponseStream, Converse, and ConverseStream.

Agents for Amazon Bedrock Runtime API operations are data events, and not logged by default. These are InvokeAgent, InvokeInlineAgent,Retrieve, RetrieveandGenerate,InvokeFlow, and RenderPrompt.

If you would like to get your CloudTrail data events included, you can enable data event logging.

https://docs.aws.amazon.com/bedrock/latest/userguide/logging-using-cloudtrail.html

To enable the data events for Bedrock:

First, create a new Trail in CloudTrail. Then, click into the new trail and enable data event collection.

The dropdown menu for Resource type will have Bedrock resources. Currently the resource types available are Bedrock Agent Alias, Bedrock Blueprint, Bedrock Data Automation Profile, Bedrock Data Automation Project, Bedrock Flow Alias, Bedrock Guardrail, Bedrock Invoke Inline-Agent, Bedrock Knowledge Base, Bedrock Model, Bedrock Prompt, and Bedrock Session. Choose your options and run some models to generate data.

You can navigate to CloudTrail and click the link to the S3 bucket. Analyze with your choice of tools, whether it is OpenSearch, a SIEM, or something else.

So, what are we looking for in CloudTrail and CloudWatch? Performance and observability, of course, and expenditure review. However, if I wanted to look for security related events, what could I see?

Some attacks on our AI resources could be caught further outside of the ring of fire, in the usual AWS ways, like API Gateway, CloudFront, and WAF. With a layered defense approach, we would want to look for multiple indicators of attacks or compromise throughout the AI workflow. Some things to watch in CloudWatch and CloudTrail logs and metrics include:

Prompt analysis : in CloudWatch logs, the prompt is returned in full, letting us analyze any anomalies or indicators of misuse.

The CloudWatch automatic Bedrock dashboard also can show us anomalous use of input tokens and there is a specific graph that would help us see if we are getting inputs that do not follow our standard pattern- “Requests, grouped by input tokens”.

Response analysis: in CloudWatch logs, we also get to see the full response to the prompt. We will need to evaluate these for things like sensitive or proprietary information disclosure. We will also want to look for hallucinations, illogical responses, or generation of harmful content.

This is the point where I realized that when I set up Bedrock Model invocation logging, I should have specified both S3 and CloudWatch Logs as the logging destination. I’m missing the full response, since just sending the logs to CloudWatch logs will limit output JSON bodies to 100KB in size.

https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html

I went back, created a new bucket and changed this logging destination. I ran a few more invocations and confirmed that I am now getting the full response.

Again, the CloudWatch automatic Bedrock dashboard also can show us anomalous use of output tokens and also graphs “InputTokenCount” against “OutputTokenCounts. “OutputImageCount” is another metric you could watch, but it’s not in the current automatic dashboard.

Performance Degradation: the CloudWatch Metrics automatic Bedrock dashboard is a great place to look for signs of this. Variance in invocation throttles, latency, and errors are included in the standard dashboard. Increased and unexpected invocations is also an indicator. Other metrics that you could set up monitoring for are InvocationClientErrors and InvocationServerErrors.

CloudTrail indicators: unexpected changes in API calls can have a number of origins. I mentioned enabling data event in addition to management event collection above. Specific to Bedrock, the Bedrock API Reference https://docs.aws.amazon.com/bedrock/latest/APIReference/API_Operations_Amazon_Bedrock.html can help us understand what we might be looking for. We can also look at agent specific API calls https://docs.aws.amazon.com/bedrock/latest/APIReference/API_Operations_Agents_for_Amazon_Bedrock.html. Indicators of Bedrock compromise or attack would also show up in API calls to non-Bedrock resources, like S3 or DynamoDB.

Similar to all aspects of information security, monitoring Bedrock resources for indicators of an attack is a layered and reiterated task. Also similar to the AI field, as a whole, the tools we will use to monitor performance and security will keep evolving and changing. What does that mean for us as security professionals? I don’t think we will run out of things to learn any time soon- more brains required, donuts optional. Thanks for reading and let me know if you have any questions or comments!

MCP Security: Tr-tr-tr-tricky, tricky, tricky

mgbec — Wed, 14 May 2025 01:17:21 +0000

The GitHub MCP Server looked like the perfect fit for a project I am working on. Connecting to the GitHub MCP from VSCode was actually not super tricky, but I do see some risks that will make securing MCP systems less than easy. I’ll walk through the process of adding this MCP server and some of the risks we will want to think about.

Setup

Here are the official instructions from GitHub- https://github.com/github/github-mcp-server.

I ended up using the manual install option and choosing to run it in Docker. You’ll go into user preferences and add this into Preferences: Open User Settings (JSON), replacing the personal access token with your own.

{
  "mcp": {
    "inputs": [
      {
        "type": "promptString",
        "id": "github_token",
        "description": "GitHub Personal Access Token",
        "password": true
      }
    ],
    "servers": {
      "github": {
        "command": "docker",
        "args": [
          "run",
          "-i",
          "--rm",
          "-e",
          "GITHUB_PERSONAL_ACCESS_TOKEN",
          "ghcr.io/github/github-mcp-server"
        ],
        "env": {
          "GITHUB_PERSONAL_ACCESS_TOKEN": "${input:github_token}"
        }
      }
    }
  }
}

These are the toolsets available at this point in time:

You could install it with limited toolsets, but I chose to install them all.

Once installed you can select certain ones to use.

Testing

I had a deliberately vulnerable test repository from StackHawk: https://github.com/kaakaww and ran some of the GitHub MCP tools against it. It gave me the code scanning alerts for the repo I wanted in a nicely formatted list.

It correctly determined that this particular repo had no Secret Scanning alerts and was able to list all of my repositories.

I asked which vulnerability I should fix first and was given a recommendation:

I asked for specific code fixes and received suggestions:

Fixing the code automatically seemed to require that the code was stored locally.

All cool stuff, right? But there is no way that this is the end of the story….

Let’s talk about security. In this particular scenario, I am running this MCP server locally in Docker and using a fairly tightly scoped GitHub personal access token. That could help with some risk reduction but there is much more I need to consider before I take this out for a ride in an enterprise.

In general, some MCP threats include:

-Prompt injection

-MCP server compromise

-Tool poisoning

-Data leakage

-Unauthorized access

-Potential for DDoS, excessive resource utilization and costs

-Lack of humans in the loop, for consent, feedback, and transparency

-Lack of observability

MCP is still new enough to have plenty of security kinks to work out but efforts are being made by multiple sources. The MCP specification https://modelcontextprotocol.io/specification/2025-03-26 has a specific section for “Security and Trust & Safety”. User consent and privacy is a key piece of this. The protocol roadmap also includes features that will increase security.

I can’t write an article without mentioning OWASP and yes, they do have MCP security advice- https://genai.owasp.org/2025/04/22/securing-ais-new-frontier-the-power-of-open-collaboration-on-mcp-security/.

AWS has guidance for secure deployment on their platform- https://aws.amazon.com/solutions/guidance/deploying-model-context-protocol-servers-on-aws/. There is an included architectural diagram that shows a layered security approach toward MCP server interactions, including OAuth 2.0 authentication using Cognito and token storage with DynamoDB.

The centerpiece of the authentication and authorization is an MCP Auth Service, which runs in a secure virtual private cloud (VPC) on AWS Fargate. MCP Auth Service works with DynamoDB and Cognito to send tokens to the MCP client, routing through the AWS Application Load Balancer and CloudFront.

Some of the other factors accounted for are observability using CloudWatch, rate limiting, web attacks, isolation with security groups, and through containerization. In-transit encryption is provided through CloudFront and throughout the AWS environment.

The CDK code for this project is included here: https://github.com/aws-solutions-library-samples/guidance-for-deploying-model-context-protocol-servers-on-aws?tab=readme-ov-file#overview

Finally, If you would like to experiment with some of the threats you might encounter with an MCP deployment, there is a deliberately vulnerable MCP server to try to exploit at https://github.com/harishsg993010/damn-vulnerable-MCP-server.

Some of the vulnerabilities you can try to exploit are:

Prompt Injection: Manipulating LLM behavior through malicious inputs
Tool Poisoning: Hiding malicious instructions in tool descriptions
Excessive Permissions: Exploiting overly permissive tool access
Rug Pull Attacks: Exploiting tool definition mutations
Tool Shadowing: Overriding legitimate tools with malicious ones
Indirect Prompt Injection: Injecting instructions through data sources
Token Theft: Exploiting insecure token storage
Malicious Code Execution: Executing arbitrary code through vulnerable tools
Remote Access Control: Gaining unauthorized system access
Multi-Vector Attacks: Combining multiple vulnerabilities

It is safe to say that threats to MCP servers and our associated resources will continue to evolve as fast as the AI landscape has been changing. Hold on to your hats and thanks for reading!

Let’s Build Agents!

mgbec — Fri, 25 Apr 2025 19:27:41 +0000

I’ve been doing a bit of research into generative AI and security. My last article focused on security risks in a simple bot- https://medium.com/@mgbecken/for-those-about-to-bedrock-751fbc804012. This time around I wanted to experiment with Bedrock agents and investigate some of the things that are specific to agentic AI. Some distinct issues include:

Complexity and orchestration of tasks performed

Potential access of enterprise data sources

Ability to perform real time tasks with real time data

Tooling integration

Memory- both long term and short term

Once again, OWASP has some excellent material to read- https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/ directly addresses agent scenarios.

Another great article — Agentic AI Threat Modeling Framework: MAESTRO | CSA. breaks threat modeling and potential mitigation into a layered framework.

A wonderful GitHub repo with incredible information is — github.com/precize/OWASP-Agentic-AI

As I build a simple single agent application, I wanted to think through and comment on some of the security issues. I broke my categories of possible issues to address into three rough steps at this point- “Prebuild”, “During Build”, and “Postbuild”. Many of these topics are not specific to agentic AI.

PREBUILD

Marketplace :

Choosing our foundation model- possible threats are a noncompliant model and trustworthy model information. I talked about this in my last article, so I won’t go into detail here.

This time around I am using Amazon Nova Micro as a foundation model for cost and performance reasons. I am building my own agents in Bedrock but in a different situation, I would need to think about secure agent selection.

Threat Modeling:

The articles above have some great places to start. There are quite a few factors to consider depending on your architecture.

Data:

Is my data AI ready? Is it labeled with access restrictions and other metadata?

Operations planning:

— Plan for keeping our CICD workflow secure to prevent IaC manipulation. You could also use code signing and integrity verification.

— Plan on control- what boundaries and constraints can we set up between components, users, roles, and agents?

— Plan for updates and vulnerability management. An AI BOM would be helpful here to detail all the components you will be using from data and algorithms to dependencies and metadata. We would want to have a system for supply chain monitoring.

— Plan your operations monitoring and incident response strategy. How will we detect problems and attacks and how will we respond to them? What type of communications will we have in place and will we have a feedback loop to self correct? Will we need system redundancies in case of failure or resource exhaustion?

— Plan to train people. We might need to train our model, but we also need to train the humans in the loop. What should the end user expect? How will support deal with issues? What is the business goal?

DURING BUILD

Data - what kind of data am I using, and who is allowed access? What type of authentication am I using? Am I making sure data is encrypted with modern methods? I am specifically thinking of the DeepSeek iOS app here using 3DES with hard-coded encryption credentials.

Tools — How can I make sure my tools remain safe and isolated? What are our isolation requirements? We would want to make sure we have authentication, boundaries, and monitoring in place. I would also want to make sure I kept the tools up to date, so we aren’t sitting on any old vulnerabilities. We could scan the Lambda based tooling and report on outdated components in AWS to help with that.

It will be interesting to see our risks evolve with MCP usage. That is next on the bucket list.

Agents - we will want to think about potential hijacking and keeping multi-agent communications secure. What monitoring can we build to catch any problems? Agent to agent communication could have mutual authentication and continuous reauthentication for long running processes.

Prompts - am I being careful to have sufficient input validation? We would want to prevent code injection and system compromise. Another threat vector might be computationally expensive inputs to create resource exhaustion. An interesting threat to read about is temporal manipulation in prompts — https://github.com/precize/OWASP-Agentic-AI/blob/main/agent-temporal-manipulation-timebased-attack-13.md

I was picturing the look on a coworker’s face if we triggered something resource intensive at peak usage hours.

Memory- we would want to prevent both memory poisoning and possible data exfiltration. Memory content validation, session isolation, encryption, and authentication would be important. We would also want to monitor memory logs for anomalies in memory storage. If we are working with sensitive data, we would want to be especially careful and make sure we are not unnecessarily retaining the data. Bedrock encrypts session information with an AWS provided key but we could use a customer managed key if necessary.

Logs- are they encrypted, sensitive data masked? Are they aggregated and stored for as long as is required by regulations?

Decisions — Is there decision traceability and transparency? Are these being analyzed? Is there human intervention for high risk decisions?

Output — is there validation and constraints to detect and prevent hallucinations, etc?

Resource exhaustion - possibly we can have process control policies that would trigger throttling or auto suspension if one of our parameters goes over a defined limit?

Redundancy — is this in place for high value workflows?

Alerts and feedback loop- is there a way to alert on big problems and potentially fix them as we run?

Orchestration — if we are using multiple agents or tools, are we able to monitor coordination and integration? I am picturing an evil orchestrator here. “Bwahaha, let me add more double bass.”

POSTBUILD

Logs and Traces — now it’s time to look at all the logs and traces we hopefully created in the operational step. Do we have everything we need? Are they decently easy to analyze? Can we find a digestible way to share them with non-technical managers?

Analysis- is this providing the functionality and value we need? What are the costs associated with this workflow?

Vulnerabilities and observed attacks- what issues do we need to fix and what is the priority? How are we going to communicate these to users who need to perform the work?

AI BOM — do we need to update this or update components in the build?

Foundation model — do we need to switch out the model? Is it doing what we want? Could we use a cheaper model?

Continuous improvements — Can we optimize anything? On this build, I definitely see some more prompt engineering needs to be done.

Now it is on to my actual build procedure….

1.) DynamoDB creation for two tables: I will use these two tables to look at the Dependabot Alerts and Developer Team Info. I had a previously created Dynamodb table populated with GitHub Dependabot alerts. I used GitHub CLI to get the alerts and associated data, if you want details on the process, here is a very quick article I wrote: https://medium.com/@mgbecken/operation-dynamodb-wrangle-some-github-data-into-dynamodb-9947a1e99077. Security notes here, using roles, so IAM protection, baked in encryption at rest. Don’t put sensitive names or plaintext data in your Primary Key and Global Secondary Indexes, since the names will show up in your table definition.

The table with alerts is called ‘concat_dependabotalert’. I also created some fake Team data to put into a DynamoDB, ‘concatTeamInfo’.

2.) Create two Lambda functions to look up information in the DynamoDb tables: ‘concatAlertNumber’ and ‘concatTeamInfo’. The code for these is in my github at https://github.com/mgbec/0425AgentSec. You will also want to bump up the timeout for both of them. The default time out is 3 seconds and you will probably want more time than that.

You also need to add DynamoDB permissions to your execution role for each Lambda. In Configuration > Permissions click on the Role name. You will be redirected to IAM where you can click the “Add permissions” button and add the ability for this role to query the DynamoDB.

3.) Create an Open API schema: The tools my agent needs to run here are both using an OpenAPI schema to tell them how to work. You can write this in yaml or json and then put it in an S3 bucket. Security note: a potential threat vector is a malicious change to this file. We could scan this upload or file for evil doings and make sure our bucket is locked down. My OpenAPI yaml files are ConcatDependabotAlertStatus.yaml and concatTeamInfo.yaml, also available in my GitHub ‘https://github.com/mgbec/0425AgentSec’

4.) Now we get to build our agent. I am using Nova Micro, which is cheap, fast, and is working fine for the task at hand. More details about the Nova models:

Amazon Nova models, specifically Nova Micro, Nova Lite, and Nova Pro, have varying costs per 1,000 tokens. Nova Micro is the cheapest, with input tokens at $0.000035 and output tokens at $0.00014. Nova Lite is in the middle, costing $0.00006 for input tokens and $0.00024 for output tokens. Nova Pro is the most expensive, with input tokens at $0.0008 and output tokens at $0.0032.

https://aws.amazon.com/bedrock/pricing/

One of the most important parts of building the agent is creating the ‘Instructions for the Agent’. I tried to be as specific as possible and went back and adjusted as I tested.

There are additional settings that you can customize as needed. I left these as default but one potential setting that could affect security is the ‘idle session timeout’- your user’s submitted data remains in short term memory for the length of the session. The agents have additional memory configuration options, depending on the foundation model.

Another setting is ‘KMS Key selection’. You could opt to use your own KMS key if that is a requirement.

5.) Creating your action groups: (not the same thing as Azure action groups, as you might suspect.)

This is where our previous work creating our Lambda functions and OpenAPI schemas pays off. We will create an action group for each DynamoDB lookup we want to perform.

We can select the OpenAPI schema and Lambda function that corresponds with each tool.

Save your Agent and take note of the Agent ARN, we will use it in the next step.

6.) For both of your Lambda functions, you will want to let the agent have the permissions to be able to invoke them. Under Configuration>Permissions, you can add a resource based policy and enter the ARN of your agent in the Conditions.

7.) Now we can test our Bedrock agent. In the right side panel, you can try your input.

The traces will help you troubleshoot errors or unexpected output, and also look at the agent’s “thinking process”.

You could possibly return the information in another format or determine what functionality you wish was there for future development.

8.) Our next step could be to keep tweaking everything and adding more tooling. We might want to add a knowledge base or test MCP. GitHub does have an MCP server — https://github.com/github/github-mcp-server. At the time I wrote this article, we would be able to add CodeQL or Secrets alerts. Other MCP servers I would like to check out for this particular project can create Excel workbooks and look up CVE’s.

9.) Finally, if we are ready to deploy, we can create and publish a user interface in the method of our choice. This is another project on my list, to be continued. Thanks for reading!

Operation DynamoDB- wrangle some GitHub data into DynamoDB

mgbec — Sun, 20 Apr 2025 20:34:39 +0000

For a Bedrock project I am working on, I needed to get my GitHub Dependabot alerts into an AWS DynamoDB table. This may be a complete edge case, but in case it helps anybody, here is how it went:

First, get your data out of GitHub using the REST API:

Install GitHub CLI — https://cli.github.com/

Once the CLI is installed, you will need to authenticate to your GitHub account-

gh auth login

Now you can call the API, see variations here:

https://docs.github.com/en/rest/dependabot/alerts?apiVersion=2022-11-28

If you are using data from your enterprise or organization, you will have a slightly different command line than I am showing for my personal GitHub repositories. You will also need to have the correct permissions to query the data. Also, if you are calling for an entity with quite a few alerts, you will probably want to add — paginate to get them all.

gh api -H “Accept: application/vnd.github+json” -H “X-GitHub-Api-Version:2022–11–28” /repos/yourgithubaccount/yourrepo/dependabot/alerts > dependabot.json

I redirected to a file and then I did this for the three repos I wanted to analyze. You can do it all in one fell swoop for an enterprise or organization.

I am not a developer, nor do I play one on TV, but this is what worked for me in Python.

## import your shenanigans
import pandas as pd
import boto3
import json
import awswrangler as wr

##read your json files produced by the GitHub CLI into dataframes
file_path = r"C:\Users\User\dependabot.json"
df = pd.read_json(file_path)

file_path = r"C:\Users\User\dependabot2.json"
df2 = pd.read_json(file_path)

file_path = r"C:\Users\User\dependabot3.json"
df3 = pd.read_json(file_path)

##Concatenate the files
concatdf= pd.concat([df, df2, df3])

#I ended up converting converting all the datatypes into string, which was fine in this case
concatdf = concatdf.astype(str)

## Function to trim URL path, I just want the repo name of this column. I also renamed that column to “repo”

def trim_url_path(url):
    return url.split('/')[4]  

concatdf['repo'] = concatdf['html_url'].apply(trim_url_path)

## Now let ’s wrangle my data with AWS SDK for pandas, formerly known as AWS Wrangler
wr.dynamodb.put_df(
    df=concatdf,
    table_name='concat_dependabotalert'
)

Yay, and now it will upload. Note: I already had a table created in DynamoDB when I started this process. I did this a few separate times with the same amount of data and the time to upload to DynamoDB varied quite a bit. It might definitely be a user side issue- a watched table never populates. You can also use this process for GitHub CodeQL alerts, and Secrets alerts. Hope this helps somebody. Thanks for reading!