Forem: Gernot Glawe

LAPD - Lambda Python Deployment for special "get out of jail" cases

Gernot Glawe — Sun, 10 Dec 2023 18:45:46 +0000

Sometimes you develop with a framework that is restrictive, or an environment which is event more restricted.

Get out of jail with zipping your own Lambda package.

Without packaging requirements.txt each time, but just adding it to the zip you wayyy faster. So fast High Speed Police will come.

Did you know you can also download the code from the lambda function itself? Then you got also the correct dependencies. Because sometimes if C libraries involved, it has to be linux. And not everybodys Laptop is Linux, is it? :)

https://github.com/megaproaktiv/lapd

So enjoy this little helper, you might not need it. But in a special project it saved me a lot of time.

Because its GO, you can just copy your binary from https://github.com/megaproaktiv/lapd/releases/tag/v0.1.2

GO-ing to production with Amazon (AWS) Bedrock RAG Part 1

Gernot Glawe — Sun, 10 Dec 2023 18:25:02 +0000

The way from a cool POC (proof of concept), like a walk in monets garden, to a production-ready application for an RAG (Retrieval Augmented Generation) application with Amazon Bedrock and Amazon Kendra is paved with some work. Let`s get our hands dirty.

With streamlit and langchain, you can quickly build a cool POC. This two-part blog is about what comes after that.

See this AWS blog post here as an example of a POC. The github repo uses streamlit and langchain to build a POC.

The story

Let's assume you are an AWS architect in the “fancy GO knowledge company. You have shown the example above to the board, and they are excited.

Hugo, is the CEO and says:

"Well this is really cool stuff. We will really enhance the way we share knowledge in our company."

You know what comes next: the BUT.

"But, ... it will be used by many people, so it should scale really well. Can you do that?

You begin to remember a video you saw a while ago on youtube: The expert.

"And of cause, it should be fast, secure and cost-effective at the same time"

OK, you think 3 BUTs. I can handle that.

"I also might have a few ideas about some more features. To start with, give us the text snippets from the documents and page numbers also."

You begin to sweat. You look to your boss Gordon (the CIO), for help. He may have slightly misinterpreted your look and added some more wishes: "And it should be serverless and have fast development cycles. And it should be easy to maintain."

Time to think

You remember what the expert in the video said: "I can do anything, I can do absolutely anything." and start.

The POC architecture

The new requirements are:

0) Make the server available as a web app
1) Scale fast
2) Secure
3) cost-effective
4) features page numbers, excerpts
5) serverless (this is not really a requirement, but a solution architecture hint)
6) easy to maintain
7) fast development cycles

Solution architecture

0) Make the server available as a web app

You decide to deploy the streamlit server as a fargate container. You use AWS CDK ApplicationLoadBalancedFargateService as a fast way to deploy a web server.

Containerization of the application is relatively easy. Here is the dockerfile:

FROM --platform=linux/amd64 python:3.11-slim

WORKDIR /app

RUN apt-get update && apt-get install -y \
build-essential \
&& rm -rf /var/lib/apt/lists/*

COPY streamlit/ streamlit/

WORKDIR /app/streamlit
RUN pip install --no-cache-dir -r requirements.txt

EXPOSE 8501

HEALTHCHECK CMD curl --fail http://localhost:8501/_stcore/health

ENTRYPOINT ["streamlit", "run", "app.py", "openai", "--server.port=8501", "--server.address=0.0.0.0"]
`

However, the monolithic architecture needs to scale better. So, you decide to split the application into two parts: the frontend and the backend.

The frontend can run streamlit now, but it could be changed afterwards. To have a clean separation, the backend is an API Gateway with a Lambda function.

1) Scale fast

With AWS Lambda now scaling 12 times faster, this part is covered. You are aware of the bedrock runtime scaling limits but you decide to deal with that later. Maybe with the throttling feature of API Gateway. You think of a SQS to decouple, but decide against premature optimization, because it is the root of all evil.

2) Secure

You know that with lambda you have checked many boxes of an AWS Well-Architected Framework.

A short commercial break:
You decide to do an Well Architected Review with tecRacer later.

The API gateway will run in a private subnet, and the lambda function will have no internet access. To secure development, you use API Keys.

3) cost effective

Lambda is a good choice to run unpredicted workloads. After reading a comparism about the storage requirements of Python vs GO in the article Stop LLM/GenAI hallucination fast: Serverless Kendra RAG with GO
, you decide to try GO as Lambda development language. This step will reduce storage and Lambda execution costs, especially cold starts.

4) features page numbers, excerpts

To analyse the POC, you look at the code of the langchain. In the langchain you look at the results given back from the Kendra query. They are created by site-packages/langchain/retrievers/kendra.py

json { "question": "my question", "chat_history": [], "answer": "some god answer", "source_documents": [ { "Document": "page_content= Document Title: ... Document Excerpt: ...", "metadata": { "source": "https: //s3.eu-west-1.amazonaws.com/a_bucket/a_document.pdf", "title": "a title", "excerpt": "...\n" } } ] }

No page numbers are returned. You decide not to change the langchain, but to code the calls directly. For bedrock, you find many examples in Python, Javascript and GO, e.g. here.

As you have to recode the logic anyway, you decide the fastest language, GO. See comparism.

5) serverless (this is not really a requirement, but a solution architecture hint)

Yes.

6) easy to maintain

The Lambda constructs implementations are good, but you decide to give AWS SAM a try. As it is focused on Serverless, it is less complex than AWS CDK. That means it is easier to learn and maintain.

The monolithic architecture will be split into two parts in the solution architecture: frontend and backend. This decoupled solution architecture is also easier to maintain.

7) fast development cycles

Frontend

The deployment of the Streamlit server as fargate container with CDK takes a few minutes. As you can fully develop the streamlit application locally, this is not a problem. You develop and do integration tests locally.

You can change the app.py code and with a refresh of the browser see the changes.

Backend

AWS Hero Yan Cui has pointed out in "Test Honeycomb", that with serverless environments, you should focus on integration tests. Although this is a ongoing discussion, you decide to focus on integration tests.

That means you must deploy the lambda function to AWS to test it. Because you test hundreds of times a day, you need a fast deployment.

The great thing is that you do not have to implement this yourself, as described in sub seconds deployment.

You give the new AWS SAM feature sync a try. So you may do a full deployment with sam deploy if resources change, or a fast `sync for code changes.

With sam sync --code --stack-name mystackname --watch code changes are deployed to AWS in a few seconds in the background.

So in the time it takes you to switch the programm from editor to postman, the code is deployed. So the question now is: Is "no time" fast enough?

On the terminal it looks like:

task sync
task: [sync] sam sync --code --stack-name godemo --watch

The SAM CLI will use the AWS Lambda, Amazon API Gateway, and AWS StepFunctions APIs to upload your code without
performing a CloudFormation deployment. This will cause drift in your CloudFormation stack.
**The sync command should only be used against a development stack**.

You have enabled the --code flag, which limits sam sync updates to code changes only. To do a complete infrastructure and code sync, remove the --code flag.
CodeTrigger not created as CodeUri or DefinitionUri is missing for RagAPIGateway.
Sync watch started.
Syncing Lambda Function RagFunction...
Cache is invalid, running build and copying resources for following functions (RagFunction)
Building codeuri: /Users/gglawe/letsbuild/git-megaproaktiv/go-rag-kendra-bedrock/lambda/query runtime: provided.al2 metadata: {'BuildMethod': 'makefile'} architecture: arm64 functions: RagFunction
RagFunction: Running CustomMakeBuilder:CopySource
RagFunction: Running CustomMakeBuilder:MakeBuild
RagFunction: Current Artifacts Directory : /Users/gglawe/letsbuild/git-megaproaktiv/go-rag-kendra-bedrock/.aws-sam/auto-dependency-layer/RagFunction
env GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -tags lambda.norpc -ldflags="-s -w" -o bootstrap
cp ./bootstrap /Users/gglawe/letsbuild/git-megaproaktiv/go-rag-kendra-bedrock/.aws-sam/auto-dependency-layer/RagFunction/.
cp ./prompt.tmpl /Users/gglawe/letsbuild/git-megaproaktiv/go-rag-kendra-bedrock/.aws-sam/auto-dependency-layer/RagFunction/.
Missing physical resource. Infra sync will be started.
You have enabled the --code flag, which limits sam sync updates to code changes only. To do a complete infrastructure and code sync, remove the --code flag.

SAM calles the Makefile in the lambda directory. It takes less than a second to build the bootstrap file. You decide to go with this architecture concept. In Part 2 I will talk about how to implement it.

Conclusion

You are happy with your architecture. You start to implement it. - This is the cliffhanger for part 2.

Disclaimer

The characters in this post are fictitious. Any resemblance to persons living or dead is purely coincidental.

What I have learned

In the last half year, my focus was on Generative AI and the RAG approach. It is a great way to enhance knowledge base access for larger companies or compleyx topics.

This story shall entertain you and give you some ideas how to implement a RAG architecture and to think outside the "its all python/langchain" box. For your project it could also be JavaScript, but as you have seen, GO has some advantages too.

Second part

The backend implementation will be covered in the second part of this blog post. All source code will be available on github and with some basic AWS knowledge you are ready to try it out yourself in minutes.

Enjoy building!

If you need developers and consulting for your next GenAI project, don't hesitate to contact the sponsor of this blog, tecRacer.

For more AWS development stuff, follow me on dev https://dev.to/megaproaktiv.
Want to learn GO on AWS? GO here

Stop LLM/GenAI hallucination fast: Serverless Kendra RAG with GO

Gernot Glawe — Wed, 20 Sep 2023 16:56:58 +0000

RAG is a way to approach the "hallucination" problem with LLM: A contextual reference increases the accuracy of the answers. Do you want to use RAG (Retrieval Augmented Generation) in production? The Python langchain library may be too slow for your production services. So what about serverless RAG in fast GO Lambda?

The AWS RAG sample solution

In evaluating whether a technical solution is suitable, the focus is on the simplicity of development. So, the AWS sample uses the well-known langchain library and a streamlit server for the chat sample.

Shifting to production ready solutions, the focus goes on speed and scalability.

In Quickly build high-accuracy Generative AI applications on enterprise data using Amazon Kendra, LangChain, and large language models a StreamLit Python Server is used. See kendra_retriever_samples for Code.

As this is a good choice for working with low volume data science and LLM projects, a serverless solution does scale better. in Simplify access to internal information using Retrieval Augmented Generation and LangChain Agents a few examples are shown.

And with a serverless solution we enter the polyglott world and can decide on the best language for the job. So what about GO?

RAG with langchain, OpenAI and Amazon Kendra

The sample server uses Python and the langchain library. However, despite its functionality, speed and memory consumption could be better.

To check this assumption, I test the speed of the solution. I use just the langchain library in kendra_retriever_open_ai.py from amazon-kendra-langchain-extensions/kendra_retriever_samples.

Measure speed and size

Python Speed

The Kendra index from the AWS sample solution queries some AWS documentation. So a question should be about that topic.

Therefore to measure speed I use a single query "What is lex?" and perform it 10 times to reduce statistical randomness. This is only done locally, but should give a good measurement for lambda cold starts.

The test calls python3 kendra_chat_open_ai.py 10 times.

./test.sh  35,19s user 7,26s system 57% cpu 1:13,63 total

So time measuring rod is 73 seconds per 10 iterations.

Python Size

The app kendra_retriever_open_ai.py itself is only a few K large.
Most of the size goes to the pleora of libraries:

499M    .venv
108K    .venv/bin
490M    .venv/lib

So size measure is around 490 M.

Production

Shifting to Lambda/GO

The problem with Python Lambda is the cold start, see custom-runtime-on-amazon-linux-2-go-outperforms-node-3x-and-python-2x-with-aws-lambda-cold-start-time.html. Also what often is forgotten is the cost of updating the Lambda Python runtime over the years. With the GO backward compatibility promise you have less cost of code maintenance.

See the comparism of cold starts Node vs Python vs GO custum runtime.

Implementation in Python

To reproduce the behaviour of kendra_retriever_open_ai.py you have to understand the Langchain library. The general behaviour is:

1) Retrieve Kendra data
2) Generate Prompt with langchain "stuff" chain
3) Call LLM for inference

The steps in Python are:

1) Retrieve Kendra data

see .venv/lib/python3.11/site-packages/langchain/retrievers/kendra.py

2) / 3) in kendra_retriever_open_ai.py:

  llm = OpenAI(batch_size=5, temperature=0, max_tokens=300, model="gpt-3.5-turbo")
  retriever = AmazonKendraRetriever(index_id=kendra_index_id,region_name=region)

The llm is initialized with the OpenAI model and the retriever with the Kendra index.

The the chain is started

RetrievalQA.from_chain_type(
      llm,
      chain_type="stuff",
      retriever=retriever,
      chain_type_kwargs=chain_type_kwargs,
      return_source_documents=True
  )

and the results are printed.

result = chain(prompt)

Resulting in:

python kendra_chat_open_ai.py
Hello! How can I help you?
Ask a question, start a New search: or CTRL-D to exit.
> QueryText: what is lex?
 Amazon Lex is an AWS service for building conversational interfaces for applications using voice and text.
 With Amazon Lex, the same conversational engine that powers Amazon Alexa is now available to
 any developer, enabling them to build sophisticated, natural language chatbots into their new and existing applications.
 Amazon Lex provides the deep functionality and flexibility of natural language understanding (NLU)
 and automatic speech recognition (ASR) so developers can
 build highly engaging user experiences with lifelike, conversational interactions, and create new categories of products.
Sources:
https://docs.aws.amazon.com/lex/latest/dg/what-is.html
https://docs.aws.amazon.com/lex/latest/dg/security-iam.html
https://docs.aws.amazon.com/lex/latest/dg/security-iam.html

Implementing in GO

1) Retrieve data from kendra

I use the Amazon Kendra API directly:

parameters := &kendra.RetrieveInput{
        IndexId:   aws.String("1d936da5-4a20-462c-826f-bdddaa59e5e8"),
        QueryText: aws.String("What is lex?"),
    }
    // do retrieve
    resp, err := client.Retrieve(context.Background(), parameters)

2) Stuff/Langchain

In the end this should be a template solution, but to measure timing I just concat strings:

    prompt := pre
    max := 3
    for i, doc := range query.ResultItems {
        if i >= max {
            break
        }
        prompt = prompt + " Document Title: " + *doc.DocumentTitle
        prompt = prompt + " Document Excerpt: " + *doc.Content
    }

So some checks and using templates will add a few microseconds to the timing.

3) OpenAI

OpenAI just uses a REST api, for the request ther is a commnunity solution: github.com/sashabaranov/go-openai.

With that I call the Chat completion API call of openAI

    resp, err := client.CreateChatCompletion(
        context.Background(),
        openai.ChatCompletionRequest{
            Model: openai.GPT3Dot5Turbo,
            MaxTokens: 300,
            Messages: []openai.ChatCompletionMessage{
                {
                    Role:    openai.ChatMessageRoleAssistant,
                    Content: prompt,
                },
            },
            Temperature: 0,
        },
    )

GO Size

With GO you get a compile binary with all libraries included:

ls -lkh dist
total 9728
-rwxr-xr-x@ 1 gernotglawe  staff   9,5M 15 Sep 18:03 rag

So overall size is 9,5 M.
Compared to 490M, this is about 50 times smaller.

With ARM bases Lambda it is even smaller:

ls -lkh dist
total 9348
-rwxr-xr-x@ 1 gglawe  staff   6,6M 20 Sep 18:34 bootstrap
-rw-r--r--@ 1 gglawe  staff   2,6M 20 Sep 18:34 bootstrap.zip

GO Speed

GO Run:

Amazon Lex is an AWS service for building conversational interfaces for applications using voice and text.
It allows developers to build sophisticated, natural language chatbots into their new and existing applications,
using the same conversational engine that powers Amazon Alexa. With Amazon Lex, developers can create highly
engaging user experiences with lifelike, conversational interactions and
take advantage of natural language understanding (NLU) and automatic speech recognition (ASR) capabilities.
Additionally, Amazon Lex enables developers to quickly build conversational chatbots.
Sources:
https://docs.aws.amazon.com/lex/latest/dg/what-is.html
https://docs.aws.amazon.com/lex/latest/dg/security-iam.html
https://docs.aws.amazon.com/lex/latest/dg/security-iam.html

And 10 iterations:

./test.sh  0,11s user 0,13s system 0% cpu 46,088 total

73 seconds/46 seconds = 1,58

So speed is about 46 seconds, which is about 50% faster. The 8% is rounded because there is still some checking etc to be done.

Moving to Lambda

A few considerations:

Language

Note the default language of the kendra data source. You need to set the language of the query to the same language.
For German you need to:

parameters := &kendra.RetrieveInput{
        IndexId:   &index,
        QueryText: &query,
        AttributeFilter: &types.AttributeFilter{
            AndAllFilters: []types.AttributeFilter{
                {
                    EqualsTo: &types.DocumentAttribute{
                        Key: aws.String("_language_code"),
                        Value: &types.DocumentAttributeValue{
                            StringValue: aws.String("de"),
                        },
                    },
                },
            },
        },
    }

In Python it would look like :

attribute_filter_json='''
  { "AndAllFilters": [
              {
                  "EqualsTo": {
                      "Key": "_language_code",
                      "Value": {
                          "StringValue": "de"
                      }
                  }
              }
          ]
  }
  '''

attribute_filter = json.loads(attribute_filter_json)
retriever = AmazonKendraRetriever(index_id=kendra_index_id, attribute_filter = attribute_filter)

Timeout

The default timeout of lambda is 3 seconds. As the results from the LLM api can take some time, you need to increase the timeout to at least a minute.

A typical log entry looks like this:

2023/09/20 16:34:53 INFO Lambda start
2023/09/20 16:34:53 INFO Kendra start
2023/09/20 16:34:54 INFO Kendra end
2023/09/20 16:34:54 INFO OpenAI start
2023/09/20 16:35:15 INFO OpenAI end
...LLM output...
2023/09/20 16:35:15 INFO Lambda end

AWS Managed Role

As the Kendra retrieve API is quite new, the AWS managed role does not have the permission to call it.
You need to add the permission to the role.

Conclusion

Langchain great for testing several choices. When you have found your model and type of langchain, you can optimize for size and speed. With GO you can get 50 times less size and 50% faster speed.

If you need consulting for your serverless project, don't hesitate to get in touch with the sponsor of this blog, tecRacer.

For more AWS development stuff, follow me on dev https://dev.to/megaproaktiv.
Want to learn GO on AWS? GO here

AWS Lambda recursive loop detection - but not (yet) for S3

Gernot Glawe — Mon, 17 Jul 2023 16:05:44 +0000

Recursive loops in AWS Lambda are the root of all evil because you create a large bill quickly.

You might think the danger is over because of new AWS recursive loop detection. But what about S3 buckets and Lambda? It's time for a closer look:

A word of caution: Be very careful when you test this. You could create large AWS bills.

TL;DR: Never ever use the same bucket for invoking a Lambda and write the output.

Some testing infrastructure:

1) A S3 bucket (CDK, GO)

    bucky := awss3.NewBucket(stack, aws.String("incoming-ring"), &awss3.BucketProps{
        BlockPublicAccess: awss3.BlockPublicAccess_BLOCK_ALL(),
        RemovalPolicy:     awscdk.RemovalPolicy_DESTROY,
    })

2) A Lambda Function Resource

    ringFunction := awslambda.NewFunction(stack, aws.String("ring"),
        &awslambda.FunctionProps{
            Description:            aws.String("ring - test recursive loop stop"),
            FunctionName:           aws.String("ring"),
            LogRetention:           awslogs.RetentionDays_THREE_MONTHS,
            MemorySize:             aws.Float64(1024),
            Timeout:                awscdk.Duration_Seconds(aws.Float64(10)),
            Code:                   awslambda.Code_FromAsset(&lambdaPath, &awss3assets.AssetOptions{}),
            Handler:                aws.String("main"),
            Runtime:                awslambda.Runtime_GO_1_X(),
            DeadLetterQueueEnabled: aws.Bool(true),
            DeadLetterQueue:        dlq,
        })

3) A Lambda Function, which gets the PutObject Event and write to the same bucket:

    someString := "hello world\nand hello go and more"
    myReader := strings.NewReader(someString)

    resp, err := client.PutObject(context.TODO(), &s3.PutObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(s3input),
        Body:   myReader,
    })

4) An event, which triggers the Lambda function each time an object is Put to the bucket:

    myHandler.AddEventSource(event.NewS3EventSource(bucky, &event.S3EventSourceProps{
        Events:  &[]awss3.EventType{awss3.EventType_OBJECT_CREATED,},
    }))

Now we have:

And because I am very careful, I decide to create a recursive loop stopper inside the Lambda function:

    GlobalCounter++
    if GlobalCounter > 20 {
        log.Fatal("Counter exceeded\n")
        os.Exit(1)
    }

And in the Console, we set the concurrency to 1:

When we test the recursive loop, we always have a browser window with the concurrency open. Because: Setting the concurrency to 0 is the fastest way to stop an endless loop.

The test

Lets test whether S3-Lambda-S3 recursive loops are still possible:

We use saw to tail lambda log:

saw watch /aws/lambda/ring

And then start the recursive loop:

aws s3 cp README.md s3://ring-incomingring3eb9a9b9-1a8q8cp8qm2ch/readme1.md
upload: ./README.md to s3://ring-incomingring3eb9a9b9-1a8q8cp8qm2ch/readme1.md

Very fast you get the logs:

[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) INIT_START Runtime Version: go:1.v18 Runtime Version ARN: arn:aws:lambda:eu-central-1::runtime:ccb68acb59818f9df9b10924cc6c83ca6eaf4067f70ba861c0e211b59e8af729
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) START RequestId: 10fba020-0e93-4430-a9d7-728d34b146c9 Version: $LATEST
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) Counter: 1
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) Etag:  446cce24a7c945503232494e881150ec
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) Seq:  0064B55F8E25E34383
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) END RequestId: 10fba020-0e93-4430-a9d7-728d34b146c9
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) REPORT RequestId: 10fba020-0e93-4430-a9d7-728d34b146c9   Duration: 92.67 ms  Billed Duration: 93 ms  Memory Size: 1024 MB    Max Memory Used: 39 MB  Init Duration: 110.22 ms
[2023-07-17T17:34:40+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) START RequestId: 97cf261e-b272-437f-9637-b7573cfb0fa8 Version: $LATEST
[2023-07-17T17:34:40+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) Counter: 2
[2023-07-17T17:34:40+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) Etag:  a9c4c5f405a3e4cc6ba3b6b355da0e71
[2023-07-17T17:34:40+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) Seq:  0064B55F8FD24FBE14
[2023-07-17T17:34:41+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) END RequestId: 97cf261e-b272-437f-9637-b7573cfb0fa8
[2023-07-17T17:34:41+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) REPORT RequestId: 97cf261e-b272-437f-9637-b7573cfb0fa8   Duration: 27.20 ms  Billed Duration: 28 ms  Memory Size: 1024 MB    Max Memory Used: 40 MB

The interesting part is:

Counter: 1
Counter: 2
...
Counter: 19
Counter: 20
Counter exceeded

And because AWS BLOG about recursive loops says:
"Lambda now detects the function running in a recursive loop between supported services after exceeding 16 invocations. It returns a RecursiveInvocationException to the caller."

We are now sure that the loop detection for S3 is not working because we counted to 20!

And although I implemented a loop stopper very fast, I started a recursive loop:

So now I set the concurrency to 0 again and delete the Lambda.

Summary

It's proven:

Recursive Loop detection does not stop S3 recursive loops, so do use different buckets for event trigger and Lambda output!

So be careful out there.
And checkout my website go-on-aws or - even better :) - use my new udemy course GO on AWS - Coding, Serverless and Infrastructure as Code

Enjoy building!
Gernot

A new simple approach to diagram as code on AWS with CDK and D2

Gernot Glawe — Sun, 02 Apr 2023 16:54:09 +0000

A diagram should convey a clear message about the intention of the architecture. For this message, you only need a few primary resources. Most generated diagrams are overloaded. This new app generates a diagram as code from your annotations in the CDK code.

When I am doing AWS Training, participants often state the wish that diagrams are automatically drawn from AWS resources.
As this is possible with solutions like cfn-diag, the generated diagrams usually need more information to be helpful.
A diagram should convey a clear message about the architecture's intention. For this message, you only need a few primary resources. The other "glue" resources are necessary for the proper functions but not to show the logical view of a diagram.

The traditional approaches to this problem are:

1) Only use it if you have a small number of resources, like the CloudFormation drawing tool.

2) Filter resources from the diagram afterwards, like cfn-dia

3) Draw manually

All solutions have a low degree of automation and do not synchronize with changes in the infrastructure.

A helpful solution should:

1) be fully automated, with no manual steps
2) Be 100% synched with deployed resources. Show what is and not what should be
3) Take meaningful names from the Code part of Infrastructure as Code. The often used CloudFormation logical ID is not helpful.

And my main requirement:

4) Show only those resources which I think are needed, to show the main purpose of the architecture

A simple approach to DaC

1) Generate D2 code from CDK code. D2 will take care of rendering.

2) Poll CloudFormation for the deployment status of the resource. Only resources which are beeing created or are created are shown.

3) Use the CDK Construct ID as title for a resources, not the logical ID or the physical ID.

4) Add very few information to the CDK code, to keep thinks simple:

Show - the Construct is shown on the diagram
Connection - a connection is drawn from source Construct to target Construct
Contains - Resources like a VPC, which contain other resources

Example with a serverless architecture

I take simple serverless architecture as a starting point.

These are the lines added for the diagramm information into the TypeScript file:

import * as d2 from "../lib/d2";
d2.Show(fnTS)
d2.Show(fnPy)
d2.Show(fnGO)
d2.Show(bucky)
d2.Show(topic)
d2.Connection(bucky, topic)
d2.Connection(topic, fnTS)
d2.Connection(topic, fnGO)
d2.Connection(topic, fnPy)
d2.Connection(fnTS, table)
d2.Connection(fnGO, table)
d2.Connection(fnPy, table)

We have a S3 Bucket named "bucky", some Lambda FNunctions, a SNS topic and a DynamoDB table.

And I copy a small TypeScript file into the lib directory.

I deploy the infrastructure:

cdk deploy

Then I generate a d2 diagramm description with cdk2d2:

 cdk2d2 generate adotstarter-auto .

Where the first parameter is the name of the stack and the second parameter is the directory of the CDK app. This creates a file adotstarter-auto.d2.

With d2 I generate a png of this file:

d2 adotstarter-auto.d2 adotstarter-auto.png

Now we got:

instead of the quite confusing CloudFormation diagram:

You see on first sight what the architecure is doing and the name of the Functions are not adotstartergo2987B222, which would be the CloudFormation logical ID, but the Construct ID from the code:

const fnGO = new aws_lambda.Function(this, 'adotstarter-go', {

Inside cdk2d2

With the information from the CDK manifest file from the cloud-assembly, q construct like the lambda function "fnGO" is generated as D2:

adotstartergo2987B222: adotstarter-go{
 icon: https://icons.terrastruct.com/aws%2FCompute%2FAWS-Lambda_Lambda-Function_light-bg.svg
 style.fill:"lightgreen"
}

Where the id of the artefact is the cfn logicalid and the title is the CDK construct ID.
The icon is set according to the CloudFormation Resource type. See icon.go for mappings. The color style.fill is set according to the deployment state of the resource.

So when you change the name of the construct, the diagram is updated with the new name.

You define the resources which should be shown on the diagramm in your code, because only you know which resources are the main resources.
For example the helper lambda which CDK uses to set the retention time on the CloudWatch logs should usually now be shown.

Now to the synchronisation part

I destroy the stack. Now we use three terminal windows:

1) cdk2d2 Script generator
2) d2 rendering
3) CDK deploy.

All running in the same CDK app base directory.

Terminal 1

On terminal 1 we start:

cdk2d2 watch adotstarter-auto .

This will update the stackname.d2 each 4 seconds with then current deployment state of the construct resource.

Terminal 2

On this terminal we start the rendering process with:

d2 --watch adotstarter-auto.d2

This will open a browser windows, where d2 renders the live state of the stack.de diagram.

Terminal 3

Here we start the CDK deployment.

cdk deploy

First picture

Will show nothing, because the stack does not exists.

Second picture

Will show the stack in the upper left with the total number of resources and the deployment percentage.

Life state change

Resources which are beeing created are colored yellow.

Resources which are created are colored green.

Fully deployed state

100% resources created.

For each resource you will see the deployment status as color.
This way you can watch CloudFormation deploying your resources. The order of the deployment will surprise you sometimes...

Dependencies

The metainformation are stored as metadata:

Show

resource.node.addMetadata("Show", "true")

Connection

resource.node.addMetadata("Connection", otherResource.node.id)

Container

resource.node.addMetadata("Container", otherResource.node.id)

Container example

Of course we are talking about a graphical container!

For a VPC you add a "Container" relationship. Only one level is supported, so vpc-subnet will not work.

Now try it out for yourself

Please send suggestions, errors, images for the gallery etc. Add prs for new features to cdk2d2 issues. There are also some more examples. Releases can be found here.

If you like it, please star the repo!

Happy building!

If you need consulting for your serverless project, don't hesitate to get in touch with the sponsor of this blog, tecRacer.

For more AWS development stuff, follow me on dev https://dev.to/megaproaktiv.
Want to learn GO on AWS? GO here

Check external links in PowerPoint automatically from the command line

Gernot Glawe — Thu, 23 Mar 2023 15:04:34 +0000

How do I check whether all my external HTTPs links in the PowerPoints slides are OK? I asked chatgp.

The answer was not right, but it included the idea of just parsing the XML from powerpoint.

So it boiled down to simple steps:

1) Looking in directory ppt/slides/_rels/slide
2) Finding links with xmlquery

elements := xmlquery.Find(doc, "//Relationship")

3) Do a

    response, errors := http.Get(web)

And check it all:

 go run main.go check testdata/someuris.pptx
❌ Target https://www.john-doe.com/ is not reachable in slide: slide3
☑️ Target https://www.google.de/ is reachable in slide: slide3
❌ Target https://www.john-doe.com/ is not reachable in slide: slide2
☑️ Target https://www.google.de/ is reachable in slide: slide2

Use the app: https://github.com/megaproaktiv/linkchecker
With binaries released: https://github.com/megaproaktiv/linkchecker/releases/tag/v0.1.4

Simple as that.
Enjoy!

For more good GO and AWS stuff:
https://www.go-on-aws.com/

Check S3 storage classes from command line

Gernot Glawe — Wed, 15 Mar 2023 17:50:07 +0000

... it's not so easy, so I created a small GO program:

https://github.com/megaproaktiv/storageclass

Scanned 1.8 Million files in 5 minutes.

It will show you the storage classes of all buckets like:

Bucket: archivemac123
STANDARD 3747
STANDARD_IA 183
8.6 GB

Enjoy!

Switch & Leapp-cli - AWS session management 100% command line

Gernot Glawe — Sun, 05 Mar 2023 13:33:59 +0000

According to the well architected framework you should not store keys as cleartext. So why do you store your AWS credentials in a credential file as clear unencrypted text? The answer is: Because it is convenient! I show you a way to handle your static or SSS AWS credentials simple and secure.

You need three little tools:

Leap, the new leapp-cli and switchaws.  You will get a zero byte credentials file, temporal credentials and command line handling with fast & easy installable tools

Quick Start for the impatient

Assuming you have an AWS SSO login and a profile called letsbuild. After installing the tools you can start the session with these two commands:

One

leapp session start letsbuild

Two

switch letsbuild

Before

ls -l ~/.aws/credentials
-rw-------@ 1 jdoe  staff  0  3 Mär 12:45 /Users/jdoe/.aws/credentials

After

ls -l ~/.aws/credentials
-rw-------@ 1 jdoe  staff  831  3 Mär 12:45 /Users/jdoe/.aws/credentials

and also filled environment variables like:

AWS_DEFAULT_REGION=eu-central-1
AWS_DEFAULT_PROFILE=letsbuild
AWS_REGION=eu-central-1
AWS_ACCESS_KEY_ID=ASIA3SHER36FBEBMXR22
AWS_SECRET_ACCESS_KEY=P9kWKJKgsOWBMOAW7a5aRI7apt31CXAuXpfNsoeC
AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjECwaCWV1LXdlc3QtMSJGMEQCIE0KfNquOOCxf9UuXxgnWnvCeK6JeYWnqXmmz48fnzP+AiAwWRh7qnXXR8FkEfpkc5...9UmXa9PxI4Qj0ObcxLP8/YQBbIkCgs0+C7xWj/e1lmKhSLlhjRI04Mlj1Y9EomihaH/YEGEAXJ1sySpcgZJAHW6n02E7LvUAhV9ODYX66AFbRdqRrFZXIlDN5J0MalU18gNts3d1OA==

So you can start using the profile:

aws sts get-caller-identity
{
    "UserId": "AIDAAAABBBBCCCAW",
    "Account": "777555666888",
    "Arn": "arn:aws:iam::777555666888:user/jdoe"
}

Alternative approaches

Using profiles only with leap

1) start session
leapp session start letsbuild

2) use profiles with each call:

aws sts get-caller-identity --profile letsbuild

Configure leaps for default profile

The downside:  The aws cli first looks for credentials in the environment variables. If it finds AWS_ACCESS_KEY_ID & co , the profile in the credentials file will not be used.

What do you get out of this approach?

Secure storage of credentials keys

Not stored as clear text as file, but in the MAC key chain

Secure usage of temporal credentials

With AWS SSO you always get temporal credentials. With a static IAM user access key, you would use static credentials. leapp uses these static keys to generate temp credentials.

Easy installation and long term stability

I have used awsume a long time. Then I got a new Macbook and lost 1/2 hour installing different python versions. So I programmed switchaws in go to get a single executable. And, yes: I declare guilty of the "not invented here" syndrom :) .

Installation is straightforward:

 1) copy the matching binary link in a directory which is in your $PATH 
2) copy the wrapper tile also in that directory 
3) Set an alias

 and you are done!

Summary

Leapp works great with either static ACCESS_KEY or sso.

Thanks

Photo by Isaac Li Shung Tan on Unsplash

Building an AWS Lambda Telemetry API extension for direct logging to Grafana Loki

Gernot Glawe — Fri, 17 Feb 2023 09:04:47 +0000

In hybrid architectures, serverless functions work together with container solutions. Lambda logs have to be translated when you don`t choose CloudWatch Logs. The old way of doing this is through subscription filters using additional Lambda functions for log transformation. With the Lambda Telemetry API there is a more elegant, performant and cost-effective way. I am using Grafana Loki as a working example and show you how to build a working Lambda-Loki Telemetry APi extension.

Using a centralized Logging solution in hybrid environments

Using e.g. Grafana Loki as your centralized logging solution, where container and Lambda Functions should ship there logs is quite common.

As described in the Lambda-Promtail documentation you have to use a CloudWatch Logs subscription filter to read Lamba logs. As serverless hero Yan Cui stated in this article, the Transform Lambda eats up your Lambda concurrency and adds additional costs. You could leverage that by using additional Kinesis streams, or you use the Lambda Telemetry API as an Lambda extension.

That makes this lighter architecture possible:

Introducing the Lambda Telemetry API

With the Lambda Telemetry API, Lambda extensions can directly receive telemetry data from Lambda.

Lambda Extensions

With the Lambda Extension API, you can use a Lambda Layer, which is called as an independent process in the execution environment and can continue to run after the function invocation is fully processed.

Lambda Telemetry API

The Telemetry API gets events from these telemetry streams:

Platform telemetry – Logs, metrics, and traces, which describe events and errors related to the execution environment runtime lifecycle, extension lifecycle, and function invocations.
Function logs – Custom logs that the Lambda function code generates.
Extension logs – Custom logs that the Lambda extension code generates.

An extension app can get all logging output from a Lambda Function by subscribing to the Function events. The event types are defined in the developer guide.

By not subscribing to Platform events, you will not get events like the platform.start event, which creates the START log entries. You will only get the output from the Lambda Function itself. This way, the logs will be better parseable.

Grafana Labs Loki

The Grafana tools are often used in containerized environment. So I will build the extension for Grafana Loki, which is the logging solution.

Fun fact: My first approach was using parseable. But it turned out that parseable is not capable of working with IAM roles at the moment.

From Lambda Telemetry API Event to promtail

You can push events into Loki with the promtail api or the newer Loki api. The api is described in the documentation. I will use the /api/prom/push call.

The Extension Code

As AWS states in Creating extensions using the Telemetry API
that it is recommended to use a compiled language such as Golang or Rust. So of course, I am using GO.

There are some extension examples github. I am using go-example-telemetry-api-extension as a starting point. You can see the code in my Lambda-Telemetry-API-Loki repository.

Who's calling who

The extension works in these main steps:

1) Register the extension with Extensions API
2) Start an HTTP listener to receive events
3) Subscribe to any of the types of Platform, Function, Extension
4) Receive events and dispatch them to the target, here Loki

1) Register

This is done via the Register call of the extension api.

See extensionApi/client.go:

baseUrl := fmt.Sprintf("http://%s/2020-01-01/extension", os.Getenv("AWS_LAMBDA_RUNTIME_API"))
//...
const action = "/register"
url := e.baseUrl + action
///
httpRes, err := e.httpClient.Do(httpReq)

2) Listener

The struct is composed of an HTTP server and a queue. For performance reasons, events are not directly pushed to Loki but are queued up to the DISPATCH_MIN_BATCH_SIZE environment variable.

The Lambda extension gets its environment from the Lambda Function, so you set the values in the Lambda Function API.

type TelemetryApiListener struct {
    httpServer *http.Server
    // LogEventsQueue is a synchronous queue and is used to put the received log events to be dispatched later
    LogEventsQueue *queue.Queue
}

3) Subscribe

In telemetryApi/client.go the function Subscribe calls the baseUrl of the telemetry extension:

baseUrl := fmt.Sprintf("http://%s/2022-07-01/telemetry", os.Getenv("AWS_LAMBDA_RUNTIME_API"))

The possible parameters are described in the Lambda Telemetry API reference.
For example, if you expect high-volume data, you should tune the batch size and the buffering parameter of the subscribe call.

Now that we have an extension, we need to build it & install it to Lambda Functions.

Build & Install the Telemetry Layer

In the repository I build the layer binaries with the help of the great taskfile:

cd loki_extension
task build

That calls:

GOOS=linux GOARCH=amd64 go build -o dist/extensions/grafana-loki-extension main.go

Which creates an executable binary for amd64 architecture. If you use arm architecture, you have to change the GOARCH variable. The created binary is zipped and published as an Lambda Layer.
To set the Layer ARN in the Lambda Function, I store the ARN.
With the extension, you define the compatible runtimes --compatible-runtimes for the Lambda Layer.

The whole call:

ARN=`aws lambda publish-layer-version --layer-name "grafana-loki-extension" --region eu-central-1 --compatible-runtimes nodejs16.x go1.x python3.9 --zip-file  "fileb://extension.zip" --query "LayerVersionArn" --output text` && echo $ARN

Output:

arn:aws:lambda:eu-central-1:012345679812:layer:grafana-loki-extension:1

Now the layer is installed in the current AWS account. We can use the Layer in Lambda Functions without changing one line of the function code.

The Test Architecture Setup

The setup has three components:

1) The telemetry layer with the Loki extension
2) A Grafana Loki containers and other Telemetry tools as docker-compose setup
3) Test Lambda Functions for the Telemetry API written in TypeScript, Python and Go

1) The telemetry Layer

This is the Layer I installed. For the Lambda Function you reference the ARN:

arn:aws:lambda:eu-central-1:012345679812:layer:grafana-loki-extension:1.

2) Grafana Tools

I am using a setup from the book Cloud-Native Observability with OpenTelemetry

This is not for production. For better setups, see the other articles of this series.

The setup includes:

The Opentelemetry-collector
Prometheus
Loki
Promtail
Grafana

For a test environment, we start the containers on a Cloud9 Server in a public network.

2.1. Create a Cloud 9 environment

2.2. Add rules to the security group of the instance open to your IP, and open the Loki/Promtail api for Lambda IP addresses like:

For production you would have an authentication setup or/and Lambda in a VPC in a private subnet, so that you got fixed IPs.

2.3. Install Docker Compose on Cloud9:

sudo curl -L "https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version

Output:

AWSReservedSSO_AWSAdministratorAccess_b58ba5bc1d953bb1:~/environment $ docker-compose --version
docker-compose version 1.23.1, build b02f1306

2.4. Clone the repo:

git clone https://github.com/megaproaktiv/Lambda-Telemetry-API-Loki.git
cd Lambda-Telemetry-API-Loki/grafana-container/

2.5. Start the Grafana environment:

docker-compose up

Output:

Creating network "grafana-container_observability" with the default driver
...

2.6. Open Grafana

Now the Grafana tools should be running, so open the public IP of your Cloud9 instance on port 3000.

Example: http://3.67.195.192:3000/?orgId=1

You should see something like:

3) Lambda Functions

In the directory lambda_telementry_api the Lambda Resources are defined with CDK.

If you use another layer version, you have to adapt the url in lib/lambda_telementry_api-stack.ts:

const extensionName = "grafana-loki-extension"
const layerVersion="1"
const loki_ip="3.67.195.192"
const distpatch_min_batch_size = "10"

const lambdatelematryApiLayerArn = "arn:aws:lambda:"+region+":"+account+":layer:"+extensionName+":"+layerVersion
const ltaLayer = lambda.LayerVersion.fromLayerVersionArn(this, "ltalayer", lambdatelematryApiLayerArn)

Also set the Loki IP from the last step.

Make sure you got docker running and deploy the CDK stack with:

task deploy

This will build all Lambda Functions and deploy them.

Now we have established a simpler logging setup:

Test the setup

The Lambda functions take all PutObject events from an S3 Bucket and write the object key into a DynamoDB.

There is a test script inside the lambda_telementry_api directory, which puts object in the created bucket:

/test/traffic.sh

Stop the script after approx. 20 copy calls.

The output looks like:

So 12 Feb 2023 14:42:17 CET
upload: ./readme.md to s3://lambdatelementryapistack-incoming0b397865-hjxvhc2842jy//test-4-0-0-
So 12 Feb 2023 14:42:18 CET
upload: ./readme.md to s3://lambdatelementryapistack-incoming0b397865-hjxvhc2842jy//test-4-0-1-

The CloudWatch logs of the Lambda Function now show the output from the extension also. The log level is set to verbose, so you can see many events:

Log events from the extension

time="2023-02-12T13:42:18Z" level=info msg="[main] Starting the Telemetry API extension" pkg=main
time="2023-02-12T13:42:18Z" level=info msg="[main] Registering extension" pkg=main
time="2023-02-12T13:42:18Z" level=info msg="[client:Register] Registering using baseURLhttp://127.0.0.1:9001/2020-01-01/extension" pkg=extensionApi
time="2023-02-12T13:42:18Z" level=info msg="[client:Register] Registration success with extensionId 5b43e095-292c-4d3c-94d0-be9cea43c78d" pkg=extensionApi
time="2023-02-12T13:42:18Z" level=info msg="[main] Registation success with extensionId5b43e095-292c-4d3c-94d0-be9cea43c78d" pkg=main
time="2023-02-12T13:42:18Z" level=info msg="[main] Starting the Telemetry listener" pkg=main
time="2023-02-12T13:42:18Z" level=info msg="[listener:Start] Starting on addresssandbox:4323" pkg=telemetryApi
time="2023-02-12T13:42:18Z" level=info msg="[main] Subscribing to the Telemetry API" pkg=main
time="2023-02-12T13:42:18Z" level=info msg="[client:Subscribe] Subscribing using baseUrl:http://127.0.0.1:9001/2022-07-01/telemetry" pkg=telemetryApi

The queue waits for shutdown or if the distpatch_min_batch_sizeis reached:

time="2023-02-12T13:58:16Z" level=info msg="[listener:http_handler] logEvents received:1 LogEventsQueue length:2" pkg=telemetryApi
time="2023-02-12T14:02:18Z" level=info msg="[listener:http_handler] logEvents received:2 LogEventsQueue length:11" pkg=telemetryApi
time="2023-02-12T14:02:18Z" level=info msg="[dispatcher:Dispatch] Dispatching :11 log events" pkg=telemetryApi

Log events from the Platform

INIT_START Runtime Version: python:3.9.v16  Runtime Version ARN: arn:aws:lambda:eu-central-1::runtime:07a48df201798d627f2b950f03bb227aab4a655a1d019c3296406f95937e2525
END RequestId: 1af7f8ab-94e7-4a7a-b3c8-db06eedf874b
REPORT RequestId: 1af7f8ab-94e7-4a7a-b3c8-db06eedf874b  Duration: 132.32 ms Billed Duration: 133 ms Memory Size: 1024 MB    Max Memory Used: 83 MB  Init Duration: 783.94 ms

Log events from the Function

The Python Lambda lambda_telementry_api/lambda/py/app.py logs the DynamoDb Response:

    for record in message['Records']:
        itemKey = record['s3']['object']['key']
        response = putDynamoItem(environment['Table'], itemKey)
        print(response)

In the cloudwatch logs we see:

{'ConsumedCapacity': {'TableName': 'items', 'CapacityUnits': 1.0}, 'ResponseMetadata': {'RequestId': 'DPKG7GR6V8G0SA2ATEPR2M4UB3VV4KQNSO5AEMVJF66Q9ASUAAJG', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'Server', 'date': 'Sun, 12 Feb 2023 13:42:19 GMT', 'content-type': 'application/x-amz-json-1.0', 'content-length': '62', 'connection': 'keep-alive', 'x-amzn-requestid': 'DPKG7GR6V8G0SA2ATEPR2M4UB3VV4KQNSO5AEMVJF66Q9ASUAAJG', 'x-amz-crc32': '2296128304'}, 'RetryAttempts': 0}}

Now we should see the same event in Grafana.

Query the Function logs in Grafana Loki / Grafana

Loki works with labels. To find the Lambda Functions, the promtail client is initialized with the Function name as tag in loki/promtail.go:

    function_name = os.Getenv("AWS_LAMBDA_FUNCTION_NAME")
    labels := "{source=\""+source_name+"\",function=\""+function_name+"\"}"
    lokiIp := os.Getenv("LOKI_IP")
    if len(lokiIp) == 0 {
        panic("LOKI Ip undefined")
    }
    conf = promtail.ClientConfig{
        PushURL:            "http://"+lokiIp+":3100/api/prom/push",
        Labels:             labels,
        BatchWait:          5 * time.Second,
        BatchEntriesNumber: 10000,
        SendLevel:          promtail.INFO,
        PrintLevel:         promtail.ERROR,
    }

All Lambda Function event are tagged as Source=Lambda

In the Grafana browser start Loki (2) with the explore icon (1):

So after the first events arrived in Loki, you see these labels:

The Label source shows that we have Lambdaevents and the three different function tags let us query each of the Functions.

When you query these labels, you get only the Function logs, not the Platform or extension logs as in CLoudWatch:

So the extension is working and all Function Logs are shipped to Loki!

Summary

The extension API and the telemetry extensions of Lambda open up possibilities for connecting telemetry data to AWS and non-AWS targets.

If the extension is up and running, you have much more flexibility than with CloudWatch alone. On the other hand, Lambda is highly integrated with CloudWatch logs, and you have to configure no additional Layers to get that.

So for the standard serverless App, the extensions should not be necessary. But for high volume or hybrid projects, it provides more functionality and saves costs because you do not need additional resources in the form of Log Functions.

The main advantage is that logging Lambda Functions do not use up the Lambda concurrency. Also, you can decide which Log events you want to see.

This article concludes the series about Serverless Observability.

If you need consulting for your serverless project, don't hesitate to get in touch with the sponsor of this blog, tecRacer.

For more AWS development stuff, follow me on dev https://dev.to/megaproaktiv.

Solved: go mod tidy: warning: "all" matched no packages / GO debugger does not work in VSCode

Gernot Glawe — Fri, 27 Jan 2023 08:11:09 +0000

In just some projects go mod tidy did not work.

Now I figured out, why:

If you are inside a working directory, which is a symbolic link, it fails with warning: "all" matched no packages..

When you cd to the linked directory, it works.

"The expert is amazed and the layman is confused!" - Ancient german proverb!

Update: This also happens with the VSCode debugger.
Could be only a MAC M1 problem...

Serverless Spy Vs. Spy Chapter 3: X-Ray vs Jaeger - Send Lambda traces with open telemetry

Gernot Glawe — Tue, 20 Dec 2022 10:06:10 +0000

In modern architectures, Lambda functions co-exist with containers. Cloud Native Observability is achieved with open telemetry. I show you how to send open telemetry traces from Lambda to a Jaeger tracing server. Let's see how this compares to the X-Ray tracing service.

As the Lambda setup with Typescript and Python already had a good coverage in chapter 2, I will stick to GO here. The CDK code is easy to migrate.

Setting

)
Architecture overview

The Lambda Function (2) sends traces to the jaeger backend with the OpenTelemetry Protocol. Because we do not want to accept requests from the internet, Lambda has to run within the network of the VPC called basevpc. This VPC is created (1) at first. The jaeger container announces its IP via the AWS Serviced-Discovery service.

To access the frontend/UI of jaeger a Load Balancer is created between the internal jaeger service private IP and the internet.

The CDK code, the application code and jaeger itself are written on GO.

Lambda

Lambda Resources

AWS Lambda Resources

  1   lambdaPath := filepath.Join(path, "../dist/main.zip")
  2   adotLayer := lambda.LayerVersion_FromLayerVersionArn(this, aws.String("adotlayer"),
  3     aws.String("arn:aws:lambda:eu-central-1:901920570463:layer:aws-otel-collector-amd64-ver-0-62-1:1"))
  4   fn := lambda.NewFunction(this, aws.String("adotlambda"),
  5   &lambda.FunctionProps{
  6     Vpc: vpc,
  7     Handler: aws.String("main"),
  8     Runtime: lambda.Runtime_PROVIDED_AL2(),
  9     Tracing: lambda.Tracing_ACTIVE,
 10     Environment: &map[string]*string{
 11       "OPENTELEMETRY_COLLECTOR_CONFIG_FILE" : aws.String("/var/task/config.yml"),
 12       // "https://opentelemetry.io/docs/concepts/sdk-configuration/general-sdk-configuration/"
 13       "OTEL_SERVICE_NAME" : aws.String("documentcounter"),
 14     },
 15     AllowPublicSubnet: aws.Bool(true),
 16     Layers: &[]lambda.ILayerVersion{
 17         adotLayer,
 18     },
 19     },
 20   )

You have to define the following configuration, see chapter 2:

Line 2:3 - The Lambda Layer for the otel collector
Line 6 - run in the VPC
Line 1 - Set the configuration file location
Line 16 - Activate the layer

Lambda Code

In the application you have to do:

1. Configure the middleware to send traces

otelaws.AppendMiddlewares(&cfg.APIOptions)
ClientDDB = dynamodb.NewFromConfig(cfg)

2. Propagate the context through all functions:

From main:

tp, err := xrayconfig.NewTracerProvider(ctx)
//...
lambda.Start(otellambda.InstrumentHandler(HandleRequest, xrayconfig.WithRecommendedOptions(tp)... ))

to HandleRequest

func HandleRequest(ctx context.Context, s3Event events.S3Event) (string, error) {
//...
putItem(ctx,s3input)

to putitem

func putItem(ctx context.Context, itemID string){
//...
result, err := ClientDDB.PutItem(ctx,input)

)

In the app, at the end an s3 listobjects is performed, so that you have two AWS services in the traces.

See chapter 2 for more details.

Now Lambda could send traces, so we need a target. I chose Jaeger, an open-source, end-to-end distributed tracing, originally provided by Uber Technologies.

Jaeger Installation

VPC

We provide a VPN to run the ECS service - just a VPC with a private subnet.

Fargate Service

)
The JAEGER service

The front end will be provided on port 16686, the OTEL request will go to port 4317 via gRPC. All jaeger ports are described in the deployment part of the jaeger documentation.

To access the jager front end with a DNS name, you have to have a domain. So change the following configurations in jaeger/cluster.go:

var SERVICE_NAME = "jaeger"
var NAMESPACE = "otel.letsbuild-aws.com"
var HOSTED_ZONE_ID = "Z042038724KH99T9LFKK6"
var DNS_NAME = "service.letsbuild-aws.com"

In this example, I have created a subdomain "service.letsbuild-aws.com" for the Load Balancer. The NAMESPACE is used for service discovery. You do not need a real domain for service discovery.

To get jaeger up and running, there is an all-in-one image we use:

jaegertracing/all-in-one:1.39.0

The jaeger container can be configured via the environment:

"SPAN_STORAGE_TYPE":      aws.String("memory"),
"COLLECTOR_OTLP_ENABLED": aws.String("true"),
"LOG_LEVEL":              aws.String("debug"),

To keep it (almost) simple, the storage is set to memory. In production, you could use Cassandra, elasticsearch and other backends. As stated in the jaeger documentation, all CLI parameters can be set via ENV variables. To be able to receive otlp data, its enabled.

The management ui and otlp ports are configured for the container:

task.AddContainer(aws.String("jaegerContainer"),
    &ecs.ContainerDefinitionOptions{
        Image:         ecs.ContainerImage_FromRegistry(aws.String("jaegertracing/all-in-one:1.39.0"), nil),
        ContainerName: aws.String("jaeger-all"),
//...
        PortMappings: &[]*ecs.PortMapping{
            {
                ContainerPort: MANAGEMENT_PORT,
                HostPort:      MANAGEMENT_PORT,
                Protocol:      ecs.Protocol_TCP,
                // management
            },
            {
                ContainerPort: aws.Float64(4317),
                HostPort:      aws.Float64(4317),
                Protocol:      ecs.Protocol_TCP,
                // "otel-grpc"
            },
//...

See the jaeger/cluster.go file for the complete source.

Connect Lambda to Jaeger

On the jaeger side a namespace is configured:

namespace := awsservicediscovery.NewPrivateDnsNamespace(this, aws.String("oteltrace-namespace"),
    &awsservicediscovery.PrivateDnsNamespaceProps{
        Name:        aws.String(NAMESPACE),
        Description: aws.String("DNS service discovery subdomain"),
        Vpc:         vpc,
    },
)

This creates an entry in the private domain otel.letsbuild-aws.com:

On the Lambda side the first thing is to tell the adot Layer, where to find the config file:

"OPENTELEMETRY_COLLECTOR_CONFIG_FILE" : aws.String("/var/task/config.yml"),

Because Lambda apps are deployed into the directory /var/task on the Lambda micro-vm, you have to prepend the path /var/task.

You find the file here: app/config.yml

The configuration is added to the Lambda deployment package:

env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-s -w" -o ../dist/main main/main.go
chmod +x ../dist/main
mv ../dist/main ../dist/bootstrap
cp config.yml ../dist
cd ../dist && zip main.zip bootstrap config.yml

The lambda package build script

Configuration

In the configuration, we have three parts

1) The local receiver:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

It receives the traces.

2) The exporter

exporters:
  otlp:
    endpoint: jaeger.otel.letsbuild-aws.com:4317
    tls:
      insecure: true

Here the dns name from the awsservicediscovery is used for the ENDPOINT.

3) The pipelines

Now incoming receiver is piped to the outgoing exporter

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [ otlp]

Details are described in the OpenTelemetry documentation. As stated in chapter 2, not all configurations are valid here.

Compare X-Ray UI to Jager UI

X-Ray now

As the collector is not configured for X-Ray traces, we just see the data from the Lambda service, not the function:

Switch the configuration back to x-rays:

1) Change app/config.yml

Samples for the configurations are provided in

app/config-otel.yml
app/config-xray.yml

2) Deploy app

cd app
task fastdeploy

Then some traffic:

cd ..
task traffic

Then we see all nodes in the X-Ray Map view:

And some traces.

X-Ray Trace Map

Jaeger/Otel

Switch the configuration back to otel and deploy Lambda app again. After creating some traffic, you see traces in the jaeger ui.

Access the jaeger UI from the loadbalancer dns entry or your domain name.

Jaeger Trace Map

1) Choose Service documentcounter here
This is the name I set with the environment variable
OTEL_SERVICE_NAME, configured in the Lambda Resource.
2) The button [Find Traces] shows a graphical view (4) and the single traces

Click on a trace (3) to see the detail view:

Jaeger Timeline

Comparing both trace maps we notice the missing nodes with the AWS service icons.

Comparing the timelines, you see that the segments otellambda AWS::Lambda and otellamba AWS::Lambda::Function only appear in X-Ray, not on jaeger.

That is because only the Lambda Function sends traces, not the Lambda Service. In the "Cloud-Native" container world, usually, it is assumed that the container is already running. So the startup time is not interesting. In Lambda the micro-vm is started, when a request hits a cold start. If that happens often, it may affect your overall latency, so you want to have data. You may get the init duration also from the Lambda Logs. If you need information from the Lambda Resource, you might use the Lambda extensions and the AWS Lambda Telemetry API, which I will cover in the last chapter.

The detail information are almost the same:

Is there a winner?

Functionality

If you have to decide whether to use X-Ray or other services for your traces, ADOT is the more flexible choice. It provides more support from various sources.

For services that have a large AWS part, the X-Ray service provide some more functionality like creating nodes.

Cost

It depends on your metrics!

Otel open source tracing e.g. jaeger

I have seen some other posts, which stated that an extra tracing service would be cheaper, "because it is open source". If you compare the costs the tco have some more parts:

Costs of the running container

Price in eu-central-1
per vCPU per hour $0.04656
per GB per hour $0.00511
With 2 vCPU | 4 GB

Which would be 62.01 €/month

Cost of storage: depends on backend

In the production environment, you would like to set up an application load balancer with cognito authentication with additional costs.

X-Ray

The X-Ray server costs are $5.00 per 1 million traces in eu-central-1. You can also adapt the sample rate to not have a trace with each call.

Operations

The telemetry infrastructure setup is done only once. Once you have it running, there should be not much to do.

With X-Ray, there is no additional operational cost.

Usage

Because you provide the jaeger container yourself, you can adapt the size to the speed you need. In my tests the jaeger frontend seemed very much faster than the X-Ray aka CloudWatch Service Map.

Conclusion

With the sample apps from the opentelemetry-lambda repository the Lambda part itself was easy to implement. What took me some time was to provide the jaeger Fargate service with IaC ouside of an k8s environment. But with ECS and ServiceDiscovery that was easy in the end. This should be even more simple in an EKS environment with the jaegertracing helm-charts.

Using something else as tracing solution instead of X-Ray not looks like a good choice for AWS serverless projects.
But if you have a container solution up and running, otel would be a good choice for an environment, where container traces and Lambda traces are stored together.

Appendix: Quick Walkthrough

Clone repository

git clone https://github.com/megaproaktiv/adot-otelstarter.git
cd adot-otelstarter

Set region export AWS_REGION=yourregion, e.g.

  export AWS_REGION=eu-central-1

If CDK is not bootstrapped:

  task bootstrap

Create VPC

  task jaeger:deploy-vpc

Set Domain and Service configuration

Edit jaeger/cluster.go:

  var SERVICE_NAME = "jaeger"
  var NAMESPACE = "otel.letsbuild-aws.com"
  var HOSTED_ZONE_ID = "Z042035555KH99T9LFKK6"
  var DNS_NAME = "service.letsbuild-aws.com"

Create ECS cluster with jaeger service

  task jaeger:deploy-jaeger

Deploy Lambda Resources and function

  task deploy

Note: because of the ENI this could take a few minutes

Create Traffic

  task traffic

First surprising Contact with ChatGP: Can it solve simple AWS SDK tasks?

Gernot Glawe — Sun, 18 Dec 2022 08:21:41 +0000

I heard so many things, so I wanted to give https://chat.openai.com/chat a GO.

Can it help me in programming or even replace me?

My question:

"show me the code to list all CloudFormation stack names in an AWS account in GO with SDK v2 in clean code"

Does the code work out of the box?

Short answer: No.
But it would save me google search time.

What did I have to change?

A surprisingly small amount of code:

diff --git a/main.go b/main.go
index 01d6286..2b02055 100644
--- a/main.go
+++ b/main.go
@@ -4,34 +4,33 @@ import (
        "context"
        "fmt"

-       "github.com/aws/aws-sdk-go-v2/config"
+       "github.com/aws/aws-sdk-go-v2/aws"
        "github.com/aws/aws-sdk-go-v2/service/cloudformation"
-       "github.com/aws/aws-sdk-go-v2/service/cloudformation/types"
 )

 func main() {
        // Initialize the AWS configuration
-       cfg, err :=  config.LoadDefaultConfig(context.Background())
+       cfg, err := aws.NewConfig()
        if err != nil {
                panic("Unable to create AWS configuration: " + err.Error())
        }

        // Create a new CloudFormation client
-       cfnSvc := cloudformation.NewFromConfig(cfg)
+       cfnSvc := cloudformation.New(cfg)

        // Set up the input parameters for the ListStacks API call
        input := &cloudformation.ListStacksInput{
-               StackStatusFilter: []types.StackStatus{
-                       types.StackStatusCreateComplete,
... some more states
-                       types.StackStatusRollbackFailed,
-                       types.StackStatusUpdateComplete,
-                       types.StackStatusUpdateRollbackComplete,
-                       types.StackStatusUpdateRollbackFailed,
+               StackStatusFilter: []cloudformation.StackStatus{
+                       cloudformation.StackStatusCreateComplete,
+                       cloudformation.StackStatusUpdateRollbackFailed,
                },
        }

So with a few commands and some patches:

go mod init test1
vi main.go
go mod tidy
go run main.go

It worked!

Conclusion

Indeed something to watch in the future!
Try https://chat.openai.com/

Thanks

Photo by Thomas Park on Unsplash

And ChatGP.

Forem: Gernot Glawe

LAPD - Lambda Python Deployment for special "get out of jail" cases

GO-ing to production with Amazon (AWS) Bedrock RAG Part 1

The story

Time to think

The POC architecture

Solution architecture

0) Make the server available as a web app

1) Scale fast

2) Secure

3) cost effective

4) features page numbers, excerpts

5) serverless (this is not really a requirement, but a solution architecture hint)

6) easy to maintain

7) fast development cycles

Frontend

Backend

Conclusion

Disclaimer

What I have learned

Second part

See also

Stop LLM/GenAI hallucination fast: Serverless Kendra RAG with GO

The AWS RAG sample solution

RAG with langchain, OpenAI and Amazon Kendra

Measure speed and size

Python Speed

Python Size

Production

Shifting to Lambda/GO

Implementation in Python

Implementing in GO

GO Size

GO Speed

Moving to Lambda

Language

Timeout

AWS Managed Role

Conclusion

AWS Lambda recursive loop detection - but not (yet) for S3

The test

Summary

A new simple approach to diagram as code on AWS with CDK and D2

A simple approach to DaC

Example with a serverless architecture

Inside cdk2d2

Now to the synchronisation part

Terminal 1

Terminal 2

Terminal 3

First picture

Second picture

Life state change

Fully deployed state

Dependencies

Show

Connection

Container

Container example

Now try it out for yourself

See also

Check external links in PowerPoint automatically from the command line

Check S3 storage classes from command line

Switch & Leapp-cli - AWS session management 100% command line

Quick Start for the impatient

Before

After

Alternative approaches

Using profiles only with leap

Configure leaps for default profile

What do you get out of this approach?

Secure storage of credentials keys

Secure usage of temporal credentials

Easy installation and long term stability

Summary

Thanks

Building an AWS Lambda Telemetry API extension for direct logging to Grafana Loki

Using a centralized Logging solution in hybrid environments

Introducing the Lambda Telemetry API

Lambda Extensions