Forem: Alexandre Bruffa

Generative AI With Amazon Bedrock and Unity3D

Alexandre Bruffa — Fri, 06 Oct 2023 23:12:18 +0000

A few days ago, Amazon made a big announcement: the service Bedrock was finally released.

If you missed the news, here it is: Amazon decided to enter massively into the generative AI field and invested in the company Anthropic for $4 billion in order to compete with popular AI services like ChatGPT, Dall-E, MidJourney, etc. The first proposal of Amazon is Bedrock, a fully managed AI service that offers multiple foundational models such as Claude, Jurassic, and Stable Diffusion XL, among others.

In my previous articles, I created Unity3D applications communicating with ChatGPT and Dall-E. Let's do the same with Bedrock!

Do you prefer something more interactive instead of reading? Check my video!

General Architecture

Here is the general architecture of our application:

We use 3 Amazon services:

API Gateway to create, expose, and consume endpoints.
Lambda to receive the client's requests and communicate with Bedrock.
Bedrock to generate answers based on the client's prompts.

If you want to get further adding a login mechanism with Cognito, please refer to my previous article: Connecting Unity3D, AWS, and ChatGPT Together.

Bedrock

First, we have to enable the foundational models we want to use. By default, all the models are disabled. Go to the Bedrock console and enable the models you will use in the model access section. If you have an updated payment method, activating the models only takes a few minutes.

If you don't enable the models in the Bedrock console, you will receive this error in Lambda:

An error occurred (AccessDeniedException) when calling the InvokeModel operation: Your account is not authorized to invoke this API operation.

Lambda, Boto3 and Bedrock

If you follow my work, you know I'm a Python guy, so I will use the Python SDK for AWS, Boto3. Good news: Bedrock is available on Boto3; bad news: Lambda runs an old version of Boto3 in which Bedrock does not exist yet.

💡 Here is a workaround: creating a new Lambda layer with an up-to-date version of Boto3. This is how to do it:

1) In PyCharm, create a new project and install boto3 in a new virtual environment. Boto3 and all the dependencies will be installed.

2) Go to the venv folder of your project and copy the lib folder into a new folder called python. Then, zip it.

3) Go to the Layers section of the Lambda console and create a new layer. Upload the zip file we have created before and choose the same Python version as your local Python virtual environment for the runtime.

The Lambda Functions

We create 2 Lambda functions, one for text generation and the other one for image generation, both with the same version as previously for the runtime (In my case, Python 3.9), and we add the layer we have created before.

The text generation function

For the text generation, I use the Claude model and implement it as described in the Amazon documentation.

The image generation function

For the image generation, I use the Stable Diffusion XL model and implement it, as described in the Amazon documentation.

Note that the model IDs can be found in the Bedrock documentation.

The Lambda Policy

In order to allow Lambda to invoke a Bedrock model, we create a new managed policy and add it to the Lambda functions roles:

API Gateway

In the API Gateway console, we create a new REST API with 2 resources: image and text. Each resource has a post method integrating with the Lambda functions we have created before.

When deploying the API, you will be asked to create a new stage. After the creation of the stage, the final endpoint will be shown. You will have to use this URL in the Unity client to perform the request.

The Unity Client

The Unity client allows the user to send a message to generate a text or an image as an answer:

Handling the request

First, we need to create serializable classes to embed the sending and receiving data:

Sending a request in Unity is an asynchronous process. Therefore, we create a coroutine that sends the request to the server:

Dealing with the generated image

If we ask for an image generation, the result sent by Bedrock is a base64 image. To show it on the Unity UI, we must convert it into bytes and create a Texture2D. Then, we assign the texture to the RawImage component of the UI.

Result

Here is the result with text and image generations:

Costs

Let's check with the AWS Calculator and the Bedrock pricing sheet what our system cost would be for a very pessimistic scenario: you love the app, and you perform 2,000 requests a month. 1,000 for text generation with the Claude model and 1,000 for image generation with Stable Diffusion XL.

API Gateway: With 2,000 requests to our REST API, the monthly cost is 0,01 USD
Lambda: With 2,000 requests with an average time of 5 seconds and 512 MB of memory allocation, the monthly cost is 0.08 USD.
Bedrock: 1,000 text generations by month with a maximum of 300 output tokens and 20 input tokens per request would result, in the worst case, in $0.2204 + $9.804. 1,000 image generations would give us a bill of $18.

Total: The total bill for our system would be approximately 28 USD monthly. It's quite affordable!

Final Thoughts

Using Bedrock is very intuitive, and integrating it with other Amazon services is absolutely straightforward. The fact that Bedrock is an in-house product makes the integration easier: there is no need for API keys or external requests to make it work, resulting in greater performance and security.

Every code of this article has been tested using Unity 2021.3.3 and Visual Studio Community 2022 for Mac. The mobile device I used to run the Unity app is a Galaxy Tab A7 Lite with Android 11.

Thanks for reading until the end!

Resources

Here is the resource mentioned in the article:

The Unity package of the application
The text generation function in Python
The image generation function in Python

Connecting Unity3D with AWS services

Alexandre Bruffa — Fri, 30 Jun 2023 00:43:57 +0000

In my last articles, I realized direct integrations between Unity3D and AWS using the AWS SDK for .NET. I generally don't dive into details about how the connection between Unity3D and the AWS services is made in order to focus on the primary topic. Still, it deserves a whole article because it’s not that simple!

There are many ways to access AWS resources from Unity. In this article, I will summarize three of them: using an IAM user, using a Cognito guest user, and using a Cognito authenticated user.

General considerations: For this article, we will use the AWS SDK for .NET. Since Unity uses .NET Standard 2.1, you will need the .NET Standard 2.0 assemblies available here and unzip the file in your Unity project's Plugins folder. Since we will perform asynchronous calls, you will also need the AsyncInterfaces package.

If you prefer to watch a video instead of reading this article, here you have!

1. Using an IAM User

Suppose you want to allow a specific user (i.e., a third-party app) to upload files to one of your S3 buckets. However, you only want to give programmatic access without implementing a signup or login flow. Using an IAM user is a perfect fit in this case.

Let’s simulate the process:

In the S3 console, create a new S3 private bucket. I will call mine awsresourcesbucket.

In the IAM console, create an IAM user with programmatic access and create a new Access Key; you will be given an Access Key and a Secret Access Key. Once created, don't forget to copy the Secret Access Key. It is shown just one time!

Attach a new policy to the IAM user. The policy gives the required permissions to write into the S3 bucket.

In Unity, build an S3 Client thanks to the IAM user's Access Key and Secret Access Key, and upload the file.

Comments about the above code:

We use the NativeGallery package to pick an image from the device.
We use the PutObjectAsync function as described in the AWS documentation.
The UploadFileAsync function is asynchronous; you can call it with the await operator inside an async function.

2. Using a Cognito Guest User

You may remember my last article, where I built a real-time multiplayer game with Unity3D and Amazon GameLift. The Unity client calls a Lambda function to start playing, and we want that everybody who has downloaded the game can play without being logged in. You can achieve it using guest users of a Cognito Identity Pool.

Let’s simulate it:

In the Lambda console, create a simple Lambda function returning a dummy value (I will call it awsResourcesFunction). I will do it in Python, but it's up to you to choose your favorite programming language.

In the Cognito console, create a Cognito Identity Pool with guest access. During the Pool creation, you will be asked to create a new guest role; I will call them respectively awsResourcesIdentityPool and awsResourcesCognitoGuestRole.

Once the Identity Pool is created, enter the guest role and attach a new policy. The policy gives the required permissions to invoke the Lambda function we have created before.

In Unity, we first build the AWS Credentials thanks to the Identity Pool ID and then the Lambda Client with those credentials.

Comments about the above code:

The AWS documentation indicates that the InvokeAsync function has been deprecated, but remember that we are using the .NET Standard 2.0 assemblies, so it's valid for us!
The InvokeAsync function is asynchronous; you can call it with the await operator inside an async function.

3. Using a Cognito Authenticated User

Now imagine that you want to allow the players of your game to upload a profile picture, but you require a login to do it. This case is a bit more complex because your game will handle guest and authenticated users. You can achieve it by creating a User Pool and adding authenticated access to our Identity Pool.

Let's try:

In the Cognito console, create a new User Pool. You also be asked to create an Application Client during the User Pool creation. Don't forget to enable the ALLOW_USER_PASSWORD_AUTH as authentication flow.

For this example, we assume we already have a user registered and confirmed. Please take a look at this article of mine to learn how to do it.

Then, go back to the Identity Pool we created before, click on "Add identity provider" of the Authenticated access section, choose "Amazon Cognito User Pool", and link the Cognito User Pool we have created previously.

Go then to "Edit role", create a new role, and attach a new policy to the role (or you can reuse the policy we created for the IAM User). The policy gives the required permissions to write into the S3 bucket.

In Unity, we first login to the Cognito User Pool with the user Credentials; a Token ID is returned. Then we build the AWS Credentials thanks to the Identity Pool ID, and we add the login data to the AWS Credentials: the Token ID and the User Pool Provider Name. We can finally build the S3 Client with the AWS Credentials and upload a file.

Comments about the above code:

We log in to the Cognito User Pool thanks to the InitiateAuthAsync function.
We add the login data to the AWS Credentials thanks to the AddLogin function.
The Provider Name is the concatenation of the following: cognito-idp. + region + .amazonaws.com/ + Cognito User Pool ID.

Final Thoughts

Connecting Unity3D with AWS services is challenging work; I hope that this article could help you!

All ids and tokens shown in this article are fake or expired; if you try to use them, you will not be able to establish any connections.

Every code of this article has been tested using Unity 2021.3.3 and Visual Studio Community 2022 for Mac. The mobile device I used to run the Unity app is a Galaxy Tab A7 Lite with Android 11.

A special thanks to Gianca Chavest for designing the fantastic illustration.