Forem: Jared Donboch

Restricting Unauthenticated AWS Access by Referer and IP, similar to Google API Keys

Jared Donboch — Sat, 24 Jul 2021 03:08:22 +0000

For anyone who has used Google's JavaScript Map APIs, you know you use an API key to authenticate but you can lock down the API key so that only requests from certain referers or IPs are accepted.

In experimenting with creating a simple web-based map for the AWS Builders Community using the new AWS Location Service, it wasn't intuitive to me how to lock down the unauthenticated access provided by the Cognito Identity Pool as described in the AWS Location Service developer guide and I couldn't find any advice in the developer guides.

Andrew Johnson pointed me in the right direction and there are conditions that can be added to the Cognito Identity Pool unauthenticated role for both referer, IP addresses, and many others.

I would hope the Amazon Location Service Developer Guide is eventually updated with this advice as this should definitely be a best practice similar to what is recommended for Google Maps.

That being said, this does not prevent a sophisticated user from spoofing the referer and/or IP and any anonymous access should be minimized to the absolute least privilege. Here is the warning from the AWS conditions page:

[The aws:Referer] key should be used carefully. It is dangerous to include a publicly known referer header value. Unauthorized parties can use modified or custom browsers to provide any aws:referer value that they choose. As a result, aws:referer should not be used to prevent unauthorized parties from making direct AWS requests. It is offered only to allow customers to protect their digital content, such as content stored in Amazon S3, from being referenced on unauthorized third-party sites.

Check out the examples below.

Restricting by referer

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "MapsReadOnly",
            "Effect": "Allow",
            "Action": [
                "geo:GetMapStyleDescriptor",
                "geo:GetMapGlyphs",
                "geo:GetMapSprites",
                "geo:GetMapTile"
            ],
            "Resource": "arn:aws:geo:us-west-2:xxxxxxx:map/my-map",
            "Condition": {
                "StringLike": {
                    "aws:Referer": [
                        "http://www.example.com/*",
                        "http://example.com/*"
                    ]
                }
            }
        }
    ]
}

Restricting by IP

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "MapsReadOnly",
            "Effect": "Allow",
            "Action": [
                "geo:GetMapStyleDescriptor",
                "geo:GetMapGlyphs",
                "geo:GetMapSprites",
                "geo:GetMapTile"
            ],
            "Resource": "arn:aws:geo:us-west-2:xxxxxxx:map/my-map",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": "123.45.67.89"
                }
            }
        }
    ]
}

I've tried this out and it works great and have since added it to my public map project(s).

Let me know if you have any other best practices for least privilege permissions for public/unauthenticated users.

Want more? Follow me on Twitter and connect with me on Linked In!

AWS Location Service: Where my AWS Community Builders at?

Jared Donboch — Fri, 11 Jun 2021 20:46:06 +0000

TLDR; I built a map of AWS Community Builders using AWS Location Services. Click here for the map, click here for the code.

I was recently accepted into the AWS Community Builders program and have been blown away by how geographically diverse the group is. The program is split up into different technology-based cohorts--although this is very loose coupling and members are free to join discussions and sessions for any set of technology they are interested in--but there is no location based grouping.

This led to many asking the same question: are there any other AWS Community Builders near me?

The manager of the program, Jason Dunn, agreed to share some anonymized location data with me and I volunteered to draft a quick proof-of-concept map that would display the locations.

I initially implemented this via Google Maps JS API (code can be found here) but with the release of AWS Location Service, I decided it was worth trying to experiment with this new service.

What is Amazon Location Service?

Amazon Location Service is a new set of resources to support maps and location-based use cases such as geocoding, routing, asset/device tracking and geo-fence events.

As of the writing of this, the AWS Location Service has been GA for less than 2 weeks. I look forward to new features and integrations that will inevitably come in the future as the service continues to get some exercise.

Initial Impressions (of Maps and Geocoding)

Here are my initial impressions. I have only really experimented with maps and geocoding to support this simple use case.

The routing, tracking and geofencing components look very interesting but I don't have any direct experience with them yet.

Good: Great integration with other AWS Services

If you are already in the AWS eco-system, using Amazon Location Services will be very easy as you can continue to use the tools and SDKs you are already using. You can control access and security via IAM, automate deployments via CloudFormation, monitor with CloudWatch, be notified of location-based events via Amazon EventBridge, etc.

Good: Creates an abstraction for various location providers.

Amazon currently supports its clients using Esri or HERE data sets and services. When you create Amazon resources for maps, places, etc you choose which provider you want to supply the data. Each provider has their own strengths and weakness but you can now switch easily without needing to integrate with each unique location provider API. As new providers are added, you can continue to leverage the same interface.

Good: Better privacy and security

AWS does not have rights to sell your data or use it for advertising purposes (can Google say that?). They also anonymize all your requests before submitting them to vendors. Any tracking and geofence data is stored only in your AWS account (not 3rd parties) which gives you the power to control and secure that information.

Not so good: Free Tier/Pricing

AWS offers a free tier but it is only a 3-month trial. After 3 months, you are on the hook for your requests based usage. Google Maps free-tier equivalent (free mobile usage and $200 maps credit each month) does not expire and is much more generous. The maps and geocoding prices are competitive with other services but Amazon charges on a per-tile basis when many other providers charge on a per map-load basis. This means if your users are zooming and panning frequently, you will likely be paying more for Amazon Location Services.

This pricing makes Amazon Location Services a little less desirable for individuals, small prototyping efforts, and mobile map applications that are extremely price sensitive as you'll probably end up paying when other providers (i.e. Google) you can do a lot more with their free tier.

I hope that Amazon creates a more permanent free-tier for the mapping service at least.

#### Not so good: Lacking access controls for anonymous access to public maps.
Amazon Location Services provides an API that can be connected to other map libraries (i.e. MapLibre and Tangram). This is sort of nice but it requires you to use a Cognito Identity Pool with anonymous access enabled if you want to display on a public website. This is well documented in their developer guide but I feel like there are some access-controls lacking as there is no way to lock down your anonymous credentials to a particular IP or HTTP referrer like you can do with Google's API Keys. This means someone could grab the Cognito Identity Pool ID from the Javascript source code in the browser and use it for their own map requests. This seems like a big flaw and hoping some additional Conditional support could be added to IAM policies to better control anonymous access. Someone pointed me in the right direction here and this is possible: check out my article here on how to lock down AWS Location Service by referer or IP, similar to Google Maps API keys.

Creating the map and geocoding the data

To complete this proof of concept, I needed the following:

Given a list of locations, I needed to 1) geocode the locations to get their latitudes and longitudes and then 2) create a static map webpage to display the locations as markers on a map. Bonus points if we can add some metadata on the markers.

The map

There was not much innovation for the map part of the project and just a quick proof of concept was the goal. It was accomplished by walking through the following sections of the AWS Location Service Developer Guide which is a great resource to get started! :

Page 27: Allowing unauthenticated guest access to your application using Amazon Cognito
Page 33: Creating a Map Resource
Page 34: Authenticating your requests
Page 35: Using MapLibre GL JS with Amazon Location Service

I also used some MapLibre tutorials below to help create the markers, icons and pop ups on the map:

The geocoding process

We need to create a Place Index resource to support the geocoding.

To do the geocoding and create the GeoJSON that is displayed by the map, I wrote a quick Python script to perform that. This was fairly easy since the geocoding actions have been added to boto3 (i.e. the Python AWS SDK).

Unfortunately, the only location information that we had for Amazon Community Builders was Country. This resulted in many duplicate markers being generated which the mapping library was not happy about. In the end, I adjusted the logic of the geocoding script to create a single point for each country but then add properties to the point that indicated the number of people in that country. This member count property could then be displayed as a pop up on the map when the user hovered over the map marker.

Final Results

To see the final code and the geocoding script, check out the acbmap-aws repo on GitHub. I tried to add more details to that README with the specifics if you want to try and run this locally.

You can find the a hosted version of the AWS Community Builders map, powered by Amazon Location Services, at the following location:

https://acbmap.humbleg.com/aws/

Using Athena data connectors to visualize DynamoDB data with AWS QuickSight

Jared Donboch — Sun, 21 Mar 2021 13:20:53 +0000

Some context and history

Almost 2 years ago, I started experimenting with QuickSight to solve some of the BI issues of the company I was working for. I appreciated QuickSight's first-class integration with many AWS data services and low cost in comparison to other similar tools. It afforded us the ability to rapidly prototype analyses and dashboards. One glaring missing feature that left us scratching our heads was the lack of DynamoDB has a data source option.

I asked the StackOverflow hive-mind how to Visualize DynamoDB in AWS Quicksight on StackOverflow and it has become the most upvoted QuickSight question on the platform because there is a demand for this feature and there was no direct answer... until recently.

Most of the work-around solutions proposed involved exporting (i.e. duplicating) your DynamoDB data to a different repository such as S3 or RDS that could then be added as a data source in QuickSight. We ended up creating scheduled Glue jobs that would move DynamoDB data to S3. The S3 data was then crawled via AWS Glue Crawlers and exposed as AWS Athena tables which were then added as Quick Sight data sets. This worked but was more custom infrastructure than was desirable and also didn't allow for real time direct queries.

In late 2019, AWS announced you could query any datasource with Amazon Athena's new federated query feature.. This was cool and they even showed in diagrams the concept of querying DynamoDB but this required us to develop and maintain our own implementation of this connector and also this feature was only in Preview and AWS QuickSight integration was not updated to allow the usage of this new Athena feature.

In March 2020, I saw that AWS Athena announced a prebuilt Data Connector for DynamoDB. This was exciting as I was able to quickly setup a data connector in Athena that could actually view and query DynamoDB data without any custom code but this feature was still in Preview and there was no support in QuickSight yet.

Fast forward, it's a year later and Athena Data Connectors have been become GA with the Athena Engine Version 2 in a handful of regions and the integration features in QuickSight that allow you to select the required Athena work groups and data sources are present.

This should be enough to make things work until DynamoDB becomes a first-class data source in Amazon QuickSight.

The Detailed Process

Using the Athena Data Connectors as part of Athena Engine Version 2, I was able to finally visualize DynamoDB data in QuickSight without creating any bespoke resources or duplicating the data to another data source.

Prerequisites/Assumptions

You have data in DynamoDB tables that you want to visualize in QuickSight
You have access to a user/role in your account with IAM access
You have a bucket created that can be used to store data Athena query results and data connector spill data.
- If this going to be used in production with large queries, I'd recommend adding S3 lifecycle policies to this bucket to ensure this bucket doesn't grow uncontrollably.
You are using a region that supports both
- Athena Engine Version 2
- QuickSight

Switch to using Athena Engine Version 2

If you don't already have an Athena workgroup that uses Athena Engine Version 2, this needs to be created.

At the time of this writing, the default primary workgroup uses Athena Engine Version 1 but they also indicate that these workgroups will be automatically upgraded at some point in the future.

Ain't nobody got time for AWS to upgrade so we need to go ahead (create, if needed) and switch to a workgroup that uses Athena Engine Version 2.

In the Athena console, select the workgroups tab.
Ensure that a workgroup exists that is "Athena Engine version 2" and switch to it.
1. Most likely, you will not have one and you will need to create one by clicking the "Create Workgroup" button. You can pretty much use the defaults for everything such ensure you select "Manually choose an engine version now." and ensure "Athena engine version 2 (recommended)" is selected when creating the workgroup.

Create the DynamoDB Athena Data Connector

Once we an Athena Engine version 2 workgroup created, let's go ahead and create our DynamoDB Athena Data Connector.

Navigate to the Data Sources tab of the Athena console and choose "Connect data source" button.
In first step of the Data Sources wizard, select the "Query a data source" option, then select "Amazon DynamoDB", then click the next button.
The second and final step, AWS wants you to specify the connector lambda. This does not exist yet so we need to deploy it by clicking the "Configure new AWS Lambda function" button.
1. This button opens a new window to the Lambda console to deploy the prebuilt AthenaDynamoDBConnector application.
2. Under Application settings, at a minimum to need to provide values for SpillBucket and AthenaCatalogName parameters as they do not have any defaults.
3. Click Deploy
4. After the the Lambda is deployed, head back to your Athena console window.
Now that our AthenaDynamoDBConnector function is deployed, click the refresh icon next to the "Choose Lambda function" dropdown list and you should now see the newly deployment lambda function. Select your function, give catalog a name, and click the "Connect" button.
At this point, you should be able to use Athena to query your DynamoDB data using the data source and catalog name you gave your resources.

Granting QuickSight IAM role Lambda permissions

The Athena Data Connector works by invoking a Lambda to query and return DynamoDB data. Therefore we need to give QuickSight's service role permissions to invoke the Lambda function.

Navigate to the IAM Console and select "Roles"
Find and click the QuickSight service role. The role name should begin with something like "aws-quicksight-service-role", i.e. "aws-quicksight-service-role-v0".
Click "Attach Policies"
Select the "AWSLambdaRole" and click "Attach Policy"

Configuring QuickSight to use the new connector

Now, let's get this working with QuickSight!

Open QuickSight and navigate to the Data Sets menu.
Click "New Dataset" and select Athena as data source.
Select the workgroup you created previously that uses the Athena Engine version 2.
Now select the catalog that we created for the DynamoDB connector in the previous steps.
1. If you get a "You don’t have sufficient permissions to connect to this dataset or run this query." error, this is most likely because you did not properly add Lambda Invoke permissions to the QuickSight role. See the step above to ensure this is setup.
Now choose if you want to cache the data via SPICE or directly query the data. Directly querying will give you the ability to see real-time data in DynamoDB but the performance and cost may suffer since it is not cached.
1. You can add additional DynamoDB table data sets and continue to reuse the Athena Engine V2 data source you created in Step 2 above.

Create your visualizations!

Now that we have a DynamoDB data set (via Athena and the DynamoDB data connectors) created, we can finally visual DynamoDB data via analyses and dashboards.

I am just using a tiny sample DynamoDB dataset for this example so it's not the most interesting visualization but hopefully you get the idea!

Final Thoughts

It's been a long time coming but glad it's finally possible to get DynamoDB data into QuickSight without custom resources and duplicate data. This is just one example of the many data sources that can now be more easily be added via the Athena Engine Version 2 data connectors and the new ability to choose Athena workgroups and Athena catalogs in QuickSight.

I'm sure this article will eventually be obsolete sooner rather than later as technology changes and Amazon continues to release new features but it was a good exercise to explore some of these new features.

Please feel free to drop me a line if this was helpful or if I you have any suggestions on thing that could be improved or added.