Forem: Cobus Bernard 🌩🥑🇿🇦 in 🇺🇸

Making my aliased commands in zsh better

Cobus Bernard 🌩🥑🇿🇦 in 🇺🇸 — Fri, 07 Mar 2025 00:00:59 +0000

Just read the post about AWS Q Developer CLI update this morning. TL;DR: you can now run prompts from the CLI with a prompt to update files, execute system commands, and more. This opens up some very interesting ways to make my life easier so I can focus on my real priority: enjoying more coffee. Read till the end to see where this is going, and what I'll be building next. ;)

Before we get started with the fun, this is only one of the many capabilities. I was watching my friend Darko use it to create EC2 instances today from a prompt... Follow the instructions in the AWS Docs to install it.

Thinking of ways I want to use it, I know that keeping a README.md file updated is not always the highest priority, so I tried it out with the following prompt to see how past-Cobus did:

Can you update the readme based on the code in this repo. Please maintain the Security and License sections at the bottom of the file exactly as they are.

And off it goes!

I tested it out on the Terraform Samples I worked on last year, and it picked up that I had forgotten to document one of the sample modules I built. The README.md wasn't rewritten from scratch, Q Dev picked up on the format (using a table), and then added a new row for the missing module (you can see the extra row on line 47 in the code block below):

Write to filesystem
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Path: README.md

Replacing:

-  1 ## Terraform samples
-  2 
-  3 Collection of examples of how to use Terraform with AWS.
-  4 
-  5 List of modules:
-  6 
-  7 | Module | Description|
-  8 |-|-|
-  9 |[bootstrap-aws-account](./modules/bootstrap-aws-account/)|Helps bootstrap a new AWS account to use Terraform by creating the resources needed to store the state file.|
- 10 |[bootstrap-cicd-aws-codebuild](./modules/bootstrap-cicd-aws-codebuild/)|Sets up a CI/CD pipeline for Terraform in a GitHub repo, with workflows to add the plan as a comment on pull requests, and apply changes when a PR is merged using AWS CodeBuild.|
- 11 |[bootstrap-cicd-github-actions](./modules/bootstrap-cicd-github-actions/)|Sets up a CI/CD pipeline for Terraform in a GitHub repo, with workflows to add the plan as a comment on pull requests, and apply changes when a PR is merged using GitHub Actions.|
- 12 |[bootstrap-cloudtrail](./modules/bootstrap-cloudtrail/)|Sets up a basic AWS CloudTrail configuration for the account.|
- 13 |[aws-billing-budget-notification](./modules/aws-billing-budget-notification/)|Sets up an AWS Budget with an email alert.|
- 14 
- 15 ## Security
- 16 
- 17 See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.
- 18 
- 19 ## License
- 20 
- 21 This library is licensed under the MIT-0 License. See the LICENSE file.

With:

+  1 ## Terraform samples
+  2 
+  3 Collection of examples of how to use Terraform with AWS.
+  4 
+  5 List of modules:
+  6 
+  7 | Module | Description|
+  8 |-|-|
+  9 |[bootstrap-aws-account](./modules/bootstrap-aws-account/)|Helps bootstrap a new AWS account to use Terraform by creating the resources needed to store the state file.|
+ 10 |[bootstrap-cicd-aws-codebuild](./modules/bootstrap-cicd-aws-codebuild/)|Sets up a CI/CD pipeline for Terraform in a GitHub repo, with workflows to add the plan as a comment on pull requests, and apply changes when a PR is merged using AWS CodeBuild.|
+ 11 |[bootstrap-cicd-github-actions](./modules/bootstrap-cicd-github-actions/)|Sets up a CI/CD pipeline for Terraform in a GitHub repo, with workflows to add the plan as a comment on pull requests, and apply changes when a PR is merged using GitHub Actions.|
+ 12 |[bootstrap-cloudtrail](./modules/bootstrap-cloudtrail/)|Sets up a basic AWS CloudTrail configuration for the account.|
+ 13 |[aws-billing-budget-notification](./modules/aws-billing-budget-notification/)|Sets up an AWS Budget with an email alert.|
+ 14 |[temp-ec2-instance](./modules/temp-ec2-instance/)|Creates a temporary EC2 instance with Terraform pre-installed for bootstrapping purposes. Includes VPC, security group, and IAM role configuration.|
+ 15 
+ 16 Examples:
+ 17 
+ 18 | Example | Description|
+ 19 |-|-|
+ 20 |[bootstrap-new-aws-account](./examples/bootstrap-new-aws-account/)|Example showing how to bootstrap a new AWS account with the necessary Terraform infrastructure.|
+ 21 
+ 22 ## Security
+ 23 
+ 24 See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.
+ 25 
+ 26 ## License
+ 27 
+ 28 This library is licensed under the MIT-0 License. See the LICENSE file.

Enter y to run this tool, otherwise continue chatting.

This will come in handy quite often. Along with some other use-cases. It will however get very tiring to constantly type out this prompt...

Dotfiles

First a confession: I've know about dotfiles for a long time, but I just created my personal repo 5 minutes ago... I had a quick look at options on how to alias commands that I would need, and the following seems like an ok approach, but please let me know in the comments how I can make this better. Step one is to create a folder where you can store some bash scripts. I want to be able to use these to define the aliases that I can just call later by adding them to ~/.bashrc or ~/.zshrc. I'm going to group my aliases together in a script per topic or tool, e.g. I'll use q-helpers.sh for Q Dev, and likely git-helpers.sh for some git commands.

Inside q-helpers.sh I have the following:

alias q-readme='q chat "Can you update the readme based on the code in this repo. Please maintain the Security and License sections at the bottom of the file exactly as they are. If it makes sense, also include a Mermaid diagram in the Readme.'

alias q-add-license='q chat "Can you please add the SPDX line to all source code files with the appropriate string for the license based on which one has been defined?"

The code block makes it easy to copy/paste, but hard to read, so here are the 2 lines on their own - note how the command is wrapped with a single quote ' and the prompt with double quotes ".

alias add q-readme='q chat "Can you update the readme based on the code in this repo. Please maintain the Security and License sections at the bottom of the file exactly as they are. If it makes sense, also include a Mermaid diagram in the Readme."

alias q-add-license='q chat "Can you please add the SPDX line to all source code files with the appropriate string for the license based on which one has been defined?"

Then I need to make them executable by running chmod +x q-helpers.sh. The last step now is to add them to ~/.zshrc (I use zsh with oh-my-zsh):

# Source all files in dotfiles directory
for file in ~/dotfiles/*.sh; do
    [ -r "$file" ] && source "$file"
    echo "Dotfiles loaded."
done

Now all I need to do is run source ~/.zshrc, and I can call q-readme and q-add-license from anywhere. Thinking about it, I could probably combine a number of instructions in a single prompt to cover a lot of common scenarios...

And let's not forget about setting up some git hooks...

I'd love to hear what commands you add to your dotfiles, and if this will replace any existing ones, let me know in the comments below!

Adding parameters to my Serverless Web Scraper API

Cobus Bernard 🌩🥑🇿🇦 in 🇺🇸 — Thu, 04 Jul 2024 00:16:55 +0000

Introduction

In the previous articles, I built a serverless API that scrapes the data from the USCIS monthly visa bulletins, and stores them in an Amazon DynamoDB table. After jumping in and defining the whole infrastructure with Terraform, I realised that I should probably first ensure the code is working. Now it is time to add some parameters to our function so that it will return the data from categories other than the hard-coded EB-3 one, with some help from Amazon Q Developer :)

Updating the Scrape and Storage Logic

Before we get to that though, we need to update the logic around the lookups and storing the data. Currently, it will loop over all the URLs on the main page, extract the ones that contain /visa-bulletin-for-, and then do a lookup in the DynamoDB table ProcessedURLS to see if that URL was processed. Only the data from pages that are not in that database table will be scraped, added to a single collection, then that whole collection will be stored. We need to update the code so that it stores the data as it scrapes a URL, and before writing the processed entry - if there was an error, it currently will not reprocess that page.

Fixing Up Processing and Storing Our Data

To get started, let's see what we need to change. Ok, only a small change is needed to lambda_handler for the loop where we process and store the data, updated version is now:

    # Scrape data from each visa bulletin page
    for link in visa_bulletin_links:
        if '2022' in link or '2023' in link or '2024' in link:
            # Check if the URL has been processed
            response = processed_urls_table.get_item(Key={'url': link})
            if 'Item' in response:
                print(f"Skipping URL: {link} (already processed)")
                continue

            # Process the URL
            print(f"Processing URL: {link}")
            url = f"https://travel.state.gov{link}"
            url_data = scrape_visa_bulletin(url)
            data.extend(scrape_visa_bulletin(url_data))

            # Store the data
            store_data(url_data)

            # Store the processed URL in DynamoDB
            processed_urls_table.put_item(Item={'url': link})

Surprisingly little effort, so not much of a distraction.

Defining Some Enums

Looking at the existing parameters (with current defaults), I suspect an enum would work, but I don't know how this is done in Python:

def read_data_locally(data, filing_type = 'Final Date', category = '3rd', country = 'All Chargeability Areas Except Those Listed'):

The response looks to be what I need (minus the incorrect country, plus the missing ones), so let's add some enums!

from enum import Enum

class FilingType(Enum):
    FINAL_DATE = 'Final Date'
    DATES_FOR_FILING = 'Dates for Filing'

class Category(Enum):
    FIRST = '1st'
    SECOND = '2nd' 
    THIRD = '3rd'
    FOURTH = '4th'
    OTHER_WORKERS = 'Other Workers'

class Country(Enum):
    ALL_AREAS = 'All Chargeability Areas Except Those Listed'
    CHINA = 'CHINA-mainland born'
    INDIA = 'INDIA'
    MEXICO = "MEXICO"
    PHILIPPINES = 'PHILIPPINES'

Side-note: Interesting how Amazon Q autocompleted PHILIPPINES for me, I'm assuming it picked it up from the context of the linked URLs - I did a reboot of my laptop this morning and started a new chat, so don't think it would be previous chat context, but could be mistaken:

Running the app now doesn't return any data, so I suspect that I need to reference the enums differently, and yup, I do. Just append .value.

Adding Parameters

Now we need to change the code so we can pass these values in. Currently, I'm running the code locally via python3 local_test.py which calls the handler.py code via:

mock_context = MockContext()

mock_event = {
    "key1": "value1",
    "key2": "value2",
    # Add any other relevant data for your event
}

result = lambda_handler(mock_event, mock_context)

Hrm, this could be interesting, how would I pass the enum as a request? I could use integers, and do some kind of mapping between the number and the enum value, but honestly, I'm not sure how this works in Python. Time to find out though! The first part of how I add them to the event seems reasonable:

mock_event = {
    'queryStringParameters': {
        'filing_type': 'FINAL_DATE',
        'category': 'THIRD',
        'country': 'ALL_AREAS'
    }
}

I'm not quite sure that the way of extracting them looks right, adding in the 2nd parameter looks like hardcoding it to me:

    query_params = event.get('queryStringParameters', {})
    filing_type_str = query_params.get('filing_type', 'FINAL_DATE')
    country_str = query_params.get('country', 'ALL_AREAS')
    category_str = query_params.get('category', 'THIRD')

After clarifying how the get() method works, this does appear to be the correct way - that 2nd parameter is a default if it doesn't find the value in the input. Having it as a string bugs me - if I ever change the name of the enum, this will not work. After a bit of back and forth (1, 2, 3) I had the following:

    query_params = event.get('queryStringParameters', {})
    filing_type_str = query_params.get('filing_type', FilingType.FINAL_DATE.name)
    country_str = query_params.get('country', Country.ALL_AREAS)
    category_str = query_params.get('category', Category.THIRD.name)

    # Convert string values to enum members
    filing_type = FilingType[filing_type_str.upper()]
    country = Country[country_str.upper()]
    category = Category[category_str.upper()]

    ...
    # Updated call to read_data() with the parameters

    eb3_data = read_data(filing_type, category, country)

Additionally, I removed the hardcoded defaults on read_data() and read_data_locally(). Running the code locally returns the data as expected, I also checked with 'category': 'SECOND', just to make sure.

Splitting the Scrape and Retrieve Methods

We are now at a point where the code does what we want, except that every time we retrieve data, it would also loop through all the bulletins and try to process them. Ideally this should be split into 2 separate functions, with a way to call the code to scrape any new bulletins one a schedule. Before I try to find the answer, I'm thinking I could create a 2nd functions similar to lambda_handler(event, context), and then set up a 2nd Lambda function using the same Lambda layer for the requirements. It does mean I will include some libraries not needed by each function as it will contain all the ones across both functions, but I'm fine with that approach. Splitting this into 2 different projects, or even just splitting the requirements.txt file feels like even more over the top than we are already are.

The suggested approach is to split the project into 2 files, scraper.py and retriever.py, and then to have a lambda_handler function in each. I'm tempted to just create a 2nd function inside the 1 file, but let's go with the suggestion and split it into those 2 files. After I rename handler.py to scraper.py and create a copy named retriever.py, I realise we will need to split it further. At the top of the code, we set the tables via table = dynamodb.Table('VisaBulletinData') and processed_urls_table = dynamodb.Table('ProcessedURLs'), and we also have the enums defined as classes, and both of the functions need them. I look how I would do that, and merrily follow the suggestion.

After I remove the code not needed in each of the new functions, each in their own file, extract the enums into enums.py, and add the import statement, I stare at the import line in scraper.py for a few seconds:

Why are the 3 classes I import the darker colour of an unused import? Aaaaah! While I use the enums to retrieve the data, I never use them to store the data. I suddenly realise that I should probably change the code to use them as a mapping from the text in the bulletins. I noticed in some of the older ones that the 5th category was split less granularly, so we would need to address that as well.

Taking a Step Back

I've been building this app over the last 2 - 3 weeks spending an hour here and there between other work to build it out. Initially I intended to sit down and build it out in a day or 2, and expected it to not take any longer. I started out with the intention that I wanted to use a Lambda function, store the data in DynamoDB, and then be able to query it. Putting on my Captain Obvious cape, I realise that this is a lot of additional complexity for a problem I could have solved with a spreadsheet where I paste in a new row or two once a month. As the saying goes:

If all you have is a hammer, then all your problems look like a nail.

Ok, new plan. I'm going to continue down this path for only a short while longer till I have the Lambda functions deployed, and I can call them.

Updating the Infrastructure

Since we now have 2 separate functions, we also need to deploy them. This requires changing the existing Lambda since it referenced handler.py. Pasting all the Terraform resources again would take up quite a bit of space, but you can look at the current version at this point in time. We can keep the following resources as-is:

resource "null_resource" "pip_install" - we aren't splitting requirements.txt per source file, so a single one stays the same.
data "archive_file" "layer" - used to trigger updating the Lambda layer we create for our dependencies.
resource "aws_lambda_layer_version" "layer" - creates the Lambda layer.
data "archive_file" "app" combined with the line source_code_hash = data.archive_file.app.output_base64sha256 to ensure we update the functions for any code changes. This will trigger for both functions even if we only update one of them, but I don't feel it is worth the effort for this project.

I do want to ensure we provide the least amount of IAM permissions per function, so will duplicate the existing IAM role, policy, and attachments, and then reduce the scope of each IAM policy for each function with only the access needed. The last step will be to define a 2nd Lambda function after updating the existing one to use the renamed source file. Sorry, I lied, the last-last step will also be to update the DynamoDB table names to use the environment variables defined in the Lambda function, with a fallback if they are not set.

Adding the Environment Variables for DynamoDB Table Names

This change requires updating the Terraform resource for our Lambda function, below is the updated scraper.py function:

# Define the Lambda function
resource "aws_lambda_function" "visa_bulletin_scraper" {
  function_name    = "visa-bulletin-scraper"
  handler          = "scraper.lambda_handler" # <--- I've updated the handler's filename as well
  runtime          = "python3.12"
  filename         = data.archive_file.app.output_path
  source_code_hash = data.archive_file.app.output_base64sha256
  role             = aws_iam_role.lambda_role.arn
  layers           = [aws_lambda_layer_version.layer.arn]

  environment {
    variables = {
      BULLETIN_DATA = aws_dynamodb_table.visa_bulletin_data.name,
      PROCESSED_BULLETIN_URLS = aws_dynamodb_table.processed_urls.name
    }
  }
}

We also need to change the currently hard-coded table names at the top of scraper.py to the following to use the environment variables, with a fallback value:

import os # <--- new import added

...

# Initialize DynamoDB client
dynamodb = boto3.resource('dynamodb')

table_name = os.environ.get('BULLETIN_DATA', 'VisaBulletinData')
table = dynamodb.Table(table_name)

processed_urls_table_name = os.environ.get('PROCESSED_BULLETIN_URLS', 'ProcessedURLs')
processed_urls_table = dynamodb.Table(processed_urls_table_name)

For retriever.py, we only need to add the VisaBulletinData via BULLETIN_DATA.

Create a 2nd Lambda Function

As mentioned above, we will reuse the same Lambda layer, and only create a 2nd function for retriever.py, along with its own IAM policy and role. While doing this, I notice that past-Cobus was lazy with the resource names:

resource "aws_iam_role" "lambda_role" {
  name = "visa-bulletin-scraper-role"

  ...

}

# Attach the necessary IAM policies to the role
resource "aws_iam_policy_attachment" "lambda_basic_execution" {
    name = "lambda_basic_execution"
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
  roles      = [aws_iam_role.lambda_role.name]
}

resource "aws_iam_policy_attachment" "dynamodb_access" {
    name = "dynamodb_access"
  policy_arn = aws_iam_policy.dynamodb_access_policy.arn
  roles      = [aws_iam_role.lambda_role.name]
}

# Define the IAM policy for DynamoDB access
resource "aws_iam_policy" "dynamodb_access_policy" {
...
}

We now need to duplicate the Lambda function resource, along with reduced permissions for the IAM role needed by retriever.py. It is also a good time to split out our Terraform resources into smaller files to make it easier to find each one of them. I decided to split them from the single app.tf into the following:

dynamodb.tf - defines the 2 tables we use.
lambda_layer.tf - builds and creates the Lambda layer used by both functions.
lambda_zip.tf - zips up all the application code into a single .zip used by both functions.
lambda_scraper.tf - defines the Lambda function, IAM policy, and IAM role for scraper.py.
lambda_retriever.tf - defines the Lambda function, IAM policy, and IAM role for retriever.py.

After making these changes, and running terraform apply, the following error is returned:

│ Error: creating IAM Role (visa-bulletin-scraper-role): operation error IAM: CreateRole, https response error StatusCode: 409, RequestID: 94248aee-5615-4fbf-936e-81ceb9c24f0f, EntityAlreadyExists: Role with name visa-bulletin-scraper-role already exists.
│ 
│   with aws_iam_role.scraper_role,
│   on lambda_scraper.tf line 20, in resource "aws_iam_role" "scraper_role":
│   20: resource "aws_iam_role" "scraper_role" {
│ 
╵
╷
│ Error: creating IAM Policy (visa-bulletin-scraper-dynamodb-access): operation error IAM: CreatePolicy, https response error StatusCode: 409, RequestID: da27e143-5a7e-4008-81a8-49bf23fb99d8, EntityAlreadyExists: A policy called visa-bulletin-scraper-dynamodb-access already exists. Duplicate names are not allowed.
│ 
│   with aws_iam_policy.scraper_dynamodb_access_policy,
│   on lambda_scraper.tf line 53, in resource "aws_iam_policy" "scraper_dynamodb_access_policy":
│   53: resource "aws_iam_policy" "scraper_dynamodb_access_policy" {
│ 
╵

Since I renamed the aws_iam_role resource from lambda_role to scraper_role, we ran into a race condition where the delete for the old one didn't complete before the new one's create started. IAM role names need to be unique, and this is why we encountered this issue. Running terraform apply a second time will fix this - worth keeping in mind if you ever run into this after doing a cleanup.

Apply complete! Resources: 4 added, 1 changed, 0 destroyed.

Pro-tip: terraform fmt will format your .tf files in the current directory and fix the indentation, be kind and ~rewind~ clean up your source files before you commit them (unlike past-Cobus who also forgot that step).

But Does It Work

If you deploy a Lambda function, but you never call it, does it even exist?

In theory, we now have our 2 Lambda functions that would scrape or return the data. We haven't called them yet though, so let's see how we can call them from our terminal:

aws lambda invoke \
  --function-name visa-bulletin-retriever \
  --cli-binary-format raw-in-base64-out \
  --payload '{"queryStringParameters": {"filing_type": "FINAL_DATE", "category": "THIRD", "country": "ALL_AREAS"}}' \
  response.json

Which returns the lovely error:

{
    "StatusCode": 200,
    "FunctionError": "Unhandled",
    "ExecutedVersion": "$LATEST"
}

No It Does Not

Using the AWS Console to test the retriever function with the same payload as above, I see the issue:

When splitting out the enums into their own source file, I forgot to also add the import statement. After fixing that, I run terraform apply again, and I'm quite curious to see what the change looks like since we're using Lambda layers. The following change to the IAM role makes me suspicious:

  # aws_iam_policy_attachment.retriever_lambda_basic_execution will be updated in-place
  ~ resource "aws_iam_policy_attachment" "retriever_lambda_basic_execution" {
        id         = "retriever_lambda_basic_execution"
        name       = "retriever_lambda_basic_execution"
      ~ roles      = [
          - "visa-bulletin-scraper-role",
            # (1 unchanged element hidden)
        ]
        # (3 unchanged attributes hidden)
    }

  # aws_iam_policy_attachment.scraper_lambda_basic_execution will be updated in-place
  ~ resource "aws_iam_policy_attachment" "scraper_lambda_basic_execution" {
        id         = "scraper_lambda_basic_execution"
        name       = "scraper_lambda_basic_execution"
      ~ roles      = [
          - "visa-bulletin-retriever-role",
            # (1 unchanged element hidden)
        ]
        # (3 unchanged attributes hidden)
    }

This rings a bell from years ago, but I can't quite remember, so off I go and do a search for aws_iam_policy_attachment so I can look at the documentation. Right at the top of the page, it has a big, red warning:

Aha! 🎵 It's all coming back to me nooooow! 🎵 This was similar to setting the security group rules inside the aws_security_group - in the case of aws_iam_policy_attachment, it will play whack-a-mole since we need to attach that policy to 2 different IAM roles. On the first run, one of them will end up succeeding, but for future runs, it will try to attach it again since whichever one finished last would remove the one that finished first. To fix this, we need to use aws_iam_role_policy_attachment instead:

resource "aws_iam_role_policy_attachment" "retriever_lambda_basic_execution" {
  role       = aws_iam_role.retriever_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

Updates done, and ran terraform apply again, only for it to timeout after 2 minutes with:

╷
│ Error: reading IAM Role Policy Attachment (visa-bulletin-retriever-role:arn:aws:iam::631077426493:policy/visa-bulletin-retriever-dynamodb-access): empty result
│ 
│   with aws_iam_role_policy_attachment.retriever_dynamodb_access,
│   on lambda_retriever.tf line 45, in resource "aws_iam_role_policy_attachment" "retriever_dynamodb_access":
│   45: resource "aws_iam_role_policy_attachment" "retriever_dynamodb_access" {
│ 
╵
╷
│ Error: reading IAM Role Policy Attachment (visa-bulletin-scraper-role:arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole): empty result
│ 
│   with aws_iam_role_policy_attachment.scraper_lambda_basic_execution,
│   on lambda_scraper.tf line 40, in resource "aws_iam_role_policy_attachment" "scraper_lambda_basic_execution":
│   40: resource "aws_iam_role_policy_attachment" "scraper_lambda_basic_execution" {
│ 
╵
╷
│ Error: reading IAM Role Policy Attachment (visa-bulletin-scraper-role:arn:aws:iam::631077426493:policy/visa-bulletin-scraper-dynamodb-access): empty result
│ 
│   with aws_iam_role_policy_attachment.scraper_dynamodb_access,
│   on lambda_scraper.tf line 45, in resource "aws_iam_role_policy_attachment" "scraper_dynamodb_access":
│   45: resource "aws_iam_role_policy_attachment" "scraper_dynamodb_access" {
│

The hint here is the empty result - when you try to attach a policy the 2nd time, it will not return an error if it is already attached. So we have another race condition (I think): the request to remove the attachment (since we replaced the aws_iam_policy_attachment with aws_iam_role_policy_attachment) was was made in parallel to the new one in an order where it didn't return a successful response for the new one. At least this is what I would speculate without digging into it too much. Regardless, you can get around this issue by just running terraform apply a 2nd time.

Let's Try Again

Second time is the charm, right? Running our aws invoke-lambda command again returns:

{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
(END)

And then it will wait for you to exit the output view - this is the standard behaviour, and you can exit it via ctrl + c or cmd + c. Alternatively, you can add the --no-cli-pager parameter:

aws lambda invoke \
  --function-name visa-bulletin-retriever \
  --cli-binary-format raw-in-base64-out \
  --payload '{"queryStringParameters": {"filing_type": "FINAL_DATE", "category": "THIRD", "country": "ALL_AREAS"}}' \
  --no-cli-pager \
  response.json

Looking at the contents in response.json, we can see the data!

{
    "statusCode": 200,
    "body": [
        {
            "date": "2021-12-01",
            "filing_type": "Final Date",
            "bulletin_date": "2024-07-01",
            "category": "3rd",
            "sk": "BULLETIN_DATE#2024-07-01",
            "pk": "FILING_TYPE#Final Date#CATEGORY#3rd#COUNTRY#All Chargeability Areas Except Those Listed",
            "country": "All Chargeability Areas Except Those Listed"
        },

        ...

While the data is all there, it isn't in a very readable format - we are only really interested in the bulletin_date along with the date as a sorted list since all the rest of the data is what we sent in to filter on.

Making the Response Better

A better response would be one where it shows the filing_type, category, and country once, and then have a list of key-value pairs for bulletin_date and date. It looks like we can do this by changing the return statement to the following:

    response = {
        'filing_type': filing_type,
        'category': category,
        'country': country,
        'data': data
    }

    return {
        'statusCode': 200,
        'body': response
    }

This does assume that the response from read_data is in the format that we need, which is that key-value pair list. We need to also update the return of read_data to accomplish this to the following:

...

    date_list = []    

    for item in sorted_items:
        date = item['date']
        bulletin_date = item['bulletin_date']
        date_list.append({"bulletin_date":bulletin_date, "date": date})

    return date_list

And another terraform apply later, it is deployed. You can also see from the Terraform output below that it only replaced the Lambda function code, not the Lambda layer as we didn't add any additional dependencies - json is built into Python:

Terraform will perform the following actions:

  # aws_lambda_function.visa_bulletin_retriever will be updated in-place
  ~ resource "aws_lambda_function" "visa_bulletin_retriever" {
        id                             = "visa-bulletin-retriever"
      ~ last_modified                  = "2024-07-03T22:28:49.000+0000" -> (known after apply)
      ~ source_code_hash               = "BmwRVIejGX4B1ual5JdhfulogEDuyTxyW9/3/g95WBE=" -> "eTHf0nHRxTy/Pi4u+tOvpKATsPWqagYcD3rEl0wwwcQ="
        tags                           = {}
        # (27 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

  # aws_lambda_function.visa_bulletin_scraper will be updated in-place
  ~ resource "aws_lambda_function" "visa_bulletin_scraper" {
        id                             = "visa-bulletin-scraper"
      ~ last_modified                  = "2024-07-03T22:29:02.000+0000" -> (known after apply)
      ~ source_code_hash               = "BmwRVIejGX4B1ual5JdhfulogEDuyTxyW9/3/g95WBE=" -> "eTHf0nHRxTy/Pi4u+tOvpKATsPWqagYcD3rEl0wwwcQ="
        tags                           = {}
        # (27 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

The body of the response will be JSON-encoded, so if we used this function as an API, we would need to deserialize the body into an object to access the data. The contents of response.json looks as follows:

Wrapping Up

As with most things, this took quite a bit longer than expected. The intention was just to add some parameters to the Lambda function for us to call, but we had to take a few detours along the way. As mentioned further up, I think I'm over-complicating this just a little bit, but I'm still enjoying myself, so I'll do one more article tomorrow to set up a simple CI/CD pipeline that will deploy any future changes to this little app. Oh, and I need to set up a daily schedule to call the scraper function to auto-update if a new file is found. Actually, what would be really nice to also include a notification if it finds a new bulletin - today is 3 July 2024, and I'm already refreshing the page to see when the draft of the August bulletin will be published...

(PS: By "tomorrow", I mean Friday 5 July as tomorrow is a public holiday here in the US.)

Final Code

As for the previous piece, I've tagged this with article-3, and pushed the tag.

Source code for this article.

Prompts used in this article

Prompt 1

Can you please show me how to update the highlighted code so that it will store the data after processing the URL, but before it saves the URL via processed_urls_table.put_item?

Response:

Sure, here's how you can modify the highlighted code to store the data after processing the URL, but before saving the URL via processed_urls_table.put_item:

for link in visa_bulletin_links:
    if '2022' in link or '2023' in link or '2024' in link:
        # Check if the URL has been processed
        response = processed_urls_table.get_item(Key={'url': link})
        if 'Item' in response:
            print(f"Skipping URL: {link} (already processed)")
            continue

        # Process the URL
        print(f"Processing URL: {link}")
        url = f"https://travel.state.gov{link}"
        url_data = scrape_visa_bulletin(url)
        data.extend(url_data)

        # Store the data
        store_data(url_data)

        # Store the processed URL in DynamoDB
        processed_urls_table.put_item(Item={'url': link})

store_data(data)

Here's what's changed:

After scraping the visa bulletin data from the URL using scrape_visa_bulletin(url), the scraped data is stored in a new variable url_data.
The url_data is then appended to the data list using data.extend(url_data). 3.A new line store_data(url_data) is added to store the data scraped from the current URL.

After storing the data, the URL is marked as processed in DynamoDB using processed_urls_table.put_item(Item={'url': link}).

This way, the data is stored immediately after being scraped from the URL, and before the URL is marked as processed in DynamoDB.

Note: This assumes that the store_data function is defined elsewhere in your code and correctly handles storing the data in the desired location (e.g., a database, a file, etc.).

Prompt 2

Looking at the highlighted line, does Python have an equivalent of C#'s enum? What would the approach be in Python to use something similar? I would like to define an easier way to provide the filing_type, category, and country?

Response:

Yes, Python has an equivalent to C#'s enum called the enum module, which was introduced in Python 3.4. It allows you to define a set of symbolic names (members) bound to unique, constant values. This can make your code more readable and less prone to errors caused by typos or invalid values. [1]

Here's an example of how you could use the enum module to define the possible values for filing_type, category, and country in your read_data_locally function:

from enum import Enum

class FilingType(Enum):
    FINAL_DATE = 'Final Date'
    DATES_FOR_FILING = 'Dates for Filing'

class Category(Enum):
    FIRST = '1st'
    SECOND = '2nd' 
    THIRD = '3rd'
    OTHER_WORKERS = 'Other Workers'

class Country(Enum):
    ALL_AREAS = 'All Chargeability Areas Except Those Listed'
    CHINA_MAINLAND = 'China-mainland born'
    EL_SALVADOR = 'El Salvador, Guatemala, Honduras'
    # Add more countries as needed

def read_data_locally(data, filing_type=FilingType.FINAL_DATE, category=Category.THIRD, country=Country.ALL_AREAS):
    # Your function logic here
    pass

In this example, we define three enum classes: FilingType, Category, and Country. Each class has members representing the valid values for that category.

When calling read_data_locally, you can now use the enum members instead of strings, like filing_type=FilingType.FINAL_DATE. This has several advantages:

Type safety : The function will only accept values from the defined enums, preventing typos or invalid values.
Readability : The code becomes more self-documenting and easier to understand.
Consistency : You ensure that the same values are used throughout your codebase.

Note that you can also assign custom values to enum members if needed, and you can add methods and properties to the enum classes.

Sources:

Is there any way to define the enum in python?

Prompt 3

Using the enum values in this way doesn't return any data, could you please explain how I would access the string value for each enum in filtered_data code?

Response:

To access the string value of an enum member in the filtered_data code, you can use the value attribute of the enum member or call the str() function on the member.

Here's an example of how you can modify the read_data_locally function to filter the data based on the string values of the enum members:

def read_data_locally(data, filing_type=FilingType.FINAL_DATE, category=Category.THIRD, country=Country.ALL_AREAS):
    # Filter the data based on filing_type, category, and country
    filtered_data = [entry for entry in data
                     if entry['filing_type'] == filing_type.value
                     and entry['category'] == str(category)
                     and entry['country'] == country.value]

    # Sort the filtered data in descending order by bulletin_date
    sorted_data = sorted(filtered_data, key=itemgetter('bulletin_date'), reverse=True)

    # Print the sorted data
    for entry in sorted_data:
        print(f"Bulletin Date: {entry['bulletin_date']}, Date: {entry['date']}")

    return sorted_data

In this modified version:

For filing_type, we use filing_type.value to get the string value of the enum member.
For category, we use str(category) to get the string representation of the enum member.
For country, we use country.value to get the string value of the enum member.

This way, the filtered_data list will contain only the entries where the filing_type, category, and country values match the string values of the corresponding enum members.

Note that when you call the read_data_locally function, you can still use the enum members directly, like read_data_locally(data, filing_type=FilingType.FINAL_DATE, category=Category.THIRD, country=Country.ALL_AREAS). By using the value attribute or str() function, you can access the string representation of the enum members, which is necessary for filtering the data based on the string values stored in the dictionary entries.

Prompt 4

Thanks, how would I add the names of the enums to the event when calling lambda_handler - I want to be able to inject it into the the mock_event object for local testing, but also be able to call the Lambda using query string parameters? It should then insert the values from the event when calling read_data().

Response:

To add the names of the enums to the event when calling lambda_handler, you can modify the function to accept the enum values as query string parameters or as part of the event payload. Then, you can use these values when calling read_data().

Here's an example of how you can modify the lambda_handler function to accept the enum values as query string parameters:

import urllib.parse

def lambda_handler(event, context):
    # ... (existing code)

    # Parse query string parameters
    query_params = event.get('queryStringParameters', {})
    filing_type_str = query_params.get('filing_type', 'FINAL_DATE')
    country_str = query_params.get('country', 'ALL_AREAS')

    # Convert string values to enum members
    filing_type = FilingType[filing_type_str.upper()]
    category = Category[category_str.upper()]
    country = Country[country_str.upper().replace(' ', '_')]

    # Call read_data with the enum values
    eb3_data = read_data(filing_type=filing_type, category=category, country=country)

    return {
        'statusCode': 200,
        'body': eb3_data
    }

In this modified version:

The query_params dictionary is created from the queryStringParameters in the event object.

The filing_type_str, category_str, and country_str values are retrieved from the query_params dictionary, with default values provided if the parameters are not present.

The string values are converted to their corresponding enum members using the FilingType, Category, and Country enums. The upper() method is used to handle case-insensitive values, and the replace(' ', '_') is used to handle spaces in the country string.

The read_data function is called with the enum values.

To test this locally, you can create a mock_event object with the desired query string parameters:

mock_event = {
    'queryStringParameters': {
        'filing_type': 'FINAL_DATE',
        'category': 'THIRD',
        'country': 'ALL_AREAS'
    }
}

result = lambda_handler(mock_event, None)
print(result)

When invoking the Lambda function, you can pass the query string parameters in the URL, like https://your-lambda-function.aws/path?filing_type=FINAL_DATE&category=THIRD&country=ALL_AREAS.

If you prefer to pass the enum values as part of the event payload instead of query string parameters, you can modify the lambda_handler function accordingly:

def lambda_handler(event, context):
    # ... (existing code)

    # Parse query string parameters
    query_params = event.get('queryStringParameters', {})
    filing_type_str = query_params.get('filing_type', 'FINAL_DATE')
    country_str = query_params.get('country', 'ALL_AREAS')

    # Convert string values to enum members
    filing_type = FilingType[filing_type_str.upper()]
    category = Category[category_str.upper()]
    country = Country[country_str.upper().replace(' ', '_')]

    # Call read_data with the enum values
    eb3_data = read_data(filing_type=filing_type, category=category, country=country)

    return {
        'statusCode': 200,
        'body': eb3_data
    }

In this case, you would pass the enum values as part of the event payload when invoking the Lambda function.

Prompt 5

Can you please explain how the line category_str = query_params.get('category', 'THIRD') works? Is the 2nd parameter "THIRD" a default value if it can't extract it?

Response:

Yes, you're correct. The line category_str = query_params.get('category', 'THIRD') is using the get() method of a dictionary (or dictionary-like object) to retrieve the value associated with the key 'category'.

The get() method takes two arguments:

The key to look up in the dictionary
An optional default value to return if the key is not found

In this case, query_params is likely a dictionary containing query parameters extracted from a URL or form data. The get('category', 'THIRD') part is attempting to retrieve the value associated with the 'category' key from query_params.

If the 'category' key exists in query_params, its associated value will be returned and assigned to category_str. However, if the 'category' key is not present in query_params, the default value 'THIRD' will be used instead and assigned to category_str.

So in summary, yes - the second argument 'THIRD' is acting as a default value that will be used for category_str if the 'category' key cannot be found in query_params.

Prompt 6

If I wanted to replace the string FINAL_DATE with the name of the enum, how would I do that? Would something like FilingType.FINAL_DATE.str() work?

Response:

In Python, you can access the value of an enum member directly using its name. There is no need for a separate method like str(). You can simply use FilingType.FINAL_DATE.value to get the string value associated with the FINAL_DATE member of the FilingType enum.

Here's an example of how you can replace the string 'FINAL_DATE' with the value of the FilingType.FINAL_DATE enum member:

filing_type_str = query_params.get('filing_type', FilingType.FINAL_DATE.value)

In this line, if the 'filing_type' key is not present in query_params, the value of FilingType.FINAL_DATE.value (which is 'Final Date') will be used as the default value for filing_type_str.

Similarly, you can use the same approach for the country_str and category_str assignments:

filing_type_str = query_params.get('filing_type', FilingType.FINAL_DATE.value)
country_str = query_params.get('country', Country.ALL_AREAS.value)
category_str = query_params.get('category', Category.THIRD.value)

By using the value attribute of the enum members, you can directly access the string values associated with each enum member, without needing any additional methods like str().

Prompt 7

Thanks, but wouldn't that set the value of the enum instead of the name since further down we look up the value of the enum via filing_type = FilingType[filing_type_str.upper()]?

Response:

You're absolutely right, my previous suggestion would not work correctly in this case. Since you are using the string value to look up the corresponding enum member later on with FilingType[filing_type_str.upper()], you need to use the string name of the enum member as the default value, not the value attribute. [1]

To replace the string 'FINAL_DATE' with the name of the FilingType.FINAL_DATE enum member, you can use the following:

filing_type_str = query_params.get('filing_type', 'FINAL_DATE')

This way, if the 'filing_type' key is not present in query_params, the string 'FINAL_DATE' will be assigned to filing_type_str. Then, when you do filing_type = FilingType[filing_type_str.upper()], it will correctly look up the FilingType.FINAL_DATE member.

Similarly, for the other enum types, you can use the string names of the desired default enum members:

filing_type_str = query_params.get('filing_type', 'FINAL_DATE')
country_str = query_params.get('country', 'ALL_AREAS')

This way, the default values will be the string names of the enum members, which can then be used to look up the corresponding enum members correctly.

Sources:

Entity Framework(EF) Enum issue/bug/feature - DEV Community

Prompt 8

This isn't quite what I'm looking for. I want a way to avoid using a hard-coded string for the enum key FINAL_DATE, and instead somehow reference FilingType.FINAL_DATE so that if I decide to rename that specific enum value to e.g. FINAL_DATE_NEW, I would get an error when trying to the code as FilingDate.FINAL_DATE would no longer exist.

Response:

I understand now. You want to avoid hard-coding the string name of the enum member and instead reference the enum member itself, so that if the enum member name changes, you'll get an error at compile/run time.

To achieve this, you can use the name attribute of the enum member. The name attribute returns the string name of the enum member.

Here's how you can modify the code:

from enum import Enum

class FilingType(Enum):
    FINAL_DATE = 'Final Date'
    # other enum members

default_filing_type = FilingType.FINAL_DATE

filing_type_str = query_params.get('filing_type', default_filing_type.name)
filing_type = FilingType[filing_type_str.upper()]

In this example:

We define a default_filing_type variable and assign it the FilingType.FINAL_DATE enum member.
When getting the filing_type_str from query_params, we use default_filing_type.name as the default value.

This way, if you rename the FilingType.FINAL_DATE enum member to something else (e.g., FilingType.FINAL_DATE_NEW), you'll get an error when trying to access FilingType.FINAL_DATE, because it no longer exists.

The name attribute of the enum member is a string, so it can be used as the default value for filing_type_str. Later, when you look up the enum member using FilingType[filing_type_str.upper()], it will work as expected, because you're using the string name of the enum member.

This approach ensures that if you rename or remove an enum member, you'll get an error at compile/run time, rather than silently using an incorrect hard-coded string value.

Prompt 9

The current lambda_handler function has 2 purposes, the first is to scrape the dates from the USCIS website and store them in the database, and the 2nd is to return the data based on the parameters specified. What is the best approach given the current structure (Python for the code, and Terraform for the infrastructure) to split this into 2 different functions?

Response:

To split the current lambda_handler function into two separate functions, you can follow these steps: [1]

Create a new Lambda function for scraping and storing data

In your Terraform configuration, create a new AWS Lambda Function resource. This function will be responsible for scraping the USCIS website and storing the data in the database. [2]

resource "aws_lambda_function" "scraper_function" {
filename         = data.archive_file.scraper_lambda_zip.output_path
function_name    = "uscis-scraper"
role             = aws_iam_role.lambda_exec_role.arn
handler          = "scraper.lambda_handler"
runtime          = "python3.9"
}

Create a new Lambda function for retrieving data

Create another AWS Lambda Function resource in your Terraform configuration. This function will be responsible for retrieving data from the database based on the provided parameters.

resource "aws_lambda_function" "retriever_function" {
filename         = data.archive_file.retriever_lambda_zip.output_path
function_name    = "uscis-retriever"
role             = aws_iam_role.lambda_exec_role.arn
handler          = "retriever.lambda_handler"
runtime          = "python3.9"
}

Separate the Python code

In your Python code, create two separate files: scraper.py and retriever.py.
scraper.py will contain the code for scraping the USCIS website and storing the data in the database. The lambda_handler function in this file will be responsible for this task.
retriever.py will contain the code for retrieving data from the database based on the provided parameters. The lambda_handler function in this file will be responsible for this task.

Update the Lambda function handlers

In your Terraform configuration, update the handler argument for each Lambda function to point to the correct file and function name. [3]

resource "aws_lambda_function" "scraper_function" {
# ...
handler          = "scraper.lambda_handler"
}

resource "aws_lambda_function" "retriever_function" {
# ...
handler          = "retriever.lambda_handler"
}

Configure triggers and permissions

Depending on your use case, you may want to configure triggers or permissions for each Lambda function. For example, you could set up a CloudWatch Event Rule to trigger the scraper function periodically, or configure API Gateway to invoke the retriever function based on HTTP requests.

Sources:

Prompt 10

Can you show me how I would extract the enum classes to their own file so I can reference them from both the scraper.py and retriever.py files?

Response:

Sure, here's how you can extract the enum classes to their own file and reference them from both scraper.py and retriever.py:

Create a new file called enums.py in the same directory as scraper.py and retriever.py.

In enums.py, define your enum classes:

# enums.py
from enum import Enum

class FilingType(Enum):
    FINAL_DATE = 'Final Date'
    INITIAL_DATE = 'Initial Date'
    # Add other enum members as needed

class Country(Enum):
    ALL_AREAS = 'All Areas'
    CHINA_MAINLAND_BORN = 'China Mainland-born'
    # Add other enum members as needed

class Category(Enum):
    FIRST = 'First'
    SECOND = 'Second'
    THIRD = 'Third'
    # Add other enum members as needed

In scraper.py and retriever.py, import the enum classes from enums.py:

# scraper.py
from enums import FilingType, Country, Category

# Your scraper code here

# retriever.py
from enums import FilingType, Country, Category

# Your retriever code here

Now, you can use the enum classes FilingType, Country, and Category in both scraper.py and retriever.py.

For example, in retriever.py, you can use the enum classes like this:

# retriever.py
from enums import FilingType, Country, Category

def lambda_handler(event, context):
    query_params = event.get('queryStringParameters', {})

    filing_type_str = query_params.get('filing_type', FilingType.FINAL_DATE.name)
    filing_type = FilingType[filing_type_str.upper()]

    country_str = query_params.get('country', Country.ALL_AREAS.name)
    country = Country[country_str.upper()]

    category_str = query_params.get('category', Category.THIRD.name)
    category = Category[category_str.upper()]

    # Your retrieval logic here

By extracting the enum classes to a separate file (enums.py), you can easily import and use them in both scraper.py and retriever.py. This approach promotes code reusability and maintainability, as you only need to define the enum classes in one place, and any changes to the enum classes will be reflected in both files.

Prompt 11

Can you please show me how I would update the line table = dynamodb.Table('VisaBulletinData') to use an environment variable called "BULLETIN_DATA" instead of the hard-coded string, and only use that hard-coded string as a fallback if the environment variable is not set?

Response:

Sure, here's how you can update the line table = dynamodb.Table('VisaBulletinData') to use an environment variable called BULLETIN_DATA and fallback to the hard-coded string if the environment variable is not set:

import os

table_name = os.environ.get('BULLETIN_DATA', 'VisaBulletinData')
table = dynamodb.Table(table_name)

Here's what's happening:

import os imports the os module, which provides a way to interact with the operating system, including accessing environment variables.
os.environ.get('BULLETIN_DATA', 'VisaBulletinData') retrieves the value of the environment variable BULLETIN_DATA. If the environment variable is not set, it will use the fallback value 'VisaBulletinData'.
The retrieved value (either from the environment variable or the fallback value) is assigned to the table_name variable.
dynamodb.Table(table_name) creates a DynamoDB Table object using the value stored in table_name. [1]

By using os.environ.get(), you can easily switch between using the environment variable or the hard-coded string, depending on whether the environment variable is set or not.

To set the BULLETIN_DATA environment variable when running your Lambda function, you can configure it in the AWS Lambda console or through your Terraform configuration. Here's an example of how you can set the environment variable using Terraform:

resource "aws_lambda_function" "your_lambda_function" {
  # ... other configuration ...

  environment {
    variables = {
      BULLETIN_DATA = "YourTableName"
    }
  }
}

Replace "YourTableName" with the desired name of your DynamoDB table.

With this configuration, your Lambda function will use the value of the BULLETIN_DATA environment variable as the DynamoDB table name. If the environment variable is not set (or set to an empty string), it will fallback to using the hard-coded string 'VisaBulletinData'.

Sources:

aws-lambda-dynamodb - AWS Solutions Constructs

Prompt 12

Can you please show me how I would call the visa_bulletin_retriever Lambda function using the AWS CLI on my local terminal with an event payload of:

{
    'queryStringParameters': {
        'filing_type': 'FINAL_DATE',
        'category': 'THIRD',
        'country': 'ALL_AREAS'
    }
}

Response:

Sure, here's how you can invoke the visa_bulletin_retriever Lambda function using the AWS CLI on your local terminal with the provided event payload:

aws lambda invoke \
  --function-name visa_bulletin_retriever \
  --cli-binary-format raw-in-base64-out \
  --payload '{"queryStringParameters": {"filing_type": "FINAL_DATE", "category": "THIRD", "country": "ALL_AREAS"}}' \
  response.json

Let's break down this command:

aws lambda invoke: This is the AWS CLI command to invoke a Lambda function.
--function-name visa_bulletin_retriever: Specifies the name of the Lambda function you want to invoke. Replace visa_bulletin_retriever with the actual name of your Lambda function.
--cli-binary-format raw-in-base64-out: This option tells the AWS CLI to send the payload as raw binary data and to decode the response from Base64.
--payload '{"queryStringParameters": {"filing_type": "FINAL_DATE", "category": "THIRD", "country": "ALL_AREAS"}}': This is the JSON payload that you want to pass to your Lambda function. It matches the structure you provided in your example.
response.json: This is the name of the file where the Lambda function's response will be saved. You can choose any file name you prefer.

After running this command, the Lambda function will be invoked with the provided payload, and the response will be saved in the response.json file.

Make sure you have the AWS CLI configured with the appropriate credentials and permissions to invoke the Lambda function.

If you want to see the response in the terminal instead of saving it to a file, you can remove the response.json part from the command:

aws lambda invoke \
  --function-name visa_bulletin_retriever \
  --cli-binary-format raw-in-base64-out \
  --payload '{"queryStringParameters": {"filing_type": "FINAL_DATE", "category": "THIRD", "country": "ALL_AREAS"}}'

This will print the Lambda function's response directly in the terminal.

Note: If you need to pass more complex payloads or payloads from a file, you can use the --payload-json file://path/to/payload.json option instead of embedding the JSON payload directly in the command.

Prompt 13

Can you please show me how to update the return method for lambda_handler in retriever.py so that the data is in the following shape please?

{
            'filing_type': 'Final Date',
            'category': '3rd',
            'country': 'All Chargeability Areas Except Those Listed',
            'data': [
                {'bulletin_date': '2024-07-01', 'date': '2021-12-01' },
                {'bulletin_date': '2024-08-01', 'date': '2021-12-05' }
            ]
        }

Response:

To update the return method for lambda_handler in retriever.py to return the data in the desired shape, you can modify the code as follows:

import json

def lambda_handler(event, context):
    query_params = event.get('queryStringParameters', {})

    filing_type_str = query_params.get('filing_type', FilingType.FINAL_DATE.name)
    filing_type = FilingType[filing_type_str.upper()].value

    country_str = query_params.get('country', Country.ALL_AREAS.name)
    country = Country[country_str.upper()].value

    category_str = query_params.get('category', Category.THIRD.name)
    category = Category[category_str.upper()].value

    data = read_data(filing_type, category, country)

    response = {
        'filing_type': filing_type,
        'category': category,
        'country': country,
        'data': data
    }

    return {
        'statusCode': 200,
        'body': json.dumps(response)
    }

Here's what's happening:

We import the json module to convert the Python dictionary to a JSON string.
Inside the lambda_handler function, we retrieve the filing_type, country, and category values from the query_params dictionary, using the corresponding enum values as defaults.
Instead of using the string names of the enum members, we use the value attribute to get the actual string values associated with each enum member (e.g., FilingType.FINAL_DATE.value returns 'Final Date').
We call the read_data function to get the data based on the filing_type, category, and country values.
We create a new dictionary response with the desired shape, including the filing_type, category, country, and data keys.
We return a dictionary with a statusCode of 200 and a body containing the JSON-encoded response dictionary.

With this modification, the Lambda function will return a response in the desired shape:

{
    "statusCode": 200,
    "body": "{\"filing_type\": \"Final Date\", \"category\": \"3rd\", \"country\": \"All Chargeability Areas Except Those Listed\", \"data\": [{\"bulletin_date\": \"2024-07-01\", \"date\": \"2021-12-01\"}, {\"bulletin_date\": \"2024-08-01\", \"date\": \"2021-12-05\"}]}"
}

Note that the body is a JSON-encoded string, so you may need to parse it on the client-side (e.g., using JSON.parse() in JavaScript) to work with the data as a JavaScript object.

Fixing... err... I mean "implementing" my Amazon DynamoDB approach with Gen-AI

Cobus Bernard 🌩🥑🇿🇦 in 🇺🇸 — Tue, 02 Jul 2024 17:32:52 +0000

Introduction

In the previous article, I started building a serverless app to scrape the USCIS priority dates from their website. I was able to extract the dates from the website, but the schema for my DynamoDB table needed a change to store the data. Today, I'm going to tackle this using Amazon Q Developer.

The issue I ran into was how I was storing my data. I had the primary key as pk = f"{filing_type}#{category}" and the secondary key as sk = f"{country}". The data that I'm trying to store has a date for when the Visa Bulletin was published (bulletin_date), and each bulletin has two tables I'm interested in for the priority date (filing_type for Final Date and Application Date). Each of those tables has a row per visa category (category), and column for the country groupings. The date for each combination of country and category is what I'm trying to store. This will allow me to query the historic data specific to me, with category='3rd', filing_type='Final Date', and country='All Chargeability Areas Except Those Listed'.

Table Schema

First item on my list today is to ensure I have a table schema to store my data without overwriting entries like I had in my previous attempt. First, I update the schema using the current data structure, update the store_data function with structure below (and also implement handling table request throttling), with a quick detour to dig into PROVISIONED vs PAY_PER_REQUEST for my table access. Now we can update our Terraform with the new table definition:

resource "aws_dynamodb_table" "visa_bulletin_data" {
  name           = "VisaBulletinData"
  billing_mode   = "PAY_PER_REQUEST"

  hash_key       = "pk"
  range_key      = "sk"

  attribute {
    name = "pk"
    type = "S"
  }

  attribute {
    name = "sk"
    type = "S"
  }
}

The bulletin_date will be used for the primary key (pk), and a composite secondary key (sk) using the filing_type, country, and category to group the data. Our pk and sk will be defined in our code as:

pk = f"BULLETIN_DATE#{bulletin_date.strftime('%Y-%m')}"
sk = f"FILING_TYPE#{filing_type}#COUNTRY#{country}#CATEGORY#{category}"

I went back and forth between using provisioned capacity vs on-demand, I do like the idea of using the retry logic, but in the end, I settled on going for a simpler approach. I know that once I have everything working, I will do another pass to improve the code for the app. Ensuring that I combine the database write / retry logic with how I currently handle getting the data will cause issues. In the scrape_visa_bulletin function, I'm looping over each of the bulletins to extract the data, and adding that URL to the ProcessedURLs table so I don't process it again, but if writing to the database fails, it won't reprocess the pages that weren't saved. We now have the following for our store_data function:

def store_data(data):
    try:
        print("Storing data")
        for item in data:
            filing_type = item['filing_type']
            country = item['country']
            category = item['category']
            bulletin_date = datetime.strptime(item['bulletin_date'], "%Y-%m-%d")
            date = datetime.strptime(item['date'], "%Y-%m-%d")

            pk = f"BULLETIN_DATE#{bulletin_date.strftime('%Y-%m')}"
            sk = f"FILING_TYPE#{filing_type}#COUNTRY#{country}#CATEGORY#{category}"

            table.put_item(
                Item={
                    'pk': pk,
                    'sk': sk,
                    'filing_type': filing_type,
                    'country': country,
                    'category': category,
                    'bulletin_date': bulletin_date.strftime("%Y-%m-%d"),
                    'date': date.strftime("%Y-%m-%d")
                }
            )
        print("Done storing data")
    except Exception as e:
        print(f"Unable to store the data, error: {e}")

After running terraform apply, grabbing some fresh coffee, finishing it, and cleaning my keyboard, I can test the updated code.

TIL: it takes time to change a DynamoDB table's access mode.

aws_dynamodb_table.visa_bulletin_data: Modifications complete after 17m41s [id=VisaBulletinData]
aws_lambda_function.visa_bulletin_scraper: Modifying... [id=visa-bulletin-scraper]
aws_lambda_function.visa_bulletin_scraper: Still modifying... [id=visa-bulletin-scraper, 10s elapsed]
aws_lambda_function.visa_bulletin_scraper: Still modifying... [id=visa-bulletin-scraper, 20s elapsed]
aws_lambda_function.visa_bulletin_scraper: Modifications complete after 25s [id=visa-bulletin-scraper]

Apply complete! Resources: 1 added, 3 changed, 1 destroyed.
(package) ➜  uscis-priority-date-tracker git:(main) ✗

Commented out the lines that handle checking if the URL was previously processed, and I can run the app again. Woohoo!! I can see my data in my table! It looks like what I need, but now I need to update the read_data method to retrieve it.

Reading the Data

Updating read_data to use the new structure, and printing the output provides the following code (I'm using defaults for now) after needing to query using the wrong pk and sk:

def read_data(filing_type='Final Date', category='3rd', country='All Chargeability Areas Except Those Listed'):
    pk_prefix = "BULLETIN_DATE#"
    sk_prefix = f"FILING_TYPE#{filing_type}#COUNTRY#{country}#CATEGORY#{category}"

    response = table.query(
        KeyConditionExpression=Key('pk').begins_with(pk_prefix) & Key('sk').begins_with(sk_prefix),
        ScanIndexForward=False
    )

    items = response['Items']

    # Sort the items by bulletin_date in descending order
    sorted_items = sorted(items, key=lambda x: x['bulletin_date'], reverse=True)

    # Print the header
    print(f"Retrieved the [{filing_type}] data for [{category}] for [{country}]:")

    # Print each item
    for item in sorted_items:
        bulletin_date = item['bulletin_date']
        date = item['date']
        print(f"Bulletin {bulletin_date}: {date}")

    return sorted_items

And 💥. It does not work!!!

Traceback (most recent call last):
  File "/Users/cobusb/projects/uscis-priority-date-tracker/src/local_test.py", line 21, in <module>
    result = lambda_handler(mock_event, mock_context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cobusb/projects/uscis-priority-date-tracker/src/handler.py", line 190, in lambda_handler
    eb3_data = read_data()
               ^^^^^^^^^^^
  File "/Users/cobusb/projects/uscis-priority-date-tracker/src/handler.py", line 123, in read_data
    response = table.query(
               ^^^^^^^^^^^^
  File "/Users/cobusb/projects/terraform-samples/us-visa-dates-checker/src/package/lib/python3.12/site-packages/boto3/resources/factory.py", line 581, in do_action
    response = action(self, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cobusb/projects/terraform-samples/us-visa-dates-checker/src/package/lib/python3.12/site-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cobusb/projects/terraform-samples/us-visa-dates-checker/src/package/lib/python3.12/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cobusb/projects/terraform-samples/us-visa-dates-checker/src/package/lib/python3.12/site-packages/botocore/client.py", line 1021, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the Query operation: Query key condition not supported

Turns out that the primary key needs to be an "equality condition". I also notice that the files referenced are from a different directory than where I moved my code to. Initially this was going to be part of a larger repo with Terraform examples, but decided to split it out. I think this is due to using venv from that folder, but as long as my code runs, I'm going to ignore it for now. Let's fix the database first.

Redoing the Table Again

After checking if a schema change would fix it, I update both the store_data and read_data methods to use the new schema:

def store_data(data):
    try:
        for item in data:
            filing_type = item['filing_type']
            country = item['country']
            category = item['category']
            bulletin_date = datetime.strptime(item['bulletin_date'], "%Y-%m-%d")
            date = datetime.strptime(item['date'], "%Y-%m-%d")

            pk = f"FILING_TYPE#{filing_type}#CATEGORY#{category}#COUNTRY#{country}"
            sk = f"BULLETIN_DATE#{bulletin_date.strftime('%Y-%m-%d')}"

            table.put_item(
                Item={
                    'pk': pk,
                    'sk': sk,
                    'filing_type': filing_type,
                    'country': country,
                    'category': category,
                    'bulletin_date': bulletin_date.strftime("%Y-%m-%d"),
                    'date': date.strftime("%Y-%m-%d")
                }
            )
    except Exception as e:
        print(f"Unable to store the data, error: {e}")

def read_data(filing_type='Final Date', category='3rd', country='All Chargeability Areas Except Those Listed'):
    pk = f"FILING_TYPE#{filing_type}#CATEGORY#{category}#COUNTRY#{country}"

    response = table.query(
        KeyConditionExpression=Key('pk').eq(pk),
        ScanIndexForward=False  # Reverse the order to get the latest bulletin_date first
    )

    items = response['Items']

    # Sort the items by bulletin_date in descending order
    sorted_items = sorted(items, key=lambda x: x['sk'], reverse=True)

    # Print the header
    print(f"Retrieved the [{filing_type}] data for [{category}] for [{country}]:")

    # Print each item
    for item in sorted_items:
        date = item['date']
        print(f"Bulletin {bulletin_date}: {date}")

    return sorted_items

While I've used DynamoDB quite a bit in the past, I haven't had to deal with data when doing a schema migration. I'm going to save that for another day, for now I just want a way to delete all the data with the AWS CLI. Looking at the list of methods, I don't see an equivalent to SQL's TRUNCATE TABLE statement, and it doesn't look like it exists. I'm now very curious what will happen to the data, so I'm going to update the table definition in Terraform and YOLO it.

Oh.

The table schema isn't changing, only how I store the data.

💡!!!

Ok, this makes sense. We aren't changing the table schema, only the data we are storing in it. I feel like I need to generate on of those 300 memes with "THIS IS NOSQL!!!!!". To clear out the "old" data, I can either manually click to delete them in batches via the AWS Console, write some extra code, or drop and recreate the table - dropping the table since this is Sparta :P

After another terraform apply, the code works! Here's the first few lines of output:

Retrieved the [Final Date] data for [3rd] for [All Chargeability Areas Except Those Listed]:
Bulletin 2024-07-01: 2021-12-01
Bulletin 2024-06-01: 2022-11-22
Bulletin 2024-05-01: 2022-11-22
Bulletin 2024-04-01: 2022-11-22
Bulletin 2024-03-01: 2022-09-08
Bulletin 2024-02-01: 2022-09-01
Bulletin 2024-01-01: 2022-08-01
Bulletin 2023-12-01: 2021-12-01

What I Learnt

After the back and forth to get the table structure to store my data, and then that last change of the values of the pk and sk for the data that didn't change the table schema, I think I'm starting to think about the data structures in the right way. And repeating the recap from the first piece, the overall process of trial and error by patching different solutions together is different. Instead of the searching and multiple tabs, I now rarely leave my IDE.

I've added a todo for myself to read up on how to handle migrating existing data with DynamoDB, think that will be very interesting. In the past, I've done many large SQL based data migrations, and I honestly have no idea how I would do it for this app other than code to handle it. Probably would need to have that in a function, call it from the main method, and do another deployment to delete it. Looks like I have my next article topic after this series is done.

While generative-ai has helped speed up creating applications, you do need to take into account that it is very literal. Looking at the response when I asked it for the AWS CLI command to delete all items in a DynamoDB table, it still provided a technically accurate and feasible solution, just not one you would really use.

Final Code

As for the previous piece, I've tagged this with article-2, and pushed the tag.

Source code for this article.

Conclusion

I've been adding the prompts at the bottom of each piece so far, and I'll keep doing it for the rest of this series as I'm trying to simulate watching and listening to someone live code, but in written form. The intent is to share the thought process instead of just the final, polished solution. This will still be turned into a single solution of how to do it from start to finish without the back and forth, but for that, I first need to finish up all the parts still waiting for me. Next up, I'll update the function so I can pass the parameters that are currently hard-coded via defaults for the read_data function. At least, that is the plan, right after I double check the message I had after pushing this code:

remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
remote: 
remote: GitHub found 1 vulnerability on build-on-aws/uscis-priority-date-tracker's default branch (1 moderate). To find out more, visit:
remote:      https://github.com/build-on-aws/uscis-priority-date-tracker/security/dependabot/1

Stay tuned for the next installment of this series...

Using Provisioned Capacity

Initially I used provisioned capacity for my DynamoDB table, along with logic to handle the retries for me, here is the code that was removed with this logic:

resource "aws_dynamodb_table" "visa_bulletin_data" {
  name           = "VisaBulletinData"
  billing_mode   = "PROVISIONED"
  read_capacity  = 5
  write_capacity = 5

  hash_key       = "pk"
  range_key      = "sk"

  attribute {
    name = "pk"
    type = "S"
  }

  attribute {
    name = "sk"
    type = "S"
  }
}

def store_data(data):
    try:
        print("Storing data")
        for item in data:
            filing_type = item['filing_type']
            country = item['country']
            category = item['category']
            bulletin_date = datetime.strptime(item['bulletin_date'], "%Y-%m-%d")
            date = datetime.strptime(item['date'], "%Y-%m-%d")

            pk = f"BULLETIN_DATE#{bulletin_date.strftime('%Y-%m')}"
            sk = f"FILING_TYPE#{filing_type}#COUNTRY#{country}#CATEGORY#{category}"

            max_retries = 5
            retry_delay = 1
            max_delay = 60

            for attempt in range(max_retries):
                try:
                    table.put_item(
                        Item={
                            'pk': pk,
                            'sk': sk,
                            'filing_type': filing_type,
                            'country': country,
                            'category': category,
                            'bulletin_date': bulletin_date.strftime("%Y-%m-%d"),
                            'date': date.strftime("%Y-%m-%d")
                        }
                    )
                    break  # Exit the retry loop if the write succeeds
                except table.meta.client.exceptions.ProvisionedThroughputExceededException as e:
                    if attempt == max_retries - 1:
                        raise e
                    else:
                        # Apply exponential backoff with jitter
                        delay = retry_delay * (2 ** attempt) + random.uniform(0, 1)
                        delay = min(delay, max_delay)
                        print(f"Write capacity exceeded, retrying in {delay} seconds...")
                        time.sleep(delay)
                except Exception as e:
                    print(f"Error writing item to DynamoDB: {e}")
                    break
        print("Done storing data")
    except Exception as e:
        print(f"Unable to store the data, error: {e}")

Prompts used in this article

Prompt 1

Can you help me define the table structure for DynamoDB using Terraform, I'm trying to store the following data? My data consists of a collection of objects, each with a BulletinDate for the month/year the data is from, and then split into 2 categories for the filing type called "priority date" and "filing date". For each of these filings, I will have a collection of data defined by 2 dimensions: Country, and Category. So for a single BulletinDate, I would have a collection of data using Country and Category as the key, with PriorityDate as the value, and the collection needs to be grouped by FilingType for "priority date" and "filing date". Can you please show me how to create this table?

Response:

Sure, I can help you define the table structure for DynamoDB using Terraform. Based on your data structure, we can create a single table with a composite primary key consisting of a partition key and a sort key. [1]

Here's how you can define the table structure in your Terraform code:

resource "aws_dynamodb_table" "visa_bulletin_data" {
  name           = "VisaBulletinData"
  billing_mode   = "PROVISIONED" # or "PAY_PER_REQUEST"
  read_capacity  = 5             # Set appropriate read/write capacity units for provisioned mode
  write_capacity = 5

  hash_key       = "pk"          # Partition key
  range_key      = "sk"          # Sort key

  attribute {
    name = "pk"
    type = "S" # String type
  }

  attribute {
    name = "sk"
    type = "S" # String type
  }

  # Additional settings, global secondary indexes, etc.
}

In this structure, we're using a composite primary key with a partition key (pk) and a sort key (sk). Both keys are defined as strings (S).

To store your data, you can use the following format for the primary key:

pk = "BULLETIN_DATE#<BulletinDate>"
sk = "FILING_TYPE#<FilingType>#COUNTRY#<Country>#CATEGORY#<Category>"

For example, if you have a BulletinDate of "2023-05", a FilingType of "priority_date", a Country of "India", and a Category of "EB3", the primary key would be: [2]

pk = "BULLETIN_DATE#2023-05"
sk = "FILING_TYPE#priority_date#COUNTRY#India#CATEGORY#EB3"

With this structure, you can store the PriorityDate value as an attribute in the DynamoDB item.

You can also consider creating a Global Secondary Index (GSI) on the pk and sk attributes to allow efficient querying based on different combinations of the primary key attributes.

Note: Make sure to replace the placeholders (<BulletinDate>, <FilingType>, <Country>, and <Category>) with your actual values when constructing the primary key.

Sources:

Prompt 2

Thanks, given that I have my source data added to a flat collection with the code below, can you please provide the python code to store this data into the table above? Here's how I store the data in my collection:

employment_based_data.append({
    'filing_type': filing_type,
    'country': countries[cell_id - 1],
    'category': category_value,
    'bulletin_date': bulletin_date.strftime("%Y-%m-%d"),
    'date': cell_date.strftime("%Y-%m-%d")
})

Response:

Sure, I can provide you with the Python code to store the data in the DynamoDB table using the structure we defined earlier. Here's an example: [1]

import boto3
from datetime import datetime

# Initialize DynamoDB resource
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('VisaBulletinData')

def store_data(data):
    for item in data:
        filing_type = item['filing_type']
        country = item['country']
        category = item['category']
        bulletin_date = datetime.strptime(item['bulletin_date'], "%Y-%m-%d")
        date = datetime.strptime(item['date'], "%Y-%m-%d")

        pk = f"BULLETIN_DATE#{bulletin_date.strftime('%Y-%m')}"
        sk = f"FILING_TYPE#{filing_type}#COUNTRY#{country}#CATEGORY#{category}"

        table.put_item(
            Item={
                'pk': pk,
                'sk': sk,
                'filing_type': filing_type,
                'country': country,
                'category': category,
                'bulletin_date': bulletin_date.strftime("%Y-%m-%d"),
                'date': date.strftime("%Y-%m-%d")
            }
        )

# Assuming you have the employment_based_data list populated
store_data(employment_based_data)

Here's how the store_data function works:

We iterate over each item in the employment_based_data list.
For each item, we extract the filing_type, country, category, bulletin_date, and date values.
We construct the partition key (pk) using the BULLETIN_DATE prefix and the bulletin_date in the format YYYY-MM.
We construct the sort key (sk) using the FILING_TYPE, filing_type, COUNTRY, country, and CATEGORY, category values.
We use the put_item method of the DynamoDB table to store the item with the constructed primary key ( pk and sk) and the other attributes ( filing_type, country, category, bulletin_date, and date). [2]

Note that we're using the datetime module to convert the bulletin_date and date strings to datetime objects, and then back to strings in the desired format (YYYY-MM for bulletin_date and YYYY-MM-DD for date).

After running this code, your data should be stored in the DynamoDB table with the defined structure. [3]

Sources:

Prompt 3

Can you update this code to be able to handle exceeding the write capacity for my table with some kind of graceful backoff and retry logic?

Response:

Sure, I can update the code to handle exceeding the write capacity for your DynamoDB table by implementing a backoff and retry logic. Here's an example of how you can modify the store_data function to achieve this:

import boto3
from datetime import datetime
import time
import random
import math

# Initialize DynamoDB resource
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('VisaBulletinData')

def store_data(data):
    for item in data:
        filing_type = item['filing_type']
        country = item['country']
        category = item['category']
        bulletin_date = datetime.strptime(item['bulletin_date'], "%Y-%m-%d")
        date = datetime.strptime(item['date'], "%Y-%m-%d")

        pk = f"BULLETIN_DATE#{bulletin_date.strftime('%Y-%m')}"
        sk = f"FILING_TYPE#{filing_type}#COUNTRY#{country}#CATEGORY#{category}"

        max_retries = 5  # Maximum number of retries
        retry_delay = 1  # Initial retry delay in seconds
        max_delay = 60   # Maximum retry delay in seconds

        for attempt in range(max_retries):
            try:
                table.put_item(
                    Item={
                        'pk': pk,
                        'sk': sk,
                        'filing_type': filing_type,
                        'country': country,
                        'category': category,
                        'bulletin_date': bulletin_date.strftime("%Y-%m-%d"),
                        'date': date.strftime("%Y-%m-%d")
                    }
                )
                break  # Exit the retry loop if the write succeeds
            except table.meta.client.exceptions.ProvisionedThroughputExceededException as e:
                if attempt == max_retries - 1:
                    raise e  # Raise the exception if the maximum number of retries is reached
                else:
                    # Apply exponential backoff with jitter
                    delay = retry_delay * (2 ** attempt) + random.uniform(0, 1)
                    delay = min(delay, max_delay)
                    print(f"Write capacity exceeded, retrying in {delay} seconds...")
                    time.sleep(delay)
            except Exception as e:
                print(f"Error writing item to DynamoDB: {e}")
                break

# Assuming you have the employment_based_data list populated
store_data(employment_based_data)

Here's what we've added to the store_data function:

We define the maximum number of retries (max_retries) and the initial retry delay (retry_delay) in seconds.
We also define the maximum retry delay (max_delay) in seconds to prevent excessive delays.
We wrap the put_item operation in a for loop that runs up to max_retries times.
If the ProvisionedThroughputExceededException is raised, indicating that the write capacity has been exceeded, we apply an exponential backoff with jitter before retrying the write operation.
The exponential backoff delay is calculated as retry_delay * (2 ** attempt) + random.uniform(0, 1), where attempt is the current retry attempt. This introduces a random jitter to prevent multiple clients from retrying at the same time.
The calculated delay is capped at max_delay to prevent excessive delays.
If the maximum number of retries is reached, the ProvisionedThroughputExceededException is raised, allowing you to handle it as needed.
For any other exceptions, we print an error message and break out of the retry loop.

With this implementation, the code will retry writing to the DynamoDB table if the write capacity is exceeded, with an exponential backoff and jitter to avoid overwhelming the table with retries. If the maximum number of retries is reached, the exception is raised, allowing you to handle it appropriately (e.g., logging, retrying later, etc.).

Note that this implementation assumes that you're using the provisioned capacity mode for your DynamoDB table. If you're using the on-demand capacity mode, you don't need to handle the ProvisionedThroughputExceededException specifically, as DynamoDB will automatically handle the scaling for you.

Prompt 4

Can you explain the difference between the provisioned capacity and on-demand for DynamoDB, and include how it would affect accessing my data?

Response:

Certainly! Amazon DynamoDB offers two capacity modes: provisioned capacity and on-demand. The choice between these modes affects the way you manage capacity and pay for the service. Here's an explanation of the differences and how they impact costs: [1]

Provisioned Capacity Mode:

In this mode, you specify the number of read and write capacity units you expect your application to require.
You are billed for the provisioned capacity, regardless of whether you consume it or not.
Provisioned capacity is suitable for workloads with predictable or consistent traffic patterns, where you can forecast your capacity requirements. [2]
You can enable auto scaling to automatically adjust the provisioned capacity based on your utilization, helping to optimize costs and performance.
If your application exceeds the provisioned capacity, you may experience throttling, which can be mitigated by increasing the provisioned capacity or enabling auto scaling.

On-Demand Capacity Mode:

In this mode, you don't need to provision any capacity upfront.
You pay for the data reads and writes your application performs, based on the number of read and write request units consumed.
On-demand capacity is ideal for workloads with unpredictable or [3]

Sources:

On-demand capacity mode - Amazon DynamoDB

Prompt 5

Thanks, can you show me how to update the read_data function using this new table schema so that I can pass filing_type, category, and country to return a sorted list? It should use the date bulletin_date for the sort.

Response:

Sure, here's how you can update the read_data function to retrieve and sort the data based on the filing_type, category, and country parameters, using the bulletin_date for sorting:

from boto3.dynamodb.conditions import Key

def read_data(filing_type='Final Date', category='3rd', country='All Chargeability Areas Except Those Listed'):
    pk = f"FILING_TYPE#{filing_type}#CATEGORY#{category}"
    sk_prefix = f"COUNTRY#{country}"

    response = table.query(
        KeyConditionExpression=Key('pk').eq(pk) & Key('sk').begins_with(sk_prefix),
        ScanIndexForward=False  # Reverse the order to get the latest bulletin_date first
    )

    items = response['Items']

    # Sort the items by bulletin_date in descending order
    sorted_items = sorted(items, key=lambda x: x['bulletin_date'], reverse=True)

    return sorted_items

Here's how the read_data function works:

We construct the partition key (pk) using the FILING_TYPE, filing_type, and CATEGORY, category values.
We construct the sort key prefix (sk_prefix) using the COUNTRY and country values.
We use the query method of the DynamoDB table to retrieve items where the partition key ( pk) matches the constructed value, and the sort key ( sk) begins with the constructed prefix ( sk_prefix).
We set the ScanIndexForward parameter to False to reverse the order of the results, so that the latest bulletin_date is returned first.
We store the retrieved items in the items variable.
We sort the items list in descending order based on the bulletin_date attribute using the sorted function and a lambda function as the key.
We return the sorted list of items.

To use the read_data function, you can call it with the desired filing_type, category, and country parameters:

# Retrieve and sort data for 'Final Date', '3rd' category, and 'All Chargeability Areas Except Those Listed' country
sorted_data = read_data()

# Retrieve and sort data for 'Filing Date', 'EB2' category, and 'India' country
sorted_data = read_data(filing_type='Filing Date', category='EB2', country='India')

The sorted_data variable will contain a list of items sorted in descending order by bulletin_date, filtered by the specified filing_type, category, and country.

Note that the query operation in DynamoDB is efficient for retrieving data based on the primary key (partition key and sort key). If you need to filter the data based on other attributes, you may need to consider using a Global Secondary Index (GSI) or performing a scan operation, which can be less efficient for large datasets.

Prompt 6

Thanks, can you now show me how I would use the returned data to print out a line per entry, with the values from the input parameters? It should have the following first (with the [] around the values for the variables):
Retrieved the [Final Date] data for [EB3] for [All Chargeability Areas Except Those Listed]:

And then a line per entry with this format:
Bulletin 2024-07-01: 2021-11-01

The code provided by this response isn't quite correct, it is using the previous table schema for the pk and sk, so I just copied the lines from store_data to replace them.

Response:
Certainly! Here's how you can modify the read_data function to print the desired output format:

from boto3.dynamodb.conditions import Key

def read_data(filing_type='Final Date', category='3rd', country='All Chargeability Areas Except Those Listed'):
    pk = f"FILING_TYPE#{filing_type}#CATEGORY#{category}"
    sk_prefix = f"COUNTRY#{country}"

    response = table.query(
        KeyConditionExpression=Key('pk').eq(pk) & Key('sk').begins_with(sk_prefix),
        ScanIndexForward=False  # Reverse the order to get the latest bulletin_date first
    )

    items = response['Items']

    # Sort the items by bulletin_date in descending order
    sorted_items = sorted(items, key=lambda x: x['bulletin_date'], reverse=True)

    # Print the header
    print(f"Retrieved the [{filing_type}] data for [{category}] for [{country}]:")

    # Print each item
    for item in sorted_items:
        bulletin_date = item['bulletin_date']
        date = item['date']
        print(f"Bulletin {bulletin_date}: {date}")

    return sorted_items

In this updated version of the read_data function, we've added the following changes:

After sorting the items, we print the header line using the print function and an f-string. The header line will be in the format:
```
Retrieved the [Final Date] data for [EB3] for [All Chargeability Areas Except Those Listed]:
```
We then iterate over the sorted_items list using a for loop.
For each item, we extract the bulletin_date and date values.
We print a line for each item using the print function and an f-string. The line will be in the format:
```
Bulletin 2024-07-01: 2021-11-01
```

To use the updated read_data function, you can call it with the desired filing_type, category, and country parameters:

# Retrieve and print data for 'Final Date', '3rd' category, and 'All Chargeability Areas Except Those Listed' country
read_data()

# Retrieve and print data for 'Filing Date', 'EB2' category, and 'India' country
read_data(filing_type='Filing Date', category='EB2', country='India')

The output will be printed to the console, with the header line followed by a line for each item in the sorted list, displaying the bulletin_date and date values.

Note that this implementation assumes that you have the necessary data in your DynamoDB table with the expected attributes ( bulletin_date and date). If the data structure or attribute names are different, you'll need to adjust the code accordingly.

Prompt 7

This doesn't look correct, can you please recheck as earlier in the conversation we defined pk and sk for the table when storing data using this:

pk = f"BULLETIN_DATE#{bulletin_date.strftime('%Y-%m')}"
sk = f"FILING_TYPE#{filing_type}#COUNTRY#{country}#CATEGORY#{category}"

Response:

You're absolutely right, my apologies. I should have used the correct primary key structure that we defined earlier. Here's the updated read_data function with the correct primary key construction:

from boto3.dynamodb.conditions import Key

def read_data(filing_type='Final Date', category='3rd', country='All Chargeability Areas Except Those Listed'):
    pk_prefix = "BULLETIN_DATE#"
    sk_prefix = f"FILING_TYPE#{filing_type}#COUNTRY#{country}#CATEGORY#{category}"

    response = table.query(
        KeyConditionExpression=Key('pk').begins_with(pk_prefix) & Key('sk').begins_with(sk_prefix),
        ScanIndexForward=False  # Reverse the order to get the latest bulletin_date first
    )

    items = response['Items']

    # Sort the items by bulletin_date in descending order
    sorted_items = sorted(items, key=lambda x: x['bulletin_date'], reverse=True)

    # Print the header
    print(f"Retrieved the [{filing_type}] data for [{category}] for [{country}]:")

    # Print each item
    for item in sorted_items:
        bulletin_date = item['bulletin_date']
        date = item['date']
        print(f"Bulletin {bulletin_date}: {date}")

    return sorted_items

Here are the changes:

We define the pk_prefix as "BULLETIN_DATE#" to match the partition key structure we defined earlier.
We construct the sk_prefix using the filing_type, country, and category values, following the sort key structure we defined earlier.
In the KeyConditionExpression of the query method, we use the begins_with condition for both the partition key ( pk) and the sort key ( sk) to retrieve items where the pk begins with "BULLETIN_DATE#" and the sk begins with the constructed sk_prefix.

With these changes, the read_data function should now correctly retrieve and print the data based on the primary key structure we defined earlier.

Thank you for catching my mistake. Please let me know if this updated version looks correct.

Prompt 8

When I use this code, it errors with:

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the Query operation: Query key condition not supported

Response:

Ah, I see the issue now. The ValidationException is occurring because the KeyConditionExpression we're using is not supported for the query operation in DynamoDB. [1]

The query operation in DynamoDB requires the partition key to be an equality condition, and the sort key can either be an equality condition or a range condition (using comparison operators or the begins_with function).

To fix this issue, we need to modify the read_data function to use the partition key as an equality condition and the sort key as a range condition. Here's the updated code:

from boto3.dynamodb.conditions import Key

def read_data(filing_type='Final Date', category='3rd', country='All Chargeability Areas Except Those Listed'):
    pk = f"BULLETIN_DATE#{bulletin_date.strftime('%Y-%m')}"
    sk_prefix = f"FILING_TYPE#{filing_type}#COUNTRY#{country}#CATEGORY#{category}"

    response = table.query(
        KeyConditionExpression=Key('pk').eq(pk) & Key('sk').begins_with(sk_prefix),
        ScanIndexForward=False  # Reverse the order to get the latest bulletin_date first
    )

    items = response['Items']

    # Sort the items by bulletin_date in descending order
    sorted_items = sorted(items, key=lambda x: x['bulletin_date'], reverse=True)

    # Print the header
    print(f"Retrieved the [{filing_type}] data for [{category}] for [{country}]:")

    # Print each item
    for item in sorted_items:
        bulletin_date = item['bulletin_date']
        date = item['date']
        print(f"Bulletin {bulletin_date}: {date}")

    return sorted_items

Here are the changes:

We construct the partition key ( pk) using the BULLETIN_DATE prefix and the bulletin_date in the format YYYY-MM.
In the KeyConditionExpression of the query method, we use the eq condition for the partition key (pk) to match the constructed pk value exactly.
For the sort key (sk), we use the begins_with condition with the constructed sk_prefix.

By using the eq condition for the partition key and the begins_with condition for the sort key, we should be able to query the data correctly without encountering the ValidationException.

Note that you'll need to pass the bulletin_date parameter to the read_data function, or you can use a default value if you want to retrieve the latest data by default.

Please let me know if this updated code works for you or if you have any further questions.

Sources:

Key condition expressions for the Query operation - Amazon DynamoDB

Prompt 9

Ok, given the requirement for an equality condition on the primary key, can you suggest what data we should use for the primary key given this constraint? I want to be able to retrieve the data for a specific filing_type, category, and country.

Response:

Given the requirement for an equality condition on the partition key, we can modify the primary key structure to accommodate retrieving data for a specific filing_type, category, and country.

Here's a suggested primary key structure:

pk = "FILING_TYPE#<filing_type>#CATEGORY#<category>#COUNTRY#<country>"
sk = "BULLETIN_DATE#<bulletin_date>"

With this structure, the partition key ( pk) will be a composite key consisting of the filing_type, category, and country values, separated by a delimiter (#). The sort key ( sk) will be the bulletin_date.

Here's how you can update the store_data and read_data functions to use this new primary key structure:

def store_data(data):
    for item in data:
        filing_type = item['filing_type']
        country = item['country']
        category = item['category']
        bulletin_date = datetime.strptime(item['bulletin_date'], "%Y-%m-%d")
        date = datetime.strptime(item['date'], "%Y-%m-%d")

        pk = f"FILING_TYPE#{filing_type}#CATEGORY#{category}#COUNTRY#{country}"
        sk = f"BULLETIN_DATE#{bulletin_date.strftime('%Y-%m-%d')}"

        table.put_item(
            Item={
                'pk': pk,
                'sk': sk,
                'filing_type': filing_type,
                'country': country,
                'category': category,
                'bulletin_date': bulletin_date.strftime("%Y-%m-%d"),
                'date': date.strftime("%Y-%m-%d")
            }
        )

def read_data(filing_type, category, country):
    pk = f"FILING_TYPE#{filing_type}#CATEGORY#{category}#COUNTRY#{country}"

    response = table.query(
        KeyConditionExpression=Key('pk').eq(pk),
        ScanIndexForward=False  # Reverse the order to get the latest bulletin_date first
    )

    items = response['Items']

    # Sort the items by bulletin_date in descending order
    sorted_items = sorted(items, key=lambda x: x['sk'], reverse=True)

    # Print the header
    print(f"Retrieved the [{filing_type}] data for [{category}] for [{country}]:")

    # Print each item
    for item in sorted_items:
        date = item['date']
        print(f"Bulletin {bulletin_date}: {date}")

    return sorted_items

In the store_data function:

We construct the partition key ( pk) using the FILING_TYPE, filing_type, CATEGORY, category, and COUNTRY, country values.
We construct the sort key ( sk) using the BULLETIN_DATE prefix and the bulletin_date value in the format YYYY-MM-DD.

In the read_data function:

We construct the partition key ( pk) using the filing_type, category, and country values passed as arguments.
We use the query method with the KeyConditionExpression set to Key('pk').eq(pk) to retrieve items where the partition key matches the constructed pk value exactly.
We sort the retrieved items by the sort key ( sk) in descending order.
When printing the data, we extract the bulletin_date from the sort key ( sk) by splitting it on the # delimiter and taking the second part.

With this primary key structure, you can retrieve data for a specific filing_type, category, and country combination by passing those values to the read_data function:

# Retrieve data for 'Final Date', 'EB3', and 'All Chargeability Areas Except Those Listed'
read_data('Final Date', 'EB3', 'All Chargeability Areas Except Those Listed')

# Retrieve data for 'Filing Date', 'EB2', and 'India'
read_data('Filing Date', 'EB2', 'India')

This approach ensures that you can query the data efficiently using the partition key equality condition, while still being able to retrieve data for specific combinations of filing_type, category, and country.

Please note that if you need to query the data based on different combinations of these attributes, you may need to consider creating additional Global Secondary Indexes (GSIs) to optimize those queries.

Prompt 10

Thanks, what is the easiest way to delete all my data from my DynamoDB table since the schema has changed? If I can do it with the AWS CLI, can you show me the command please?

Response:

To delete all data from your DynamoDB table using the AWS CLI, you can use the scan operation to retrieve all items in the table, and then use the batch-write-item operation to delete them in batches. [1]

Here's the command to scan and retrieve all items from the table:

aws dynamodb scan --table-name VisaBulletinData --output json > items.json

This command will scan the VisaBulletinData table and store the retrieved items in a JSON file named items.json.

Next, you can use the following Python script to delete the items in batches using the batch-write-item operation:

import json
import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('VisaBulletinData')

# Load the items from the JSON file
with open('items.json', 'r') as f:
    items = json.load(f)['Items']

# Batch size for the write operation
batch_size = 25

# Delete items in batches
for i in range(0, len(items), batch_size):
    batch = items[i:i+batch_size]
    with table.batch_writer() as batch_writer:
        for item in batch:
    print(f"Deleted batch {i//batch_size+1}")

print("All items deleted from the table.")

Here's how the script works:

It loads the items from the items.json file created by the scan operation.
It defines a batch_size for the batch-write-item operation. In this example, it's set to 25, but you can adjust it based on your requirements.
It iterates over the items in batches of batch_size.
For each batch, it uses the batch_writer context manager to perform the batch-write-item operation.
Inside the batch_writer context, it iterates over the items in the batch and calls the delete_item method with the primary key ( pk and sk) of each item.
After each batch is processed, it prints a message indicating the batch number.
Finally, it prints a message indicating that all items have been deleted from the table.

Make sure to replace 'VisaBulletinData' with the name of your DynamoDB table.

Note: This approach assumes that your table doesn't have any Global Secondary Indexes (GSIs). If you have GSIs, you'll need to delete the data from the GSIs as well, or consider recreating the table if it's more convenient.

Sources:

Use DeleteTable with an AWS SDK or CLI - Amazon DynamoDB

Prompt 11

Does DynamoDB have the equivalent of SQL's "Truncate table"?

Response:

Yes, DynamoDB does have an equivalent of SQL's "TRUNCATE TABLE" operation. It's called the DeleteTable operation, which deletes the entire table and all its data. [1]

Here's how you can use the AWS CLI to delete (truncate) a DynamoDB table:

aws dynamodb delete-table --table-name VisaBulletinData

Replace VisaBulletinData with the name of your DynamoDB table.

This command will delete the entire table, including all its data and any Global Secondary Indexes (GSIs) associated with the table.

After running this command, the table will be in the DELETING state for a short period, and then it will be completely removed from DynamoDB.

If you want to recreate the table with the same name and structure after deleting it, you'll need to wait for the DeleteTable operation to complete before creating a new table with the same name.

Alternatively, you can use the AWS Management Console, AWS SDKs, or the DynamoDB API to perform the DeleteTable operation. [2]

It's important to note that the DeleteTable operation is an irreversible operation, and all data in the table will be permanently deleted. If you need to keep a backup of the data, you should export it before running the DeleteTable operation.

Sources:

Prompt 12

Thanks, can you provide me with the updated Terraform to create this table please? Also change it to use on-demand pricing instead of provisioned.

Interesting that ttl and tags suddenly pop up in this response, I'm going to strip that out as I don't need or want that.

Response:

Sure, here's the updated Terraform code to create the DynamoDB table with the new schema and using on-demand pricing: [1]

resource "aws_dynamodb_table" "visa_bulletin_data" {
  name           = "VisaBulletinData"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "pk"
  range_key      = "sk"

  attribute {
    name = "pk"
    type = "S"
  }

  attribute {
    name = "sk"
    type = "S"
  }

  ttl {
    attribute_name = "TimeToExist"
    enabled        = false
  }

  tags = {
    Name        = "VisaBulletinData"
    Environment = "production"
  }
}

Here's what the code does:

It defines an aws_dynamodb_table resource with the name visa_bulletin_data.
The name attribute sets the name of the DynamoDB table to VisaBulletinData.
The billing_mode is set to PAY_PER_REQUEST, which enables on-demand pricing for the table. [2]
The hash_key and range_key attributes define the partition key ( pk) and sort key ( sk) for the table, respectively.
The attribute blocks define the attributes for the partition key ( pk) and sort key ( sk), both of which are strings ( S).
The ttl block is used to configure the Time to Live (TTL) feature for the table. In this case, it's disabled by setting enabled to false.
The tags block defines two tags for the DynamoDB table: Name and Environment.

After applying this Terraform code, it will create a new DynamoDB table named VisaBulletinData with the specified primary key structure ( pk and sk) and using on-demand pricing.

Note that you'll need to have the AWS provider configured in your Terraform code, and you'll need to run terraform init and terraform apply to create the DynamoDB table.

Also, keep in mind that with on-demand pricing, you'll be charged for the read and write requests made to the table, rather than provisioning read and write capacity units upfront. You can refer to the AWS DynamoDB pricing documentation for more details on the on-demand pricing model.

Sources:

Amazon DynamoDB Pricing for On-Demand Capacity

Building a Serverless Web Scraper with a ~little~ lot of help from Amazon Q

Cobus Bernard 🌩🥑🇿🇦 in 🇺🇸 — Mon, 01 Jul 2024 18:23:00 +0000

Introduction

Update: 2024-06-14: This turned out to be more than I initially though, so will be turned into a series of articles, this being the first. I'll be posting the followups in this series, and link them as a list.

I've been itching to build a little serverless API for a while now, so I decided to use Amazon Q Developer to help decide how to build it. I didn't feel like building yet another Todo or note taking app, so I picked a practical problem I currently have: US Green Card priority dates. Since I moved to the US at the end of 2021, I've been going through the process to apply for a Green Card. An important aspect of this is your "priority date". The process is a convoluted one, the short summary is that each month, they publish the dates per visa category. While the latest date is useful, I wanted to look at the historic data as the bulletins go back to 2002. There are many factors influencing how these dates progress, so I'll be looking at the last 5 years (at least, that's my plan right now).

The idea is to write an API that pulls the values from all the bulletins, and then add it to a database table so I can query it. I'll be using Amazon Q Developer to help me along the way - both for asking general questions, and to help generate my code. So let's get started. I'm going to try my best to follow the instructions as-is, and only make small changes to see how far I can get.

All the code for this app can be found on GitHub, with the code as it was at the end of this article on the article-1 tag.

Understanding the data

For each Visa Bulletin, there are 2 tables that I need to keep an eye on, the TL;DR: version is there are 2 important tables under "final action date" and "filing date", and only 1 of these is applicable at a given time for my category. At the moment, it is the "final action date" one, and the table looks like this for the July 2024 bulletin:

Architecture Choices

The first step is to decide how I want to build this. I want to start off using AWS Lambda functions, DynamoDB, Amazon S3, and then Terraform to create all the infrastructure, and then deploy my function(s). I'm expecting a lot of back and forth questions with Amazon Q Developer, so I'll add all the prompts/responses and my comments about them at the bottom of this page. I'm not sure which programming language to use yet, and will see what is suggested. Overall, it seem doable based on the first prompt's response, so let's start with the Terraform code.

Setting up Terraform

Setting up Terraform for a new project always has a chicken & egg problem: you need some infrastructure to store you state file, and you need IaC to create your infrastructure. After looking at the answer, I create the following files to deal with it. I started using Terraform in ~2015 from v0.5, so I'm incorporating what I've learned using it in production, and will do the same for all the responses from Amazon Q Developer throughout this article.

# variables.tf

variable "aws_profile" {
    default = "development"
}

variable "aws_region" {
  default = "us-west-2"
}

variable "state_file_bucket_name" {
  default = "tf-us-visa-dates-checker"
}

variable "state_file_lock_table_name" {
  default = "tf-us-visa-dates-checker-statelock"
}

variable "kms_key_alias" {
  default = "tf-us-visa-dates-checker"
}

# providers.tf

provider "aws" {
  region = var.aws_region
}

# _bootstrap.tf

# Bucket used to store our state file
resource "aws_s3_bucket" "state_file" {
  bucket = var.state_file_bucket_name

  lifecycle {
    prevent_destroy = true
  }
}

# Enabling bucket versioning to keep backup copies of the state file
resource "aws_s3_bucket_versioning" "state_file" {
  bucket = aws_s3_bucket.state_file.id

  versioning_configuration {
    status = "Enabled"
  }
}

# Table used to store the lock to prevent parallel runs causing issues
resource "aws_dynamodb_table" "state_file_lock" {
  name           = var.state_file_lock_table_name
  read_capacity  = 5
  write_capacity = 5
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

# (Optional) KMS Key and alias to use instead of default `alias/s3` one.
resource "aws_kms_key" "terraform" {
  description = "Key used for Terraform state files."
}

resource "aws_kms_alias" "terraform" {
  name          = "alias/terraform"
  target_key_id = aws_kms_key.terraform.key_id
}

# statefile.tf

terraform {
  ## The lines below are commented out for the first terraform apply run

  # backend "s3" {
  #   bucket         = "tf-us-visa-dates-checker"
  #   key            = "state.tfstate"
  #   region         = "us-west-2"
  #   dynamodb_table = "tf-us-visa-dates-checker-statelock"
  # }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.53.0"
    }
  }

  required_version = ">= 1.8.5"
}

You may have noticed the lack of any AWS credentials in the code above, and that is intentional. I know I will run into AWS credential issues when I try to run terraform plan - I specifically never have the default profile configured due to the many different accounts I use. I would rather have an error that forces me to think where I want to run something than accidentally deploy / make any API calls to the incorrect account. I have set up a profile for this project called development, so I export it via the AWS_PROFILE environment variable by calling:

export AWS_PROFILE=development

Now I can run terraform apply to create those resources for the state file, uncomment the lines in statefile.tf, and then run terraform init to store the state file in S3.

Important: You cannot use variables in the terraform block for the backend, so make sure to copy the exact names for the resources from _bootstrap.tf.

Ok, we can now start building the app.

Building the Lambda Function

I ask the following to help me get started:

Thanks, can you show me how I would write a webpage scraper with Lambda to scrape all the pages linked from https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin.html with the format https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-july-2024.html - they should be linked from the first page listed. Each page has multiple sections and tables, I would like to only store the rows in the tables listed under the heading "A. FINAL ACTION DATES FOR EMPLOYMENT-BASED PREFERENCE CASES" and "B. DATES FOR FILING OF EMPLOYMENT-BASED VISA APPLICATIONS". Can you show me how to create a Lambda function to do this and store the data in DynamoDB?

The provided code needs a few changes, and I now have this:

import requests
from bs4 import BeautifulSoup
import boto3
from datetime import datetime

# Initialize DynamoDB client
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('VisaBulletinData')

def scrape_visa_bulletin(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')

    # Find the tables with the relevant data
    employment_based_tables = soup.find_all('div', {'class': 'field-item even'})

    # Extract data from the tables
    employment_based_data = []
    for table in employment_based_tables:
        table_heading = table.find_previous('h3').text.strip()
        if 'A. FINAL ACTION DATES FOR EMPLOYMENT-BASED PREFERENCE CASES' in table_heading or 'B. DATES FOR FILING OF EMPLOYMENT-BASED VISA APPLICATIONS' in table_heading:
            rows = table.find_all('tr')
            for row in rows[1:]:  # Skip the header row
                cols = row.find_all('td')
                cols = [col.text.strip() for col in cols]
                employment_based_data.append({
                    'table_heading': table_heading,
                    'data': cols
                })

    return employment_based_data

def lambda_handler(event, context):
    base_url = 'https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin.html'
    response = requests.get(base_url)
    soup = BeautifulSoup(response.content, 'html.parser')

    # Find all links to visa bulletin pages
    links = soup.find_all('a', href=True)
    visa_bulletin_links = [link['href'] for link in links if '/visa-bulletin-for-' in link['href']]

    # Scrape data from each visa bulletin page
    for link in visa_bulletin_links:
        url = f"https://travel.state.gov{link}"
        data = scrape_visa_bulletin(url)
        for item in data:
            table_name = datetime.now().strftime('%Y-%m-%d') + '_' + item['table_heading'].replace(' ', '_')
            table.put_item(
                Item={
                    'table_name': table_name,
                    'data': item['data']
                }
            )

    return {
        'statusCode': 200,
        'body': 'Visa bulletin data scraped and stored in DynamoDB'
    }

I do see that it needs a DynamoDB table, so I added it after asking for the Terraform code - I will add everything for the application to app.tf for now, may split it out later, here is the table definition so far (foreshadowing... :) ):

resource "aws_dynamodb_table" "visa_bulletin_data" {
  name           = "VisaBulletinData"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "table_name"

  attribute {
    name = "table_name"
    type = "S" # String type
  }
}

Next up, I want to ensure we can deploy the application. There are may different ways to accomplish this, and since I'm already using Terraform for the rest of the infrastructure, I want to use it to zip up the file and deploy it as a Lambda function as well. The first attempt is close, but requires me to manually install the dependencies and zip up the code, so I try again, being more specific. While this attempt looks closer to what I need, I can see it won't work due to how requirements.txt is handled. I find this article and like the approach more:

Install the dependencies in requirements.txt
Zip these up
Deploy it as a Lambda layer

This means that my build will only update this layer if my dependencies in requirements.txt change. I also move handler.py into /src, add src/package and layer to my .gitignore. Now I need to make sure that requirements.txt have the correct imports - I've used python enough to know about venv to install dependencies, but haven't had to do it from scratch before. After poking around for a while, learn that the "easiest" way would be look at the import statements at the top of my code, and then call pip3 install <module> for each one, currently I have requests, boto3, and BeautifulSoup, and then run pip3 freeze > requirements.txt. This does highlight a problem in my current approach: I've been merrily setting up all the infrastructure with Terraform, but not run my code at all. Once I have the infrastructure creation working, I will then focus on making sure the code works. I end up with the following Terraform in app.tf:

resource "aws_dynamodb_table" "visa_bulletin_data" {
  name           = "VisaBulletinData"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "table_name"

  attribute {
    name = "table_name"
    type = "S" # String type
  }
}

# Install dependencies and create the Lambda layer package
resource "null_resource" "pip_install" {
  triggers = {
    shell_hash = "${sha256(file("${path.cwd}/src/requirements.txt"))}"
  }

  provisioner "local-exec" {
    command = <<EOF
        cd src
        echo "Create and activate venv"
        python3 -m venv package
        source package/bin/activate
        mkdir -p ${path.cwd}/layer/python
        echo "Install dependencies to ${path.cwd}/layer/python"
        pip3 install -r requirements.txt -t ${path.cwd}/layer/python
        deactivate
        cd ..
    EOF
  }
}

# Zip up the app to deploy as a layer
data "archive_file" "layer" {
  type        = "zip"
  source_dir  = "${path.cwd}/layer"
  output_path = "${path.cwd}/layer.zip"
  depends_on  = [null_resource.pip_install]
}

# Create the Lambda layer with the dependencies
resource "aws_lambda_layer_version" "layer" {
  layer_name          = "dependencies-layer"
  filename            = data.archive_file.layer.output_path
  source_code_hash    = data.archive_file.layer.output_base64sha256
  compatible_runtimes = ["python3.12", "python3.11"]
}

# Zip of the application code
data "archive_file" "app" {
  type        = "zip"
  source_dir  = "${path.cwd}/src"
  output_path = "${path.cwd}/app.zip"
}

# Define the Lambda function
resource "aws_lambda_function" "visa_bulletin_scraper" {
  function_name    = "visa-bulletin-scraper"
  handler          = "lambda_function.lambda_handler"
  runtime          = "python3.12"
  filename         = data.archive_file.app.output_path
  source_code_hash = data.archive_file.app.output_base64sha256
  role             = aws_iam_role.lambda_role.arn
  layers           = [aws_lambda_layer_version.layer.arn]

  environment {
    variables = {
      DYNAMODB_TABLE = aws_dynamodb_table.visa_bulletin_data.name
    }
  }
}

# Define the IAM role for the Lambda function
resource "aws_iam_role" "lambda_role" {
  name = "visa-bulletin-scraper-role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}

# Attach the necessary IAM policies to the role
resource "aws_iam_policy_attachment" "lambda_basic_execution" {
    name = "lambda_basic_execution"
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
  roles      = [aws_iam_role.lambda_role.name]
}

resource "aws_iam_policy_attachment" "dynamodb_access" {
  name       = "dynamodb_access"
  policy_arn = aws_iam_policy.dynamodb_access_policy.arn
  roles      = [aws_iam_role.lambda_role.name]
}

# Define the IAM policy for DynamoDB access
resource "aws_iam_policy" "dynamodb_access_policy" {
  name = "visa-bulletin-scraper-dynamodb-access"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem"
      ],
      "Resource": "${aws_dynamodb_table.visa_bulletin_data.arn}"
    }
  ]
}
EOF
}

Looking at this code, I do see a problem for later: the dynamodb:PutItem Action will only allow my Lambda to write to the table, but not read. I'm going to leave it here for now till we get there. On the first terraform plan, I ran into the following error, and had to run terraform init -upgrade to install the null and archive providers:

➜  us-visa-dates-checker git:(main) ✗ terraform plan 
╷
│ Error: Inconsistent dependency lock file
│ 
│ The following dependency selections recorded in the lock file are inconsistent with the current configuration:
│   - provider registry.terraform.io/hashicorp/archive: required by this configuration but no version is selected
│   - provider registry.terraform.io/hashicorp/null: required by this configuration but no version is selected
│ 
│ To update the locked dependency selections to match a changed configuration, run:
│   terraform init -upgrade

Running the code locally

I now have my Lambda function and dependencies deployed, but no idea yet if it works, so I look how I can run it locally and create src/local_test.py:

from handler import lambda_handler

class MockContext:
    def __init__(self):
        self.function_name = "mock_function_name"
        self.aws_request_id = "mock_aws_request_id"
        self.log_group_name = "mock_log_group_name"
        self.log_stream_name = "mock_log_stream_name"

    def get_remaining_time_in_millis(self):
        return 300000  # 5 minutes in milliseconds

mock_context = MockContext()

mock_event = {
    "key1": "value1",
    "key2": "value2",
    # Add any other relevant data for your event
}

result = lambda_handler(mock_event, mock_context)
print(result)

I can now run this by calling python3 local_test.py, and it will be able to access my AWS resources we exported the environment variable AWS_PROFILE. And it looks like it works since I don't see any errors:

(package) ➜  src git:(main) ✗ python3 local_test.py           
{'statusCode': 200, 'body': 'Visa bulletin data scraped and stored in DynamoDB'}

Time to add some outputs to my code so I can see what is happening - I'm not going to try to implement proper logging at this point, will leave that and all the other productionizing for a future date. I add in a print line in the scrape_visa_bulletin function:

def scrape_visa_bulletin(url):
    print("Processing url: ", url)

When I now run local_text.py, I can see that it is processing all the bulletins, but there is no data in my DynamoDB table:

python3 local_test.py
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-june-2024.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-july-2024.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-july-2024.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-june-2024.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-may-2024.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-april-2024.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-march-2024.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-february-2024.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-january-2024.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-december-2023.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-november-2023.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-october-2023.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2023/visa-bulletin-for-september-2023.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2023/visa-bulletin-for-august-2023.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2023/visa-bulletin-for-july-2023.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2023/visa-bulletin-for-june-2023.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2023/visa-bulletin-for-may-2023.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2023/visa-bulletin-for-april-2023.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2023/visa-bulletin-for-march-2023.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2023/visa-bulletin-for-february-2023.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2023/visa-bulletin-for-january-2023.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2023/visa-bulletin-for-december-2022.html
Processing url:  https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2023/visa-bulletin-for-november-2022.html

...
{'statusCode': 200, 'body': 'Visa bulletin data scraped and stored in DynamoDB'}

Looking at the page source for one of the bulletins, I can see that there aren't any <div> fields that match field-item even for the class as defined in this line in my app:

employment_based_tables = soup.find_all('div', {'class': 'field-item even'})

After trying again, and a few more times, I realise that I'm not able to clearly articulate exactly how I want to extract the data, so instead of continuing, I just take a somewhat brute-force approach and loop over the tables, rows, and cells to inspect the data. The code afterwards for scrape_visa_bulletin now looks like this (and it isn't storing the data in the DB yet):

def scrape_visa_bulletin(url):
    print("Processing url: ", url)

    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')

    employment_based_tables = soup.find_all('tbody')
    employment_based_data = []

    # Date pattern for the table cell dates
    date_pattern = r"(\d{2})([A-Z]{3})(\d{2})"

    # Extract the date from the URL
    bulletin_date_pattern = r'visa-bulletin-for-(\w+)-(\d+)\.html'
    match = re.search(bulletin_date_pattern, url)
    if match:
        month_name, year = match.groups()
        month_abbr = month_name[:3].lower()
        month_num = datetime.strptime(month_abbr, '%b').month
        date_obj = datetime(int(year), month_num, 1)
    else:
        date_obj = None

    employment_table_id = 0

    for table in employment_based_tables:
        rows = table.find_all('tr')
        countries = []
        # From 2022 till 2024 the number of rows differ
        if len(rows) < 9 or len(rows) > 12: 
            continue
        filing_type = 'Final Date' if employment_table_id == 0 else 'Filing Date'
        employment_table_id += 1
        for row_id, row in enumerate(rows):
            cells = row.find_all('td')
            for cell_id, cell in enumerate(cells):
                clean_cell = cell.text.replace("\n", "").replace("  ", " ").replace("- ", "-").strip()
                if row_id == 0:
                    if cell_id == 0:
                        table_heading = clean_cell
                        print("Table heading: ", table_heading)
                    else:
                        countries.append(clean_cell)
                else:
                    if cell_id == 0:
                        category_value = clean_cell
                    else:
                        match = re.match(date_pattern, clean_cell)
                        if match:
                            day = int(match.group(1))
                            month_str = match.group(2)
                            year = int(match.group(3)) + 2000 # Year is only last 2 digits

                            month = datetime.strptime(month_str, "%b").month
                            cell_date = datetime(year, month, day)
                        else:
                            cell_date = date_obj

                        try:
                            employment_based_data.append({
                                'filing_type': filing_type,
                                'country': countries[cell_id - 1],
                                'category': category_value,
                                'bulletin_date': date_obj,
                                'date': cell_date
                            })
                            print("Date: [", date_obj.strftime("%Y-%m-%d"), "], Filing Type: [", filing_type, "], Country: [", countries[cell_id - 1], "], Category: [", category_value, "], Value: [", cell_date.strftime("%Y-%m-%d"), "]")
                        except:
                            print("ERROR: Could not process the row. Row: ", row)


    return employment_based_data

It does however prompt me with some autocomplete suggestions that I use and the modify a bit more - below you can see what it looks like, the grayed out text is the suggestion:

Using the URL format of visa-bulletin-for-july-2024.html to determine the date of the bulletin, I extract that, and store it in the bulletin_date. The debug output from the print line after I add the data to employment_based_data looks mostly right to me at this point:

Table heading:  Employment-based
Date: [ 2023-05-01 ], Filing Type: [ Filing Date ], Country: [ All Chargeability Areas Except Those Listed ], Category: [ 1st ], Value: [ 2023-05-01 ]
Date: [ 2023-05-01 ], Filing Type: [ Filing Date ], Country: [ CHINA-mainland  born ], Category: [ 1st ], Value: [ 2022-06-01 ]
Date: [ 2023-05-01 ], Filing Type: [ Filing Date ], Country: [ INDIA ], Category: [ 1st ], Value: [ 2022-06-01 ]
Date: [ 2023-05-01 ], Filing Type: [ Filing Date ], Country: [ MEXICO ], Category: [ 1st ], Value: [ 2023-05-01 ]
Date: [ 2023-05-01 ], Filing Type: [ Filing Date ], Country: [ PHILIPPINES ], Category: [ 1st ], Value: [ 2023-05-01 ]
Date: [ 2023-05-01 ], Filing Type: [ Filing Date ], Country: [ All Chargeability Areas Except Those Listed ], Category: [ 2nd ], Value: [ 2022-12-01 ]
Date: [ 2023-05-01 ], Filing Type: [ Filing Date ], Country: [ CHINA-mainland  born ], Category: [ 2nd ], Value: [ 2019-07-08 ]
Date: [ 2023-05-01 ], Filing Type: [ Filing Date ], Country: [ INDIA ], Category: [ 2nd ], Value: [ 2012-05-01 ]
Date: [ 2023-05-01 ], Filing Type: [ Filing Date ], Country: [ MEXICO ], Category: [ 2nd ], Value: [ 2022-12-01 ]
Date: [ 2023-05-01 ], Filing Type: [ Filing Date ], Country: [ PHILIPPINES ], Category: [ 2nd ], Value: [ 2022-12-01 ]

... (many more rows)

Transforming the data

Now that I have the raw data, storing it in DynamoDB is the next step. The shape of the data is not what I want to store, currently it is an object per table cell. Ideally I want the data in the format:

data: [
  {
    filing_type: Final Action Date
    category: EB-3
    countries: [
      {
        country: "All Chargeability Areas Except Those Listed"
        history: [
          { bulletin_date: 2024-07-01, date: 2021-12-01},
          { bulletin_date: 2024-06-01, date: 2022-11-22},
          { bulletin_date: 2024-05-01, date: 2022-11-22},
          { bulletin_date: 2024-04-01, date: 2022-11-22},
          { bulletin_date: 2024-03-01, date: 2022-09-08},
        ]
      }
    ]
  }
]

The response looks ok, but I am by no means proficient in python so let's copy & paste and use it, here is what was added:

def transform_data(employment_based_data):
    # Assuming employment_based_data is your initial collection
    transformed_data = []

    # Group the data by filing_type and category
    grouped_data = defaultdict(lambda: defaultdict(list))
      for item in employment_based_data:
          filing_type = item['filing_type']
          category = item['category']
          country = item['country']
          bulletin_date = item['bulletin_date']
          date = item['date']
          grouped_data[filing_type][category].append({
              'country': country,
              'bulletin_date': bulletin_date,
              'date': date
          })

    # Transform the grouped data into the desired structure
      for filing_type, categories in grouped_data.items():
          for category, country_data in categories.items():
              countries = defaultdict(list)
              for item in country_data:
                  country = item['country']
                  bulletin_date = item['bulletin_date']
                  date = item['date']
                  countries[country].append({
                      'bulletin_date': bulletin_date,
                      'date': date
                  })

              transformed_data.append({
                  'filing_type': filing_type,
                  'category': category,
                  'countries': [
                      {
                          'country': country_name,
                          'history': history_data
                      }
                      for country_name, history_data in countries.items()
                  ]
              })

The last step is to add transformed_data = transform_data(data) to the lambda_handler. All good so far, except when I run the code, it returns None for transformed_data. After wrapping the code in try blocks, I still can't see the issue, and even with some more help, it stays None. I even try json-serializing the objects to better inspect them, which results in me learning about custom serialization in Python via:

# Custom serialization function for datetime objects
def datetime_serializer(obj):
    if isinstance(obj, datetime):
        return obj.strftime("%Y-%m-%d")
    raise TypeError(f"Type {type(obj)} not serializable")

...

print(json.dumps(transformed_data, default=datetime_serializer, indent=4))

With a bunch of print statements and json.dumps later, I still see the correct data in grouped_data and transformed_data (inside the transform_data method though). Then it hits me: if I'm calling a method and using the result of it, that method should probably have a return call in it... Right, let's pretend that didn't happen and I had return transformed_data in there the whole time... Ahem!

Storing the data

Since we started on this merry adventure, I didn't really think about the shape of the data and how I want to store it, I just went with the recommendation from Amazon Q Developer. Now that we have the data in the shape we need, let's see what changes are needed. I need to update the DynamoDB table, and then also add in the logic to store the data, and then add a method to retrieve the specific data set I'm interested in. The table definition now is:

resource "aws_dynamodb_table" "visa_bulletin_data" {
  name           = "VisaBulletinData"
  billing_mode   = "PROVISIONED"
  read_capacity  = 5
  write_capacity = 5
  hash_key       = "pk" # Partition key
  range_key      = "sk" # Sort key

  attribute {
    name = "pk"
    type = "S" # String type
  }

  attribute {
    name = "sk"
    type = "S" # String type
  }

  global_secondary_index {
    name            = "CountryIndex"
    hash_key        = "pk"
    range_key       = "sk"
    projection_type = "ALL"
  }
}

When I try to terraform apply this change, it returns the error write capacity must be > 0 when billing mode is PROVISIONED. After looking at the dynamodb_table resource definition, it appears you need to specify write_capacity and read_capacity for the global_secondary_index as well. With that fixed, I try to run the "final" version for today's efforts, but it errors with:

Traceback (most recent call last):
  File "/Users/cobusb/projects/terraform-samples/us-visa-dates-checker/src/local_test.py", line 21, in <module>
    result = lambda_handler(mock_event, mock_context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cobusb/projects/terraform-samples/us-visa-dates-checker/src/handler.py", line 203, in lambda_handler
    eb3_data = read_data()
               ^^^^^^^^^^^
  File "/Users/cobusb/projects/terraform-samples/us-visa-dates-checker/src/handler.py", line 175, in read_data
    KeyConditionExpression=Key('pk').eq(pk) & Key('sk').begins_with(sk_prefix)
                           ^^^
NameError: name 'Key' is not defined

Luckily my handing coding assistant is extremely polite, and instead of telling me "didn't you pay attention to the code I gave you", it tells me "The issue here is that the Key function is part of the boto3.dynamodb.conditions module, and you need to import it explicitly in your Python script.". Yes. This is me not paying attention to the output. After adding the import, the error is gone, but I do notice that after I changed the logic in lambda_handler it errors.

Changed from:

    for link in visa_bulletin_links:
        if '2022' in link or '2023' in link or '2024' in link:
            print("Processing link: ", link)
            url = f"https://travel.state.gov{link}"
            data = scrape_visa_bulletin(url)
            transformed_data = transform_data(data)

To:

    for link in visa_bulletin_links:
        if '2022' in link or '2023' in link or '2024' in link:
            print("Processing link: ", link)
            url = f"https://travel.state.gov{link}"
            data.append(scrape_visa_bulletin(url))

    transformed_data = transform_data(data)
    store_data(transformed_data)

There is an error "Unable to group the data, error: list indices must be integers or slices, not str", and I remember seeing this as part of a previous response:

Note: This code assumes that the employment_based_data collection contains unique combinations of filing_type, country, category, and bulletin_date. If there are duplicate entries, you may need to modify the code to handle them appropriately (e.g., by keeping only the latest entry or aggregating the data in some way).

Which makes sense when I look at how I am grouping the data:

        for item in employment_based_data:
            filing_type = item['filing_type']
            category = item['category']
            country = item['country']
            bulletin_date = item['bulletin_date']
            date = item['date']
            grouped_data[filing_type][category].append({
                'country': country,
                'bulletin_date': bulletin_date,
                'date': date
            })

After checking again and again I find the error:

data.append(scrape_visa_bulletin(url))

The returned object from scrape_visa_bulletin is a list, not a single object, so using .append() causes the issue as it will add that whole list as a single object in my new list. Instead, I need to use .extend(). While debugging this and looking at how the data is stored, I also realise that we don't need the transform_data at all, if you look at what is being stored via the table.put_item, the flat list of objects we extract from the web pages would work:

pk = f"{filing_type}#{category}"
sk = f"{country}#{bulletin_date}"
table.put_item(
    Item={
            'pk': pk,
            'sk': sk,
            'filing_type': filing_type,
            'category': category,
            'country': country,
            'bulletin_date': bulletin_date,
            'date': date
        }
    )

So all I would need to store the data is:

for item in data:
    filing_type = item['filing_type']
    category = item['category']
    country = item['country']
    bulletin_date = item['bulletin_date']
    date = item['date']
    pk = f"{filing_type}#{category}"
    sk = f"{country}"
    table.put_item(
        Item={
                'pk': pk,
                'sk': sk,
                'filing_type': filing_type,
                'category': category,
                'country': country,
                'bulletin_date': bulletin_date,
                'date': date
            }
        )

Realisation #3: so far, I have only approached this as a single run, if I were to run the code a 2nd time, it would add the full history a 2nd time. So let's fix this by adding another DynamoDB table called ProcessedURLs and updating the code by adding processed_urls_table = dynamodb.Table('ProcessedURLs') at the start of handler.py under our existing reference to VisaBulletinData, and then update lambda_handler to:

def lambda_handler(event, context):
  ... #skipped for brevity

    # Scrape data from each visa bulletin page
    for link in visa_bulletin_links:
        if '2022' in link or '2023' in link or '2024' in link:
            # Check if the URL has been processed
            response = processed_urls_table.get_item(Key={'url': link})
            if 'Item' in response:
                print(f"Skipping URL: {link} (already processed)")
                continue

            # Process the URL
            print(f"Processing URL: {link}")
            url = f"https://travel.state.gov{link}"
            data.extend(scrape_visa_bulletin(url))

            # Store the processed URL in DynamoDB
            processed_urls_table.put_item(Item={'url': link})

And create the table via Terraform:

resource "aws_dynamodb_table" "processed_urls" {
  name           = "ProcessedURLs"
  billing_mode   = "PROVISIONED"
  read_capacity  = 5
  write_capacity = 5
  hash_key       = "url"

  attribute {
    name = "url"
    type = "S"
  }
}

Now the only step left is to update the IAM policy to allow read and write access to this new table. Remember earlier in this article where I mentioned we will need to also add read permission to the VisaBulletinData table? Let's combine that into a single request! At first glance, I'm confused why it created 2 statements instead of adding the 2 resources as an array in the Resources: section. Looking a 2nd time, I spot the difference in the statement Action sections:

resource "aws_iam_policy" "dynamodb_access_policy" {
  name = "visa-bulletin-scraper-dynamodb-access"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem"
      ],
      "Resource": "${aws_dynamodb_table.processed_urls.arn}"
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:Query"
      ],
      "Resource": "${aws_dynamodb_table.visa_bulletin_data.arn}"
    }
  ]
}
EOF
}

When we call the VisaBulletinData table to read from it, we are not using a .get() call, but a .query() call. I would have missed that 😳 ... Nice to see that someone else is at least paying attention.

Finishing up for now

Once all these changes were in place, I ran the app again locally to process all the bulletins for 2022 - 2024. While I was building so far, I had limited it to only the most recent one (July 2024). The first few pages it processed scrolled past very quickly, and the it just hung in my terminal. No error. And I realised what the issue was:

The way I'm processing and storing the data results in more calls than the 5 read / write units I had provisioned for my DynamoDB table. I called it a day, and when I came back this morning, the process had finished without breaking. At least without any exceptions, my table design needs to be completely redone though. From the code further up, this is how I was storing my data, see if you can spot the issue:

table.put_item(
pk = f"{filing_type}#{category}"
sk = f"{country}"
table.put_item(
    Item={
            'pk': pk,
            'sk': sk,
            'filing_type': filing_type,
            'category': category,
            'country': country,
            'bulletin_date': bulletin_date,
            'date': date
        }
    )

I'll give you a hint: using the above, my primary key (pk) for the data I'm interested in would be Final Date#3rd, and it can only store 1 entry per country with 1 bulletin_date and date stored:

At this point, my todo list has grown to:

Fix the data structure to store all the data
Test the Lambda function to see if it works
Add parameters to the function to be able to only select the specific set of dates you want
Decide how to expose the Lambda function - either using AWS API Gateway as I have before, or setting up an invocation URL
Set up a CI/CD pipeline to deploy it
Add in tests

This would make this article way too long, so my plan now is to update the code to take the data I've retrieved, pull the subset I need, and not try to store it in DynamoDB. Then I can tackle that and the other items in follow up pieces.

Changing the code to use the collection of data I already have results in the following:

def read_data_locally(data, filing_type = 'Final Date', category = '3rd', country = 'All Chargeability Areas Except Those Listed'):
    # Filter the data based on filing_type, category, and country
    filtered_data = [entry for entry in data
                    if entry['filing_type'] == filing_type
                    and entry['category'] == category
                    and entry['country'] == country]

    # Sort the filtered data in descending order by bulletin_date
    sorted_data = sorted(filtered_data, key=itemgetter('bulletin_date'), reverse=True)

    # Print the sorted data
    for entry in sorted_data:
        print(f"Bulletin Date: {entry['bulletin_date']}, Date: {entry['date']}")

    return sorted_data

I'm setting some defaults for now as I'm very keen to see if it works. And it does! Yay!!!! Here is the output from that print statement:

Bulletin Date: 2024-07-01, Date: 2021-12-01
Bulletin Date: 2024-06-01, Date: 2022-11-22
Bulletin Date: 2024-05-01, Date: 2022-11-22
Bulletin Date: 2024-04-01, Date: 2022-11-22
Bulletin Date: 2024-03-01, Date: 2022-09-08
...
Bulletin Date: 2022-02-01, Date: 2022-02-01
Bulletin Date: 2022-01-01, Date: 2022-01-01
Bulletin Date: 2021-12-01, Date: 2021-12-01
Bulletin Date: 2021-11-01, Date: 2021-11-01
Bulletin Date: 2021-10-01, Date: 2021-10-01

I'm going to pause at this point, and continue in the next article.

What I Learnt

This was actually a lot of fun, I love trying out something I have no idea how to do, making mistakes along the way, and learning. One of my favourite interview questions, after asking what projects a person has worked on, is to ask them "How did it break / go wrong, and how did you go about fixing it". I've found that if a person can't answer that, they were either not that involved, or didn't really build it. But that is a post for another time.

From this little adventure, I would summarise what I learnt into the following sections.

Using Amazon Q Developer

I've been using Amazon Q Developer for a few months now while building an internal reporting tool in C# (my primary coding language at the moment), and this was definitely a very different experience. I've used Python quite a few times over the last 19 years I've been building full time, but never from scratch, or with AWS services other than single-purpose, one-off Lambda functions. And never deployed it with Terraform. Getting requirements.txt (correctly) populated took me longer than I expected. It was definitely a lot faster and easier with Amazon Q Developer than with my old approach of googling things piecemeal. If I had to summarise it, the conversational aspect along with not needing to sit and think exactly what keywords to add, makes a huge difference. Usually I would open multiple of the search results that are kind-of-but-not-quite what I'm looking for, spend the first 10 seconds to assess the source by scrolling the page, looking at when it was published, and if it feels trust-worthy. Then I would need to string together multiple parts from different sources unless I was really lucky and found exactly what I was looking for.

As for not even having to switch to my browser, that is probably the biggest win. I've found myself even using it for non-coding questions, e.g. the difference between "learnt" and "learned". While it makes me much more productive, that doesn't mean I can copy and paste everything. Large language models (LLMs) are still improving every day, and while I didn't run into any hallucinations in this exercise, I did have to evaluate each response for accuracy, and if it was the way I should be solving the problem. Just look at the part where I built that whole transform_data method only to realise I didn't need it all. All my experience over the last 32 years (GET OFF MY LAWN YA KIDS, I ENJOYED CHANGING DOS 6.22 MENUS!!!) is still very relevant.

One pain point I had was when I accidentally closed my IDE, and with that, lost the conversation thread I had going. I'll definitely be passing on that feedback to our service teams, and also seeing if it doesn't keep a local log somewhere. If anyone knows, please drop that in a comment below please.

Approach and Tech Choices

This was probably the biggest one for me. My background the last 10+ years has been very / mostly DevOps focussed, so I approached this by starting with the infrastructure automation, how I would use a Lambda function to store my data in DynamoDB. Instead, I should have started with just the Python code to retrieve and process the data, then turn that into a Lambda function, deploy it, etc etc.

If I wanted to just get this done, I would have used C#, but I wanted to play around and spend some time learning. Amazon Q Developer suggested using Python, so I went with it, and it definitely stretched my brain to use a dynamic language. Looking back at the first prompt, I thought I would be using Bedrock for this, but I didn't really think it through, and even though it was included it in the list (and even called it a "core service you'll use to build your generative AI application"), I didn't end up using it at all. At least, not yet ;)

Technical

The most interesting part for me was the 💡 moment regarding Lambda Layers. While I have known about them for a while, as well as how they worked, it had yet to fully "click" in my brain. Using a layer for all the dependencies that pip installs makes complete sense to me, and this is a really good way to split deploying changes between the application and the dependencies. I can see some interesting ways how you could use this test out upgrading the dependency version and then testing them before you roll it out to production...

While doing the Terraform code, I quickly looked at the difference between path.cdw and path.module, and I will need to read up more. Most of the time, I've seen path.module used, but it looks like path.cwd should be used instead.

Final Code

As this is going to turn into a series of articles, I've tagged the code at this point in time, you can have a look at the source for what was covered in this article and is "working", I'll include instructions in the README.md on how to run it locally.

Source code for this article.

Conclusion

To wrap things up, I would say this was a (mostly) successful attempt to solve what I trying to do, with a number of detours. I'm going to start working on the next part later today, I really want to get to a point where I can just call an URL to get the information instead of finding the pinned tab in my browser, and refreshing it daily to see if there are any updates. As the next articles come out, I'll add links to them below.

And yes, I am 100% aware that I could just have copied the dates into a spreadsheet by hand and slapped on a graph in about 1h, but what fun would that be? I'll leave you with two of my favourite pieces of advice from xkcd.

Prompts used in this article

Prompt 1

I want to build an app using Amazon Bedrock that can pull in data from a public website using Terraform to build the infrastructure, can you start by telling me if this is possible, and which services I will need to use?

Response:

Yes, it is possible to build an application using Amazon Bedrock that can pull data from a public website, and you can use Terraform to provision the necessary infrastructure. Here's an overview of the services you might need to use:

Amazon Bedrock: This is the core service you'll use to build your generative AI application. With Bedrock, you can choose from various foundation models (FMs) like Anthropic's Claude, AI21 Labs' Jurassic-2, and Amazon's Titan models. You can also create Knowledge Bases and Agents to integrate external data sources and execute complex tasks. [1]
AWS Lambda: You can use AWS Lambda functions to fetch data from public websites and integrate with Amazon Bedrock. Lambda functions can be triggered by various events, such as API Gateway requests or scheduled events.
Amazon API Gateway: If you want to expose your application as an API, you can use API Gateway to create, publish, and manage APIs that can invoke your Lambda functions.
Amazon S3: You might need to store the data fetched from public websites in Amazon S3 buckets, which can then be used by your Bedrock Knowledge Bases or Agents.
AWS IAM: You'll need to manage permissions and roles using AWS Identity and Access Management (IAM) to allow your Lambda functions and Bedrock resources to access other AWS services securely.
Terraform: You can use Terraform to define and provision the necessary AWS resources, such as Lambda functions, API Gateway, S3 buckets, IAM roles, and Bedrock resources, in a declarative and reproducible manner.

To get started, you can create a Terraform configuration that provisions the required resources, such as Lambda functions to fetch data from public websites, S3 buckets to store the data, and Bedrock resources like Knowledge Bases and Agents to integrate the data and build your application logic.

You can also explore the AWS Bedrock Studio, a web-based development environment that simplifies the process of building generative AI applications using Bedrock's capabilities.

Sources:

Generative AI App Development - Amazon Bedrock Studio - AWS

This is mostly ok, except I prefer not to create the infrastructure to store the statefile by hand, so I usually have _bootstrap.tf that contains the details of my base infrastructure for Terraform, create the S3 bucket, DynamoDB locking table, and KSM key with that, and then I add in the backend config to use those resources.

Prompt 2

Thanks, let's start by getting the Terraform set up storing the statefile in an S3 bucket

Response:

Sure, let's start by setting up Terraform to store the state file in an Amazon S3 bucket. This is a recommended practice for managing Terraform state in a team or CI/CD environment, as it provides a centralized and durable location for the state file. [1]

Here are the steps to set up an S3 bucket for storing the Terraform state file: [2]

Create an S3 bucket: First, you'll need to create an S3 bucket to store the state file. You can do this using the AWS Management Console, AWS CLI, or Terraform itself. Make sure to choose a unique bucket name and configure the appropriate access controls and versioning settings.

Configure the Terraform backend: In your Terraform configuration, you'll need to specify the S3 backend for storing the state file. Create a backend.tf file with the following content:

terraform {
  backend "s3" {
    bucket         = "XXXXXXXXXXXXXXXX" # Replace with your S3 bucket name
    key            = "path/to/state/file.tfstate" # Specify the path and name of the state file
    region         = "your-aws-region" # Replace with your AWS region
    dynamodb_table = "your-dynamodb-table-name" # Optional, for state locking
  }
}

Replace the placeholders with your actual S3 bucket name, the desired path and name for the state file, your AWS region, and an optional DynamoDB table name for state locking (recommended for team environments).

Initialize the backend: Run terraform init to initialize the Terraform working directory and configure the S3 backend.
Configure AWS credentials: Terraform needs access to your AWS credentials to interact with the S3 bucket and other AWS resources. You can configure your AWS credentials using environment variables, shared credentials file, or other methods supported by the AWS provider.
Apply the configuration: Run terraform apply to create the S3 bucket (if it doesn't exist) and store the initial state file in the bucket.

After completing these steps, Terraform will store the state file in the specified S3 bucket and use it for subsequent operations. This ensures that your Terraform state is centralized, durable, and accessible to your team or CI/CD pipeline.

Remember to follow best practices for securing your S3 bucket and state file, such as enabling versioning, configuring appropriate bucket policies, and using server-side encryption.

Sources:

Prompt 3

Thanks, can you show me how I would write a webpage scraper with Lambda to scrape all the pages linked from https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin.html with the format https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-july-2024.html - they should be linked from the first page listed. Each page has multiple sections and tables, I would like to only store the rows in the tables listed under the heading "A. FINAL ACTION DATES FOR EMPLOYMENT-BASED PREFERENCE CASES" and "B. DATES FOR FILING OF EMPLOYMENT-BASED VISA APPLICATIONS". Can you show me how to create a Lambda function to do this and store the data in DynamoDB?

Response:

Sure, I can help you with that. Here's an example of how you can create an AWS Lambda function in Python to scrape the pages you mentioned and store the relevant data in DynamoDB.

First, let's install the required Python libraries:

import requests
from bs4 import BeautifulSoup
import boto3
from datetime import datetime

# Initialize DynamoDB client
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('VisaBulletinData')

Next, we'll define a function to scrape the data from a given URL:

def scrape_visa_bulletin(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')

    # Find the tables with the relevant data
    employment_based_tables = soup.find_all('div', {'class': 'field-item even'})

    # Extract data from the tables
    employment_based_data = []
    for table in employment_based_tables:
        table_heading = table.find_previous('h3').text.strip()
        if 'FINAL ACTION DATES' in table_heading or 'DATES FOR FILING' in table_heading:
            rows = table.find_all('tr')
            for row in rows[1:]:  # Skip the header row
                cols = row.find_all('td')
                cols = [col.text.strip() for col in cols]
                employment_based_data.append({
                    'table_heading': table_heading,
                    'data': cols
                })

    return employment_based_data

This function takes a URL as input, sends a GET request to fetch the page content, and then uses BeautifulSoup to parse the HTML. It finds the relevant tables by looking for <div> elements with the class field-item even. For each table, it checks if the heading matches the desired sections ("FINAL ACTION DATES" or "DATES FOR FILING"). If so, it extracts the data from the table rows, skipping the header row, and appends it to a list.

Now, let's define the Lambda handler function:

def lambda_handler(event, context):
    base_url = 'https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin.html'
    response = requests.get(base_url)
    soup = BeautifulSoup(response.content, 'html.parser')

    # Find all links to visa bulletin pages
    links = soup.find_all('a', href=True)
    visa_bulletin_links = [link['href'] for link in links if '/visa-bulletin-for-' in link['href']]

    # Scrape data from each visa bulletin page
    for link in visa_bulletin_links:
        url = f"https://travel.state.gov{link}"
        data = scrape_visa_bulletin(url)
        for item in data:
            table_name = datetime.now().strftime('%Y-%m-%d') + '_' + item['table_heading'].replace(' ', '_')
            table.put_item(
                Item={
                    'table_name': table_name,
                    'data': item['data']
                }
            )

    return {
        'statusCode': 200,
        'body': 'Visa bulletin data scraped and stored in DynamoDB'
    }

In the lambda_handler function, we first fetch the base URL (https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin.html) and find all the links to visa bulletin pages using BeautifulSoup. Then, for each visa bulletin page link, we call the scrape_visa_bulletin function to extract the relevant data.

For each item of data extracted, we construct a table name based on the current date and the table heading, and store the data in DynamoDB using the put_item method of the DynamoDB table resource.

Finally, the function returns a success message with a status code of 200.

To deploy this Lambda function, you'll need to create a DynamoDB table named VisaBulletinData with a partition key named table_name (string type). You can do this using the AWS Management Console, AWS CLI, or Terraform.

You can then create the Lambda function using Terraform, specifying the Python code as the function source code, and configuring the necessary IAM permissions for the Lambda function to access DynamoDB.

Note: This code assumes that you have configured your AWS credentials correctly, either by setting environment variables or using a shared credentials file. Additionally, make sure to handle any potential errors or exceptions that may occur during the scraping process.

There is a change needed here, it should use the full heading I supplied as there are other sections starting with FINAL ACTION DATES and DATES FOR FILING. I do like how Q extracted '/visa-bulletin-for-' for the individual bulletin links.

Prompt 4

Can you please show me the Terraform code to create the dynamodb table with the partition key mentioned above?

Response:

Sure, here's an example of how you can create the DynamoDB table using Terraform: [1]

resource "aws_dynamodb_table" "visa_bulletin_data" {
  name           = "VisaBulletinData"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "table_name"

  attribute {
    name = "table_name"
    type = "S" # String type
  }
}

In this Terraform code:

resource "aws_dynamodb_table" "visa_bulletin_data" { ... } defines a new DynamoDB table resource.
name = "VisaBulletinData" sets the name of the DynamoDB table.
billing_mode = "PAY_PER_REQUEST" specifies that the table should use the pay-per-request billing mode, where you pay for the read and write throughput consumed by your application.
hash_key = "table_name" defines the partition key (hash key) for the table as table_name.
attribute { ... } block defines the attribute for the partition key, specifying its name (table_name) and type (S for String).

After defining this resource in your Terraform configuration, you can run terraform apply to create the DynamoDB table with the specified settings.

Make sure to include the required AWS provider configuration in your Terraform code, specifying the AWS region and your AWS credentials (either through environment variables or a shared credentials file).

provider "aws" {
  region = "your-aws-region"
  # Configure AWS credentials if not using environment variables or shared credentials file
}

Once the DynamoDB table is created, you can use it in your Lambda function to store the scraped data as shown in the previous example.

Note: If you need to make any changes to the DynamoDB table later, such as adding a sort key or configuring additional settings, you can modify the Terraform code and run terraform apply again to update the table.

Sources:

Step 1: Create a table - Amazon DynamoDB

Since we already defined the AWS provider for Terraform, we can discard that part.

Prompt 5

Thanks, can you now show me how I would deploy this Python code to Lambda using Terraform?

Response:

Sure, here's an example of how you can deploy the Python code to AWS Lambda using Terraform: [1]

# Define the Lambda function
resource "aws_lambda_function" "visa_bulletin_scraper" {
  filename         = "lambda_function.zip"
  function_name    = "visa-bulletin-scraper"
  role             = aws_iam_role.lambda_role.arn
  handler          = "lambda_function.lambda_handler"
  runtime          = "python3.9"
  source_code_hash = filebase64sha256("lambda_function.zip")

  environment {
    variables = {
      DYNAMODB_TABLE = aws_dynamodb_table.visa_bulletin_data.name
    }
  }
}

# Define the IAM role for the Lambda function
resource "aws_iam_role" "lambda_role" {
  name = "visa-bulletin-scraper-role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}

# Attach the necessary IAM policies to the role
resource "aws_iam_policy_attachment" "lambda_basic_execution" {
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
  roles      = [aws_iam_role.lambda_role.name]
}

resource "aws_iam_policy_attachment" "dynamodb_access" {
  policy_arn = aws_iam_policy.dynamodb_access_policy.arn
  roles      = [aws_iam_role.lambda_role.name]
}

# Define the IAM policy for DynamoDB access
resource "aws_iam_policy" "dynamodb_access_policy" {
  name = "visa-bulletin-scraper-dynamodb-access"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem"
      ],
      "Resource": "${aws_dynamodb_table.visa_bulletin_data.arn}"
    }
  ]
}
EOF
}

Here's what this Terraform code does:

1. Define the Lambda function: The aws_lambda_function resource defines the Lambda function. It specifies the filename of the ZIP archive containing the Python code (lambda_function.zip), the function name, the IAM role for the function, the handler function (lambda_function.lambda_handler), and the Python runtime version (python3.9). The source_code_hash attribute is used to trigger updates when the source code changes. The environment block sets the DYNAMODB_TABLE environment variable with the name of the DynamoDB table.

2. Define the IAM role: The aws_iam_role resource defines the IAM role that the Lambda function will assume. The assume_role_policy specifies that the role can be assumed by the Lambda service.

3. Attach IAM policies: The aws_iam_policy_attachment resources attach the necessary IAM policies to the Lambda role. The AWSLambdaBasicExecutionRole policy provides basic permissions for Lambda functions, and the dynamodb_access_policy (defined below) grants access to the DynamoDB table.

4. Define the DynamoDB access policy: The aws_iam_policy resource defines an IAM policy that grants the PutItem permission on the DynamoDB table. This policy is attached to the Lambda role in the previous step.

Before applying this Terraform configuration, make sure to:

Create a ZIP archive (lambda_function.zip) containing your Python code (lambda_function.py).
Update the filename attribute in the aws_lambda_function resource with the correct path to the ZIP archive.
Replace the runtime attribute with the appropriate Python version if you're using a different version.
Configure the AWS provider with your AWS credentials and region.

After applying the Terraform configuration, Terraform will create the Lambda function, IAM role, and IAM policies. The Lambda function will have the necessary permissions to write data to the DynamoDB table.

Note: If you need to update the Lambda function code, you'll need to create a new ZIP archive and update the source_code_hash attribute in the Terraform configuration. Then, run terraform apply again to update the Lambda function.

Sources:

AWS Lambda adds support for Python 3.10

This is close, but not quite there, I don't want to add a manual step to zip up the Python app. I'm also assuming I need to run a command to install the dependencies, or store them somewhere. I also notice that the IAM permission is only for writing to the table, not reading, but that is fine for now, we haven't gotten to the part where we need to access the data. Let's follow-up with another question on how to avoid the manual zipping of the file.

Prompt 6

Is there a way to handle the zipping of the file and installing any dependencies with Terraform?

Response:

Yes, Terraform provides a way to handle the zipping of the Lambda function code and installing dependencies without requiring manual steps. You can use the archive_file data source and the null_resource resource with local-exec provisioners to achieve this. [1]

Here's an example of how you can modify the Terraform code to handle the zipping and dependencies: [2]

# Create a ZIP archive of the Lambda function code
data "archive_file" "lambda_zip" {
  type        = "zip"
  source_dir  = "${path.module}/lambda_code/" # Directory containing the Lambda function code
  output_path = "${path.module}/lambda_function.zip"
}

# Install dependencies and create the Lambda package
resource "null_resource" "lambda_package" {
  triggers = {
    source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  }

  provisioner "local-exec" {
    command = <<EOF
      python3 -m venv package
      source package/bin/activate
      pip install -r ${path.module}/lambda_code/requirements.txt -t package/python
      deactivate
      cd package/python
      zip -r9 ${path.module}/lambda_function.zip .
      cd ../..
      rm -rf package
    EOF
  }
}

# Define the Lambda function
resource "aws_lambda_function" "visa_bulletin_scraper" {
  filename         = data.archive_file.lambda_zip.output_path
  function_name    = "visa-bulletin-scraper"
  role             = aws_iam_role.lambda_role.arn
  handler          = "lambda_function.lambda_handler"
  runtime          = "python3.9"
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256

  environment {
    variables = {
      DYNAMODB_TABLE = aws_dynamodb_table.visa_bulletin_data.name
    }
  }

  depends_on = [null_resource.lambda_package]
}

# ... (IAM role and policy definitions remain the same)

Here's what the new code does:

The data "archive_file" "lambda_zip" block creates a ZIP archive of the Lambda function code located in the lambda_code/ directory.
The resource "null_resource" "lambda_package" block is used to install dependencies and create the final Lambda package. The triggers block ensures that this resource is re-created whenever the source code changes.
- The local-exec provisioner runs a series of commands:
  - Create a Python virtual environment (venv).
  - Activate the virtual environment and install the dependencies listed in the requirements.txt file into the package/python directory.
  - Deactivate the virtual environment.
  - Change to the package/python directory and create a new ZIP archive containing the installed dependencies and the Lambda function code.
  - Remove the temporary package directory.
The aws_lambda_function resource is updated to use the filename and source_code_hash values from the archive_file data source. The depends_on attribute ensures that the null_resource is executed before creating or updating the Lambda function.

With this setup, Terraform will automatically handle the zipping of the Lambda function code and the installation of dependencies whenever you run terraform apply. The final ZIP archive will be created in the same directory as your Terraform configuration.

Make sure to replace ${path.module}/lambda_code/ with the actual path to your Lambda function code directory, and ${path.module}/lambda_code/requirements.txt with the path to your Python requirements file (if you have one).

Note: The null_resource with local-exec provisioners should be used as a last resort solution, as it can introduce potential issues with remote execution environments like Terraform Cloud. If possible, consider using a more robust packaging solution like AWS Lambda Layers or Docker containers for managing dependencies.

Sources:

Deploying a simple Python script and it’s dependencies as AWS Lambda Function with only Terraform | by Maxim Pisarenko | Medium

The mention of ${path.module} makes me think this code is from a module, so I will need to update it. Looking at the steps, I don't think this will work. The resource "null_resource" "lambda_package" block uses the data.archive_file.lambda_zip.output_base64sha256 as a trigger to run the commands in provisioner "local-exec", which makes sense as it is a way to figure out if it should trigger a new build based on if anything has changed. Further down it doesn't use the zip created with this local-exec though, just the original with just the source code, so will need to update this.

Prompt 7

How would I add the dependencies for my python app to requirements.txt if I only have the .py file with imports at the top?

Response:

If you only have the Python file (e.g., lambda_function.py) with the imports at the top, you can generate the requirements.txt file automatically using the pipreqs tool. Here are the steps: [1]

Install pipreqs using pip:
```
pip install pipreqs
```
Navigate to the directory containing your Python file (lambda_function.py).
Run the pipreqs command with the path to your Python file:
```
pipreqs /path/to/lambda_function.py
```
This will generate a requirements.txt file in the same directory, containing the list of dependencies and their versions based on the imports in your Python file. [2]

Alternatively, you can use the following command to generate the requirements.txt file in a specific directory:
```
pipreqs /path/to/lambda_function.py --force --savepath /path/to/requirements/directory
```
This command will create (or overwrite) the requirements.txt file in the specified /path/to/requirements/directory.
Review the generatedrequirements.txt file and make any necessary adjustments or additions. [3]

In your Terraform configuration, update the local-exec provisioner in the null_resource block to use the correct path to the requirements.txt file:

provisioner "local-exec" {
  command = <<EOF
    python3 -m venv package
    source package/bin/activate
    pip install -r /path/to/requirements/directory/requirements.txt -t package/python
    deactivate
    cd package/python
    zip -r9 ${path.module}/lambda_function.zip .
    cd ../..
    rm -rf package
  EOF
}

Replace /path/to/requirements/directory/requirements.txt with the actual path to your requirements.txt file.

After making these changes, when you run terraform apply, Terraform will use the requirements.txt file to install the necessary dependencies for your Python application.

Note that the pipreqs tool analyzes the import statements in your Python file to generate the requirements.txt file. It may not always be 100% accurate, especially for complex projects with nested dependencies or non-standard import patterns. In such cases, you may need to manually review and update the requirements.txt file.

Sources:

Prompt 8

How would I call the lambda_handler function in my code while doing local development?

Response:

To call the lambda_handler function in your code during local development, you can create a separate Python script that imports and invokes the lambda_handler function with the appropriate event and context objects. [1]

Here's an example of how you can do this:

Create a new Python file, e.g., local_test.py, in the same directory as your lambda_function.py file.
In local_test.py, import the lambda_handler function from your lambda_function.py file:
```
from lambda_function import lambda_handler
```

Define a mock event object and a mock context object. The event object should mimic the structure of the event that your Lambda function expects to receive when invoked. The context object can be a simple dictionary or a custom class that implements the required methods and attributes.

Here's an example of a mock event object:

mock_event = {
    "key1": "value1",
    "key2": "value2",
    # Add any other relevant data for your event
}

And here's an example of a mock context object:

class MockContext:
    def __init__(self):
        self.function_name = "mock_function_name"
        self.aws_request_id = "mock_aws_request_id"
        self.log_group_name = "mock_log_group_name"
        self.log_stream_name = "mock_log_stream_name"

    def get_remaining_time_in_millis(self):
        return 300000  # 5 minutes in milliseconds

mock_context = MockContext()

In local_test.py, call the lambda_handler function with the mock event and context objects:
```
result = lambda_handler(mock_event, mock_context)
print(result)
```
Run the local_test.py script:
```
python local_test.py
```

This will execute the lambda_handler function with the mock event and context objects, and print the result returned by the function.

You can modify the mock event and context objects to match the expected input for your Lambda function, and add any additional logic or assertions in local_test.py to verify the behavior of your lambda_handler function.

By separating the local testing code from the actual Lambda function code, you can easily run and debug your Lambda function locally without the need for deploying it to AWS or invoking it through AWS Lambda. [2]

Note: When running locally, you may need to mock or simulate any external dependencies or services that your Lambda function interacts with, such as DynamoDB, S3, or other AWS services. You can use libraries like moto or placebo to mock AWS services during local development and testing.

Sources:

Define Lambda function handler in Python - AWS Lambda

Prompt 9

For this prompt, I had unfortunately closed my VSCode instance, and had to restart the conversation.

For the html in src/sample.html, can you show me how I would extract the table values using Python, BeautifulSoup, and store each cell in DynamoDB? For the DynamoDB table, please show me the terraform code to create it. The html in src/sample.html is a local copy of a file, please assume that the URL will be passed as a parameter to the function that extracts the data.

Response:

To extract the table values from the HTML file using Python and BeautifulSoup, and store each cell in DynamoDB, you can follow these steps:

Parse the HTML file using BeautifulSoup

import requests
from bs4 import BeautifulSoup

def extract_table_data(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')
    # Find the table element
    table = soup.find('table')
    # Extract data from table rows and cells
    data = []
    for row in table.find_all('tr'):
        row_data = []
        for cell in row.find_all(['th', 'td']):
            row_data.append(cell.get_text().strip())
        data.append(row_data)
    return data

Store the data in DynamoDB using Boto3 [1]

import os
import boto3

def create_dynamodb_table(table_name):
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.create_table(
        TableName=table_name,
        KeySchema=[
            {
                'AttributeName': 'row_index',
                'KeyType': 'HASH'
            },
            {
                'AttributeName': 'col_index',
                'KeyType': 'RANGE'
            }
        ],
        AttributeDefinitions=[
            {
                'AttributeName': 'row_index',
                'AttributeType': 'N'
            },
            {
                'AttributeName': 'col_index',
                'AttributeType': 'N'
            }
        ],
        ProvisionedThroughput={
            'ReadCapacityUnits': 5,
            'WriteCapacityUnits': 5
        }
    )
    return table

def store_data_in_dynamodb(table, data):
    for row_index, row in enumerate(data):
        for col_index, value in enumerate(row):
            table.put_item(
                Item={
                    'row_index': row_index,
                    'col_index': col_index,
                    'value': value
                }
            )

if __name__ == '__main__':
    url = os.environ.get('URL')
    table_name = 'table_data'
    data = extract_table_data(url)
    table = create_dynamodb_table(table_name)
    store_data_in_dynamodb(table, data)

This code assumes that you have the necessary AWS credentials configured in your environment. It creates a DynamoDB table with a composite primary key consisting of row_index (hash key) and col_index (range key). The extract_table_data function retrieves the HTML content from the provided URL, parses the table, and returns a list of lists containing the table data. The store_data_in_dynamodb function iterates over the table data and stores each cell value in DynamoDB using the put_item method. [2]

Terraform code to create the DynamoDB table [3]

resource "aws_dynamodb_table" "table_data" {
  name           = "table_data"
  billing_mode   = "PROVISIONED"
  read_capacity  = 5
  write_capacity = 5
  hash_key       = "row_index"
  range_key      = "col_index"

  attribute {
    name = "row_index"
    type = "N"
  }

  attribute {
    name = "col_index"
    type = "N"
  }
}

This Terraform code creates a DynamoDB table named table_data with a composite primary key consisting of row_index (hash key) and col_index (range key). The billing_mode is set to PROVISIONED, and the read_capacity and write_capacity are both set to 5. You can adjust these values based on your requirements.

Note: Make sure to replace os.environ.get('URL') with the actual URL or a method to obtain the URL dynamically. Additionally, ensure that you have the necessary AWS credentials configured in your environment or use an AWS credentials provider for Boto3.

Sources:

Working with DynamoDB tables in Java - Amazon DynamoDB

Prompt 10

Given that the URLs I'm processing are all in the following format:
https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-july-2024.html
How can I extract the date for which it is based on the last part of the URL "-july-2024.html" and add it to the employment_based_data data I'm storing as a field called "Date" which should be a date-only object as the first day of that month?

Response:

To extract the date from the URL and add it as a field called "Date" in the employment_based_data list, you can modify the scrape_visa_bulletin function as follows:

from datetime import datetime
import re

def scrape_visa_bulletin(url):
    print("Processing url: ", url)

    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')

    employment_based_tables = soup.find_all('table')
    employment_based_data = []

    # Extract the date from the URL
    date_pattern = r'visa-bulletin-for-(\w+)-(\d+)\.html'
    match = re.search(date_pattern, url)
    if match:
        month_name, year = match.groups()
        month_abbr = month_name[:3].lower()
        month_num = datetime.strptime(month_abbr, '%b')
        date_obj = datetime(int(year), month_num, 1)
    else:
        date_obj = None

    for table in employment_based_tables:
        rows = table.find_all('tr')
        table_heading = rows[0].find('td').text.strip()
        if 'Employment-based' in table_heading:
            print("Table heading: ", table_heading)
            headers = [th.text.strip() for th in rows[0].find_all('td')[1:]]
            for row in rows[1:]:
                cols = row.find_all('td')
                type_value = cols[0].text.strip()
                for i, header in enumerate(headers):
                    value = cols[i + 1].text.strip()
                    employment_based_data.append({
                        'group_name': 'Employment-based',
                        'country': header,
                        'type': type_value,
                        'Value': value,
                        'Date': date_obj
                    })

    return employment_based_data

Here's what has been added or modified:

Imported the re module for regular expression operations and the datetime module from the Python standard library.
Added a regular expression pattern date_pattern to match the date portion of the URL (e.g., "july-2024" in "visa-bulletin-for-july-2024.html").
Used re.search to find the match in the URL, and extracted the month name and year from the match groups.
Created a datetime object date_obj using the extracted month name and year, setting the day to the first of the month.
Added the date_obj to the dictionary in the employment_based_data list as the value for the "Date" key.

With these changes, the employment_based_data list will contain dictionaries with the following keys: "group_name", "country", "type", "Value", and "Date". The "Date" value will be a datetime object representing the first day of the month extracted from the URL.

Note that this assumes the URL follows the format "visa-bulletin-for-{month}-{year}.html", where {month} is a three-letter abbreviation of the month name (e.g., "jan", "feb", "mar", etc.), and {year} is a four-digit year. If the URL format is different, you may need to modify the regular expression pattern accordingly.

Prompt 11

The code blocks are added here only for readability, the prompt contained it as pasted in-line code.
Given that I have the following collection built up using:

employment_based_data.append({
    'filing_type': filing_type,
    'country': countries[cell_id - 1],
    'category': category_value,
    'bulletin_date': date_obj,
    'date': cell_date
})

Can you show me how I would transform this collection into a data shape grouped in the following structure?

data: [
  {
    filing_type: Final Action Date
    category: EB-3
    countries: [
      {
        country: "All Chargeability Areas Except Those Listed"
        history: [
          { bulletin_date: 2024-07-01, date: 2021-12-01},
          { bulletin_date: 2024-06-01, date: 2022-11-22},
          { bulletin_date: 2024-05-01, date: 2022-11-22},
          { bulletin_date: 2024-04-01, date: 2022-11-22},
          { bulletin_date: 2024-03-01, date: 2022-09-08},
        ]
      }
    ]
  }
]

Response:

Sure, I can help you transform the collection into the desired data structure. Here's an example of how you can achieve this in Python:

from collections import defaultdict

# Assuming employment_based_data is your initial collection
transformed_data = []

# Group the data by filing_type and category
grouped_data = defaultdict(lambda: defaultdict(list))
for item in employment_based_data:
    filing_type = item['filing_type']
    category = item['category']
    country = item['country']
    bulletin_date = item['bulletin_date']
    date = item['date']
    grouped_data[filing_type][category].append({
        'country': country,
        'bulletin_date': bulletin_date,
        'date': date
    })

# Transform the grouped data into the desired structure
for filing_type, categories in grouped_data.items():
    for category, country_data in categories.items():
        countries = defaultdict(list)
        for item in country_data:
            country = item['country']
            bulletin_date = item['bulletin_date']
            date = item['date']
            countries[country].append({
                'bulletin_date': bulletin_date,
                'date': date
            })

        transformed_data.append({
            'filing_type': filing_type,
            'category': category,
            'countries': [
                {
                    'country': country_name,
                    'history': history_data
                }
                for country_name, history_data in countries.items()
            ]
        })

print(transformed_data)

Here's how the code works:

We start by importing the defaultdict from the collections module, which will help us group the data efficiently. [1]
We create an empty list transformed_data to store the transformed data.
We use a nested defaultdict to group the data by filing_type and category. The outer defaultdict groups by filing_type, and the inner defaultdict groups by category. For each item inemployment_based_data, we append a dictionary containing the country, bulletin_date, and date to the appropriate group.
Next, we iterate over the grouped data (grouped_data). For each filing_type and category combination, we create another defaultdict to group the data by country.
For each country group, we create a list of dictionaries containing the bulletin_date and date for that country.
Finally, we append a dictionary to transformed_data with the filing_type, category, and a list of dictionaries containing the country and its history (the list of bulletin_date and date dictionaries).

After running this code, the transformed_data list will contain the data in the desired structure.

Note: This code assumes that the employment_based_data collection contains unique combinations of filing_type, country, category, and bulletin_date. If there are duplicate entries, you may need to modify the code to handle them appropriately (e.g., by keeping only the latest entry or aggregating the data in some way).

Sources:

defaultdict

Prompt 12

Thanks, when I use this code, I can see that "grouped_data" has data in it, but the "transformed_data" object is returned as a "None", can you help me troubleshoot this to fix it?

Response:

Sure, I can help you troubleshoot this issue. The problem might be that the print(transformed_data) statement is not being executed because of an error or exception occurring earlier in the code.

To troubleshoot this, we can add some print statements and exception handling to the code to see where the issue is occurring. Here's the modified code with some debugging statements:

from collections import defaultdict

# Assuming employment_based_data is your initial collection
transformed_data = []

# Group the data by filing_type and category
grouped_data = defaultdict(lambda: defaultdict(list))
try:
    for item in employment_based_data:
        filing_type = item['filing_type']
        category = item['category']
        country = item['country']
        bulletin_date = item['bulletin_date']
        date = item['date']
        grouped_data[filing_type][category].append({
            'country': country,
            'bulletin_date': bulletin_date,
            'date': date
        })
except Exception as e:
    print(f"Error while grouping data: {e}")

print("Grouped data:")
print(grouped_data)

# Transform the grouped data into the desired structure
try:
    for filing_type, categories in grouped_data.items():
        for category, country_data in categories.items():
            countries = defaultdict(list)
            for item in country_data:
                country = item['country']
                bulletin_date = item['bulletin_date']
                date = item['date']
                countries[country].append({
                    'bulletin_date': bulletin_date,
                    'date': date
                })

            transformed_data.append({
                'filing_type': filing_type,
                'category': category,
                'countries': [
                    {
                        'country': country_name,
                        'history': history_data
                    }
                    for country_name, history_data in countries.items()
                ]
            })
except Exception as e:
    print(f"Error while transforming data: {e}")

print("Transformed data:")
print(transformed_data)

In this modified code, we've added try-except blocks to catch any exceptions that might occur during the grouping and transformation steps. We've also added print statements to display the grouped data and the transformed data.

When you run this code, it will print the grouped data first. If there are no issues with the grouping step, it will then attempt to transform the data. If an exception occurs during the transformation step, it will print the error message and continue executing the code.

Finally, it will print the transformed data. If the transformed_data list is empty or None, it will print an empty list or None.

By running this modified code, you should be able to identify where the issue is occurring and potentially fix it based on the error message or the printed data.

If you still can't identify the issue, please provide the output of running this modified code, along with a sample of the employment_based_data collection, so that I can further assist you in troubleshooting the problem.

Prompt 13

Again, codeblock is just for readability.
Thanks, that helped me find the issue. I now want to relook how the data will be stored in DynamoDB. I currently have the table defined using Terraform as:

resource "aws_dynamodb_table" "visa_bulletin_data" {
  name           = "VisaBulletinData"
  billing_mode   = "PROVISIONED"
  read_capacity  = 5
  write_capacity = 5
  hash_key       = "row_index"
  range_key      = "col_index"

  attribute {
    name = "row_index"
    type = "N"
  }

  attribute {
    name = "col_index"
    type = "N"
  }
}

Can you show me how I should change this so I can do a look up of the data using "filing_type", "category", and by a specific "country" in "countries"? Please only show the change the the DynamoDB table's Terraform code, the snippet for storing the whole data set, and the snippet for retrieving a subset of the data based on the 3 lookups I defined.

Response:

Sure, I can help you with that. To enable lookups based on filing_type, category, and country, we need to modify the DynamoDB table schema and use a composite primary key. Here's how you can change the Terraform code for the DynamoDB table:

resource "aws_dynamodb_table" "visa_bulletin_data" {
  name           = "VisaBulletinData"
  billing_mode   = "PROVISIONED"
  read_capacity  = 5
  write_capacity = 5
  hash_key       = "pk" # Partition key
  range_key      = "sk" # Sort key

  attribute {
    name = "pk"
    type = "S" # String type
  }

  attribute {
    name = "sk"
    type = "S" # String type
  }

  global_secondary_index {
    name            = "CountryIndex"
    hash_key        = "pk"
    range_key       = "sk"
    projection_type = "ALL"
  }
}

In this updated Terraform code:

The hash_key is set to "pk" (partition key), and the range_key is set to "sk" (sort key). Both keys are of type "S" (String).
A global_secondary_index named "CountryIndex" is defined, which has the same hash_key and range_key as the table's primary key. The projection_type is set to "ALL", which means that all attributes will be projected into the index. [1]

With this schema, you can store the data in DynamoDB using a composite primary key that combines the filing_type, category, and country values. Here's an example of how you can store the data:

import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('VisaBulletinData')

for item in transformed_data:
    filing_type = item['filing_type']
    category = item['category']
    for country_data in item['countries']:
        country = country_data['country']
        history = country_data['history']
        for entry in history:
            pk = f"{filing_type}#{category}"
            sk = f"{country}#{entry['bulletin_date'].isoformat()}"
            table.put_item(
                Item={
                    'pk': pk,
                    'sk': sk,
                    'filing_type': filing_type,
                    'category': category,
                    'country': country,
                    'bulletin_date': entry['bulletin_date'],
                    'date': entry['date']
                }
            )

In this code snippet, we construct the pk (partition key) by combining the filing_type and category with a delimiter (#). The sk (sort key) is constructed by combining the country and bulletin_date (converted to an ISO-formatted string) with a delimiter.

To retrieve a subset of the data based on filing_type, category, and country, you can use the following code:

import boto3
from boto3.dynamodb.conditions import Key

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('VisaBulletinData')

filing_type = 'Final Action Date'
category = 'EB-3'
country = 'All Chargeability Areas Except Those Listed'

pk = f"{filing_type}#{category}"
sk_prefix = f"{country}#"

response = table.query(
    KeyConditionExpression=Key('pk').eq(pk) & Key('sk').begins_with(sk_prefix)
)

items = response['Items']

In this code snippet:

We construct the pk (partition key) by combining the filing_type and category with a delimiter (#).
We construct the prefix for the sk (sort key) by combining the country with a delimiter (#).
We use the query method of the DynamoDB table to retrieve items where the pk matches the constructed value, and the sk begins with the constructed prefix.
The retrieved items are stored in the items variable.

This approach allows you to efficiently retrieve data based on the filing_type, category, and country values. You can modify the query conditions as needed to retrieve different subsets of the data.

Note: Make sure to replace the read_capacity and write_capacity values in the Terraform code with appropriate values based on your application's read and write requirements. Additionally, consider enabling auto-scaling for the DynamoDB table to automatically adjust the provisioned capacity based on your application's traffic.

Sources:

Creating an index - Amazon DynamoDB

Prompt 14

The code below is causing an error, I see that "Key" is highlighted in my IDE with the warning:
"Key is not defined"
Here is the code causing the issue:
response = table.query(
KeyConditionExpression=Key('pk').eq(pk) & Key('sk').begins_with(sk_prefix)
)
And the error when I run it:
NameError: name 'Key' is not defined

Response:

The issue here is that the Key function is part of the boto3.dynamodb.conditions module, and you need to import it explicitly in your Python script.

To fix this error, add the following import statement at the top of your Python file:

from boto3.dynamodb.conditions import Key

After adding this import statement, the Key function will be available, and you can use it in your code without encountering the NameError.

import boto3
from boto3.dynamodb.conditions import Key

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('VisaBulletinData')

filing_type = 'Final Action Date'
category = 'EB-3'
country = 'All Chargeability Areas Except Those Listed'

pk = f"{filing_type}#{category}"
sk_prefix = f"{country}#"

response = table.query(
    KeyConditionExpression=Key('pk').eq(pk) & Key('sk').begins_with(sk_prefix)
)

items = response['Items']

With this change, your code should run without the NameError: name 'Key' is not defined error.

The Key function is part of the boto3.dynamodb.conditions module, which provides a way to construct condition expressions for DynamoDB queries and scans. It allows you to create conditions based on the primary key attributes (partition key and sort key) and other attributes of the DynamoDB table.

By importing the Key function from the boto3.dynamodb.conditions module, you can use it to construct the KeyConditionExpression for your DynamoDB query, specifying conditions on the pk (partition key) and sk (sort key) attributes.

Prompt 15

Again, codeblock added for readability.

Can you help me update the following code to handle multiple items with the same "country" field? The "bulletin_date" should be unique per "country". Here is the code causing the issue:

    grouped_data = defaultdict(lambda: defaultdict(list))
    try:
        for item in employment_based_data:
            filing_type = item['filing_type']
            category = item['category']
            country = item['country']
            bulletin_date = item['bulletin_date']
            date = item['date']
            grouped_data[filing_type][category].append({
                'country': country,
                'bulletin_date': bulletin_date,
                'date': date
            })
    except Exception as e:
        print(f"Unable to group the data, error: {e}")

Response:

To handle multiple items with the same "country" field and ensure that the "bulletin_date" is unique per "country", we need to modify the grouping logic slightly. Here's the updated code:

from collections import defaultdict

# Assuming employment_based_data is your initial collection
transformed_data = []

# Group the data by filing_type, category, and country
grouped_data = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
try:
    for item in employment_based_data:
        filing_type = item['filing_type']
        category = item['category']
        country = item['country']
        bulletin_date = item['bulletin_date']
        date = item['date']
        grouped_data[filing_type][category][country].append({
            'bulletin_date': bulletin_date,
            'date': date
        })
except Exception as e:
    print(f"Unable to group the data, error: {e}")

# Transform the grouped data into the desired structure
for filing_type, categories in grouped_data.items():
    for category, countries in categories.items():
        countries_list = []
        for country, history_data in countries.items():
            countries_list.append({
                'country': country,
                'history': history_data
            })

        transformed_data.append({
            'filing_type': filing_type,
            'category': category,
            'countries': countries_list
        })

print(transformed_data)

Here's what has changed:

We're using a nested defaultdict with three levels: filing_type, category, and country. This allows us to group the data by all three keys.
In the grouping step, we're appending the bulletin_date and date dictionaries to the innermost defaultdict for each unique combination of filing_type, category, and country.
In the transformation step, we iterate over the countries dictionary for each filing_type and category combination. For each country, we create a dictionary with the country key and the history data (the list of bulletin_date and date dictionaries).
We append these dictionaries to a countries_list, which is then added to the transformed_data list.

With this updated code, the "bulletin_date" will be unique per "country" because we're grouping the data by all three keys (filing_type, category, and country). The history list for each country will contain only unique bulletin_date entries.

Note: This code assumes that the employment_based_data collection does not contain duplicate entries for the same combination of filing_type, country, category, and bulletin_date. If there are duplicate entries, you may need to modify the code to handle them appropriately (e.g., by keeping only the latest entry or aggregating the data in some way).

Prompt 16

I am still getting the same error, can you show me the code to add to detect and print duplicates that would cause this issue?

Response:

Sure, I can help you with that. Here's the updated code that includes a check for duplicate entries based on the combination of filing_type, country, category, and bulletin_date:

from collections import defaultdict

# Assuming employment_based_data is your initial collection
transformed_data = []

# Group the data by filing_type, category, and country
grouped_data = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
duplicates_found = False

try:
    for item in employment_based_data:
        filing_type = item['filing_type']
        category = item['category']
        country = item['country']
        bulletin_date = item['bulletin_date']
        date = item['date']

        # Check for duplicates
        key = (filing_type, country, category, bulletin_date)
        if key in grouped_data[filing_type][category][country]:
            print(f"Duplicate entry found: {key}")
            duplicates_found = True
        else:
            grouped_data[filing_type][category][country].append({
                'bulletin_date': bulletin_date,
                'date': date
            })

except Exception as e:
    print(f"Unable to group the data, error: {e}")

if duplicates_found:
    print("Duplicates found, please review the data.")
else:
    # Transform the grouped data into the desired structure
    for filing_type, categories in grouped_data.items():
        for category, countries in categories.items():
            countries_list = []
            for country, history_data in countries.items():
                history_data.pop()  # Remove the key tuple from the history list
                countries_list.append({
                    'country': country,
                    'history': history_data
                })

            transformed_data.append({
                'filing_type': filing_type,
                'category': category,
                'countries': countries_list
            })

    print(transformed_data)

Here's what has been added or modified:

A duplicates_found flag is initialized to False.
Before appending the bulletin_date and date dictionaries to the innermost defaultdict, we create a tuple key with the combination of filing_type, country, category, and bulletin_date.
We check if this key already exists in the innermost defaultdict for the corresponding filing_type, category, and country. If it does, it means we have a duplicate entry, and we print a message indicating the duplicate entry.
If the key doesn't exist, we append the bulletin_date and date dictionaries to the innermost defaultdict, and we also append the key tuple to the same list. This will help us identify and remove the key tuple later.
After the grouping step, we check the duplicates_found flag. If it's True, we print a message indicating that duplicates were found, and the user should review the data.
If no duplicates were found, we proceed with the transformation step. However, before appending the history data to the countries_list, we remove the key tuple from the history_data list using the pop() method.

With these changes, the code will now detect and print any duplicate entries based on the combination of filing_type, country, category, and bulletin_date. If duplicates are found, it will print a message and skip the transformation step. If no duplicates are found, it will proceed with the transformation and remove the key tuples from the history data before appending it to the transformed_data list.

Note: This code assumes that you want to keep the first occurrence of a unique entry and discard any subsequent duplicates. If you have a different requirement for handling duplicates (e.g., keeping the latest entry or aggregating the data), you'll need to modify the code accordingly.

Prompt 17

I have a list called "visa_bulletin_links" which contains a list of strings. I then loop over the list to process the URLs in that list, can you show me how I would store the URL in DynamoDB after I process it successfully so that on the next run, I skip that URL? If another DynamoDB table is needed, please provide the Terraform code to create it.

Response:

Sure, I can help you with that. To store the processed URLs in DynamoDB and skip them on subsequent runs, you can create a separate DynamoDB table to keep track of the processed URLs. Here's how you can do it:

Create a new DynamoDB table using Terraform: [1]

Create a new file called processed_urls_table.tf and add the following Terraform code to create a DynamoDB table for storing the processed URLs:
```
resource "aws_dynamodb_table" "processed_urls" {
  name           = "ProcessedURLs"
  billing_mode   = "PROVISIONED"
  read_capacity  = 5
  write_capacity = 5
  hash_key       = "url"

  attribute {
    name = "url"
    type = "S"
  }
}
```
This Terraform code creates a DynamoDB table named ProcessedURLs with a partition key (hash key) named url of type string (S). You can adjust the read_capacity and write_capacity values based on your application's requirements.

Check if the URL has been processed before:

Before processing a URL, you can check if it has already been processed by querying the ProcessedURLs table. Here's an example Python code snippet:

import boto3

dynamodb = boto3.resource('dynamodb')
processed_urls_table = dynamodb.Table('ProcessedURLs')

for url in visa_bulletin_links:
    # Check if the URL has been processed
    response = processed_urls_table.get_item(Key={'url': url})
    if 'Item' in response:
        print(f"Skipping URL: {url} (already processed)")
        continue

    # Process the URL
    print(f"Processing URL: {url}")
    # Your code to process the URL goes here

    # Store the processed URL in DynamoDB
    processed_urls_table.put_item(Item={'url': url})

In this code:

* We create a DynamoDB resource and get a reference to the `ProcessedURLs` table.

* For each URL in the `visa_bulletin_links` list, we use the `get_item` method to check if the URL already exists in the `ProcessedURLs` table.

* If the URL exists (i.e., `'Item'` is in the response), we print a message indicating that the URL has already been processed and skip to the next URL using `continue`.

* If the URL hasn't been processed, we print a message indicating that we're processing the URL, and then execute your code to process the URL.

* After successfully processing the URL, we store the URL in the `ProcessedURLs` table using the `put_item` method.

By following this approach, you can keep track of the processed URLs in a separate DynamoDB table and skip them on subsequent runs of your application.

Note: Make sure to run terraform apply after creating the processed_urls_table.tf file to create the ProcessedURLs table in your AWS account. Also, remember to adjust the read_capacity and write_capacity values based on your application's requirements and enable auto-scaling if necessary.

Sources:

Creating a DynamoDB Table with CloudFormation and Adding Items at Creation Time | AWS re:Post

Prompt 18

Can you show me how to update the aws_iam_policy in the terraform code to allow access to this new DynamoDB table, and also include the required permissions to read from it and the VisaBulletinData table?

Response:

Sure, I can help you with that. Here's how you can update the AWS IAM policy in Terraform to allow access to the new ProcessedURLs DynamoDB table and the existing VisaBulletinData table:

Assuming you have an existing IAM policy resource in your Terraform code, you can add the following statements to the policy document:

resource "aws_iam_policy" "your_policy_name" {
  name        = "YourPolicyName"
  description = "Policy for your application"
  policy      = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    # Existing statements...

    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem"
      ],
      "Resource": "${aws_dynamodb_table.processed_urls.arn}"
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:Query"
      ],
      "Resource": "${aws_dynamodb_table.visa_bulletin_data.arn}"
    }
  ]
}
EOF
}

In this updated policy:

The first new statement allows the GetItem and PutItem actions on the ProcessedURLs DynamoDB table. This will allow your application to check if a URL has been processed and store processed URLs in the table.
The second new statement allows the GetItem and Query actions on the VisaBulletinData DynamoDB table. This will allow your application to retrieve data from the VisaBulletinData table.

Note that you'll need to replace your_policy_name with the actual name of your IAM policy resource, and visa_bulletin_data with the actual resource name of your VisaBulletinData DynamoDB table.

If you don't have an existing IAM policy resource in your Terraform code, you can create a new one with the following code:

resource "aws_iam_policy" "your_policy_name" {
  name        = "YourPolicyName"
  description = "Policy for your application"
  policy      = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem"
      ],
      "Resource": "${aws_dynamodb_table.processed_urls.arn}"
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:Query"
      ],
      "Resource": "${aws_dynamodb_table.visa_bulletin_data.arn}"
    }
  ]
}
EOF
}

After adding or updating the IAM policy resource in your Terraform code, you'll need to run terraform apply to apply the changes and update the IAM policy in your AWS account.

Remember to attach this IAM policy to the IAM role or user that your application is using to access the DynamoDB tables.

Prompt 19

Given that I have the following code that populates my collection:

employment_based_data.append({
    'filing_type': filing_type,
    'country': countries[cell_id - 1],
    'category': category_value,
    'bulletin_date': bulletin_date.strftime("%Y-%m-%d"),
    'date': cell_date.strftime("%Y-%m-%d")
})

How would I extract all the entries where I provide the filling_type, category, and country, and then sort and print the matching results in descending order by bulletin_date?

Response:

To extract all the entries where you provide the filing_type, category, and country, and then sort and print the matching results in descending order by bulletin_date, you can use the following Python code:

from operator import itemgetter

# Assuming employment_based_data is your initial collection
filing_type = 'Final Action Date'
category = 'EB-3'
country = 'All Chargeability Areas Except Those Listed'

# Filter the data based on filing_type, category, and country
filtered_data = [entry for entry in employment_based_data
                 if entry['filing_type'] == filing_type
                 and entry['category'] == category
                 and entry['country'] == country]

# Sort the filtered data in descending order by bulletin_date
sorted_data = sorted(filtered_data, key=itemgetter('bulletin_date'), reverse=True)

# Print the sorted data
for entry in sorted_data:
    print(f"Bulletin Date: {entry['bulletin_date']}, Date: {entry['date']}")

Here's how the code works:

We define the filing_type, category, and country values for which we want to extract the data.
We use a list comprehension to filter the employment_based_data collection based on the filing_type, category, and country values. The resulting filtered_data list contains only the entries that match the specified criteria.
We use the sorted function to sort the filtered_data list in descending order based on the bulletin_date value. The key parameter of the sorted function is set to itemgetter('bulletin_date'), which tells the sorted function to use the bulletin_date value as the sorting key. The reverse=True argument ensures that the sorting is done in descending order.
The sorted data is stored in the sorted_data list.
Finally, we iterate over the sorted_data list and print each entry's bulletin_date and date values.

After running this code, you should see the matching entries printed in descending order by bulletin_date.

Note: This code assumes that the bulletin_date and date values in the employment_based_data collection are strings in the format "YYYY-MM-DD". If they are in a different format or data type, you may need to modify the code accordingly.

Prompt 20

Can you explain the difference between "learnt" and "learned" please?

Response:

"Learnt" and "learned" are both past participle forms of the verb "to learn," but they are used differently in different varieties of English.

In American English, "learned" is the preferred past participle form, while in British English, "learnt" is more commonly used.

Here are some examples:

American English:

I have learned a lot from this experience.
She had learned to play the piano when she was a child.
The students learned their lesson after failing the test.

British English:

I have learnt a lot from this experience.
She had learnt to play the piano when she was a child.
The students learnt their lesson after failing the test.

In American English, "learnt" is generally considered incorrect or non-standard, although it may be used in some regional dialects or informal speech. In British English, both "learned" and "learnt" are considered correct, but "learnt" is more common in everyday speech and writing.

It's worth noting that in both varieties of English, the present participle form is "learning," as in "I am learning to code" or "She is learning French."

In summary, while "learnt" and "learned" mean the same thing, "learned" is the preferred form in American English, and "learnt" is more commonly used in British English. [1]

Sources:

Learning to Learn (again)

📰 AWS Compiled #006 - Architecting Resilient Microservices

Cobus Bernard 🌩🥑🇿🇦 in 🇺🇸 — Fri, 09 Oct 2020 14:59:07 +0000

# AWS Compiled is a bi-weekly curation of 
# AWS developer content and news. 
# This covers the topics that matter most to developers from 
# across AWS, the developer communities we support, 
# and the projects we contribute to.

$ git checkout -b AWS_Compiled_006
$ git commit -a -m 'Architecting Resilient Microservices'
$ git checkout main
$ git merge AWS_Compiled_006

Share your thoughts to better fit your needs by leaving comments or emailing us. You can also find additional developer news here.

Table of Content

TECH SPOTLIGHT: Architecting Resilient Microservices
UPCOMING EVENTS
FROM THE COMMUNITY
GUIDES AND TUTORIALS
RELEASE NOTES

TECH SPOTLIGHT: Architecting Resilient Microservices

This week's Tech Spotlight is on Architecting Resilient Microservices on AWS. The popularity of breaking up monolithic applications in favour of microservice-based applications has increased greatly due to the ability to solve modern development challenges such as velocity, scalability, and reliability. Monolithic applications are fast to develop. However, they become difficult to scale and update as the codebase grows and many teams get involved. This is because each aspect of the application is tightly coupled, unlike microservice applications which are built with modular independent components. At AWS, we consider microservice architectures a key component of how we build.

A good starting point is to understand how the architecture of microservice-based applications differ from traditional ones, and the first step is to understand the challenges with distributed systems, which in many cases are quite unique. This will help you think about how to design your application using microservices. An example of this, McDonald’s Home Delivery system was built and launched in under 4 months using microservices architecture. Their microservices architecture allows the platform to scale to 20,000 orders per second, with sub 100ms latency.

Building distributed, microservice-based applications is quite different from desktop or monolithic application development and our Amazon engineers have learned a lot from their successes and failures over the years. While building Amazon and AWS, we learned valuable lessons, like how to think about leader election or how we avoid overload in distributed systems by putting the smaller service in control. You can access these, and other learnings via the Amazon Builders’ Library to take advantage of our experiences. As we like to say, there is no compression algorithm for experience, but these articles will help you reason about building your modern applications.

If you want to learn more about architecting and running microservices-based applications on AWS, come join us at the upcoming AWS Dev Day on Modern Application Development on October 15, where we have 12 sessions to help you learn how to build modern application on AWS.

Dive Deeper
Amazon ECS workshop
Amazon EKS workshop
Serverless workshops
Serverless with AWS CDK

UPCOMING EVENTS

DevDay, Oct 15, 2020
Join us at AWS DevDay Online and deepen your modern app development knowledge with this free, virtual event. Participants can dive deep into any of the 12 sessions across our three tracks covering a broad range of topics including containers, serverless, event driven applications, infrastructure as code, application security and more.
Register here

Building CI/CD Workflows for Serverless Applications, Oct 22, 2020
Continuous Integration and Continuous Delivery are more than just buzzwords, they are accepted best practices of building modern software. With serverless applications, building your CI and CD workflows becomes even easier and faster to get up and running with a code to production flow. In this tech talk, we’ll cover the basics of building a pipeline for your serverless applications as well as how you can use serverless in your pipelines. We’ll cover the advanced capabilities of the AWS Code services and how you can modify your pipeline to use services such as AWS Lambda and AWS Step Functions to handle complicated tasks and perform sophisticated workflows on each run.
Register here

AWS re:Invent, Nov 30 - Dec 18, 2020
Join the AWS developer community and subject matter experts as we embark on a 3-week digital re:Invent experience from Nov. 30 – Dec. 18. This year, re:Invent is fully virtual and free to attend. Sign up for emails to get the latest re:Invent news and information.
Subscribe for updates

FROM THE COMMUNITY

AWS releases open source diagramming and visualization tool
AWS announces Diagram Maker, an open source graphical user interface library for IoT application developers.
Learn more

GUIDES AND TUTORIALS

Getting started with Travis-CI.com on AWS Graviton2
Learn how to get started building, testing, and deploying quickly through Travis-CI.com on Graviton2.
Get started

Managing AWS Organizations using the open source org-formation tool
In this three-part series, the creator of the open source tool org-formation explains how to manage AWS Organizations resources using infrastructure as code, how to integrate management of resources across accounts using task files, and how to deploy CloudFormation resources to multiple accounts using Organization Bindings.
Dive in

Using the K3s Kubernetes distribution in an Amazon EKS CI/CD pipeline
We explore how to add unit and integration testing to an Amazon EKS CI/CD pipeline, using the open source, lightweight K3s Kubernetes distribution.
Start exploring

RELEASE NOTES

AWS SSO credential profile support in the AWS Toolkit for JetBrains IDEs
With this new release of the AWS Toolkit for JetBrains, customers can use federated credentials, MFA and AWS Single Sign-On (AWS SSO) to connect their IDEs to AWS. Using AWS SSO, your organization's users can sign in to Active Directory, a built-in AWS SSO directory, or another external identity provider (IdP) connected to AWS SSO and get mapped to an AWS Identity and Access Management (IAM) role.
Read more

The waiters feature in the AWS SDK for Java 2.x
The waiters feature in the AWS SDK for Java 2.x is an abstraction that enables you to validate that AWS resources, such as DynamoDB tables or Amazon S3 buckets, are in a specified state before performing operations on those resources.
Read more

AWS CodePipeline now Supports git clone for Source Actions
AWS CodePipeline Source Actions for GitHub.com, GitHub Enterprise Server, and Bitbucket.org now support cloning of your git repositories. With the new feature, when you create a connection to an external source provider in the Source Action, CodePipeline will clone the git repository to fetch the commit history and metadata.
Read more

Upcoming AWS Events for Sub-Saharan Africa (SSA) in June / July.

Cobus Bernard 🌩🥑🇿🇦 in 🇺🇸 — Tue, 30 Jun 2020 07:34:03 +0000

The last month has been a blur, I’m still confused as to how we can be in June already, feeling like a time traveler as I’m busy booking out talks in September already and the weather making me think it is still April (nice 22C day here in South Africa for me today). This will be a quick listing of what events there are coming up in the next 1.5 months, it is fairly packed :)

AWS EMEA Online Summit

First off, we had the AWS EMEA Online Summit on 17 June, with 55+ sessions. Good news is if you missed it, you can watch all the sessions via the platform for the next 3 months. This was a first for our summit series (which are usually in-person events) and I am really excited to be able to watch all the sessions — prior to joining AWS I always had to pick only 5 on the day as you can’t clone yourself and attend all of them.

AWS Africa Region Virtual Day

For those not yet away, the AWS Cape Town Region was launched on 22 April, and we have created a one-day event with 10 technical sessions to help you build your first cloud project and a business track focused on digital transformation and moving to the cloud. All the details and RSVP can be found here.

AWS Africa Office Hours on Twitch

This is a new type of live, AMA style event we are trying out. Currently we are streaming it to Twitch on channel, with the first 2 sessions on were just myself trying to answer questions, we’ve decided to make it more engaging by having some of my team mates join as well. We had the first one of these on 18 June, and you can watch the session on Youtube. If you want a reminder for the upcoming sessions, use this registration link, otherwise join us live on the following dates/times:

Streaming on the Rocks

Myself and Darko Mesaroš have been trying a new format where we discuss specific tech topics (Ep 1: Infrastructure as Code and Ep 2: T(w)ooling [My #dadjoke of tooling + episode 2 …]) on a Friday evening while enjoying a beverage — “Streaming on the Rocks”. We are looking to also do versions of it called “Streaming Beans” / Some other badly named event that will be roughly during office hours for most EMEA based timezones. Keep an eye on our social feeds for the next ones.

Webinars

With the restrictions on in-person events, we have shifted a lot of focus to doing more digital content, and webinars are a great way for providing content and interacting with the attendees. We’ve been pre-recording them as it allows me to answer questions while the sessions are running. If you haven’t joined them yet, here is a list of the coming ones (the last Infrastructure as Code ones are being scheduled next week, will update with the link when it is ready):
22 June: Multi-region architecture: Using data in more than 1 region
23 June: Multi-region architecture: Using services in more than 1 region
24 June: Getting started with Data Warehouses on AWS
25 June: Getting started with Data Lakes on AWS
13 July: Understanding your AWS bill
14 July: Understanding Total Cost of Ownership on AWS
15 July: Understanding Disaster Recovery using AWS
21 July: Understanding Operations: Add super powers to your operations
22 July: Understanding Operations: The 3 Ps (Preventing Patching Problems)
23 July: Understanding Operations: Operations in a hybrid world
27 July: Infrastructure as Code: An overview
28 July: Infrastructure as Code: Amazon Cloudformation
29 July: Infrastructure as Code: Hashicorp Terraform
30 July: Infrastructure as Code: AWS CDK
And that is a what we currently(*) have lined up till the end of July. If you have any opinions / ideas / suggestions, please respond in the comments, or on social media via Twitter or LinkedIn. Hope to see you at some (all!?!?) of these events :)

(*) There are some French webinars & office hours as well, please let me know if you are interested in more French content, we are doing these as an experiment at the moment to gauge interest.