Forem: Josh Vincent

Amplify API & Lambda

Josh Vincent — Wed, 17 Mar 2021 05:26:53 +0000

Part 2

Okay great now we have data streaming into our DynamoDB table, but how do we use it? What if we want to allow other people to have access to this same data, Maybe for the front end of an application just to display the latest prices.

AppSync.

This is where app sync comes in. We want to be able to have users search this table to get the most up-to-date prices.

Back to VScode we go...

Now we run amplify add api

amplify add api

? Please select from one of the below mentioned services: GraphQL
? Provide API name: marketdataapi
? Choose the default authorization type for the API API key
? Enter a description for the API key: thisisatestapikeyformarketdata
? After how many days from now the API key should expire (1-365): 30
? Do you want to configure advanced settings for the GraphQL API No, I am done.
? Do you have an annotated GraphQL schema? No
? Choose a schema template: Single object with fields (e.g., “Todo” with ID, name, description)
? Do you want to edit the schema now? Yes

Now you should have a file named schema.graphql open, it will first look like this.

type Todo @model {
  id: ID!
  name: String!
  description: String
}

Now we update it to contain our table schema.

type Company @model {
  id: ID!
  symbol: String
  name: String
  price: String
  daily_change: String
  yearly_change: String
  marketcap: String
  timestamp: String
}

This time it will create our API and also create a new folder called graphql with some queries subscriptions and mutations inside.

To test this is working let's login to the console and create an entry in our newly created table.
Amplify appends a string to the end so this table will look different from the last one.

Open up the app sync console.

Let's change the Explorer from query to Mutation & click the + button.

You should now see 3 options
createCompany
deleteCompany
updateCompany

Enter this in a new mutation

mutation MyMutation {
  createCompany(
    input: {
      daily_change: "10%"
      id: "123"
      marketcap: "145 B"
      name: "MarketData Company"
      price: "$8.00"
      symbol: "MDC"
      timestamp: "2021-03-16T08:55:21.827Z"
      yearly_change: "300%"
    }
  ) {
    createdAt
    id
    daily_change
    marketcap
    name
    price
    symbol
    timestamp
    updatedAt
    yearly_change
  }
}

Click run and you should see the following JSON returned.

{
  "data": {
    "createCompany": {
      "createdAt": "2021-03-17T04:37:21.519Z",
      "id": "123",
      "daily_change": "10%",
      "marketcap": "145 B",
      "name": "MarketData Company",
      "price": "$8.00",
      "symbol": "MDC",
      "timestamp": "2021-03-16T08:55:21.827Z",
      "updatedAt": "2021-03-17T04:37:21.519Z",
      "yearly_change": "300%"
    }
  }
}

Now your DynamoDB table should have 1 new entry.

To do a get entry all you need to do is query getCompany(id: "123") which will return the record just created.
or listCompanys which will return a list of all the companies in the table.

Now we have a few other fields and we will need to switch our lambda function over to write to our new table.

{
  "__typename": "Company",
  "createdAt": "2021-03-17T04:37:21.519Z",
  "updatedAt": "2021-03-17T04:37:21.519Z"
}

these are the 3 fields that are excluded from our other lambda-only table lets update our other function and include them now.

Now we open amplify/backend/function/addMarketDatatoDb/src/index.js

and update the Table and Item

TableName: "Company-xpssoal4fbaadbdtx3urmue33m-dev", // UPDATE THIS WITH THE ACTUAL NAME OF THE FORM TABLE ENV VAR (set by Amplify CLI)
    Item: {
      "__typename": "Company",
      "createdAt": timestamp,
      "updatedAt": timestamp,
      id: context.awsRequestId,
      timestamp: timestamp,
      ...parsedBody,
    },

run amplify push once again to update our lambda function

amplify push --y

Now our functions should be updated we have to update our function to allow access to DynamoDB go to the IAM role that is created and grant it access to DynamoDB or if you would like to lock it down just include the ARN and dynamodb:putItem, let's have a look

Open Lambda again go to test and Invoke our marketdataPuppeteer-dev function again.

Now check your Company table you should have 10 new entries!

Great that is all working now let's run our list items query in AppSync

Run in the console window

query MyQuery {
  listCompanys {
    items {
      id
      createdAt
      daily_change
      marketcap
      name
      price
      symbol
      timestamp
      updatedAt
      yearly_change
    }
  }
}

You should get a json list back of all the items from the table now!

{
  "data": {
    "listCompanys": {
      "items": [
        {
          "id": "85cb6706-6be8-562d-9bb1-3fcdf98cef44",
          "createdAt": "2021-03-17T05:14:12.777Z",
          "daily_change": "-0.05%",
          "marketcap": "$63.15 B",
          "name": "FMG Fortescue Metals Group Ltd",
          "price": "$20.51",
          "symbol": "FMG",
          "timestamp": "2021-03-17T05:14:12.777Z",
          "updatedAt": "2021-03-17T05:14:12.777Z",
          "yearly_change": "+91.86%"
        },
        {
          "id": "65a4e33a-5030-5ca3-a8a0-bbf1c2548bb0",
          "createdAt": "2021-03-17T05:14:12.646Z",
          "daily_change": "-0.32%",
          "marketcap": "$154.17 B",
          "name": "CBA Commonwealth Bank of Australia",
          "price": "$86.90",
          "symbol": "CBA",
          "timestamp": "2021-03-17T05:14:12.646Z",
          "updatedAt": "2021-03-17T05:14:12.646Z",
          "yearly_change": "+28.47%"
        },
        etc...
      ]
    }
  }
}

Now we are able to get data via Appsync, the backend is done and all set up to be used from a front end.

AWS Amplify & Puppeteer

Josh Vincent — Tue, 16 Mar 2021 12:49:11 +0000

Recently working on a project that required getting hourly market data of the top 10.

I have put this together in the hope that it will help someone else out using Amplify, Puppeteer, Lambda, DynamoDB, and scraping public data off the web.

As we found it very difficult to find any Api's that could solve this issue for us.

We Discovered a way in which we could leverage lambda every hour to run and get the top 10 companies by marketcap, insert the data into a database which we would fetch later (another future blog post to come...)
This is how we did it.

Here is an image of what the architecture looks like.

This has a 1-hour timer that triggers a cloud watch alarm
Cloudwatch calls our first Lambda function
Lambda function pushes messages as it reads from the table to an SQS queue
SQS triggers a series of lambda functions that write data to our DynamoDB

Creating our new project.

In VS Code run the following



amplify init

Create a name for your project and then accept all defaults



Note: It is recommended to run this command from the root of your app directory
$ Enter a name for the project MarketData
$ Enter a name for the environment dev
$ Choose your default editor: Visual Studio Code
$ Choose the type of app that you're building javascript

Please tell us about your project
$ What javascript framework are you using none
$ Source Directory Path:  src
$ Distribution Directory Path: dist
$ Build Command:  npm run-script build
$ Start Command: npm run-script start
Using default provider  awscloudformation

$ Do you want to use an AWS profile? Yes
$ Please choose the profile you want to use marketdata

If you don't have your credentials saved in ~/.aws/credentials you can just say no to the last question.

Creating our first Lambda function

Next, we get to create our first lambda function by doing the following.



amplify add function

Follow the prompts which should look like this



$ amplify add function
$ Select which capability you want to add: Lambda function (serverless function)
$ Provide a friendly name for your resource to be used as a label for this category in the project: marketdataPuppeteer
$ Provide the AWS Lambda function name: marketdataPuppeteer
$ Choose the runtime that you want to use: NodeJS
$ Choose the function template that you want to use: Hello World
$ Do you want to access other resources in this project from your Lambda function? Yes
$ Select the category 

You can access the following resource attributes as environment variables from your Lambda function
        ENV
        REGION
$ Do you want to invoke this function on a recurring schedule? Yes
$ At which interval should the function be invoked: Hourly
$ Enter the rate in hours: 1
$ Do you want to configure Lambda layers for this function? No
$ Do you want to edit the local lambda function now? (Y/n) y

Set Up the dependencies

next, browse into the name of the function you just created



cd amplify/backend/function/marketdataPuppeteer/src/

Now install the following.



npm install chrome-aws-lambda --save-prod
npm install puppeteer-core --save-prod

Great now open your index.js file inside the amplify/backend/function/marketdataPuppeteer/src/ folder.

Add the required dependencies



//amplify/backend/function/marketdataPuppeteer/src/index.js

var AWS = require("aws-sdk");
var SQS = new AWS.SQS({ region: "ap-southeast-2" });
const chromium = require("chrome-aws-lambda");

You will also need to add in the URL of your SQS queue which we will set up shortly.



var QUEUE_URL =
  "https://sqs.ap-southeast-2.amazonaws.com/012345678/Market-Data-Que";

here is our full function



//amplify/backend/function/marketdataPuppeteer/src/index.js

var AWS = require("aws-sdk");
var SQS = new AWS.SQS({ region: "ap-southeast-2" });
const chromium = require("chrome-aws-lambda");

var QUEUE_URL =
  "https://sqs.ap-southeast-2.amazonaws.com/012345678/Market-Data-Que";

exports.handler = async (event, context, callback) => {
  let result = null;
  let browser = null;

  try {
    browser = await chromium.puppeteer.launch({
      args: chromium.args,
      defaultViewport: chromium.defaultViewport,
      executablePath: await chromium.executablePath,
      headless: chromium.headless,
      ignoreHTTPSErrors: true,
    });

    let page = await browser.newPage();
    await page.goto("https://www.marketindex.com.au/asx-listed-companies");
    await delay(5000); // Wait 5 seconds for page to load

    const selector = "tbody tr"; //We are finding the table here
    const row = await page.$$eval(selector, (
      tableRows //looping through each row in the table
    ) =>
      tableRows.map((tableRow) => {
        const tableDataElement = [...tableRow.getElementsByTagName("td")]; //Get each table data element
        return tableDataElement.map((tableData) =>
          tableData.textContent.trim() //Return the text content from the table data element.
        ); 
      })
    );
    //All the rows now exist in the row variable, console.log(row[0][2]); will get you the first element in rows and the third column
    row.slice(0, 10).map((company) => {
      //Create an object for each of the rows to add to our table
      const item = {
        symbol: company[2],
        name: company[3],
        price: company[4],
        daily_change: company[5],
        yearly_change: company[6],
        marketcap: company[7],
      };
      //console.log(item); //In our logs we should now have a list of objects with the above data

      var params = {
        MessageBody: JSON.stringify(item),
        QueueUrl: QUEUE_URL,
      };
      console.log(item);
      SQS.sendMessage(params)
        .promise()
        .then((result) => console.log("Successfully sent message", result))
        .catch((error) => console.log("Error failed to send message", error));
    });
    //console.log(table);
    await browser.close();
  } catch (error) {
    return callback(error);
  } finally {
    if (browser !== null) {
      await browser.close();
    }
  }
  return callback(null, result);
};

cd back to the project root and push the backend to aws by running



amplify push --y

you should see something like this



$ amplify push
✔ Successfully pulled backend environment dev from the cloud.

Current Environment: dev

| Category | Resource name       | Operation | Provider plugin   |
| -------- | ------------------- | --------- | ----------------- |
| Function | marketdataPuppeteer | Create    | awscloudformation |
$ Are you sure you want to continue? (Y/n) y

This should push your code to AWS Amplify backend and create your lambda function.

Open your console in AWS. You should see something that looks like this

Creating the SQS Queue

Now we have our lambda function setup you might have noticed the QUEUE_URL we haven't created yet.
As of the time this is posted Amplify doesn't have the built-in functionality to create SQS queues so we will do it via the console.

Click create queue

Give your queue a name as we have here Market-Data-Que

Keep all the defaults & Create a queue

Now get your URL from the endpoint address, go back to your function, and replace the QUEUE_URL  with your value.

Run another amplify push to update our recent changes.



amplify push --y

As this is a function that runs for a little longer than normal and the way Puppeteer works we have increased both the execution times and memory for this function.

Go to your Lambda function

Click the configuration tab
Click Edit
Increase Memory to 512 MB
Timeout to 2 min 30 seconds (increase this if you need to)
Click on the View the marketdataLambdaRole12313-dev-role on the IAM console.
We want to enable lambda to contact SQS, so add SQSFullAccess to the role. (update to ARN to lock down access)

Attach the policy and return to your lambda function.

Now we are ready to Test!

Open up the test tab, create a demo event using the hello-world template and Invoke.
You should now see something that looks like this

If you then visit your SQS Queue you should see a heap of messages like this.

Okay great, So far we have a lambda function opening the window in puppeteer, extracting the data, and pushing it to an SQS queue.

Now we want to create a new function that uses the SQS queue as a trigger to enter our entries into the DynamoDB table. Let's set them up.

Creating the storage table.

First let's create the Storage with amplify add storage



$ amplify add storage 
$ Please select from one of the below mentioned services: NoSQL Database

Welcome to the NoSQL DynamoDB database wizard
This wizard asks you a series of questions to help determine how to set up your NoSQL database table.

$ Please provide a friendly name for your resource that will be used to label this category in the project: marketdata
$ Please provide table name: marketdata

You can now add columns to the table.

$ What would you like to name this column: id
$ Please choose the data type: string
$ Would you like to add another column? Yes
$ What would you like to name this column: symbol
$ Please choose the data type: string
$ Would you like to add another column? Yes
$ What would you like to name this column: name
$ Please choose the data type: string
$ Would you like to add another column? Yes
$ What would you like to name this column: price
$ Please choose the data type: string
$ Would you like to add another column? Yes
$ What would you like to name this column: daily_change
$ Please choose the data type: string
$ Would you like to add another column? Yes
$ What would you like to name this column: yearly_change
$ Please choose the data type: string
$ Would you like to add another column? Yes
$ What would you like to name this column: marketcap

Please choose partition key for the table: id
$ Do you want to add a sort key to your table? Yes
$ Please choose sort key for the table: symbol

You can optionally add global secondary indexes for this table. These are useful when you run queries defined in a different column than the primary key.
To learn more about indexes, see:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html#HowItWorks.CoreComponents.SecondaryIndexes

$ Do you want to add global secondary indexes to your table? No
$ Do you want to add a Lambda Trigger for your Table? No

Do another amplify push and see the DynamoDB database created



amplify push —y

Now open the console and go to DynamoDB you should see your table here.

Now that the table is created we can now create another lambda function that will pull messages from the SQS queue and then input the data into our newly created table.

Creating Lambda function 2

We create another function with amplify add function



amplify add function



$ Select which capability you want to add: Lambda function (serverless function)
$ Provide a friendly name for your resource to be used as a label for this category in the project: addMarketDataToDB
$ Provide the AWS Lambda function name: addMarketDataToDB
$ Choose the runtime that you want to use: NodeJS
$ Choose the function template that you want to use: Hello World
$ Do you want to access other resources in this project from your Lambda function? Yes
$ Select the category storage
Storage category has a resource called marketdata
$ Select the operations you want to permit for marketdata create

You can access the following resource attributes as environment variables from your Lambda function
        ENV
        REGION
        STORAGE_MARKETDATA_ARN
        STORAGE_MARKETDATA_NAME
$ Do you want to invoke this function on a recurring schedule? No
$ Do you want to configure Lambda layers for this function? No
$ Do you want to edit the local lambda function now? (Y/n) y

cd into the function again



cd amplify/backend/functions/addMarketDataToDB/src

Add the following dependencies to the top of your index.js



const AWS = require("aws-sdk");
const docClient = new AWS.DynamoDB.DocumentClient();



/* Amplify Params - DO NOT EDIT
    ENV
    REGION
    STORAGE_MARKETDATA_ARN
    STORAGE_MARKETDATA_NAME
Amplify Params - DO NOT EDIT */

const AWS = require("aws-sdk");
const docClient = new AWS.DynamoDB.DocumentClient();

exports.handler = async (event, context) => {
  const { body } = event.Records[0];
  const parsedBody = JSON.parse(body);
  const timestamp = new Date().toISOString();
  var params = {
    TableName: "marketdata-dev", // UPDATE THIS WITH THE ACTUAL NAME OF THE FORM TABLE ENV VAR (set by Amplify CLI)
    Item: {
      id: context.awsRequestId, //this uses the id of the invoked lambda function from context
      timestamp: timestamp,
      ...parsedBody,
    },
  };
  console.log(params);
  await docClient
    .put(params)
    .promise()
    .then((data) => console.log("Success!", data))
    .catch((err) => console.log("error!", err));
};

/* SQS event looks like this.
{
    "Records": [
      {
        "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
        "receiptHandle": "MessageReceiptHandle",
        "body": "{\n      \"symbol\": \"FMG\",\n      \"name\": \"FMG Fortescue Metals Group Ltd\",\n      \"price\": \"$20.63\",\n      \"daily_change\": \"+1.23%\",\n      \"yearly_change\": \"+113.78%\",\n      \"marketcap\": \"$63.52 B\"\n    }",
        "attributes": {
          "ApproximateReceiveCount": "1",
          "SentTimestamp": "1523232000000",
          "SenderId": "123456789012",
          "ApproximateFirstReceiveTimestamp": "1523232000001"
        },
        "messageAttributes": {},
        "md5OfBody": "{{{md5_of_body}}}",
        "eventSource": "aws:sqs",
        "eventSourceARN": "arn:aws:sqs:ap-southeast-2:123456789012:MyQueue",
        "awsRegion": "ap-southeast-2"
      }
    ]
}
*/

Update our cloud backend with another amplify push —y



amplify push --y

  wait for addMarketDataToDB function to appear in the console. 
Once it has appeared go to the test tab and insert the following event with this example SQS event.



{
    "Records": [
      {
        "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
        "receiptHandle": "MessageReceiptHandle",
        "body": "{\n      \"symbol\": \"FMG\",\n      \"name\": \"FMG Fortescue Metals Group Ltd\",\n      \"price\": \"$20.63\",\n      \"daily_change\": \"+1.23%\",\n      \"yearly_change\": \"+113.78%\",\n      \"marketcap\": \"$63.52 B\"\n    }",
        "attributes": {
          "ApproximateReceiveCount": "1",
          "SentTimestamp": "1523232000000",
          "SenderId": "123456789012",
          "ApproximateFirstReceiveTimestamp": "1523232000001"
        },
        "messageAttributes": {},
        "md5OfBody": "{{{md5_of_body}}}",
        "eventSource": "aws:sqs",
        "eventSourceARN": "arn:aws:sqs:ap-southeast-2:123456789012:MyQueue",
        "awsRegion": "ap-southeast-2"
      }
    ]
}

If this is working you should now be able to see an entry in your DynamoDB table like this.

Next, we need to configure the lambda function to get the messages from the SQS queue.
Go back to your lambda function in the console.

Click Triggers
Find SQS
Enter the name of your SQS queue
Make sure to make the batch size 1
Click create

Once you have created this if you go to your SQS queue you will notice the count of messages decreasing, Lambda is now querying the SQS queue and inserting the data into the table.

Now you have a fully automated web scraper running every hour. To Test the whole process just wait for 1Hr or Trigger the first lambda function we created!