Forem: Dera Okeke

How to Dynamically Scrape Data from a Webpage with a "Load More" Button

Dera Okeke — Wed, 26 Mar 2025 13:50:17 +0000

Web scraping is a powerful tool for gathering information from websites. When web scraping in NodeJS, we can choose from an array of web NodeJS has several web scraping libraries. However, we'll use Puppeteer to handle the web scraping for this tutorial.

In this tutorial, we'll walk through a simple, step-by-step guide to scraping data from a website with dynamic content. We'll cover making requests using NodeJS, loading additional content, parsing information, and exporting this information to a CSV document.

Let's get right to it!

View implementation on GitHub.

Step 1: Prerequisites

In this section, we’ll go through all the steps and installations required for this tutorial. To follow along with the rest of this guide, you will need the following:

NodeJS: You can download and install a NodeJS version with Long-Term Support (LTS) from the NodeJS download page. This installation adds NodeJS to your machine's directory and allows you to install dependencies with the Node Package Manager (NPM).
A code editor of choice.

With the environment setup completed, you can start setting up NodeJS for your project. To begin, create a new folder in a directory of choice and create a scrape.js file.

Next, you must initialize NodeJS in the newly created folder. To initialize NodeJS, run the following command at the root of the folder's directory:

npm init -y

This command creates a project scaffold with the node_modules folder and package.json file that contains our project's dependencies. Then, we need to download some NPM packages to scrape the website. These packages are:

puppeteer: Puppeteer is a powerful NodeJS library that automates tasks in a headless browser. It allows interaction with page elements, making it helpful in scraping dynamic content.
json2csv: A fast and configurable JSON to CSV converter.

All required packages are available on the NPM package registry. To download them to your project, run the following command in the terminal at the root of your project's directory:

npm install puppeteer json2csv

Step 2: Get Access to the Content

After installing all required dependencies, the next step is to access the HTML content on the page. To do this, you’ll use Puppeteer to send the request to the URL and then log the returned response to the console as shown:

// scrape.js

const puppeteer = require("puppeteer");

const url = "https://www.scrapingcourse.com/button-click";

const scrapeFunction = async () => {
  // Launch the browser and open a new blank page
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Navigate to the target URL
  await page.goto(url);

  // Get the URL’s HTML content
  const content = await page.content();
  console.log(content);

  // Close the browser and all of its pages
  await browser.close();
};

scrapeFunction();

You can run the code in the scrape.js file using NodeJS by running the following command in the terminal:

node scrape.js

The output of the code returns the full HTML content of the website in the terminal window:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Load More Button Challenge to Learn Web Scraping   - ScrapingCourse.com</title>

    <!-- Bootstrap CSS -->
</head>
<body data-new-gr-c-s-check-loaded="14.1174.0" data-gr-ext-installed="" itemscope itemtype="http://schema.org/WebPage">
    <main>
    <!-- ... -->
    </main>

    <!-- Bootstrap and jQuery libraries -->
    <script src="https://code.jquery.com/jquery-3.5.1.slim.min.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/@popperjs/core@2.5.2/dist/umd/popper.min.js"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/js/bootstrap.min.js"></script>
</body>
</html>

Step 3: Load More Products

Now that you have access to the URL content, you can interact with the page's content to load more products. For this tutorial, you’ll simulate an interaction with the "Load More" button at the bottom of the page.

To do this, you will use Puppeteer's page.click() and page.waitForSelector() methods. The page.click() method accepts the selector of a DOM element, scrolls it into view if required, and then simulates a mouse click at the center of the element. The page.waitForSelector() method, on the other hand, pauses the script until a node that matches a specific selector appears on the page. This method is helpful when waiting for a dynamic element on a page.

You must have retrieved the selector of the "Load More" button——button#load-more-btn—on the target page before using the page.click() method. Also, for this tutorial, we'll be using a for loop to repeat the button click event till we have a minimum of 48 product cards on the target page.

// scrape.js

//   Click the "Load More" button five times
for (let i = 0; i < 5; i++) {
  await page.click("button#load-more-btn");
}

// wait for the 48th product card
await page.waitForSelector(".product-grid .product-item:nth-child(48)");

The page.waitForSelector() method accepts the selector of the 48th product item in the product-grid element class. This ensures that the rest of the program runs only after the 48th item is rendered.

At this point, your scrape.js file should look like this:

// scrape.js

const puppeteer = require("puppeteer");

const url = "https://www.scrapingcourse.com/button-click";

const scrapeFunction = async () => {
 // Launch the browser and open a new blank page
 const browser = await puppeteer.launch();
 const page = await browser.newPage();

 // Navigate to the target URL
 await page.goto(url);

 //   Click the "Load More" button a fixed number of times
 for (let i = 0; i < 5; i++) {
   await page.click("button#load-more-btn");
 }

 // wait for the 48th product card
 await page.waitForSelector(".product-grid .product-item:nth-child(48)");

 // Get the URL's HTML content
 const content = await page.content();
 console.log(content);

 // Close the browser and all of its pages
 await browser.close();
};
scrapeFunction();

Again, you can run the NodeJS script in your terminal using the earlier command. The output of the code returns the full-page HTML content of the website in the terminal window:

<!DOCTYPE html><html lang="en"><head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Load More Button Challenge to Learn Web Scraping   - ScrapingCourse.com</title>

    <!-- Bootstrap CSS -->
     <main>
      <!-- ... -->
     </main>

    <!-- Bootstrap and jQuery libraries -->
    <script src="https://code.jquery.com/jquery-3.5.1.slim.min.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/@popperjs/core@2.5.2/dist/umd/popper.min.js"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/js/bootstrap.min.js"></script>
</body></html>

Step 4: Parse Product Information

You can parse the product information since you now have the raw HTML data from the target page. Doing this converts the raw HTML data into a more readable format.

To begin, you’ll use Puppeteer's page.evaluate() method. This method allows you to run JavaScript functions within the context of the target page and returns a result. Then, create an empty array to collect the extracted data and retrieve the selector of the container element as shown:

 // extract product information and return an array of products
 const products = await page.evaluate(() => {
   const productList = [];
   productElements = document.querySelectorAll(".product-grid .product-item");

   // parsing function goes here

 return productList;
 });

Next, loop through each product container returned to scrape through the data using each element’s unique selector. Then, push the data into the array created:

// scrape.js

// extract product information and return an array of products
 const products = await page.evaluate(() => {
   const productList = [];
   productElements = document.querySelectorAll(".product-grid .product-item");

// loop through each product to extract the data
   productElements.forEach((product) => {
     const name = product.querySelector("div span.product-name").textContent;
     const imageLink = product
       .querySelector("img.product-image")
       .getAttribute("src");
     const price = product.querySelector("div span.product-price").textContent;
     const url = product.querySelector("a").getAttribute("href");


    // push the extracted data to the array created
     productList.push({ name, imageLink, price, url });
   });
   return productList;
 });

Finally, use NodeJS' file system module—fs—to create a new products.json file and write the array content to this file. You’ll also use JavaScript's JSON.stringify() method to convert the array data to JSON for readability.

// scrape.js
const fs = require("fs");

// create a new JSON file and parse all the data to the file
fs.writeFileSync("products.json", JSON.stringify(products, null, 2));

console.log("Data saved to products.json");

At this point, your scrape.js file should look like this:

// scrape.js
const puppeteer = require("puppeteer");
const fs = require("fs");

const url = "https://www.scrapingcourse.com/button-click";

const scrapeFunction = async () => {
 // Launch the browser and open a new blank page
 const browser = await puppeteer.launch();
 const page = await browser.newPage();

 // Navigate to the target URL
 await page.goto(url);

 //   Click the "Load More" button a fixed number of times
 for (let i = 0; i < 5; i++) {
   // Click the "Load More" button
   await page.click("button#load-more-btn");
 }

 // wait for the 48th product card
 await page.waitForSelector(".product-grid .product-item:nth-child(48)");

 // extract product information and return an array of products
 const products = await page.evaluate(() => {
   const productList = [];
   productElements = document.querySelectorAll(".product-grid .product-item");

   // loop through each product to extract the data
   productElements.forEach((product) => {
     const name = product.querySelector("div span.product-name").textContent;
     const imageLink = product
       .querySelector("img.product-image")
       .getAttribute("src");
     const price = product.querySelector("div span.product-price").textContent;
     const url = product.querySelector("a").getAttribute("href");

     // push the extracted data to the array created
     productList.push({ name, imageLink, price, url });
   });
   return productList;
 });

 // create a new JSON file and parse all the data to the file
 fs.writeFileSync("products.json", JSON.stringify(products, null, 2));

 console.log("Data saved to products.json");

 // Close the browser and all of its pages
 await browser.close();
};

scrapeFunction();

The output of the code returns a structured JSON in the products.json file with the product data:

[
 {
   "name": "Chaz Kangeroo Hoodie",
   "imageLink": "https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh01-gray_main.jpg",
   "price": "$52",
   "url": "https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie"
 },
 {
   "name": "Teton Pullover Hoodie",
   "imageLink": "https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh02-black_main.jpg",
   "price": "$70",
   "url": "https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie"
 },
 {
   "name": "Bruno Compete Hoodie",
   "imageLink": "https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh03-black_main.jpg",
   "price": "$63",
   "url": "https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie"
 },
  {
   "name": "Ajax Full-Zip Sweatshirt",
   "imageLink": "https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh12-green_main.jpg",
   "price": "$69",
   "url": "https://scrapingcourse.com/ecommerce/product/ajax-full-zip-sweatshirt"
 },
...
 {
   "name": "Mars HeatTech&trade; Pullover",
   "imageLink": "https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj10-red_main.jpg",
   "price": "$66",
   "url": "https://scrapingcourse.com/ecommerce/product/mars-heattech&trade;-pullover"
 }
]

Step 5: Export Product Information to CSV

Alternatively, you can also export the product data into a CSV file. This will be an easy step since you already learned how to parse the information earlier.

To do this, you will use the json2csv package installed at the beginning of the tutorial.

// scrape.js

const json2csv = require("json2csv").Parser;
const fs = require("fs").Parser;
 // . . .   

 // initialize the package
 const parser = new json2csv();

 // create a new CSV file and parse the data to the file
 const productsCSV = parser.parse(products);
 fs.writeFileSync("products.csv", productsCSV);

 console.log("Data saved to products.csv");

At this point, your scrape.js file should look like this:

const puppeteer = require("puppeteer");
const fs = require("fs");
const json2csv = require("json2csv").Parser;

const url = "https://www.scrapingcourse.com/button-click";

const scrapeFunction = async () => {
 // Launch the browser and open a new blank page
 const browser = await puppeteer.launch();
 const page = await browser.newPage();

 // Navigate to the target URL
 await page.goto(url);

 //   Click the "Load More" button a fixed number of times
 for (let i = 0; i < 5; i++) {
   // Click the "Load More" button
   await page.click("button#load-more-btn");
 }

 // wait for the 48th product card
 await page.waitForSelector(".product-grid .product-item:nth-child(48)");

 // extract product information and return an array of products
 const products = await page.evaluate(() => {
   const productList = [];
   productElements = document.querySelectorAll(".product-grid .product-item");

   // loop through each product to extract the data
   productElements.forEach((product) => {
     const name = product.querySelector("div span.product-name").textContent;
     const imageLink = product
       .querySelector("img.product-image")
       .getAttribute("src");
     const price = product.querySelector("div span.product-price").textContent;
     const url = product.querySelector("a").getAttribute("href");

     // push the extracted data to the array created
     productList.push({ name, imageLink, price, url });
   });
   return productList;
 });


 // initialize the package
 const parser = new json2csv();

 // create a new CSV file and parse the data to the file
 const productsCSV = parser.parse(products);
 fs.writeFileSync("products.csv", productsCSV);

 console.log("Data saved to products.csv");

 // Close the browser and all of its pages
 await browser.close();
};

scrapeFunction();

Again, you can run the NodeJS script in your terminal using the earlier command. The output of the code returns a structured CSV in the products.csv file with the product data:

"name","imageLink","price","url"
"Chaz Kangeroo Hoodie","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh01-gray_main.jpg","$52","https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie"
"Teton Pullover Hoodie","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh02-black_main.jpg","$70","https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie"
"Bruno Compete Hoodie","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh03-black_main.jpg","$63","https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie"
"Frankie  Sweatshirt","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh04-green_main.jpg","$60","https://scrapingcourse.com/ecommerce/product/frankie--sweatshirt"
"Hollister Backyard Sweatshirt","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh05-white_main.jpg","$52","https://scrapingcourse.com/ecommerce/product/hollister-backyard-sweatshirt"
"Stark Fundamental Hoodie","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh06-blue_main.jpg","$42","https://scrapingcourse.com/ecommerce/product/stark-fundamental-hoodie"
"Hero Hoodie","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh07-gray_main.jpg","$54","https://scrapingcourse.com/ecommerce/product/hero-hoodie"
"Oslo Trek Hoodie","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh08-brown_main.jpg","$42","https://scrapingcourse.com/ecommerce/product/oslo-trek-hoodie"
"Kenobi Trail Jacket","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj04-black_main.jpg","$47","https://scrapingcourse.com/ecommerce/product/kenobi-trail-jacket"
"Jupiter All-Weather Trainer","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj06-blue_main.jpg","$56.99","https://scrapingcourse.com/ecommerce/product/jupiter-all-weather-trainer"
"Orion Two-Tone Fitted Jacket","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj07-red_main.jpg","$72","https://scrapingcourse.com/ecommerce/product/orion-two-tone-fitted-jacket"
"Lando Gym Jacket","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj08-gray_main.jpg","$99","https://scrapingcourse.com/ecommerce/product/lando-gym-jacket"
"Taurus Elements Shell","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj09-yellow_main.jpg","$65","https://scrapingcourse.com/ecommerce/product/taurus-elements-shell"
"Mars HeatTech&trade; Pullover","https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj10-red_main.jpg","$66","https://scrapingcourse.com/ecommerce/product/mars-heattech&trade;-pullover"

Conclusion

In this tutorial, we’ve covered the basics of web scraping:

Sending requests to access content on a webpage.
Loading additional content in a dynamic webpage.
Parsing information retrieved from the webpage.
Exporting information to JSON and CSV files.

Whether you’re using open-source libraries or a web scraping service, these steps will help you get started with scraping dynamic content efficiently. Happy scraping!

How to Implement Seat-based Billing with Stripe Checkout

Dera Okeke — Wed, 12 Mar 2025 17:28:06 +0000

Modern SaaS businesses need a pricing model that aligns with customer usage and scales effectively. Seat-based billing is a common pricing model for SaaS businesses, allowing customers to pay a fee based on the number of seats or users they need each month.

In this article, we’ll explore how to implement a seat-based subscription model using Stripe Hosted Checkout for initial subscription creation and Stripe Customer Portal for post-purchase seat management.

View implementation on Github.

Prerequisites

In this section, we'll go through the requirements for the tutorial. To follow along with the rest of this guide, you will need the following:

A Stripe account: You can use an account in test mode for now. However, ensure you enable live mode on your account when in production.
Node.js: You can download and install a NodeJS version with Long-Term Support (LTS) from the NodeJS download page.
A code editor of choice. The tutorial assumes that you already have a product with a recurring billing setup. If you don't have one, see Create products and prices on the Stripe documentation for more information.

Step 1: Install the Stripe library

To get started, we must first install the Stripe Node Library. This library is required to create a new checkout session on our application. To install the library, run the following command in the root of your project's directory:

npm install stripe express

The command also installs Express to the project. Express is a minimalistic back-end web application framework used for building applications with Node.js.

Step 2: Building the UI

Next, we need to build out the UI for the application. This step involves building a lightweight UI to handle the transition to Stripe's hosted checkout, as well as the success.html and cancel.html pages required by Stripe checkout.

Create the `checkout.html`

For the checkout.html, we'll create a simple HTML page with a form and button that triggers Stripe's checkout session when clicked, as shown:

<body>
  <section>
    <form action="/create-checkout-session" method="POST">
      <button type="submit" id="checkout-button">Checkout</button>
    </form>
  </section>
</body>

Create the success and cancel pages

Next, let's create the success and cancel pages—success.html and cancel.html. Stripe requires these pages as they determine how our application handles redirects after a successful checkout or upon cancellation.

<!-- success.html -->
<body>
  <section>
    <p>
      We appreciate your business! If you have any questions, please email
      <a href="mailto:orders@example.com">orders@example.com</a>.
    </p>
  </section>
</body>

For the cancel.html:

<!-- cancel.html -->
<body>
  <section>
    <p>
      Forgot to add something to your cart? Shop around then come back to pay!
    </p>
  </section>
</body>

Step 3: Setup the server

Now that we've built the UI, we can begin setting up the backend server that handles the checkout session.

First, we must import and initialize the Stripe and Express libraries that we installed earlier as shown:

// server.js
const stripe = require("stripe")("YOUR_STRIPE_SECRET_KEY");
const express = require("express");
const app = express();
app.use(express.static("public"));

const YOUR_DOMAIN = "http://localhost:4242";

Create the checkout session

Now, we can begin setting up the checkout session. To begin, we must define an endpoint on the server that handles the form submission request from the front end. This endpoint must match the value of the action attribute on the form element in the checkout.html file. For this tutorial, we will be using /create-checkout-session as shown:

// server.js
app.post("/create-checkout-session", async (req, res) => {
  const session = await stripe.checkout.sessions.create({
    line_items: [
      {
        // Provide the Price ID of the recurring product
        price: "PRICE_ID",
        quantity: 4,
      },
    ],
    mode: "subscription",
    success_url: `${YOUR_DOMAIN}/success.html`,
    cancel_url: `${YOUR_DOMAIN}/cancel.html`,
  });

  res.redirect(303, session.url);
});

The stripe.checkout.sessions.create method creates a Checkout Session. A Checkout Session controls what your customer sees on the payment page, such as line items, the order amount, and acceptable payment methods.

Within the line_items array, provide the price ID of the product for which you want to implement seat-based billing within the price field and the number of initial seats within the quantity field. Each object in the array represents a product in your product catalog. To create seat-based billing, the price IDs of all products in the array must have a recurring price.

Next, include the values of the success and cancel URLs. These values must match the HTML pages created earlier. Since seat-based billing is a type of recurring payment, we must set the billing mode to subscription using the mode parameter.

At this point, our server.js should look like this:

const stripe = require("stripe")("YOUR_STRIPE_SECRET_KEY");
const express = require("express");
const app = express();
app.use(express.static("public"));

const YOUR_DOMAIN = "http://localhost:4242";

app.post("/create-checkout-session", async (req, res) => {
  const session = await stripe.checkout.sessions.create({
    line_items: [
      {
        // Provide the exact Price ID (for example, pr_1234) of the product you want to sell
        price: "PRICE_ID",
        quantity: 4,
      },
    ],
    mode: "subscription",
    success_url: `${YOUR_DOMAIN}/success.html`,
    cancel_url: `${YOUR_DOMAIN}/cancel.html`,
  });

  res.redirect(303, session.url);
});

app.listen(4242, () => console.log("Running on port 4242"));

Now, we can test the application to ensure it works as desired. To begin:

Start the application server and navigate to http://localhost:4242/checkout.html.
Click the Checkout button to trigger the checkout session. This will redirect you to the Stripe Checkout page.
Fill in the required information and click the Subscribe button to create the subscription. This creates a new subscription for the product. You can find the new subscription on the Subscriptions page on your Stripe dashboard.

Upgrade and downgrade seats

Stripe allows us to manage the subscription by making updates after creating a new seat-based billing subscription.

However, since Stripe Checkout only supports creating a new subscription, we must use an alternative method for updating our seat-based billing subscription. For this tutorial, we’ll use Stripe’s customer portal. The customer portal allows customers to self-manage their payment details, invoices, and subscriptions.

To get started:

Go to the customer portal configuration page on the Stripe dashboard.
Click the Activate test link button.
Go to the Subscriptions menu on the portal configuration section.
Enable the Customers can switch plans and Customers can update the quantity of their plans options.
Next, select the product for which you created a subscription.
Finally, click the Save changes button to save the latest changes.

After completing these steps, Stripe will generate a customer portal link that your customers can use to manage their subscriptions.

To update a subscription:

Go to your customer portal link on a browser.
Enter your customer email to log in. The customer email you provide must be associated with an existing customer. Stripe automatically creates a new customer when a subscription is created, so you can use the customer email provided at checkout.
Click on the login link sent to the customer email provided.
Click the Update subscription button and then set the new number of seats to be purchased.
Click Continue > Confirm to confirm the update.

Note that in test mode, Stripe will only send authentication emails if you have a test mode customer associated with the email address provided.

Next steps

Using Stripe's Hosted Checkout and Customer Portal, you have learned how to create a new seat-based billing subscription and upgrade and downgrade seat count after purchase.

While this provides a solid foundation for per-seat billing, some notable limitations can create challenges for businesses managing seat-based subscriptions. For example, Stripe's proration invoices can be difficult to interpret, especially when customers upgrade or downgrade their seat count mid-cycle. The resulting invoice lacks clear line items that reflect standard industry practices.

Secondly, the simplicity of Stripe's approach forces businesses to build additional logic to enforce seat limits within their application. These gaps can lead to increased engineering effort and commitment for companies looking for a more robust and scalable solution.

How to Build an AI Agent for Voice-Enabled Payments with the Tether Wallet Development Kit

Dera Okeke — Tue, 04 Mar 2025 00:02:52 +0000

Voice-enabled payments are revolutionizing how users interact with financial applications, offering a seamless and intuitive way to manage transactions. Using Wallet Development Kit (WDK) by Tether, developers can create powerful, AI-driven solutions for modern blockchain payment.

In this guide, we'll walk through building an AI Agent for voice-enabled payments using the Tether WDK. You'll learn how to set up the development environment, understand the core components of the WDK, and implement a fully functional voice-enabled payment agent. By the end of this tutorial, you'll have a working example and the knowledge to expand its capabilities further.

Let's get right to it!

What is the Tether WDK?

The Tether WDK is a multi-asset cryptocurrency wallet library developed by Tether. It enables businesses and developers to seamlessly integrate advanced wallet-related functionalities and user experiences for Bitcoin and USD₮ into any website, app, or device.

The library is prebuilt with all the necessary components to create a wallet. These include:

Wallet seed: This component is a sublibrary within the Tether WDK library that handles BIP39 seed generation and management. The component is used to generate BIP39 seed phrases for all assets.
Wallet store: This component is a sublibrary within the Tether WDK library wallet that stores data. It supports multiple storage engine implementations, allowing developers to implement the storage engine that best suits their projects' needs.
Wallet indexer: This component is a remote blockchain data provider. It seamlessly integrates JSON-RPC and WebSocket APIs in the background to fetch and deliver real-time blockchain data efficiently.
Wallet test-tools: This component contains tools used for developing and testing the Tether WDK. It supports setting up test environments for both Bitcoin and Ethereum local networks.

In addition to the Tether WDK's prebuilt components, the library also supports the creation of custom components, allowing you to build components to address your use case.

Building the AI Agent

Now that we've learned about the Tether WDK and its components, let's use the library to build an AI agent for voice-enabled payments.

Step 1: Prerequisites

In this section, we'll go through all the steps, dependencies, and installations required for this tutorial. To follow along with the rest of this guide, you will need the following:

Fulcrum Electrum: Fulcrum is a fast and light Simplified Payment Verification (SPV) server for Bitcoin Cash, Bitcoin BTC, and Litecoin.
speaches: speaches is an OpenAI API-compatible server supporting streaming transcription, translation, and speech generation. An active speaches instance is required to follow along with this tutorial.
Ollama: Ollama is a popular Large Language Model (LLM) backend used to run and serve LLMs offline. An active Ollama instance is required for this tutorial.
NodeJS: You can download and install a NodeJS version with Long-Term Support (LTS) from the NodeJS download page. This installation adds NodeJS to your machine's directory and allows you to install dependencies with the Node Package Manager (NPM).
A code editor of choice.

With the environment setup completed, you can start setting up NodeJS for your project. To begin, create a new folder in a directory of choice and create an index.js file.

Next, you must initialize NodeJS in the newly created folder. To initialize NodeJS, run the following command at the root of the folder's directory:

npm init -y

This command creates a project scaffold with the node_modules folder and package.json file that contains our project's dependencies. Next, we need to download the Tether WDK. To do this, run the following command in the terminal at the root of your project's directory

npm install github:tetherto/lib-wallet#v0.0.1

This command pulls the Tether WDK library—lib-wallet—directly from the Github repository and updates both the node_modules folder and package_json file with the necessary dependencies.

Step 2: Render the UI

After installing all the dependencies and libraries, we can begin working on the UI. In this step, we'll create an example address book to enable us interact with our final application. We'll also define two JavaScript functions—renderAddressBook and renderAddresses— that enable us interact with the addresses on the screen. These functions set up the basic UI for recording audio, transcribing it, and performing wallet actions based on the transcription.

// Example address book
const book = {
  bob: {
    btc: "bcrt1qrfd2ujntu7la5vjqpjr69u8tc8rl6fxvx6hrzm",
  },
  alice: {
    btc: "bcrt1q7mm7seyccvf4dyc2je97zumh4aes7xhgetwc6m",
    eth: {
      usdt: "0x9ede22b627388b5db43c3488f27480b45d22d238",
    },
  },
};

function renderAddressBook(book) {
  const container = document.createElement("div");
  container.className = "address-book";

  for (const [name, addresses] of Object.entries(book)) {
    const personElement = document.createElement("div");
    personElement.className = "person";

    const nameElement = document.createElement("h3");
    nameElement.textContent = name;
    personElement.appendChild(nameElement);

    const addressList = document.createElement("ul");
    renderAddresses(addresses, addressList);
    personElement.appendChild(addressList);

    container.appendChild(personElement);
  }
  const node = document.getElementById("addr");
  node.appendChild(container);
}

function renderAddresses(addresses, parentElement, prefix = "") {
  for (const [key, value] of Object.entries(addresses)) {
    const listItem = document.createElement("li");

    if (typeof value === "string") {
      listItem.textContent = `${prefix}${key}: ${value}`;
    } else if (typeof value === "object") {
      listItem.textContent = `${prefix}${key}:`;
      const nestedList = document.createElement("ul");
      renderAddresses(value, nestedList, "  ");
      listItem.appendChild(nestedList);
    }

    parentElement.appendChild(listItem);
  }
}

You can add appropriate styling to these elements as required.

Step 3: Setup the wallet

Now, we can begin setting up the wallet to be used for the project. To do this, we will use Tether's WDK library. Before using the library, we must first import the required libraries into the index.js file. For this tutorial, we will import both the Tether WDK library and the Wallet seed sublibrary. To do this, copy the following snippet and paste at the top of your index.js file:

const Wallet = require("lib-wallet/src/wallet-lib");
const Bip39Seed = require("lib-wallet-seed-bip39");

Next, let's generate a seed phrase and define the wallet configuration. To generate a new seed phrase, we used the generate method within the Wallet seed library's Bip39Seed class:

// Generate a new phrase on page refresh
const PHRASE = await Bip39Seed.generate();

// Wallet config
const wconfig = {
  network: "regtest",
  electrum_host: "ws://localhost",
  electrum_port: "8001",
  token_contract: "0x959922bE3CAee4b8Cd9a407cc3ac1C251C2007B1",
  web3_indexer_ws: "ws://localhost/eth/hardhat/indexer/ws",
  web3_indexer: "http://localhost/eth/hardhat/indexer/rpc",
  web3: "ws://localhost/eth/hardhat/indexer/web3",
  seed: {
    mnemonic: PHRASE,
  },
};

Finally, let's define a function to initialize the wallet as shown:

async function initWallet() {
  renderAddressBook(book);
  const w = await Wallet.wallet(wconfig);

  await w.syncHistory({ all: true });
  document.getElementById("seed").textContent = w.seed.mnemonic;
  Wallet.demoWallet = w;

  return w;
}

When run, the initWallet function will initialize the wallet using the configuration defined in wconfig, sync the wallet's transaction history, and then display the generated seed phrase on the page.

Step 4: Wallet interactions

Now that we have successfully set up the wallet, we can define interactions that a user can perform on the wallet. To do this, we need to create a function that performs actions on the wallet based on the parsed transcription. The function also includes conditionals that determine the error messages if a user attempts to perform an unsupported action or use an unsupported asset.

async function walletAction(msg) {
  const wallet = Wallet.demoWallet;

  const asset = wallet.pay[msg.asset.toLowerCase()];
  if (!asset) return setStatus(`asset: ${msg.asset} is not supported`);
  if (!asset[msg.action])
    return setStatus(`action ${msg.action} not supported by wallet`);

  if (msg.args) {
    msg.args.fee = 10;
  }

  const res = await wallet.pay[msg.asset.toLowerCase()][msg.action](
    { token: msg.token?.toLowerCase() },
    msg.args
  );

  try {
    setStatus(JSON.stringify(res, null, 1));
  } catch (err) {
    console.log(err);
    setStatus("command failed");
  }
}

function setStatus(txt) {
  const statusDiv = document.getElementById("status");
  statusDiv.textContent = txt;
}

Step 5: Set up voice recording

The first phase involved in this step is to define constants for the active Ollama and speaches instances. To do this, copy the following lines of code and paste them into your index.js file. The link to each instance must be the same as the link on your local device.

// Edit these to your local instances
const SPEACHES = "http://localhost/whispr/audio/transcriptions";
const OLLAMA = "http://localhost:11434/api/chat";

Next, let's define a function that handles audio interaction on the app. The initMic function initializes the microphone, starts/stops recording, and then uploads and sends the audio to the SPEACHES service for transcription.

async function initMic() {
  let mediaRecorder;
  let audioChunks = [];
  let isRecording = false;

  const recordButton = document.getElementById("record");
  const audioPlayback = document.getElementById("audio");

  recordButton.onclick = toggleRecording;

  async function toggleRecording() {
    if (!isRecording) {
      await startRecording();
    } else {
      stopRecording();
    }
  }

  async function startRecording() {
    audioChunks = [];
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    const mediaRecorder = new window.MediaRecorder(stream, {
      mimeType: "audio/ogg; codecs=opus",
    });

    mediaRecorder.ondataavailable = (event) => {
      audioChunks.push(event.data);
    };

    mediaRecorder.onstop = async () => {
      const audioBlob = new Blob(audioChunks, {
        type: "audio/ogg; codecs=opus",
      });
      const audioUrl = URL.createObjectURL(audioBlob);
      audioPlayback.src = audioUrl;

      await uploadAudio(audioBlob);
    };

    mediaRecorder.start();
    isRecording = true;
    recordButton.classList.add("recording");
    setStatus("Recording.... (Press again to stop recording)");
  }

  function stopRecording() {
    mediaRecorder.stop();
    isRecording = false;
    recordButton.classList.remove("recording");
  }

  async function uploadAudio(audioBlob) {
    const formData = new FormData();
    formData.append("file", audioBlob, "recording.ogg");
    formData.append("language", "en");

    setStatus("Uploading...");
    const response = await fetch(SPEACHES, {
      method: "POST",
      body: formData,
    });

    if (response.ok) {
      const result = await response.json();
      setStatus(`transcribed: ${result.text}. processing ....`);
      parseTranscribe(result.text);
    } else {
      setStatus("Upload failed");
    }
  }
}

Step 6: Parse and process audio transcriptions

Upon retrieving the transcribed text, we can send it to the Ollama LLM instance service for further processing. It expects a structured JSON response. Therefore, we must parse the returned data before sending it for processing, as shown:

async function parseTranscribe(txt) {
  const data = {
    model: "llama3.1",
    stream: false,
    messages: [
      {
        role: "user",
        content: `${JSON.stringify({
          asset: "",
          token: "",
          action: "",
          args: {
            amount: "",
            unit: "",
            address: "",
          },
          addressBook: book,
          text: txt,
        })}`,
      },
    ],
  };

  let msg;
  try {
    const response = await fetch(OLLAMA, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify(data),
    });

    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }

    const result = await response.json();
    msg = JSON.parse(result.message.content);
  } catch (error) {
    console.error("Error:", error);
    throw error;
  }
  walletAction(msg);
}

Now, we can bring all the above steps together. But first, we need to define an asynchronous function—main—that calls the initWallet and initMic functions. Then, we call the main function when the DOM is fully loaded.

async function main() {
  await initWallet();
  document.getElementById("record").textContent = "Record";
  initMic();
}

document.addEventListener("DOMContentLoaded", function () {
  main();
});

Conclusion

By following the steps outlined in this tutorial, you've covered the following:

Building an AI Agent for voice-enabled payments using the Tether WDK.
Setting up the development environment, configuring essential components, and integrating the wallet library to handle blockchain transactions.
Implemented voice commands to streamline user interactions using an AI agent.

With this foundation, you can extend the AI Agent to support more advanced use cases and customize features to meet specific requirements.

The possibilities are endless—happy building!

Data Visualization: How to Create Styled Cryptocurrency Candlesticks with Highcharts

Dera Okeke — Sat, 28 Dec 2024 15:10:06 +0000

What is Data Visualization?

Data visualization is the practice of representing data/information in pictorial or graphical formats. It is a means by which large data sets or metrics are converted into visual elements like maps, graphs, and charts, which are much more appealing to an end-user.

The JavaScript ecosystem currently has several reliable, first-rate libraries for data visualization. Some of which include D3.js, Highcharts, Charts.js, Rechart, etc. However, in this article, we will be using Highcharts to create our charts.

Highcharts is a JavaScript library for creating SVG-based, responsive, and interactive charts for web and mobile. It provides deep customization of charts via JavaScript or CSS. Highcharts offers four product categories for creating charts. These include:

Highcharts: This is the basic Highcharts module that is required in all charts. It can be used to create simple line, bar and pie charts.
Highcharts Stock: This is used for creating general stock and timeline charts for your applications. Some examples include simple Stock charts, Candlesticks & Heikin-Ashi, OHLC. You can also make use of the Stock Tools module which provides a GUI that permits interaction with charts.
Highcharts Maps: Highcharts also provides an option to generate schematic maps which allow developers add interactive, customizable maps to their web application. It offers options to either use map collection provided by Highcharts or create custom SVG maps to suit your purpose.
Highcharts Gantt: This is a special type of bar chart used for illustrating project schedules.

We will use the Highcharts Stock to create styled candlesticks with oscillators and technical indicators provided by the Stock Tools module.

Creating the Candlestick

A candlestick chart( or Japanese candlestick chart) is a style of financial chart used by traders to determine possible price movements of a stock, security, or currency based on previous patterns. It makes use of key price points or OHLC(open, high, low, close) values taken at regular intervals for a specified period of time.

Not to be confused with the typical candlestick chart is the Heikin-Ashi('average bar') chart. Although identical to the candlestick chart, it is mostly used in conjunction with the candlestick as it helps make candlestick chart trends easier to analyze. Hence, making it more readable.

The Highcharts API provides options for creating both candlestick charts and Heikin-Ashi charts. This article focuses on candlestick charts; however, I will point out the tradeoffs required for creating an Heikin-Ashi chart along the way. Let's get our hands dirty, shall we?!

Getting Started

To begin using Highcharts, we must first download Highcharts. Highcharts provides several options to introduce Highcharts into your project. You can choose to either:

Download the entire Highcharts library. Depending on your use case, you can also download the Highcharts Stock, Maps, or Gantt libraries.
Install Highcharts via NPM and import as modules. These are best for Single Page Applications like React and Vue.
Use the Highcharts CDN to access files directly.

We will be using the Highcharts CDN for this article.

The HTML

The bulk of the HTML contains script tags used to load the Highcharts CDN. The first three are required modules for all stock charts created with Highcharts.

<script src="https://code.highcharts.com/stock/highstock.js"></script>
<script src="https://code.highcharts.com/stock/modules/data.js"></script>
<script src="https://code.highcharts.com/stock/modules/exporting.js"></script>

Unlike candlestick charts, if you need to create a Heikin-Ashi chart, you will need to bring in the module separately as below:

<script src="https://code.highcharts.com/stock/modules/heikinashi.js"></script>

The final CDN we need to bring into our application is the Stock Tools module. This enables us to make use of technical indicators. The Stock Tools module must be loaded last so it can pick up all available modules from above.

<script src="https://code.highcharts.com/stock/indicators/indicators-all.js"></script>

Rather than loading all technical indicators from the Stock Tools module, you can also load specific indicators depending on your needs:

<script src="https://code.highcharts.com/indicators/indicators.js"></script>
<script src="https://code.highcharts.com/indicators/rsi.js"></script>
<script src="https://code.highcharts.com/indicators/ema.js"></script>
<script src="https://code.highcharts.com/indicators/macd.js"></script>

Lastly, we need to create an HTML element to hold our chart that we can reference from the JavaScript:

<div id="container" style="height: 500px; min-width: 310px"></div>

The JavaScript

Bringing in our Data
The first item on our itinerary is to bring in the data we will be plotting. Highcharts provides a .getJSON method similar to that of jQuery, which is used for making HTTP requests. It also provides a stockChart class for creating the chart. The stockChart class takes in two parameters:

The first parameter, renderTo, is the DOM element or the id of the DOM element to which the chart should render.
The second parameter, options, are the options that structure the chart.

Highcharts.getJSON('https://api.coingecko.com/api/v3/coins/ethereum/ohlc?vs_currency=usd&days=365', 
function (candlestick) {
  // create the chart
  Highcharts.stockChart('container', {
    title: {
      text: 'Untitled Masterpiece'
    },

    series: [
      {
        type: "candlestick",    //heikinashi for Heikin-Ashi chart
        name: "Ethereum",      //chart name
        id: "eth",             // chart id, useful when adding indicators and oscillators
        data: candlestick,      //data gotten from the API call above
      },
    ], 

yAxis: [
      {
        height: "100%",       // height of the candlestick chart
        visible: true,  
      }
    ]
  });
});

The above code gives us a simple candlestick with basic styling provided by Highcharts.

Stock Tools

The Highcharts Stock Tools is an optional feature in Highcharts. You can either enable the entire Stock Tools Graphical User Interface(GUI), which allows users to add indicators and oscillators based on their needs, or add specific Stock Tools to your web app via Javascript.

We will add an indicator(Acceleration bands) and an oscillator(awesome oscillator) to our chart. To do this, we need to edit both the series and yAxis objects above:

series: [
      {
        type: "candlestick",
        name: "Ethereum",
        id: "eth",           // chart id, useful when adding indicators and oscillators
        data: data,
      },
         {
        type: "abands",      //acceleration bands indicator
        id: "overlay",       // overlays use the same scale and are plotted on the same axes as the main series.
        linkedTo: "eth",    //targets the id of the data series that it points to
        yAxis: 0,           // the index of yAxis the particular series connects to
      },
      {
        type: "ao",          // awesome oscillator
        id: "oscillator",    // oscillators requires additional yAxis be created due to different data extremes.
        linkedTo: "eth",    //targets the id of the data series that it points to
        yAxis: 1,           // the index of yAxis the particular series connects to
      },
    ],
    yAxis: [
      {
        //index 0
        height: "80%",      //height of main series 80%

        resize: {
          enabled: true,     // allow resize of chart heights
        },
      },
      {
        //index 1
        top: "80%",         // oscillator series to begin at 80%
        height: "20%",      //height of oscillator series
      },
    ],

Here is what we have now:

Styling the Chart

Before we can begin styling the chart, we need to understand first the different parts that make up the chart.

Highcharts provides two ways of styling charts:

Highcharts.CSSObject: This is the default method of styling charts. It builds upon the options object within the stockChart class provided by Highcharts to define the visual appearance of individual SVG elements and HTML elements within the chart.
styledMode: boolean: This defaults to false. However, when in styled mode, no presentational attributes are applied to the SVG via the options object. Hence, CSS rules are required to style the chart.

We will be making use of the Highcharts default styling for this article. Therefore, within the options object:

{
     // general chart styles
     chart: {
      backgroundColor: "#1c1b2b",       
      borderRadius: 15,
      height: 500,
      styledMode: false,            //if true, CSS rules required for styles
    },

    // config styles for the title
    title: {
      text: "Candlestick & Awesome Oscillator", 
      style: {
        color: "#fff",
      },
    },

    // styles for range selectors
    rangeSelector: {
       // styles for range selector buttons
       buttons: [ 
        {
          type: "month",
          count: 1,
          text: "1m",
          title: "View 1 month",
        },
        {
          type: "month",
          count: 4,
          text: "4m",
          title: "View 4 months",
        },
        {
          type: "month",
          count: 8,
          text: "8m",
          title: "View 8 months",
        },
        {
          type: "ytd",
          text: "YTD",
          title: "View year to date",
        },
        {
          type: "all",
          count: 1,
          text: "All",
          title: "View All",
        },
      ],
     selected: 2,            //The index of the button to appear pre-selected on page load.
      buttonTheme: {
        fill: "none",
        stroke: "none",
        "stroke-width": 0,
        r: 8,
        style: {
          color: "#4F6C89",
          fontWeight: "bold",
        },

        states: {
        // styles for different button states; hover, active etc
          select: {
            fill: "transparent",
            style: {
              color: "#D76F2A",
            },
          },
        },
      },

       // styles for range selector input
      inputBoxBorderColor: "#4F6C89",
      inputBoxWidth: 110,
      inputBoxHeight: 18,
      inputStyle: {
        color: "#4F6C89",
        fontWeight: "bold",
      },
      labelStyle: {
        color: "#cbd1d6",
        fontWeight: "bold",
      },
    },

    // The plotOptions is a wrapper object for config objects for each series type. Main structure/ styling can be defined here
    plotOptions: {
      series: {
        marker: {
          enabled: false,         //disables point markers on chart
        },
      },

      // config style for candlestick series
      candlestick: {                  
        lineColor: "#FB1809",       // The line color of the candlestick.
        color: "#FB1809",            // The default fill color candlestick. Can be used for falling candlesticks
        upColor: "#4EA64A",            //The fill color of the candlestick when values are rising.
        upLineColor: "#4EA64A",        // The line color for rising candlesticks.
      },

     // config style for acceleration bands indicator
      abands: {
        lineWidth: 1,
        lineColor: "#20a0b1",
        bottomLine: {
          styles: {
            lineWidth: 0.5,
            lineColor: "#fcfc27",
          },
        },
        topLine: {
          styles: {
            lineWidth: 0.5,
            lineColor: "#2efc27",
          },
        },
      },
    },

   // config styles for xAxis
    xAxis: {
      lineWidth: 0.1,
      tickColor: "#2f2952",
      crosshair: {
        color: "#8e8aac",
        dashStyle: "dash",
      },
    },

   // styles and config for yAxis
    yAxis: [
      {
        gridLineColor: "#201d3a",
        lineWidth: 0,
        visible: true,
        height: "80%",
        crosshair: {
          dashStyle: "dash",
          snap: false,
          color: "#696777",
        },
        labels: {
          align: "right",
          x: -2,
        },
        resize: {
          enabled: true,
          lineWidth: 2,
          lineColor: "#1d1c30",
        },
      },

      {
        top: "80%",
        height: "20%",
        gridLineColor: "#201d3a",
      },
    ],

   // styles and config for tooltip
    tooltip: {
      split: true,
      shape: "rect",
      shadow: false,
      valueDecimals: 2,

      // position tooltip
      positioner: function (width, height, point) {
        var chart = this.chart,
          position;

        if (point.isHeader) {
          position = {
            x: Math.max(
              // Left side limit
              0,
              Math.min(
                point.plotX + chart.plotLeft - width / 2,
                // Right side limit
                chart.chartWidth - width - chart.marginRight
              )
            ),
            y: point.plotY,
          };
        } else {
          position = {
            x: point.series.chart.plotLeft,
            y: point.series.yAxis.top - chart.plotTop,
          };
        }

        return position;
      },
    },

    // disable stocktools GUI
    stockTools: {
      gui: {
        enabled: false,
      },
    },

    // config styles for navigator
    navigator: {
      enabled: true,
      height: 50,
      margin: 10,
      outlineColor: "#8380a5",
      handles: {
        backgroundColor: "#8380a5",
        borderColor: "#e9d5d5",
      },
      xAxis: {
        gridLineColor: "#8380a5",
      },

    },

   // config styles for scrollbar
    scrollbar: {
      barBackgroundColor: "#8380a5",
      barBorderColor: "#8380a5",
      barBorderRadius: 8,
      buttonArrowColor: "#fff",
      buttonBackgroundColor: "#405466",
      rifleColor: "#fff",
      trackBackgroundColor: "#e9d5d5",
    },

    // disable credits
    credits: {
      enabled: false,
    },
  }

This is what our final chart becomes:

Conclusion

Creating styled cryptocurrency candlesticks with Highcharts allows you to transform raw data into visually compelling and actionable insights. By leveraging Highcharts’ flexibility, you can customize candlestick charts to align with your branding, enhance user experience, and effectively communicate market trends. Whether you're building a financial dashboard or enhancing a trading platform, the ability to design and implement tailored visualizations is a critical skill in today’s data-driven landscape.

With the steps outlined in this guide, you now have a foundation for working with Highcharts to create dynamic candlestick charts. Explore additional customizations and experiment with Highcharts’ extensive API to bring your cryptocurrency visualizations to the next level.

Eliminate CORS Policy Errors on a Frontend Application

Dera Okeke — Mon, 10 Oct 2022 10:44:29 +0000

You just signed up for an API service and have been provided with your unique API key to work with. You've tested calls to the endpoints with Postman or VSCode's Thunder Client extension, and everything seems to work fine. Now, it's time to incorporate this in your frontend application, and you get hit with an unexpected error from your browser console:

This error on the browser can be a real pain when working with APIs, especially for a new developer. This article provides a clear understanding of what CORS policy errors are and shows ways to avoid them when developing applications.

Understanding the error

Cross-Origin Resource Sharing (CORS) is a mechanism that allows a server to control which domains (or origins) outside its own from which a browser may allow resource requests. This mechanism is implemented in line with the same-origin policy — a security model whereby a browser can permit scripts in a document or webpage to interact with resources from another webpage only if both web pages are of the same origin. An advantage of this is that it combats malicious attacks by preventing scripts on one page from accessing sensitive information on another page and relaying this data to the attacker.

Two URLs have the same origin if the protocol, port (if specified), and host are the same for both.

CORS serves as a way to relax a browser's same-origin policy by allowing cross-origin requests from subdomains and trusted third parties. When a request is made from a browser to an API endpoint, the browser sends a preflight request to its server looking for permissions. The preflight request precedes the actual request to the endpoint and checks whether or not this request will be allowed by the server. It contains information about the actual request, such as HTTP headers and methods, and determines if the browser should send the actual request by assessing the response from the server.

A CORS policy error like the image above is returned as a response to the preflight request sent by the browser stating that one of the required HTTP request headers — Access-Control-Allow-Origin — is missing. So, how do we fix this?

Finding a solution

There are several ways to fix CORS policy errors encountered in an application. The quickest of these is to install a browser extension that includes the Access-Control-Allow-Origin option to the request header. There are extensions available in the Chrome Web Store that allow browsers to perform seamless cross-domain requests in a web application when enabled. However, it is misleading to solve CORS errors this way because the extension merely tricks the browser into thinking the Access-Control-Allow-Origin header is included in the request when that is not the case. Hence, the extension only fixes the issue while enabled on your browser.

Fixing CORS errors the right way

Since the solution described above only temporarily fixes the CORS error by including the missing request header, it is not a recommended fix because this request header remains missing on browsers without the extension, leading to the same errors. Another way to fix CORS errors is to send all requests to the endpoint through a proxy server. The proxy acts as an intermediary server between the client and the endpoint server and includes the Access-Control-Allow-Origin header on all requests to the endpoint. When a request is made from a browser to the API endpoint, the browser immediately sends the request to the proxy server. The proxy server adds the CORS headers to the request and then forwards it to the actual endpoint server.

There are a few free CORS proxies available for use on the internet. However, it is not advisable to use these proxies on production applications due to security, speed, and performance concerns. A more commonly used and accepted alternative is to build an application's proxy from scratch. Building your proxy offers the same benefits as the free CORS proxies while also eliminating the security and latency concerns associated with them. In this article, you will learn how to create your own proxy server to fix CORS policy errors.

Requirements

In this section, I will show you all the steps and installations required to build this in your application. To follow along with the rest of this guide, you will need the following:

Node and npm enabled on your device.
An API resource that restricts cross-origin HTTP requests. For this guide, I'll be using the CoinMarketCap API.
A frontend application with npm initialized.

If you do not have npm on your project, you must initialize it by running the command below at the root of your project's directory:

npm init

This command creates a project scaffold with the package.json file that contains our project's dependencies. Next, we need to download some packages to enable us to create the proxy server and make requests from it. These packages are:

express: This is a minimalistic back-end web application framework for Node.js.
cors: A Node.js package that enables CORS in an Express middleware.
axios: This is a promise-based HTTP client for Node.js. This is my preferred choice for making requests but feel free to use any alternative you desire.

All required packages are available on the npm package registry. To download them to your project, run the command below at the root of your project's directory:

npm i express cors axios

Getting started

Before we begin building our proxy, let's take a look at the current request we're making to the endpoint from our application.

  const options = {
   method: "GET",
   url: "https://pro-api.coinmarketcap.com/v2/cryptocurrency/quotes/latest",
   headers: {
      "X-CMC_PRO_API_KEY": process.env.REACT_APP_MARKET_CAP_KEY,
    },
   params: {
     slug: "bitcoin,ethereum,band-protocol,tezos",
    },
  };

  Axios.request(options)
    .then((response) => {
      console.log(response.data);
    })
    .catch((error) => {
      console.log(error);
    });

The snippet above makes a client-side request with JavaScript from our frontend application. Certain API providers — like ours — tend to block requests of this nature for several reasons, usually to protect their users' API keys since they can still be accessed on the browser. This request returns an error that we can view in the "Console" tab of the browser's Developer Tools.

Now, let's start creating our proxy server. Begin by creating a new file named index.js in your project's root directory. This file will contain all the Node.js code for the server. Next, paste the following code in the index.js file created:

const PORT = 8080;
const express = require("express");
const cors = require("cors");
const axios = require("axios");

require("dotenv").config();

const app = express();

app.use(cors());

app.get("/", (req, res) => {
  const options = {
    method: "GET",
    url: "https://pro-api.coinmarketcap.com/v2/cryptocurrency/quotes/latest",
    headers: {
      "X-CMC_PRO_API_KEY": process.env.REACT_APP_MARKET_CAP_KEY,
    },
    params: {
      slug: "bitcoin,ethereum,band-protocol,tezos",
    },
  };

  axios
    .request(options)
    .then((response) => {
      res.json(response.data);
    })
    .catch((error) => {
      res.json(error);
    });
});

app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

Let's go over the code above, shall we?

First, we define a port for the server to run on our local machine. Ours is set to port 8080.
Next, we bring in all the required modules and assign them to constants.

require("dotenv").config();

The line above is required to load environment variables defined in a .env file. To use this in your application, you must install the dotenv module as seen on the npm registry.
Next, we initialize the express package and assign it to a constant called app. Then, we enable all CORS requests by calling the cors package inside the app.use method.
Below that, we make an HTTP request similar to our previous client-side request inside the app.get method. This method lets us define a route handler for GET requests. Ours is set to the root route —/— in the snippet above.
Lastly, the app.listen method is used to bind and listen for connections on the specified port — http://localhost:8080/.

Now that we have the server setup, we must create a start script for starting the server. Go to the package.json file at the root of your project directory and add the line below to the scripts object:

"start:backend": "node index.js",

This defines the start script for running the server locally. Depending on your original setup, your package.json file should look something like this:

{  
  "name": "project_name",  
  "description": "project_description",  
  "scripts": {  
    "start": "react-scripts start",
    "start:backend": "node index.js"
  },  
  "dependencies": {
    "axios": "^0.21.0",
    "cors": "^2.8.5",
    "dotenv": "^16.0.1",
    "express": "^4.18.1"
  }
}

Notice all the installed packages listed as dependencies. Also, notice there are separate start scripts for the frontend and backend to prevent any confusion.

To start up the server, run the command below in your editor's terminal:

npm run start:backend

This command spins up the backend server at the defined port. We can view the response from the server in JSON format by visiting the port URL on a browser— http://localhost:8080/.

Now, we can make GET requests to the server URL on our frontend rather than making requests to the API endpoint directly. Let's head over to our previous front-end request to implement this change:

const options = {
  method: "GET",
  url: "http://localhost:8080/",
};

Axios.request(options)
  .then((response) => {
    console.log(response.data);
  })
  .catch((error) => {
    console.log(error);
  });

Unlike our earlier client-side request that returns an error, the above request returns the data from the API, which we can view in the browser console or use in our application.

Conclusion

CORS errors can be a real pain for front-end developers. However, understanding the underlying browser behavior helps us to properly visualize our approach to solving them. In this post, I briefly explained what CORS errors are, why they occur, and discussed possible ways of solving these errors. Finally, I implemented the recommended solution to fix CORS issues on a frontend application.

Authentication and Authorization in Firebase

Dera Okeke — Thu, 06 Oct 2022 11:14:10 +0000

Introduction

The importance of developing secure applications cannot be overstated these days. Users trust applications with their most sensitive data and expect it to be protected.

When building applications, app security should be a top priority. Developers must confirm users' identities and ensure that users can only access data they're allowed.

On this week's 8weeks of Firebase series, I'll cover Firebase's most fundamental offering — authentication.

What is Firebase

Firebase is a Google-backed Backend-as-a-Service (BaaS) solution that provides developers with a tool kit to build, improve, and grow their applications. Backend-as-a-Service solutions like Firebase eliminate the need for developers to build and manage various back-end services and focus solely on application development. These services include authentication, file storage, databases, hosting, testing, performance monitoring, analytics, and so on.

Authentication in Firebase

Authentication identifies and verifies users of an application. Firebase Authentication utilizes easy-to-use SDKs and ready-made UI libraries to authenticate users and create personalized experiences for users across all devices.

Under the hood, Firebase Authentication uses JSON Web Tokens (JWTs) to authenticate users. It supports authentication mechanisms like email/password and email/link combinations. It also allows phone number authentication and supports social logins with popular federated identity providers like Google and Github. In this tutorial, I will focus only on Google Authentication.

Firebase Authentication allows us to incorporate Google sign-in into our authentication workflow, allowing users to sign in with their Google accounts. To enable this on our application, we must first enable Google Authentication on the Firebase project console:

Head to the "Authentication" section on the Firebase console.
Click on the "Add New Provider" button on the "Sign-in providers" panel.
Select "Google" from the list of native providers shown.
Finally, enter a project support email in the popup displayed and hit "Save".

Next, we must create separate instances for Firebase auth and the Google auth provider object to utilize this sign-in method in our application. Finally, we call the signInWithPopup function and pass in the Firebase Auth instance as its first parameter and the Google auth provider instance as the second

import { getAuth, signInWithPopup, GoogleAuthProvider } from "firebase/auth";

const auth = getAuth();
const provider = new GoogleAuthProvider();

signInWithPopup(auth, provider)
  .then((result) => {
    // do something
  }).catch((error) => {
   // handle error
  }

Once authenticated, we can listen for users' authentication states in real-time and also make requests to other Firebase services with users' unique IDs.

Authorization in Firebase

Upon validating the identity of the user accessing resources on our application, it is also necessary to control what operations they can perform. Authorization is the process of specifying the users' privileges and controlling what resources they access. Firebase secures and controls access to every backend service using security rules.

Firebase Security Rules stand between your data and users, ensuring that your application's users can only read and write data they're allowed. Firebase Security Rules allow us to use an expression language to determine how an end user is allowed to access data in Firebase products. It works by matching a pattern against database or bucket paths and then applying custom conditions to allow read and write access to data at those paths.

Using Firebase Authentication, we can access users' authentication information and write security rules based on this. For example, the rules below define read and write operations for a path, users/<uid>, where the uid subpath could be the unique identifier of any user on our application:

{
  "rules": {
    "users": {
      "$uid": {
        ".read": true,
        ".write": "$uid === auth.uid"
      }
    }
  }
}

In the above, we allow all read operations to the users/<uid> path, but grant write access only to users whose uid matches the subpath value.

Conclusion

Authentication and authorization are essential components of any secure application. It is necessary to implement good authentication and authorization strategies to protect your application's users and their data from attackers. Firebase provides mechanisms that protect all backend services, making sure that only users can see their own data. In this post, I briefly discussed Firebase Authentication, demonstrated an implementation of Google sign-in, and finally, I discussed securing our application's data using Firebase Security Rules.

To learn more about authentication and authorization in Firebase, check out my Firebase course on Educative.

New Course Alert

Dera Okeke — Wed, 28 Sep 2022 13:34:16 +0000

Hello all,

I'm THRILLED to announce that my Firebase course is now live on Educative, Inc. This course took a lot of time to create, and I am extremely proud of it.

It truly was an exciting experience for me, and I thoroughly enjoyed working with the team at Educative Inc to create this course. It was also a huge learning opportunity for me as I found myself looking through Docker and tmux documentation multiple times, haha!

The course is tailored for beginner-intermediate Firebase developers and contains all you need to know to get started with Firebase. It comprehensively discusses Firebase services and demonstrates the usage and implementation of these services in a real-world application using Educative's in-app SPA widget.

Kindly like, share, and recommend it to anyone who wants to learn Firebase!

Link to course

Styled Market Depth Chart

Dera Okeke — Thu, 11 Nov 2021 10:22:47 +0000

Beautifully styled market depth chart.

Highcharts: Styled Candlestick vs Heikin Ashi

Dera Okeke — Thu, 11 Nov 2021 10:01:08 +0000

Beautifully styled stock chart showing the differences between Candlesticks and Heikin Ashi(Average Bar) charts on Bitcoin over the past 14 days.

Highcharts: Styled Candlestick & Awesome Oscillator

Dera Okeke — Tue, 09 Nov 2021 08:45:03 +0000

Beautifully styled candlestick with awesome oscillator and acceleration bands

Highcharts: Styled Heikin Ashi with Bollinger Bands

Dera Okeke — Tue, 09 Nov 2021 08:43:41 +0000

Beautifully styled Heikin Ashi with Bollinger Bands.

Highcharts: Styled OHLC with MACD Oscillator

Dera Okeke — Tue, 09 Nov 2021 08:42:04 +0000

Beautifully styled OHLC chart with MACD oscillator

Forem: Dera Okeke

How to Dynamically Scrape Data from a Webpage with a "Load More" Button

Step 1: Prerequisites

Step 2: Get Access to the Content

Step 3: Load More Products

Step 4: Parse Product Information

Step 5: Export Product Information to CSV

Conclusion

How to Implement Seat-based Billing with Stripe Checkout

Prerequisites

Step 1: Install the Stripe library

Step 2: Building the UI

Create the checkout.html

Create the success and cancel pages

Step 3: Setup the server

Create the checkout session

Upgrade and downgrade seats

Next steps

How to Build an AI Agent for Voice-Enabled Payments with the Tether Wallet Development Kit

What is the Tether WDK?

Building the AI Agent

Step 1: Prerequisites

Step 2: Render the UI

Step 3: Setup the wallet

Step 4: Wallet interactions

Step 5: Set up voice recording

Step 6: Parse and process audio transcriptions

Conclusion

Data Visualization: How to Create Styled Cryptocurrency Candlesticks with Highcharts

What is Data Visualization?

Creating the Candlestick

Getting Started

The HTML

The JavaScript

Stock Tools

Styling the Chart

Conclusion

Eliminate CORS Policy Errors on a Frontend Application

Understanding the error

Finding a solution

Fixing CORS errors the right way

Requirements

Getting started

Conclusion

Authentication and Authorization in Firebase

Introduction

What is Firebase

Authentication in Firebase

Authorization in Firebase

Conclusion

New Course Alert

Styled Market Depth Chart

Highcharts: Styled Candlestick vs Heikin Ashi

Highcharts: Styled Candlestick & Awesome Oscillator

Highcharts: Styled Heikin Ashi with Bollinger Bands

Highcharts: Styled OHLC with MACD Oscillator

Create the `checkout.html`