Forem: Tyler Reicks

What is data scraping?

Tyler Reicks — Thu, 20 Oct 2022 18:01:38 +0000

Suppose you want to get large amounts of information from a website as quickly as possible. How can this be done? In this article, we will talk about data scraping and how to scrape the web. Additionally, we'll get into what data scraping is, why you would want to do it, how data scrapers work, and lastly, we'll go over different processes for scraping the web. I'll also include a quick example to reference.

What is data scraping?

If you've ever copied and pasted content from a website into a different location, you are doing a very manual version of data scraping. In this article, we will be using software applications to do the data scraping for us.

Data scraping is the process of using an application to extract valuable information from a website. This will allow us to obtain large amounts of data from websites in a short amount of time. Many of the larger websites like Google, Facebook, and GitHub have APIs that allow you to access their data. This is super convenient because the data will be given to you in a structured format that is easy to consume.

Unfortunately, that isn't always the case. There will be other times when you will have to collect content and data for your specific use case. This is where web and data scraping applications come in handy. You can program these scraping applications to visit websites and extract the content/data that you want. The obvious benefit of this is being able to get the precise data that you want easily and efficiently.

Data scraping is comprised of two parts, the crawler, and the scraper. The crawler is the algorithm that we can create to browse the web and find the exact data that we want. An example of this would be navigating to a specific website and clicking on the page where the content you want exists. Once you have found that data, we will utilize the scraper. The scraper is used to "scrape" the data from the website. With the scraper, you can detect the data points you want and export them to a format that would work best for you.

Once the data has been exported the fun can begin. You are free to use that data however you need.

Use cases for data scraping

While reading this article you've probably wondered, "what are some good use cases for web/data scraping?" Let's go over a couple of these use cases.

The first one that we will be talking about is my favorite, price monitoring. You can use price monitoring to keep track of prices and make sure you are finding the best deal. Who doesn't want to save money? I have written a previous blog post about scraping Amazon.com to monitor the prices of specific products. Companies can also use price scraping to see what competitors are pricing similar products. This provides them with an
advantage of being able to provide optimal pricing for their products so they can obtain maximum revenue.

Contact scraping is another way data scraping is used. Many companies and individuals can scrape the web for contact information to use for e-mail marketing. A good example of this would be scraping locations like an online employee directory or a bulk mailing list. This use case is very controversial and often requires permission to collect this kind of data. If you have ever visited a website and given them access to your contact information in exchange for using their software, you have permitted them to collect personal data like your e-mail address and phone number.

The last use case we'll go over is news monitoring. Many individuals and companies can scrape news sites to stay current on stories and issues relevant to them. This could be especially useful if you are trying to create a feed of some type, or if you just need to keep up with day-to-day reports.

How do data scrapers work?

Next, let's go over how data scrapers work. Scrapers can take all the content on web pages or just the specific data that you want. In many situations, it is best to pinpoint the specific data you want so that the data scraper can quickly extract it. For example, in the Amazon web scraping blog post that I mentioned earlier, we look at pricing for office chairs. In that instance, we are only trying to identify the price of the chairs and the title of the item. That allowed the data scraper to swiftly filter out any unneeded clutter resulting in the script being run relatively fast.

Now when the process of scraping a site first begins there needs to be a URL that is provided to the script or application software. Based on the provided URL the scraper will navigate to that web page. Next, it will load the HTML code of that site. After that code has been loaded, the scraper can then begin to collect the data that is wanted/needed. Lastly, the collected data is outputted in a predefined format determined by the user. Usually, this is a JSON file, but it can also be saved in other formats like an excel spreadsheet or a CSV file.

Stepping through the scraping process

Now that we know how a data scraper functions let's identify some preliminary steps that are needed before you try to scrape a website yourself. There are many cool tools and software applications out there that help with scraping websites. Because of this we will stay at a high level and focus on the basics.

First, you want to find the URLs you want to scrape. This might seem obvious, but it is a key factor and how well your data scrape will work. If the URL you give the scraper is even slightly incorrect the data you get back will not be what you want, or even worse, your scraper won't work at all. For instance, if you're trying to do a price monitoring data scrape you want to make sure that your URL goes to a relevant site like Amazon or Google shopping.

Secondly, you want to inspect the webpage (f12 on most keyboards) to identify what your scraper needs to scrape. If we use the same Amazon price monitoring example, you could go to a search result page on Amazon, inspect that page, and locate where the price is in the HTML code.

Once you have that, you want to identify the unique tags that are around the price so you can use that in your data scraper. Some good tags would be div tags with IDs or very specific class names.

<div id="price">$1.99</div>

After finding the code you want to collect and use, you'll want to incorporate this into your data scrape. This could be writing a script with the IDs or class names that you found in the previous step or simply inputting the tags into scraping software. You'll also probably want to add supporting information to help when displaying your data. Sticking with our Amazon example, if you are collecting the price of office chairs on Amazon it would be nice to also have the title of the item to accompany the price.

Once you've specified the tags in your script or scraping application, you'll want to execute the code. This is where all the magic happens. Everything that we talked about in the above section about how data scrapers work comes into play here.

Hopefully, you now have the data you need to start building your application. Whether that be a dashboard of charts, a cool table, or a sweet content feed the data is yours to do with it what you like. Lots of times you may get data back that you don't expect. That is completely normal. Just like anything else in the engineering world, if one tiny thing is off it can often lead to things being incorrect. Don't get discouraged! Practice makes perfect and you will catch on.

Data scraping with Browserless

All the basics have been covered for scraping the web. Before we end, I want to mention a cool tool that allows you to do data scraping. Browserless is a headless Chrome browser as a service. You can use Browserless with libraries like puppeteer or Selenium to automate web-based tasks like data collection. To learn more, make sure to visit the Browserless website where you can find blog posts, documentation, debuggers, and other resources.

Data Scraping Example with Browserless

In this example we are going to do a simple data scrape of the Y Combinator Hacker News feed. You can also run this example in the Browserless debugger tool. For this, we will use two main tools, Puppeteer and Browserless. In the above paragraph I mentioned these tools with corresponding links. I highly recommend you check them out before diving into the example.

Initial setup

Alrighty, let's get to it! We are going to start with the initial setup. Luckily for us, there aren't many dependencies we need to install. There's only one... Puppeteer.

npm i puppeteer

Once you've run that command you are good to go! 👍

Browserless setup

Let's look at how to set up Browserless real quick.

For starters, you'll need to set up an account. Once you get your account set up, you'll be directed to your Browserless dashboard.

Here you can find your session usage chart, prepaid balance, API key, and all of the other account goodies.

Keep your API key handy as we'll use that once we start writing our script. Other than that we are ready to start coding!

The code

Ok now for the exciting part! Here is the code example:

const puppeteer = require("puppeteer");

const scrape = async () => {
    const browser = await puppeteer.launch({ headless: false });
    // const browser = await puppeteer.connect({ browserWSEndpoint: 'wss://chrome.browserless.io?token=[ADD BROWSERLESS API TOKEN HERE]' })

    const page = await browser.newPage();
    await page.goto('https://news.ycombinator.com');

    // Here, we inject some JavaScript into the page to build a list of results
    const items = await page.evaluate(() => {
        const elements = [...document.querySelectorAll('.athing .titleline a')];
        const results = elements.map((link) => ({
            title: link.textContent,
            href: link.href,
        }));
        return JSON.stringify(results);
    });

    // Finally, we return an object, which triggers a JSON file download
    return JSON.parse(items);

};
scrape();

Lines 4 and 5 are very important in this example.

Line 4 uses Puppeteer when running your script. This is good for testing so you can view how the browser is interacting with your script.

Line 5 is the Browserless connection. This is where you can add your API key which will link up to your Browserless account and allow you to run your script with Browserless.

Make sure one of these two lines are commented out. You only need one.

Final product

Alrighty, that is all you need for this example. Once things are installed and code is implemented, you can open up your preferred command line interface in your project and run node [insert name of js file here].

The output should be a json file with real time titles and links from the Hacker News feed!

I hoped this article on data scraping was intriguing and exciting. There are endless possibilities as to what you can accomplish with web and data scraping. I hope this sparks some cool projects for some of you.

Happy coding! ❤️

Getting started with Website Test Automation

Tyler Reicks — Mon, 29 Aug 2022 17:46:00 +0000

The software development ecosystem is in a constant state of movement. This is especially true when it comes to building websites and other applications. Not only is the technology that you are using changing, but so are the requirements and acceptance criteria. This is where technologies like headless architecture can help with continuous integration and continuous delivery (CI/CD).

In this article, we will be going over website test automation. Specifically, we will be talking about UI automation testing using Puppeteer and Browserless. There are other libraries that you can use for website test automation alongside Browserless (i.e. Playwright), but for the sake of this article, we will be using Puppeteer. If you want more info on other browser automation tools check out this other article on the Browserless blog. Browserless and Puppeteer integrate seamlessly within your project and allow you to stay on top of testing as well as changing requirements. We'll go through what both tools are, and do some simple examples.

Headless Browser Testing

First, let's explain what headless browser testing is. Headless browser testing allows for automated control of a webpage without any graphical user interface (GUI). This allows a tester/user/QA/developer to do automated testing on a web application without having to manually test functionality on browsers that we visually see on our monitors. Imagine having a robot test your application for you! That's pretty much what headless browser testing is only often times it's a simple script. Headless browser testing speeds up the testing process while also providing quick feedback during development.

Puppeteer and Browserless

So, you may be thinking how do Puppeteer and Browserless fit into this equation?

Puppeteer provides a great high-level API to control Chrome (both headless and non-headless). Puppeteer is maintained by the Chrome DevTools team. Most things you can do manually in the browser can be accomplished using Puppeteer. This lets us easily perform actions to test our UI's functionality. Puppeteer also allows us to generate screenshots and PDFs of the pages while performing actions on the application for testing.

Puppeteer alone is a great tool to use for website test automation. However, Browserless can take your website test automation to the next level. Let's say you want to run tests in your production environment, but you don't want to add Puppeteer as an extra dependency. This is where Browserless comes to the rescue. With Browserless there is no need to install extra packages or dependencies, and it's compatible with Puppeteer. All you need to do is plug and play. This takes a huge load off your application. You can run your tests with Puppeteer in your dev environments, and meanwhile, run Browserless in your prod environments without having to worry about the extra baggage.

Setup

Before we get to the examples let's set up our environments. Luckily, there aren't many dependencies we need to install. There is only one ... Puppeteer.

npm i puppeteer

Once you have run that command you are good to go!

The great part about the other tool we're using, Browserless, is that there's not any setup involved other than creating an account. Let's go to www.browserless.io and create an account. Once you get your account set up you will be directed to your Browserless dashboard.

Here you can find your session usage chart, prepaid balance, API key, and all of the other account goodies. Keep your API key handy as we will use that for our examples.

I should also mention, Browserless provides a great online debugger tool that lets you test out your scripts to see how they work. If you don't want to go through the hassle of setting up your local machine, this is a great option to use.

Example 1:

In this example, we are going to test the functionality of a cool website created by Tim Holman (@twholman) and Tobias van Schneider (@vanschneider) called "The Passive Aggressive Password Machine". Their website asks you to enter a password, and it will tell you how good your password is. This test is going to be super simple. We are going to test the functionality of the website by using Puppeteer to type a password in the input area. We will then take a screenshot to verify the website gave the user feedback. Taking a screenshot is the simplest form of validation. If you wanted to make this test even more robust, you could programmatically check to make sure the website was giving updated feedback on the password that was entered.

See the code for example 1 below. Notice the commented section on line 5. This is where you would add in your Browserless API token to run in your Browserless environment.

const puppeteer = require("puppeteer");

const scrape = async () => {
    const browser = await puppeteer.launch({headless: false});
    // const browser = await puppeteer.connect({ browserWSEndpoint: 'wss://chrome.browserless.io?token=[ADD BROWSERLESS API TOKEN HERE]' })

    const page = await browser.newPage();

    await page.goto('https://trypap.com/');

    await page.setViewport({ width: 1920, height: 1080 });

    await page.type('input[placeholder="Please Enter a Password"]', 'BrowserlessIsCool1234');

    await page.waitForTimeout(2000);

    await page.screenshot({ path: "screenshot.png", fullPage: true });

    await browser.close();
};
scrape();

Once you have this script on your local machine, run node [title of your script] in your terminal and you should get a screenshot outputted into the root of your directory.

Here is what it looks like:

Congrats! You've officially done a simple UI test.

Example 2:

In this example, I want to show off the /stats API that Browserless provides to us. The /stats API is powered by Google's Lighthouse project and gives you metrics about performance, accessibility, size, and much more. As mentioned in the documentation, the /stats API is pretty simple to use. To gather this data, send a POST request to https://chrome.browserless.io/stats with a simple JSON payload containing a valid URL. This will output a large amount of data, about 370kb. You can narrow down what gets outputted by adding a config object with specific categories you want to see. This is similar to how you would use the Lighthouse node API. For our example, we will only be checking the performance category.

We are going to run a curl command to receive the output, but if you wanted to run this in JavaScript you could accomplish the same thing with the fetch API. For this example we'll run the Lighthouse tests on the Passive Aggressive Password Machine website we used in the first example. Below is the curl command.

curl -X POST \
  https://chrome.browserless.io/stats?token=[ADD BROWSERLESS API TOKEN HERE] \
  -H 'Cache-Control: no-cache' \
  -H 'Content-Type: application/json' \
  -d '
{
  "url": "https://trypap.com/",
    "config": {
        "extends": "lighthouse:default",
        "settings": {
            "onlyCategories": ["performance"]
        }
    }
}'

This should return a JSON object with the Lighthouse performance score. At the time of writing this, trypap.com received a 0.9 performance score (on a scale from 0-1). Pretty good!

I hope this article helped open up a whole new world of website test automation for you. As always, make sure to check out the Browserless blog and Browserless YouTube channel for more exciting web browser automation content! Happy coding!❤️

Scraping Google Shopping using Puppeteer and Browserless

Tyler Reicks — Wed, 29 Jun 2022 03:23:32 +0000

In this article, we are going to completely automate the process of searching for a product on Google Shopping. Specifically, we are going to search for board games and scrape the Google Shopping results page for the product name and price. Below is a visual of what we are going to be targeting on the page.

Here's the kicker, all we are going to use is one script and exactly 50 lines of code. How are we going to do this you may ask? The magic of JavaScript. Furthermore, a node library called Puppeteer and a web service called Browserless. If you don't know much about these tools I'd highly recommend you check them out further. They offer a lot of neat capabilities! We will go over some of them in this article.

For further info on Puppeteer, I'd encourage watching Eric Bidelman's talk at the Google I/O conference in 2018. Also, if you're hungry for more Browserless material after reading this article checkout the Browserless YouTube channel and the Browserless blog. Both the channel and blog offer great tips and tricks to use in your Browserless and Puppeteer environments.

Why Puppeteer?

Next, let's very quickly explain why we are using Puppeteer to accomplish our task of automating a Google Shopping search. First, Puppeteer provides a great high-level API to control Chrome (both headless and non-headless). Puppeteer is maintained by the Chrome DevTools team. These engineers know what they are doing. Lastly, most things you can do manually in the browser can be accomplished using Puppeteer. This will allow us to easily manipulate our script to perform the actions we want in the browser.

Initial setup

Alrighty, let's get to it! We are going to start with the initial setup. Luckily for us, there aren't many dependencies we need to install. There's only one... Puppeteer.

npm i puppeteer

Once you've run that command you are good to go! 👍

Browserless

Technically you are all set to start coding, but I want to use Browserless in tandem with Puppeteer. Why am I going to use this tool you're wondering? Let's get into that.

The script that we are going to be coding is one of many things that can be done with Puppeteer and headless Chrome. Let's say you want to run this script in one of your production applications, but you don't want to have to add Puppeteer as an extra dependency. What can you do? This is one scenario where Browserless comes to the rescue. With Browserless there is no need to install extra packages or dependencies, and it's compatible with Puppeteer. All you need to do is plug and play (We'll get more into the details of setting up Browserless below). This takes a huge load off of your application. You can test and run Puppeteer scripts in your dev environments and meanwhile run Browserless in your prod environments without having to worry about the extra baggage. Talk about a performance boost!

Another area where Browserless shines is in remote development environments (i.e. repl.it, gitpod, codepen, codespaces, etc.). Since there is no need to install any software you can easily hook up to the Browserless API and run your Puppeteer scripts in all your remote environments.

Before we get to the actual script, let's look at how to set up Browserless real quick.

For starters, you'll need to set up an account. Once you get your account set up, you'll be directed to your Browserless dashboard.

Here you can find your session usage chart, prepaid balance, API key, and all of the other account goodies.

Keep your API key handy as we'll use that once we start writing our script. Other than that we are ready to start coding!

The code

Ok now for the exciting part! I like to show the final product at the beginning and then dissect each piece from top to bottom. Here is our script:

const puppeteer = require("puppeteer");
const fs = require("fs");

const scrape = async () => {
  try {
    const browser = await puppeteer.launch({headless: false});
    // const browser = await puppeteer.connect({
    //   browserWSEndpoint:
    //     "wss://chrome.browserless.io?token=[ADD BROWSERLESS API TOKEN HERE]",
    // });

    const page = await browser.newPage();
    await page.goto("https://shopping.google.com/");
    await page.type(
      "input[placeholder='What are you looking for?']",
      "board game"
    );
    await page.click("button[aria-label='Google Search']");
    await page.waitForSelector("#pnnext");

    // Go to next results page
    await page.click("#pnnext");
    await page.waitForSelector("#pnnext");

    // Gather product title
    const title = await page.$$eval("div.sh-dgr__grid-result h4", (nodes) =>
      nodes.map((n) => n.innerText)
    );

    // Gather price
    const price = await page.$$eval(
      "div.sh-dgr__grid-result a.shntl div span span[aria-hidden='true'] span:nth-child(1)",
      (nodes) => nodes.map((n) => n.innerText)
    );

    // Consolidate product search data
    const googleShoppingSearchArray = title.slice(0, 10).map((value, index) => {
      return {
        title: title[index],
        price: price[index],
      };
    });
    const jsonData = JSON.stringify(googleShoppingSearchArray, null, 2);
    fs.writeFileSync("googleShoppingSearchResults.json", jsonData);
    // await browser.close();
  } catch (err) {
    console.error(err);
  }
};
scrape();

Okie doke, let's dive into this. The first 2 lines are pulling in JavaScript modules. We'll want to pull in Puppeteer for obvious reasons. The second module is what we will use to export our scraped data to a JSON file.

const puppeteer = require("puppeteer");
const fs = require("fs");

The next section is the start of our actual scrape() function. At the very beginning of this arrow function, we hook up to the Browserless API using the API key you obtained from the Browserless dashboard.

  const browser = await puppeteer.connect({
    browserWSEndpoint:
      "wss://chrome.browserless.io?token=[ADD BROWSERLESS API TOKEN HERE]",
  });

Alternatively, you could replace this section of code with puppeteer.launch like so:

const browser = await puppeteer.launch({ headless: false });

This is good to do in your dev environment to test out the script. That will save your Browserless resources and help with debugging if there is any issue with your code. One thing I wanted to point out was the {headless: false} property that is being passed in. This tells Puppeteer to launch Chrome on the screen so you can visually see how the code is interacting with the webpage. Pretty cool to watch, and extra useful when building out scripts!

In the next section, we create a new browser page, navigate to Google Shopping's website, find the search bar, type in whatever we want to search for (in this case we are searching for a board game), click the search button, and finally waiting for the page to load. More specifically, waiting for the pagination at the bottom of the page to load. You'll see why in a minute.

    const page = await browser.newPage();
    await page.goto("https://shopping.google.com/");
    await page.type(
      "input[placeholder='What are you looking for?']",
      "board game"
    );
    await page.click("button[aria-label='Google Search']");
    await page.waitForSelector("#pnnext");

This following section is kind of a bonus. What if you wanted to navigate a couple of pages deep into the search results? That is what we accomplish in the next code block.

Remember how we waited for the pagination to load in the last part of the code? In this section, we click on that "next page" button that we were waiting on. This will navigate our browser to the next page of board games.

    await page.click("#pnnext");
    await page.waitForSelector("#pnnext");

Alrighty, now to gather the data we want from the page! The following piece of code is what finds the product title and the price of the board game. I do want to add a little disclaimer here. When inspecting Google Shopping's search page, you'll notice their compiled code is a little convoluted. If these specific attributes aren't working for you, make sure to inspect the search results page yourself and find the HTML attributes that work for your scenario. This is where using that puppeteer.launch({ headless: false }) property can help.

    // Gather product title
    const title = await page.$$eval("div.sh-dgr__grid-result h4", (nodes) =>
      nodes.map((n) => n.innerText)
    );

    // Gather price
    const price = await page.$$eval(
      "div.sh-dgr__grid-result a.shntl div span span[aria-hidden='true'] span:nth-child(1)",
      (nodes) => nodes.map((n) => n.innerText)
    );

Ok, so once we've gotten our product titles and prices it's time to consolidate our data. For this I'm only going to take the first 10 results I gathered.

    // Consolidate product search data
    const googleShoppingSearchArray = title.slice(0, 10).map((value, index) => {
      return {
        title: title[index],
        price: price[index],
      };
    });

Lastly, we want to display or save our data. If you simply want to see what results get returned, you can console.log(googleShoppingSearchArray);. But let's say you want to export this data to a JSON file and maybe use that JSON to make a chart of the results. To save your data into a JSON file you'll use the code in our example.

    const jsonData = JSON.stringify(googleShoppingSearchArray, null, 2);
    fs.writeFileSync("googleShoppingSearchResults.json", jsonData);

I added extra parameters to the stringify method for formatting purposes. The last param is a space parameter.

And to finish off our script, we close the browser.

await browser.close();

Below is a visual display of what our script is doing in the browser.

As you can see, Puppeteer is a pretty cool API. Pair that with Browserless and you have a super scalable, web scraping, automation tool. Did this article spark an idea or thought? Let me know in the comments, or on Twitter at @tylerreicks. Happy coding ❤️!

Scraping Amazon using Puppeteer and Browserless

Tyler Reicks — Tue, 07 Jun 2022 17:51:49 +0000

In this article, we are going to completely automate the process of searching for a product on Amazon. Specifically, we are going to search for office chairs and scrape the Amazon results page for the product name and price. Below is a visual of what we are going to be targeting on the page.

Here's the kicker, all we are going to use is one script and under 50 lines of code. How are we going to do this you may ask? The magic of JavaScript. Furthermore, a node library called Puppeteer and a web service called Browserless. If you don't know much about these tools I'd highly recommend you check them out further. They offer a lot of neat capabilities! We will go over some of them in this article.

For further info on Puppeteer, I'd highly recommend watching Eric Bidelman's talk at the Google I/O conference in 2018. Also, but if you're hungry for more Browserless after reading this article checkout Joel Griffith's webinar about cool ways to optimize web scraping. Joel is the CEO of Browserless and offers great tips and tricks to use in your Browserless and Puppeteer environments.

Why Puppeteer?

Next, let's very quickly explain why we are using Puppeteer to accomplish our task of automating an Amazon search. First, Puppeteer provides a great high-level API to control Chrome (both headless and non-headless). Puppeteer is maintained by the Chrome DevTools team. These engineers know what they are doing. Lastly, most things you can do manually in the browser can be accomplished using Puppeteer. This will allow us to easily manipulate our script to perform the actions we want in the browser.

Initial setup

Alrighty, let's get to it! We are going to start with the initial setup. Luckily for us, there aren't many dependencies we need to install. There's only one... Puppeteer.

npm i puppeteer

Once you've run that command you are good to go! 👍

Browserless

Technically you are all set to start coding, but there is another tool I want to use in tandem with Puppeteer. That tool is a web service called Browserless. Why am I going to use this tool you're wondering? Let's get into that.

Before we get to the actual script, let's look at how to set up Browserless real quick.

For starters, you'll need to set up an account. Once you get your account set up, you'll be directed to your Browserless dashboard.

Here you can find your session usage chart, prepaid balance, API key, and all of the other account goodies.

Keep your API key handy as we'll use that once we start writing our script. Other than that we are ready to start coding!

The code

Ok now for the exciting part! I like to show the final product at the beginning and then dissect each piece from top to bottom. Here is our script:

const puppeteer = require("puppeteer");
const fs = require("fs");

const scrape = async () => {
  const browser = await puppeteer.connect({
    browserWSEndpoint:
      "wss://chrome.browserless.io?token=[ADD BROWSERLESS API TOKEN HERE]",
  });

  const page = await browser.newPage();
  await page.goto("https://www.amazon.com");
  await page.type("#twotabsearchtextbox", "office chair");
  await page.click("#nav-search-submit-button");
  await page.waitForSelector(".s-pagination-next");

  // Go to next results page
  await page.click(".s-pagination-next");
  await page.waitForSelector(".s-pagination-next");

  // Gather product title
  const title = await page.$$eval("h2 span.a-color-base", (nodes) =>
    nodes.map((n) => n.innerText)
  );

  // Gather price
  const price = await page.$$eval(
    "[data-component-type='s-search-result'] span.a-price[data-a-color='base'] span.a-offscreen",
    (nodes) => nodes.map((n) => n.innerText)
  );

  // Consolidate product search data
  const amazonSearchArray = title.slice(0, 5).map((value, index) => {
    return {
      title: title[index],
      price: price[index],
    };
  });

  const jsonData = JSON.stringify(amazonSearchArray, null, 2);
  fs.writeFileSync("amazonSearchResults.json", jsonData);
  await browser.close();
};
scrape();

const puppeteer = require("puppeteer");
const fs = require("fs");

  const browser = await puppeteer.connect({
    browserWSEndpoint:
      "wss://chrome.browserless.io?token=[ADD BROWSERLESS API TOKEN HERE]",
  });

Alternatively, you could replace this section of code with puppeteer.launch like so:

const browser = await puppeteer.launch({ headless: false });

In the next section, we create a new browser page, navigate to Amazon's website, find the search bar, type in whatever we want to search for (in this case we are searching for an office chair), click the search button, and finally waiting for the page to load. More specifically, waiting for the pagination at the bottom of the page to load. You'll see why in a minute.

  const page = await browser.newPage();
  await page.goto("https://www.amazon.com");
  await page.type("#twotabsearchtextbox", "office chair");
  await page.click("#nav-search-submit-button");
  await page.waitForSelector(".s-pagination-next");

This following section is kind of a bonus. What if you wanted to navigate a couple of pages deep into the search results? That is what we accomplish in the next code block.

  await page.click(".s-pagination-next");
  await page.waitForSelector(".s-pagination-next");

Alrighty, now to gather the data we want from the page! The following piece of code is what finds the product title and the price of the office chair. I do want to add a little disclaimer here. When inspecting Amazon's search page, you'll notice their compiled code is a little convoluted. If these specific attributes aren't working for you, make sure to inspect the search results page yourself and find the HTML attributes that work for your scenario. This is where using that puppeteer.launch({ headless: false }) property can help.

  // Gather product title
  const title = await page.$$eval("h2 span.a-color-base", (nodes) =>
    nodes.map((n) => n.innerText)
  );

  // Gather price
  const price = await page.$$eval(
    "[data-component-type='s-search-result'] span.a-price[data-a-color='base'] span.a-offscreen",
    (nodes) => nodes.map((n) => n.innerText)
  );

Ok, so once we've gotten our product titles and prices it's time to consolidate our data. For this I'm only going to take the first 5 results I gathered.

  // Consolidate product search data
  const amazonSearchArray = title.slice(0, 5).map((value, index) => {
    return {
      title: title[index],
      price: price[index],
    };
  });

Lastly, we want to display or save our data. If you simply want to see what results get returned, you can console.log(amazonSearchArray);. But let's say you want to export this data to a JSON file and maybe use that JSON to make a chart of the results. To save your data into a JSON file you'll use the code in our example.

  const jsonData = JSON.stringify(amazonSearchArray, null, 2);
  fs.writeFileSync("amazonSearchResults.json", jsonData);

I added extra parameters to the stringify method for formatting purposes. The last param is a space parameter.

And to finish off our script, we close the browser.

await browser.close();

Below is a visual display of what our script is doing in the browser.

Creating a Slack clone with Supabase and Next.js

Tyler Reicks — Thu, 11 Mar 2021 23:00:01 +0000

Intro

Recently, I came across a new cloud backend service, called Supabase, that claimed to be a firebase alternative. I think Firebase is a great tool but I have had my fair share of headaches working with it. So I decided to try out Supabase and see what it was all about.

On Supabase's GitHub page they have a lot of cool example projects to help you learn their platform. One that really caught my eye was their Slack clone project. In this article, I'm going to go over the steps in that example. I'll talk about my experience and what my opinions are. Enjoy!

Here is the repo that I forked from Supabase to make this project. You can follow along there with the README if you want as well.

Setup

Not much setup was needed for this project. The three things you need are all accounts: GitHub account, Vercel account (Next.js), and a Supabase account. Sidenote, you can use your GitHub credentials to make a Vercel and a Supabase account, so if you want to be technical, you really only need one account. Either way, once you have all of that setup you are ready to start the project!

Supabase things

Once you have set up your Supabase account you'll see a dashboard page. When you first sign up you won't have any projects, but we shall change that soon. On the left side of the screen, you'll see a vertical toolbar. Select the third icon that looks like a code icon/terminal logo. This will bring you to a page with specific scripts and quick start guides to help you get up and running with the Supabase platform. Select the "Slack Clone" option in the Quick start section.

Once you click on the Slack clone script you'll be directed to a page where you can see the code for the script query. On the right side of that screen, there will be a "Run" button. Make sure to give that a click. Look at that! You're well on your way to making your own Slack clone with Supabase!

Next, we are going to click on the Settings button on the left vertical toolbar. It will be the last icon in the column and looks like your typical settings gear. Once there, you'll want to click on the API tab from the project menu on the left. There you will find the API URL and the anon key for your project.

If you're wondering what the anon key is here is how the Supabase documentation explains it:

"The anon key is your client-side API key. It allows "anonymous access" to your database until the user has logged in. Once they have logged in, the keys will switch to the user's own login token. This enables row-level security for your data."

Deploy to Next.js

Ok, so now that we have created our slack clone backend, we have to host it somewhere. The easiest way to do this is by following step 4 in the GitHub repo that Supabase provides. Essentially what you will do:

Fork their repo into your own GitHub account.
After you fork, click the deploy button on step 4 in your newly created repo.
This will take you to the Vercel website where you will have to log in with your Vercel account (It's way easier if you just log in with your GitHub credentials).
Once you log in, it will ask you to create a project name. I kept it the same as what was generated, but you can change it to whatever you want.
This is where I got a little confused but hang in there. In the previous step, you created the project, for this step you are now creating the repository. You'll want to link your GitHub again by clicking on the GitHub button. The steps look very identical and it almost feels like your doing the same thing over again, but trust me you're on the right track.
Once you have created your project and repo, you'll be brought to an "import project" screen where you will be asked for a NEXT_PUBLIC_SUPABASE_URL and NEXT_PUBLIC_SUPABASE_KEY. Use the API URL and anon key from step 3 of the GitHub README. This is also what we went over just previously at the end of the Supabase things section above.
Lastly, all you have to do is click the "Deploy" button on the Vercel website!

Final touches

Ok so after you click the deploy button, everything will build and hopefully succeed. You'll see a congratulations message and two buttons. One to go to the dashboard and the other to visit your newly created Slack clone!

Before I get into the actual Slack clone, I wanted to touch on the Vercel dashboard really quickly. The only change I made here was the domain name. When on the dashboard, head over to the settings tab in the top navigation. Then click on "Domains" to see and edit what domain your site is under. Ok, that's it for the dashboard stuff.

Now to the actual, newly created Slack clone site! Initially, you will be prompted with a login screen. I was initially confused when trying to sign up for an account because when I clicked the "Sign up" button nothing happened. It took me a couple of minutes to realize that when I clicked the "Sign up" button it sent me a confirmation email to confirm my account. I think a good fun exercise for this app would be to give some sort of feedback to the user when they sign up for an account. Just something to let them know to check their email for a confirmation link. Anyways, after I figured that out I was able to log in and start messaging away!

Conclusion

Supabase seems like a really well-built tool. I can definitely see it competing with Firebase. It is very easy to use, and setup. I think the key differentiator for me will be the price. Currently, Supabase is in its alpha version. In other words, they are still not a final product yet. Since it's not officially in its final version, Supabase is offering its service for free. But when they do finally get to a version one I'm curious to see how much it costs compared to Firebase.

That's my only outstanding question so far. I think Supabase is really cool and will be a force to be reckoned with. I can't wait to see how it evolves in the future!

That's all for now. Thank you everyone for following along, and happy coding!

Create A Data Visualization Map Using Mapbox

Tyler Reicks — Sun, 14 Feb 2021 23:03:46 +0000

Intro

In this article, we are going to make a map with a software called Mapbox. This won't involve any coding at all, but I think it is a cool tool to use if you want to add a professional-looking map to your website or app. Mapbox's software is used by apps of many popular companies that all of us utilize every day.

Snap Inc.
The Weather Company
Shopify
Facebook
AllTrails
Ancestry
Peloton
And many more...

For the map that we make today, we will be using the data visualization component that is provided to us in Mapbox Studio (aka Mapbox's map editor). The data we will be working with is the temperature change for US counties from 1895 to 2019. This is already pre-loaded into Mapbox studio which is really handy. Without further ado, let's get into it!

Setup

The setup for this is pretty simple. All you need is an internet connection and a Mapbox account. Once you have those 2 things you are ready to go!

The Process

Now that you are signed in you'll want to go to the Styles page. This is where all of your different designed maps would be housed. If you just set up your account you won't have anything in here.

Next, we are going to click the "New style" button on this page. This will take you to a screen where you can choose a map template you'll want to use. For this scenario I'm going to choose the Basic map template, I think any template will work here (other than the blank template of course). After you have chosen your template, click the "Customize" button at the bottom of the modal.

This will direct you to your new Mapbox map! Pro tip: your map will be centered in Paris. Make sure to navigate to the United State before moving on to the rest of the steps.

This next paragraph is completely optional, in my opinion, but to make the map feel a little less cluttered we will delete some of the components that are added on by default. Click on the component in the menu on the left-hand side. Then click the trash can icon at the top of the menu. Below is the list of components that I deleted.

Buildings
Place labels
Points of interest
Road network
Transit
Walking, cycling, etc.

Adding the data

For this exercise, we are going to create a choropleth map with data from The Washington Post's "2ºC: Beyond the Limit" series about rising temperatures, which analyzes warming temperatures in the United States. Our choropleth map will visualize the change in average annual temperature from 1895 to 2019.

In the Components panel, click the plus icon and then select "Data visualization" from the menu that pops up. You'll then be shown a dark map of the United States. Click on where it says "None selected" under Source. Then click "Add source by ID". Paste mapbox.brt3djy1 into the search bar and click "Find".

Yay! We have pulled in the data. It was that easy. Now we have to decide how we want our data to look on our map. Click "Select data visualization type". Then click the choropleth option, and lastly, the "Select Choropleth" button.

There you have it. The component will be added to your map and you'll see the United States illuminate with color.

Finishing touches

The rest is just messing around with the styles and colors. You'll notice initially that your map will have 3 colors populated. The field tempchg has values above and below 0. To make the visualization reflect divergence from 0 with color, the second stop should be 0 instead of the default value of 1.774. So let us change the middle value (2) in the Color section to 0.

Also, it would be nice to make this look a little bit more like a heat map. We can do that by going down to the bottom section of the Components panel under the Colors section. Click on "Choropleth" with the 3 colors to the left. This will expand and display a dropdown. In that dropdown, select the "Temperate" option (the "Adverse" color palette is a good choice too). Finally, our colors are switched around so we need to click the "Reverse palette" button next to the dropdown.

Conclusion

There you have it! A super slick looking map that shows temperature change data of the United States. How easy was that? I encourage you to play around a little more in the Mapbox studio and see what other cool manipulations you can make. Some easy ones are: adding more colors to show more details of the data, change the stroke to show or hide the polygon outlines, toggle labels on or off, and even change the opacity of your map.

You can preview your map by copying the share link in the top right corner, as well as publish it so you can include it in your dev project.

Lastly, if you're curious about diving deeper into this, Mapbox has lots of great lessons and tutorials to help guide you through all of the available functionality. I think a logical next step after this would be to add your own data to a Mapbox map. Maybe you could look at one of my previous articles about scraping websites for data then including that data into your very own custom map!

That's all for now. I hope you enjoyed this article. Happy coding!

Create a Chess game with React and Chessboardjsx ♟️

Tyler Reicks — Fri, 22 Jan 2021 05:53:41 +0000

Intro

In this article we are going to create a chess game with React. I based this article off of another article that I read recently by Varun Pujari. Here's the link to that if you want to check it out.

We'll use a package called chessboardjsx, which will give us an easy way to display the chess game. On top of that we will use the chess.js library to implement moves and how the game should be played.

This chess game will have one player playing against an AI that will make a random move for every turn. Lastly, we will add a timer to our chess game so we can time how fast we beat the AI!

Setup

Setup is pretty simple. First, we'll run a couple commands in the terminal/command prompt to get everything installed.

Run yarn create react-app chess-game --template typescript. You can also run npx create-react-app chess-game --template typescript but yarn worked better for me. I was getting an error saying my create-react-app was out of date. Every time I would uninstall and try to run the npx command I would get the same out of date error. So yarn was what I went with.
Next we'll install chessboard.jsx with this yarn add chessboardjsx command.
Now let's install the brains of the game. Run yarn add chess.js. This package is what we will use for the AI logic.
Since we are also using typescript we have to add types for chess.js. We can do this by running yarn add @types/chess.js.
Lastly, I was thinking it would be cool to add a timer to this game so we can see how long the game took. Let's set that up by running yarn add react-compound-timer.

Code

Now for the fun part, the actual code behind the game. Below you will find the code for the only file you'll need to edit in this project, the App.tsx file. I've tried to comment on the main parts so it's easier to understand what is going on.

import React, { useState } from "react";
import "./App.css";
import Timer from "react-compound-timer";
// Lines 5-8: Bring in chessboard and chess.js stuff
import Chessboard from "chessboardjsx";
import { ChessInstance, ShortMove } from "chess.js";

const Chess = require("chess.js");

const paddingStyle = {
  padding: 5
}

const marginStyle = {
  margin: 5
}

const App: React.FC = () => {
  const [chess] = useState<ChessInstance>(
    // Set initial state to FEN layout
    new Chess("rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1")
  );

  const [fen, setFen] = useState(chess.fen());

  // Logic for the setting up the random computer move.
  const handleMove = (move: ShortMove) => {
    // Line 29 validates the user move.
    if (chess.move(move)) {
      setTimeout(() => {
        const moves = chess.moves();
        // Lines 33-28: Computer random move.
        if (moves.length > 0) {
          const computerMove = moves[Math.floor(Math.random() * moves.length)];
          chess.move(computerMove);
          setFen(chess.fen());
        }
      }, 300);
      // Sets state of chess board
      setFen(chess.fen());
    }
  };

  return (
    <div className="flex-center">
      <h1>Random Chess Game</h1>
      <Chessboard
        width={400}
        position={fen}
        // onDrop prop tracks everytime a piece is moved.
        // The rest is handled in the the handleMove function.
        onDrop={(move) =>
          handleMove({
            from: move.sourceSquare,
            to: move.targetSquare,
            // This promotion attribute changes pawns to a queen if they reach the other side of the board.
            promotion: "q",
          })
        }
      />
      {/* Timer code */}
      <Timer initialTime={0} startImmediately={false}>
        {/* I thought this was weird. Definitely a better way to do this, but I just wanted it to work. */}
        {({ start, resume, pause, stop, reset, timerState } : {start:any, resume:any, pause:any, stop:any, reset:any, timerState:any}) => (
            <>
                <div>
                    <span style={paddingStyle}><Timer.Minutes /> minutes</span>
                    <span style={paddingStyle}><Timer.Seconds /> seconds</span>
                    <span style={paddingStyle}><Timer.Milliseconds /> milliseconds</span>
                </div>
                <div style={paddingStyle}>{timerState}</div>
                <br />
                <div>
                    <button style={marginStyle} onClick={start}>Start</button>
                    <button style={marginStyle} onClick={pause}>Pause</button>
                    <button style={marginStyle} onClick={resume}>Resume</button>
                    <button style={marginStyle} onClick={stop}>Stop</button>
                    <button style={marginStyle} onClick={reset}>Reset</button>
                </div>
            </>
        )}
      </Timer>
    </div>
  );
};

export default App;

Once you have updated your App.tsx file to look like this you should be able to run your project with yarn start and play chess against an AI that you created. Don't forget to start the timer and see how quick you can win!

FEN (Forsyth-Edwards Notation)

One thing I thought was really interesting about this project was Forsyth-Edwards Notation, or FEN. It is the notation that describes a chess position. You'll notice it being used on line 21 of the App.tsx code. It stood out to me because when I first saw it I was certain it was just a bunch of gibberish. Can you figure out what the letters on the initial starting state of the Forsyth-Edwards Notation mean? I'm sure you'll pick it up quick, but if you need a hint, it has to do with names of the pieces on the chess board.

Conclusion

Welp, that’s about it. Pretty short and simple. I hope you enjoyed making this chess game and hopefully have more fun playing it. If you want a little bit of extra credit, try deploying this out online somewhere and see how fast your friends and family can beat the AI. I would recommend deploying it to Netlify. That's my go to hosting service.

Like always, happy coding! Love y'all. Peace.

Scraping Wikipedia for data using Puppeteer and Node

Tyler Reicks — Wed, 06 Jan 2021 04:41:23 +0000

Intro

In this article, we'll go through scraping a Wikipedia table with COVID-19 data using Puppeteer and Node. The original article that I used for this project is located here.

I have never scraped a website before. I've always seen it as a hacky thing to do. But, after going through this little project, I can see the value of something like this. Data is hard to find and if you can scrape a website for it, in my opinion, by all means, do it.

Setup

Setting up this project was extremely easy. All you have to do is install Puppeteer with the command npm install puppeteer. There was one confusing issue I had during setup, however. The puppeteer package was not unzipped correctly when I initially installed it. I found this out while running the initial example in the article. If you get an error that states Failed to launch browser process or something similar follow these steps:

Unzip chrome-win from node_modules/puppeteer/.local-chromium/
Then add that folder to the win64 folder in that same .local-chromium folder.
Make sure the chrome.exe is in this path node_modules/puppeteer/.local-chromium/win64-818858/chrome-win/chrome.exe
This is for windows specifically. Mac might be similar, but not sure.

Here is the link that lead me to the answer. It might be a good idea to do this no matter what to make sure everything is functioning properly.

The code

I had to make a couple of small changes to the existing code.

First example

The first example didn't work for me. To fix the problem I assigned the async function to a variable then invoked that variable after the function. I'm not sure this is the best way to handle the issue but hey, it works. Here is the code:

const puppeteer = require('puppeteer');

const takeScreenShot = async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto('https://www.stem-effect.com/');
    await page.screenshot({path: 'output.png'});

    await browser.close();
};

takeScreenShot();

Wikipedia scraper

I also had an issue with the Wikipedia scraper code. For some reason, I was getting null values for the country names. This screwed up all of my data in the JSON file I was creating.

Also, the scraper was 'scraping' every table on the Wikipedia page. I didn't want that. I only wanted the first table with the total number of cases and deaths caused by COVID-19. Here is the modified code I used:

const puppeteer = require('puppeteer');
const fs = require('fs')

const scrape = async () =>{
    const browser = await puppeteer.launch({headless : false}); //browser initiate
    const page = await browser.newPage();  // opening a new blank page
    await page.goto('https://en.wikipedia.org/wiki/2019%E2%80%9320_coronavirus_pandemic_by_country_and_territory', {waitUntil : 'domcontentloaded'}) // navigate to url and wait until page loads completely

    // Selected table by aria-label instead of div id
    const recordList = await page.$$eval('[aria-label="COVID-19 pandemic by country and territory table"] table#thetable tbody tr',(trows)=>{
        let rowList = []    
        trows.forEach(row => {
                let record = {'country' : '','cases' :'', 'death' : '', 'recovered':''}
                record.country = row.querySelector('a').innerText; // (tr < th < a) anchor tag text contains country name
                const tdList = Array.from(row.querySelectorAll('td'), column => column.innerText); // getting textvalue of each column of a row and adding them to a list.
                record.cases = tdList[0];        
                record.death = tdList[1];       
                record.recovered = tdList[2];   
                if(tdList.length >= 3){         
                    rowList.push(record)
                }
            });
        return rowList;
    })
    console.log(recordList)
    // Commented out screen shot here
    // await page.screenshot({ path: 'screenshots/wikipedia.png' }); //screenshot 
    browser.close();

    // Store output
    fs.writeFile('covid-19.json',JSON.stringify(recordList, null, 2),(err)=>{
        if(err){console.log(err)}
        else{console.log('Saved Successfully!')}
    })
};
scrape();

I wrote comments on the subtle changes I made, but I'll also explain them here.

First, instead of identifying the table I wanted to use by the div#covid19-container, I pinpointed the table with the aria-label. This was a little more precise. Originally, the reason the code was scraping over all of the tables on the page was because the IDs were the same (I know, not a good practice. That's what classes are for, right?). Identifying the table via aria-label helped ensure that I only scraped the exact table I wanted, at least in this scenario.

Second, I commented out the screenshot command. It broke the code for some reason and I didn't see the need for it if we were just trying to create a JSON object from table data.

Lastly, after I obtained the data from the correct table I wanted to actually use it in a chart. I created an HTML file and displayed the data using Google charts. You can see the full project on my Github if you are curious. Fair warning, I got down and dirty (very hacky) putting this part together, but at the end of the day, I just wanted an easier way to consume the data that I had just mined for. There could be a whole separate article on the amount of refactoring that can be done on my HTML page.

Conclusion

This project was really fun. Thank you to the author, Mohit Maithani, for putting it together. It opened my eyes to the world of web scraping and a whole new realm of possibilities! At a high level, web scraping enables you to grab data from anywhere you want.

Like one of my favorite Youtubers, Ben Sullins likes to say, "When you free the data, your mind will follow".

Love y'all. Happy coding!

Creating a Todo app with Svelte and Meteor

Tyler Reicks — Thu, 24 Dec 2020 00:27:10 +0000

Intro

For my next project, I decided to create a todo list app using Svelte and Meteor. I initially got the motivation to do this through Meteor's documentation site. On their site, they have a tutorial on how to create a svelte app with a meteor back end (and lots of other great tutorials). Which is exactly what I was looking for! In this article, I'll take you through my experience of the tutorial and a couple of extra things that I added.

Before I start, I want to share a great quote I heard about Meteor from Scott Tolinski on the Syntax podcast:

Meteor is a back end tool that allows you to "Bring your own front end".

I couldn't agree more, and Meteor shows this off with all of the different front end frameworks you can use with their platform. This is comparable to how Next.js operates. Next allows you to bring in whatever back end platform you want to hook up to your Next front end.

Setup

The setup for Meteor was weird to me. First, I had to install some software called Chocolatey. I'm not sure if this a popular software among the dev community, but I had definitely never heard of it.

Once I had that installed, I was able to run the command choco install meteor to actually get Meteor on my machine. After that, it was a pretty straightforward setup process. Running the meteor create <name of project> was all I needed to do to get going.

There were a couple of odd/cool things during this setup process that I want to note. Meteor forces you to run their commands in Powershell. Which is another practice that I had never experienced before. I'd be curious to know why that is. Also, while in a Meteor project, Meteor replaces npm in a lot of ways. For example, instead of using npm install <package> you would use meteor add <package>. Lastly, I was super stoked to find out that Meteor automatically hooks up your project to a MongoDB back end! Right out of the box!

Cool things I learned

Here are a couple of things that I learned or thought were cool about Svelte and Meteor.

Meteor

Meteor offers a lot of really cool features (on top of the automatic MongoDB hook up) that I haven't really encountered with other projects I've done. I'm still very new to development so these might not be too special but they stood out to me.

Running your app with Meteor makes it extremely easy to develop for mobile environments. In this tutorial, they actually go over how to run your app on iOS and Android emulators. They explain the steps very well, and it's a nice added functionality bonus.

Meteor has built-in account login. No need for additional setup. I thought that was really handy. Other cloud-based platforms have made hooking up logins from other accounts pretty easy nowadays, but what made Meteor stand out to me was their default username and password login functionality. It was very straightforward and easy to understand. Other login accounts you could hook up in your Meteor project included: Google, GitHub, and Facebook.

Because Meteor is such a focused back end solution they include a great security section in their tutorial. In this section, Meteor goes over how to set up your data calls correctly. This covers how you submit, update, and delete data. Really beneficial especially in times like these where hacking is a huge problem in the tech world.

Last but certainly not least, testing. Every developer loves writing tests! (subtle sarcasm) In this Meteor tutorial, they walk you through writing some tests. It was actually pretty easy and helped me understand writing tests better overall. Side note: One of my tests didn't work but that's not the point! The point is, I wrote my own tests.

Svelte

The main reason I did this tutorial was to get more exposure to Svelte so I thought I should share something I learned about that framework as well.

The way Svelte uses reactivity was very confusing for me at first, but this tutorial helped me understand it a little better. I also looked at some Svelte documentation to help guide me as well. Long story short, the $ is awesome!

This doesn't really have to do with Svelte, but I'm going to add it in here anyway. I had never used the double bang syntax !! until this tutorial. What does the double bang syntax do? It casts a javascript variable to a boolean. Pretty cool, huh? If you're still confused here's a great article that helped me grasp the concept. Running the author's example in the web console dev tools helped me visualize it even better.

Not so great things

There were a couple of hiccups during this tutorial. Luckily only one code hiccup which is always nice.

Code Error

When I finished the project my to-do list wasn't functioning correctly. The checkboxes weren't checking and the delete button wasn't deleting. Here is what I had to change in my code. This change occurred in the api/tasks.js file:

// Lines 35-53 in my tasks.js file
    'tasks.remove'(taskId) {
        check(taskId, String);

        const task = Tasks.findOne(taskId);
        if (task.private && task.owner === this.userId) {
            // If the task is private, make sure only the owner can delete it
            Tasks.remove(taskId);
        } 
    },
    'tasks.setChecked'(taskId, setChecked) {
        check(taskId, String);
        check(setChecked, Boolean);

        const task = Tasks.findOne(taskId);
        if (task.owner === this.userId) {
            // If the task is private, make sure only the owner can delete it
            Tasks.update(taskId, { $set: { checked: setChecked } });
        } 
    },

Deployment

The last downside for me was deploying this Meteor app. Meteor provides a wonderful service called Galaxy. It looks like it handles all of your Meteor deployment needs. What's the downside? There isn't a free tier. The cheapest you can get is 7 dollars a month. Which, for me, is too expensive to host a tutorial project.

So naturally, I looked into other ways of deploying this out to the public. The next best resource seemed to be Heroku. There are a couple of GitHub repos and articles that explain how to get this deployed onto the web. The major hurdle that I ran into was the recent discontinuation of the mongoLab Heroku add-on. Since Meteor uses a mongo database under the hood a lot of the ways to deploy this via Heroku involved using this recently non-existent Heroku add-on. So that was a bummer. I'm sure there's still a way to deploy this thing, but there was no easy way. The easy way was what I was looking for.

Conclusion

Meteor seems like a really great platform. I like how structured it is, and how it does all the complicated back end work for you while allowing you to use whatever front end framework you want. I can definitely see why many companies rely on Meteor for their web needs. However, the lack of easy deployment puts a wrench in the possibility of casual devs latching onto this. Unless I'm really dedicated to a project I'm working on, I'm not shelling out 7 bucks a month to host my fun web project. Especially when there are so many awesome free tools out there. I rate this dev experience a 3.4 out of 5.

That's all folks! Happy coding!

Building a blog with Strapi, Netlify, and React

Tyler Reicks — Wed, 09 Dec 2020 06:08:09 +0000

Intro

For this blog post, I am going to go over how I built my blog site with Strapi and React. I'm going to keep this as short and simple as possible. I followed along with the tutorial from Strapi themselves but there were hiccups along the way that I will address. Also, the tutorial only goes over how to make the blog locally. In this post, I'll explain what I did to deploy it out to the worldwide web.

Here is the link to the Strapi blog post that I followed along with. Overall it was great. It clearly described every step of the process and explained why things were done the way they were.

Initial hurdles

The first small issue I noticed was in their instructions. One of the first commands you have to run is yarn strapi install graphql. Before this command is run in the terminal you must ensure you are in the correct directory. Make sure to cd backend in your terminal. You make this backend folder in the step before this, they just don't tell you to navigate to the newly created folder before the next step. If you don't do this you will install graphql in the root folder of the project and your backend for the blog will not work.

Another problem I ran into was how to organize the project for version control. I use VSCode's built-in source control almost 100% of the time for my projects. The way this tutorial gets setup (I'm starting to learn many projects are set up this way), you'll end up with two separate directories for your frontend and backend code. Since the backend was largely already set up for me by Strapi I only committed my frontend code to GitHub. My backend directory also has version control but it is managed through the Heroku CLI. I'm sure there is a better way to manage this, but at the time I was too lazy to look into other options. Especially, when my current setup was working perfectly fine.

Lastly, I ran into an issue that I couldn't quite understand while running the frontend locally. Here was the error:

Could not find "client" in the context or passed in as an option. Wrap the root component in an , or pass an ApolloClient instance in via options

Luckily a couple of users had already solved this problem and put it in the comments. Link to comment here.

Everything else in the tutorial was pretty self-explanatory though! If I did have any confusion on something Strapi was so kind to add a video to the tutorial which also guided people through the process of creating the blog. I was able to have the blog finished and running locally on my machine pretty quickly. The next step was deploying the blog out to Heroku and Netlify.

Deploying to Heroku and Netlify

If you haven't used this stack with other projects before, from what I've gathered, Heroku is used to host your backend server and Netlify is used to host the frontend (at least that is what the tutorial recommended).

Heroku

Strapi has great documentation on how to deploy their backend to Heroku. Here is the link to that. Quick note, I created a Heroku project for this blog while I was still going through the tutorial. Pro tip: don't do that. The Strapi documentation shows you how to create the Heroku project in your backend folder which is so much easier. But if you took the same route I did, make sure to skip those create Heroku project steps (step 6) in the Strapi Heroku docs and connect to your existing project.

I used PostgreSQL for this. They also give you an option to use MongoDB. I'm sure both are great, but seemed like the PostgreSQL route was simpler.

Other than that the Heroku backend set up was pretty straight forward. Let's get to deploying the frontend.

Netlify

I think the Netlify set up might have been the easiest part of the project. Like I mentioned above, I had already pushed up my frontend directory to GitHub. All I had to do was make a Netlify account and connect my GitHub repo to my Netlify server (they step you through this process when you make a Netlify account).

Just like that my frontend code was being hosted on Netlify, but there was one problem. My Heroku backend wasn't connected to my Netlify frontend. That issue can easily be solved in your .env file in your frontend directory. Originally, your backend is set to your localhost URL. I think by default in the Strapi tutorial it is https://localhost:1337. You'll need to change that to the link of your Heroku app you just created. Pro tip: Make sure there is no trailing slash in the URL. For example, your .env file should have this one line in it: REACT_APP_BACKEND_URL="https://*your-app-name*.herokuapp.com".

There we go! Now you should have your blog uploaded to the internet for the whole world to see. Next, I'll go over what I'll call an "optional issue".

Image upload with Cloudinary

Figuring out how to upload an image for a blog post was a real pain. What is the issue you may ask? Here is the answer straight from the Strapi docs:

Heroku's file system doesn't support local uploading of files as they will be wiped when Heroku "Cycles" the dyno. This type of file system is called ephemeral, which means the file system only lasts until the dyno is restarted (with Heroku this happens any time you redeploy or during their regular restart which can happen every few hours or every day). Due to Heroku's file system, you will need to use an upload provider such as AWS S3, Cloudinary, or Rackspace.

One very easy way to get around this is to simply not add any pictures to your blog posts (this is why I consider this an optional issue). If you do go this route, make sure an image is not a required field in the Strapi admin portal (it was by default when I created my Articles content type).

I decided to use Cloudinary, but in hindsight, I think AWS would have been easier. I'm not giving Cloudinary enough credit though. It really wasn't that hard. I was just being stupid.

First, you want to head over to the Strapi Provider for Cloudinary npm package page. This page will show you how to add the Cloudinary provider to your Strapi blog.

Second, I set up my configuration differently than they recommended. Before I get into what I did, I should disclose that this is probably not a best practice. Instead of using an .env file for the Cloudinary config variables I just put them in the plugins.js file statically as strings. Yes, I know. For some reason when I was trying to use environment variables Cloudinary wasn't getting hooked up correctly. Here is a code example of what I did in my backend/config/plugins.js file:

module.exports = ({ env }) => ({
  // ...
  upload: {
    provider: "cloudinary",
    providerOptions: {
      cloud_name: "add_cloudinary_name_here",
      api_key: "add_api_key_here",
      api_secret: "add_api_secret_here",
    },
  },
  // ...
});

Definitely not the best way to solve that problem but it indeed worked.

Conclusion

Overall this was a really fun project. If you're new to coding, this tutorial (along with the video) walks you through the process very well. There may be a couple of hiccups to figure out, but hopefully, this article solves most of them. Like always, peace ✌️ and happy coding!