Forem: Lena Jeremiah

Picshaw - An image sharing app

Lena Jeremiah — Mon, 14 Oct 2024 04:43:58 +0000

This is a submission for the The Pinata Challenge

What I Built

Imagine attending an exciting event — like a developer conference—where you meet new people, take tons of photos and vidoes with them, and make lasting memories. But once the event wraps up, you realize you have no easy way to get all those great pictures.

Or consider a wedding ceremony: You know your guests captured beautiful moments throughout the day, but gathering all those photos means individually reaching out to each person. It is definitely not the way to go.

Picshaw offers a simple solution. As an event organizer, you can create a dedicated event folder on the platform, generate a shareable link, and invite your guests to upload the pictures they’ve taken. This way, everyone gets to relive the event from the unique perspectives of all attendees. No more missing moments, just memories shared effortlessly.

Picshaw Features

Creating Public and Private events. Public events are showcased on the “Discover Events” page, allowing anyone to browse and explore them. Private events, on the other hand, remain accessible only via a shareable link provided by the event organizer.

Upload Files Effortlessly Guests can easily upload their event photos to the designated event folder, making sure all memorable moments are gathered in one place.

Browse Uploaded Photos with Two Viewing Modes.
Users can explore photos either in list mode, which snaps images into a feed similar to Instagram, or in grid mode, offering a gallery-style layout.
Easily Share Event Links
Organizers can generate and share a unique link to invite guests to upload their pictures, streamlining the process of gathering photos.
Discover Public Events
Explore and browse all public events from the “Discover Events” page, opening up new ways to experience moments shared by others.

Seamless Authentication Users can sign in quickly and securely using Google sign-in or magic links, making the experience smooth and hassle-free.

Support for dark mode and light mode

Demo

Try out Picshaw live: https://picshaw.vercel.app

My Code

The full code is available on GitHub. Feel free to star the repo and follow me! 😊

lenajeremy / hack-photobomb

This is a Next.js project bootstrapped with create-next-app.

Getting Started

First, run the development server:

npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev

Open http://localhost:3000 with your browser to see the result.

You can start editing the page by modifying app/page.tsx. The page auto-updates as you edit the file.

This project uses next/font to automatically optimize and load Geist, a new font family for Vercel.

Learn More

To learn more about Next.js, take a look at the following resources:

Next.js Documentation - learn about Next.js features and API.
Learn Next.js - an interactive Next.js tutorial.

You can check out the Next.js GitHub repository - your feedback and contributions are welcome!

Deploy on Vercel

The easiest way to deploy your Next.js app is to use the Vercel Platform from the creators of Next.js.

Check out our Next.js deployment documentation for more…

View on GitHub

Using Pinata

Integrating with Pinata was one of the smoothest parts of the project. Here’s a breakdown of how I implemented it:

1. Initialize the Pinata SDK

I set up a dedicated file, @/lib/pinata.ts, to manage the Pinata configuration:

import { PinataSDK } from "pinata"

export const pinata = new PinataSDK({
    pinataJwt: `${process.env.PINATA_JWT}`,
    pinataGateway: `${process.env.NEXT_PUBLIC_GATEWAY_URL}`
})

2. In my API Route:

Ensured my user was logged in: I ensured the user was logged in before uploading.

const session = await auth()

if (!session || !session.user) {
   return respondError(new Error("User not authenticated"), "Failed to authenticate user", 401)
    }

Validate the Request: I checked if the event exists and ensured the file upload is within limits.

const form = await request.formData()
const files = Array.from(form.values())
const eventSlug = params["event-slug"]

try {
    const event = await prisma.event.findUnique({ where: { slug: eventSlug } })

    if (!event) {
        return respondError(new Error("Event not found"), undefined, 404)
    }

    if (files.length > 50) {
        return respondError(new Error("Limit of 50 files per request"), "Too many files", 400)
    } else if (files.length === 0) {
        return respondError(new Error("At least one file is required"), "No files", 400)
    }

Upload files to Pinata The next step is uploading the files.

const res = await Promise.all(files.filter(f => typeof f !== 'string').map(f => pinata.upload.file(f)))
        const successfulUploads = res.filter(r => r.status === "fulfilled")

        const uploadsWithPublicURL = await Promise.all(successfulUploads.map(async r => {
            const publicURL = await pinata.gateways.createSignedURL({ cid: r.value.cid, expires: 60 * 60 * 24 * 365 })
            return { ...r.value, publicURL }
        }))

Note: A better implementation would include checks for failed uploads. This way, users are notified about failed files and can retry uploading them.

Save Upload Data to the Database Finally, I saved the upload information into the database using Prisma.

const dbUpload = await prisma.upload.create({
            data: {
                text: `${session.user.name} uploaded ${files.length} images for ${event?.name}`,
                ownerId: session.user?.id ?? "",
                eventId: event?.id ?? "",
                files: {
                    createMany: {
                        data: uploadsWithPublicURL.map(file => {
                            return {
                                ownerId: session.user?.id ?? "",
                                eventId: event?.id ?? "",
                                cid: file.id,
                                publicURL: file.publicURL,
                            }
                        })
                    }
                }
            }
        })

        return respondSuccess(dbUpload, "Uploaded successfully", 201)

This flow ensures the event photos are uploaded securely and efficiently to Pinata, with successful uploads tracked and stored in the database for easy access later.

Rendering the uploaded images in the browser

1. Updating `next.config.js`

I first had to update the images field in the next.config.js file to allow NextJS optimize images from remote URLs.

/** @type {import('next').NextConfig} */
const nextConfig = {
  images: {
    remotePatterns: [
      {
        hostname: "sapphire-obliged-canidae-626.mypinata.cloud",
        protocol: "https",
      },
    ],
  },
};

export default nextConfig;

2. Fetching Event Data

In my client, I use a custom hook useFetch to fetch the details about the selected event

 const params = useParams();
  const eventSlug = params["event-slug"];
  const [viewMode, setViewMode] = React.useState<"grid" | "list">("grid");
  const router = useRouter();

  const {
    loading,
    data,
    trigger: getEventDetails,
  } = useFetch<void, GetUploadsResponse>(`/api/e/${eventSlug}`, undefined, {
    fetchOnRender: true,
  });

3. Rendering the Images:

I render the retrieved images inside a responsive grrrrrrrrrrid

  <div className="grid grid-cols-3 md:grid-cols-4 gap-1 my-6">
        {photos.map((file) => (
          <Image
            key={file.id}
            src={file.publicURL}
            width={400}
            height={400}
            alt=""
            className="aspect-square object-cover"
          />
      ))}
 </div>

Ideas for improvement

To further improve the app, it would be necessary to add some content moderation features. This would ensure that users don't post NSFW content on public groups. I started integrating with Google Cloud Vision but I didn't have enough time to complete it before the deadline.

Conclusion

Integrating Pinata’s Files API into Picshaw has greatly enhanced how images are uploaded and managed. Pinata provided seamless performance, and its intuitive API made implementation straightforward, allowing me to focus on building the core features of the app while trusting Pinata to handle file storage and delivery efficiently.

Overall, Pinata has been an essential tool in building a reliable and smooth image management system for Picshaw.

Also, I really enjoyed building Picshaw.

Follow me on X @jeremiahlena13

Build AI Chat Apps in Minutes: Unleashing the Power of NeonChat Starter Kit

Lena Jeremiah — Mon, 02 Sep 2024 03:00:22 +0000

This is a submission for the Neon Open Source Starter Kit Challenge : Ultimate Starter Kit

My Kit

Building AI chat apps can be a real challenge. You've got to juggle user logins, make sure AI responses appear instantly, save all those conversations, and tie everything to a solid database. It's a lot to handle, even for experienced developers. But here's the good news: I've cooked up something that's about to make your life a whole lot easier.

Welcome to NeonChat, the ultimate open-source starter kit for building AI-powered chat applications! This kit combines the power of Next.js, Prisma, and Neon's serverless Postgres to create a flexible, scalable, and developer-friendly foundation for your next AI chat project.

The main features? Well, we've got:

A clean chat interface that doesn't look like it's from the 90s
Login with email or GitHub
Real-time chatting with AI
Saving conversations (Prisma doing it's thing with Neon's Postgres)
Ultimately, a CLI tool which allows developers to easily select whichever AI provider they want to use and get started in minutes.

Link to Kit

You can find the NeonChat starter kit on GitHub https://github.com/lenajeremy/nextjs-ai-neon-starter
The CLI tool is https://github.com/lenajeremy/nextjs-ai-starter-script

Or you can simply run this command to get started

npx neonchat

My Journey

I chose this stack for several reasons:

Next.js because it's just awesome.
Prisma makes database stuff really simple and intuitive.
Neon's Postgres for it's scalability and its ease of use. Also, because I needed to 😂😂
I used Vercel's AI SDK because I wanted people to be able to use whatever AI service they wanted to and not be restricted to the one (the one that was kind enough to give me free credits). And also because it was really easy to integrate.
Shadcn because it's the best.

Things I learned:

Trying to make something flexible AND simple is like juggling flaming swords while balancing on a tightrope.
Documentation is Key: Writing clear documentation is as important as writing good code. I put extra effort into creating a README that guides developers through every step of the process

To help you get up and running quickly, here's a brief overview of how to use NeonChat:

Get Started

Run npx neonchat in your terminal.
Follow the prompts to select your preferred AI provider and configure your project.
The CLI tool will set up your project, customize the starter templates based on the selected AI provider and install the required dependencies for you.
All you have to do is navigate to the created project directory and
Start the development server with npm run dev.
Visit http://localhost:3000 to see your NeonChat instance in action!

Conclusion

NeonChat aims to empower developers to build AI-powered chat applications quickly and efficiently. By combining a feature-rich frontend, a flexible backend, and an intuitive setup process, we hope to inspire a new wave of innovative AI projects.
We're excited to see what the community builds with NeonChat. Happy coding!

Practical Guide to Web-scraping in Python

Lena Jeremiah — Fri, 17 Mar 2023 14:04:58 +0000

Web scraping, in simple terms, is a technique used to collect useful data from a website. You'd agree that there's a ton of information on the internet, and most times, this information might not be structured exactly how we want. So, web scraping allows us to extract data from the internet, which can then be restructured or rearranged so that it is useful to us.

In this article, I'll be taking you step-by-step through the web scraping process, providing you with useful advice on how to develop your skills, tips on what to do and what not to do when scraping websites (copyright infringement in particular), and much more. Additionally, we'd scrape two websites, CoinMarketCap and Bitcoin.com, to get the prices of cryptocurrencies and to get latest news about Bitcoin.

Prerequisites

Basic programming skill. Although we will be using Python in this article, you can still benefit from it even if you are familiar with other programming languages.
A fundamental understanding of HTML, HTML elements and attributes, etc.
Knowledge of Chrome DevTools (not required)
A computer or other tool for running Python code.

Let me quickly introduce you to two packages: requests and BeautifulSoup. They are Python libraries that you use to scrape websites. In this article, we'd require them.

The Web-scraping Process

When you go to your browser and input the url of the website you want to visit, your browser makes a request to the server. The server then responds to your browser. The browser receives data from the server as text, HTML, JSON, or a multipart request (media file).

When the browser receives HTML, it parses it, generates DOM nodes, and renders it to the page, displaying the lovely website on your screen.

When scraping websites, we need this HTML content, and that introduces us to the first package requests.

Requests

As the name suggests, it's used for sending requests to servers. If you have any experience in frontend development or Javascript, it's the equivalent of fetch API in Javascript.

For us to be able to scrape website content, we need to ensure that end server endpoint we're calling returns HTML.

After getting the HTML, the next course of action is to collect the data we need. And this brings us to the second package BeautifulSoup.

Beautiful Soup

Beautiful Soup is a library used for pulling data out of HTML and XML files. Beautiful soup contains a lot of very useful utilities, which makes web-scraping really easy.

Building the Scrapers (Writing Code)

After much talk, let's get into the crux of the matter, writing code and building the actual web-scrapers.

I've created a Github repo which contains the code for the scrapers. The complete code is at the completed branch, checkout to the get-started branch to get started.

I'll be creating the project from scratch in this article, for those who might not want to clone a Github repo.

PROJECT SETUP

Create a new folder. Name it whatever you want.
Set up a new virtual environment.

Run the following commands to set up a new virtual environment


 shell
# if you've not already installed virtualenv
pip install virtualenv

# you can name your environment whatever you want
python -m venv <name_of_environment>

After running the previous commands successfully, you should see folder with the name of the virtual environment you just created.

You also need to activate the virtual environment. To do that, run the following commands depending on your OS.


 shell
# macOS / linux
source ./<virtual_env_name>/bin/activate

# windows
<virtual_env_name>/Scripts/activate.bat   # In CMD
<virtual_env_name>/Scripts/Activate.ps1   # In Powershell

Install the required packages.

Having activated our virtual environment, let's go on ahead to install the packages. To do so, run the following in your terminal:


 shell
pip install requests beautifulsoup4

Generate a requirements.txt file.

If you'll be sharing with code with someone or a team, it's important that the other party installs the exact same package version you used to build the project. There could be breaking changes to the package used, so it's important that everybody uses the same version. A requirements.txt file lets others know the exact version of the packages you used on the project.

To generate a requirements.txt file, just run:


 shell
pip freeze > requirements.txt

After running this, you should see a file containing the packages required to run the project.

If you're working in a team and there's a requirements.txt file in the project, run pip install -r requirements.txt to install the exact version of the packages in the requirements.txt file.

How many times did I say requirements.txt file?🤡

Add a .gitignore file.

This is used to indicate to git the files or folders to ignore when you run git add . Add /<virtual_env_name> to your .gitignore file.

Scraping Bitcoin.com.

Now, we'll be scraping the first website. The first thing we need to do is to get the HTML with requests.

So, create a file bitcoinnews.py (you can name it whatever you want).

Add the following code to the file


 python
import requests

def get_news():
    url = '<https://news.bitcoin.com>'

    response = requests.get(url) # making a request
    response_text = response.text # getting the html

    # here's where we should get the news content from the html, we're just printing the content to the terminal for now
    print(response_text)

get_news()

A bit on HTML and frontend development

When scraping websites, it's important that we understand basic HTML as this would help us decide the best way to extract our data.

HTML (HyperText Markup Language) is the standard markup language for creating webpages. HTML is used to describe the structure of a web page. HTML elements are the building blocks of every webpage. HTML elements are related to one another in a way that resembles a tree.

In the diagram above, we have the root element, the <html> element, which has two children, the <head> and <body> elements. The <head> tag has a single child, the <title> tag which has text as simple text as its child. The <body> has three children (one heading tag and two paragraph tags).

This tree-like structure facilitates the composition of elements and the definition of relationships between them. It makes it simple to write a paragraph and include links within it.

HTML elements, in addition to having children, have attributes, which add a lot more functionality to them. An 'image' element would require the source of the image it is supposed to display. When you click on a link (anchor tag), it needs to know where you want to go. Attributes can also be used to distinguish between a single element and a group of elements.

Understanding HTML attributes is really important because when scraping a website, we need specific data and it's important we can target the elements that have the data we need.

If you want to learn more about HTML, you can check out W3schools.com

Scraping top crypto prices from CoinMarketCap.

Before we get back to our Bitcoin News scraper, let's scrape the prices of the top cryptocurrencies from CoinMarketCap, alongside the images.

TIP: Before you start scraping a website, disable Javascript and open the page in that browser. This is significant because some websites display content on the webpage using client-side Javascript, which cannot be used by our scrapers. So it only makes sense to see the exact HTML that the server is returning.

With Javascript running

Without Javascript running

If you open coinmarketcap.com on a browser with Javascript disabled, you'd notice that you can only get the prices for the first ten cryptocurrencies. You'd also notice that on a browser where Javascript is enabled, you can scroll to get prices of other cryptocurrencies. This difference in the look of page (HTML) can cause you a lot of trouble when scraping a website.

Back to the code editor

Create a new file. You can call it whatever you want. I called mine coinmarketcap.py
Add the following code to it.


 python
import requests
from bs4 import BeautifulSoup

def get_crypto_prices():
    url = '<https://coinmarketcap.com>'
    response_text = requests.get(url).text

    soup = BeautifulSoup(response_text, 'html.parser') #added this line

    print(soup)

get_crypto_prices()

This looks almost exactly like the previous file, I only changed the URL and function name, and also added the line with BeautifulSoup.

That line of code parses the HTML text we get from the endpoint and converts it to a BeautifulSoup object which has methods we can use to extract information from the text.

Now that we have access to the HTML, let's figure out how to scrape the data we need: in our case, prices of cryptocurrencies.

Open up DevTools on your web browser. If you're using Chrome on a Mac, you can press Cmd + Option + I to open up DevTools.

Or you can just right click on the target element and click inspect.

If you look at the elements in the Inspect Element section of Chrome DevTools, you'd notice that there's a table element which children. Inside the table element, we have the colgroup, thead, tbody elements. The tbody element contains all the content of the table. Inside the tbody element, you'd notice that we have a tr which denotes a table row, which also contains multiple td elements. td denotes table data.

Now, just knowing that the data we need is inside of table isn't enough. We have to dig deeper into the HTML tree to be able to extract the exact data we need. In this case, we need the name of the cryptocurrency, the shortened name e.g. BTC for Bitcoin, the current price and the image of the cryptocurrency.

If you look closely, you'd notice that the name of the cryptocurrency is in a paragraph tag with the class sc-e225a64a-0 ePTNty. The shortened name of the cryptocurrency is also in a paragraph tag with a different class name sc-e225a64a-0 dfeAJi coin-item-symbol.

We use attributes like class and id to uniquely identify HTML elements or groups of familiar HTML elements. When we have these distinct attributes, we can use them to target the elements and extract the values we need from them.

What we've done so far?

From analysing Coinmarketcap's website, we've seen that data of each cryptocurrency is in a row and each row has children that contains data we'd like to scrape.

Let's get back to the code editor and update our coinmarketcap.py file


 python
import requests
from bs4 import BeautifulSoup

def get_crypto_prices():
    url = '<https://coinmarketcap.com>'
    response_text = requests.get(url).text

    soup = BeautifulSoup(response_text, 'html.parser')

    # get all the table rows
    table_rows = soup.findAll('tr')

    # iterate through all the table rows and get the required data
    for table_row in table_rows:
        crypto_name = table_row.find('p', class_ = 'sc-e225a64a-0 ePTNty')
        shortened_crypto_name = table_row.find('p', class_ = 'sc-e225a64a-0 dfeAJi coin-item-symbol')
        coin_img = table_row.find('img', class_ = 'coin-logo')

        print(crypto_name, shortened_crypto_name)

    get_crypto_prices()

Notice the difference between findAll and find

If you run the above code, you'd get this

You can see that some of the data is returning None. This is as a result of the remaining table rows which are empty. What we can do here is to check to see if there is a value before printing the values.

Updating our for loop, we have this:


 python
# iterate through all the table rows and get the required data

for table_row in table_rows:
    crypto_name = table_row.find('p', class_ = 'sc-e225a64a-0 ePTNty')
    shortened_crypto_name = table_row.find('p', class_ = 'sc-e225a64a-0 dfeAJi coin-item-symbol')
    coin_img = table_row.find('img', class_ = 'coin-logo')

    if crypto_name is None or shortened_crypto_name is None:
        continue
    else:
        crypto_name = crypto_name.text
        shortened_crypto_name = shortened_crypto_name.text
        coin_img = coin_img.attrs.get('src')

        print(crypto_name, shortened_crypto_name)

If there's a value for crypto_name or shortened_crypto_name, we get the text from the HTML element and print it to the console. We also get the src of the crypto image.

Running the updated code, we should have this:

Now, let's get the prices for each cryptocurrency.

Going back to our Chrome Devtools and right-clicking on the price text, we should see this:

We see that the price of the currency is in a span tag, which is wrapped in an anchor tag: the a tag, which has a class value of cmc-link. However, using the class of the anchor tag to scrape the price won't work because the class cmc-link does not uniquely identify the element we're attempting to target.


 javascript
const bitcoinRow = document.querySelectorAll('tr')[1]
const cmcLinks = bitcoinRow.querySelectorAll('.cmc-link')

console.log(cmcLinks) // NodeList(4) [a.cmc-link, a.cmc-link, a.cmc-link, a.cmc-link]

If you run the Javascript code above in the browser console, you'd see that there are four links with the class name cmc-link in every row. This is definitely not the best way to get the price of the crypto at that row.

Let's look at the parent: the div with the class name sc-8bda0120-0 dskdZn

Notice that when the element is hovered on the console, the price is also hovered on the web page. So, this proves to be a better way to get the price of the cryptocurrency.

Updating the code, we have:


 python
import requests
from bs4 import BeautifulSoup

def get_crypto_prices():
    url = '<https://coinmarketcap.com>'
    response_text = requests.get(url).text

    soup = BeautifulSoup(response_text, 'html.parser')

    # get all the table rows
    table_rows = soup.findAll('tr')

    # iterate through all the table rows and get the required data
    for table_row in table_rows:
        crypto_name = table_row.find('p', class_ = 'sc-e225a64a-0 ePTNty')
        shortened_crypto_name = table_row.find('p', class_ = 'sc-e225a64a-0 dfeAJi coin-item-symbol')
        coin_img = table_row.find('img', class_ = 'coin-logo')
        crypto_price = table_row.find('div', class_ = 'sc-8bda0120-0 dskdZn')

            if crypto_name is None or shortened_crypto_name is None or crypto_price is None:
                continue
            else:
                crypto_name = crypto_name.text
                shortened_crypto_name = shortened_crypto_name.text
                coin_img = coin_img.attrs.get('src')
                crypto_price = crypto_price.text

                print(f"Name: {crypto_name} ({shortened_crypto_name}) \\nPrice: {crypto_price} \\nImage URL: {crypto_img_url}\\n")

get_crypto_prices()

Running the updated code, we should get this:

Whooopppsssssss... That was a lot to process. I hope you were able to follow through.
Grab that cup of coffee, you deserve it👍

Going back to scraping Bitcoin News

Now, with the knowledge we've gained from scraping Coinmarketcap, we can go on ahead to complete our Bitcoin News scraper. We'd be using this scraper to get the latest news in the crypto space.

Right-clicking on the first news and opening the Inspect Elements section of Chrome Devtools would reveal that news headlines have a story class. However, if you right-click on more news headlines, you'd discover that they are variations to the news. There's the medium story, small, huge, tiny, etc, with different class names to uniquely identify each type.

A very important skill required for web-scraping is the ability to take a close look at how the HTML of a webpage is structured. As long as you understand the structure of a webpage, extracting useful content from the elements is a small task.

Heading back to the code editor:


 python
import requests

def get_news():
    url = '<https://news.bitcoin.com>'

    response = requests.get(url) # making a request
    response_text = response.text # getting the html

    print(response_text)

    soup = BeautifulSoup(response_text, 'html.parser')
    all_articles = soup.findAll('div', class_ = 'story')

    for article in all_articles:
        print(article.text.strip())

get_news()

When you run the above code, you might get this:

To prevent malicious bots from accessing their website, some websites employ Cloudflare Bot Management. Cloudflare maintains a list of known good bots that are granted access to the website, such as search engines, copyright bots, chat bots, site monitoring bots, and so on. Unfortunately for web-scraping enthusiasts like you and me, they also assume all non-whitelisted bot traffic is malicious.

However, they are a number of ways this can be bypassed and the ease depends on how much of a threat our bot poses to Cloudflare and the Bot Protection plan the website owner subscribed for.

A list of ways to avoid Cloudflare can be found here. Click here to learn more about Cloudflare Bot Management.

In this article, we'll look at the most basic, which is setting the User-Agent header. By doing so, we pretend to the server that the request is coming from a regular web browser.


 python
def get_news():
    url = '<https://news.bitcoin.com>'
    headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
    }

    response = requests.get(url, headers=headers)

    # the rest of the code

After adding the above line of code and running it, we should get this:

Now that we have the actual web content, let's go to our browser to inspect the webpage:

Right-clicking on the first news there, we should see this:

Scraping news headline

Notice that the h6 element has a class of story__title story--medium__title. If you right-click on another news, you might see something like story__title story--huge__title, or story__title story--large__title. Notice how there's already a pattern: the headline of each news always has the class of story__title. That seems like the best way to target the headline of the news.

Scraping news URLs

If you look closely, you'd notice that the news title has a parent which is a link. This link contains the URL of the news in it's href attribute.

Putting all of these together, we can write the code for the scraper.


 python
import requests
from bs4 import BeautifulSoup

def get_news():
    url = '<https://news.bitcoin.com>'
    headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
    } # headers to bypass Cloudflare Protection

    response = requests.get(url, headers=headers)
    response_text = response.text

    soup = BeautifulSoup(response_text, 'html.parser')
    all_articles = soup.findAll('div', class_ = 'story')

    for article in all_articles:
        news_title_element = article.select_one('.story__title')
        news_url = news_title_element.parent.attrs.get('href')
        news_title = news_title_element.text.strip()

        print(f"HEADLINE: {news_title} \\nURL: {news_url}\\n")

get_news()

Running the above code, we have the news headline and the URL

Utilising scraped data

Now that we've successfully scraped required data from the websites, let's save the scraped data. If you're working with a database, you could immediately save it to the data, you could perform computations with the data, you could save it for future use, whatever the case is, data you've scraped is not very useful in the console.

Let's do something really simple: save the data in a JSON file


 python
import requests
from bs4 import BeautifulSoup
import json

# some code
all_articles = soup.findAll('div', class_ = 'story')
scraped_articles = []

for article in all_articles:
    news_title_element = article.select_one('.story__title')
    news_url = news_title_element.parent.attrs.get('href')
    news_title = news_title_element.text.strip()

    scraped_articles.append({
        "headline": news_title,
        "url": news_url
    })

with open ('news.json', 'w') as file:
    news_as_json = json.dumps({
        'news': scraped_articles,
        'number_of_news': len(scraped_articles)
    }, indent = 3, sort_keys = True)

    file.write(news_as_json)

Now, we're saving saved news in a JSON file, which can be used for whatever you want.

Your JSON file should look something like this.

Whooopppsssss.⚡️⚡️ That was a whole lot to take in.

What not to do when scraping websites

Overloading the servers with too many requests at a time. The reason why many websites frown against bots is their frequent abuse. When you make a request, the server uses resources to be able to process the request. Making a ton of requests might make the server run out of resources, which isn't good.


 python
"""
⛔️ DON'T DO THIS
"""
for i in range(10000):
    request.get('<https://reallyamazingwebsite.com>')

Your IP address might get blacklisted.

Disrespecting copyright rules. Web scraping is completely legal if you scrape data publicly available on the internet. But some kinds of data are protected by international regulations, so be careful scraping personal data, intellectual property, or confidential data. Some websites might openly state that content on the page should not be distributed by any other means, that should be respected.

You've finally made it to the end of the article, well-done champ.

Conclusion

In conclusion, web scraping in Python can be a useful tool for gathering information from websites for a variety of purposes. Python provides a user-friendly and efficient way to extract structured data from websites through the use of popular libraries such as BeautifulSoup. However, it is critical to remember that web scraping should always be done ethically and in accordance with the terms of service of the website. It is also critical to be aware of any legal constraints that may apply to the data that is being scraped. With these factors in mind, Python web scraping can be a useful skill for anyone looking to collect and analyse data from the web.

If you enjoyed reading this article, you can follow help share on your socials and follow me on social media.

What is Debouncing?

Lena Jeremiah — Sun, 13 Feb 2022 18:57:45 +0000

Performance is one of the many things that are prioritized when building websites and software generally. As software engineers, it's imperative that we write code with performance in mind, as this would help a great deal to improve the overall user experience of our software.

In this article, we'd be taking a look at Debouncing, a very useful technique for improving the performance of client-side applications.

Before we look at what debouncing is, let's take a brief look at event listeners.

Event Listeners

When building client-side applications, event listeners are things we can't do without. Every client-side application would require that the user interacts with it for it (the app) to be useful, these interactions could be clicking a button, scrolling to view more content, typing into an input field, submitting a form and so many more. These event listeners have callbacks that fire whenever the event they're listening for is triggered.

On some occasions, these event listeners would have performant-heavy callbacks, hence, the need for us to control how and when these callbacks are called. And this is where debouncing comes to play.

Let's assume that we have a search bar that makes a request to an API whenever a user makes a change to the input field. That means if a user wants to search for the term 'What is debouncing?', the browser would have to make a total of 19 requests to the API.

Here's a code pen so you can test it out.

Now, with this approach, our browser makes a request for every single keystroke the user makes on the keyboard, which leaves us with multiple useless requests.

How about we find a way to prevent the request from being made until the user has finished typing? Would this solve the problem? Now, this is exactly what debouncing is.

Debouncing

Debouncing is a method in which a function is prevented from running until a certain amount of time has elapsed without the function being called. In our case, the request won't be made until the user has stopped typing.

Implementing debouncing, our event callback would look something like this:

let timeout;
// other codes
inputField.addEventListener('input', () => {
    clearTimeout(timeout);
    timeout = setTimeout(makeRequest, 500);
})

From the above snippet, whenever a user types, we clear a timeout, which does not exist when the function is called initially. We then create another timeout using setTimeout, which calls the makeRequest function whenever the time has elapsed. That means that if the timeout has not exceeded and the user types, we clear the previous timer and create another one. This way, only the last timeout would run. Hence, solving our problem of having multiple requests. n

This is what it looks out after implementing debouncing:

Makes more sense, right?

Here's a codepen if you want to take a close look at the implementation

An extra something

Instead of manually writing the debouncing function every time we want to implement this amazing functionality, we can just create a utility function that takes a callback and a timer and then return a function that has the whole debouncing functionality.

Something like this:

function debounce(func, timeINMS) {
  let timeout;

  return function () {
    clearTimeout(timeout);
    timeout = setTimeout(func, timeINMS);
  };
}

let debouncedHello = debounce(() => console.log("say hello", Date.now()), 800);

Here, the debounce function takes two arguments, a function that logs say hello and a number that represents time(in milliseconds) through which the function should be delayed, then returns a function that has the function of debouncing.

Debouncing is a very simple and intuitive technique but also considerably improves performance.

I hope you were able to follow through with the concept. In my next article, I'd be talking about another technique which is a little like debouncing: Throttling.

Stay tuned and stay safe❤✌️

Memory Leaks, How to avoid them in a React App.

Lena Jeremiah — Sat, 08 Jan 2022 09:58:39 +0000

What is a memory leak?

According to Wikipedia, a memory leak is a type of resource leak that occurs when a computer program incorrectly manages memory allocations in a way that memory that is no longer needed is not released. A memory leak may also happen when an object is stored in memory but cannot be accessed by the running code.

Simply put, a memory leak is said to occur whenever inaccessible or unreferenced data exists in memory. Nowadays, many modern programming languages have techniques for clearing out data that is no longer needed, garbage collection, but it turns out there are other not-so-popular errors which can expose your React app to memory leaks and, to a great extent, reduce the performance of your app.

Let's look at some causes of memory leaks.

Causes of Memory Leaks in a React Application

Memory leaks in React applications are primarily a result of not cancelling subscriptions made when a component was mounted before the component gets unmounted. These subscriptions could be a DOM Event listener, a WebSocket subscription, or even a request to an API.

The first two are not too much of a challenge, as we can easily remove an event listener or unsubscribe from the WebSocket before the component gets unmounted. But the last one might require a little bit of extra work.

A typical React workflow

import { useState, useEffect } from 'react';
import { Link } from 'react-router-dom';
import axios from 'axios';

const MyCompany = function() {
    const [ company, setCompany ] = useState(null);
    useEffect(() => {
        (async () {
             const { data } = await axios.get(
                 'https://random-data-api.com/api/company/random_company'
             );
             setCompany(data);
        })();
    }, []);

    return (
        <>
            <pre>{JSON.stringify(company, null, 3)}</pre>
            <Link to = '/anotherpage'>Another Interesting Page</Link>
        </>
    )
}

In the code snippet above, we have a simple component MyCompany which when mounted, makes a request to get a random company and sets value of company state to the value gotten from the API.

The Problem

Assuming our user has a very slow internet connection and then decides to leave the current page for another interesting page, the request would have already been made and our browser would be expecting a response, which when received, would lead us to call setState on a component that's no longer mounted.

Aside from setting state, we would now have unimportant data in our app with no means of accessing them. This process is repeated multiple times while the user uses the app, filling up useful memory with useless and inaccessible data and leading to serious performance issues.

We've seen the problems and I believe you understand, now let's look at how to solve this problem.

The Way Forward: AbortControllers

Having understood the problem, what we'd do to salvage the situation is cancel the request the moment our component unmounts, ensuring we don't get any data from the API.

So, how do we cancel requests? AbortControllers

According to MDN, the AbortController represents a controller object that allows you to abort one or more Web requests as and when desired. That's quite explanatory!!

AbortControllers are created with the new AbortController() syntax, initializing an instance of the AbortController class. Every AbortController object has a read-only signal property which is passed into requests, and an abort() method which is whenever you want to cancel a request.

Now using AbortControllers, our code should look like this:

import { useState, useEffect } from 'react';
import { Link } from 'react-router-dom';
import axios from 'axios';

const MyCompany = function() {
    const [ company, setCompany ] = useState(null);

    useEffect(() => {
         let abortController;
        (async () {
             abortController = new AbortController();
             let signal = abortController.signal;    

             // the signal is passed into the request(s) we want to abort using this controller
             const { data } = await axios.get(
                 'https://random-data-api.com/api/company/random_company',
                 { signal: signal }
             );
             setCompany(data);
        })();

        return () => abortController.abort();
    }, []);

    return (
        <>
            <pre>{JSON.stringify(company, null, 3)}</pre>
            <Link to = '/anotherpage'>Another Interesting Page</Link>
        </>
    )
}

Now, when our user navigates to a new page, our AbortController cancels the request and we don't have to worry about having data leaks.

NOTE: Calling abortController.abort() after the request has been completed doesn't throw any errors. The abortController simply does not take any action on an already complete request.

Using AbortControllers in your web apps can help improve performance and prevent memory leaks, so it's something you should actually use.

Thanks for reading❤❤

Forem: Lena Jeremiah

Picshaw - An image sharing app

What I Built

Picshaw Features

Demo

My Code

lenajeremy / hack-photobomb

Getting Started

Learn More

Deploy on Vercel

Using Pinata

1. Initialize the Pinata SDK

2. In my API Route:

Rendering the uploaded images in the browser

1. Updating next.config.js

2. Fetching Event Data

3. Rendering the Images:

Ideas for improvement

Conclusion

Build AI Chat Apps in Minutes: Unleashing the Power of NeonChat Starter Kit

My Kit

Link to Kit

My Journey

Things I learned:

Get Started

Conclusion

Practical Guide to Web-scraping in Python

Prerequisites

The Web-scraping Process

Requests

Beautiful Soup

Building the Scrapers (Writing Code)

PROJECT SETUP

Scraping Bitcoin.com.

A bit on HTML and frontend development

Scraping top crypto prices from CoinMarketCap.

Back to the code editor

What we've done so far?

Going back to scraping Bitcoin News

Heading back to the code editor:

Scraping news headline

Scraping news URLs

Utilising scraped data

What not to do when scraping websites

Conclusion

What is Debouncing?

Event Listeners

Debouncing

An extra something

Memory Leaks, How to avoid them in a React App.

What is a memory leak?

Causes of Memory Leaks in a React Application

A typical React workflow

The Problem

The Way Forward: AbortControllers

So, how do we cancel requests? AbortControllers

1. Updating `next.config.js`