Forem: Lauren C.

Web Scrapper: E-commerce Price Tracker

Lauren C. — Mon, 11 Nov 2024 02:13:03 +0000

Overview

Extract data from file using terminal
Extract data from a file using a script
Putting it all together to build a web scraper

Introduction

Each step of the tutorial has a purpose. Try not to skip a step, because you will miss valuable skills needed to become a proficient software engineer. These small steps are the skills you need to own your craft.

Section 1: Extracting data from a file using terminal teaches command line skills and narrowing down on arguments needed to properly extract data.

Section 2: Extracting data from a file using a script give learners practice in writing scripts which are precise. In computer science you want to be accurate, because the computer can handle precise. Writing scripts give learners practice in creating accurate scripts, which will be needed for the final step.

Section 3: Putting it all together to build a web scraper is the final step to show you have mastered the skill of web scraping. You will build a web scraper you can now use to crawl the website. You can gather information or build scrappers for freelancing clients.

How to Build a Web Scrapper

Assignment: create a E-commerce Price Tracker

Section 1:

select 1 website

For this tutorial I will be using LinkedIn.

Download 1 job listing html page onto your computer: wget

Website pages can be downloaded as html files by right clickin on the page, selecting "View page source" then right click again and "Save as". There is another way using wget.

Wget is a command line application which will save the file onto our computer at our current directory. The -O flag is used to specify the output file name we select. You can see more informaiton about wget and other flags to use by using the help command below:

$wget -O name_your_file.html www.websiteYouAreScraping.com

$ wget --help

Using command line applications

grep is a command line application which searches a file for a specified term. The vertical bar (|) feeds the output for the grep command into the wc command. wc is another terminal application, but this application can count the number of lines, words, and characters in the input data:

$grep "search term" file_name.html | wc -l

Sort will organize the data. It takes in the output of the grep command, orders it and outputs sorted data. the carrot symble (>) pipes the sorted data into a new file you specify.

$grep "search term" file_name.html | sort > output_file.txt

Alias Commands & Shortening default prompt

When working with a long command which you use often, it can be useful to create an alias for that command. Alias commands for the terminal can be made by adding the comment to the .bash_profile or .bashrc file

When the prompt in the terminal becomes long, because the list of directories are all shown, it can be easily shortened. Use the comment export PS1=[\!:\w]$. The forward slash exclamation point and the forward slash w are special characters meaning which parts of omit and which parts to include in the prompt. You can read more on https://ss64.com/bash/syntax-prompt.html

Section 2:

Section 3:

Closing Remarks

Now that you have completed the tutorial, I encourage you to select a website and practice scraping data from it. Try using a web scraping framework to or get paid for your new skills by finding freelancing work. Go onto UpWork.com or Freelancer.com and try to create a web scraper for a client and get paid for your skills.

Web Scrapper Project Ideas

E-commerce Price Tracker: Scrape product prices from various e-commerce websites and track price changes over time.
Job Listings Aggregator: Extract job listings from multiple job portals and compile them into a single, searchable database.
Social Media Sentiment Analysis: Scrape social media platforms for mentions of a particular brand or product and analyze the sentiment of the comments.
Real-Time News Aggregator: Extract news articles from various news websites and create a real-time news feed.
Stock Market Trend Analysis: Scrape stock market data and analyze trends to predict future movements.
Recipe Recommendation Engine: Extract recipes from cooking websites and create a recommendation engine based on user preferences.
Automated Travel Itinerary Planner: Scrape travel websites for flight, hotel, and activity information to create personalized travel itineraries.
Sports Statistics Collector: Extract sports statistics from various sources and create a comprehensive database for analysis.
Product Review Aggregator: Scrape product reviews from e-commerce sites and compile them into a single, searchable database.
Real Estate Market Analysis: Extract real estate listings and analyze market trends to provide insights for buyers and sellers.

Hugging Face: Interacting with Roberta and Hugging Face for the first time

Lauren C. — Sat, 14 Sep 2024 23:41:49 +0000

This is my first interaction with Hugging Face. The free and open Ai option

Date: September 15, 2024

Prerequisites: I bought a 1T external SSD and created this tutorial using the external hard drive. I am using a windows computer. This is where the tutorial begins.

Change directory to D drive
$D:
create directory on external SSD titled version of python and pip I'm using

$mkdir python312
$cd python312
$mkdir Scripts

install python and pip file from website. Save in python312 directory and install virtual environment with command:
$pip install virtualenv
verify it's working by checking the version

python --version
pip --version
virtualenv --version

Create virtual environment. Tool is called virtualenv. Name of the virtual env. is my_venv.
Next command "python" invokes interpreter. -m runs module as script. venv module is invoked and my_venv is name of virtual environment

$ virtualenv my_venv
$python -m venv my_venv

Activate virtual environment

$.\my_venv\Scripts\activate

Install Transformers, datasets, and pytourch

$pip install transformers
$pip install datasets
$pip install torch
$pip install tensorflow

install Visual Studio from Microsoft website to edit python files and create python files
Open Visual Studio Installer
Select Modify and add Python development by checking the box
click modify
in Scripts directory create new application in visual studios. Save in script file in D drive. Name "roberta_hugging_face" and save solution and project in same directory

add code to file:

from transformers import pipeline

# Load the sentiment analysis pipeline
sentiment_pipeline = pipeline("sentiment-analysis")

# Perform sentiment analysis
result = sentiment_pipeline("Hugging Face is creating amazing tools for NLP!")
print(result)

run file. You should be in:

D:\python312\Scripts\roberta_hugging_face

$python roberta_hugging_face.py

install backward compatible package and retry running
$pip install tf-keras
$python roberta_hugging_face.py
When download is complete you will see the result for the pipeline "Hugging Face is creating amazing tools for NLP!"
To deactivate
$deactivate
Safely disconnect SSD

Pedagogy for Self Taught Developers: Introduction to Identifying Your Technical Skill Level with Bloom’s Taxonomy

Lauren C. — Tue, 08 Aug 2023 15:10:58 +0000

Intro

When you're stuck in tutorial hell, the right answer is not to just keep on going. Developers need a real way to measure growth in a coding skill. Thanks to the work of teachers, and educators we can identify growth. We can be purposeful about the way we learn, so to maximize growth and minimize time needed to master the skill. Learning hard things isn't new. Educators have been teaching tough topics since the dawn of education.

If you are new to code or you are leading a team of junior developers, everyone could use a marker to measure progress. Understanding what actionable steps can take you from zero to mastery will help you reach success sooner rather than later.

Bloom's taxonomy is a well respected system of classification taught in every college of education across America. Simply put, Bloom's taxonomy is a tool for teachers to identify which activities to best support student's growth.

As a blog post series, I will be diving into Bloom's taxonomy for real world application for learning a new technical skill. Anyone should be able to use this blog post series for any technical skill and see growth in their understanding. Seems like a lofty goal for a blog, but Bloom's Taxonomy has been used for many decades. Bloom's taxonomy can be used by anyone learning highly technical skills.

Articles

In the Articles section, I will summarize one to three high quality reading on the topic of Bloom's taxonomy as it relates to computer science education.

Actionable Steps

In this section I will bullet some tasks anyone could work on to grow and show their understanding. An example of a deliverable could be summarize a video tutorial or completing a coding challenge. I will provide some PDF templates to help you get started.

Actionable Steps are meant to get you started working quickly. Feel free to make up your own action items. If you do, please share links to what you have built or learned in the comments below.

High Quality Learning Materials

Sometimes Bloom's taxonomy can be seen in learning materials online today. When I find good learning materials, I will share them with you. I will also give information on why I believe it is a good tool to learn from.

Final Mentions

Each blog post follow the same format: Intro, Academic Articles, Actionable Steps, High Quality Learning Materials, and Final Mentions. The blog posts in this series will keep to these sections.

My goal for writing this series is to help anyone easily apply Blooms taxonomy in a straight forward way and for readers to feel a sense of growth in their learning.

Image source: Vanderbilt University Center for Teaching (Click here for link to image) / (Click here for link to image license)