Forem: vejendla

Day 1: Learning full-stack development

vejendla — Tue, 18 Apr 2023 02:13:16 +0000

Hello! One of the technical skills I want to learn in 2023 is full-stack development. More precisely, I want to continue learning full-stack development.

I started my career as a junior engineer at Floyd, a d2c e-commerce brand based in Detroit. When I was in college, I did not expect that I was going to go down this path. I only did a minor in Computer Science in college and most of my friends were in the business school. Because of that, I assumed I was going to be a consultant of some kind, or a similar analytical or business role.

Because of my background, and the fact that I had very few friends who were in software engineering, I have always felt like an imposter in technical roles. I found myself feeling like I didn't know enough to consider myself an engineer, and that I wasn't good enough to continue down the career path. After my role at Floyd ended, I didn't think I was cut out for another software engineering role and that I should look towards roles that are adjacent to software engineering or tech, but tend to flex softer skills (I'm still open to that career path as well, especially in the long term).

I want to prove to myself that I can learn a framework from scratch, and I want to blog about my experience to help other people who don't come from a traditional pure Computer Science background. Hopefully, you can relate to some of the feelings I'm going through, and it helps you work through similar challenges.

Where I'm starting from

I am not starting this challenge as a beginner, because I have worked as a software engineer (for close to two years) using a modern front-end framework. Specifically, I worked with Solidus, which was built on Ruby on Rails, and a Vue.js heavy front-end. Our app also worked with a content management system, called Contentful (the organization has since switched to using a different set of tools).

Also, I have been hobby programming since I was 11 or 12. I got my start with HTML/CSS, and then graduated to messing around with Wordpress. I completed an undergraduate minor in Computer Science at the University of Michigan, which shares core classes with the major, and is a well-reputed computer science program more generally. Those have given me a strong base of knowledge to approach this course.

Day 1

Let's begin! To teach myself Full Stack I'll be using this website, which is a free resource. They recommend doing one part at a time. I'm starting with Part 1 - Introduction to React.

The first challenge with any new framework is figuring out how to get the d**** thing running. This is by far the most frustrating part of development.

Specifically, I'm trying to run the following command

npx create-react-app part1

npx is a command that you can use to install npm package binaries (link I Googled). npm is a package manager for Javascript. I always thought it stood for Node Package Manager but technically that's not true (though that was the name of the original commit).

I'm not sure how to run this command - I think I need to install npm, and use that to install npx. I already have nvm installed. Based on this section of create-react-app docs, I think I can use nvm to use npx/npm. Let's try following these steps to see if that works. I ran the following command to install node:

nvm install node

Now let's see if I can use npx. I followed the commands here, and it worked! Excellent.

I'm not going to replicate every step in the tutorial, but will outline any cool things I notice, or any problems I run into as time goes on.

Here are some things

React dynamically updates the content - I didn't have to refresh my react app after saving code, unlike Vue.js

Reflections

I spent probably one to two hours on this. I'm starting a part-time job soon, and that took up the remainder of my day. I think to be successful with this, I should focus on doing it at the beginning of the day, and setting a goal of 2 hours of learning.

Geocoding addresses

vejendla — Thu, 06 Apr 2023 04:12:30 +0000

I want to teach myself Leaflet.js, so I’m working on a project to rebuild this map as a Leaflet.js app. If you’re curious why I picked a website cataloguing old synagogues in Detroit as a project, read the readme on my GitHub to learn more.

So far, I’ve scraped the names and addresses of each former synagogue from the website, and have a .csv of them all that looks like this:

Some of the locations are listed as addresses (2625 Tyler), while some are listed as cross streets (Holbrook at Brush). A few are listed as general locations like “Curtis just west of Livernois”.

The next step in my project is to convert these addresses to coordinates.

Which service?

To do so, I need to geocode these locations. Geo-coding is the process of transforming a description of a location (like 2625 Tyler or Holbrook at Brush) to a set of coordinates referring to a specific place on the Earth.

There are several geo-coding services out there. While researching geocoding services, I found this StackExchange thread which lists a few options for geocoding. There’s also this article for Python which lists Google Maps. Here are some of the options I found:

First, I thought of using Nominatim (an Open Street Maps API), since it’s free. However, Open Street Maps (not the API) doesn’t recognize “Holbrook at Brush, Detroit, MI” as an address. I’m assuming that the API wouldn’t recognize it either, which is a problem.

Next, I thought about using Google Maps, which does recognize the address. However, their API requires me to get a key, which is a lot of work and may require me to set up a billing account. There’s one more place I can look…

Census to the rescue

The Census Bureau offers a free tool to geocode a .csv of addresses or locations in bulk. It works best for fully formed addresses (i.e., 400 Main St, Anywhere, NY). I can try uploading my .csv, and then filling in the missing addresses manually. Let’s give that a try:

It worked! Only 14 of the addresses didn’t have matches, which should be easy to fill in. Next, I copied the coordinates column back into my original spreadsheet. I ran into a roadblock here because it turns out that Numbers (on Mac) doesn’t actually have a SPLIT function. That seems like a tomorrow problem (it’s 9pm as of writing this), so I’ll move forward on filling in the missing values.

Using Google Maps, I found the first missing value, and right-clicked on the map to get the coordinates. It turns out that the Census and Google Maps return their coordinates in a different order. This also sounds like a tomorrow problem, so I’ll just copy them as is. 10 minutes later – I have a spreadsheet full of coordinates.

Next steps

Next, I need to:

reverse the order of the coordinates I copied from Google Maps
split the coordinates into their own columns
refer to this GitHub repo to read the .csv into my Leaflet.js map
figure out how to add both the name and the address of the each lost synagogue as part of the title card

Plotting ZIP Codes using Kepler.gl

vejendla — Mon, 03 Apr 2023 18:44:56 +0000

Do you have a set of data that contains ZIP Codes, and want to display that data using a free, open-source tool?

Then read on to learn how I did exactly that to help a non-profit in my hometown understand their visitor data over a three year period.

This article contains three sections. First, I'll generate a list of vists by ZIP Codes using Pivot Tables.

Second, I'll describe how I used a free dataset and Python to convert the ZIP Codes into geographic coordinates.

Finally, I'll upload the data to Kepler.gl.

Here's what the final data looked like:

1. Creating a pivot table of Zip Codes

First, I previewed my data using Google Sheets:

Then, I used pivot tables to display the count of visitors by ZIP Code. See the result (left) and settings (right) below. You can access this view by highlighting your data range and clicking "Insert > Pivot Tables".

2. Converting the Zip Codes to latitudes and longitudes

Since Kepler.gl requires coordinates to plot data, I had to find a way to convert ZIP Codes into latitudes and longitudes (As an aside, ZIP Codes aren't always the best datatype for business purposes. See this post for more.)

There are a number of ways to convert ZIP Codes to coordinates. One free way is to download this dataset from Kaggle, and use a VLOOKUP style operation in Pandas. That's exactly what I did. Below is the code that I used. You will need to have pandas and geopy installed prior to using this code.

import pandas as pd
import geopy
import math

# replace this with your input csv
df = pd.read_csv('20-21/20-21pv.csv')

# this is the name of the column in your input CSV that contains the ZIP Codes
df['Zip Code'] = df['Zip Code'].astype('int')

# replace this with the URL to the Kaggle csv
zipdf = pd.read_csv('zip_lat_long.csv')

def geocode_lat(row):
    if math.isnan(row['Zip Code']):
        return None

    searchZip = int(row['Zip Code'])
    tempRow = zipdf.loc[zipdf.ZIP == searchZip, "LAT"]
    if tempRow.empty:
        return None
    else:
        return tempRow.values[0]

def geocode_long(row):
    if math.isnan(row['Zip Code']):
        return None

    searchZip = int(row['Zip Code'])
    tempRow = zipdf.loc[zipdf.ZIP == searchZip, "LNG"]
    if tempRow.empty:
        return None
    else:
        return tempRow.values[0]

df['lat'] = df.apply(lambda row: geocode_lat(row), axis=1)
df['long'] = df.apply(lambda row: geocode_long(row), axis=1)

# replace this with the name of your output file
df.to_csv('20-21pv_with_latlong.csv')

and here's the result (see the coordinate columns):

Note 1: I did some mild data cleaning between step 1 and 2 that involved removing invalid ZIP Codes from the dataset. This included foreign ZIP Codes, and improperly formatted US ZIP Codes.

Note 2: The free dataset I used only contains ZIP Codes up to 2010. The US Postal Service has added a small number since then that weren't included in my final visualization.

3. Plotting visits in Kepler.gl

The final step is the easiest. With our .csv, I simply uploaded it into Kepler.gl.

The user interface should be self-explanatory, but there's a helpful guide if you get stuck.