Forem: Stefanie Davis

Data Explained: ETL

Stefanie Davis — Tue, 04 Apr 2023 23:00:59 +0000

Extract.

Transform.

Load.

This can be a complex concept. Here’s an easy way to understand it:

McDonalds. Yes. McDonalds.

You order from the drive-through and receive your order (Extract)

Add ranch. People like ranch.

Add Barbeque Sauce to your Fries.

And Take off the Onions. Eww. (Transform)

Eat. (Load)

Garbage In. Garbage Out.

Edgar Codd and The Modern Data Stack

Stefanie Davis — Wed, 31 Mar 2021 04:29:51 +0000

The modern data stack has the obvious definition of “it depends”. Some of the things that determine the data stack include:

Price
Size (Terabytes, Petabytes, etc.)
Company Relationships
Scale

This is not an exhaustive list. But, it’s what I’ve seen determine the technologies used at different companies from startups to large corporations. It’s almost a given to say these technologies are based in the cloud. This adds to the scalability and programmatic nature of the code used for data. And as there are now three major cloud providers (AWS,GCP, and Azure) engineers adapt to the services provided by these companies. This shift allowed business to scale faster and do more with data. The common phrase is “Storage is cheap and Compute is expensive”. These cloud providers allowed for cheaper compute giving many companies the focus on solving problems with data, at any size. So, who’s going to maintain this influx of data? Some analysts or traditional DBAs skilled up and the programmatic nature called for other engineers to become data engineers.

Google Trends for Big Data

Google Trends for Data Engineer

These are search terms. However, you can see the trend (no pun intended). As big data emerged, the need for someone to maintain the data grew.

What about Codd??

Edgar Codd is known for creating the relational database model and the process of normalization. His name is usually brought up in relation to normal forms (1NF,2NF,3NF,etc). These foundational elements aren’t commonly discussed when evaluating data. But, relational databases are everywhere and you can have your pick. They range from lightweight like sqlite to Oracle or Teradata. These principles exist in Redshift and any other relational database. Codd even coined the term OLAP (Online Analytical Processing). On Top of this foundation exists Business Intelligence, Data Visualization, Data Science, Artificial Intelligence, Machine Learning and more.

The Point: Data Engineers should know SQL and Python, as tools to do the job. However, there’s domain specific foundational knowledge that exists in data. And good data modeling can accomplish more (save money, scale data science, democratize data, ensure data governance, etc.) than writing more code can sometimes. I’ve seen it happen. I’ve made it happen.

Coder’s Block

Stefanie Davis — Mon, 11 May 2020 03:39:20 +0000

Coding is an art. I don’t mean just the design. The way you design your schemas, create APIs, and write the code for the front-end. There’s an imprint of the creator. If you spend enough time with someone and see their code you can get an understanding of how they write code and see their personality expressed in the code they create.

Currently, I’m having Coder’s Block. Going through Flatiron School provided challenges and time restraints. I was still in control of what kind of projects I created. However, that pressure and what felt like impossible timelines, didn’t leave space for anything except creating and building.

There’s no shortage of app ideas, tutorials, etc. None of those are the problem, ideas are plenty. If you give me enough time, I could fill that time discussing my own. However, what’s missing is the drive to code again. Holding the weight of personal and world issues is a heavy burden. This burden is usually eased by coding. I can code (with music) and for that time not be concerned with the outside world, just the code. Building something.

So, where do I turn when I have Coder’s Block? Movies? Music? I tried, it just didn’t cut through. It’s one feeling to consume art and another level to create art. The science backs it up. This is your brain creating art:

Tl;dr Your brain likes art. And so does mine. I miss creating. I miss learning. I believe I’ll get back to it.

Until then, music will stay on rotation and I’ll continue to take in inspiration. And give myself space to feel and focus on gratitude. I’m grateful to have more tools to create. And for the little victories every day.

Coding during COVID-19 as a Recent Coding Bootcamp Grad

Stefanie Davis — Sun, 05 Apr 2020 23:45:08 +0000

This pandemic has changed our society mentally and the obvious physical impact.

As a recent coding bootcamp grad (Flatiron School), there is pressure to apply to jobs and build side projects to add to your portfolio. However, a major event occurred towards the end of my bootcamp and it's still ramping up.

I still feel this pressure. However, there's a stronger pressure on all of us. One area I figured I could help was in social isolation. I knew therapists were booked and many people can't afford therapy. However, empathy is free. I created a project to crowdsource empathy by allowing people a vehicle to talk to each other.

The project is built in React. And I used a Calendly and Zoom integration. This allows users to book time on the calendar to speak with someone. Once you receive the meeting notification, you also receive the Zoom link. The other party receives the Zoom link as well and you're able to have a conversation at a convenient time and connect with someone new during these trying times.

Next Steps:

I need volunteers to speak to people and a vetting process.

https://www.ronaconnect.com/

Using iTunes API with React && Redux && Rails

Stefanie Davis — Wed, 18 Mar 2020 09:20:50 +0000

For my final Flatiron Project, I created a Music Journal. You can save songs and add them to your journal with the feeling and other text associated with that song. For example, "I remember this song from High School". Expanding this project may include more music details for Music Producers or Songwriters to have collections that help them electronically save notes about songs, including their own.

While creating this project, I realized Redux is tough and not using Redux is tough. This paradox came when using the iTunes Search API.

This data comes nicely formatted in JSON and you don't need a key to access. So, that eliminated another step.

Here is an example of one song:

In order to get this data I needed to fetch. I didn't use axios, though I may in the future. For my project, I needed songs and I had to give the API a search term. I could hard code the media type as music. And I needed to get the search term from user input. I fetched the data in my handle submit function after a user inputted the term they wanted to search.

The tricky part.

I now have search results. However, I'm not going to hold them in state. But, I need to do 3 things:

Display the search results on the page.
Display more details of the chosen song on a view page.
Add that song to a list of saved songs, if chosen.

To accomplish this, I have to bypass the Redux and rely on React within the application.

I took the search results from the fetch and pass those results to a Song Item component to hold the details of the song.

I display the Song Title and Image through the SongItem component. This addresses: 1. Display the search results on the page.

            <Link
              to={{
                pathname: `/song/${song.trackId}`,
                state: { song }
              }}
            >

I then send the same song information as state to a new path that will address: 2.Display more details of the chosen song on a view page. That path goes to the SongView Component. This is accomplished through the Routes.

<Route exact path="/song/:trackId" component={SongView} />

Moving to #3, the SongView component displays the songs information and a button to add the song to the database and save it to the Redux store.

And once the song is added the user is sent to the '/saved' path to see all of the songs saved to the database, including the new song that has just been created.

So, that's all three accomplished:

Display the search results on the page.
Display more details of the chosen song on a view page.
Add that song to a list of saved songs, if chosen.

Hopefully, this is helpful for someone else looking to integrate these technologies and use an external API while understanding what should be stored with Redux as global state.

Master Plan to Execution: Javascript && Rails

Stefanie Davis — Wed, 19 Feb 2020 07:12:46 +0000

All great projects begin with a great plan.

Before starting this project, I wanted to make sure I had an plan for how I wanted it to look and the functionality I want it to have. (See hand-drawn UI image above)

This was a very exciting challenge. This was our first project adding Javascript. And we weren't allowed to use any frameworks. So, vanilla JS it is.

The Process

After going through this roller coaster and talking to other people in my cohort this seemed to be the general process of creating this project.

1. ok cool

2. OMG (The Wall)

3. Calm a little

4. Get Back to Work

5. ok. I'm just going to turn in something

What was my something?

I created a web application called BookTap. The concept is to have all your books on tap and add comments related to the book. Plenty of times, I come back to a book I've read. But, I don't remember any exact quotes or thoughts I had while reading the book. So, I created something I would use as my web application.

After the idea phase, I had to start building.

I decided to use Postgres instead of the out of the box sqlite3 that comes with creating a new rails project. This was to help with future functionality I wanted to add and hosting.

I kept track of the commands I ran, because they aren't stored anywhere. You just see the product of the commands. So, to create the Rails backend with Postgres I used this command:

rails new booktap-backend --api --database=postgresql

My database structure was simple for this project:

Then, I went on to add the coordinating Models and Controllers. I didn't need Views, because Javascript would take that place.

I also added serializers for Json to determine exactly what information I wanted to pass through. For example, I didn't need to pass through the created_at and updated_at timestamp columns from my database.

Javascript

With my created and configured for my project, I could begin to work on the Javascript. I went through three phases of the javascript section

Get everything to work
Get things to look decent
Get things to look decent and work
Refactor

As you can image #1 took the longest amount of time. I only had one file index.js and I was able to get everything to work in there.

Side-Note: I still think it's magic to have forms appear on a page

On very helpful piece, while trying to make things work is testing and testing in small chunks.

console.log() was a friend when it came to testing my fetch requests to my api.

When the page loads, one part I didn't know before was that I would have to create DOM elements each time the page loads to render the data from my fetch request. A piece of that is below:

Below the image of the book I have an area on the book card for comments and a button to add a new comment.

There are many pieces of code that allow that comment button to work. To simplify it I have:

An event listener for the button
A function to create a form that allows the user to create a comment
A function to create actually create the comment and save it in the database sending a POST request through fetch.
Displaying the new comment on the DOM for the user to see

Below is the function to create the comment form to match the second item above.

There are a lot of moving parts to this project. And both creating and debugging felt surgical.

I hit the wall and got stuck plenty of times during this project, especially on the short deadline.

I learned the only way to get over the wall is by getting help, whether that's someone pushing or lifting hands to pull me over the wall that's how I got over it. I'm thankful to my Flatiron cohort and instructor for helping me to this point.