Forem: Lara

Explaining Data Engineering to a 5 Year-Old (data is mud)

Lara — Sat, 12 Feb 2022 16:44:26 +0000

You are at the playground, and it starts raining. What happens to the soil? It gets muddy - and then you can try to play with it and make shapes out of it. Maybe get a bit creative and colors it or add engravement?

What does that have to do with data engineering? Well, data in this story is mud, and trust me data can get muddy.

A data engineer's job is to collect the mud, see if it can be used to create clay. Remove any leaves or rocks from it. Then figure out where to put it in a place so that it stays in a good condition. Finally, they do their best to ensure that the mud is as clean as possible, and is stored in a safe and comfortable environment.

In data terms, data engineer's responsibilities are but are not limited to:

- Data Collection (collecting mud)
They are responsible for and sometimes advise on the best tools used for data collection. Support in the data collection process. Ensuring that the collected data is as clean and error-free as possible.

- Data Storage (storing the mud for easy access)
They would be responsible for storing the data. It is not an easy task, they need to ensure that data is stored in a way that is most suitable for the data type. For example, videos and images are stored differently than text. Also, they need to ensure that data retrieval is as optimal as possible by data analysts and data visualist. They also need to ensure that data is stored securely and protected from any breach attempt.

- Data Clean up (removing leaves and rocks from the mud)
They are sometimes responsible for cleaning data entry mistakes, outliers, discrepancies in the data to some extent.

- Data Analysis (figuring out the mud type and what we can make of it)
Though this is the job of a data analyst. Data engineers do not always get the joy of digging deep into the data and understanding its hidden messages. But they do take on primal data analysis.

On a closing note, data engineers may not be at the frontline, but they pave the road for data visualization and analysis. They are the foundation that data science is built on. They are our data avengers.

How would you describe a data engineer?

How to combine and summarize excel/CSV files in Python?

Lara — Sat, 12 Feb 2022 16:03:00 +0000

This is the begging of a series to share simple and short lines of code in Python that helped me to automate my work.

Scenario: You have monthly sales reports for the year 2021, and your need to put them all in one file to create a yearly report and generate simple statistics. How can we do that?

Prerequisites:

All files must have the same set of columns (same name, spelling, and number)
Have all files in one folder
All files must have the same extension (either .xlsx, .cvs)
Any version of Python - mine is 7.29.0.
You will need the below libraries:
- glob
- pandas
- os

Set up

First, install the above libraries if you do not have them installed already. You can use the below commands and tweak them depending on the IDE you use, I use Spyder.

Spyder:
pip install pandas pip install os pip install glob

Importing the files

1. Set up the path
path = r'C:/Folder Name'
2. Get the list of files
This will get for us the names of all the excel/csv files in the particular folder we specified in the path.
file_list = glob.glob(path + "/*.xlsx")
Just change .xlsx to .csv throughout the code and it will work the same.

3. Importing the files
This is a for loop that will go through each file we got in the file_list, read it as a data frame, and put it inside a list of data frames by appending it to the list.

excel_list = []

for file in file_list: excelFile =pd.read_excel(file) excelFile['fileName']=os.path.basename(file) excel_list.append(excelFile)

Merging the files into one

1. Concatenating the files into one data frame
excel_merged = pd.concat(excel_list, ignore_index=True)

2. Export the file to the same path
excel_merged.to_excel(path+'/All Lists.xlsx', index=False)

Create summary and basic statists

Get all columns in the data

excel_merged.columns.tolist()

Generate a general summary of the data

excel_merged.describe()

This will give stats of all numeric columns (mean, medium, STD, count, max, min, etc.)

You can use the below to get stats on a certain numeric column

excel_merged['column name'].describe()

Get stats group by certain month
summary=excel_merged.groupby(["Month"])[["Sales Value"]].describe() summary

This will show us statistics on sales per month - it is a good indicator to see if we have peaks or low sales in certain months.

Plot the data

f, ax = plt.subplots()
ax.bar(x=excel_merged['Month'], height=excel_merged['Sales'], color="purple")

ax.set(title="Plot of Sales per Month") plt.show()

Troubleshooting

In case a file was missing ensure that the file is in the file_list, and it has the same extension and is in the folder.
If you have more columns than the original lists -this means there is a file with either extra columns or does not have the same spelling.
Throughout the codes, just change the path from .xlsx to .csv and it will work the same.

Well done!

5 Tips for Working From Home - Developers Edition

Lara — Sat, 12 Feb 2022 15:18:22 +0000

2020 was the year everyone in the world got to experience what feels like the norm to the rest of us - developers!
We all at some point worked from home before the year that shall not be named. Either as part of a remote contract, taking our work home - as usual, or working on a side project. Thus what felt weird to the rest of the world was another Monday to us. For those of you just getting on board with the Work From Home (WFH) thingy, this article is for you.

1. Change your PJs to a work PJs

It is really hard to switch your mind from work mode to relax mode. You might set on the computer for 45 minutes while your brain is still in hibernate mode. What helps is to find a habit to make your mind do the switch - a tic. What I found helpful was to change my PJs to a work PJs that I got ONLY for work. I wake up, make coffee, put on my serious stripe pajamas, and set on my desk. Then BOOM I'm in work mode. Once I stop working I change back to my regular PJs, and my brain stops thinking about work. It is really hard to draw the line between personal life and work-life, especially when working from home. Therefore, you need to find your switch.

2. Have a dedicated space for work ONLY

Not all of us might have the option to have a dedicated space for work, but this is where we need to get creative. If you work in the same area where you eat and chill, it is not easy for your brain to work in that same space. As it does not associate it with being productive. That is why you need a dedicated area for work. Maybe one of your dining table chairs can be dedicated to work, or putting on a cloth on the table when you start working. This will train your brain to associate work with that space/change.

3. Have a two-monitor setup

Trust me this will enhance your productivity - A LOT. As you do not have to constantly switch between windows, you will have for example one for the code and one for Stack Overflow. Even if you are not a fan of dual monitors, at least get one. Most laptops monitors are bad for you, they will break your back and strain your eyes. Development requires a constant focus on the screen for long hours. You need to ensure that what you are looking at is comfortable for you.

4. It might be scary but TALK TO YOUR TEAM

Some of us might be socially anxious or people are not their thing. But it is important to stay in touch with your team. This will give you a sense of belonging, and commitment. Try to use the first 30 minuets of the day (or the last) to go through and organizing your to-do list whilst also checking up on your team members/team lead. Working from home can feel isolating, hence try to reach out to your team as much as you can. This will also give a sense to the team that you are present. Then you or them will not hesitate to reach out in case someone needs help. People are not always charming but talking to someone about an issue that you are stuck with for hours will help your brain to come up with a solution as you are explaining the issue. Check out the rubber duck debugging.

5. Keep track of your mental health

Again, we are at the end of the day social beings. Being away from people - as good as it seems - might fire back. Ensure that you have a clear end of your working day. Have a routine in place and stick to it, structure helps!.
Ensure that you do at least 3 of the following on daily-basis they will help you and increase your productivity:

Take a lunch break away from your desk.
Go for a walk outside, in nature - graphics are amazing trust me.
Dedicate 15 minutes for watching something FUN (short video, TikTok, scrolling through social media, etc.)
Stretch
Clean your desk
Do digital decluttering

Let me know what your tips for WFH, what worked and did not work for you?

Disclaimer: This is based on my personal experience and lessons learned over the years, which may or may not apply to you.

2021 Online-Learning List

Lara — Sat, 06 Feb 2021 16:19:12 +0000

Intro

The new year comes with the excitement to learn new things, specially in a dynamic field like software development. My new years resolutions for 2021 had some few items for programming. I wanted to work on:

Mastering Python
Building GUIs Using Python
Data Science As we all know, for the new years resolutions to stick beyond the first week of January, we need to make them SMART!

I wanted to make programming part of my daily routine. Specially with quarantines and lockdowns, sitting on the couch finishing marathons of Netflix series, refreshing social media and keeping up with tik-tok trends made me go crazy.

Thus I have decided to register for 3 online courses on the topics above. Scheduling the time for learning by giving 2 hours a day per working-day and 5 hours during the weekend for the period from Jan till April 2021. Below are the courses I signed up for, with a brief on each, with my personal opinions on them, and some links if someone wants to get on board or just check them out.

The courses are sorted from the beginner level to professional level.

I. 100 days of Code

2021 did not seem like a bad year to finally get on board with #100DaysOfCode boat. I wanted to dedicate 1 hour a day for coding outside my job, dedicated to earning new things. That's why I signed up for the 100 Days of Code - The Complete Python Pro Bootcamp for 2021 on Udemy.

It it divided to four levels: Beginner, Intermediate, Advanced and Professional. The first two levels are super slow based for someone with a software development background but the rest are pretty interesting. It takes only one to one and a half hour every day, I usually watch the material while eating dinner, and have the problems at the end for dessert! This gives the feeling of fulfillment after crossing one day of my list of 100 days!

Learning topics:

Building a small project at the end of every day, thus linking the learning material to real life applications
A good dive into useful libraries like: NumPy, Pandas, Matplotlib etc..
Building GUIs with Python

Course Link

Cost: 20$ | Length: 60 hours on demand videos

II. Python for Machine Learning & Data Science

It is another course on Udemy for Python, combined with Machine Learning and Data science concepts. It's called Python for Machine Learning & Data Science Masterclass.

Below are the topics that are covered in the course:

It has an hour crash course on python - perfect for a quick refresher and review -
A quick guide through the major Python Libraries NumPy, Pandas Matplotlib, and Seaborn Dara Visualization
A dive into statistics: Linear regression, Feature Engineering and Logistics Regression
Combining statistical concepts with code

I personally found the course clear, concise and super efficient. I am enjoying my time with it, and it has fun exercise at the end of every module, which are a nice recap for the covered material.

Course Link

Cost: 20$ | Length: 32 hours on demand videos

III. Data Science and Big Data Analytics

This course is part of MIT online learning professional courses, similar to edx material. It is called MIT xPRO: Data Science and Big Data Analytics: Making Data-Driven Decisions. As the first two courses were a bit for a beginner level, I wanted to pursue a more professional learning path. I was a bit hesitant for two reasons, first the cost and second the commitment that this course will require from me. It is a two hours session per week, each require 4+ hours of research, self learning and working on assignments. But since it was for 7 weeks, I could manage.

I found the course very demanding, and require lots of time on self learning, specially if your background in not statistics, as it will take you extra time to google and learn the new terminologies. The learning style is more like college, with lectures and grading assignments. So far, I found the course a bit difficult to follow, and very demanding. I had to pause quiet often to search on my own, and what's given in the lecture requires lots of side reading. As the lectures only provide general headlines. Unlike the other two Udemy courses, where you follow a step by step coding exercises and then apply on your own some example. Nevertheless, it has been pushing my outside of my comfort learning zone, and I am excited for the outcomes and the added value from this course.

Learning topics:

Regression and prediction
Classification, Hypothesis Testing and Deep Learning
Predictive Modeling

Course Link

Cost: 807$ | Length: 7 weeks

Conclusion

I am still going through these three courses, and I will be till April 2021. Perhaps by then I can write a second post on my learning experience and lessons learned from these courses, time management, and money for value. You might think that these courses are redundant, but I tend to learn concepts better when I get them from various resources. That's just how I am wired. Additionally, since the courses are of various levels, they somehow support each other and feed into one another, to make my learning path easier.

Questions

Do you have any tips for me? Have you tried any of these courses or other online courses that you'd like to share or recommend? What are your thoughts on online courses for learning how to code? What is your style of learning? Books, courses, hands on projects ?

Remark

You might think that online courses alone are not enough and you need to write code on your own. Well you are absolutely right, what I have been doing is that I have a dataset from my work, I've been experimenting with the various methods I'm learning on my dataset, and seeing some pretty interesting results.. which is super exciting, and these methods help see the progress I am making at my work.

Nevertheless, Lara Coded

Lara — Mon, 05 Mar 2018 20:45:16 +0000

In my country unlike the rest of the world almost 80% of the students in Computer Engineering major are females.
The top 5 in my batch were girls.
when we get into the work force the percentage drops down to 50%.
When I look at the senior developers and team leaders they are all males!

Something is not right going on in here!

For some reasons girls are at the top in the classroom, but falling behind in the real world!

Well guess what, time's up!

My advice for allies to support women who code is....

we are not your enemies but your allies as well. Gives us a chance to prove what we are capable of. Don't judge us based on our gender but based on our work. Ask yourselves all the time, if gender is not real, would I do this and that? would I hire this person? would I make this person feels more welcome? would I have said that? etc..

Finally, just be kind to everyone, we all have our own battles to fight, let us just be kind and tender to make this world a better place.

I.T. Certifications

Lara — Mon, 05 Mar 2018 19:56:25 +0000

I am interested in advancing in .NET MVC, I started looking for certificates that I can take to empower my knowledge and skills, I came across several including Microsoft Certifications and I wanted to discuss with you all the following:

What are your believes on I.T. certificates?
Do you think that taking certificate would help in a career advancement?
Would they really open up new opportunities?
Are they worth the time, effort and money?
What kind of certificates would you recommend?

My journey in code

Lara — Thu, 01 Mar 2018 19:52:07 +0000

This is my journey throughout the years in learning how to code.

1. High School and code

My journey with code started when I was in high-school, we were introduced to FrontPage at first and I was like wow me make websites, me is genius. Afterwards, QuickBASIC came with it's blue screen I thought that I was officially a hacker! and then we moved into Visual Basic where I wrote my first if statement, my first for loop and made my first calculator -tears of joy-, the idea of me talking to a machine and telling it what to do was mind blowing for me. Yet, it took me a while to grasp and understand all these programming concepts.

2. College, code and tears

Then I got into college, and got into depth in programming. I basically studied C++ and Java as programming languages, then I was introduced to OOP concepts, Algorithms, Data Structure and Database. It was pretty tough, all I did during college was crying because I did not get good grades, every semester my GPA would fall, my tears and my self esteem.

3. Work and code

Following that, I did an internship in iOS development using Objective-C.
And for the past 2 years, I've been working as a .NET developer using MVC C#.

I know that am still at the beginning of my career, yet things are not easy for me and I'm doing my best to keep getting better and improving. I tried taking some online courses, and reading few books about coding, but I'm still not getting there.

The Struggle is REAL

Do you know when you like get something? There is a spark in your head that lights up and then you got it, you mastered that skill I am still not there yet but I really wanna get there.

So, do you have any tips for me? Also, I am curious to know how did you guys learned programming (self taught, school, boot camps, online courses etc..) and what was the thing that got you into programming?