Forem: Huy Bui

Datathon 2019

Huy Bui — Sat, 09 Nov 2019 18:01:20 +0000

I attended the Datathon 2019 on October 20th and it was so much fun.
This year the event was hosted by Texas A&M University where I just graduated 3 months ago.

For the first time, TAMU Datathon is banding together data enthusiasts from across multiple disciplines, skill levels, and universities. TAMU Datathon is a 24-hour non-stop coding event where companies, researchers, and hundreds of students immerse themselves in data science.

This was probably the largest Data Science Hackathon in the US. There were 2032 Applications, 110 universities worldwide, and 15 company sponsors. Some big companies that sponsored the event are Facebook, ConocoPhillips, CBRE, Goldman Sachs, Shell, Walmart.

There were learning and competition track. The learning track was for beginners who have little or no experience in data science. Attendees on this track could choose to attend a series of lectures or a number of different workshops created by global lead data scientists. On the competition track, participants can show off their programming skills which could help them win prizes or earn an internship opportunity with a company.

I chose the competition track because it sounded so much fun. I joined the event Slack and found 3 friends who were also interesting in competition. Together, we created a team (4 math students) and worked non-stop on the problem.

When I first came, I was overwhelmed by the size of the event. Here are some pictures describe the activity at Recreational Center conference room throughout the day

and night...

Challenge

Every company brought here a challenge and the competitors could freely choose which one they want to do. There were three problems that I found entertaining:

Walmart: use the image processing technique to detect the availability of products on the shelves. This is mainly to deal with Black Friday!

ConocoPhillips: given a time series of about 150 sensors data on a drilling rig, predict when the machine has a malfunction.

Facebook: create a mini google map for the Dodoma region, given the satellite images and residential houses location.

I ended up choosing the Facebook challenge because of its complexity. The data consist of 64 high-resolution pictures described road network of the region approximately 30 km x 30 km in the vicinity of Dodoma, the capital of Tanzania

A sample of 4 images

Putting 64 images together we have the whole map

Besides the pictures, there was a folder containing 64 .csv files with detail about each pixel of the map and their coordinates!. Each pixel approximately represents 0.5 meters square in real space.

Another .csv file consists of population features

The questions were:
1) Find the closest street to each residential location
2) Find the closest patch from one place to another.

The data was two gigabytes and solving the second question required hours of training the neural networks. It was really hard working with big data and deliver a result in just one day. In fact, some of us did not sleep at all.

Presentation

The presentation starts at 2 p.m of the following day. The host set up a big timer on the projector. Every time it hit 4 minutes stop, all judges move to the next table to grade a different project. The best part about the presentation was I could talk to Facebook data scientists and listen to their valuable feedbacks.

The picture of me and my team presenting to a Facebook judge

After a tiring day, we took a group picture together. We were just strangers the day before and now we became much closer. This was because we tried so hard to complete our project. I was excited to come back home and shared my experience with everyone in my data science Bootcamp and later, the dev.to community.

For more pictures of the event, please visit here.

Monty Hall Problem

Huy Bui — Fri, 27 Sep 2019 19:28:50 +0000

Monty Hall Problem is a famous probability puzzle in statistics. It is named after Monty, the host of the television game show "Let's Makes a Deal". The brain teaser loosely replicates the game show concept and it goes like this:

There are 3 doors. You will have to choose a door, and you will win whatever behind it. There is one door with a car. Each remaining door has a goat. First, you are asked to pick one of the doors. Next, Monty, who knows what's behind each of the doors, opens up one of the two doors you didn’t pick and reveal a goat. Finally, you are given the opportunity to either “stay” with your original choice, or “switch” to the remaining door. What is the best strategy to win the car?

I first encountered this problem 4 years ago in a statistics class. It was hard enough that I spent the whole day thinking about it and still insist on 1/2 for both strategies. My explanation was: there are 3 doors, 1 door has a car. If the host opens one door that has the goat, one out of two remaining doors must have the car. So regardless of "switching" or "staying", the probability should be 1/2!

I was wrong. The tricky part of this problem is Monty knows what behind those doors so his decision has an influence on the outcome. Take a minute for yourself to think if this is true.

1) Intuitive answer

Assuming that the three doors are: 1, 2, and 3
Since probability = Event/Sample space we just need to find these variable and plug in the equation.
Sample space here is the total of different ways that we choose the door and where the car actually is. There are 6 possibilities which are illustrated in the following table:

Note that in this table, Monty has already eliminated the door with the goat.
As you can see, the probability of winning for "staying" strategy is 1/3 while "switching" is 2/3.

2) Bayes formula method

This is another approach for people who loves Bayes!
Let P(A|B) denote the probability of event A given that event B already happened

If we know how to translate the problem into events, then everything becomes straight forward. Let:

A be the event that is choosing the door with the car on the first choice.
B be the event that Monty eliminates one door that has a goat. So we are looking for P(A|B), the "staying" strategy.

Let write down things that we know:

P(B|A) = 1, since Monty always choose the right door regardless of any given condition
P(A) =1/3, there is a 1/3 chance of opening the car door, without knowing anything.
To understand how to find P(B), let's look at the diagram:

There are only 2 possible strategies, either stay or switch. Thus, "switching" probability is the complement of staying, which is 1-1/3=2/3.

If Monty does not know which door has the goat. This is equivalent to P(B|A)=1 and P(B|¬A)=1/2. Plug them back to the equation and it returns P(B|A)=1/2. This is the part that I was confused.

3) Simpler answer

With the "staying" strategy, you can only win if you choose the door with a car at first. Since there are 3 doors, the probability of choosing a door with a car is 1/3. So the "switching" strategy is 2/3. This provide us a general observation for more than 3 doors Monty Hall problem.

4) What if?

What if Monty Hall decides to reveal k doors out of n doors where the maximum number of k is n-2, will it affect the strategy at all?

"Switching" in this case means not keeping the original choice

Case A: k=n-2

This case is similar to the original problem. Note that you cannot switch to the door that you chose. So "Staying" win rate is always 1/n and switching win rate, in this case, is (n-1)/n

Case B: k<n-2

The probability of choosing the door with a car at the beginning is still 1/n. Thus "staying" = 1/n
In the "switching" strategy, your final choice cannot be your first choice. Thus, you don't want to choose the first door which has the car at the beginning. The probability of not choosing it is (n-1)/n. The k-th switch is also when your pick, there are n-k door(s) left but you must exclude your first door. Thus, the probability of picking the prized door in the last step is 1/(n-k-1). Together we have:

It shows that the "switching" strategy is always better than "staying" strategy.

5) Simulation

Let run it on python to see if our formula work. Let n be the number of doors and r is the number of door Monty wants to reveal

The graph shows that the statistical result getting close to our actual answer when increasing the number of iteration.

So, you know what you should do if you play a similar game like this next time.

For more information about the code, please visit my Githup repository.

Blockbuster Movie Trends

Huy Bui — Fri, 20 Sep 2019 00:40:55 +0000

Imagine that you are the CEO of a trillion-dollar company like Microsoft. One night, after watching a movie at the AMC with your girlfriend, you thought about entering the Hollywood industry and make a lot of money for your company. Before doing that, you need to hear a little insight about the current movies market. Consequently, you consult with a data scientist and here is what he has:

Data

The objective of this project is to find the trend of popular movies in the U.S. This blog will explain the data through visualization that requires no technical knowledge to understand. However, a GitHub repository is provided for those interested in the technical details of this project in regards to Data Science.

The data come from 220 top gross revenue movies from 2009 to 2019. This is obtained through the API requests from the movies database. Also, a list of the 1000 best actor is collected from imdb.
The program is written in Python and is presented by Tableau.

Question

1) What is the average profit and rating?

The average rating is 6.845, ranging from 4.8 (Independent Day) to 8.3 (Inception).

Avengers: Endgame unsurprisingly made more money than any movie in history with $2.4b in profit. The lowest one come from Dark Phoenix with $45m. The average profit of this top 220 movies is $538m.

2) Which MPAA rating attracts the most audience?

Teen friendly movies, PG-13, are top of the chart for the Motion Picture Association of America (MPAA) rating with over 50% of the top-grossing movie market. The reason is simple, it has the widest range of audience.

3) When is the best time to release a movie?

The best months for going to the box office are :
Summer blockbusters: May, June, July.
Thanksgiving and Christmas time: November, December.

4) Best genres to invest

No 1: Action-Adventure such as Wonder Woman, Captain America, The Hunger Games.
No 2: Action-Fantasy such as Lord of the Rings, Maleficent, Star Wars.
No 3: Animation-Family such as Lion King, Inside Out, Wall-E.

5) Profit expectancy

The linear regression line shows the correlation between your profit and investment with the formula: profit=1.55*budget+3.08*10^8.

For example, let's assume that every step from marketing, production, leadership, to choosing the right studio is nearly perfect, if your company invest in $100m, then you should expect to see $455m in return!

It also worths to mention some of the special cases:
Avatar is the most profitable movie of all time, where profitability is derived from profit/budget ratio. One reason for that is Avatar is the first 3D movie released in the movie theater.

Avengers: Endgames is the next outliner. The sequence is the culmination of 22 movies released over 11 years!

Therefore, new technology or careful sequel planning is the game changer!

Justice League is on the other side of the spectrum. The movie falls at the bottom of this list due to the failure of its leadership. More about it here.

6) The association of genres and profit

Take a look closer to the best three genres. The trend lines show that if the budget is:

Less than $125m, you should invest in Animation.
Between $125 and $182, you should invest in Action.
More than $182m, you should invest in Adventure.

7) How many good actors should we hire?

The casting is a big factor in our calculation. Hiring popular actors can significantly increase the anticipation/attention of the audience. A stunning moment of Gal Gadot could go viral on the internet for a few weeks.

We also want to hire actors that can roller coaster audience's emotion. The list of 1000 best actor capture all of these features. Some of the people on the list are Jack Nicholson, Leonardo DiCaprio, Morgan Freeman.

The graph shows that action, adventure, and family genres have an average of 5 A-rank actors. Interestingly, history movies have the most, 7 actors on average. Though, it has been proved that visual effects can overshadow acting skill.

Summary:

Top 3 genres for a high grossing film are Action, Animation, and Adventure with action being the highest rank. The profitability also depends on the budget range.
Release movies during the summer months and before and during winter break.
Hire at least 5 movie stars.
Rate it PG13.

First Post: Transition from Mathematics to Data Science

Huy Bui — Thu, 05 Sep 2019 14:05:56 +0000

This post is a personal story about how I "quitted" math and found data science. I do not intend to encourage anyone to do the same. There are a lot of factors that influence my choice, so please be careful.

Why not Math?

Three months ago, I was still a Ph.D. student in Mathematics. I was struggling with the decision of dropping out of the program. The process of making that decision was a two-year nightmare. Galois theory, expander graph, chaos theory, discharging method... They are all interesting topics that I spent a lot of time studying them myself. But the more I studied the more I pushed myself away from then reality. I woke up every morning feeling completely lost. I will explain my internal conflict with an example in set theory. The Continuum Hypothesis states that:

There does not exist a set with a size less than the reals and no set strictly greater than the natural numbers. - Georg Cantor

If we can find the 1-1 correspondence map between two sets then they have the same size (cardinality). In another word, natural number, integer, and a rational number have the same cardinality, which is infinity. Is there anything bigger than infinity?

The answer is yes! There are sets such as the set of real numbers, that it is impossible to find the 1-1 map between itself and the set of natural number. You can quickly see why by investigating the countability of each set. Natural number set is countable while the real number is not. The Continuum hypothesis emphasis on what truly in-between natural number and real number?

The study of different type of infinity is very interesting. However, the results are impractical to society. Why are there people (Set Theorist) spend their whole life study about infinity? I can't answer this question myself. I respect mathematician in general because of their unspoken contribution to science or art. Nonetheless, my mentality is not built for processing this 9-to-5. This truth took me a long time to learn.

One important point is that my family lives under the poverty line. They always support my career decision and I am grateful about that. But as the eldest son in the family, I still feel seriously bad for not being able to provide. Graduate-level pure math is a form of art which takes seven years to master. On the contrary, a Ph.D. in pure math is unhireable in most industry and also competitive in academia. Thus, it is not sensible for me to continue.

Let says that family was not an issue. Would you continue?

I started the Ph.D. program in math with full funding. I imagined myself finished the program with my head held high in a standing ovation. However, Ph.D. in math takes more than just the liking. It requires a lot of commitment, sacrifice, and mental capability. I was not ready for it at all. I came to the battle arena with just a wooden sword.

In the very first month of grad school, I suffered from imposter syndrome. My school was one of the tops of the nation in abstract algebra. I freaked out by my classmates because they always "quietly nod their head" whenever the teacher wrote, both literally and metaphorically, Greek on the board. I worked my butt off to get better but the progress is hardly noticed. I was a big fish in a small pond before I came, but here I am nobody in the sea of sharks and whales. By the end of the first years, I got an existential crisis. I was so depressed that I sent out apology letters to some professors in my undergrad, telling them that I was a failure, I did not live up to their expectation. Luckily, I managed to pass all the classes and maintain good standing in the program. However, I felt tired and my head was cloudy all the time. I learned that something was not right. But, I was too afraid to think about it.

What makes me choose data science?

After the trauma, I reevaluated my goal in life. I tried to convince myself about opportunities outside academia. However, I felt ashamed for not be able to finish what I started. One day, I saw a post from a Ph.D. fellow that changed my mind,

The advice soothed the wound. A few days later, I talked to the department chair about my situation. This was four months before my graduation.

I looked for a job that required a Master in a quantitative field and Data Science caught my attention. I know a little bit Python, combinatorics, and graph theory. I just hoped that my time here would not be wasted.

How do I feel now?

I came back home after graduation and live with my parents and three younger siblings. It takes me a while to adjust my daily routines. I have been gone for two years and a lot of things had changed. My brother had grown a lot. He endures the anxiety disorder and gave my parents a headache many times. I just wish that I could be here sooner to be his brother. I also had a lot of mom food which is like heaven compared with what I ate back then.

I started a Data Science program at Flatiron a week ago. I had to pay a lot of money to get in but I feel good about it so far. The program is challenging enough to be fun. There is a lot of logical thinking in programming which is similar to math. Although the curriculum is condensed and laborious, the staffs are very helpful. They just graduated from the program so they understand our feeling and give a lot of input. The students here came from different backgrounds: oil and gas, geology, IT, teacher,... We have one thing in common that we all try hard to advance our career.

Ending notes

I never regret my two years in graduate school. I made a bunch of smart friends and experienced college life. I also have learned a lot about myself through hardship.

My piece of advice for friends out there who are still finding their passion. It's hard to change but it worths the try!