Forem: Paul Leclercq

Meetups, the right way

Paul Leclercq — Sun, 19 May 2019 15:04:55 +0000

Meetups, the right way

Under this provocative, clickbait title, I hope these feedback can help meetups organizers to better target their audience and make meetups more valuable for everyone.
These feedback come from meetups of different people, cities and countries: Montpellier, Paris, Marseille, New York, Montréal and San Francisco.

Rule #1 : No Q&A (Question & Answer)

I'm a simple person : I hate wars, I hate global warming deniers, I hate poverty, and… I hate Q&A at the end of conferences or meetups. I wonder why this is a absolute norm.
During a meetup, you have the great power to have rare collective time, so please use it wisely! Instead of a Q&A session (often as long as the speaker's talk), add a tiny talk with no slides from someone of the audience.
Example: "I am working on this interesting project at my company because we did this thing this particular way."

Everyone should be able to express his/her opinion

It's hard to express ourselves in front of an audience, and let's be honest, the tech community has more introverts than other communities. Having an open mic Q&A is not fair to everyone.

Alternatives

Propose an interactive quiz with Kahoot, an interactive survey on a slide at the beginning/end to know your audience. Have a MC to animate the audience, group/filter questions, or a person who has already prepared questions for a round table discussion. Another thing, they can highlight someone in the audience they know by doing 3 minute interview on what they are working on.

Katia Aresti

@karesti

I loved the idea of being able to send questions to the speaker through a form during the talk and someone from the organization asking at the end of the talk for us instead of passing microphones. More questions, more efficient, less trolling #bilbostack2019 @BilboStack

12:58 PM - 26 Jan 2019

6 32

Rule #2: No Q&A

Just to be sure 😉
Plus, following these rules, you will be able to filter out assholes, who waste collective time by asking a specific questions about version 6.2.X of an obscure software. They just want to look smart in front of everybody by saying something that nobody can understand and it will make you feel shitty about not knowing this.
We all experienced something like this during meetups, it's time to say no more.

Comfort and location

To attract more people, the venue must be able to host people. That could appear like an obvious one, but no. I'm sure we've all been to venue with not enough (personal) space. An university's lecture classroom is ideal.

Food

We want beers and pizzas 🍕

No. Developers are normal person, they can eat different type of foods, be sure to propose something that does not contain meat for everyone to feel welcomed

Set and say your rules

Have an agenda of the evening.

Say it's totally fine to leave the room at any time, it will not be received as an insult to the speaker. Attendees can also wait for a break to leave the room if they want to show more respect to the speaker.
It's totally OK to not applaud at the end of a talk. If you liked the talk, say it directly to the speaker, or contact her/him on twitter/email later.

By setting some rules, some people blocked by the stress of answering live questions would dare to present a subject, which would be great for the meetup diversity.

Start early, finish early.

Companies should support employees leaving early when a free meetup happens, it's in their own interest.

I do not want to and I cannot be attentive for several hours after 6pm, especially for technical subjects, and especially if I'm hungry.
Make it short, and be sure to… (read below 😛)

Have enough time for networking

What I mean by networking, is not to share a business card and to recruit people by talking about how disrupting your company is. It's to make sure to say hi, being friendly and smile to other people. They share the same passion as you, you share at least one interest together, it's rare in this world, so meet new people!

A great icebreaker is to ask to all attendees to discuss for 4 minutes to the person next to them before the conference begins.
Nothing is more painful than seeing a HR in a 2 hour technical meetup, they clearly want to be somewhere else. Be empathetic, say hi to them, and if you are looking for some fresh air you can help each other exchanging information 😄

Record talks

It's the best advertising you can have to attract more people next time. I recommend this article by Thomas Gasc @meltybro (french)

Organizing meetups is no joke, thanks for all people who does it. I would not be the dev I am today without them : Ubuntu.

What are your top advises for meetups organizers ?

Special thanks to:

https://twitter.com/NDuforet
https://twitter.com/JDarsel
.ltag__user__id__7256 .follow-action-button { background-color: #0030b3 !important; color: #ffffff !important; border-color: #0030b3 !important; }

Aurel Tyson

/aurel_tyson

Aurel_Tyson

Twitter Without Short Links

Paul Leclercq — Tue, 13 Feb 2018 03:44:49 +0000

As dev.to, I created my own Chrome extension (sorry Firefox), I was tired of not seeing the complete URL of a tweet. I felt the right to know where I'm going to land. Also, a lot of information can be found in a URL (URLs can be UI), and my click can depend on them. Example :

What's the complete name of the repo?
I've been using my extension, called fasterLinks (worst name ever?), for 3 years now, and even now I have trouble to use Twitter without it. So I wanted to share it with my favorite online community.! It replaces every short URLs (bitly, etc.) and display long, for twitter, URLs at their full length. fasterLinks!

Marketers love to add several UTMs to track tweets performance, as they are visual pollution, the extension removes them as well.

fasterLinks is free, open source, it does not contains any trackers, except the one used to count active users (28!!) on the Chrome store.

Fork it https://github.com/polomarcus/faster-links and/or try it https://chrome.google.com/webstore/detail/fasterlinks/ojggkiabpbjlckhpaphgdhhojgcpimah

What I Like About Data Engineering

Paul Leclercq — Tue, 23 Jan 2018 03:51:33 +0000

I've recently been asked why I chose to specialize in data engineering and what I liked about it.

As a developer, I'm more used to technical questions/articles/tweets based on facts and long hours of development, and it felt actually good to remember why I spend some much time reading and developing by answering this question.

TL;DR: To build trustworthy analytical dashboards, whereas more data engineers I know seem to enjoy more building systems that scale well

The beginning

I started to work as front-end dev/technical support in a high-paced advertising start-up (from 50 to 500+ people in a few years), dealing with lot of advertising openRTB events such as click, view, complete... And got interested by knowing how the business actually runs and why customers use the platform, simply because I want my work to make sense to me.

I could notice trust was the number 1 reason, beside a wide network of publishers, why customers invest their advertising campaigns. And, dashboards were the key to it. Late, changing, or simply wrong metrics happen and can hurt business badly. And don't think it happens only to others, even Facebook keeps having "bug" with their ad metrics

After having to support several cases like metrics discrepancies with remorseless customers, it led me to understand how metrics were calculated, Hadoop MapReduce, Spark jobs, where those metrics were stored, Cassandra, Postgres, Hadoop Distributed File System, what could we do if something goes wrong, Lambda/Kappa architecture, how to send data between services, Kafka, and what format should data be, Avro, Parquet, to be sure to provide the best metrics possible that benefit business.

Thanks to this willingness to learn, another startup trusted me to become their first data engineer. I'll be forever thankful for everything I learned there. I know I sound cliché :p

My advice to start Data Engineering

Data engineering, is still software development, at the point where I want to call myself more and more software engineer/developer/your favorite words to describe a developer, instead of data engineer.
The only difference is you might spend a bit more time in your database/system config files or documentation, one of my running joke is that I'm not a data engineer but a configuration engineer, so you don't be afraid to start, you can do it.

Learn

You can start today to write your first Spark code using Scala/Java/Python/SQL or R on Databricks, that provides a notebook platform with a free 6G server to analyze your favorite dataset or use one of their introduction notebooks. Or play with BigQuery, Google's data warehouse for analytics, to analyse open datasets like Github.

Learn more

Maxime Beauchemin (AirBnb/Lyft) writes beautiful articles about Data Engineering

I name-dropped a lot of technologies that look fancy and complicate, Martin Kleppman's Designing data-intense applications book contains everything you need to know about the subject. Remember, deep down it's all about simple config files.

Now, I'm also going to ask you a simple question

Why do you do what you do ?

I Suck At Whiteboard Interviews

Paul Leclercq — Fri, 07 Jul 2017 09:18:08 +0000

Also published on Medium
Nowadays my main objective is to find my dream job, why? Simply because I am young, don’t have any kids, don’t have any loans, and I think I have the right positive energy and knowledge to work on everything, so why not focus on these kind of jobs?
I’ve recently done interviews for Spotify in their Stockholm office for a Data Engineer position thanks to 2 blog articles (first, and second) I wrote about their API. I had my first video interview in March 2017 and my last on-site interview on May 31st.

Why I consider this position as a possible dream job:

They’ve worked a lot on their engineering culture
Excellent reads on their tech blog, meaning that you can collaborate with one of the best engineers and learn from them
They used a managed solution on the Google Cloud Platform, which means spending more time on your product and less time managing serversÂ
build the tools to work efficiently on top of GCP and use new tech for defining batch and streaming data processing jobs

This kind of job makes you grow as a engineer.

Preparation

I’ve read ton of articles, read Martin Kleppman’s “designing data-intense applications”, Holden Karau’s “High Performance Spark”, tried a lot of new tech, presented my first talk at 2 Meetups, thanks to the interview pressure I felt.

// Detect dark theme var iframe = document.getElementById('tweet-483688700447326209-881'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=483688700447326209&theme=dark" }

*The more you know the less you know, Trump version*

I also had a few technical interviews with others companies to practice my presentation skills, well… that’s the game and also you might get lucky enough to find excellent opportunities, where I was sometimes weirdly questioned about stuffs I learned during my computer science studies: how collisions are handled within a hashtable, or exercises about Linked List. I spent a lot of time to prepare for it, it made me a better engineer, and it feels good.
I’ve also noted everything I could while doing interviews, to be sure to work on my weak points for the next ones.
Also to boost my confidence, just before my on-site interview, I spent the whole day at the National Library of Stockholm studying for my interview that will take place the next day.

On-site Interviews

I got welcomed by the HR I had on the phone, he explains me kindly what was going to happen and how I should act during interviews. During chit-chat he asks me how I was, I sincerely answered him that I felt stressed and that’s I didn’t have much sleep the night before, but I was really excited and curious about these interviews.

They use easygoing-style interviews to be sure you act normal as possible, or at least the most normal you can, in front of a whiteboard. A lot of smile, soft drinks, bathroom breaks invits…

Programming interview and SystemÂ design

I’ve always have this weird feeling when I got asked something that the interviewer know, actually I have the same feeling when I speak English to a French-speaking person that speaks better English than I do. The advice I have, and that I have trouble to follow, is to imagine you want to explain your solution to one of colleague that doesn’t work in your team, as a data engineer, for example a front-end developer: he has the potential to understand it, but hasn’t spent as much time thinking about it as you have.
So we’ve talked about how we could design a real-time dashboard for the most played songs, and an exercise about removing duplicates from a list while maintaining the order. I’ve had this great feeling that I’ve almost done everything right when going out of these interview, and it gave me confidence for the others.

Lunch

I had a one-hour break, that I thought I was going to spend alone, so I took a book with me, but actually, the Engineers that were my interviewers for the day ate with me, I really enjoyed it, and I felt like a true team member, even for one hour.
Fun anecdote: I once got interviewed by a huge company, where my 2 interviewers, that have apparently already eaten, couldn’t leave me alone while I was having my lunch, so I had the strangest lunch of my entire life, where 2 people that you don’t know try to make small talks while watching you eating.

Culture interview

I would rather had a written culture interview, because some questions cannot be answered without thinking them throughÂ : what is the best advice you gaveÂ ? Hmm.. Well.. Exercice and eat healthy, I guess?. Other questions were what is a team? What is a successful one? And you should answer these questions with concrete examples.
I try to be honest when answering these questions, for example I said that I need a team to backup the solution that I propose otherwise I don’t feel 100% sure. But I guess it can be quite easy to embellish answers. My advice would be try to do so, because that’s simply the interview game.
Another advice, you should have no doubts in which team you want to work with, every answer you give should say, I am the right fit for this particular team. To know this, you need to do your homework by looking for on LinkedIn what people actually do in their team.

Case study interview

With a real-life problem they had, they want to know how you would debug the problem that is hidden somewhere inside monitoring tools. It was a role-play kind of interview, and by the end he had fun with the 2 interviewers. One of them knew that I’ve built the #1 online alarm clock. It feels good when interviewers know your profile, you can say that’s a obvious thing to do when interviewing people, but it’s really rare. You would be surprised by the number of times I had to present my profile during different interviews at the same company, this is one of the reason I currently feel a fatigue of interviews, although I love talking about data engineering. One time on a 5-hour on-site interview, I had to answer to a question that has been asked to me at a previous interview on the same day, even if I told the interviewer that I’ve already been asked this, he suggested me: try to give a different answer. After the on-site interviews, I knew I won’t accept any jobs there.Â
Some advices I would give to interviewers, please be agile and act as an human being, as every interviewee is different.

Results

After 5 hours at the Spotify’s office, being interviewed, I felt weirdly great and full of energy, meaning that I had good times. I summed up my day with the HR I met when I arrived at the beginning of the day, and he proposed me very nicely to visit the whole office: I’ve never seen some many microwaves in my life haha
2 days later, after discovering Stockholm, the city where I was going to move because I did so great, I received this email from the same HR

I am sorry that I have not been able to get in touch earlier with the outcome of the onsite. It took longer than expected. Unfortunately, I have to let you know that we will not proceed with you for this role.

He also offers me to go through each interview by phone so that I can understand their final answer. It was really appreciated. To sum up their decision, I tend to lack of confidence, that I don’t feel sure when I explain my solution. But I guess this is what whiteboard interviews is all about, see how you can handle a new problem without your usual tools, a proper amount of time to check your different solutions and 2 people observing all your movements. I would had loved to be asked a bit more about the solutions I presented to show that I could answer, most, questions. But also, to confort me, the HR told me that all interviewers said that I was friendly, it’s a good point for Gryffondor, I guesss? And he also gave me hope that I could be a good fit in one year.Â

My thoughts on 5-hours on-site interviews

The on-site interviews happened after being selected among a tons of CVs, two 45-minute phone/video interviews with HRs and engineers to make sure you can fit the role they offer, and a 1:30 hour video technical interview with 2 engineers. If we sum the hours spent with the on-site interviews, it’s 8 hours.
If I compare that with my experience of working with new hires, I could say in 30 minutes if the person knows how to code, if I will get along well with him/her, and most importantly, if the person is passionate about what she/he does. How could we need 8 hours to see that?

Although I had a great time being interviewed at Spotify, I have to say that for a second I couldn’t help feeling frustrated and seriously thinking about questioning my professional skills. Especially when you think you are going to get an offer because you were quite happy with all the answers you gave, and sometimes even proud. And also when you went to the extra miles by writing 2 show cases of your skills based on their API.

On the other hand, how would you handle 10 000 applicants for a position? You have to select on details, like Spotify did with me, and I can totally understand that.

In my opinion, on-site interviews should be casual, no code needed, as they’re done during video-interviews. Only case study ones or architecture design to see your point of view on different subjects and what you could bring to the company, with the whole team you are being interviewed for, followed up by a comfort meal all together to be sure that you can fit well among the team.

Try.

To finish this article, and encourage you taking a proper amount of time to apply for your dream/better job,have a look to this quote from Ben of the Practical Dev

But if your goal is to land your dream jobâ€Š–â€Šor just get your first jobâ€Š–â€Šplay the game and take more risks. It is up to the hiring firm to disqualify you, it’s not your job. Somebody is going to be the benefactor of randomness, it may as well be you.

Analyze one year of radio station songs aired with SQL, Spark, Spotify, and Databricks

Paul Leclercq — Mon, 20 Mar 2017 15:50:16 +0000

Note: This post was originally published on Medium

Whenever I drive or code, I listen to music, as this happens a lot, and in order to find new songs, I listen to the radio or I listen to Spotify's discover weekly playlist, which made me like Mondays (because they release it every Monday)

A french old-school institute called MediamÃ©trie analyzes radio stations' songs. Since I have seen their study (that I can't find anymore) some years ago, I have been obsessed with creating my own.

*Mediametrie's image you can find online â†’ old school*

This article will present the year 2016 for 4 main french radio stations through fun SQL queries, then we will connect each song to the Spotify API to create the radio stations' musical profile.

We will use the Databricks community version to visualize our data. All SQL queries and all results are available on this notebook. It's the “backstage” of this article, where the magic happens if we can say.

Protip: don't miss the bonuses at the end of the article

Radio stations introduction

We all have a favorite radio station, mine is Radio Nova for their diversity, their humor, and as a hip hop fan this is the only national radio where we can hear listenable hip hop songs

Radio nova had 1,4% of the audience in September 2016 (PDF to download from Mediametrie).

*Most 5 broadcasted songs on Nova in 2016*

In order to see how a radio becomes number 1, we are also going to analyze the number 1 music radio called NRJ who has 10,8% of the audience and 2 others : Virgin (5%) which, we'll see, sounds like NRJ, and Skyrock (6%), don't mind the name it's a rap radio… haha

*Most 5 broadcasted songs on NRJ in 2016*

The main question is, after we compared these radios, should we give to Radio Nova the tips of how to be the number one based on NRJ's analyze? What do you say, Nova? Learn from the best, right?!

Getting the Radio's songs data

*“What was this title?” Nova page*

In order to extract the songs lists, artist, song title and timestamp, we are going to parse each Radio “What was this song?” HTML pages, except for Skyrock which has a handy RESTful web service.

Every song extracted will be converted into this Song class to query them easily with (Spark) SQL:

    Song(timestamp:Int, humanDate:Long, year:Int, month:Int, day:Int, hour:Int, minute: Int, artist:String, allArtists: String, title:String)

In 2016 300K broadcasts were collected:

Nova : 95K broadcasts of 5000 different songs
NRJ : 50K broadcasts of 800 different songs
Virgin: 60K broacasts of 1200 different songs
Skyrock: 100K broadcasts of 1000 different songs

Every songs is stored in a parquet format to extract only once the data (you're welcome radios servers :p) and to speed up SparkSQL queries. Btw, if you are interested by the file I can export it to you in CSV, or parquet.

Remember that the best way to speed up, the Spark doc says often by more than 10x, queries, if you have to use the same SQL table (or Dataset/Dataframe) again and again, is to cache them in memory (Thanks Databricks for the 6Go RAM server!) with the dataframe.cache() method.

Let's dive into our analysis now !

How many songs by day?

Some days were not recorded by the radios' history system, so the real numbers should be a bit higher.

*Songs broadcasted by day*

Fun to see that both radio stations broadcast more songs during summer (if we do not take in consideration the one-week bug of Radio Nova, in blue, in August), this is certainly due to summer holidays. They do a good job all year long, so, it's OK to take some days off, I guess!

We can see that Skyrock and Nova broadcasts the same number of songs each day, whereas NRJ and Virgin a bit less, certainly due to more talk shows or untracked DJs night shows.

How many different songs by day?

The real difference comes from the number of different songs played, see by yourself the number of different tracks per day:

*Different songs by day*

More mainstream radios such as NRJ, Virgin and Skyrock top 100/120 different songs a day whereas Nova is more about 280. If you want to discover more songs, it's clearly on Nova.

How many different songs by month?

If we have a look to the monthly different songs, the gap between radios is even bigger.

*Different songs by month in 2016*

Top 10 played titles by each radio station

It's interesting to see how “hits” are played through the year.

We can notice summer hits:

Kaytranada for Nova
Enrique Iglesias for NRJ
Kent Jones and Drake for Skyrock
Imany and Kungs for Virgin.

We can tell that the most broadcasted songs are mostly aired during summer. So artists, play smart here and release your songs between February and June to have more chance to become number one, or to have more people hating your music because they heard it too many times?

Nova

NRJ

Skyrock

Virgin

Percentage of music by day

If we take the average broadcasted songs by day and the mean duration of a song, 3.30 minutes, we can guess the percentage of music by day. The other percentage is likely to be talk shows, advertising or untracked songs.

*Percentage of music per day*

To understand more these percentages, we should see what a normal day is for our analyzed radio stations.

What is a typical Monday for our radio stations?

Let's have a look to the average of number of songs for all radio stations for Mondays

*Average number of songs for Mondays*

We can distinguish 2 gaps, during the morning and evening shows for every radio stations. Amazing. More seriously, no discovery here, it's a known fact that most radios have morning and evening shows during which there is less music and more talk.

Advertising time

If we recalculate the average percentage of music at noon, when there is no shows for all radio stations, we can estimate the percentage of advertising by radio by hour. We estimate that the radio hosts speak 5 minutes during the whole hour. We have to note that radios may advertise more during prime time when they have a larger audience.

For 60 minutes, we get from 7 minutes of advertising time, for Skyrock, to 15 minutes, for Virgin. In details, we have this table:

*Average minute of music and advertising for every Monday at noon*

Radios brainwashing?

An annoying feeling we have sometimes with radios is we keep listening to the same songs over and over. As we are men and women who believe in science and not in our instinct we are going to use basic statistics to verify this weird feeling.

How many times is the same song aired on the same day?

*Average number of times the same song isbroadcasted by day*

These pie charts below tells us a lot about radio stations's habits, more mainstream radios such as Virgin, NRJ or Skyrock are more about to broadcasts the same songs multiple times.

*How many times the same song is broadcasted?*

When is the next time we will listen to the same song during the same day?

*Minimum difference in hours between the same songs broadcasted on same day*

Again, the most mainstream radios, NRJ, Skyrock and Virgin tend to broadcast the same song most often 2/3 hours since it was first aired. Nova's value is more about 7/8 hours.

While we have different distribution, the average for our 4 radios is between 7 and 8 hours.

How many new songs* are added and when?

*“New songs” means songs that are not yet broadcasted in 2016.

*New songs by month distribution*

If we look at the average after April 2016, we see that's Nova is ahead, but don't forget Nova plays 2500 different songs each month, so it's normal statistically speaking

*Average new songs by month*

New songs are distributed equally along the week for all radios

*New songs per weekday*

Common songs between radio stations

On the table below, we can see NRJ has 25% of common songs with Virgin and 12% with Skyrock.

Virgin has 18% with NRJ while Skyrock has 9% of common songs with NRJ.

*Number of same songs broadcasted between radios*

Nova has a few similar songs with the others radio, there are mostly legendary artists such as Bob Marley, Daft Punk, Aloe Blacc, Kavinsky, BeyoncÃ©… If you are interested by the full list look for the “Similar songs between radios” cell in the “backstage” AKA the blog article's Databricks notebook

Our 4 radio stations are different, for sure, but do they have common songs between them? Surprisingly the answer is yes

I would classify these songs as songs that everybody likes, you can play them at your party without any stress of being booed

If we use a visualization for our previous table it will look like this, the blue bar is the similar songs, the orange and the green bar are the total of different songs.

*Similar songs*

What are the secrets to be #1?

We have analyzed 4 radio stations based on the artist name, the title name and the day and time the songs were broadcasted. Beside letters and numbers, these 3 values mean nothing, if we want to make a deeper analyze we have to learn more about the songs played: how popular is the song right now? what is the genre of the song? How many followers does the artist have?

Hopefully, by connecting each song to the Spotify API we will get a lot of data we can play with :

https://api.spotify.com/v1/search?q=ARTISTTITLE&type=track&limit=1

In 2016, we have collected 8000 different songs from the radios, so to get the artist, the track and the tracks' audio features from the Spotify API we have to make :

Number of songs x (Artist + track + audiofeatures) = 24K requests

*From [HTTP STATUS DOGS](https://httpstatusdogs.com/)*

That's a lot. Plus, Spotify has a limit of request in time, so we have got to do it slowly, 20 request every 2 seconds, why not you know.

BUT, with this slow rate one thing I didn't plan is we could see the number of followers change when we requested a song's artist, as most artists have multiple songs been broadcasted, the artist information was asked from twice to 10 times. No problemo, right? No…This will mess up our SQL join between artist and track data later just because the DISTINCT on artists information were fake due to followers.total

I have to say this led me to craziness, because I had more songs after my join than before haha

*Spotify API Stats from [https://developer.spotify.com/my-applications](https://developer.spotify.com/my-applications)*

Songs Popularity By Radio

Definition by Spotify

The popularity is calculated by algorithm and is based, in the most part, on the total number of plays the track has had and how recent those plays are.

*Percentage of songs popularity distribution by radio*

No surprises here, mainstream radios NRJ, Virgin or Skyrock, tend to play more popular songs, that's why I use the term mainstream, clever, right?

*Popularity average*

But the real question is : was the song popular before it was broadcasted on the radio?

Audio features

The Spotify API gives audio features extracted from the Song's soundwaves, thanks to these we can display a musical profile of each radio:

In my opinion, the most meaningful audio features are :

danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity.
energy is a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy.
valence describes the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).

Let's see their average and their distribution, as a average alone can be sometimes misleading, among the radios' tracks. As Nova got more different songs than the others we are going use percentage to compare our radios to add more context to our stats.

If you have read my Facebook Interview Journey, you know this is where I failed during my SQL interview, this code is specially for you, dear Mr. Interviewer, no hard feelings though :p

SELECT ROUND( (COUNT(t.*) / subTotal.total_radio * 100),2) AS percentage_of_songs,
      subTotal.total_radio,
      ROUND(popularity / 10) * 10 AS popularity,
      t.radio
FROM AudioFeatureArtistTrackRadios t
JOIN (
    SELECT COUNT(*) AS total_radio, radio
    FROM AudioFeatureArtistTrackRadios
    GROUP BY radio
) AS subTotal
ON subTotal.radio = t.radio
GROUP BY subTotal.total_radio, ROUND(popularity / 10) * 10, t.radio
ORDER BY popularity

*Brilliant SQL code*

Energy

*Energy by radio distribution*

Mainstream radios tend to play more energetic songs, I guess there are more easy to listen to? Some example of song with a lot of energy are We Are Your Friends - JUSTICE, Steppin' stone - Davy Jones, and of course, the classics from the classics Jerk It Out – Caesars, I've first heard it while playing SSX3 on GameCube 8)

Danceability

This chart tells us both radio broadcasts the same kind of danceable songs. Some example of danceable song are Trick Me – Kelis, Around the world – Daft Punk or Anaconda – Nicki Mina

*Danceability average*

Valence / Positiveness

*Valence / Positiveness By Radio distribution*

Same as the danceability, both radio broadcasts the same kind of positive tracks. Some examples : September – Earth Wind & Fire,Ska-Boo-Da-Ba – The Skatalites or Hey Ya! – OutKast

*Valence / Positiveness average by radio*

2 others interesting data, which are not Spotify (Echo Nest) specific, are the BPM (beats per minute) and the songs duration

Tempo / Beats per minute

Duration

*Duration distribution*

Nova seems to be a bit different from the other radios by playing shorter or longer tracks. Virgin, NRJ and Skyrock are really into 3-minute tracks.

When I first saw this graph, I couldn't help myself to think about this Hocus Pocus' song called “Voyage immobile” (motionless journey) and this sentence about our undiversified musical environment :

“Je ne voyais que blocs longs de 3 minutes taillÃ© dans le roc et dans le mÃªme but”

“I could only see 3-minute blocks from the same base with the same goal”

*The duration average in minute by radio*

Music genres

Spotify got some pretty weird music genres, have you noticed “post-teen pop”, “pop christmas”, pop songs you listen during christmas I guess? haha

We can clearly see that NRJ and Virgin, which are very alike, are more about pop/dance/electro music, their top 3 genres are : pop, dance pop and tropical house. Nova is about soul, funk and indie music, and Skyrock is more about rap, dance and pop

*Number of different music genres by radio*

Hip hop genres

*First on rap*

Skyrock is famous for its motto “1st on Rap”, let's compare Hip hop/Rap genres (genres with “rap”, “hip” or “hop” inside the name) with the others radios.

*Number of hip hop, rap songs by radio*

OK, that's a close match between Skyrock and Nova, let's compare the internal hip hop genres now.

I don't really care about genres, but there are a lot of confusion between Hip hop, which is a culture, and rap, which is the actual fact of rapping, if you want to learn more check this Wikipedia Chapter, I also recommend the excellent Netflix's documentary “Evolution of Hip Hop”

*Genres with Hip or Hop or Rap by Radios*

Nova, in orange, is more about indie/alternative/undergroup hip hop music, and Skyrock, in blue, is really more into French rap/trap/hiphop and also popular rap. So let's fix Skyrock's motto by “1st on French rap” haha

Music classifier for Radios' selection idea

In my last article, I explained how to create your own music recommendation system thanks to these audio features.

A fun project (the link is a tribute to the Scala Guru Martin Odersky, he tends to say too many times that his Scala exercises are fun whereas they are brain melt haha) would be to create an algorithm that will help music selectors to find radios style's songs.

Spotify recommendation system

Spotify's system is not only based on the audio features we saw earlier. It also analyzes what others similar users listen to. This slide contains a nice schema that explains their whole system.

What's next?

Thanks to this project I have built solid foundations to query the Spotify API in Scala, process it thanks to Spark SQL, and visualize it thanks to Databricks. I think more projects are about to come, plus Spotify has just released, March 2017, this new endpoint “Recently Played Tracks” and ideas are coming.

Databricks pros and cons

Pros

Free community edition with 6Go RAM server
Awesome and easy-to-use Data Viz

Cons (or more, what can be better)

Can only visualize a maximum of 10 elements when using a GROUP BY, the others elements go to one category called “Others”
Not possible to choose the color of an entity, so a Radio can be blue on a graph and red on another, it can be sometimes confusing
Cannot export graph as iframe, so we have to export pictures from the interactive graphs
Cannot modify SQL on the Data Viz interface

Thanks

Databricks, for their awesome platform
Spotify, for their easy-to-use API and their human-readable documentation
Radio Nova for being a top music selecta, I would not listen to the same music that I listen to today without you
Marc H'LIMI, Radio Nova's advisor, for our exchanges
Pierre Trussart, engineer and DJ, Benjamin Thuillier, scala rockstar, Nicolas Duforet, data science master, Justine Mouron, engineer
My friends for hearing me talking about this project too often

Bonus – Spotify Playlists

To thank for reading, I created 4 playlists of the most ~200 songs broadcasted sorted by the number of broadcast for :

Bonjour, I'm Paul Leclercq

Paul Leclercq — Thu, 16 Mar 2017 23:25:32 +0000

I have been coding for 8 years.

You can find me on Twitter as @polomarcus or Medium as ... @polomarcus

I live in Montpellier, sunny South of France. But I am currenly looking for a data position in North America.

I mostly program in these languages: Scala (with Spark), SQL, or JS

I am currently learning more about Statistics and Data science.

Fun fact:

I've created the #1 online alarm clock in France when I was a student
the "cq" in my family name is silent

Nice to meet you :)

Forem: Paul Leclercq

Meetups, the right way

Meetups, the right way

Rule #1 : No Q&A (Question & Answer)

Everyone should be able to express his/her opinion

Alternatives

Rule #2: No Q&A

Comfort and location

Food

Set and say your rules

Start early, finish early.

Have enough time for networking

Record talks

Aurel Tyson

Twitter Without Short Links

What I Like About Data Engineering

The beginning

My advice to start Data Engineering

Learn

Learn more

Why do you do what you do ?

I Suck At Whiteboard Interviews

Preparation

On-site Interviews

Programming interview and SystemÂ design

Lunch

Culture interview

Case study interview

Results

My thoughts on 5-hours on-site interviews

Try.

Analyze one year of radio station songs aired with SQL, Spark, Spotify, and Databricks

Radio stations introduction

Getting the Radio's songs data

How many songs by day?

How many different songs by day?

How many different songs by month?

Top 10 played titles by each radio station

Nova

NRJ

Skyrock

Virgin

Percentage of music by day

What is a typical Monday for our radio stations?

Advertising time

Radios brainwashing?

How many times is the same song aired on the same day?

When is the next time we will listen to the same song during the same day?

How many new songs* are added and when?

Common songs between radio stations

What are the secrets to be #1?

Songs Popularity By Radio

Definition by Spotify

Audio features

Energy

Danceability

Valence / Positiveness

Tempo / Beats per minute

Duration

Music genres

Hip hop genres

Music classifier for Radios' selection idea

Spotify recommendation system

What's next?

Databricks pros and cons

Pros

Cons (or more, what can be better)

Thanks

Bonus – Spotify Playlists

Bonjour, I'm Paul Leclercq