Forem: Tilo

How I aggregated a million datapoints and built a travel website

Tilo — Tue, 03 Mar 2020 19:24:11 +0000

It was December 2019. The gloomy weather and frigid temperatures in Toronto made me crave for a warm getaway. As I navigated through search results, I realized I had dozens of tabs open to answer a pretty simple question: "What are some warm places to travel to in December with cheap flights?"

Being a software engineer, I realized that all I was doing was trying to apply a filter to a broad set of travel results. What if I had a lot of data on places in the world? Can I build some type of filtering to find cool places?

I dove in over the next week to see how much data I could find on the internet to build a travel website that would help me find interesting destinations.

60 days later, Visabug was born and soft-launched on Reddit where it went to #1 on the sideproject and reactjs subreddits. 🎉

My goals for Visabug were:

Build something that is genuinely useful: I didn't want to just build something because it was technically interesting.
Make data freely available to help people make better decisions: Travel opens our eyes to other cultures and makes us more tolerant. I didn't want to hide data behind paywalls.

Getting Country Information

The first piece of data that I was interested in was country information. I wanted to answer the question, "What countries can I travel to easily, and how much would it cost on average to fly there?"

To start, I got a list of all the countries in the world. Then, I used data from the World Bank, along with Google's Geocoding API to get common data such as location, population, capitals, languages, and more.

This was my starting point. I created a JSON file for each country, so I had 238 JSON files titled as canada.json, india.json, etc.

With a little bit of extra work, I was able to also find data on:

Related countries and nearby countries
Regions and Continents
Population
Weather patterns (Temperature and Rainfall)

Getting Visa and Travel Information

Next, I wanted to get tourist visa data. If I were a citizen of a country and wanted to travel to another country, what kind of visa would I need?
I tried looking around for APIs but there weren't any that were easily available.

I turned towards Google Search. It turns out when you search this on Google, you get a nice card with some useful information:

I wrote a script that created an array of Google search queries with each country permutation. It looked something like this:

const visaSearchSet = [ 
  "usa visa requirement for canada citizens", 
  "brazil visa requirement for algeria citizens" 
  ...
]

Then, I wrote a Puppeteer script that queried Google and scraped the result set from the card. This was piped into another JavaScript function that parsed the string into a data structure. The result was something like this:

"united-kingdom": { 
  "nepal": { 
    visaCategory: "required", 
    validity: "three to six months from date of issue", 
    embassy: "12A, Kensington Palace Gardens, London, 48 4KU" 
  }, 
  ...
}

This was my initial prototype. Since then, I've improved the algorithm to double check the visa requirements against some other sites, so a single wrong result doesn't give me incorrect data.

After fetching visa information, I wanted to also fetch travel advisories. A country may be easy to get to, but you may not want to go there because its dangerous!

Luckily, there's a site called SmartTraveller that makes it really easy to get travel advisories, customs information, immunizations and more.

Currently, Visabug is able to track:

Visa requirements between any two countries in the world
Classifies visas as "required", "not required", "e-visas", and "visas-being-refused"
Support for Schengen area visas
Embassy locations
Travel Advisories
Customs Information (coming soon)
Immunizations (coming soon)
Multi-country visas (coming soon)

Getting City Data

Next, I wanted to get city data. To do this, I had to first figure out the most popular cities in the world. I couldn't just use population because many popular cities are relatively small. I used this free dataset for my initial set of cities. As a bonus, that dataset let me map cities to their parent country.

Next, I wanted to collect some useful metrics about these cities.

What is the city known for? To solve this, I used Tripadvisor to get the most popular things to do, and classified them.
What is the cost of living? The Cost of Living index from Numbeo helped provide relative costs per city.
Is Uber available? Uber's website has a list of all the cities in which they operate.
How safe is it? Numbeo also has a safe cities index!

I collect a lot more data than the list above, but that should give you an idea of how it works. By piecing together data from different providers, I was able to understand the unique characteristics of all cities.

Currently, I also collect:

Average Flight Prices between two countries
Cost of Meals
Internet Speeds
Popular SIM Providers
Whether water is safe to drink
Air Quality (coming soon 🤫)
Popular Tourist Attractions (coming soon 🤫)

Creating Filters

In Visabug, you are able to use filters to find unique destinations. Here's a screenshot of the filter box.

Apart from the Visa Requirement filters, the filtering actually works on the city-level, not at a country-level. So when you apply a filter like "Sand and Beaches", Visabug finds all cities that it thinks are close to beaches, and bubbles the result up to the country-level.

There is some averaging that is done to ensure that countries are not marked as false-positives. For example, you wouldn't say Canada is close to sand and beaches, but Toronto is. I have written some code to verify that a single city doesn't influence the overall country's classification.

The reason I went with this approach is that I like the information to be living at a more granular level. It would let me do city-level searching in the future.I like the information to be living at a more granular level. It would let me do city-level searching in the future.

This is why you can see city level information in Visabug. Cities are what actually power most of the non-visa data, and it's one of the areas of the site I want to improve.

Getting Images

I'm really happy with how the Visabug User Interface looks, and a large part of that is due to the imagery. It just makes me want to travel!

Images were very easy to get. I signed up for an Unsplash Developer Account that gave me access to 50 requests/hour through the Unsplash API. 5 hours later, I had images for all 238 countries in the world.
To determine what image to show for a country, I ordered Unsplash's images by likes and picked the highest-liked one.

Recently, I was approved to get an Unsplash Partner Account, which now gets me 5000 requests/hour. I intend to use this to have better images for Cities in the near future.

All image data is stored as JSON files, so I don't have to make any API queries in real-time.

Storing the data

The funny thing is that I wanted to build Visabug out really quickly, to see if there was any interest in the product. To speed things up, I actually launched the site without a database. 😅

Currently, Visabug has 2 JSON files: one with all country data, and another with all the city data. Together, they are about 300 MB. When the application starts up, this data is loaded into memory. This isn't ideal but has worked up until now.

Of course, I can't send 300MB of data to the client, so Visabug has a NodeJS server that does processing on this data and only sends back what the client wants. Everything is server-rendered and I don't have a public API yet.

What happened next?

I had acquired all this data by January, and spent the next month actually building out the product. This is what Visabug looked like in January.

Here’s what Visabug looked like 60 days ago. I have spent about 1hr on it everyday since then and it has come a long way!

Persistence and consistency are underrated. https://t.co/Lsy8sbqc2e
— Tilo (@tilomitra ) February 29, 2020

I am going to write about how I designed the website in the next post. I received help from Nathan Barry who generously helped me shape my Home Page messaging, and Chris Messina gave me many useful product tips.

Follow me on Twitter or here on Dev.to if you want to be notified when that post comes out. Of course, please do check out Visabug and let me know what you think!

How my JavaScript App went to #1 on Reddit (and how yours can too)

Tilo — Tue, 25 Feb 2020 16:06:10 +0000

Working on a side-project can be lonely. It helps to get validation from others that what you are doing is genuinely useful. Does that sound familiar? It’s the same boat I was in (and still am, somedays). I've been building a free web application called Visabug which helps people find unique destinations given their passports and travel interests.

In this post, I’ll highlight my journey over the last week where I went from no product validation to people thanking me for building my product.

These comments of validation from strangers really help to motivate and to guide future product development.

I have found that Reddit is a great website for mini-launches, as people cluster in groups (subreddits) related to one topic. My product is in the travel space, but it’s also a side project and built using React. I chose to target the sideproject (46k members) and reactjs (130k members) subreddits.

My general process was simple:

💻 Show the site
😃 Tell people why I built it, and tailor my post to something that would interest them
🙋‍♂️Be available to ask all questions.

Post #1

My first post was on the sideproject subreddit as it was smaller and I didn't want my site to go down if a lot of people visited it. Here’s the post that I made.

The side project Reddit group is entrepreneurial so I spoke about some of the product decisions that I made. I answered questions such as “what makes my product unique”, and “why I built this”. I also openly asked for feedback. I’ve been keeping a Trello sheet of all feedback so I can see what the most important features to build are.

The post ended up getting ~80 likes, stayed #1 for 24 hours, and sent 3,000 visitors to my website. The bounce rate was super low (30%) which is great! It told me I was on the right path.

Incorporating Feedback

After a few days, traffic went down again. This is natural and I expected it to happen. However, it brings back sad feelings and thoughts. They call it the trough of sorrow.

I took this time to address some of the feedback that was provided by the Reddit community. I improved my SEO by building out a better way to filter, updated the user interface, and added Analytics to practically every user interaction. This would help me track usage metrics to figure out which features were actually being used.

Post #2

After a few days, I posted on another Subreddit. This time it was the reactjs subreddit. It’s 3x larger (140k members). Since my application is built on React and Next, I figured people would be interested in it. Here’s my post.

Since this audience is mostly developers, I created a post with more technical post. I answered questions like “what is my tech stack”, “how I collected data”, “how I designed the web app”, etc. Once again, the post took off and reached #1 almost immediately. As I write this, it’s still #1 and has around 200 upvotes and 60 comments.

There’s a lot of great feedback in here for me. I’ve been diligent in collecting them to iterate on this idea. I’d like to thank Redditors who took the time to respond.

What I learned

Here’s what traffic looks like this week (the week still has a few days left!)

8,200 users (up 3,800%) ✅
9,220 sessions (up 3700%) ✅
32% bounce rate (down 44%) ✅
2m 12s avg duration (up 124%) ✅

So what did I learn through this exercise? Here are a few things I would keep in mind:

Create something polished even if it’s an MVP. A lot of people liked and upvoted me because what I built was considered to actually look like a real product.
Write a thorough post explaining your product and tailor it to the audience in mind.
Be open and available to answer questions.
Collect the feedback in an organized manner and iterate on it.

Dealing with failure

Lastly, if you post somewhere and it doesn’t go anywhere, don’t get too disheartened. Before these two posts took off, I had posted in some other subreddits which didn’t get any upvotes. There is an element of luck to all of this. However if you see the same pattern occurring 3-4 times, then you may want to analyze the situation further.

Let me know if you have any questions, and check out Visabug if you want to learn more about what I’ve been building.