Forem: G Adventures

Migrating our API Gateway from Flask to Django

Adam McKerlie — Tue, 17 Apr 2018 16:12:56 +0000

“Fresh pineapple sitting on the ocean shoreline” by Pineapple Supply Co. on Unsplash

At G Adventures, we’ve been building a publicly accessible API for the past three years. We’ve been pushing hard to make the G Adventures API a standard within the Adventure Travel industry, and we believe we’re doing so by providing superb documentation & support, as well as enabling our customers with a standards-compliant JSON/XML API, powered by a suite of services and tools.

When we first tackled the construction of our API, we wanted to keep things simple. We have many systems at G Adventures which own data, and we wanted to ensure that the API would not be an owner of data, simply focused on delivering data.

Thus, we looked towards frameworks that kept things simple. In the Python ecosystem, one of the most popular frameworks for keeping things simple is Flask.

Flask is a micro-framework that truly does not get in your way. It ensures you can write plain Python, with additional batteries included. This was great for our needs, as our mandate was:

Build a system that can capture a single request, and transform that into potentially multiple requests to source systems. Effectively, an API Gateway
Ensure we don’t re-invent the wheel when it comes to HTTP Standards
Keep it lean and fast. Speed is important. Both from development ramp-up and data delivery perspectives

Thus we began iterating on our API prototype. Flask truly did not get in our way. In a couple of days, we had a prototype that would proxy a single client request into multiple requests behind-the-scenes. We iterated on this prototype until we came closer to a reasonable product. Building out our resource definitions in pure Python, we were able to define common patterns, and store them as simple dictionaries. We’ll touch a bit more on what this looks like today shortly.

Eventually, we launched our API. It was a success! We had many internal, and external clients consuming G Adventures data. We began expanding our system by offering more resources, tools like OAuth, payload validation, and so on. As we began to add these tools, we introduced concepts like an ORM (Via SQLAlchemy) and migrations (Alembic) to our infrastructure.

These aren’t foreign concepts, and the tools mentioned are superb, but we’d began to slide away from our core fundamentals. Our API was becoming larger in scope, we began re-inventing the wheel, and our simple approach of dictionaries for resource definition worked, but didn’t. Essentially, we’ve hit a point where we could identify that if we continue to build upon this micro-framework, we’d build upon our technical debt. This was an opportunity to rethink

We began to think of how we move forward. We want to keep the API layer simple, and allow other microservices to provide the various features of the G Adventures API, but we wanted to have a consistent, well-structured set of ideas we could leverage and build from.

And thus, we waved hello to Django Rest Framework. Django Rest Framework is a perfect companion to a system where you have Django Models, and you want to expose an API to them. It’s so simple that most of our internal systems that expose an API to our API Gateway use it. However, we were in a different situation than a typical Django project.

We were actively running an API service which was receiving millions of hits per week, we wanted to leverage Django Rest Framework, but would need to pivot it to fit our needs of multiple HTTP backend services. Essentially, we had to make Django Rest Framework work in the ways we’ve structured our API. Let’s take a look at what that looked like.

Migrating while the machine is running

On the API, we support 90 resources, each with a different degree of complexity. A read-only resource on one backend is easier to test than a read-write resource using many backends. At the end of the day, migrating more than 5 simple resources at once is asking for trouble. Our decision was to migrate a few at a time and proxy the requests to the appropriate process (Flask or DRF). This allowed us to migrate slowly, communicate with our clients in case of errors and easily route back to Flask if needed.

Being able to revert to Flask served us well. You can migrate resources with precise respect to the documentation, but that does not guarantee success. Django Rest Framework could reject a request while the Flask framework wouldn’t. For example, Flask will happily process:

POST to /bookings POST to /bookings/resource

While Django Rest Framework will only accept:

POST to /bookings

Our documentation always said to use bookings. bookings/resource was not prohibited in our documentation, it simply wasn’t mentioned. One of our clients used bookings/resource, it went through and they continued to use it. When we migrated bookings to DRF, the client messaged us about it. Flask overlooked this error but DRF did not. Well, we were surprised bookings/resource was a valid request on our own API. I told the client to remove resource for all our resources. Now imagine we migrated all 90 resources at once — all integrations using <resource>/resource would be sad.

Customizing our endpoints

With Django Rest Framework, a resource is served by binding an endpoint to a view. For example, to serve bookings on the API, we simply had to register it:

router.register(bookings, BookingView)

Using DRF generic router, this will support requests on:

bookings/\<booking\_id\>

Our Flask supported other URL variations, and one of our challenges was to properly route all existing requests. Here is what we needed to serve:

\<resource\>/\<id\>/ \<resource\>/\<id\>/\<resource\_2\> \<resource\>/\<id\>/\<variation\_id\> \<resource\>/\<id\>/\<variation\_id\>/\<resource\_2\>

First, we found how DRF parses URL: https://github.com/encode/django-rest-framework/blob/1c53fd32125b4742cdb95246523f1cd0c41c497c/rest_framework/routers.py #L241.

This means our bookings/<id> will be valid through DRF’s default regex:

base\_regex=’(?P\<{lookup\_prefix}{looking\_url\_kwarg}\>{lookup\_value})’

URLs are constructed in get_urls, we can easily overwrite this function:

def get\_urls(self): urls = super().get\_urls() return urls + \<our custom urls\>

Our custom URLs will take care of:

\<resource\>/\<id\>/\<resource\_2\> \<resource\>/\<id\>/\<variation\_id\> \<resource\>/\<id\>/\<variation\_id\>/\<resource\_2\>

Let’s say we had an itineraries resource with ID 2222, a possible variation ID of 3333 and also a nested resource of itinerary_maps. These are all valid requests:

Variation 1: itineraries/2222/3333 Variation 2: itineraries/2222/itinerary\_maps Variation 3: itineraries/2222/3333/itinerary\_maps

First, we overwrote this method https://github.com/encode/django-rest-framework/blob/1c53fd32125b4742cdb95246523f1cd0c41c497c/rest_framework/routers.py#L230 to add extra regexes, such as:

\<base\> + "/\*(?P\<variation\_id\>[^/]\*)"

Notice in our get_urls method, we call super() before we create our nested URLs. super().get_urls() calls self.get_lookup_regex(viewset) which constructs our variation_ids before constructing our nested URLs (i.e. those ending with <resource_2>).

The call to super() creates:

itineraries/2222 itineraries/2222/3333

While making our nested URLs, we append these urls, also creating:

itineraries/2222/itinerary\_maps itineraries/2222/3333/itinerary\_maps

By constructing our regex orderly, we are able to create these custom URLs and bind them to the appropriate views.

Routing our backends

I mentioned backends a few times in this article, but never formally defined them. A backend is what we call one of our internal systems that provides data into the API. Our reservation system provides bookings, our Salesforce piece provides contact data, our operations tool provides tour data and so on. This means our API must know which backend(s) to use for which resource. We did this by defining a set of backends and binding resources (example):

class Backend: config\_variable = 'B1'

def request: \<initializes a request with the system\> def get\_object: \<uses request to get an object\> def update\_object: \<uses request to update an object\> def create\_object: \<uses request to create an object\>

class Resource: object\_backends = [(READ\_WRITE, 'B1'), (READ\_ONLY, 'B2')]

The above declares Resource to read and write from B1 and then use B2. Notice the order, B2 will actually override data from B2 if fields collide. Now we just need to patch DRF to use this structure, and this is easy. Here is just the function to override: https://github.com/encode/django-rest-framework/blob/7078afa42c1916823227287f6a6f60c104ebe3cd/rest_framework/generics.py#L77 . We did the same for update_object and create_object, allowing us to select backends for resources. Our Flask application operated the same way, but the code for this functionality was much more complex.

Now and future of the API

Our DRF framework has been serving all resources for over a year now and we could not be happier. While Flask served us well initially, we eventually found ourselves reinventing the wheel. With Flask, all the API basics had to be done from scratch. With DRF, we get basic functionalities and an easy way to add our own. A structured framework allowed us to focus on the actual API while allowing us to slightly customize the framework. DRF is just flexible enough to give us endless possibilities. We are currently customizing the API to route different requests to different reservation systems, stay tuned!

Keep following us on Medium and Twitter for more snippets of the Tech world according to G!

Want to help G Adventures change people’s lives and travel the world? Check out all our jobs and apply today.

Why we created a REST API at G Adventures

Adam McKerlie — Wed, 11 Apr 2018 13:43:55 +0000

Photo by Artem Sapegin on Unsplash

As a senior dev at G Adventures, I’ve had the great opportunity to work on the REST API here. This post is the first of many detailing how we got to where we are.

Our core business at G Adventures is small group travel, a social enterprise that strives to show the world to our travellers and in turn improve both the lives of our travellers and the lives of those who live at the destinations we visit.

As one can imagine lots goes into planning a tour on the other side of the planet to ensure everything goes smoothly from a day one. As our business grew so did our various systems. Eventually, it was necessary to have one common interface (or one common language) that our systems could use to talk to each other. We chose REST API which stands for Representational State Transfer and API stands for Application Programming Interface.

We use this interface to expose our business systems to developers both internal and external. As an example

contains all kinds of useful details about our Peru Panorama tour, including things such as tour’s itinerary or links to specific departures.

This means of communication has been a huge factor for our business growth as it allows our partners to integrate with us in a much more cost-effective way (in the past booking a tour with us required the use of a specialized website and often even a phone call). With REST API it’s a few HTTP requests instead.

Internally it allowed us to integrate our various systems and hide implementation details so that application developers no longer have to have intimate knowledge of every single system that we have. Once a developer becomes familiar with the REST API they can use any of our systems in uniform fashion whether they are interacting with our inventory management system or our customer management system (CRM), or anything else we expose via API.

The existence of this uniform interface even allowed whole new applications previously not possible. Such as our automated email engine or universal search functionality.

Not long after we launched the API did we discover the need to also be aware of when things change. E.g. if the above Peru Panorama trip no longer starts in Lima, we need to notify the website, our booking engine, all our API clients, who often display the trip on their own websites.

One way to do this is for every developer to check the above URL at specific intervals (polling). But this isn’t really a good strategy because we run around a thousand different tours with most of them having at least one departure every 10 days. That is a lot of URLs to poll.

Webhooks (don’t call us we will call you)

Webhooks can be compared to push notifications and are a simple mechanism to inform our API users when something in our ecosystem changes. In other words, rather than our users always checking the above URL to see if anything changed we will let everyone subscribed know if there are changes to the Peru Panorama tour.

These webhooks enable a universal mechanism of notifications that various business units use to obtain relevant info as soon as possible. Side effects included us being able to keep a replica of our data that we used for uniform searching and caching to increase functionality and responsiveness of our API.

Our external partners can use the webhooks to observe our inventory, watch for special deals or be immediately informed when anything happens on a booking they made with us.

I am proud of what we were able to accomplish with our API, but in my next installment I’ll get more technical as the story of REST API at G Adventures is far from finished.

Stay tuned.

Keep following us on Medium and Twitter for more snippets of the Tech world according to G!

Want to help G Adventures change people’s lives and travel the world? Check out all our jobs and apply today.

Deploying Code at G Adventures

Adam McKerlie — Mon, 22 Jan 2018 13:35:32 +0000

At G Adventures we deploy 20+ times a day to many different services that handle millions of requests and millions of dollars in transactions by keeping it simple

Danil Polshin from the Noun Project

One of my guiding principles for developing and deploying software has always been to keep it as simple as possible for as long as possible. Premature optimization wastes valuable developer time, solving problems that may never be real. One of the areas where this is apparent at G Adventures is how we deploy code. Today I’m going to talk about the common ways we deploy code, the pros and cons of each way and what the future may look like for us.

Environment

At G, we pride ourselves in using the right tool for the job. While we’re primarily a Python shop, using Django for our websites, we also have a few services written in Go and have started building out our front ends using React. Each environment comes with different challenges and best practices when it comes to deployment, which I’ll describe these below.

We use GitHub to host our code, Jenkins and Travis CI to automate our tests, and right now most of our applications and code live in VMs in a Colo just outside of Toronto. We’re in the process of transitioning to AWS but I’ll leave that for another post.

Other common technologies we use are Postgres (primary DBs), Redis (cache and key/value store), RabbitMQ (messaging broker) and Celery (task queue). In future blog posts, we’ll break down the individual teams and stacks.

Historically each team developed their own processes around code reviews, migrations and deployments. As we’ve grown we’ve made an effort to standardize our processes to make it easier for teams to share developers and resources. It’s still apparent in how we deploy with each team doing it slightly differently.

Current Deploy Strategies

Fabric + Git Pull

This is our simplest and oldest method of deploying our applications. It uses Fabric, a command line tool for simplifying SSH connections and git to update the code on our servers and deploy a new version of our applications.

The above gist is pretty close to what we do in production for our Django applications. We do have some code that determines which environment to deploy to as well as posting the status of deploys to Slack.

As you can see this deploy strategy is extremely simple and has worked across multiple teams for many years. The simplicity allows us to deploy new applications quickly and confidently and bring new employees up to date and deploying their changes within the first week of them starting.

There are some downsides to such a simple deploy process. Server, GitHub and PyPI connection issues can leave the application in an inconsistent state. In theory, running multiple deploys at the same time could cause migration issues. In practice, we rarely run into connection issues and because we post the status of deploys to a slack channel all it takes is for a developer to check that other deploys have finished before starting their own.

Symlink

An approach we’ve taken more recently is to build out a new environment entirely on each of the machines and then update the symlinks Nginx uses to serve the content. This allows us to deploy and verify that the application is in a consistent state before switching the symlinks and restarting the application servers.

Below is an example of how we deploy one of our React applications.

This deploy is almost as simple as the Git Pull method but adds an additional step of cloning into a new folder (e.g. app-2018010410224 ) and only switching the symlink once it’s been deployed across all of our servers.

The main benefit to this strategy is that if any of the requirements fail (connecting to a server, cloning the repo, building the application) across any of the servers the entire deploy will halt and production won’t be updated. This has been extremely useful as we’ve started moving to AWS and noticed an increase in connection timeouts. Rolling back deploys is also easier with this strategy. You simply update the symlinks to a previous working deploy and restart Nginx.

There are still a couple of downsides to this deploy strategy. Servers can still have different dependencies if you don’t lock down your pip requirements or node requirements and you need to manage how many past deployments you keep on the server or you can potentially run into using up the entire disk space. For a React application, you’re also still installing all of your node modules on each server which not only takes a long time but also uses a lot of disk space. This can be easily fixed by building locally and shipping off the application to each server.

Future

As you can see from the above examples we’ve been able to deploy tens of thousands of times by using simple deployment strategies. As we start to push past 50 developers we’re looking at ways to improve how we deploy code.

The biggest thing we’re looking at right now is continuous deployment after all of our tests are run. We already have Continuous Integration for all of our applications so the next logical step is to bundle up the environment, requirements and code and ship it off after the tests pass successfully. We’re also looking at containerizing all of our applications to help us manage deployment (we’ve just started looking at this).

How do you test and deploy your code? What strategies are you most comfortable with? We’d love to hear from you down in the comments.

Keep following us on Medium and Twitter for more snippets of the Tech world according to G!

Want to help G Adventures join our growing team and travel the world? Check out all our jobs and apply today.

Originally posted on tech.gadventures.com

Translating React Apps

Adam McKerlie — Wed, 27 Sep 2017 20:40:28 +0000

Originally posted on tech.gadventures.com

Imagine you’re in a Roman train station in the middle of the night, and nobody’s around. You need to purchase a ticket to Venice. The ticketing software is in Italian. You don’t speak Italian. After a bit of a struggle, you manage to get a ticket. You get on the train. More Italian. A 3 hour trip. You look at the welcome sign. â€˜Welcome to Florence’. Sigh. Florence is not Venice.

Roma Termini TrainÂ Station

In the ideal world, we’d all speak many languages, like some of our European travelers. But not everyone understands English, so it’s better to support as many languages as possible.

As some of you may know, G has a travel app called Good to Go. There are certain requirements you may need to complete before you travel. These include sending us a copy of your visa, your jacket size, your food preference, etcâ€Š–â€Šwe refer to these as “trip requirements”. Before travelers start their adventure, Good to Go helps them make sure they have all the things they need before they travel.

Thousands of travelers use it to complete their trip requirements before they begin. Every single one of our departures goes through Good to Go. It’s a big star. In fact, G-Stock 23 marks the 1 year anniversary of Good to Go. We continue to work on it and make it better every single day.

Last month, we brought German support to Good to Go, which is written in React. It was pretty easy thanks to Yahoo’s React-Intl library. This post details my experience adding internationalization to Good to Go.

Firstly, when it comes to translating, the order of the words is important, as well as the formatting of dates and names. Grouped together, these things are known as “locale data”.

Installing react-intl is simple enough, but there are a few additional steps you need to complete after the initial yarn add react-intl. Let’s go over them now.

Babel

I’m going to illustrate how to do this with Babel and Webpack. First, install the babel-plugin-react-intl plugin (yarn add babel-plugin-react-intl). It will extract the strings marked for translation. To activate the plugin, you have to then add its name to your .babelrc file.

"plugins": [
    ...,
    ["react-intl", {
        "messagesDir": "./src/translations",
        "enforceDescriptions": false
    }],
    ...
]

In the example above, react-intl will extract tagged messages into src/translations. The messages are stored in JSON files, inside directories that are named after your components.

When it’s time to send the messages to a translation service, you will probably want all of them in a single file. For this reason, the creators of the library have written a script that you can use to combine all the strings into a single file. To see a more detailed example, see this post.

Locale Data

The locale data for each language is provided and imported independently so you’re only sending down what you need. In the following example, we’re importing the locale data for English and German.

import { addLocaleData } from 'react-intl'
import enLocaleData from 'react-intl/locale-data/en'
import deLocaleData from 'react-intl/locale-data/de'

So the process is:

Install the library
Install the babel plugin
Import the locale data

Finally, you wrap your root component with IntlProvider, so that all the translated messages and locale data can be loaded into the React Intl Components used by your app.

<IntlProvider locale="de" messages={messages}>
     <Root />
</IntlProvider>

Another method is to inject these values right into the intl reducer during the “boot up part of your app’s lifecycle, like index.js or a configureStore method.

{
    ...,
    intl: {
        defaultLocale: 'en',
        locale: 'en',
        messages: allMessages.en
    },
    ...
}

Formatting with React-Intl

The heart of react-intl is its tagging system. You can tag (or mark) strings for translation by wrapping them in a Rect component or using the API.

A React Component

import { FormattedMessage } from 'react-intl'

<FormattedMessage
    id="header.selectPax"
    defaultMessage="Select the traveller for whom you'd like to enter requirement information." 
/>

Using the component is straight forward, and powerful. For example, they can take values, illustrated in this example:

<FormattedMessage
    id="form.name"
    defaultMessage="{name}"
    values={{ name }} 
/>

The object you provide to values can be another FormattedMessage, meaning you can string them along like a chain to create a message with HTML tags!

<FormattedMessage
    id="body.text"
    defaultMessage="{bodyText}"
    values={{
        bodyText:
            <div>
                <h2>
                    <FormattedMessage 
                        id="bodyText.1"
                        defaultMessage="Ready for a G Adventure?" />
                </h2>
            </div>
    }} 
/>

import { defineMessages } from 'react-intl'

const formButtons = defineMessages({
    login: {
        id: 'formButtons.login',
        defaultMessage: 'Login'
    }
})

With the API, you define your message ahead of time in a simple Javascript object.

The API provides you with translated strings in places where a React component would not be appropriate. For example, inside of a <button>.

import { injectIntl } from 'react-intl'

const myForm = ({ intl }) => {
    <button
        type="submit"
        onClick={handleSubmit}>
        {intl.formatMessage(formButtons.login)}
    </button>
}

export default injectIntl(Waiver)

You’ve seen two ways you can use the react-intl library to translate your strings. I recommend using it everywhere. You will delight your users when they see your content in their native language.

Look out for Part 2 of this series on translation, where we go over how to build out a Language Picker that uses react-intl.

Keep following us on Medium and Twitter for more snippets of the Tech world according to G!

Want to work at G Adventures? Join our growing team and travel the world! Check out all our jobs and apply today.