Forem: Gordon Wintrob

A new approach to an old-school API

Gordon Wintrob — Tue, 17 Oct 2017 15:03:44 +0000

Welcome to GET PUT POST, a newsletter all about APIs. Each edition features an interview with an API-first business and killer app ideas for developers to build on their platform. Want the latest interviews in your inbox?

This edition, I spoke with Zack Kanter, Founder & CEO at Stedi. They’re modernizing an antiquated protocol that powers millions of transactions but you’ve probably never heard of itâ€Š–â€ŠEDI. We talk about this founder’s journey from the pain point of interacting with retailers to devising an modern API-first strategy. We also dive into their serverless architecture and how to gather feedback pre-launch. Enjoy!

How did you get started with Stedi?

I founded a brand of 2,200 auto parts manufactured in Taiwan and China. I was running the business myself and when it came time to sell my products to retailers, I didn’t want to manually enter orders one by one.

I reached out to all the retailers and said, “Can you send me your API spec? They all responded, “What’s an API? I explained, “I’m trying to automate all the orders. They told me, “You’re looking for EDI.”

EDI is a technology and taxonomy for thinking about how companies can talk to each other. It handles ~300 different transaction types, including everything from a purchase order and confirmation to instructions on how to load something onto an ocean carrier. It’s also a file format and a transfer protocol.

The taxonomy is elegant, but the file format and transfer protocol is hideous. If someone wants to sign up for EDI today, it’s an arcane process that involves scheduling a demo with sales on one of our competitors’ websites. They reach out, do a WebEx demo, receive a contract, negotiate the pricing, print it out, scan it, fax it back in, etc. Six months later, after a dozen screensharing sessions with an implementation specialist, they can start using EDI.

Our short-term vision is building a self-serve experience for EDI that feels like Shopify or Stripe. Any non-technical, business-minded user can sign up for EDI and automate their transactions or receive purchase orders from places like Wal-Mart, Amazon, or Best Buy within hours instead of weeks and months.

Why is there a custom protocol for commerce versus something like an XML feed?

EDI was developed in the 60s. It was originally mainframe-to-mainframe communication. This format was then picked up and popularized by Wal-Mart in the 80s. Wal-Mart was one of the leaders in popularizing EDI, automating transactions with their suppliers, and giving their suppliers data feeds that showed sales velocity so that they could do upstream planning.

This was before XML and JSON. Since data was extremely expensive back then (Wal-Mart was one of the the first companies to have their own satellite system for sharing this sort of data, which was expensive to do), it is optimized for what’s known in the EDI industry as kilocharacters (a kilocharacter is 1000 characters, which is roughly a kilobyte). They tried to optimize for keeping the kilocharacter count as low as possible. It’s this complex, hierarchical flat-file. It’s very odd, but it’s all designed this way in order to minimize data usage.

Then, when Amazon came about in the 90s and early 2000s, they wanted to do business with all the companies that were supplying Wal-Mart. The fastest way to onboard those supplier was to adopt the same standard the Wal-Mart was using. Fast forward to today, everybody uses EDI.

The specific type of EDI that is popular in the US is called the X12 standard. The amazing thing is that the entire retail commerce industry has basically adopted the same standard, which means that people can code to one standard instead of having to do 500 different one-off integrations for each retailer.

I’m divided on the merits EDI. If we all were to redo this from scratch today, under no circumstance would I say, “Let’s do it the same way with the same file format, optimize for kilocharacters, etc. However, if each retailer went out and designed their own awesome API and we had to integrate with thousands of different trading partners with their own APIs that changed over time, that would be a nightmare.

Even though it’s a standard, everybody has a different implementation or interpretation of it. If I’m looking at a given field in an EDI document, I know that it’s going to be one of 3,000 possible values with regard to units of measure. Each company might use five different units of measure, so there is some degree of configuration and customization involved in every EDI mapping.

And Stedi’s function is to abstract away those idiosyncratic differences?

That’s right. Our customers should never have to think about EDI. We handle all of the complexity and just give them our internal, canonical, JSON format.

Will you integrate with retailers individually or get open access to everyone?

We do have to integrate with each retailer individually, but because each is based on the same standard, we’re able to build a somewhat universal translation engine that is configured differently for each retailer. You can feed us purchase orders from Wal-Mart, Target, Amazon, and we map each one to our internal format.

There’s a fair amount of overlap from one to the other. Once we finish the first 50 retailers, we might be approaching 98% of the mappings that need to be done.

What has the feedback been from retailers?

EDI is a permissionless framework. As long as people can do the EDI handshake and create valid response documents in the EDI format, they generally don’t need to get permission from retailers. If someone wants to integrate via EDI with Amazon, for example, Amazon doesn’t have to approve the company that is handling the EDI integration.

For example, let’s say a water bottle brand sells $5mm worth of water bottles on Kickstarter. If Amazon wants to start stocking those water bottles in their warehouses, Amazon would set up a vendor relationship with the brand and start issuing purchase orders. If the manufacturer wanted to automate those orders, they would walk through Amazon’s EDI onboarding process. Amazon would ask, “What is your EDI server address? and that address could be Stedi’s EDI server, a competitor’s’ EDI server, or a server running in your garage.

There are ways that we can change that over time. Because so many of the EDI companies out there take so long to do integrations and their process is so difficult, we think that over time, if we do 50 awesome supplier integrations for a single retailer, the retailer might say, “Hey, this is working really well. This is taking one-tenth the time that our standard integration takes. Once that happens, we can work out some sort of deal with the retailers.

But in the short-term, it’s an invisible processâ€Š–â€Šthe retailer might not even know that Stedi exists.

Can you give some other examples of companies that would want to work with Stedi?

Anyone who’s selling a physical product to a major retailer could be a customer of ours.

Each EDI transaction has a numberâ€Š–â€Še.g. 850 (purchase order) or 855 (purchase order confirmation) or 810 (invoice). There isn’t a purchase order for auto parts and another for electronics; it’s a single purchase order spec that encompasses any type of product that might be ordered. With the integrations we’ve built, we can automate transactions for any type of product from groceries to consumer electronics.

What’s your go-to market?

Six months ago, I would have said, “We’re going to start off by signing people who are not yet using EDI, because it’s going to be pretty hard to get people to switch. That was our first assumption.

Source: Zack’s TEDÂ Talk

But we’ve done some content marketing and I wrote a piece about Amazon for TechCrunch that went viral. We got a ton of beta signups. The crazy thing is that 9 out of 10 of our beta signups are current customers of our competitors.

We thought it would be the opposite, but Stedi resonates with people who have gone through or are going through the pain of an EDI implementation. It’s hard to imagine how bad it is until you’ve tried.

The trap that I’ve fallen into before is the “If you build it they will come assumption. Build a beautiful API and then assume people will just use itâ€Š–â€Šbut it doesn’t work that way.

I can’t go into our strategy in detail but there are a number of idiosyncrasies specific to EDI that we’ve figured out how to leverage.

Think forward 10 years when everyone who needs to integrate with retailers is doing it through Stedi.

There are two types of EDI: “webforms and “integrated.”

Webforms is really just a browser-based interface for manually processing EDI transactions. Believe it or not, the vast majority of EDI users today are using this manual webforms process where they are actually copying and pasting data for every single transaction.

Webforms is basically a portal that pulls together orders from different retailers allows the user to create response documents. For example, a company receives a purchase order, downloads that as a PDF or CSV, manually creates a purchase order confirmation, and then a ship notification and an invoice.

We think this whole process is crazy. The original dream of EDI was to automate all the transactions between two trading partners. Webforms isn’t automationâ€Š–â€Šit’s just a marginally-faster way of manually processing transactions. We’re focusing on “integrated EDI, where we are fully automating the end-to-end transaction flow. We’re doing that with both our API and with prebuilt integrations to ERPs and other platforms like Flexport.

How the Flexport API enables global trade

Once we’re processing all of this structured data and everybody’s a Stedi customer, our plan is to build out a developer marketplace or app store that allows people to build business applications on top of our pipes.

Tell me more about your stack and what different technologies you’re using.

I’m a huge believer in the idea of serverless. We’re leveraging AWS Lambda and services like API Gateway, SQS, and RDS Aurora heavily. We’re months into building and we still have no servers, and we think that is going to give us a big advantage in terms of DevOps overhead down the line.

For us, using something like AWS Lambda, where we get a million requests for 20 cents, gives us a few advantages over our competitors. You can imagine what the weeks leading up to Black Friday and Cyber Monday is like for a legacy EDI company. Serverless gives us the ability to scale massively for peak demand without paying for any unused capacity.

It also means that we can do granular cost accounting on our transactions we want to land a huge retailer as a customer, we are going to have to get competitive on price. We tag all the Lambda and API gateway calls and know what it would cost down to fractions of a cent. We can figure out exactly what our transaction cost will be for processing an EDI document. That’s a lot harder to do with auto-scaling servers.

You’re using Java?

We are. We didn’t set out to use Java from the get-go. I said, “We’re going to try to be language agnostic and pick the best backend engineers we can find.” We interviewed a ton of people and the best person we found was a Java engineer.

The nice thing about using AWS Lambda is that if we wanted to, we could have one function in Python, one function in Java, and one function in something else. It’s easy to break everything apart.

Obviously, that would create other organizational issues; we’re trying to stick with Java for the most part, or at least JVM languages. At some point, we’ll probably rewrite our EDI parsing engine in something like Scala, but for now, we’re using Java. Java isn’t cool, but serverless is cool, and we think that balances it out.

How are you gathering customer feedback before you get started?

There’s a famous saying, “No restaurant ever went out of business for being too small. And that’s something we think about often.

The problem with EDI, however, is we can’t say, “Okay, we’re going to launch an MVP and people can receive purchase orders, but they can’t send confirmations or send invoices. All those things have to be done in order for it to be useful.

And we can’t hack it together and only have 99% of the messages go through successfully. We’re dealing with people’s money and with actual orders that are shipping real product in the physical world, so it has to be 100% accurate.

On top of that, few brands only sell to one retailer, so we have to build integrations for most of the major retailers in order to be useful for our customers.

So there is a big development hurdle, which is why we had to be a well-funded company in order to build a fully-functioning EDI platform even though we’re focusing on building a little as we possibly can in order to service early customers..

The problem we ran into was that we didn’t want to build in a vacuum. We needed a way to ship often and avoid working for months without any validation, but we couldn’t afford to screw up real customers’ transactions.

We have a unique advantage here. I still own the auto parts company I previously founded (though someone else runs it now), so we are able to roll out functionality piece-by-piece for a “real alpha customer without fear of really pissing anyone off. This closed feedback loop allowed us to build really quicklyâ€Š–â€Šwe were able to quickly start processing live transactions without any potential reputational risk.

For finding bugs and figuring out all the edge cases that come up with EDI, we have a cohort of pilot users who are automating their retailer relationship using Stedi before we start onboarding paid users. Ultimately, we’re working with beta users who have real businesses and can give us brutal feedback, because that gives us the information we need to make the platform great. Once we have everything dialed in for the first cohort of early users, we’ll be ready to publicly launch.

If you like the post, I’d really appreciate you sharing it on Twitter.

Want more API interviews in your inbox?

Building an API on open data

Gordon Wintrob — Thu, 03 Aug 2017 19:01:35 +0000

Welcome to GET PUT POST, a newsletter all about APIs. Each edition features an interview with a startup about their API and ideas for developers to build on their platform. Want the latest interviews in your inbox? Subscribe here.

This edition, I spoke with Ed Freyfogle, Co-Founder of OpenCage. Their API makes it easy to turn addresses into coordinates and, unlike Google and others, it’s built on top of a completely open data set. We discuss how the business got started, their stack to fan requests to other services, and scrappy marketing efforts.

What is OpenCage?

OpenCage is a geocoding service that incorporates all the benefits of open data. We help developers create solutions in ways that they can’t when working with proprietary data services and aim to make geocoding as simple as possible.

Before we started, I had a company that did online real estate searches. That brand was called Nestoria and we built a real estate search that worked for about ten countries.

Geocoding is the process of converting addresses into geographic locations and vice versa. There are great tools available for geocoding in countries like the US, UK, and most major European states. However, it’s a lot more difficult in countries like India and Brazil where Nestoria was involved.

We built our own geocoding service because we were geocoding millions of Indian and Brazilian properties. In these countries, it’s difficult to purchase geodata (if it even exists). To build our internal service cost effectively, we had to work with open data.

In 2013, we took the geocoding technology and turned it into its own brand: OpenCage. Nestoria was acquired in 2015 by one of our competitors and we spun OpenCage off as its own company.

How much usage did you have when you made that leap to exclusively doing geocoding?

We first released OpenCage as an experiment and called it a beta service. We learned a lot by listening to our users and got great feedback. Initially, it was free and then we talked with people who were using it, found out which services they would be willing to pay for, and created a business around that feedback.

We only introduced pricing after we added a lot to the service and made it more reliable. We’ve seen slow and steady growth in terms of usage and number of customers. The pain and pleasure of a geocoder is that the work is never done since the world is a complicated and dynamic place. New mapping issues and use cases come up continually.

There’s a massive demand for geocoding. There are many different offerings on the market from proprietary players like Google and developers can theoretically build their own service on open data. However, at this point, it’s far more efficient to use a service like ours rather than try to build it from scratch.

How many people use the API?

We offer a couple of different tiers. Tens of thousands of developers have joined our free tier and a smaller percentage of them are in our premium tier.

There’s diversity in the types of geocoding that people need. Many developers have a project involving a large database of addresses that need to be geocoded and then no longer need our service. We can’t build an ongoing business around interactions like that, so we focus on customers who have an ongoing need for geocoding.

What are some examples of use cases for the API?

Often it’s people who somehow have a database of addresses which need to be converted into coordinates. That’s the bread and butter of our use case. Those addresses can come from everywhere from online shopping transactions to patient records.

One area of growth is reverse-geocoding, particularly for vehicle tracking. Every new car continually records its travels and gathers coordinates. Customers want to convert those coordinates into addresses.

The cost of a device that can record location has fallen dramatically and it’s getting to the point where high-end bicycles often have location tracking capabilities. The problem is that coordinates mean little to humans.

For example, when someone rents a car in Spain, the agency asks customers if they’re going to leave the country. If they say yes, they pay more. The agency has a tracking device in the car that records the coordinates of its journey. Periodically, agencies check a car’s data and need to convert that data into information about whether that car has crossed into France.

A large portion of our business comes from developers working on apps where users allow access to their location data. Developers find it useful to convert those coordinates into human terms.

Do you have any customers that you’d like to highlight?

The vehicle tracking industry is huge and many of our big customers are in that space. One example is Bosch (a company well known in Europe as an electronics supplier). They use us to track cargo containers.

Geocoding isn’t particularly glamorous, but it’s important with regard to the logistics of getting things from A to B.

How do you market OpenCage?

Since OpenCage was originally a smaller part of another business, we already had some contacts. Those were our first users and they typically came from the online real estate industry.

The next step was to become active in the OpenStreetMap communityâ€Š–â€Ša global community of people interested in mapping. From a technical standpoint, it involves literally mapping the entire world down to insane levels of detail. We initially sponsored a geo event in London and expanded from there.

Source: YouTube

People search for geocoding help so marketing our product means being good with SEO and being in places where people ask technical questions regarding geocoding.

Usually there are three reasons why people are unhappy with proprietary suppliers like Google.

First, proprietary suppliers are not useful in their country, which is the case in many less developed markets. Second, people dislike their terms of use, which hinder developers from fixing bugs themselves. The third reason is priceâ€Š–â€Šthey become expensive when geocoding at high volumes. We’re much more affordable because open data is available for free and we’re a smaller operation.

How large is your team?

My co-founder and I are the only ones who work full-time. Depending on the project, we may bring on freelancers and contractors.

How do you help bring on new customers?

We have a free trial to let people test our product. It’s also necessary to have great documentation that makes it easy for developers to get started.

Developers considering Google Maps are faced with an extremely wide variety of cluttered and confusing services and can easily waste days reading about each option. It’s hard to make a choice without having a background in this space.

That’s why our position is clear. We’re simple. We just do geocoding.

Conversely, there are developers who think that they can build their own system with OpenStreetMap. The reality is that it’s quite complex, both in the software necessary for geocoding and the underlying data. It’s not easy to keep a geocoding database up to date.

How do you prioritize features?

We rely on customer feedback and focus mainly on the people paying the bills. We’ve had several features that have come about in that way.

When someone asks for a feature, our first response is “Will you pay us to develop this?” We often still offer it to all users, but a paying customer really validates the need

For example, we had a customer last summer doing medical research. He had data regarding patients and needed to geocode their addresses. He was very concerned about privacy, so we added a new privacy parameter. This was very much a custom feature but it was useful to others.

Tell me more about your interview series.

Most of our data comes from sources like OpenStreetMap that have great global communities.

People all around the world are mapping their local neighborhoods and countries. There are different motivations for why people do that, but one is that they want to put that data into OpenStreetMap so that they can use it. We reach out to those communities, encourage them, and contribute to them.

We don’t have a massive budget, so we can’t be the triple gold sponsor of OpenStreetMap conferences, but we can make people aware of the OpenStreetMap community. One way we do that is by hosting an interview series on our blog where we interview people from all over the world including places like Tunisia and Costa Rica.

In some less developed parts of the world, there are no maps and people are trying to make maps for those countries. We piggyback on that global community and provide a forum where everyone can talk.

What do you wish people would build on top of OpenCage?

One of the big challenges we have isâ€Š–â€Šbecause it’s such a low-level serviceâ€Š–â€Šthat we often don’t know exactly how people are using it. We just see queries coming in and we don’t usually know what the final product is. We’re usually only a small part of very complex projects.

The most common use case consists of people who come to our website and put in their home address. On the map, the exact coordinate could be off by 20 meters, which is confusing to some. However, many of our customers aren’t interested in precise accuracy.

For example, we once had a customer that specifically did not want precision. Their use case was an app where people could make and send videos to friends. When the friend received the video, this customer wanted to show the location of the sender, but not have that location be super precise because of privacy issues.

One area where developers can build is by providing “annotations” based on the coordinate data. We can add information about time zones, currency, the country calling code, and other pieces of relevant information that might be useful.

Tell me more about your stack.

We are a simple wrapper around different open geocoders. When a query comes into us, we do some validation and authentication of the query and then fire it off to the different services. There are some geocoders that are designed for specific countries that we don’t use for every query.

We receive the results, de-duplicate them, rank them, clean them up, add annotations, and send that back to the user.

We have our own servers; we’re not on the cloud. The challenge is the sheer volume of data that we’re managingâ€Š–â€Šit’s now over 1TB. It’s much more cost effective to lease our own servers than it is to use an AWS or cloud-based approach.

Our server is pretty thin and the software is written in Perl, which is very good at text manipulation. We do authentication and validation and then send off queries to the different geocoders. The whole point of using our API is that it doesn’t really matter what our stack is. Requests are fanned out and aggregated.

We have libraries in almost every major language to make it easy for developers to interact with us. We’ve written and maintained some of those libraries, while members of the community submitted others. They’re all open source.

What happens when one of these services is down or slow?

Some of them can be slow. If a certain geocoder hasn’t responded, then we just rely on the ones that have returned.

One of the challenges we face is garbage input that slows down the geocoders. Sometimes a user will want to geocode a database, so they’ll write some for loops before really looking at their data or cleaning it up.

Sometimes users just get data from crawling the web. They often scrape all kinds of junk and then fire it at our service. We try to catch stuff like that before it gets to the geocoding services.

Want more API interviews in your inbox?

This post was originally published on medium.com