Developers need choice

Peter Cho — Mon, 26 Mar 2018 16:16:16 +0000

Close your eyes and imagine this scenario for me. You’ve been hired as sous chef of a new restaurant. You walk into the kitchen and they say “We just signed a purchase order with DepotMart so here are your DepotMart knives and moving forward all of your ingredients will be from DepotMart and also here are some DepotMart brand shoes, shirt, and pants.” You say “but I’m used to my Wüsthof knives and I quite like the pants I’ve got and I wasn’t aware that DepotMart even sells groceries." They reply “too bad, procurement signed the contract, this is what you’re working with.. for the next five years.”

Now replace DepotMart with Microsoft, Amazon, Google, or any of the other major monoclouds. Companies who are beholden to their shareholders for selling Operating Systems, Retail, and Search, they are now responsible for giving you the tools and infrastructure to build your software.

Don’t get me wrong, the big box clouds have some pretty nice tools and stable, scalable infrastructure with interwoven components that work fairly well together. (and many of us are very happy to be entirely under the warm loving embrace of Jeff Bezos for all of our needs) But where will that put us 2, 5, or 10 years from now? Do we honestly believe that they will innovate of their own volition, when there is no competitive incentive to do so? cough*Railroads and Telephone companies*cough

On the flip side, companies like Mailgun live or die by whether or not they provide an amazing experience for transactional email. LogDNA only stays in business by providing stellar logging in the cloud. There are hundreds of developer services like this, companies who do one thing extremely well, who prioritize developer experience over all else.

That’s why we started Manifold.

We offer convenience to developers by letting them use one credit card and one account to purchase Postgres, memcache, monitoring, etc. You can integrate into Terraform, Kubernetes, Laravel, and you can invite team members and manage on a per-project basis. But those are nice-to-haves, they’re vitamins. What we’re really after is giving developers the freedom to choose, to choose amongst tools and services created by developers, for developers.. a flourishing ecosystem of companies that are dedicated to making your life easier and your code more effective, always innovating and pushing the boundaries of what is possible with software.

If you’re a developer, come try out the first handpicked batch of services! We’ve got RabbitMQ, Redis, Elasticsearch, all the essentials. We’re also going to be launching new services every month through the rest of the year, so if you don’t see anything you need yet, you assuredly will in the coming months. Added bonus, use coupon code DEVTO2018 to get $10 in credit to your account. Extra added bonus, if you email me your feedback on our product I’d love to send you some swag!

If you offer an amazing developer service, let’s talk! We are always on the lookout for companies who share our goal of making developers’ lives easier. Whether you’re an up and comer or been in the game for a while, we’re going to be launching tons of services this year and would love for you to join us.

!!! @manifoldco just launched!!! Super excited for this product. Go check that shit out: https://t.co/37fa6WexK3 It's like Steam for devs.
— Randall Degges (@rdegges) May 3, 2017

When it comes to choice, as consumers we have access to the Google Play Store and Apple App Store, gamers have Steam (I can’t imagine a world in which EA or Ubisoft would have created Goat Simulator), but for some reason developers are still waiting for our Cambrian explosion. We’re still out here grappling with the choice between Microsoft’s MicroCache, MicroDB, and MicroCompute service or Amazon’s AWSCache, AWSDB, or AWSCompute service..

Hopefully we can build a future, together, in which developers have choice.

📈 A/B testing with limited data at a startup

Colin White — Fri, 23 Mar 2018 18:12:39 +0000

Originally posted on Manifold

📈 A/B testing with limited data at a startup

Practically every article you read about startup marketing stresses the importance of A/B testing. From the header on your landing page to the colour of your signup button, even the most minor thing should be tested. But, almost all of these articles you’re reading assume one thing. That you’re in growth mode.

But that’s an issue for a lot of companies.

Startups that haven’t hit growth just don’t have enough data to run A/B tests at scale. This means that testing at low numbers is a completely different ballpark than when you are scaling. I’m not saying that it’s impossible, but I’ll break down some common misconceptions and mistakes that are easy to make when testing with small amounts of data.

Let’s quickly define A/B testing

A marketing experiment where two variations of a landing page, ad, email or other piece of online content are pitted against each other to determine which produces the highest conversion rate.
https://unbounce.com/conversion-glossary/definition/ab-testing/

That’s the definition from a marketers point of view. Pit two pieces of marketing material against each other to see which converts better. This definition doesn’t do justice to the statistical methods behind A/B testing.

When you run an A/B test, you are most likely doing a comparison of two binomial distributions using some type of statistical test (there are a few possibilities).

Things like click-through rates or conversion rates are binomial distributions.

If you are using A/B testing software like Optimizely, a lot of these statistics are hidden from you, which can sometimes be to your detriment.

A/B testing pitfalls

Testing software makes running experiments approachable to all types of people. You really don’t have to know much about statistics to pull off an A/B test in Google Optimize or Optimizely. But not having much knowledge in stats can lead you down the path to some easy to avoid mistakes.

Let’s look at a few.

Ending your test too early

If you’re like me, you LOVE watching the numbers go up when you’re running experiments (or ads, or anything). And one of the best numbers to watch is what most A/B testing software has now. The “chance to beat” metric. This is a number that the software calculates on the fly using a multitude of variables available.

This number can be misleading. If your “chance to beat” is at 100% you might be thinking, “alright, let’s end this and start the next test”, but that can get you into major trouble. This is called peeking. Peeking is when you look at the data before you’ve gathered a big enough sample size and means you haven’t hit statistical significance. Following the process of ending your tests early can lead to false positives that aren’t going to be correct in the long run.

Running A/B tests takes a lot of traffic, especially as you get further into your funnel. Users drop off and you get less and less people seeing your test.

Let’s look at an example. Say you have a conversion rate on a landing page of 5% and you want to get it up to 5.5%. That’s a 10% increase. Seems pretty reasonable to get there. But to be confident in that change to the landing page, you’ll need each variation to get a sample size of ~30000 visitors.

I don’t know about you, but getting ~60k visitors to a landing page can be pretty tough when you are early in your startup’s lifecycle.

Checkout Evan Miller’s awesome sample size calculator to better understand the audience size you need: https://www.evanmiller.org/ab-testing/sample-size.html

Not understanding your audience

Not all visitors to your website are the same. As marketers & founders, we know motivation and intent are key to marketing proficiently.

A/B testing is the same. If you are running a test that needs 10000 sessions per variation and all of a sudden get 20k hits to your landing page because a blog post blew up. Is that really a good sense of what your conversion rate is going to be? Probably not. You need to make sure you understand who is visiting your site before you can make a call.

Sometimes it’s worth running a test to a higher significance if you’re not confident you’re sample has a decent diversity.

Testing all the small things

I hope you don’t have blink 182 in your head now.

I’ve talked a lot about the sample size that’s needed to run tests. In some cases it’s huge. And one of the biggest contributors to your sample size needed is the percent lift you want to see.

Let’s go back to the example above of our 5% -> 5.5% conversion rate. Testing to significance for that example, we would need to have around 30k visitors per variation. But say we wanted to see that conversion rate lift to 6% instead by changing the whole landing page and not just the CTA copy. That drops our visitors needed per variation down to ~12k. That’s half the visitors and a much more achievable number.

Making large changes to whatever you are testing is important when you you can only muster a small sample set. If you are changing something small, you won’t be hypothesizing a large lift in your test variable. But, if you make large changes to your content and go for a higher change in the test variable, your sample size needed will decrease dramatically.

Don’t just go and boost your goal lift to a crazy amount here. You still need to think of testing in a scientific way and choose all of these metrics based on a good hypothesis. Otherwise you are never going to hit significance and you’ll never learn anything.

Testing is hard

And it’s even harder at a startup. Tools like Optimizely and Google Optimize are making it easier and easier to run experiments, but they don’t give you all of the background math that needs to be done. Not knowing those statistical methods that are running in the background can lead you to false positives.

Make sure you’re setting your sample size before your test and sticking to it. Don’t call a test done just because it looks like it’s going to win. Remember, data trumps gut if you have it. Watch out for traffic spikes from one spot, it can make your audience less diverse and skew your results to the max. And lastly, don’t test the small things when your throughput is low. If you’re looking to see significant results at low sample sizes, make big changes. Switch out the landing page for a completely different one, or completely change up the ad you’re running, not just a few words.

Feel free to hit me up at colin@manifold.co if you have any questions, I love to chat about this stuff.

Forem: Manifold

Developers need choice

📈 A/B testing with limited data at a startup

📈 A/B testing with limited data at a startup

Let’s quickly define A/B testing

A/B testing pitfalls

Ending your test too early

Not understanding your audience

Testing all the small things

Testing is hard