Forem: Olga R

Boost Your Testing Strategy: The Coolest Methods to Prioritize A/B Tests Like a Pro! 🎲📊😎

Olga R — Tue, 11 Apr 2023 09:01:54 +0000

In today's fast-paced business world, continuous product improvement and innovation are key to staying competitive.

This is where A/B testing comes in as a powerful tool with both advantages and drawbacks. While A/B testing can yield valuable insights, it can also be expensive and time-consuming. Therefore, it is crucial to focus on running only the most valuable experiments.

To prioritize hypotheses, there are several methods available, including ICE, RICE, and PIE. We will discuss each of these methods to understand which one is better.

ICE. Impact. Confidence. Ease.

To effectively use the method, the team should assign a score to each of the three factors mentioned above and multiply them together. The smaller the score range used, the better, as it simplifies the process and helps to better assess the parameters.

Impact. First, consider the potential impact of the test configuration on the key metric being optimized.
Confidence. Next, assess how confident the team is in the effectiveness of this experiment based on their experience.
Ease. Finally, determine how easy it is to implement the idea by assessing the number of team members required and the amount of time it will take. The easier it is to implement, the higher the score.

Once you have scored each factor, use the following formula to calculate the overall score:

ICE Score = Impact x Confidence x Ease

RICE. Reach. Impact. Confidence. Effort.

This method is quite similar to ICE, but it helps make a decision regarding the potential reach of the test.

Reach. This is an estimate of how many users the idea will affect. For example, if the idea will affect all users, assign a value of 5. If it will impact 50% of users, assign a value of 4. If it will affect about 20% of users, assign a value of 3. If it will affect about 5% of users, assign a value of 2. Finally, if it will affect less than 1% of users, assign a value of 1.
Impact. Think about how much the idea will influence user activity, such as the user's willingness to become a payer and how it affects the conversion rate. If you expect a 100% boost in conversion rate, assign a value of 5. If you expect a 50% increase, assign a value of 4, and so on. Alternatively, you can use a measuring scale where 3 indicates a massive impact, 2 for high impact, 1 for medium impact, 0.5 for low impact, and 0.25 for minimal impact. The measuring scale is flexible and can be adjusted to suit your needs.
Confidence. How certain are you about the impact and reach scores you assigned to each idea? If you are completely confident, assign a score of 5. If you are not at all confident, assign a score of 1. Alternatively, you can use a percent score to indicate your confidence level, where 100% represents high confidence, 80% represents medium confidence, 50% represents low confidence, and so on.
Effort. Last but not least is the amount of time and resources required to implement a particular feature. For example, if an improvement requires three team members to work on it for one week, the effort score would be 3 person-weeks. If it needs one team member and three weeks of work, it would also have an effort score of 3 person-weeks. If you don't want to calculate an exact number, just do a mental tally and assign an Effort score of 5 if it takes more than 10 workdays, 4 if it takes more than 5 workdays, and 1 if it takes less than a day.

After assigning all scores, you need to calculate the final score.

RICE Score = (Reach x Impact x Confidence) ÷ Effort

For example, (5 Reach x 5 Impact x 1 Confidence) ÷ 4 Effort = ~ 6.

While doing RICE, it is essential not to get bogged down in complex scoring and calculations in order to be as accurate as possible. Scoring provides a solid foundation for rational discussions about which idea to prioritize and why.

P.I.E. Potential, Importance, and Ease.

Here you should answer the next questions:

How much improvements can be made on this part of your product as a result of a specific idea?
Is this part of your product important? How often do users interact with it?
Will it be challenging to implement the test?

To simplify the process, you can assign a score between 1 and 10 for each variable and calculate the average of the three scores. The result will give you an overall idea of the potential impact of the idea being tested.

While these methods may seem impressive in theory, it is crucial to assess their true value in practice. To effectively prioritize your A/B tests, it is important to consider your specific goals, resources, and product type. In my opinion, it is better to create a custom method tailored to your unique business processes, one that incorporates variables specific to your company. The scores of each idea should serve as a basis for discussion rather than a final judgment.

Regardless of the method used, proper prioritization of A/B test ideas is crucial to ensure that no one idea is given undue preference over others. Also prioritization should not be the responsibility of just one person, this should be a team activity that is performed regularly - for instance, once every two weeks - where team members can vote and share their opinions.

The backlog of ideas should be regularly reprioritized, as business priorities, strategies, and developer availability can all change. By staying on top of your ideas and regularly reevaluating them, you can ensure that you are always focusing on the best ideas and staying ahead of the curve.

Don't Break the Bank on SQL Queries: BigQuery On-Demand vs Flat-Rate prices. Which Saves You More? 💰😎

Olga R — Sun, 12 Mar 2023 17:52:00 +0000

BigQuery is a powerful database with numerous benefits, but like any tool, it has its concerns. First of all, it is important to be mindful of the amount of data you are going to process to avoid unexpected costs at the end of the month.

While this may not be a concern if your data is under 1 TB, you may break the bank if you have much more than this. Stay ahead of the game by familiarizing yourself with the various pricing models available in BigQuery, and unlock the full potential of it as our team did.

Discover the differences between Default and Flat-Rate pricing models 🧐

BigQuery offers two pricing models for running SQL queries: on-demand and flat-rate. You may read a little more about it here - link. By default, you will be on the on-demand pricing model when you start working with the database.

But what happens when you are dealing with massive amounts of data, say, 20 TB of it? And what if your SQL query is particularly complex?

This is where the on-demand pricing model can get pricey, as you will pay for the amount of data you process while running your query. The cost for processing 1 TB of data ranges from $5 to $9 depending on the location of data processing. To process 20 TB of data in "us" or "eu", you will have to pay $100 for just one query. However, with up to 2,000 concurrent slots available, your complex query will be finished very quickly.

But BigQuery also offers a Flat-rate pricing model where you pay for the time you have slots booked for your tasks, rather than the amount of data processed.

You can book anywhere from 100 to 10,100 slots, with prices ranging from $4 to $6 for booking 100 slots for 1 hour. The best part is that you pay for the minutes. That means that if you cancel the booking in a couple of minutes after creating it, you will only be charged $0.12 to $0.20.

So, let's go back to our earlier example of processing 20 TB of data. If you can complete your data query in less than an hour with 100 slots reserved, you will only end up paying less than $5 for the entire process! That is a whopping $95 in savings per SQL query, making the Flat-rate pricing model an attractive option for those looking to optimize their budget.

Unveiling the secret to our massive savings: Switching to a hybrid approach! 🤩

When we discovered the potential savings with the flat-rate pricing model, we explored our options. There are three subscription types to choose from, each with its own unique benefits:

Monthly flat-rate commitments starting at $2,000 per 100 slots per month
Annual flat-rate commitments starting at $1,700 per 100 slots per month
Flex slots, which offer short-term commitments starting at just $4 per 100 slots per minute.

After careful consideration, we opted for flex slots.

But when we need to process small-sized data sets quickly we still use on-demand. It is perfect when you do not want to wait.

And when we know we are going to be processing more than a few TB of data, we simply buy fixed slots, run the query, and cancel the reservation. Typically, I buy 200 slots for less than 30 minutes, and that is plenty of time to get the job done.

Overall, our hybrid approach has been a game-changer, helping us save money and optimize our data processing in a way that works best for us.

Unlock the power of Flex slots: Step-by-Step Guide 👩‍💻

Let me share with you the incredible cost savings our team has experienced by using both the on-demand and flat-rate pricing models for our data processing needs.

Using the on-demand model is simple - just log in, write your SQL query, and you are off to the races. But when it comes to the flat-rate pricing model, there are a few more steps involved.

First, you will need to choose the project you want to purchase slots for. It is important to make sure you have the proper permissions for this project, as you can buy slots with Project A and use them to process data from Project B.

Step 1.

In your Google Cloud panel go to BigQuery - Capacity management.

Step 2.

Then choose "Create commitment" and fill in the form.

While choosing the “Commitment duration” we choose “Flex” as we decided earlier. We do not need an Annual or Monthly reservation this time.

When it comes to buying slots for your SQL queries, it is important to consider the complexity of your query. If you are dealing with a lot of window functions, group by statements, and other complex operations, it may be worth investing in 300 or more slots. By purchasing more slots, you will be able to process your data faster.

Click 'NEXT', type ‘CONFIRM’ and click ‘Purchase’.

Step 3.

Congratulations on purchasing the capacity you need! Before diving into running your queries, it is important to reserve those slots for the specific project you will be working on. To do this, simply navigate to the "Create reservation" option and follow the prompts.

Fill in the fields.

Click on “Create assignment”.

Choose the project where you will write a query and click "Save".

Now you can run your query!

Step 4.

Great job on completing your request. Now it is time to clean up after yourself and delete everything you created, step by step in reverse order.

First, delete the assignment you created for the specific project. Then, cancel the reservation you made for the flat-rate capacity. Finally, head back to the "Slot commitments" page and delete your purchase altogether.

By taking the time to clean up after yourself, you will ensure that you are not wasting any unnecessary resources or incurring additional costs. Plus, it is always good to keep things tidy and organized, right?

The final view of your "Slot commitments" view should be empty.

My congratulations to you! Once you get the hang of it, using a hybrid approach for your data processing needs will become a breeze. And the best part? You will be saving a ton of money for your team!

Before you call it a day, there are two important things to keep in mind:

First, be sure to check that your "Slot commitments" page is empty after completing your request (if you use flex slots like me). Otherwise, your reservation will continue to take up space even when you do not need it. And you will have to pay for it.
Secondly, when you create an assignment for a specific project, all queries for that project will use the purchased flat-rate capacity. If your colleagues are also running queries for that project, they will be placed in a queue and you may experience some wait time.

At our team, Data Analysts, Data Engineers, and the QA team have all learned how to use this pricing model to optimize their spending and achieve maximum efficiency. So why not give it a try and see how much you can save?