Forem: Mario Casciaro

Implementing smoke testing in production and staging. DIY vs Ad-Hoc Tool

Mario Casciaro — Thu, 27 Feb 2020 11:30:28 +0000

What's smoke testing?

Smoke testing allows us to quickly asses the status of an application by running a set of end-to-end tests targeted at checking the most important (or the most significant) user flows.
It should be run just after a fresh deploy and ideally at regular intervals after that.

Smoke testing is different from a full regression testing in mainly two aspects:

The coverage of smoke testing is wide and shallow. Its purpose is to roughly test as much functionality as possible in a short time. On the other hand, regression testing is meant to be as thorough as possible to make sure that any existing functionality was not negatively affected by the new changes introduced by a new commit or release.
Smoke testing should be fast compared to regression testing, as its main purpose is to quickly assess the main user flows within an application.

For this two reasons, smoke testing are well suited to be run continuously at regular intervals to check the status of an application over time. In fact, running tests just after a fresh deploy will only validate the application while in that particular state, just after a new restart. This is why, running a suite of smoke tests at regular intervals will make sure that the application behaves as it should while in different states at different moments in time.

Now, let's see what are some frequently asked questions about smoke testing, before moving onto more practical matters.

Smoke testing FAQ

Should I run my smoke tests on production or on staging?

Ideally on both. Smoke testing is useful on staging because it allows you to be notified about important problems faster. A thorough regression testing suite usually takes a few minutes to run, sometimes even 30 minutes or an hour or even more, depending on the size of the application. This means that after a new commit or release we may have to wait a long time before knowing if our code broke something major. With smoke testing, instead, we have an almost immediate feedback on the status of a new commit or release. Usually, on staging or testing environments a smoke test is followed by a more detailed regression test.

How long should a smoke test suite take?

For a smoke test to be effective it should be fast. I'd say that a reasonable upper limit is 5 minutes for a small application, 10 for a medium application and 15 for a large application. But, of course the shortest the better.

How can I make sure that my smoke test takes as little time as possible?

There are actually a couple of tricks you can use:

Limit the use of static wait instructions, such as waiting a predefined time (e.g. 10 seconds) for a page to load. Use smart wait instructions instead, such as wait for an element to appear. Even better, use a tool that can do that for you under the hood (implicit waiting).
Run the tests in parallel. Make sure that there are no relationships between the various tests in a suite, so that they can be run in parallel. This can usually make a big difference in the total running time.
Remember to only test the most significant parts of an application. Don't go too much into the details and make sure to test features that can indicate, with a certain degree of confidence, if other parts of the application are broken too.

When should I run my smoke tests?

Smoke tests should be run at least afte revery new deploy to staging or production. Even better, you can run your smoke tests after each commit since they won't take long to run anyway. Finally, you can also use your smoke tests to check the status of your production or staging environment at regular intervals.

Is there a preferred framework for running smoke tests?

Generally speaking no. Any end-to-end testing framework should just do the job. However, as we will see later, you can also go no-framework and use a pre-packaged cloud testing solution to simplify and reduce the maintenance of your smoke testing solution.

Can smoke testing replace regression testing?

No. Smoke testing won't be able to cover most of the functionality of an application and its purpose is different than that of regression testing. For a serious product, I recommend having both. That said, if you currently don't have neither of the two, then having at least a smoke test suite is a giant leap forward because it can potentially catch the a good chunk of critical issues before they become a problem.

Option 1. Building a smoke testing solution from scratch

Now, let's see what it takes to build a smoke testing solution from scratch. The core requirement that we have, is that the smoke test suite must be able to run after a new deploy and ideally run continuously at regular interval.

The first requirement of our smoke testing solution can be actually simple if you already have an articulated build process or a CI infrastructure in place. In this case, in fact, you could just add another step in your build script to run your smoke tests.

Choose a testing framework

Next, choose a good end-to-end testing framework. If you are using JavaScript for your tests, then you have many good options:

Each of these frameworks has its pros and cons. For example Cypress and Taiko only support Chromium-based browsers. While Selenium is notoriously the most difficult to deal with. At the same time, Cypress, Taiko and Puppeteer have a very good tester experience.

Choose a test runner (optional)

Optionally, if your testing framework already doesn't integrate one, you have to choose a test runner (make sure it's compatible with the testing framework you chose). In this case too, there are many options to choose from:

Get coding

Next, code your tests manually or use a test recorder such as Selenium IDE. Note, however, that if you use Selenium IDE, your framework choices may be limited.

After you have your tests ready, make sure they are checked-in in a repository (you can decide if in the repository of the main application or somewhere else) and make sure they are available to the CI when it runs.

Finally, when a test fails, most CIs will let you know with a notification (an email, Slack message, SMS, etc.), so there is nothing else to add on that compartment.

Choose a CI platform (if you already don't have one)

It's probably redundant to say that if you don't have a CI already in place, you probably need one to run your smoke tests. Your options are many in this occasion too:

Running a test suite at regular intervals

Now, if you wanted to run your smoke tests at regular intervals, that would be a different kind of beast altogether. In this case, in fact, your CI infrastructure is mostly useless as you would need an always-on infrastructure to be able to continuously run your tests. The options in these case are mainly two:

Have a dedicated machine in the cloud for running your tests. Setup a cron job to start the smoke test suite at your desired interval.
Setup your tests to be run in a serverless environment, such as AWS Lambda, Azure functions, Google Cloud Functions. In this case the serverless function can be setup to run at specified intervals natively, without the need of any cronjob.

The second options is probably easier to maintain in the long run, but it requires some prior knowledge of serverless programming and the particular cloud platform in use. In both situations, however, you would have to implement ways to keep the test suites up to date on the remote machine or on the serverless platform, plus you would need to implement a solution for getting notified of failing tests. You can, for example, send an email to yourself or to you entire team, or a send a Slack notification on your preferred channel.

Option 2. Using a web testing tool built for the purpose

As we have seen in the previous section, implementing a web smoke testing solution from scratch can be a quite laborious matter. If you already have a CI infrastructure in place, then you may get out of it a little bit easier, but still, if you want to run a test suite continuously, then it's almost impossible to avoid to implement and maintain extra tooling. All those homemade solutions will likely become technical debt, as over time the focus shifts to implementing tests rather than maintaining the infrastructure on which they run.

Based on those assumptions, choosing a tool specifically built for the purpose (without reinventing one) is often the most sensible option, which will repay itself many times even in the short run. To give you an idea of what those tools are capable of, take a look at this list:

With a dedicated tool, usually you just need to create tests and run them. There is no need to choose frameworks, CI platforms, test runners or test recorders. All of them are integrated into one single product and completely hidden to you.
You can easily run a test suite on-demand, using webhooks.
You can schedule a test suite to run at regular intervals using a built-in scheduler.
You can have a full report of the last test run as well as stats about the previous runs.
All kinds of notifications are taken care of out-of-the-box.

In essence, the purpose of those tools is shifting the efforts of testing from setting up and maintaining an infrastructure to just actually writing and maintaining tests. The only disadvantage that I can think of, is that most of these tools are not well suited to satisfy very custom requirements, in which case you may need to go back and do it yourself. But to be honest, this is a common rule in software development.

There are a few tools out there that match the description we've given above. Each tool has its pros and cons, some are cheap other more expensive, but in general they all can improve dramatically your smoke testing experience.

One of those tools is Frontend Robot.

Frontend Robot is a fully cloud-based tool that simplifies the whole testing experience. It provides Smart Triggers to trigger the execution of a test suite on-demand, and the ability to run tests at regular intervals with its Run Scheduler. But this only scratches the surface, take a look the our homepage for more information.

Conclusions

Smoke testing ia a crucial part of a software development pipeline, however is often overlooked and misunderstood. Like any other type of tests, setting up and maintaining a custom built solution is often tedious and expensive, with results that most of the time are sub-optimal. Those are exactly the pain points that modern testing tools try to solve. Using a dedicated tools for implementing your smoke testing solution is often the best choice in terms of adoption time, costs and overall developer/tester happiness. The only exception would be if your requirements are so custom that a pre-packaged solution is not an option, but this is usually quite rare occurrence.

PS: For an overview of more testing tools with a good tester experience, refer to the following article: 5 web testing tools with the best Tester Experience (TX).

As always, if you have any questions or feedback you can send me an email at mario@frontendrobot.com.

5 web testing tools with the best Tester Experience (TX)

Mario Casciaro — Thu, 06 Feb 2020 11:36:29 +0000

Disclaimer: I work at Frontend Robot, so I'm a little biased here. Where I express personal considerations I will try to make it as clear as possible that they are my own views.

The best thing that happened to the software industry

If we take a look at the evolution of web development in the last few years, I can only think to one aspect that really changed the face of the industry and that's Developer Experience (DX). Based on the slogan "developers are people too", the DX movement is bringing the ergonomics and the concepts of User Experience (UX) into the developer's world.

Developer experience spans tools, APIs, SDKs, documentation and workflows; essentially, most aspects of a developer's job. In general, if a developer uses or interacts with a product of some kind, then, DX is involved.

Personally, the first time that I became DX-aware, was in 2015 when I started using React Hot Loader after watching a talk by Dan Abramov and his subsequent interview with Kent C. Dodds where they explicitly talk about "Developer Experience and Tools". For those not familiar with the tool, React Hot Loader allows developers to see changes made to a React application live, without the need to reload the page. It works like magic. React Hot Loader became for me the perfect example of what a good Developer Experience means; less frustration, more productivity and putting back the word fun into software development.

Tester Experience (TX), encompasses the same ergonomics principles that characterize UX and DX, but applied to software testers. Luckily, in the last few years, the industry has made giant steps in the area of TX as well. Similarly to UX and DX, Tester Experience is not just about the ease of use of a product, but also about the emotional impact that it has on the person.

From open-source tools to cloud-based services, we have witnessed a surge of web testing tools aimed at radically improving the testing experience, making it simpler, more rewarding and even fun.

The one that follows is a list of the top 5 testing tools born in the last few years, which shine for providing a great Tester Experience. Eeach one has its pros and cons, so just choose the one that works best for you.

1. Frontend Robot

Frontend Robot is a cloud platform which will get you up and running with automated testing in just a couple of minutes (literally). Frontend Robot was born two years ago from my frustration with setting up and writing automated tests for the web. Just to write a simple test, I needed to choose a framework, set it up, and have a build machine to run it on. On top of that, writing tests was a laborious matter with a constant back and forth between the code editor and the browser to check the results and create element selectors. That wasn't fun at all, I just wanted to create and run tests!

Now, with Frontend Robot it's just a matter of opening the browser, creating a test, and adding actions and assertions while there is a live test session, which immediately shows the results on screen. Paste the smart hook URL in your deployment script and have the test running at every deploy. It's as simple as that.

Frontend Robot also provides a nice set of goodies that will help with writing tests and debugging, such as:

DOM Snapshots
Smart CSS selector picking
Time travel
Variables

Frontend Robot was designed from the ground up to provide the best possible Tester Experience, studied in every detail to remove any obstacles or complications to make writing tests less frustrating and fun again. If you are mainly after simplicity and an enjoyable testing experience, then Frontend Robot is the right tool for you.

2. Ghost Inspector

Born in 2014, Ghost Inspector was created by Justin Klemm from similar motivations to those that pushed me to create Frontend Robot. I definitely consider Ghost Inspector one of the main source of inspirations for Frontend Robot. If you are curious about the story behind Ghost Inspector, you can take a look at Justin's blogpost Why Did We Build Ghost Inspector? where he highlights the main pain points that Ghost Inspector tries to solve.

I consider Ghost Inspector one of the most seasoned and stable tools in this list. If you look at their documentation, it's clear that they worked closely with their customers over the years to build a tool that can cover almost every major need in test automation. From screenshots comparison to file uploads and testing emails, they have almost everything you need to write a complete test. The only drawback in my opinion is a UI that looks outdated and a Tester Experience which can be improved.

3. Cypress

While the other tools in this list bet on codeless test authoring, Cypress focused mainly on reinventing the traditional code-based testing framework. Cypress built a test runner from scratch, allowing to provide some interesting features out-of-the-box, such as:

Spies, stubs and clocks
Network traffic control
Time travel and DOM snapshots
Real-time reload

Being code-based, plus having a custom runner built on Chromium, allow developers and testers to write very advanced and complex test routines, which would be impossible to create with the other tools in this list.

Despite being code-based, Cypress strives to provide the best Developers Experience in the industry. In fact, their API is crafted for maximum efficiency and their documentation very detailed and easy to read. They even have a dedicated Developer Experience team!

The Cypress team has big plans for the future, however being still a young product, it has some limitations. For example, as of today it only support Chromium-based browsers. Plus, it's not a 100% cloud-based solution, something that can be a good news for some but a bad news for others. In fact, Cypress has an open-source runner that everybody can use for free, and a paid dashboard service to collect and organize test results. However, it's still responsibility of the developers/tester to setup the tests to run in a separate CI (Continuous Integration) provider.

4. Testim

Testim launched in 2014 and it's currently backed by $20M in funding, which makes it one the most well funded products in this list. Testim is a hybrid tool which allows to both record tests using their Chrome extension or using code written with your favourite IDE. Testim is marketed as an AI-powered testing tool, featuring self-healing selectors which should require less maintenance. But to be honest with you, besides this feature, I'm not sure how much other AI is in the rest of the product. Said that, Testim looks like a solid, featureful product, which strives to provide a simplified and more enjoyable testing experience.

One interesting feature is the ability to integrate with third party browser grids, such as Saucelabs and Browserstack allowing you to run your tests on a variety of desktop and mobile browsers.

For some, the only real negative side of Testim may be its price (at the time of writing starting at $450/month) which will make it off limits for many individual developers or small companies.

5. Mabl

Mabl is the most well funded tool in this list with a staggering $30M invested in it. Founded by two ex-Google employees, Mabl defines itself as a "unified DevTestOps platform". In other words, Mabl is a cloud-based codeless tool that simplifies the entire testing experience, from setting up tests to running them in their Google Cloud based grid. Their test recorder, called the mabl trainer, is a Chrome plugin which allows to transform browser actions into test steps.

Mabl is the only product in this list to natively support cross-browser testing on their own cloud, with Chrome, Firefox, IE, and Safari being the available browsers. Like Testim, also Mabl has some AI embedded in it, allowing self-healing tests which adapt to UI changes, resulting in more robust tests that break less often.

Mabl doesn't provide public pricing information, so my guess is that it's mainly targeted at medium to big businesses.

Conclusions

The products we've just analyzed are, at the end of the day, very similar in their goal. They all aim to improve the Tester Experience making the tester's job simpler, more productive and more enjoyable. However each one of them is targeted at a specific group of testers/developers. Frontend Robot and Ghost Inspector are the products to choose if you don't want any fuss or extra complications with writing your tests. Cypress can be a good choice if you still want to write code for your tests and need more powerful features and control. Testim and Mabl are mainly targeted at larger companies, who need more support and enterprise features, at the cost of a more expensive subscription.

There are many other products that I've left out from this list, and that doesn't mean they are less good. However, there is an important player that I've left out just because it's still very young and in active development, and this is the new Selenium IDE, which recently took the place of the old, Selenium IDE. The team is working on a new Electron-based editor, which promises to bring some interesting features.

As always, if you have any questions or feedback you can send me an email at mario@frontendrobot.com.

Run your end-to-end tests on production like a boss

Mario Casciaro — Fri, 26 Jul 2019 10:38:10 +0000

End-to-end tests are usually run on a testing or staging environment, and for good reasons. But there are instances when running your tests on production is not that crazy after all.

Why not on production?

There are good reasons for not running your tests on a production environment.

First of all, when your code is on production it's usually already too late for testing 😓. You want to run your tests so that your customers don't get a broken product, so usually you should test your code before it arrives into your customers' hands.

Secondly, you don't want to mess with production without first messing with staging. This is really a consequence of the point made above. If you haven't run your tests on a staging environment first and your tests break things up, they'll break production and that's - to put it mildly - undesirable. Tests are meant to find bugs, and bugs are unpredictable, you can never be sure of their extent. You may end up tearing down your production environment or even destroying data, who knows. On the positive side, it's better that your tests find the problem instead of your customers, at least you get notified 😝.

Another reason for not running your tests on production is the data noise. Imagine running you tests multiple times a day for years. You will end up having a lot of noise in your production data, which will make it difficult to extract meaningful information from it. And data nowadays is a precious resource. On a testing/staging environment you can just refresh the database every so often, if you ever need to, to get rid of all the data created by tests. Clearly, this is not so simple on production.

So why running on production after all?

Even if running your tests on production may not seem a good idea at first, it has some important advantages. Of course, as we will see later, you shouldn't run your tests on production without having a proper plan in place to mitigate the negative aspects we've just talked about. That said, let's see why instead you should run some tests on production.

Well, first of all, your staging environment is not what your customers use. A staging environment should replicate as closely as possible the production setup, however, your staging environment will never be exactly the same as your production environment. At the very least you will have different URLs, but you can also have different API keys for external services or variables that enable/disable things when on production rather than on staging. These are your dark spots in your QA strategy, and can only be removed by testing your production environment as well.

Also, the good health of your application doesn't come only from its code, but from a combination of:

The integrity of the data it uses.
The status of any external system (e.g. databases, storage services, etc.) it depends on.
The status of the runtime environment on which it runs (e.g. the server, the PaaS, etc.).

All the above factors are necessarily different between your staging and your production environments. If your data becomes corrupt on production because of a bug you missed, your application may become unstable. If your storage provider has a hiccup, your user won't be able to upload their files. If you have wrongly setup some file permissions on your production server only, you may not be able to run some parts of your application or access some of the files it needs.

After all, end-to-end testing it's also a type of integration testing so you will be able to catch all the above issues - hopefully before your customers will notice them - with a properly designed test plan running on production.

How to properly run your tests on production

So we have assessed that running some end-to-end tests on production is indeed beneficial, but also that there are some caveats we should take into account. So, below is a list of good practices to consider when running some end-to-end tests on production.

Don't rely just on "production" for your tests

This is probably the most important aspect. You should at least run the same set of tests on your application before it reaches production, on a staging or testing environment. This will make sure that your tests won't break anything on production, but it will also catch any major bug before your customers do.

In an ideal setup, we would have a three-tier testing plan:

Run a full regression testing suite on staging before the code goes to production.
Run a smaller set of tests (smoke test) on staging after each commit and deploy
Run an even smaller set of tests on production at regular time intervals

Test less, test often

What we really want to do here is to monitor the production environment rather than just testing it. This means that there should be a very limited number of highly meaningful tests, running continuously at short time intervals. This will ensure that your application is actually running and its major components work as they should.

The interval at which the tests should run may vary and it depends on how long the tests take to run. But a good value falls between 15-30 minutes.

Cleanup after your tests

Your production data is important. You can use it to improve your product, your customer experience and nowadays even to build powerful Machine Learning models. So it's important not to have unnecessary noise created by our tests.

A solution is to remove all the data that we have created during our tests. There are various strategies that we can use - and we'll have a separate blog post to talk about them in more detail - but probably the easiest approach is to use the same test automation to perform the cleanup.

Let's call it frontend monitoring

The practice of regularly checking that a website or web application is running as it should is also called monitoring.

At its most low-level implementation we have uptime monitoring, which simply makes sure that our services are up and running. This is usually done by checking the response to a call to a HTTP endpoint. There are many providers out there to chose from.

If we decide to run our end-to-end tests on production at regular intervals, we are essentially implementing some frontend monitoring (some call it user flow monitoring).

The scope and purpose of the two techniques is different but it overlaps in many points. Both are intended to make sure that a production application is running as it should. Uptime monitoring usually runs very basic checks that require instants to run, and that's why the monitoring interval is usually very short, down to 1 minute in many instances. The checks (or tests) performed for frontend monitoring on the other hand require longer to run, but are also more exhaustive as they will take into account also the User Interface subsystem and its integration with the backend.

Conclusions

The words test and production rarely appear on the same sentence and that's due to very good reasons. However we have seen that with a proper testing strategy, we can safely implement a frontend monitoring solution on production.

If you are interested in setting up a frontend monitoring solution, Frontend Robot has a test scheduler specifically designed for this use case.