Forem: Assert(QA)

Best practices for test automation

Leonardo Barba Galani — Mon, 04 May 2020 20:41:25 +0000

It's great to see so many peers getting better on what they do, learn new technologies, learn new frameworks, calling themselves QAOps, QaDev, all sort of titles to show that they might know how to code or can put a test framework and an end2end test up and running in no time.

The only little problem is that some of those people kind of don't know what they are doing, don't know WHY they are applying specific techniques or why they are using certain frameworks.

If you already got bored of this intro and want to close this tab, please consider at least the most important take away from this article: 'Test Automation is a development process, end of the story'.

Understand your current project, and it's Decisions.

You might have entered on the project after the big decisions where made, but this process shouldn't be less relevant. Try asking around about the context of the decisions and exercise a little bit of critical thinking.

"Should we be using BDD for this project? Why is there no unit test? Why there are only U.I. Tests? Why don't we do stress tests Etc."

Before writing any line of code, understand the patterns that are in place and code itself, so you don't repeat yourself.

And if its a new project, you should start thinking strategy with your team and defining scope, coverage, and other stuff to guarantee the quality of the software you will deliver.

Always do a Proof of Concept

Before making a huge setup, install a bunch of dependencies or spend time thinking about how you should organize your stuff, take a moment to realize if you know what you are doing.

Do you know the language? Do you know the framework? Do you know if it's possible to automate that particular thing that the P.O. asked you?

Well, there is no answer before you try those stuff your self and this is
the reason why you should do a proof of concept as the first thing.

info: https://en.wikipedia.org/wiki/Proof_of_concept

You should test your hypotheses with little/no dependency, so other people can also execute the piece of code you just did.

POC's can also apply for new features/validation on the test code that you are not sure how to do it.
Do you need to execute all the setup to "debug, fix and retry" every time?

It's also a good practice to talk to people and tell them your intends with the future POC that you are going to do and see if they want to check the results with you OR if they have another idea that you cant try out.
I don't see any reason for people to start coding away, especially a new test framework, without talking to people and showing them how you will do it.

Check your external dependencies

Have you ever stop and look at the dependencies of your code? Did you ever check the license of the packages you use? How many issues are open, and many commits and releases there have?

If you work at a startup that really doesn't care about anything and just want to put code into production, oh well, there is nothing we can do, right?

But if you somehow care about your work and the code you put in production, you might want to check the security breaches of the dependencies you use, if they have an apache/MIT license or GPL or if they are a fork of another library.

Invest in code isolation

Ever time you code some helper, util, or a shared method, think about code isolation.
Think about yourself if a few days in the future and how happy you will be not to have errors popping out of nowhere when you remove an import that you thought it was safe to remove.

Do you have hidden imports that searches and add your entire repo into memory just because you don't want to have a few more lines? Can you feel the code smell? A tricky one to fix if you don't start your project in the Right way. Even if you do things correctly in the beginning, you should always stay vigilant on every P.R.

Code isolation can be ~~easily~~ achieve if you understand what a code Domain / Namespace is.

+info: https://en.wikipedia.org/wiki/Domain-driven_design

A helper method should NOT know any implementation details of any other thing on your project.

Sometimes it's a bit hard to understand with plain text, right? Let me give you a simple pseudo-py-code example:

def login(user):
    field.email.set(user['Email'])
    field.password.set(user['Password'])

Can you see the problem with the code above?
The login method "knows" the implementation details of the user object. What would happen if somehow we change the interface of the user? Or maybe there are multiple ways to create a user, and one of those methods might create the user with just a "magic_link". Or maybe this user object has a cellphone number instead of email as identification. Can you see where I'm going?

The good refactor might look something like this, removing the knowledge of the user interface:

def login(email, password):
    field.email.set(email)
    field.password.set(password)

Put memory allocation into consideration

"Leonardo, I think you are living in the early 2000s when people didn't have decent computers with decent processing power".

You can tell me that your company gave you a brand new MacBook with tons of memory BUT, are you going to run the CI process inside your MacBook? I guess not.

Most of your tests will run in parallel inside a CI cluster that can be expansive. Don't count on having workers with a bunch of memory and processing power because the price to maintains those machines/instances are high.

"We have a nice machine for the CI man, stop being a jerk."

You know that if you have an excellent machine, it will probably going to use docker to run your CI software, and perhaps your test will be run on a container, in other words, docker inside docker. It's okay, but if your machine is not that good, buckle up because the slow and flaky tests are coming in your way.

To avoid this, make your code isolated. Only put in memory modules/libs/packages that you going to use. Avoid using external libs for trivial things.

I might be pushing too hard on this one, but stay dependency-free as much as you can.

Ex: In ruby, there is a framework called Airborne for API tests, which is a wrap of Faraday/http_request/assert.
You can use Airbone to do your API test, but you don't need it. You have all the tools you need to test API with pure ruby.

Its just a matter of understanding the language, what it comes with, and how to make most on you got.

Code smell and Language Style

Despite your language of choice, understand that every time you touch a repository, you are singing a blood pact that you and those after you will follow the language/repository guidelines.

Every language has its ways to define vars, constants, classes, and methods. Those principles will help you to avoid most of the things that lead to code smell, so please, don't turn off your Rubocop, pylint, eslint, or whatever you have in place to check your code.

Don't think that your case is unique and disable the rules. Treat it as Technical Debit and put it on your backlog.

Once you understand the rules, your new code will pass flawlessly on those checks, and you will see the benefits when shipping or debugging your code.

Some companies also define their patterns. You should apply it to your work, but IF by doing so, you feel that you are deriving away from the industry standards, speak up.

Avoid If's as much as possible.

+info: https://francescocirillo.com/pages/anti-if-campaign

One of the most problematic behaviors that I see young test developers doing is the tsunami of if's and try expect on their code.

The conditional operator "IF" is a robust tool that you will use a lot, but bear in mind that it increases the complexity of the code exponentially.
A test script or helper should be clear and have a single purpose.

info: https://en.wikipedia.org/wiki/SOLID

Let's use the same example from before:

def login(user):
    if user['MagicLink']:
        do.login(user['MagicLink'])
        return
    elif user['Email]:
        field.email.set(user['Email'])
    elif user['Cellphone']:
        field.cellphone.set(user['CellPhone'])

    field.email.set(user['Password'])
    do.login

You may be laughing about this and saying, "I would never do such a thing," but the number of times I saw similar mistakes like that on forums and P.R.'s that I had to review is vast!

Let's refactor so you can understand what I'm saying:


def login_magic_link(magic_link):
    visit magic_link
    do.login

def login_with_email(email, password):
    field.email.set(email)
    field.password.set(password)
    do.login

def login_with_cellphone(cellphone, password):
    field.cellphone.set(cellphone)
    field.password.set(password)
    do.login

A code that is easy to read is also easy to debug and to extend.

Read code, docs and unit test

If you don't have the habit of reading other people's code or understanding what the unit tests are testing, You might want to put this process on your daily tasks.

Sometimes people will not have time to help you out on a particular task, or maybe the answer on how to instantiate a specific object is already in place on a unit test.

Make a habit of reading the implementation details before asking stuff on the internet.

CODE! CODE! CODE!

Don't code just test scripts. Code a custom tool or use a cocking recipe to create your twitter or something. Maybe a code challenge might be your thing.

Practice the craft as much as possible because "test automation" is on its way to not be a job title anymore.

Final Thoughts

I left out some topics like "pair programming" and "P.R. revision" because you can't use those technics and practices if you don't know what you are doing :)

I'm open to critics and suggestions... are you interested in a possible part 2? let me know!

Before starting automation, you should change your mindset

Wellington Avelino dos Santos — Sun, 02 Feb 2020 15:23:19 +0000

Change culture, not process**

I've decided to write this post because this topic has surfaced time and again in many projects that I've worked on during my time in software development. It is a big misconception that automation is the root cause of issues in the QA process. However, it is my belief that in order to refine a process, the surrounding culture must be a supportive environment for it to grow and thrive.

In this article, I have detailed a few initiatives that can be followed and adapted according to a team's maturity level and autonomy.

Common complains:

"We just need automation to make things better," "Automation is the solution for us.", "Just create more UI tests".

Necessary QA skills

So let's start with some self-reflection:

Is my team helping when needed?
Is quality a shared responsibility?
Is testing an activity?
If the QA is out sick, does the team know what to do?

If the answer to any of these questions is NO, then probably the culture isn't right

Based on that, I will outline some points that highlight the importance of involving each team member in quality activities.

Culture

If you have previously encountered this conversation, you probably also have heard people say that this "broken process" is caused by the company's culture; which is fine. But why is there no initiative to change it? Or why have any previous attempts to address any issues failed?

Sometimes, it is easier to keep going without trying to improve.

Considering the questions from the beginning of this article. In those questions I highlighted some common problems that might be causing issues in your company/project:

QA as a gatekeeper of quality.
The team doesn't understand what the QA is doing in the sprint (I know, it's almost 2020!).
The team recognizes the QA as a problem-maker and does not see them as someone who is part of the process that provides solutions too.
A lack of understanding that quality assurance is a phase within the development cycle.

Mindset Change

Changing people's mindset might be the biggest challenge.
You have to learn how to handle different situations and how to approach people with diverse backgrounds and characteristics. I read this book that I probably already recommended to you if we spoke face-to-face, called "Agile testing" By Lisa Crispin and Janet Gregory. Published in 2008, I would consider this to be a "bible" for the testers of today. It surprises me that there are a lot of professionals that have never heard about this book, or even know what Agile testing means.

This book will open a window of diverse possibilities in the agile process.

Below I have outlined a few small changes that you could build upon.

Start asking yourself, have I shown to my team what quality is?

Sometimes your team or manager doesn't understand what is involved in being a QA, and what it is that you actually contribute to each sprint. It's our responsibility, as QAs, to clearly outline and define what we do. To address this you could organize small sessions to discuss what you have been doing and how you can help the team to achieve the Holy Grail of quality.

Are your tests done at the end of the process?

It is ubiquitous when tests are still a phase in the process: in the traditional mindset, tests only happen at the end of the cycle. You shouldn't be testing at the end, and you should bring quality checks as early as possible; by doing this your goals will change as well. Instead of playing catch-up with bugs, you will prevent them from even happening.

Some examples could be:

Suggested activities

Shared responsibility

Shared responsibility should be what you're aiming for since you shouldn't be the only one that cares for the quality. This can sometimes be a challenge as it may require an initial shift in thinking and mindset from you and your team. Once everyone is on the same page, shared responsibility can start to become a reality.

Here's is a list of suggestions that are based on my previous experiments/experiences:

Shared responsibility doesn't mean that you will lose your job. It means that you'll have more time to explore things that matter instead of being stuck performing boring test stuff.
Your team has to see you as a critical part of the process, in other words, you should be the one that they will seek for testing knowledge.
Organize exploratory sessions with the team to help them create empathy and learn more about how the product works and what it looks like.
In general, we tend to work on small pieces of the product, and we don't have a vision of the big picture, which isn't a good thing if we want to put quality first.

You're no longer trying to find bugs, you're trying to prevent them from happening.

Yes, it sounds a bit weird… but you get used to it ;)

We're no longer the last and only person concerned about quality, so why should we still catch bugs instead of preventing them?

Maybe it doesn't look like our job, but during refining meetings, it is important to try and make sure that everyone is on the same page. From my experience, back-and-forth conversations happen during development when there is a misunderstanding of what the end result of a feature should be.

A few simples questions could help, such as:

Is everyone on the same page?
Do we have any concerns regarding how the feature should work?
Are we comfortable with the acceptance criteria?

A test engineer asking those questions could sound a bit odd right? But as I said before, shared responsibility is not only tied to quality; it's everything!

But Wellington, where do I start?

Start with one person. You need to get used to this way of working before spreading it to the rest of the team. The benefit of doing it this way is that you'll develop proficiency, and it will be much more likely that the person will be your ally in selling it to the rest of the team.
Be proactive and test early in the process. Check out the branches that your team-mates are working on and perform some validations.
Don't wait until the developers put the story in "Ready for QA" lane (or whatever kind of the process you're using).
Perform exploratory testing instead of tedious manual scripting testing.

You should keep in mind that there's no silver bullet, and you should adapt yourself to different contexts and make small changes incrementally. You soon will realize the effectiveness of doing things step-by-step rather than doing everything all at once.

To do it right, you should have allies, followers, believers, not only executors.

Automation

At this point, you should be wondering why automation isn't the first topic of this article. Let me explain the reason.

Automation is the tip of the iceberg; if you don't have a good culture or even don't have people to support you, it will be hard to get things done.

You should choose the automation framework carefully and do your research to understand what suits you and your team.

What should you consider?

If the framework is continuously maintained and updated
If it suits your needs
If you're working with backend, it doesn't make sense to create a separate test project, it will only create a wall between you and the engineers.
Consider using the same programming language that the team is using to create the tests; it will remove the knowledge gap and reduce maintenance cost.
On mobile, you have different options using BlackBox frameworks such as Appium, Robot, or even Whitebox frameworks such as Espresso, EarlGrey, XCTest. Choose carefully, and don't get too excited or you will overload the UI layer with tons of tests; you should split those validations. There could be an entire post dedicated to this topic.
If you're thinking of using any BDD framework to write your specifications, validate it first with your team. It has a HUGE benefit, but only if the entire team is sold with the idea and helps to get this done collaboratively.
You should be able to write useful unit tests and help the team to develop it if necessary. Remember quality over coverage, learn about mutation tests, and explore it with your team; you might be surprised in how many "assert true" you can find.

Continuous integration

I might be saying the basics here. Don't get me wrong, but automation has no value if it is not running on CI under a good strategy. It should run fast and provide feedback in each stage of the integration. Explore parallel execution, consider splitting your suites into smaller ones so you can have more control over each test execution.

Reporting

You should also find a way to give visibility to what you're doing. The right direction for this is to use your automation framework to provide that for you. Pretty much all of the automation tools have a way to generate reports. Most of them are following the JUnit pattern, which's easy to parse and render it on a big monitor using a dashboard.

Takeaways

People over the automation process.
Start small, get support, and go bigger.
Quality isn't only automation.
Instead of choosing tools that make you work, choose tools that work for you.
It's your job to make sure that everyone knows how important you are; quality assurance has always been a role that people don't fully understand.

I hope that this post helps you to make some improvements in your team or at least acts as a catalyst that sparks a change.

Good reference books.

Agile testing & More agile testing
Specification by example
Continuous delivery