Forem: Everton Schneider

Book: How To Lie with Statistics - Bias

Everton Schneider — Tue, 02 Apr 2024 02:06:13 +0000

The book How To Lie with Statistics, by Darrel Huff, has great views on how to analyze data. After reading this book, good questions arise from any discussion or affirmation that you may hear from friends, news, articles, and so many other sources.

"A great introduction to the use of statistics, and a great refresher for anyone who's already well versed in it." (Bill Gates)

The book contains 10 chapters and I will write down an article for each one of them. It must avoid suppressing the richness of the content found in the book.

The book chapters are:

The Sample with Built-in Bias
The Well-Chosen Average
The Little Figures That Are Not There
Much Ado about Practically Nothing
The Gee-Whiz Graph
One-Dimensional Picture
The Semiattached Figure
Post Hoc Rides Again
How to Statisticulate
How to Talk Back to a Statistic

Below is the article for the first chapter.

1. The Sample with Built-in Bias

The book begins by saying that in the New York Magazine once there was a comment saying that the AVERAGE of a man who studied at Yale, in the class of '24, "makes $25,111 a year".

What exactly does it mean?

If you send your boy to Yale, you won't have to work in your old age, and neither will he
This number is too precise

This kind of information is usually taken from surveys. Some people may say they get more than they do because they're vain, some people may say they make less, afraid to pay more taxes, and some people may not even know exactly how much they make every year, since they could have investments and other types of income.

People that has answered the survey are a sample of all students from Yale's class of '24, therefore, there source that can produce $25,111 as the "average income" of some men whose actual average may well be nearer half of that amount.

It is sad truth that conclusions from such samples, biased or too small or both, lie behind much of what we read or think we know.

How to be so sure it was taken from a sample?

What's in the magazine about the "class of '24" was published about 25 years later. Many of the addresses (or contacts) from the "class of '24" would not be reached 25 years later. Those whose addresses are known, could not respond.

So we find that the income figure is based on a sample composed of all class members whose addresses are known and who replied to the questionnaire.

Who are the lost sheep down in Yale rolls as "address unknown"? Are they big-income earners? (usually, the address of the rich are easy to find). It is a good guess that the lost names are those of the men who, 25 years or so after becoming Yale Bachelors of Arts, have not fulfilled any shining promise, part of the low-paid market.

The Psychiatrist

A psychiatrist reported once that practically everybody is neurotic.

Who has the psychiatrist been observing? It turns out that he has reached this edifying conclusion from studying his patients, who are a long way from being a sample of the population. If a man were normal, our psychiatrist would never meet him.

The Literary Digest's famed fiasco (Elections 1936)

The ten million telephone and Digest's subscribers who assured the editors of the doomed magazine that it would be Landon 370, Roosevelt 161 came from the list that had accurately predicted the 1932 election. If the list has been tested before, how could there be a bias in it?

There was. People who could afford telephones and magazine subscribers in 1936 were not a cross-section of voters.

The basic sample is the kind called "random", however, it could select all the answers from the same city or neighbourhood.

As seen, the tendency of the sample to be biased in this consistent direction can rig it automatically.

Book: Eat That Frog: 21 Ways to Stop Procrastinating

Everton Schneider — Sun, 17 Dec 2023 02:44:38 +0000

The book Eat that Frog! by Brian Tracy lists greatly, a pack of "rules" that can help people to stop procrastinating.
Of course, it is never that easy, but some techniques can always help.

This article has all the 21 rules in a quick overview.

1. Set the table

Before anything, decide what you need or must do. Your purpose. Write it down, set a deadline, make a plan and do it.
Yes, write it down.

2. Plan Every Day in Advance

Planning your day should take 10 to 12 minutes, but it is said to save you up to 2 hours in wasted time in your future day.

3. Apply the 80/20 Rule to Everything

Focus on the top 20% most important work you have to do. They will represent 80% of what must be delivered.

If you have a list of ten items to do, two of those items will turn out to be worth much more than the other eight items put together.

4. Consider the consequences

You will never have enough time to do everything you need to do.
People need to have long-term plans and be aware of what can be left aside that will not become a problem to achieve the long-term goal.

5. Practice Proactive Procrastination

The fact is that you cannot do everything that you have to do. You have to procrastinate on something.

So, do the worst first! The biggest and most complicated. Eat the big frost first.

Decide to outsource, delegate, and eliminate those activities that don't make much of a contribution to your life.

Learn to say no!, and say it politely.

To do something new, you will need to stop doing something old.

6. Use the ABCDE Method Continually

Make a list of everything you need to do for your next day. Write it down on a paper.

Beside each item, assign a letter A, B, C, D or E in an order of importance (A: most important; E less important).

Never, ever do a task marked with B, before a task marked as A.

Tasks marked as D can probably be delegated. Tasks marked as E can probably be eliminated.

7. Focus on Key Result Areas

A key result area is something that, if you don't do, does not get done.

Discuss it with your boss, understand what is your purpose for the company or team, and work on it.

Give yourself a grade. Search areas where you are doing a good job and areas that you must improve.

You may be only one critical skill away from top performance at your job.

Ask yourself:

What one skill, if I develop and do it in an excellent fashion, would have the greatest positive impact on my career?

Decide on doing these analyses regularly for the rest of your career. Never stop improving. This decision alone can change your life.

8. Apply the Law of Three

In thirty seconds, write down your three most important goals in life right now.

In 80% or more of the cases, people will have the three first things in common:

a financial and career goal
family or relationship goal
heal or fitness goal

Rule: it is the quality of time at work that counts and the quantity of time at home that matters.

Don't waste time. Time that you waste at work often has to be taken away from your family.

And, the more time you spend face-to-face with the people you love, the happier you will be.

9. Prepare Thoroughly Before You Begin

Create a comfortable workspace, be it at home or work.
Don't expect perfection the first time, or even the first few times.

You miss 100% of the shots you don't take

Feel effective, efficient, and ready to get going each time you sit down to work.

10. Take it One Oil Barrel at a Time

By the yard it's hard; but inch by inch, anything is a cinch!

You need to get your mind off the big task in front of you and focus on a single action that you can take.

One of the best ways to eat a big frog is to take it one bite at a time.

11. Upgrade Your Key Skills

Upgrading your skills is one of the most important personal productivity principles of all.

Always remember, however good you are today, your knowledge and skills are becoming obsolete at a rapid rate.

Rule: Continuous learning is the minimum requirement for success in any field.

Three Steps to Mastery

Read in your field for at least one hour every day.
Take every course and seminar available on the key skills that can help you. Dedicate yourself to becoming one of the most knowledgeable and competent people in your field.
Listen to audio programs in your car.

12. Identify Your Key Constraints

Between where you are and the goal you want to achieve, there are constraints. Your job is to identify what could be holding you back clearly.

This could be hard to find, but if there are a few constraints, you can apply the 80/20 Rule to work on them.

There are 80% of the constraints that are internal. They are within you, your qualities, abilities, and habits.

There are 20% of the limiting factors that are external - competition, markets, governments, or other organizations.

Try to identify what could be acting as a brake on the achievement of your goals.

13. Put the Pressure on Yourself

The problem is: no one is coming to rescue you.

Only 2% of people can work entirely without supervision. They are called leaders.

To reach your full potential, you must form the habit of putting pressure on yourself.

Raise your bar. Push yourself to be better every day.

Work as though you had only one day to get your most important jobs done.

By putting pressure on yourself, you will accomplish more tasks better and faster than ever. You will become a high-performance, high-achieving personality.

14. Motivate Yourself into Action

To keep yourself motivated, you must resolve to become an optimist. You must decide to respond positively to the words, actions and reactions of the people and situations around you. You must refuse to let the unavoidable difficulties and setbacks of daily life affect your mood or emotions.

Control Your Inner Dialogue

Say, "I like myself! I like myself!".
Say, "I can do it!, I can do it!".
Say, "I feel terrific!".

As speaker-humorist Ed Foreman says:

You should never share your problems with others because 80% of people don't care about them anyway, and the other 20% are kind of glad that you've got them in the first place.

Develop Mental Positive Attitude

Optimism is the most important quality you can develop for personal and professional success and happiness. Optimistic people seem to be more effective in almost every area of life.

It turns out that optimists have four special behaviors:

First: look for the good.
Second: see the valuable lesson in every setback or difficulty.
Third: look for the solution to every problem.
Fourth: think and talk continually about their goals.

15. Technology Is a Terrible Master

Technology can be your best friend or your worst enemy. It becomes the enemy when we give in to an obsessive need to communicate continually.

When people are too plugged in, communications technology quickly becomes a destructive addiction.

Apply the 80/20 Rule to your emails. There are 80% of your message with no value. Of the other 20%, only about 4% require immediate responses. The other 16% could be transferred to an action folder to be worked on later.

Unsubscribe from all unwanted newsletters.

If there is something really important that you are afraid to miss by not being connected, it always ends up that if it is really important, someone will tell you.

16. Technology Is a Wonderful Servant

You must discipline yourself to treat technology as a servant, not as a master.

Make sure that only the communication channels you need to complete your task are opened.

Show your smartphone who is the boss by disabling all notifications - both audio and visual.

Segment your communication channels so that only frogs can hop into your castle of concentration.

Take control of your time with a to-do list. It is a powerful tool for taking control of your time. It can be programmed to remind you about the most important tasks.

And, if you become frustrated, just remember that it happens to everyone; even expert programmers who make hundreds of dollars per hour will sometimes be frustrated with technology.

17. Focus Your Attention

Focused attention is the key to high performance.

Current research proves that continuously responding and reacting to e-mails, telephone calls, texts, and instant messages has a negative effect on your brain, shortening your attention span and making it difficult, if not impossible, for you to complete the tasks.

Developing an Addiction

When you check your e-mail, first thing in the morning or when you respond to the bell or other sound that indicates an incoming e-mail or message, your brain releases a tiny shot of dopamine. This shot gives you a pleasant "buzz".

When you start your day with a few shots of dopamine triggered by your e-mail or IM bell going off, you find it extremely difficult to pay close attention to your important tasks for the rest of the day.

The Multitasking Illusion

People can focus only on one thing at a time. When people say that they can do more than one thing at a time, they are doing what is called task shifting.

After an Internet interruption, it takes about 17 minutes for you to shift your total attention back to your task and continue working.

The Proven Solutions

Do NOT check your e-mail in the morning and immediately trigger the all-day dopamine addiction. Leave your devices off.

Decide to check your e-mails only twice a day, at 11am and 3pm.

Never insult the person you are meeting with by working on your laptop or answering your phone. Pay 100% attention to the other people. This applies at home as well.

Double Your Productivity

Plan each day in advance, select your most important task, and then start on that task first thing, before you do anything else.

Work for 90 minutes without any break, then give yourself a 15 minutes break. Keep doing it repeatedly.

18. Slice and Dice the Task

To cut a big task down to size is the "salami slice" method.

When you start and finish a small piece of a task, you feel motivated to start and finish another part, the other one, and so on.

I have several friends who have become bestselling authors by simply resolving to write one page or even one paragraph per day until the book was completed.

19. Create Large Chunks of Time

Schedule large chinks of time for a designated task. Let's say, use the time between 10am and 11am to call all your clients that you need to get feedback from.

As a result of the well-defined time for the task, you become more and more productive and eventually produce two times, three times as much as an average person.

Get up 3 hours earlier and work it without interruptions at home. You can get three times more work done at home without interruptions.

One of the keys to high levels of performance and productivity is to make every minute count.

20. Develop a Sense of Urgency

Do not wait: the time will never be "just right".

High-performing men and women have the quality of "action orientation". They are in a hurry to get their key tasks completed.

As a result, they seem to power through enormous amounts of work in the same amount of time that the average person spends socializing, wasting time, and working on low-value activities.

Develop a sense of urgency. A sense of urgency feels very much like running against yourself. You concentrate on the things you can do right now to get the results you want to achieve the goals you desire.

The faster you move, the more impelled you feel to do even more even faster. You enter "the zone".

Build a Sense of Momentum

Although it may be tremendous amounts of energy to overcome inertia and get started initially, it. then takes far less energy to keep going.

The faster you move, the more competent and capable you become at your work.

A sense of urgency shifts you automatically onto the fast track in your career.

Do It Now!

One of the simplest things to repeat to yourself: "Do it now! Do it now! Do it now!". "Back to work! Back to work! Back to work!".

Nothing will help you more in your career than for you to get the reputation for being the kind of person who gets important work done quickly and well.

21. Single Handle Task

By concentrating single-mindedly on your most important task, you can reduce the time required to complete it by 50% or more.

When you prepare thoughroughly and then begin, refusing to stop or turn aside until the job is done, you develop energy, enthusism, and motivation.

Once you have decided on your number one task, anything else that you do other than that is a waste of time.

Elbert Hubbart defines self-discipline as:

the ability to make yourself do what you should do, when you should do it, whether you feel like it or not.

Once you start your most important task, discipline yourself to persevere without diversion or distraction until it is 100% complete.

Conclusion

Develope the habit of eating your frog first thing every day when you start work. Repeat it every day.

Review the 21 rules and principles regularly until they become firmly ingrained in your thinkgin and actions, and your future will be guaranteed.

Make a decision to practive these principles every day until they become second nature to you.

Just do it! Eat that frog!.

What does an automation tester do?

Everton Schneider — Wed, 30 Aug 2023 14:29:49 +0000

Recently I’ve watched a great talk from Selenium Conf 2023 by Mark Winteringham.

He could expose something that I believe most professionals cannot realise.

Automation testers focused too much in their technical skills.

To help to elaborate that, Mark asks 2 questions:

what we try to say we do?
what we are really doing?

What do we try to say we do?

The first thing that called my attention was his research based on a job website (surprisingly not LinkedIn), collecting information about job roles.

Most of the automation testers try to expose that they have knowledge about tools (Java, JavaScript, Python, JUnit, Test NG, Selenium, Cucumber, Cypress, Playwright, Rest Assured, Postman, SOAP UI - the list is long).

We focus on these things, like if they represent the importance of what we do. It seems that it is more important to know how to use a tool or technology than to test something well.

From Mark's research, he sorted testing job skills in 3 different areas:

He listed jobs on the Indeed website and classified them.

Testing skills (31%)
- Risk analyses
- Writing and raising bugs reports
- Ensuring quality
Automation skills (33%)
- Maintaining flaky tests
- Set up environments
- Set up pipelines
Tools skills (36%)

After all, the majority of jobs list more tools than testing or automation skills.

But, should we focus on the tools, or the principles?

What are we really doing?

There isn’t any right answer for this question, however, Mark asks the audience to write down 3 simple things:

What do we do on a daily basis?
What do we do on a weekly basis?
What do we do on a monthly basis?

After answering these 3 questions, we can start to think more clearly about what we do.

Automation testers need different skills that are not directly related to any technology or tool:

plan tests
set up pipelines
understand system requirements
report defects
prioritise tasks
write down documentation
communicate with the team
investigate issues

None of these are specifically related to tools or technologies, but they are all testing skills required to do the daily basis work.

The image of what an automator is, is not the same as we are actually doing or getting involved in.

By the end of the talk, Mark listed some things that an automator does:

To me, it was a great talk to reflect over testing skills. We usually focus heavily on learning and specialising in tools, but we usually leave basic principles aside.

The talk from Mark is available on YouTube:

What Exactly Do You Do in Test Automation?

Tips to avoid unstable tests in Selenium

Everton Schneider — Mon, 24 Apr 2023 03:30:23 +0000

Selenium tests are hard to maintain. They can get flaky and present intermittent failures as their codebase grows.

I will list a few things that I've seen as effective to make Selenium tests stable, running a suite of ~10K tests.

TL;DR;

Test code must be treated as seriously as development code (do code review, use best practices);
Track the problems creating bugs for failures;
Use Page Objects;
Understand how Selenium wait works (implicit, explicit) and to use WebDriverWait and ExpectedConditions classes.

1. Treat test code in the same way you treat development code

Test code is not different from development code. They act in different areas of the software, but they should be treated with the same respect.

The more effort is put in having good code for automated tests, fewer problems are found when maintaining the test suites.

Do code review

Code for automated tests MUST be reviewed. They have to be treated with respect, and the reviewer has to be as strict as he or she is with the code from the developers.

E2E code is widely seen as expensive and unstable. Code review helps to organize the code in reusable components, enforce using better selectors (css, id, name or even xpath), check code semantics, ensure it has good logging, verify that it is well-documented and that errors are handled properly (etc, etc, etc).

When code review is taking seriously for E2E tests (or any automated test), the consistency of the codebase enforces standards that result in more stability in tests execution, reducing the issues seen when running E2E tests frequently, or after new releases.

Request for best practices

Page Objects have been around for a long time. They are a solid pattern to be used. However, it is not the only thing to be used.

Create random data to enter in fields for tests. There are many libraries that can help (e.g. DataFaker, jFairy). Do not use static values.

Use the Builder pattern to create instances of entities.

Inject sessions to skip repeatedly log-in operations.

Keep specifications small. Try to set up a threshold for the overall time it takes to run (I use 5 minutes as a reference).

Use DSL, if convenient.

Choose a good a testing library to perform assertions (JUnit, TestNG, ScalaTest).

Run tests over a CI server (Jenkins, TeamCity).

Scale the execution using Selenium Grid - it is easy with Docker Selenium.

Use Page Objects

As said before, Page Objects are already mature enough to be called a pattern.

They can be used not only to map pages, but also components.

Pages

Page is a UI representation that must be reusable in the codebase. Different test cases can access the same page and its code must be shared (e.g. the system's landing page).

The page class must contain fields and methods to interact with the web elements that are present on the UI.

Components

A page is composed of many different web components. They can be seen as inputs, dropdowns, custom elements and so on.

A component can be reusable along different pages. They should be wrapped in a specific class that can be instantiated in different pages, similar with how the front-end codebase declare them.

2. Create bugs for all issues

All tests that fail and need to be fixed need an issue in the issue tracker system. As much as any intermittent failure needs to have one.

When creating issues for all failures seen, issues in automated tests can be tracked. When issues can be tracked, they can be prioritized and, if they can be prioritized, they can be fixed in an order that helps the suite of tests to become more stable.

Remember:

if a test case is failing consistently (it can even be reproduced manually): create a ticket to fix it.
if a test case is failing intermittently: create a ticket to investigate and fix it.

There must be no fear for tickets. They serve to help.

3. Implement custom waiting

Page load is probably the thing that mostly impacts in Selenium tests performance. They are hard to predict and sometimes lead to intermittent failures.

If a screenshot is taken by the time that a Selenium test fails, it reveals commonly that there is something still loading on the page (the page itself or one of its components).

It is extremely important to understand how to implement a mechanism for Selenium to apply some waiting time.

Implicit Wait

Implicit wait is defined by values that are set up in the WebDriver instance.

// Specifies the amount of time the driver should wait when searching for an element if it is not immediately present.
webDriver.manage.timeouts.implicitlyWait(3) 
// Sets the amount of time to wait for a page load to complete before throwing an error
webDriver.manage.timeouts.pageLoadTimeout(6)

These are default values set in the WebDriver that will be used between starting the web browser (webDriver.get(url)) and closing it (webDriver.quit()).

Explicit Wait

Implicit wait is not always effective. Pages and their elements can take different time to load.

An exception like NoSuchElementException is commonly thrown when Selenium cannot find the element to interact with on the page.

If the element is excepted to be on the page, it is possible to use other strategies to wait for it, for longer than the Implicit Wait set into the WebDriver.

This can be achieved combining the WebDriverWait and the ExpectedConditions and is called Explicit Wait.

`WebDriverWait` and `ExpectedConditions`

The API for the ExpectedConditions is rich and very clear. It is used internally inside an instance of the WebDriverWait class, passing by parameter also the time that it should wait for the condition.

Below some self-explanatory examples:

// Instantiate the `WebDriverWait` with `15` seconds as the limt time to wait for a condition
WebDriverWait = bew WebDriverWait(driver, 15)

// Find an element on the page
WebElement element = webDriver.findElement(By.id('main'));

// Wait for an element to be stale
seleniumWait.until(ExpectedConditions.stalenessOf(element));

// Wait for an element to be visible
seleniumWait.until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector(".button"));

// Wait for an element to be invisible
seleniumWait.until(ExpectedConditions.invisibilityOfElementLocated(element));

// Combined condition: all 3 elements must be clickable
seleniumWait.until(
    ExpectedConditions.and(             
        ExpectedConditions.elementToBeClickable(By.id("a")),
        ExpectedConditions.elementToBeClickable(By.id("b)),
        ExpectedConditions.elementToBeClickable(By.id("c"))
    )
)

// Wait for 2 elements present on the DOM
seleniumWait.until(ExpectedConditions.numberOfElementsToBeMoreThan(By.cssSelector(".button"), 2))

The API for the ExpectedConditions class is rich and easy to understand. There are many methods that can help the conditions that could help to add stability to tests that rely on elements that take longer than expected to be ready to interact with.

4. It's not all

Surely this is an endless discussion, and many other things could be added to the article above that contribute to having stable tests implemented with Selenium.

System Design: Basic approaches

Everton Schneider — Mon, 27 Mar 2023 21:13:16 +0000

System design is a very important topic in the technology industry today. It doesn't matter whether working in a decision-making role or as part of a development team, having a basic understanding of system design is essential.

TL;DR;

Systems need to adapt to their requirements as they are created or as they grow:
- respond faster;
- respond to more simultaneous users;
- use more storage;
- be highly available.
To adapt, systems have to scale. They basically must use vertical or horizontal scalability techniques.
Scalability must consider the costs to implement each different solution.
Basic System Designs:
- 1 server (application + database);
- 2 servers (1 for application, 1 for database);
- Load Balancer (many application servers).
Database strategies beyond how to set up hardware can also apply:
- Database normalization or denormalization.

Introduction

One of the major challenges for professionals in IT is system scalability. Planning how to design a system to meet all non-functional requirements is challenging and uncertain.

The design must be carefully assessed before releasing a system, or it needs to be reviewed after problems arise.

This article aims to list a few "basic approaches", which can be easily found in many small systems, mostly the ones hosted on-premises.

Designing a system needs to take many aspects into account:

How many users will concurrently use the application?
How much space is needed? How will it grow?
How crucial is the availability of the service?
How much money can be expended with services and/or hardware?

Below there are some concepts and the presentation of basic system design for applications.

Types of Scalability

Scalability relies basically on two pillars:

Vertical Scale
Horizontal Scale

Vertical Scale

In short, it implies that, when the system is not responding as desired, then more hardware has to be added for it. Hardware like memory, CPUs or disk space are usually added.

Horizontal Scale

In short, it implies that, when the system is not responding as desired, more machines have to be added for it. System infrastructure is increased with other servers, considering that there are separate concerns, redundancy, etc.

In old (or very old) days

Single Server Design

In the "old days", the common design of a system consisted in having one server with all the services, usually maintained on-premises.
The same machine was responsible for hosting the web server for the application, the database, the filesystem and whatever else was needed.

This approach is very fragile, as many things can interfere in the availability or usability of the system:

if the CPU has to process a complicated query - it will make the whole application very slow;
if the disk is out of space, it will make the application unavailable;
if the web server has too many requests, it will return an HTTP server error response (500 errors).

Apart from the things mentioned above, it has a "single point of failure". It means that, if the server goes down, the whole application goes down too.

In this type of design, to scale a system, it is usually used a technique called "Vertical Scale", where the server gets more RAM, hard disk or processors (CPUs).

Even though it is an approach that has been in place for years, the Single Server Design still has space for small systems, usually sitting in a computer inside the office, which serves daily tasks for a limited number of users (e.g. 10 users or so). It can be set up on-premises or even in a cloud provider.

Application Server + Database Server Design

A better approach is to separate the database from the application server. It usually brings more resilience for the architecture, although it still has a single point of failure.

In case the CPU is overloaded in the application server, at least the database is isolated and can still operate. Even the risk of corruption is reduced since the database uses a separate machine with a dedicated hard disk and processors.

Adding a Load Balancer

Modern approach for scaling "expects" Horizontal Scale.

Another option of a system design is to set up the application behind a Load Balancer. It distributes the loading of the application between different servers.

A load balancer is a device that acts as a reverse proxy and distributes network or application traffic across a number of servers. Load balancers are used to increase capacity (concurrent users) and reliability of applications. (What is a Load Balancer?)

This approach works well if servers are "stateless", where they don't store anything that makes a request depend on what a previous request has stored.

However, it still depends on a single point of failure. If the database crashes (for any reason), the application cannot rely on any other data source.

Database Normalization

Normalization is the process of organizing data in a database. This includes creating tables and establishing relationships between those tables according to rules designed both to protect the data and to make the database more flexible by eliminating redundancy and inconsistent dependency. (Description of the database normalization basics)

Normalizing data is something very basic, and we usually tend to do it to avoid redundancy by creating more tables and relationships.
Yet, it may not be the best solution for a given problem.

The better we organize the database and avoid redundancy, the less space we use. And that is usually great. On the other hand, we usually spend more time to return query results due to joins between different tables or multiple queries to return a set of data.

It is important to define what's the trade-off. If performance a high priority and there isn't a concern about how much it can cost to store all data, denormalizing the database may be an option.

Example

The example below shows a very simple invoice stored in a normalized database. The data is stored in two different tables:

INVOICE
CLIENT

Normalized database

A normalized database uses less storage, but it requires more lookups.

Table `INVOICE`

The table has a foreign key (CLIENT_ID) that enforces the relationship with the table CLIENT.

Table `CLIENT`

The table CLIENT stores the client's information and avoids duplication.

Denormalized database

A denormalized database uses more storage, but it requires just one lookup.

Table `Invoice`

The question is: do you need to respond quickly without multiple joins and queries? Or, do you need to minimize disk usage?

Conclusion

This article presents very basic System Design for applications. They can be widely used for applications that do not require large scalability.

The database denormalization technique can help to minimize bottom necks of performance in high-availability applications, however it has its downside, which is more use of disk-space.

It is important to consider that the simplest architecture that meets the project traffic requirements should be used, but not the smallest.

More to come in next posts about System Design.

How to run Playwright tests with Selenium Grid versions after `4.5`

Everton Schneider — Sat, 18 Feb 2023 16:35:04 +0000

Compared to other automation testing libraries, Playwright is relatively new, but it supports writing tests in different languages.

If you're using Java, Playwright can take advantage of the Selenium Grid to scale test execution, helping you save time and increase efficiency.

TL;DR;

Playwright has a limitation that restricts running tests in Selenium Grid to only the version prior to 4.5.2-20221021 (at the time of this writing - February/2023).
Selenium 4 has passed through a redesign of its architecture. After a long period of stabilization, a long-standing intermittent issue was fixed in version 4.5.2-20221021.
If Playwright tests only work with versions prior to 4.5.2-20221021, it won't be able to benefit from the latest improvements made by the Selenium team to the grid or any browser version upgrades. A workaround for the issue is to use a specific property -Djdk.httpclient.websocket.intermediateBufferSize, passed as an argument to the selenium-server.jar.

Problem

Although Playwright is still relatively new compared to Selenium, it generally offers faster test execution times. In a "real-work" test that I used to compare the two tools, the results were as follows:

Spec	Library	Number of tests	Duration
selenium.specs.LoginSpec	Selenium	32	1:31 minutes
playwright.specs.LoginSpec	Playwright	32	43 seconds

The example above is a simple test run from a local machine, without using Selenium Grid.

According to the Playwright website, tests written with Playwright can run against Selenium Grid, as documented here. However, as of February 2023, there is currently an issue with Selenium Grid versions 4.5.2-20221021 or later, which can cause errors when attempting to use Playwright with Selenium Grid.

Cannot run playwright tests on Selenium Grid

If Playwright tests can run only with Selenium Grid version prior to this, then it means it is running only with older versions of Google Chrome or Microsoft Edge:

Spec	Supported versions in `4.5.0`	Supported versions in latest (`4.8.0`)
Chrome	106.0.5249.119	110.0.5481.77
ChromeDriver	106.0.5249.61	110.0.5481.77
Edge	106.0.1370.47	110.0.1587.41
EdgeDriver	106.0.1370.47	110.0.1587.41

History

When Selenium 4 was release, it changed completely how Selenium Grid is structured. It has totally changed its design, based on distributed components to handle messages between the hub and the nodes.

After its release, a critical intermittent issue received significant attention, as it was difficult to reproduce and persisted for almost two years before being resolved.

https://github.com/SeleniumHQ/selenium/issues/9528#issuecomment-947994758

This fix was officially released on docker-selenium project on version 4.5.2-20221021. The exact same version that started to present issues with the execution of tests written with Playwright.

Fix (Workaround)

The fix was reported by EndPositive in an issue opened for the Selenium team.

The issue seems to be related to WebSocket communication, with messages larger than the default configuration of Selenium Grid causing problems.

The suggestion is to increase the WebSocket buffer size passing the parameter Djdk.httpclient.websocket.intermediateBufferSize to the selenium-server.jar.

Using docker-selenium images, it is possible to do as shown in the example below:

version: "3"

services:
  selenium-hub:
    image: selenium/hub:4.8.0-20230210
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"
    environment:
      - JAVA_OPTS=-Djdk.httpclient.websocket.intermediateBufferSize=3000000
      - SE_EVENT_BUS_HOST=selenium-hub

  chrome:
    image: selenium/node-chrome:4.8.0-20230210
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - JAVA_OPTS=-Djdk.httpclient.websocket.intermediateBufferSize=3000000
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - HUB_HOST=selenium-hub
      - HUB_PORT=4444
      - SE_NODE_GRID_URL=http://localhost:4444

To run Playwright tests against Selenium Grid, the environment variable SELENIUM_REMOTE_URL must be declared:

export SELENIUM_REMOTE_URL=http://localhost:4444

After that, tests can be run in different ways:

./gradlew test

./mvnw test

Or with other build tools of preference.

Final consideration

It seems that Playwright uses Selenium Grid, but Selenium Grid does NOT officially support Playwright. The problem to run Playwright tests with the newest version of Selenium Grid were raised in Selenium GitHub project, however it hasn't been investigated so far to be fixed once it cannot be reproduced with tests implemented using Selenium (and Selenium WebDriver).

Optimising parallel tests in Selenium Grid

Everton Schneider — Sat, 07 Jan 2023 22:48:29 +0000

There are many articles trying to explain how to improve Selenium tests within the code. They show best practices and are very relevant.
On the other hand, it is hard to find pages with tips to improve the execution against Selenium Grid.
This document aims to cover some simple practices that can result in faster execution of Selenium tests using Selenium Grid.

TL;DR;

Define a maximum time to run each spec in your test suite and implement them with that constraint in mind
Sort the specs inside your suite of tests
- Measure the overall time that each spec takes to be executed
- Run the longest specs first
Cancel subsequent tests of a spec when a test case inside the spec fails

Implement automated tests with a time constraint

Specs should be small - period.
Small specs take advantage of better usage of Selenium Grid, since they are better distributed between the available nodes.

Every case must be evaluated, but in my case I try to have 5 minutes as a time limit. In the suite of tests that I work with, if a spec takes more than 5 minutes to run, I usually put some effort to split it into 2 (or more) smaller specs.

Sort the execution of the tests suite

Selenium tests are commonly run against a CI Server (Jenkins, TeamCity, etc), but even when run locally they provide a report with the list of specs executed and the duration of each spec.
Usually, the order how tests are run is arbitrary and managed by the test library (JUnit, TestNG, etc) used with Selenium.

Saving this data can be useful to sort tests.

Measure and set the overall time in each spec

Every time a spec is run, it takes a different time to execute. Page load and some other factors influence in how the duration takes a bit more or less each time that a spec is run.

However, by looking at the history of tests executed, it's possible to find an average time. It is interesting to use this average time to sort the specs before they are run.

The average time could be set in a property, internally in the spec, or be searched from a database, if that's saved.

A possible approach to sort the specs is to define a category based on the average time the spec runs:

Very Short (executed in less than 30 seconds)
Short (executed between 30 seconds and 1:30 minutes)
Medium (executed between 1:30 minutes and 3:00 minutes)
Long (executed between 3:00 minutes and 5 minutes)
Very Long (executed in more than 5:00 minutes)

If every spec has a constant containing the information above, it is easy to order the list of specs from a test suite using that criteria.

The drawback is that it needs to be updated if a spec is changed (e.g. split), but other approaches can be considered (e.g. saving every test result in the database and calculating each spec average time before every test run).

Run the longest specs first

This is my favourite trick. It must be combined with the previous steps, and it makes the test distribution between the Selenium nodes better, reducing the overall time that tests take to run and also reducing idleness of Selenium nodes.

As an example, let's say we have just 2 Selenium nodes available to run 3 different specs with the following execution time average.

Spec1: runs in 5 minutes
Spec2: runs in 2:30 minutes
Spec3: runs in 2:30 minutes

As seen in the image above, if we don't run the longest spec first, the overall execution time may take longer due to the distribution that is not optimised. Apart from the longer time to run tests, it shows that the Node 2 will be idle for some time. In a scenario with hundreds of specs, it can result in a big difference in the overall execution time and when tuned it can represent a better use of the resources available for the Selenium Grid.

Cancel subsequent tests of a spec after a failure

Selenium tests are different from Unit Test. In many cases, they depend on previous tests to run.
As an example:

1. log in to the application
  1.1. ensure the page title is as expected
  1.2. ensure the values are loaded correctly
2. click on the button "New"
  2.1. ensure the page title is as expected
3. complete the form
4. click on the button "Save"
  4.1. ensure the success message is displayed

For the example above, if the test 2.1 fails, what's the benefit of trying the next ones, which depend on it? They may just add a lot of invalid failures, which are not related to the root cause of the problem, adding unnecessary error LOGs to the results.

Just as an example, in ScalaTest it can be easily done mixing-in the trait CancelAfterFailure to the specs.

Final words

The simple changes listed above can save some time and optimise the execution of Selenium tests.

For some of the topics pointed above, it is needed to have a custom implementation, like ordering the specs to consider the longest ones to run first. However, there shouldn't be any complicated implementation.

Hopefully it's somehow useful for somebody else.

What to study for the LambdaTest Selenium Java 101 Certification

Everton Schneider — Tue, 18 Oct 2022 12:53:26 +0000

LamdaTest has a certification to test your Selenium skills. The exam is free, and it can be done all online.
Below, I will share my experience taking the tests to get the certification.

TL;DR;

If you want to take the Selenium Java 101 certification:

For the multiple-choice questions section
- there will be questions about Java
- there will be questions about JUnit and TestNG
- there won't be questions about Selenium
For the code assignment
- the implementation needs to use JUnit or TestNG
- the implementation needs to run tests in parallel and in different browsers (some LambdaTest knowledge will be required)
- the implementation of tests will require to use the LambdaTest playground and will need to use different selectors (id, xpath, cssSelector, etc)

How it is structured

The certification page explains clearly how the test is structured. It is composed of 2 rounds:

Round 1: 40 multiple choice questions to be completed in 45 minutes
Round 2: code assignment to be completed in 36 hours

What to expect from the MCQs

When I started the MCQs session, I was very surprised. Since I did not study much for the content of the certification, I did not realize something obvious:

You do not use Selenium for testing without a testing library.

The certification page has a section called Learning Material that mentions ONLY Selenium.

I was expecting to see questions about selectors, how to deal with iframes, cookies, instantiate local x remote WebDriver, but...

The MMQs have ZERO questions about Selenium.

I work daily with Selenium and use ScalaTest as the testing library, but for the Selenium Java 101 certification, you need to know JUnit and TestNG.

The MMQs, in my case, were all about Java, JUnit and TestNG.

Java

I remember very few questions about Java, and they were not trivial. I recall questions about Threads and some expected outputs from a piece of code.

JUnit annotations

It is important to know them, how to use them and the internal parameters they receive.

The ones I remember to see questions about:

@Test
@Before, @After
@Parameterized
@Timeout

TestNG annotations

Like in JUnit, it is important to know the annotations and their parameters.

The ones I remember to see questions about:

@Test
@BeforeTest, @AfterTest
@DataProvider

What to expect from the code assignment

The code assignment is sent to you after you complete the MMQ session. You only receive the code assignment if your score is at least 50% in the MCQ.

My assignment was composed of 3 tasks. All of them were to implement Selenium Tests for different scenarios using the LamdaTest playground website.

The tests to be implemented were not complicated, but in the end, they all must be executed inside LambdaTest platform.

Some assignment requirements:

implement tests using different locators (xpath, css, id, etc).
run tests in parallel

I was not familiar with LambdaTest platform, so it is good to consider some time to learn a little bit about how it works.

FAQ

There are a few questions and answers on the LambdaTest certifications page.

The ones that I think are important to know in advance:

the results "should" be sent to you in 48 hours (my results came after a week)
there is a 45 days retake policy ("Can I reappear the certification?")

Result

Book: Pragmatic Thinking and Learning

Everton Schneider — Mon, 01 Mar 2021 14:37:32 +0000

Andy Hunt, one of the authors of the famous book The Pragmatic Programmer has also written the book Pragmatic Thinking and Learning.
Pragmatic Thinking and Learning lists some interesting models and techniques that can help in the process of learning.
A few of them I will mention in this article.

The Dreyfus Model

The Dreyfus Model is a model of how learners acquire skills through formal instruction and practising (see Wiki). It is divided into 5 levels:

Novice
Advanced Beginner
Competent
Proficient
Expert

Dreyfus is applicable per ability (skill)

Novice

They have little or no prior experience in the ability area.
They are not concerned about learning the details. They want to achieve the goal asap.
They don't know how to respond to errors, therefore they're very vulnerable to confusion when something goes wrong.

Novices need recipes.

Advanced Beginner

They can run away from a set of fixed rules.
They can try to do some things on their own, but still have difficulties to solve problems.
They want information quickly.

Advanced Beginners don't want to have an overview.

Competent

They can develop conceptual models of the problem domain.
They can work with the conceptual model effectively.
They can solve problems.
They plan acts based on previous experiences.
They still have difficulties to find details when solving problems.

Competent professionals can solve problems.

Competent professionals are good to have in a team. They can solve problems, help beginners and do not disturb much the experts.

Proficient

They need an overview.
They will try to understand the concepts around the skills.
They get frustrated with simplified information.
They can review how they have done things in the past to have a better performance in the future.
They can take advantage of feedback. This is a big difference compared with the previous stages.

Proficient professionals can self-correct.

Proficient is more of a "junior expert" than an "advanced competent".

Expert

They have a great set of experiences to be explored and applied in the proper context.
They write books, articles, make speeches in seminars. They're wizards or gurus in the modern world.
They work based on intuition, not reasoning.

Statistically, there aren't many experts - probably something about 1 to 5% of the population.

Dreyfus at Work

Use rules for novices, intuition for experts.

The distribution of abilities.

You have to know what you don't know.

How To Become An Expert

You should have to work on a plan for about 10 years to become an Expert, regardless of the area or subject.
Just to work on the area or subject for 10 years may not be sufficient. You need to practice. The cognitive scientist K. Anderson Ericsson says that the practice requests 4 conditions:

You need a well-defined task;
The task needs to be hard, challenging, but doable;
You need to be able to get feedback to improve;
You need to have the possibility to act to repeat and fix errors.

At work...

A group is as good as its weakest member.

How Our Brain Works

The brain can be divided into 2 parts - Right and Left.
The book treats them in 2 different modes - Rich and Linear.

Characteristics of the mode L

Rational
Time based
Logic
Linear

Characteristics of the mode R

Not verbal
Not rational
Analogic
Holistic
Intuitive

The book claims it's important to improve the mode R to achieve intuition, one of the skills found in experts in the Dreyfus Model.

It lists a few things that can be done to help to develop the mode R:

Take notes of all ideas you have
Create metaphors
Write freely without thinking every day
Go out for a walk to clear your mind
Practice yoga, meditation, do breathing exercises
Practice martial arts

Changes are good

We have habits, but ingrained habits are not the best thing for our brain. We do NOT make new connections repeating habits and we become blind to new alternatives.

A few things that are easy to change to try to help the mode R:

Ask for a different dish at the restaurant
Cook something new
Visit a place you have never visited before
Read a book or watch a movie from a different genre (e.g. if you like comedy, try fiction)

A Few Things About Our Mind

The human mind is not open-sourced. We cannot access its code and try to fix the bugs when they happen, but we can try to understand it better to know how to manage the failing processes in our thinking.

Let's check 4 categories of problems

Cognitive Bias: How our behaviour can deviate.
Generational Affinity: How our colleagues have influence over us.
Personality Tendencies: How our personality influences our thoughts.
Hardware Bugs: How older parts of your brain can overlap parts more intelligent.

Cognitive Bias

There are many cognitive biases. Wikipedia lists about 90 common cognitive biases.
Below there is a very short list of examples:

Attribution Bias: The tendency for people to want to believe that the world is fundamentally just, causing them to rationalize an otherwise inexplicable injustice as deserved by the victim(s).
Confirmation Bias: The tendency to test hypotheses exclusively through direct testing, instead of testing possible alternative hypotheses.
Egocentric Bias: The tendency of people to see their projects and themselves as more singular than they actually are.

Generational Affinity

Have you ever thought about why you value the things that you value?

Below a list of different attitudes:

Take risk vs Do not take risk
Individualist vs Team player
Likes stability vs Likes freedom
Value family vs Value work

Different generations have different values and different attitudes.

The above are wide generalizations.

The generations:

GI, 1901-1924
Silent, 1925-1942
Boom, 1943-1960
X, 1961 to 1981
Millenium (or Y), 1982-2005
Z, 2005-???

All Together Now

Nowadays, around 2010, the workplace has all these generations present at the same time, interacting with each other and coexisting (sometimes not).

How Does It Affect You

Not everybody shares the same values you do and it doesn't mean that you are right or wrong.

You need to fit in the context you are in.

Personality Tendencies

Hell is other people - Jean Paul Sartre

Your personality affects your values and perceptions in addition to generational affinity. This is how you were born and the baggage you carry with you - your own context of personal attitude, your temper.

MBTI (Myers Briggs Type Indicator)

There is an introspective self-report questionnaire indicating differing psychological preferences in how people perceive the world and make decisions. The test attempts to assign four categories: introversion or extraversion, sensing or intuition, thinking or feeling, judging or perceiving.

You can try it on the internet. There are many free surveys that you can answer to see your type.

16personalities free test

You cannot change people.

There are different ways to develop a solution and commitment. The only thing that certainly does not work is to try to change the temper of another person to be the same as yours.

Monkey Sees, Monkey Does

We are natural imitators.
If you are close to happy and optimistic people, this tends to improve your humour. If you are going out with depressed and pessimistic people that feel like they are losers, you start to feel a depressed loser as well.
Attitudes beliefs, behaviours, emotions - they are all contagious.

Test Yourself

When you are completely convinced of something, as yourself why.

How do you know?

Make good questions:

How do you know that?
Who said that?
How exactly it works?
Compared to what?
Does it always happen? Is there any exception?
What could happen if...?

Techniques To Learn

You need continuous goals. You need to get feedback to understand your progress and approach everything in a more specific way than taking a course once a year in an uncomfortable class room.

SMART

You need to be careful if you don't know where you are going, because you may never get there.

It is important to create a SMART to achieve your goals.

PIP - Pragmatic Investment Plan

In the book Pragmatic Programmer, the author has suggested that the reader should consider his or her skills and talents like a knowledge portfolio. And like any portfolio, it should be managed over time.

Model your knowledge portfolio with the same care that you would manage an investment portfolio.

The time cannot be created or destroyed, just allocated.

There are many important aspects involved when maintaining your knowledge portfolio:

You need a concrete plan.
You need to diversify.
You need to make an active investment, not a passive one.
You need to do it on regular bases.

You need to commit to investing a minimal quantity of time regularly. Create a ritual, if necessary. Go to a place you can concentrate on. Not all sessions will be equally productive, but by scheduling regularly, you will see the benefits in the long term.

Work Together, Study Together

Studies have shown that study groups are a good thing.

Reading groups are not toxic.

Form study groups to learn and teach.

Use Enhanced Learning Techniques

Summarise material.
Use mind maps.
Teach.

Read with SQ3R

SQ3R is a reading comprehension method named for its five steps: survey, question, read, recite, and review.

The method offers a more efficient and active approach to reading textbook material.

Mind Maps

A mind map is a type of diagram that presents topics and how they are connected. Its creation is a technique widely used to improve creativity and productivity.

The main idea can be written in the middle of the page and the links will be written around, connected by arrows or just lines.

The Power Of Documentation

Write: To document is more important than the documentation.

Writing helps you to have a better understanding of the topic you are writing about.

Learn By Teaching

Another way to have a better understanding of a topic is to teach it to another person. Explaining to another person makes you organise the ideas more clearly in your head.

Gain Experience

Build to learn, not learn to build.
Fail effectively with better feedback.
Establish your neural paths for success.

Give yourself permission to fail, this is the key to success.

How To Improve Your Focus

Three ways to better manage your mind:

You need increasing focus and attention.
You need to manage your knowledge.
You need to improve your current context.

The book cited a study that has shown that practising meditation can improve the capability of a person the pay attention throughout the day.

Defocus to Focus

Creativity does not work like a time clock and usually doesn't present good results when under pressure. You need to abandon a problem with the problem in your mind, letting it aside for a while.

Take a walk to clear your mind to help to solve a problem.

Optimise Your Current Context

Multi-tasking affects heavily productivity. A study has found that on average, multi-tasking can cost from 20% to 40% of your productivity (see the article).

It takes 20 minutes to reload a context.

It means that if you're working on a task and gets interrupted, you can take an average of 20 minutes to get back to the normal pace. Now, think about how many interruptions you can have in a day - a great portion of our day will be wasted. You can explain why programmers, in general, hate to be interrupted.

Another study has observed that, if you constantly interrupt your task to check or answer text messages, your IQ drops by 10 points.

You can try to organise your tasks in an order that the context between them has minimal changes. This can enhance productivity.

Keep A Big Enough Context

Use many monitors to avoid changing contexts.

Put some effort to avoid your focus to leak to other spaces:

E-Mail
Tasks list
Chat app
Calendar

Beyond Expertise

Start with a plan: get some time and fight for it.
Inaction is an enemy, not the failure: Failing is not the enemy, not doing anything is.
New habits take time: It takes at least 3 weeks doing a new activity before it becomes a habit. Maybe even more. Give it a chance.
Take small, next steps: Start with something easy. Define a plan and a goal.

What To Do Tomorrow Morning

Just start.

Create a P.I.P. (Pragmatic Investment Plan) and define goals using SMART.
Find out where you are in the Dreyfus modal (novice - expert) in your profession.
Keep a notebook with you. Take notes, keep your thoughts floating.
Start a blog.
Take a walk to think over things once during the day.
Get a second monitor.

Take the wheel, you cannot drive in the automatic mode.

On the last page of the book, surprisingly the author asks the reader to send him a Mind Map with his learnings of the book.
Below, I am sharing mine.

Book: Clean Code

Everton Schneider — Sat, 28 Nov 2020 20:37:26 +0000

The book Clean Code is a classic. Every programmer should have it on their shelf. Every programmer should read it. Every programmer should know it.

After years of experience, I finally got the time to take some time to read this book. I felt like I should have read it years ago.
The reading progresses smoothly with some nice code examples in an enjoyable journey with tips to learn, to have some reflection and questions about how you can code better.

The chapters of the book are very well divided into categories and the more you proceed with its contents, more you want to continue.

Below I will leave a few of my favourite topics and some distinct phrases that I liked from it, which I turn to sometimes to remember some good practices we all should follow.

Chapter 1: Clean Code

The Boy Scout Rule

It's not enough to write code well. The code has to be kept clean over time.
If we all check in our code a little cleaner than when we checked it out, the code simply could not rot.

Chapter 2: Meaningful Names

Don't Pun

Avoid using the same word for two purposes. Using the same term for two different ideas is essentially a pun.
Follow the "one word per concept" rule.

Use Solution Domain Names

Remember that the people who read your code will be programmers. So go ahead and use computer science (CS) terms, algorithm names, pattern names, math terms, and so forth.

Don't Add Gratuitous Context

Don't prefix your classes or methods with some type of "project initials". If you have a project called "Corner Shop Cash Flow" it is a bad idea do prefix every class with CSCF. When you type C in your IDE, it will give you a long list of all class, not filtering anything you need. Let your IDE work for you.

Chapter 3: Functions

Small!!!

The first rule of functions is that they should be small. The second is that they should be smaller than that.

Blocks and Indenting

This means that the blocks within if statements, else statements, while statements, and so on should be one line long. Probably that line should be a function call.

Do One Thing

Functions should do one thing. They should do it well. They should do it only.

If the function does only those steps that are one level below the stated name of the function, then the function is doing one thing. After all, the reason we write functions is to decompose a larger concept into a set of steps at the next level of abstraction.

Switch Statements

We cannot always avoid using switch statements, but we can make sure that each switch statement is buried in a low-level class and is never repeated. We do this with polymorphism.
The switch statements can be tolerated if they only appear once in the code.

Use Descriptive Names

You know you're working on clean code when each routine turns out to be pretty much what you expected.

Don't be afraid to make a name long. A long descriptive name is better than a short enigmatic name. A long descriptive name is better than a long descriptive comment.

Function Arguments

The ideal number of arguments for a function is zero. Next comes one, followed by two. Three arguments should be avoided where possible. More than three required very special justification - and then shouldn't be used anyway.

Arguments are even harder from a testing point of view. It's hard to write test cases for all combinations that different values passed by parameters can execute. If there are no arguments, it's clear. If there's one argument, is not too hard. If there are two arguments the problem is already a bit challenging. More than that it starts to be complex.

Flag Arguments

Flag arguments are ugly. Passing a boolean into a function is a truly terrible practice. It complicates the signature of the method and it declares that the function does more than one thing. It does one thing if the flag is true and it does another thing if the flag is false.

Argument Objects

When a function needs more than three or four arguments, it is likely that one or more arguments are wrapped into an object.

It looks like cheating, but it's not. The object is part of a clear context that is passed to the function, not just a list of different single values.

Have no side-effects

Functions must promise to do one thing only. They cannot do hidden things that may lead to unexpected behaviours.

Command Query Separation

Functions should either do something or answer something, but not both.

This gets easier to understand with an example:

// THIS IS BAD
public boolean set(String value, int size)  // it should not set a value and return a boolean.

Imagine the readability of this:

// THIS IS CONFUSING
if (set("height", 10)) {...}  // is it a question about what was set? or is it a question about what is set now?

Extract Try/Catch Blocks

Just like mentioned before about Blocks and Indenting, the try/catch blocks are ugly because they mix error handling with business code. It's better to extract the bodies of the try and catch into functions of their own.

Don't Repeat Yourself

Duplication may be the root of all evil in software. Use OO wisely as a strategy for eliminating duplication.

Conclusion

Master programmers think of systems as stories to be told rather than programs to be written.

Chapter 4: Comments

Don't comment bad code - rewrite it!

Nothing can be more damaging as an old comment that propagates lies and misinformation.
The older the comment is, the farther away it is from the code it describes, the more likely it is to be plain wrong.

Comments Do Not Make Up for Bad Code

A good reason to write comments is bad code. It's used when the code is a mess. Someone thinks "Ah, I'd better comment that!". What should be done? Better clean it!

Explain Yourself in Code

This deserves just an example.

// Check if an employee is eligible for full benefits
if ((emplioyee.flags && HOURLY_FLAG) && (employee.age > 65)) {...}

The above comment is not necessary. Just change the condition to be a meaningful named function.

if (employee.isEligibleForFullBenefits) {...}

Bad Comments

Most comments fall into this category. Usually, they are excuses for poor code or justification for insufficient decisions, amounting to little more than the programmer talking to himself.

Any comment that required you to look into another module to check the meaning of that comment has failed to communicate to you and is not worth the bits it consumes.

Don't Use a Comment When You Can Use a Function or a Variable

This title speaks for itself. As seen above, replace comments with a meaningful named function or variable! #profit

Chapter 5: Formatting

The Purpose of Formatting

Code formatting is too important to be ignored. It is about communication, and communication is the professional developer's first order of business.

Vertical Formatting

How big should a source file be?

There are many source analyzers that can try to help when managing the size of classes and methods. The book Clean Code brings a graph with the average file length in different projects. It is interesting to have an idea of the files with the smaller and biggest length, even when it is for a personal project.

Vertical Openness Between Concepts

Nearly all code is read left to right and top to bottom. Each line represents an expression or clause, and each group of lines represents a complete thought. Those thoughts should be separated from each other with blank lines.

Vertical Distance

Concepts that are closely related should be kept vertically closed to each other.

Dependent Functions. If one function calls another, they should be vertically close and the caller should be above the callee, if at all possible.

Vertical Ordering

In general, we want function call dependencies to point in the downward direction. That is, a function that is called should be below a function that does the calling. This creates a once flow down the source code module from high level to low level.

As in newspaper articles, we expect the most important concepts to come first, and we expect then to be expressed with the least amount of polluting detail. We expect the low-level details to come last. This allows us to skim the source files, getting the gist from the first few functions, without having to immerse ourselves in the details.

Chapter 6: Objects and Structures

Hiding implementation is not a matter of putting a layer of functions between the variables. Hiding the implementation is about abstractions! A class does not simply push its variables out through getters and setters. Rather it exposes abstract interfaces that allow its users to manipulate the essence of data, without having to know its implementation.

Data/Object Anti-Symmentry

Object hide their data behind abstractions and expose functions that operate on that data. Data structure expose their data and have no meaningful functions.

Below the dichotomy between objects and data structures:

Procedural code (code using data structures) makes it easy to add new functions without changing the existing data structures. OO code, on the other hand, makes it easy to add new classes without changing existing functions.

And...

Procedural code makes it hard to add new data structures because all the functions must change. OO code makes it hard to add new functions because all the classes must change.

Mature programmers know that the idea that everything is an object is a myth. Sometimes you really do want simple data structures with procedures operating on them.

The Law of Demeter

The method should not invoke methods on objects that are returned by any of the allowed functions. In other words, talk to friends, not to strangers.
The following code appears to violate the Law of Demeter (among other things) because it calls the getScratchDir() function on the return of getOptions() and then calls getAbsolutePath() on the return of value of getScratchDir().

final String outputDir = ctxt.getOptions().getScratchDir().getAbsolutePath();

The Wrecks

This kind of code is often called a train wreck because it looks like a bunch of coupled train cars. Chains of calls like this are generally considered to be sloppy style and should be avoided. It's usually best to split them up as follows:

Options opts = ctxt.getOptions();
File scratchDir = opts.getScratchDir();
final String outputDir = scratchDir.getAbsolutePath();

Whether it is a violation of Demeter depends on whether or not ctxt, Options, and ScratchDir are objects or data structures. If they are objects, then their internal structure should be hidden rather than exposed, and so knowledge of their innards is a clear violation of the Law of Demeter. On the other hand, if ctxt, Options, and ScratchDir are just data structures with no behaviour, then they naturally expose their internal structure, and so Demeter does NOT apply.
The use of accessor functions confuses the issue. If the code had been written as follows, then we probably wouldn't be asking about Demeter violations.

final String outputDir = ctxt.options.scratchDir.absolutePath;

The issue would be a lot less confusing if data structures simply had public variables and no functions, whereas objects had private variables and public functions.

Chapter 7: Error Handling

Error handling is important, but if it obscures logic, it's wrong.

Use Unchecked Exception

When checked exceptions were introduced in Java, they seemed like a great idea. The signature of every method would list all possible exceptions it could throw to its caller. The code would not compile if the signature of the method didn't match what your code could do.
They were thought to be a great idea, but now it's known as not. Other languages like C#, C++, Python or Ruby don't have checked exception and they work well without it.

So, what's the price to use checked exceptions?
It violates the Open/Closed Principle, where software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification.
If you throw a checked exception from a method in your code and the catch is 3 levels above, you must declare the exception in the signature of the method between you and the catch. It means that a change at a low level of the software can force signature changes on many higher levels.
Also, encapsulation is broken because all functions in the path of a throw must know about details of that low-level exception.

Provide Context with Exceptions

Create informative error messages and pass them along with your exceptions. If you are logging in your applications, pass along enough information to be able to log the error in your catch.

Do NOT Return Null

When we return null, we are essentially creating work for ourselves and foisting problems upon our callers. All it takes is one mission null check to send an application spinning out of control.

Do NOT Pass Null

Returning null from methods is bad, but passing null into methods is worse. Unless you're working with an API which expects you to pass null, you should avoid passing null in your code whenever possible.

Passing null into a method can possibly result in a NullPointerException wherever a null check is missing.

In most programming languages there is no good way to deal with a null that is passed by a caller accidentally. Because this is the case, the rational approach is to forbid passing null by default. When you do, you can code with the knowledge that a null in an argument list is an indication of a problem, and end up with far fewer careless mistakes.

Chapter 8: Boundaries

Third Parties

Third-party implementation (aka libraries) has 2 important goals:

it must have broad applicability (intention of providers).
it must implement specific needs (intention of the users).

The users may want to use a library or API, but not have all its methods available to their developers.
The book brings an example of the java.util.Map, which demonstrate that the Map API can have many methods available to who uses it, even if that's not the intention of the user to provide them.
Let's say no one should remove an object from the Map, what could be done, since the method remove(Object key) is available?
One suggestion is to encapsulate the use of the API in a class containing a private variable and to expose the desired methods by creating proxy methods.
Example:

public class Products {

    private Map<Product> products = new HashMap<Product>();

    public Product getById(String: id) {
        return products.get(id)
    }

    public void delete(String: id) {
        products.remove(id)  // fell free to add a try/catch and throw a proper exception if the item does not exist
    }

}

Code that does not exist

When something needs to be developed to be integrated with a code that does not yet exist, it's possible to create a fake interface to mock the API with the external code. It helps to test the code and to keep the internal consistency, without impact from methods and variables names or method signatures.

Clean Boundaries

Boundaries with third-party must be coded in very few places to refer them. One of the advantages is that the code in the boundary has fewer maintenance points when the third-party changes.

Chapter 9: Tests

Test code must be implemented with the same standards of quality than production code. There must not be any license to break the rules in tests codebase.

Having dirty tests is a horrible practice. The problem is that tests must change as the production code evolves. The dirtier the tests, the harder they are to change.

A code must have tests to ensure that changes in the production code work as expected. If there aren't good tests, their maintainability becomes difficult, the number of tests decreases, the bugs may increase after releases, fear of changes make users believe that they will bring more harm than good. The code starts to rot.

What's make a good test?

Readability, readability, readability.

Single concept per test

We don't want long test functions that go testing one miscellaneous thing after another.

Each assertion deserves one test. If that's not possible, they should be grouped by the concept.

F.I.R.S.T.

F: Fast
I: Independent
R: Repeatable
S: Self-Validating (output must be true/false, pass/fail - nothing in between)
T: Timely (written before production code)

Chapter 10: Classes

Encapsultation

Variables and utility functions must be kept private.
If a test in the same package must call a function, it can be made protected or package scope.

Small

The first rule of classes is that they should be small. The second rule of classes is that they should be smaller than that.

A class must have only one responsibility.

We should be able to write a brief description of the class in about 25 words, without using "if", "and", "or" or "but".

The Single Responsibility Principle

The Single Responsibility Principle states that a class or module should have one, and only one, reason to change.

Classes should have one responsibility - one reason to change.
Trying to identify responsibilities (reasons to change) often helps us recognise and create better abstractions in our code.

... SRP is often the most abused class design principle.

Maintaining separation of concerns is just as important in our programming activities as it is in our programs.

Question:
Do you want your tools organised into toolboxes with many small drawers each containing well-defined and well-labelled components? Or do you want a few drawers that you just toss everything into?

Coehision

Cohesion must be high. When cohesion is high, it means that the methods and variables of the class are co-dependent and hang together as a logical whole.

Modifications

Any modifications in the class have the potential of breaking other code in the class. It must be fully retested.

Coupled vs Decoupled

Coupled code: code connected to many others - where it depends on others and others depend on it.
Decoupled code: code isolated.

A system decoupled will be more flexible and promote more reuse. The lack of coupling means that the elements of our system are better isolated from each other and from change. This isolation makes it easier to understand each element of the system.

Chapter 11: Systems

It is a myth that we can get systems "right the first time".
Software systems are unique compared to physical systems. Their architectures can grow incrementally, IF we maintain the proper separation of concerns.

BDUF

BDUF: Big Design Up Front.

The book mentions it is not necessary to do BDUF and that it is even harmful because It inhibits adapting to change, due to the psychological resistance to discarding prior effort and because of the way architecture choices influence subsequent thinking about design.

In a sufficiently large system, whether it is a city or a software project, no one person can make all the decisions.

Use standards wisely

Standards must add demonstrable value. Teams cannot be obsessed with various strongly hyped standards and lose focus on implementing value for their customers.

Chapter 12: Emergence

Kent Beck defines "simple" design with the following rules:

Runs all the tests.
Contains no duplication.
Expresses the intent of the programmer.
Minimises the number of classes and methods.

The rules are in order of importance.

Run all tests

Making a system testable pushes developers toward a design where classes are small and single purpose. It's easier to test classes that conform to the SRP. The more tests we write, the more we'll continue to push toward things that are simpler to test.

The fact is that we have these tests eliminates the fear that cleaning up the code will break it!

We can apply anything from the body of knowledge about good software design. We can increase cohesion, decrease coupling, separate concerns, modularise concerns, shrink our functions and classes, choose better names, etc.

No duplication

Duplication is the primary enemy of a well-designed system. It represents additional work, additional risk, and additional unnecessary complexity.

Expressive

When software becomes big and complex, it demands more time from a developer to understand it, and there is even a greater opportunity to misunderstanding it.

The clearer the author can make the code, the less time others will have to spend understanding it.

Below what can reduce defects and shrink the cost of maintenance.

You can express yourself by choosing good names.
You can express yourself by keeping your functions and methods small.
You can express yourself by using standard nomenclature.
You can have well-written unit tests that are expressive.

Spend a little time with each of your functions and classes. Choose better names, spit large functions into smaller functions, and generally just take care of what you've created. Care is a precious resource.

Minimal Classes and Methods

The goal is to keep the overall system small the functions and classes are also kept small. This is the lowest priority of the four rules od Simple Design.

So, although it's important to keep class and function count low, it's more important to have tests, eliminate duplication, and express yourself.

Chapter 13: Concurrency

Writing clean concurrent programs is hard - very hard.

Correct concurrency is complex, even for simple problems.

Concurrency Defense Principles - SRP

The Single Responsibility Principle states that a given method/class/component should have a single reason to change. Concurrency design is complex enough to be a reason to change in its own right and therefore deserves to be separated from the rest of the code.

Recommendation: Keep your concurrency-related code separated from other code.

Know Your Library

A few things to consider when writing threaded code in Java:

Use the provided thread-safe collections.
Use the executor framework for executing unrelated tasks.
Use non-blocking solutions when possible.
Several library classes are not thread safe.

Recommendation 1: Review the classes available to you. In case of Java, become familiar with the java.util.concurrent, java.util.concurrent.atomic and java.util.concurrent.locks.

Recommendation 2: Learn the basic algorithms and understand their solutions.

Synchronized blocks

"... if there is more than on synchronized method on the same shared class, then your system may be written incorrectly"

The synchronized keyword introduces a lock. All sections of code guarded by the same lock are guaranteed to have only one thread executing through them at any given time.

Recommendation: Keep your synchronized sections as small as possible.

Testing Threaded Code

Write tests that have the potential to expose problems and then run them frequently, with different programmatic configurations and system configurations and load. If tests ever fail, track down the failure. Don't ignore a failure just because the tests pass on a subsequent run.

Get Your Nonthreaded Code Working First

This may seem obvious, but it doesn't hurt to reinforce it.
So not try to chase down non-threading bugs and threading bugs at the same time. Make sure your code works outside of threads first.

Chapter 14: Successive Refinement

This chapter of the book is a case study. Below there are a few aspects I thought interesting mentioning.

To write clean code, you must first write dirty code and then clean it.
Much of good software design is simply about partitioning - creating appropriate places to put a different kind of code. This separation of concerns makes the code much simpler to understand and maintain.
Programmers who satisfy themself with merely working code are behaving unprofessionally.

Nothing has more profound and long-term degrading effect upon a development project than bad code.

Of course, bad code can be cleaned up. But it's very expensive. As code rots, the modules insinuate themselves into each other, creating lots of hidden and tangled dependencies.
On the other hand, keeping the code clean is relatively easy. If you made a mess in a module in the morning, it is easy to clean up in the afternoon.

The solution is to continuously keep your code as clean and simple as it can be. Bevel let the code rot.

Chapter 15: JUnit Internals

This chapter is somehow similar to Chapter 14. The JUnit framework code is analysed by the book author and criticised.
Is not relevant to write down all the suggestions he's making to change JUnit code.

Below the conclusion from the author of the book:

... we have satisfied the Scout Boy Rule. We have left this module a bit cleaner than we found it. Not that it wasn't clean already. The authors (of JUnit) had done an excellent job with it. Bit no module is immune from improvement, and each of us has the responsibility to leave the code a little better than we found it.

Chapter 16: Refactoring SerialDate

This chapter is very similar to Chapter 15, but the code inspected was from the class SerialDate. Just like in the previous chapter, it shows areas of potential improvement in the code.

Chapter 17: Smells and Heuristics

This chapter is a summary of many things seen throughout the book.

Comments

Obsolete Comment

Comments get old quickly. It is best not to write a comment that will become obsolete.

Redundant Comment

Comments should say things that the code cannot say for itself.

Poorly Written Comment

If you're gonna write a comment, take the time to make sure it is the best comment you can write.

Commented-Out Code

That code sits there and rots, getting less and less relevant with every passing day. Commented-out code is an abomination.
When you see commented-out code, delete it. Don't worry, the source code control system still remembers it.

Environment

Build Required More Than One Step

You should be able to check out the system with one simple command and then issue one simple command to build it.

Tests Require More Than One Step

Being able to run all tests is so fundamental and so important that it should be quick, easy and obvious to do.

Functions

Too Many Arguments

Functions should have a small number of arguments. More than three is very questionable.

Output Arguments

Output arguments are counterintuitive.

Flag Arguments

Boolean arguments loudly declare that the function does more than one thing.

Dead Function

Keeping dead code around is wasteful.

General

Multiple Languages in One Source Code

The idea is for a source file to contain one, and only one, language. Realistically, we will probably have to use more than one. But we should take pains to minimise both the number and extent of extra languages in our source files.

Obvious Behaviour Is Unimplemented

When an obvious behaviour is not implemented, readers and users of the code can no longer depend on their intuition about function names.

Override Safeties

Turning off certain compiler warning (or all warnings) may help you get the build to succeed but at all risk of endless debugging sessions. Turning off failing tests and telling yourself you're getting them to pass later is as bad as pretending your credit cards are free money.

Duplication

This is one of the most important rules in this book and it should be taken very seriously. Andy Hunt and Dave Thomas call it DRY (Don't Repeat Yourself). Kent Beck made it one of the core principles of Extreme Programming and called it: "Once and only once".

Every time you see duplication in the code, it represents a missed opportunity for abstraction.

Still more subtle are the modules that have similar algorithms, but that don't share similar lines of code. This is still duplication and should be addressed by using the TEMPLATE METHOD it STRATEGY pattern.

Base Classes Depending on Their Derivatives

The most common reason for partitioning concepts into base and derivative classes is so that the highest level base class concepts can be independent of the lower level derivative class concepts.

Too Much Information

A well-defined interface does not offer very many functions to depend upon, so coupling is low. A poorly defined interface provides lots of functions that you must call, so coupling is high.

Dead Code

Dead code is code that isn't executed. After a while, it starts to smell. That happens because dead code isn't updated when the designs change. It still compiles but does not follow the newer conventions or rules.

Inconsistency

If you do something a certain way, do all similar things in the same way.

Clutter

Clutters are useless things in the code (my words).

Functions that are never called.
Variables that are never used.
Comments that are useless.
Default constructor with no implementation.

All these things are clutters and should be removed.

Missplaced Responsibility

One of the most important decisions a software developer can make is where to put code.

Code should be placed where a reader would naturally expect it to be.

Innapropriate Static

If you want a function to be static, make sure that there is no chance that you'll want it to behave polymorphically.

Use Explanatory Variables

One of the more powerful ways to make a program readable is to break the calculations up into intermediate values that are held in variables with meaningful names.

Function Names Should Say What They Do

In the code below:

Date newDate = date.add(5);

What do you expect this code to add to the date?

5 seconds
5 minutes
5 hours
5 days
5 weeks
5 months
5 years
...

Variable names must be named to have an easy understanding for the reader.
A better name for the code above would be name the method addDaysTo(n) or increaseByDays(n).

Prefer Polymorphism to If/Else or Switch/Case

I use the following "ONE SWITCH" rule. There may be no more than one switch statement for a given type selection. The cases in that switch statement must create polymorphic objects that take the place of other such switch statements in the rest of the system.

Follow Standard Conventions

Every team should follow a coding standard based on common industry norms.

Replace Magic Numbers with Named Constants

This is probably one of the oldest rules in software development.

Be Precise

When you make a decision in your code, make sure you make it precisely. Know why you have made it and how you will deal with any exceptions.

Structure over Convention

Naming conventions are good, but they are inferior to structures that force compliance.

Encapsulate Conditionals

For example:
if (shouldBeDeleted(timer)) {...}
is preferable to
if (timer.hasExpired() && !timer.isRecurrect()) {...}

Avoid Negative Conditionals

For example:
if (buffer.shouldCompact())
is preferable to
if (!buffer.shouldNotCompact())

Functions Should Do One Thing

public void pay() {
  for (Employee e: employees) {
    if (e.isPayday()) {
      Money pay = e.calculatePay();
      e.deliverPay(pay);
    }
  }
}

should be split in

public void pay() {
  for (Employee e: employees) {
    payIfNecessary(e);
  }
}

private void payIfNecessary(Employee e) {
  if (e.isPayday) {
    calculateAndDeliverPay(e);
  }
}

private calculateAndDeliverPay(Employee e) {
  Money pay = e.calculatePay();
  e.deliverPay(pay);
}

Encapsulate Boundary Conditions

Observe that the snippet level + 1 appears twice in the code below.

if (level + 1 < tags.length) {
  parts = new Parse(body, tags, level + 1, offset + endTag)
}

This is a boundary condition that should be encapsulated into a variable.

int nextLevel = level + 1;
if (nextLevel < tags.length) {
  parts = new Parse(body, tags, nextLevel, offset + endTag)
}

Java

Constants versus Enums

Enums exist in Java for a while. Use them. Don't keep using the old trick public static void final ints. The meaning of int's can get lost. The meaning of enum's cannot because they belong to an enumeration that is named.

Names

Choose Descriptive Names

Make sure the name is descriptive.
This is not just a feel-good recommendation. Names in software are 90 percent of what makes software readable.

Names are too important to treat carelessly.

Tests

Insufficient Tests

A test suite should test everything that can possibly break.

Don't Skip Trivial Tests

They are easy to write and their documentary value is higher than the cost to produce them.

Exhaustively Test Near Bugs

When you find a bug in a function, it is wise to do an exhaustive test if that function. You'll probably find that the bug was not alone.

Tests Should Be Fast

A slow test is a test that won't get run.

Final Note

Professionalism and craftsmanship come from values that drive disciplines.

Forem: Everton Schneider

Book: How To Lie with Statistics - Bias

1. The Sample with Built-in Bias

The Psychiatrist

The Literary Digest's famed fiasco (Elections 1936)

Book: Eat That Frog: 21 Ways to Stop Procrastinating

1. Set the table

2. Plan Every Day in Advance

3. Apply the 80/20 Rule to Everything

4. Consider the consequences

5. Practice Proactive Procrastination

6. Use the ABCDE Method Continually

7. Focus on Key Result Areas

8. Apply the Law of Three

9. Prepare Thoroughly Before You Begin

10. Take it One Oil Barrel at a Time

11. Upgrade Your Key Skills

Three Steps to Mastery

12. Identify Your Key Constraints

13. Put the Pressure on Yourself

14. Motivate Yourself into Action

Control Your Inner Dialogue

Develop Mental Positive Attitude

15. Technology Is a Terrible Master

16. Technology Is a Wonderful Servant

17. Focus Your Attention

Developing an Addiction

The Multitasking Illusion

The Proven Solutions

Double Your Productivity

18. Slice and Dice the Task

19. Create Large Chunks of Time

20. Develop a Sense of Urgency

Build a Sense of Momentum

Do It Now!

21. Single Handle Task

Conclusion

What does an automation tester do?

What do we try to say we do?

What are we really doing?

Tips to avoid unstable tests in Selenium

TL;DR;

1. Treat test code in the same way you treat development code

Do code review

Request for best practices

Use Page Objects

Pages

Components

2. Create bugs for all issues

3. Implement custom waiting

Implicit Wait

Explicit Wait

WebDriverWait and ExpectedConditions

4. It's not all

System Design: Basic approaches

TL;DR;

Introduction

Types of Scalability

Vertical Scale

Horizontal Scale

In old (or very old) days

Single Server Design

Application Server + Database Server Design

Adding a Load Balancer

Database Normalization

Example

Normalized database

Table INVOICE

Table CLIENT

Denormalized database

Table Invoice

Conclusion

How to run Playwright tests with Selenium Grid versions after `4.5`

TL;DR;

Problem

History

Fix (Workaround)

Final consideration

Optimising parallel tests in Selenium Grid

Implement automated tests with a time constraint

`WebDriverWait` and `ExpectedConditions`

Table `INVOICE`

Table `CLIENT`

Table `Invoice`