Forem: ashleygraf_

The software tester as "Designated Ignoramus"

ashleygraf_ — Sun, 02 Feb 2025 01:59:56 +0000

As a software tester, it's basically impossible to test the software you test like a user like, except maybe at the start of your time with a company, when it's still new to you. But I maintain that you can still get close.

I use personas to try and achieve this. In particular, these ones:

User is a traveller with a different native language
Very very tired user
Drunk user
User with an urgent deadline and no time to read the manual

I then think of something even shorter than test scenarios: goals, which I derive from user stories.

For example, as a user I want to

Book tickets to the next destination
Withdraw money from the bank
Pay for my doctor's appointment
Arrange accommodation
Book a museum visit
Get a week travel pass

Those are your goals.

To achieve these goals, you don't read the code, you don't look at the Figma, and you certainly don't create test steps.

All you, the user, knows, is what is right in front of you, or maybe not even that. You may not understand the language, or the screen may be hazy in front of you. Your fingers might slip. You might press the wrong button by accident.

It's a test of how closely the software follows design patterns, how well it handles errors and mis-steps, how much sense each screen makes, and how logically the steps flow.

In this way, you can take a first pass to find "obvious errors".

Creating test data with Claude Sonnet 3.5

ashleygraf_ — Fri, 26 Jul 2024 05:42:15 +0000

Chatbot: Claude Sonnet 3.5
Testbed: the US Weather API.

Task:

Try out different queries and see how usable the result is
Develop my skills using AI chatbots

Experiments:

ran the same query over and over on the same chatbot
ran the same query on different chatbots
made queries increasingly specific
tried different ways to make the queries more specific

Focus:

Generating test data

I am also exploring using AI with test planning and automation, and I hope to share the results with you in later weeks.

I encountered a few issues. For example:

requesting a list of test data that DIDN'T match a regex, the very first item DID match that regex (although the rest did not).
requesting a list of test data that DIDN'T match a regex, two items did match the regex
the Claude Sonnet training data was less extensive than I expected, and results got pretty deterministic surprisingly fast.

References:
I took some inspiration from prompt engineering examples I've seen, including from Mark Winteringham's book, Software Testing with Generative AI, as well as my own attempts at creating some useful queries.

Test Data

Functions

Create a JavaScript function that returns an array of UTC DateTimes from the current date to 7 days ago, at 24-hour intervals, starting with mid-day of the current date.

function getLastWeekDatesFromMidDay() {
  const dates = [];
  const now = new Date();

  // Set the time to 12:00:00 UTC for today
  now.setUTCHours(12, 0, 0, 0);

  for (let i = 0; i < 8; i++) {
    const date = new Date(now);
    date.setUTCDate(now.getUTCDate() - i);
    dates.push(date.toISOString().slice(0, 19) + 'Z');
  }

  return dates.reverse();

Regular expressions

Can you create a list of dates in date formats that do not match yyyy-mm-dd hh:mm:ss for the DateTime 2020-05-14T05:40:44Z?

invalidDateFormats": [
    "2020-05-14", 
    "2020-05-14T", 
    "2020-05-14T5",
    "2020-05-14T05", 
    "2020-05-14T05:40",
    "2020-05-14T05:40:4",
    "14/05/2020 05:40:44",
    "05/14/2020 05:40:44",
    "2020/05/14 05:40:44",
    "14-05-2020 05:40:44",
    "05-14-2020 05:40:44",
    "20200514054044",
    "14052020054044",
    "May 14, 2020 05:40:44",
    "14 May 2020 05:40:44",
    "Thu, 14 May 2020 05:40:44",
    "2020-05-14 05:40:44",
    "05:40:44 14-05-2020",
    "1589433644",
    "20-05-14 05:40:44",
    "2020.05.14 05:40:44",
    "14/05/20 05:40:44",
    "05/14/20 05:40:44"

Can you give a list of text that does not match this regular expression? [\"AK\",\"AL\",\"AM\",\"AN\",\"AR\",\"AS\",\"AZ\",\"CA\",\"CO\",\"CT\",\"DC\",\"DE\",\"FL\",\"FM\",\"GA\",\"GM\",\"GU\",\"HI\",\"IA\",\"ID\",\"IL\",\"IN\",\"KS\",\"KY\",\"LA\",\"LC\",\"LE\",\"LH\",\"LM\",\"LO\",\"LS\",\"MA\",\"MD\",\"ME\",\"MH\",\"MI\",\"MN\",\"MO\",\"MP\",\"MS\",\"MT\",\"NC\",\"ND\",\"NE\",\"NH\",\"NJ\",\"NM\",\"NV\",\"NY\",\"OH\",\"OK\",\"OR\",\"PA\",\"PH\",\"PK\",\"PM\",\"PR\",\"PS\",\"PW\",\"PZ\",\"RI\",\"SC\",\"SD\",\"SL\",\"TN\",\"TX\",\"UT\",\"VA\",\"VI\",\"VT\",\"WA\",\"WI\",\"WV\",\"WY\"]"

[
  "AK",
  "ak",
  "Ak",
  "aK",
  "AAK",
  "AKA",
  "A K",
  "AK ",
  " AK",
  "AK,",
  "AK.",
  "AK-",
  "AK_",
  "(AK)",
  "[AK]",
  "{AK}",
  "\"AK\"",
  "AK1",
  "1AK",
  "AKAK",
  "AL-AK",
  "Alaska",
  "ALASKA",
  "USA",
  "US",
  "United States",
  "123",
  "AB",
  "ZZ",
  "XX",
  "AA",
  "ak,al",
  "AK AL",
  "AK.AL",
  "AK;AL",
  "A-K",
  "a.k.",
  "KA",
  "_AK",
  "AK_",
  "-AK",
  "AK!",
  "!AK",
  "AK?",
  "?AK",
  null,
  "",
  " ",
  "\\AK",
  "AK\\",
  "AK\nAL",
  "AK\tAL"
]

Can you give me test data that does not match the regular expression in the line below, and can you provide it in JSON format in an object named "invalidZoneCodes"? "^(A[KLMNRSZ]|C[AOT]|D[CE]|F[LM]|G[AMU]|I[ADLN]|K[SY]|L[ACEHMOS]|M[ADEHINOPST]|N[CDEHJMVY]|O[HKR]|P[AHKMRSWZ]|S[CDL]|T[NX]|UT|V[AIT]|W[AIVY]|[HR]I)[CZ]\d{3}$"

  "invalidZoneCodes": [
    "AKC1234",
    "NYC123",
    "CAZ12",
    "CAX123",
    "ca123",
    "AK 123",
    "AKCZ123",
    "AKC12A",
    "AKC-123",
    "AKZC123",
    "123AKC",
    "AKC",
    "123",
    "AKCA23",
    "AK123C",
    "AKC1Z3",
    "AKZ000",
    "RIC1234",
    "HIC1234",
    "AK123Z",
    "UTCABC",
    "VAZ1234",
    "WYC12",
    "ORC12Z",
    "MNC123C"
  ]

Can you give me test data that does match the regular expression in the line below, and can you provide it in JSON format in an object named "validZoneCodes"? "^(A[KLMNRSZ]|C[AOT]|D[CE]|F[LM]|G[AMU]|I[ADLN]|K[SY]|L[ACEHMOS]|M[ADEHINOPST]|N[CDEHJMVY]|O[HKR]|P[AHKMRSWZ]|S[CDL]|T[NX]|UT|V[AIT]|W[AIVY]|[HR]I)[CZ]\d{3}$"

"validZoneCodes": [
    "AKZ123",
    "CAC456",
    "DEZ789",
    "FLZ012",
    "GAC345",
    "ILC678",
    "KSZ901",
    "LAC234",
    "MNC567",
    "NJZ890",
    "OHC123",
    "PAZ456",
    "SDC789",
    "TXZ012",
    "UTC345",
    "VAZ678",
    "WAC901",
    "HIZ234",
    "RIZ567",
    "AZC890",
    "COZ123",
    "DCZ456",
    "GMC789",
    "IAZ012",
    "KYC345"
  ]

Classification

Can you create a map of the parameter keys and values used in this API response? After that, can you generate test data for those keys? Your purpose is to check the integrity of the data input by the weather stations, as the migration has not yet completed, and some agencies are using ENUMs for some values, and others are still using plain text. What data would you classify as valid, and what data would you classify as invalid, considering the training data. It should cover valid and invalid cases, with a focus on big and little. Please take into account the fact that null and empty fields are not invalid.

Classification of Valid and Invalid Data:
Valid Data:

ENUM values and their plain text equivalents
Numeric values with appropriate units
Null values
Empty arrays

Invalid Data:

Values not matching ENUM or expected plain text
Numeric values with incorrect units or format
Non-numeric values for numeric fields

Test Data:

windThreat:

Valid (Big): ["RADAR INDICATED", "OBSERVED", null, "", "POSSIBLE"]
Valid (Little): ["RADAR INDICATED"], [], null
Invalid (Big): ["RADAR_INDICATED", "STRONG", "WEAK", "UNKNOWN", 123]
Invalid (Little): ["INVALID"]


maxWindGust:

Valid (Big): ["70 MPH", "100 KPH", "50 MPS", null, ""]
Valid (Little): ["1 MPH"], [], null
Invalid (Big): ["500 MPH", "-10 MPH", "100", "STRONG", "70MPH"]
Invalid (Little): ["GUSTY"]


hailThreat:

Valid (Big): ["RADAR INDICATED", "OBSERVED", null, "", "POSSIBLE"]
Valid (Little): ["RADAR INDICATED"], [], null
Invalid (Big): ["RADAR_INDICATED", "LARGE", "SMALL", "UNKNOWN", 123]
Invalid (Little): ["INVALID"]


maxHailSize:

Valid (Big): ["0.88", "1.00", "2.75", null, ""]
Valid (Little): ["0.25"], [], null
Invalid (Big): ["10.00", "-1.00", "GOLF BALL", "LARGE", "0,88"]
Invalid (Little): ["SMALL"]


thunderstormDamageThreat:

Valid (Big): ["CONSIDERABLE", "NONE", "MODERATE", null, ""]
Valid (Little): ["NONE"], [], null
Invalid (Big): ["VERY_CONSIDERABLE", "UNKNOWN", "LIGHT", "HEAVY", 123]
Invalid (Little): ["INVALID"]


tornadoDetection:

Valid (Big): ["RADAR INDICATED", "OBSERVED", null, "", "POSSIBLE"]
Valid (Little): ["RADAR INDICATED"], [], null
Invalid (Big): ["RADAR_INDICATED", "STRONG", "WEAK", "UNKNOWN", 123]
Invalid (Little): ["INVALID"]

You are a software tester testing data integrity for a weather data API. Can you suggest a set of test data for the value "maximum hail size"? It should cover positive and negative cases. The positive cases should include edge cases.

  "maximumHailSize": {
    "validCases": [
      0.00,
      0.25,
      0.50,
      0.75,
      1.00,
      1.25,
      1.50,
      1.75,
      2.00,
      2.50,
      3.00,
      4.00,
      4.50,
      5.00,
      6.00,
      7.00,
      8.00
    ],
    "edgeCases": [
      0.01,
      0.10,
      9.99,
      10.00
    ],
    "invalidCases": [
      -0.01,
      -1.00,
      10.01,
      15.00,
      100.00,
      0.001,
      0.999,
      "0",
      "1.5",
      "",
      " ",
      null,
      undefined,
      true,
      false,
      [],
      {},
      Infinity,
      -Infinity,
      NaN,
      1.5e2,
      "trace",
      "1.5 inches",
      "1,50",
      "1/2",
      "ping pong",
      "golf ball"
    ]
  }

You are a software tester testing data integrity for a weather data API. You are generating tests for the nightly run checking that the data entered by various weather stations is valid. Can you suggest a set of test data for the value "maximum hail size"? It should cover positive, negative, edge, creative, big, little, invalid, exploratory, and boundary-related cases. It should also check for external interference with the system in a non-destructive manner.

  "maximumHailSize": {
    "validCases": [
      0.00,
      0.25,
      0.50,
      1.00,
      1.75,
      2.50,
      4.00,
      6.00,
      8.00
    ],
    "edgeCases": [
      0.01,
      7.99,
      8.00
    ],
    "bigCases": [
      100.00,
      1000.00,
      9999.99
    ],
    "littleCases": [
      0.001,
      0.009,
      0.01
    ],
    "invalidCases": [
      -0.01,
      -1.00,
      8.01,
      "0",
      "",
      " ",
      null,
      undefined,
      true,
      false,
      [],
      {},
      Infinity,
      -Infinity,
      NaN
    ],
    "creativeInputs": [
      "golf ball",
      "baseball",
      "grapefruit",
      "1,50",
      "1/2",
      "1.5 inches",
      "trace",
      "melted",
      "unknown"
    ],
    "exploratoryInputs": [
      1.5e2,
      "0xFF",
      "0b1010",
      "0o777",
      "\u0037\u002E\u0035",
      "    7.5    ",
      "7.5\n",
      "7.5\t",
      "+7.5",
      "７.５"
    ],
    "boundaryTests": [
      -0.01,
      0.00,
      0.01,
      7.99,
      8.00,
      8.01
    ],
    "externalInterference": [
      "<script>alert('XSS')</script>",
      "'; DROP TABLE weather_data; --",
      "%00%3Cscript%3Ealert(%27XSS%27)%3C/script%3E",
      "../../../etc/passwd",
      "{{7*7}}",
      "${7*7}",
      "#{7*7}",
      "=7+7",
      "@{7*7}"
    ]
  }

I want you to act as an expert software tester who works on creating test data to provide comprehensive test data coverage.
I want you to generate tests for the nightly run checking that the data entered by various weather stations is valid.
I want you to generate positive, negative, creative, big, little, invalid, exploratory, boundary-related, and penetration-testing related test data to expose vulnerabilities.

Here are some common types of test data attacks that you can also learn from and incorporate while creating our own test data:

Paths/Files: Long Name (>255 chars), Special Characters in Name, Non-Existent characters, Character with No Space.
Time and Date: Crossing Time Zones, Leap Days, Always Invalid Days (Feb 30, Sept 31), Feb 29 in Non-Leap Years, Different Formats (June 5, 2001; 06/05/2001; 06/05/01; 06-05-01; 6/5/2001 12:34), Internationalisation dd.mm.yyyy, mm/dd/yyyy, am/pm, Daylight Savings Changeover.
Numbers: 0, 32768 (215), 32769 (215 + 1), 65536 (216), 65537 (216 +1), 2147483648 (231), 2147483649 (231 + 1), 4294967296 (232), 4294967297 (232 + 1), Scientific Notation (1E-16), Negative, Floating Point/Decimal (0.0001), With Commas (1,234,567), European Style (1.234.567,89).
Strings: Long (255, 256, 257, 1000, 1024, 2000, 2048 or more characters), Accented Chars (àáâãäåçèéêëìíîðñòôõöö, etc.), Asian Characters Common Delimiters and Special Characters ( “ ‘ ` | / \ , ; : & < > ^ * ? Tab ), Leave Blank, Single Space, Multiple Spaces, Leading Spaces, SQL Injection ( ‘select * from customer), Emojis.

Provide the results in json format.
I want you to generate as many cases as you can think of for each test case type.
These are the variable names to create test data for: "maximum hail size"

{ "type": "Positive", "value": 1.0, "description": "Valid small hail size in inches" }, { "type": "Positive", "value": 2.5, "description": "Valid medium hail size in inches" }, { "type": "Positive", "value": 4.75, "description": "Valid large hail size in inches" }, { "type": "Positive", "value": 8.0, "description": "Valid extreme hail size in inches (near record)" }, { "type": "Boundary", "value": 0.0, "description": "Minimum possible size" }, { "type": "Boundary", "value": 0.01, "description": "Just above minimum size" }, { "type": "Boundary", "value": 9.99, "description": "Very large, but still plausible" }, { "type": "Boundary", "value": 10.0, "description": "Maximum plausible size" }, { "type": "Negative", "value": -1.0, "description": "Invalid negative size" }, { "type": "Negative", "value": -0.01, "description": "Small negative value" }, { "type": "Invalid", "value": 100.0, "description": "Unrealistically large size" }, { "type": "Invalid", "value": 1000.0, "description": "Absurdly large size" }, { "type": "Invalid", "value": "abc", "description": "Non-numeric input" }, { "type": "Invalid", "value": "1,5", "description": "Comma instead of decimal point" }, { "type": "Invalid", "value": "1.5.5", "description": "Multiple decimal points" }, { "type": "Invalid", "value": "1.5\"", "description": "Unit included in the value" }, { "type": "Invalid", "value": "1/2", "description": "Fraction instead of decimal" }, { "type": "Creative", "value": 3.14159, "description": "Pi as hail size" }, { "type": "Creative", "value": 9.8, "description": "Gravity acceleration as hail size" }, { "type": "Creative", "value": 299792458, "description": "Speed of light as hail size" }, { "type": "Big", "value": 999999999.99, "description": "Very large number" }, { "type": "Little", "value": 0.000000001, "description": "Very small positive number" }, { "type": "Exploratory", "value": "1E-6", "description": "Scientific notation" }, { "type": "Exploratory", "value": "1.5e2", "description": "Scientific notation for large value" }, { "type": "Exploratory", "value": "+2.5", "description": "Explicit positive sign" }, { "type": "Exploratory", "value": "0x1A", "description": "Hexadecimal value" }, { "type": "Exploratory", "value": "0b1010", "description": "Binary value" }, { "type": "Boundary", "value": 32767, "description": "2^15 - 1 (max 16-bit signed integer)" }, { "type": "Boundary", "value": 32768, "description": "2^15" }, { "type": "Boundary", "value": 65535, "description": "2^16 - 1 (max 16-bit unsigned integer)" }, { "type": "Boundary", "value": 2147483647, "description": "2^31 - 1 (max 32-bit signed integer)" }, { "type": "Invalid", "value": null, "description": "Null value" }, { "type": "Invalid", "value": "", "description": "Empty string" }, { "type": "Invalid", "value": " ", "description": "Space character" }, { "type": "Invalid", "value": " 2.5 ", "description": "Value with leading/trailing spaces" }, { "type": "Penetration", "value": "'; DROP TABLE weather_data; --", "description": "SQL injection attempt" }, { "type": "Penetration", "value": "<script>alert('XSS')</script>", "description": "Cross-site scripting attempt" }, { "type": "Internationalization", "value": "2,5", "description": "European decimal format" }, { "type": "Internationalization", "value": "२.५", "description": "Hindi numerals" }, { "type": "Internationalization", "value": "二.五", "description": "Chinese numerals" }, { "type": "Time-related", "value": "1.5@12:00", "description": "Size with time" }, { "type": "Date-related", "value": "2.0@2024-02-29", "description": "Size with leap day date" }, { "type": "Invalid", "value": "🌨️", "description": "Weather-related emoji" }, { "type": "Invalid", "value": "∞", "description": "Infinity symbol" }, { "type": "Invalid", "value": "NaN", "description": "Not a Number" }, { "type": "Creative", "value": "MAXIMUM", "description": "Text instead of number" }, { "type": "Creative", "value": -273.15, "description": "Absolute zero temperature as size" }

#TestInPublic: Charty App

ashleygraf_ — Fri, 21 Jun 2024 20:34:15 +0000

Update

These issues are in the process of being addressed where appropriate.

See this Twitter post for the first set.

Introduction

Hi! I'm Ashley. I'm a software tester who used to work in marketing and communications so I keep swapping hats back and forth. I'm recently back on the job hunt after a 6 month sabbatical which I was very lucky to be able to take. But it's a very different market to the last two that I have experienced as a QE, so I'm trying new things. I've been more than a little bit inspired by the indie hacker movement, so I want to help y'all out while I'm interviewing.

Every fortnight, I'm going to choose one early-stage project to #testinpublic with a real-world application to keep my testing skills sharp. You can nominate your product, or suggest a friend's, and I'll take a look over a couple days, and send you my thoughts.

For my first week, I'm looking at Charty App, an app where you can create animated charts. This one is just soon to come out of alpha so I expect it to be usable, but not polished. Did anyone say "Minimum Lovable Product"?

Executive Summary

2-day functional testing exercise
Tested key features: user account, payment, one type of chart, import, export
Most issues were around user experience and error handling

Summary

Most issues I found were in two categories: user experience, and error handling.

It came down to two questions

How do you avoid overwhelming users when you want to give them all the possible options?
How do you handle errors for minimum negative impact on customer experience?

With user experience, there are some issues I’d like to call out.

The main issue is that there was premature optimisation of categories, with settings for several aspects scattered across several categories. It added a lot of scrolling and short-term memory recollection to do what I wanted.
I was also expecting some categories and settings to have different names, as it was what I had come to expect from using other charting software.
The next, is that many settings were missing default values and/or units of measurement. Give the user a reference point!
The last, is that some subcategory titles were missing the title CSS styling, and they got a bit lost in that big list. For styling, with this many options available, it becomes really important to be able to separate feature from feature aspects at first glance

I’d rank the importance of the error-handling issues in the below order

500 errors that led to crashes and therefore lost work
Silent failures that would lead to confusion and lost time
Enabling inputs outside of the allowed range

Approach

When I start looking at a new piece of software, I always try and gain as much understanding of the product as possible by wandering around it for a while by myself without any deliberate provision of context from the creators. Then I'll dig in when it gets interesting. This means that I don't have any explanations of how it works to make my experience easier, and I use it much closer to how an actual new user would.

When I take this approach, I deliberately DON'T ask questions, just like most users wouldn't, and I see if I get disoriented or confused or overwhelmed, and so on. My context is instead similar products, and UX design patterns. These create unwritten expectations.

For this exercise, I'm going to limit my testing to a small part of the app. I chose The User Account, Line Charts, Import, Export, and Payment.

I have not prioritised (what I feel may be) the key issues, because while a QA may and should engage in bug advocacy they are not the ultimate decision maker. I have chosen instead key features to test, and thrown everything that I think could be of concern into an organised pile. Occasionally I will put my marketing hat back on and make some suggestions. I will always point out when that is.

This is also not exhaustive, as I set a time limit of 2 days to test. It also cannot be exhaustive. There are always bugs, just hopefully as few as possible.

For the sake of brevity, I will only be functional testing, and only on my home device. My observations will only cover possible defects, concerns, and questions.

I tested with the latest version of Chrome on Windows 10.

General Observations

I felt like the settings options were a bit scattered and the namings and labels a little inconsistent. Some titles are missing title CSS styling as well. While some settings have on-hover tooltips and others don't. I'll provide examples for each further down.
There is no unit of measurement provided for some of the different settings. Is it px? I would assume it's px.

I was able to enter numbers above the maximum set in the slider for most features using sliders to set values. This should not be possible.

Naming consistency

Placement dropdown naming was inconsistent. For Common Text Properties it used used "top, bottom". For Ticks it used "start, middle, end". But these both referred to label positions.

Box Radius is titled Angle for Background Plate Border and Box Radius for Watermark Box. Unless I am misunderstanding something.

There are some Points settings in Graph Settings and other Points settings in Points Labels. Personally, I found it annoying going to two different places to set all the Points settings.
Animations settings are in three places
-- Animation Timings
-- Graph Animation
-- Particles, which appear to be celebratory animations (unless you chose the sad or angry face emoji)
Most Ticks settings are in Ticks, but to set Tick label size, I need to go to Common Text Properties.

500s could be handled better. When they happen, the whole application crashes and I see the start of the stack-trace. I will not provide screenshots of that here.
There seem to be two ways to add and remove Grid Lines

Ticks -> Horizontal Ticks
Ticks -> Vertical Ticks

Grid -> Enable Grid X
Grid -> Enable Grid Y

However, the Grid Line settings over-rule the Ticks Line settings. This could get confusing for the user (and it's more code that the developer has to maintain), especially as the default setting for the Starter Line Chart has the Grids turned off in Grid.

Marketing hat on

The Animation options sound very mathematical and I wasn't sure what the difference was between most of them without trying them out! Perhaps a section on your features page with animations demonstrating all the options.

There are a LOT of options. The user can basically customise everything! There's celebratory graphics for social media, and slick animations for everything. For that reason, I would
consider
some renaming of some categories.
some rearranging of some categories.
clustering the animations options together.
subcategories
I personally was unsure about how to add x and y labels for a while!
ALL settings are Graph Settings. That category is probably a high candidate for renaming.

How to get to there

1) on-hover tooltips explaining what each category means (just temporarily. in the long term the new short titles should be enough)

2) running analytics to see which categories and settings are most often used and least often used for a more effective clustering.

3) talking to customers to understand why

Perhaps: Least often used - at the bottom and accordion shut by default. Most often used - At the top and accordion shut by default.

Marketing hat off

~The User Account~ (Log In, User Profile)

I believe it's a standard of web experience that the user should never lose their work. I would expect that if you click "Log In" from a page where you are editing a project, it would ask you if you want to save the project currently in progress as a JSON file (as there is not currently an option to save projects in progress on the user profile). This does not happen. Additionally, I was unable to use the Back button to rescue my work.
I can't close the User Profile modal. I have to click Open Editor to close it indirectly.

Payment

I think I was given a crypto wallet to use for DePlan when I signed up for an account but I can't be sure as it was labeled "Your UID" in my User Profile. I don't know what it's for.
Not necessarily a bug but why is it not an option to pay for Pay-As-You-Go Access from the User Profile page? Probably something about DePlan I do not understand.
Why is the dollar sign on the left for the Pay-As-You-Go option, and on the right for the Monthly Premium option? It's not clear what currency you pay in for either option. I think it's USD.

File/Import

If the user attempts to import non-compliant JSON it silently fails.
If the user attempts to import the project without the project settings it silently fails.
If the user removes a setting from the JSON they are still able to import it. The relevant setting just returns to the default.
If the user brute forces a setting to be higher than the maximum or lower than the minimum, unless it's Animation Framerate, it is allowed. It can still be successfully uploaded.
It was unclear until I tried it that Open referred to importing project files.

Marketing hat on

The purpose of File -> Open is a little unclear until you look at the icon. Then it's obvious that it means 'Import'. I would maybe change the titles like File -> Project Open -> Import Save -> Save.

Marketing hat off

Export

Why is there a + icon for "Export Image" but not "Export Animation"

Play button

No on-hover tooltip. Interesting that it's not using the start/stop video icons.

The Starter Line Chart

Animation Timings

What is the maximum animation framerate? It's not clear but it appears to be 300 because when I try a number higher than that, it changes to 300.

Background Properties

Plate Border is missing some title CSS styling

Watermark

Watermark means something else to me. I would be expecting to Free version to have the ChartyApp logo watermark in the bottom right-hand corner, but it didn't.

Marketing hat on

The watermark on the premium version seems more like somewhere to put the customer's branding. I'd suggest, if I may, an option to upload an image instead of adding text.

Marketing hat off

At the default X and Y offset, the Watermark is falling off the chart for characters like "q, j, p, g" when they are lower case for fonts with both lower and upper case, like Raleway and Montserrat.

Changing the Box Padding should probably push the box up so all the text remains in the chart, but it stays in the same place horizontally.

Common Text Properties

There was no character limit for the Graph Header.
The on-hover tooltip for the Enable Graph Header slider to show the tooltip, is not correct.

If you hid the Graph Header, the chart did not increase in size to fill up the space no longer used by it.

There was a very high range of options for the Ticks Label Text Size. It currently doesn't detect collision and set a limit when that is reached, and so they can currently overlap.

Particles

There is some sort of bug with Particles -> Graphics -> Custom Emojis at the moment generating a 500 error, and when I tried to select it, the whole application crashes and I see the start of the stacktrace.
Emoji Set 1 -> 9 didn't really mean anything to me as I don't know what's in the set before I try it. They are not descriptive.

Points Labels

It is interesting that the default is to not leave a space between the point value and point unit of measurement. It does give the user more options.

Non-numeric characters are removed for the Y-axis when adding the Label Suffix to it. For the X-axis, the NaN error appears.

Graph Animation

The dropdown options for Animation Type should be Capital Case to make the rest.

Graph Settings

The sliders are missing on-hover tooltips.
Line Curve is missing a sub heading, and the dropdown settings are still in camelCase.

Legends

The labels appear to have multi-lingual capabilities. That's good. I tried Cyrillic.

Ticks

Horizontal Ticks is missing the title CSS styling.

I found it a little confusing initially without a reference to x-axis and y-axis.

Editor

The x-axis values allow non-numeric characters. I suppose so that the user can do the time series any direction they want?

Simple Editor

For 'Import Your Data', when CSV has header, the title 'Group' Values Column (can be empty) implies it can be empty, but there were no NULL options provided. I had to select a column to choose as the Group values column.

Raw Data

It would be good to have some kind of written alert when the JSON file is no longer JSON compliant.
If you remove the x-axis value on Raw Data for one of the Groups, it disappears for all on the chart.
If you deleted the name of the group in the Raw Data editor, that group would disappear from the chart. It wouldn't just remove the name from the label, as you might expect.
Once the Raw Data JSON was no longer JSON compliant, the chart "froze" in its last compliant state (and red underlines would appear on the editor) but there was no other indicator to the user that the data was not compliant.

Conclusion

I'd like to give Naz a shout-out for some amazing work! And thank you for being a good sport about my publishing my findings of your alpha-stage software on the internet!

Heuristics for identifying legal (documentation) risks as a QA

ashleygraf_ — Wed, 05 Jun 2024 20:46:55 +0000

[This article is not a substitute for professional legal advice. This article does not create an attorney-client relationship, nor is it a solicitation to offer legal advice.]

Companies generally have to follow certain requirements for legal documentation. These are that legal documentation such as Terms and Conditions and the Privacy Policy, amongst other documents, must be

PRESENT
UP-TO-DATE
NOT SKIPPABLE
EASY TO LOCATE AND ACCESS

So if they are BOHNN, you might have an issue.

BYPASS

Find paths that allow users to BYPASS/SKIP the terms and conditions or the privacy policy page/{other relevant documentation check with your company's lawyer/legal team) when signing up for the product, signing into the platform or buying a new service.

OUTDATED

Find links to OUTDATED legal copy documents.

Is the copy on the page - if it's directly on the screen - up-to-date?

Do the links to the documentation page or document go to the correct one?

Is the outdated legal documentation still viewable if/when it's not supposed to be?

When all the legal links/documents/copy were updated on the website, were they really ALL updated? Does it match the rest of the links (if it is supposed to match)?

HIDDEN

Find HIDDEN (links to) legal copy.

A link might be hidden if it blends in with the background colour of the page.

It might also be hidden if it's not accessible from every screen.

MISSING

Find MISSING (links to) legal copy.

A page might be missing references to the relevant legal copy for that page.
A citation might be incorrect.
The footer might be missing references to the legal pages.

NOT AGREE

Find paths that allow users to NOT AGREE to the terms and conditions or the privacy policy (for example) when signing up for the product, or signing into the platform, or buying a new service.

If the site is using checkboxes, is it possible to go to the next screen without ticking the relevant checkbox?

If the site is saying, "if you click accept, you accept these terms", can you go to the next screen without accepting? Or maybe if you go backward and forward you can remove the acceptance and continue on?

Applying security principles to avoid getting pickpocketed

ashleygraf_ — Fri, 10 May 2024 22:08:30 +0000

I've been travelling for just about 6 months in Europe, including some of the world's major pickpocket hotspots, and I have some thoughts. Of course, I couldn't help but take a security mindset with me. So what does it look like applying information security principles to physical objects?

I have been lucky as well, but I am sure these steps also played a part in avoiding getting pick-pocketed. I'll also share some techniques I didn't use.

If you're interested in anti-theft, there are companies that have already done the work for you, and you could sort through the best of those. If you're not interested in looking like a hiker or a business traveller (sometimes they blend in better with the locals, sometimes they don't, depending on where you are), there's a bit more research to do.

So what gives a "fashionable" bag anti-theft properties?

The aim of the game here is to make yourself as difficult to target as possible. You are using the Swiss cheese model.

When you are wearing it, are there any exposed zips? In a backpack, Zips should be hidden between the straps or on the side closest to your body. here are some examples: https://www.tuscanyleather.it/it/zaini-in-pelle/shanghai-zaino-in-pelle-morbida-cognac-tl140963. https://www.tuscanyleather.it/it/borse-a-mano-in-pelle/tl-bag-pochette-in-pelle-verde-tl141990?
If the zips are exposed, how close does someone have to get to you to be able to unzip the bag?
Can a stranger open your bag while you are wearing it?
If they can, how long does it take them to work out how to?
Can you lock the zips?
How easy it is to lock the zips?
Does the opening use a two-way zipper?
Can you put a lock through both zips? Two-way zippers are a feature more common on backpacks and suitcases than handbags. If you can lock these exposed zips, how far the bag can be easily opened is far reduced.
Are all openings sealed and water-tight when the bag is fully closed?
How visible are the zips? The less visible or hidden, the better.
Don't forget that you must also keep the bag physically close to you at all times when outside your accommodation. So your bag should be comfortable to wear in front of your body, where you can always put a hand in front of it when you are out-and-about.

I am always mentioning bags here because at no point should you leave anything in your pockets, especially if they don't have zips. Wallets stay in your zipped up bag when you are not using them.

Preferable, you usually wouldn't even take your cards around with you. Just leave it at your accomodation. When you can't pay with your phone, you pay with cash. You shouldn't need to take out your card except at the ATM or the money exchange to withdraw cash.

In airports, major train stations, tourist hot-spots, restaurants, I would always wear my phone on a leash and my bags on my lap with an arm through my smaller daily backpack, or a leg through my larger backpacking backpack (get bags with adjustable length straps).

I also tested how easy it would be to un-clasp the phone leash. How long would it take to wear out and just fall?

If I was considering a replacement I would always compare how easy it was to unclasp the new leash versus my current one, and if it was easier, I would always pass on it. So in crowds, the leash is made shorter (get a phone leash with adjustable length leash) to make it more of a hassle to cut. I then place my phone under my shirt with the strap still shortened when I am not using it. You can get phone cases where the strap cannot be unclasped from both big phone repair stores in "Europe". However, the straps that come with can be easily cut. You may want to replace it with another fabric. Security alone is not enough, though. It is nicer to get the leash in a comfortable fabric that is not too heavy, and flat and wide, to remove the temptation to not use it. it should not strain your neck when wearing it.

Tote bags are an easy target. Deep ones with small straps are probably less easy to steal from as they have less surface area to dig into. I do not put anything irreplaceable in there. They are for food and to store clothing for when the weather changes.

In public places like airports and major train stations, wear your important documents under your clothing. In Berlin, you can buy utilitarian leather bags at the markets for specifically this purpose. Or you could get a jacket with internal pockets - generally suit jackets and hiking jackets will have these, and wear them zipped up with your documents inside. (And another layer of clothing on top of that jacket).

A tote bag with a prominently exposed zip is very popular in Western Europe, and they always wear it in front of their body. In this case, in close quarters, you always have your hand in front of your bag so it cannot be unzipped.

Workarounds for inconsistently formatted logs in Splunk

ashleygraf_ — Sun, 23 Jul 2023 12:25:56 +0000

Over the past year and a half as I've learned to use Splunk to hunt down root causes, create sequence diagrams, make reports, and generate statistics, there's a few features I keep returning to, to make inconsistently formatted logs easier to work with.

If your logs are any combination of fully parsed JSON that you can use directly, JSON blobs, or straight-up just text with predictable patterns, this may come in handy.

They are the below

rex (regular expressions)
rename (renaming fields)
coalesce (combining fields)
joins (like SQL, on fields with matching names [hence why you need rename])

Regex

Yes that's right, the tool everyone loves to hate. Regular expressions. I use regular expressions in two key ways
1) to remove any unique identifiers that would match a pattern, and replace it with text representing that pattern. This is particularly useful for creating statistics, if you don't have URI as a field. This enables you to make it a field.

For this purpose, I've found it generally unnecessary to learn complex regex. Don't overcomplicate it. You will start off needing to know 5 symbols.

words \S
numbers \D
more than 1 character until you meet the next symbol or anchor +
number of characters [n]
all characters until the next symbol or anchor .*

and then whatever flag you need for sed mode. I find g for global suffices for most searches.

rex field=<field> mode=sed "s/<regex>/<replacement>/<flags>"

2) to find text that matches a pattern, and collect it for further use.

rex field=<field> "anchor(?<new_field_name>\S)anchor"

Don't name the new field the same as an existing field if you are planning to use it in a coalesce.

Do name the new field the same as an existing field if you are planning to use it in a join.

If you need to combine data and join two searches, coalesce first, then join.

Rename

Once you are done retrieving your new field out of the ordered text, you can rename it so it looks nicer for your report or chart, or so that you can use it for joins. If you rename before you try to coalesce, you cannot coalesce them, as they have the same name.

rename <field_name> as <new_field_name>

Coalesce

This is useful to combine the values you have just created out of regex-ed values with the data you already have from the fully parsed logs. If your field used to have a name, but now it has a different one due to reasons, this comes in handy. You can compare with historical data.

eval <field_name>=coalesce(<new_field>,<existing_field)

Joins

This is useful to combine the results of two searches. You might use it to connect a field to match it to another field that is missing on some logs but not on others with a field that it does have in common.

There are two joins in Splunk - left (or outer) and inner, and this is how you do them.

index="index" "phrase" | ...... | table ..... | join type=outer field [search index="index" "phrase" | ....... | rename ..... | table ..... ]

index="index" "phrase" | ...... | table ..... | join type=inner field [search index="index" "phrase" | ....... | rename ..... | table ..... ]

Onboarding onto a new team

ashleygraf_ — Sat, 27 Aug 2022 06:49:40 +0000

I had previously worked at a development agency, and many many many small companies. This taught me how to hit the ground running with at least a basic understanding of the product I was working on within several weeks or even days.

Here's how I do it.

Make a buddy with an established team member. Ask them questions I don't need to ask my manager.
Get set up on the shared & local test environments ASAP.
Learn how to create and use test data.
Get set up the VPN & HTTP/S proxy.
Learn the little bureaucratic processes and rules around getting access.
Mosey around the code, the databases, the logs, and the user interface, etc.
Read through the Requirements, wireframes, UI designs and flows, and key tickets for the project.
Search Github for frequently used keywords to see how the functions & services connect together.
Have short video call sessions with at least 1 member of each team - backend, frontend, mobile, and so on, to get to know them as a person & build rapport.
Ask the people who've been on the team for the longest if they have any special tricks for navigating around the organisation you work for, and any productivity tips they might have.
Do some ad-hoc testing. Try stuff out on the user interface or postman, and observe the UI & logs & the DB & the proxy to see what happens.
Preference 'hanging out' in the office to 'doing work' in the office. You learn more about the team and product by talking to team members about how they do their work, and asking to see what they're working on, than sitting in a corner working on your ticket for the day.

Why code review is an underrated skill in testing

ashleygraf_ — Sun, 27 Mar 2022 01:46:49 +0000

NOTE: This was written as part of the Ministry of Testing's Blogger's Club challenge. If you are interested in learning testing, I encourage you to check out MOT.

What is code review?

It's both looking over the code personally, and letting the linter - with rules decided as a team - inspect the code.

Why code review?

Code review is the first step after the development is 'ready for testing'. It's the first step of post development testing - you've scoped out the work, analysed the requirements, had a session with the developer and designer to discuss. This is the last chance to catch bugs without intense 'traditional testing', as it were.

I like it to be the first step I take before I start testing. I may not comment, but I definitely read it. I too often regret when I don’t.

It's a place to catch issues you would take a longer time to find when testing with other methods - security flaws, small but important business rule mis-interpretations, commented out code, code that could be tested with integration and unit tests that aren't - and issues you can't find when testing with other methods that can lead to product quality issues down the line, like hard to understand code, unclear names, de-standardisation of naming styles.

It helps you learn how to code. That is useful for the inevitable programming you'll end up doing - whether it is fixing minor bugs, creating testing tools, using testing tools, or test automation. You see examples of different patterns and when they should it shouldn’t be used, out in the wild.

It helps defect replications later - you understand the code, and therefore the most prominent scenarios that the code fixes are referring to just by reading the code. If fix tickets are ever lacking the required details, or they assume knowledge you don't yet have, you can reverse engineer the intent, and then talk to the engineers and product manager when they are available to fill in the blanks.

This makes doing code reviews both a potential workaround to organisational issues, and a skill enhancer.

Ultimately, it helps you refine your questioning skills - which is ultimately what we are doing as testers. Making and recording observations and asking questions. You practice asking questions, while both broadcasting your assumptions, and assuming that you may be missing context that the developers have. This improves your communication skills.

16 ways software testability can assist manual testing

ashleygraf_ — Mon, 21 Feb 2022 10:34:47 +0000

It's just not what you're thinking right now.

I’m not, or not just referring to the existence of good test automation streamlining manual testing by reducing repetitive and tedious work, as well as the potential for error, here, which is what you may assume!

Most see testability as the ease of test automation suite creation. This concept of testability makes improving UI, API, and integration testing relatively easy to define and research. It gives a name to the process, and thus easy-to-find references on how to improve it. However, we don’t often discuss testability for what the testing industry likes to refer to as ‘manual testing’. Unless you hang around the Ministry of Testing and a few other select corners of the web, you find that most of the focus and the material is on assisting test automation. So I wanted to add to the literature outside test automation, and talk about software testability in the context of ‘manual testing’, and I do think it should use the same term, because just like for automation, it makes the testing easier.

I suppose we do call them testing tools. They are both the judicious use of tools already out there, as well as ones you make/program yourself from scratch.

Predominantly, they come in 5 categories:

they separate concerns (where possible)
they enhance the team's skills
they generate test data (easily)
they change the system's state (easily)
they search already available test data (with precision)
they search the logs & db of test envs (in detail)

The ease is what enhances a software’s testability, and overall, they increase a team’s capacity.

I think we should call this manual test assistance ‘software testability’ more often. It’s part and parcel of making software easier to test. But what does this ‘manual testing assistance’ look like? What does this aspect of testability entail?

Here are some ideas.

This is not dogmatic. It’s more of a pick-and-mix box, a buffet, or even an encyclopaedia if you will. It’s a set of ideas you can pick up and try, according to your product’s needs. And in this article I am referring to only the most simple organisational or personal aspects of you or your team. Predominantly, it’s code that you can slot in to make testing easier.

It is also a living document. If you have some ideas, any useful things you've seen and done, you are welcome to share it here.

If you’ve read Ash Winter’s Testability Advocacy Canvas you may think of this article as like a set of visits to local companies to see how they do it.

I would like to acknowledge Ady Stokes for a few contributions, and editing, as well as Sam Connelly and Alan Giles for reviewing the frame. And now, a few ideas for software testability.

View this summary document of my TestBash World 2022 talk on Testability.

In-depth logging

What
Everyone who tests the software - testers, product team, and developers alike - could have read-only access to the Terminal log in AWS and read-only access to the code in your repository software of choice so they can run it locally, enabling them to run more destructive tests that aren’t necessarily cybersecurity-related without heavy planning and coordination or potential chaos.
Within what is displayed on the Dev Tools Console log, display as much useful custom logging as you can when the code is in pre-prod mode.

Why
It’s more direct to read than a stack trace. Of course, you want both. Combined, you know the cause, and where you look in the code to fix it.

How
Tools like Github give read-only access to the code as a matter of course, the natural consequence of giving someone access to a private repo, so it doesn’t have to be a big step.
Instrumentation, like Honeycomb, and monitoring, like DataDog, that adds it for you, automatically notifying you of some types of anomalies and making it easier to set up the rest

Showing everyone how to read multipage JSON requests and responses

What
JSON is a major method used in API requests, having taken over from XML. It's important for all testers to be able to understand how to read it.

Why
When JSON payloads and requests become multi-pagers, they can be draining to read! That can be unavoidable, so let’s make it easier!

How
Explain to your team that they can be converted into CSVs, and prettified with JSON prettifier websites and programs, so it’s easier to read when they go on for a bit.

Clear error messages

What
Clear error messages make software easier to use for everyone. What makes a clear error message?

Why
They’re just as useful an indicator to testers as users, to what has ‘gone wrong’.

How
Clear error messages are sent out by the error handlers when the code is in pre-prod mode. Some understandably have to be obfuscated from the API for security reasons, and the rest should be clear anyway for a good user experience. When they do have to be obfuscated, Ady Stokes points out that another approach is to hide them behind a debug menu and make them accessible in any pre-prod environment. The debug menu has to be hard to find, and protected (by a passcode, for example) but it can work really well to see the whole picture.

Reduce silent failures

Why
Silent failures can make testing frustrating. It’s the reason behind the bug reports or messages in Slack that are just like ‘I can’t go any further here’ and then you pull up the Network tab and there’s a 500 error.

How
Every 500 could show an alert UI element and be added to the debug log as per above in pre-prod mode.

Display all entity database records and make entity model values editable

What
Database records are all the information we have on users and related entities. It’s where it’s stored and it is what it is.

Why
Displaying the output of the program, in the information created and generated by the entities that use it, where the admins can see everything all users see, in a way easily by all who test the software. Database records. All of them. This is typically an administrative section with different levels of permissions for different levels of users.
Making entity model values that influence their treatment by the software changeable in Admin where reasonable. For example, from basic elements like whether they are active or inactive, up to like the ‘age’ of the account. But perhaps you would exclude ‘has sent a message’ and reserve sending messages for the user platform, and then check if the user has sent a message on the Admin platform.

How
Give ‘super admins’ access to edit all entity model values. You can turn off everything that won’t be needed or wanted in production with feature flags.

Did you know?
Some systems have these viewing capabilities built into admin UIs complete with change history for audit purposes, particularly in highly regulated environments.

Read-only access to the Database

What
You can view, and search, but you can’t modify the database, with read-only access.

Why
Cut through all the time and red tape of adding it all to the admin interface. Useful in cases where there just isn’t a need for one and it truly would take up unnecessary, unavailable developer or SDET time.

How
You might like to use tools like Redash as it is just for viewing and querying data, and you don’t have to navigate through the rest of a platform like AWS to find it, which can be intimidating

Regular error alerts

What
Know when errors are happening, when they happen. You might get an alert immediately, or receive a summary of alerts for the day, or be able to ping a service to see if it’s still up, but probably some combination of the above.

Why
In particular for integrations with external software. Whose software failed? Yours or theirs? Is it important for the end-user to know who? Or just that it failed? It sure is important for the development team to know.

How
For example, it comes plugged into AWS, You have to set it up to react to certain events though.

Controlling variables and settings placed in Config files

What
Config files can place controlling variables, settings and API keys in a central location

Why
This makes it easier for everyone who tests to modify system-wide settings without needing in-depth knowledge of programming, and you can change the system without needing a full re-deploy. This can be a blessing and a curse. You’ll need to have a stringent .gitignore file to make sure no unintended changes go through. You can find plenty of base .gitignore files for different projects to start you off.

How
Store the base config files in source control.
NOTE: Some API keys should not be saved to version control, for security reasons. There is instead a reference to them in the file, and they are stored as program arguments instead after the keys are retrieved elsewhere.
Creating and managing this system is part of what is known as Infrastructure as Code.

Anonymising production data

What
This is placing (a subset of) anonymised production data into your pre-prod environments

Why
You can’t get any more realistic than production data, and so it enables more realistic testing. Anonymising production data allows you to load production data onto your test systems, if you have the legal restriction of GDPR, and it’s just generally good practice regardless. Or as I always say, email is always production. If that’s someone’s real email address, that email is getting seen, and that use of your email send quota is real.

How
A script that anonymises the data, so you reduce the likelihood of compromising the privacy of users. Simple anonymisation is likely not to be enough, and you may have to add randomisation to it. You will likely also have to do a few random manual checks to check that all values have been anonymised in a sensical way.

Testing features and their crontabs separately

What
Cron tabs set off functions to run at set intervals, typically when they detect a change in the ‘status’ of an entity.

Why
If you change the ‘status’ of an entity, it’s handy to be able if the functions behind the crontab are set up correctly, and will go off appropriately, right after you have made the change. You would test that the set intervals are set correctly separately.

How
Set each crontab up so they can be set off manually by clicking a button on a hidden screen in admin. For this, you remove the cron, and effectively, what you are doing is saying 'run the function that uses this cronjob'.
Put crontabs in the config file so they can be easily changed to run on a frequent basis and you don't feel the need to set up a button to set the function off. The issue with this, is that while it is more efficient in the short run it can clutter the terminal log, making it more difficult to debug issues you find locally.

Making copies of raw and processed files downloadable (from the same place)

What
Is uploading and processing files one of the key functions of the software? Have copies of files before they are processed by the software, and after they are processed by the software

Why
The changes may be something you didn’t expect. Raw files enable you to immediately see the difference. There’s a unix utility called diff derived from different. But if you don’t have access to that, you can copy the content from the raw (text) files you now have access to into Diffchecker for example, and it will run a comparison for you.

Downloads of aggregated data

What
Have bulk downloads of all the data that goes in, so you can analyse it with other software, and see if it matches the results from your software

Why
If you need to analyse user/entity data to check the internal statistics engine is running correctly.

Sped-up individual user creation

What
Creating users, but in Admin/Swagger/Postman, because it's more efficient, and gives you fresh new users faster.

Why
Chances are it’s probably faster than generating it as a user. If you need a fresh user and you’re not testing the registration process, and you’ve already tested that there are no differences between a user created on the user platform and the admin portal, it may be a better option in a hurry.

How
Your own custom setup in Admin. Running a single API request Swagger or Postman, or a set in Postman with a Runner, if that is what is required.

Bulk user generation

What
Bulk test user creation is more than being able to create individual users in Admin. Although you should be able to do that too.

Why
You have just created many test user accounts to use later for testing, and you can use them later for load testing as well (assuming you aren’t also testing registration and verification in the test)

How
Bulk upload of users through uploading a CSV file in Organisation-level Administration or central Administration
User generation through adding number of users through an API request ‘served’ through Swagger, the Admin section or even a shared Postman collection

Tip
One of the pitfalls here is you do have to have put in a little extra work to make the users realistic, or it has the potential to mess up your load testing later. You will also get better results when testing if you base these auto-generated users on your personas.

Size of change

Let’s think a little more conceptually yet again.

What
Smaller releases to staging before each bout of testing

Why
Smaller releases to staging make it easier for testers to understand the impact of each change, and give the testers more time to test in general, enabling more exploration.

How
The developers release their code to the pre-prod environments more often. This requires the developers to release more often. This eventually requires the automation of build pipelines.

Mocks and stubs

What
If you’ve worked on any e-commerce projects, or projects with payments, you’ve already used this. All the payment APIs have test cards that test various transaction and validation states.
But you can also use it in other scenarios. For example, to simulate users. For example, if a user with a particular email address make a purchase, they are treated like a frequent purchaser and sent an email giving a special offer for spending $500 in the last 3 months. This is added through the API request for purchasing which lives in the production code, so pick obscure email addresses (or from your own domains) and hide that section of the code with a feature flag in prod, just in case.

Why
If you or your testers are unable to interact with the database directly, this gives you a way to set up your project so that it interacts with the database for you.

How
Add predetermined responses to certain inputs. Within say the ‘registration API’ you could say ‘if the user has email address x then act as if they are a new customer, or a frequent customer, or a customer with an expired membership, and so on’.

Example

A set of accounts are set up to show different account personas
Card payments give responses based on the expiry date used

Default One Time Password Responses

What
This is for projects that have 2FA. You can hardcode the same phone number and password.

Why
It saves time diving for the phone, and you don't end up with any important test accounts tied to individual employees through their mobile number.

How
Registering test phone numbers - Google Cloud
How To Use Twilio Test Credentials with Magic Phone Numbers

Coming soon

Feature flags
Feature flag overrides
Using HTTP proxies / HTTP monitors / Reverse Proxies
Using mock servers

What can high ‘manual’ software testability do for you?

A better understanding of the system you are working with enables better testing and support.

Better bug reports while in testing
If all those who do testing know how to write strong bug reports, and there’s a mandatory template for bug reports. You could even have different templates for different classes of bugs. Either way, it gives you all the information you could possibly access or need.

Faster chasing down bugs in production
It’s easier to work backwards when you have a more detailed trail leading back to the cause(s) of the issue.

It could make exploratory testing easier
Improved manual testability typically makes more variables alterable at the click of a button or some writing in a text field or the changing of some numbers or a date-field or the send of an API request. Although remember to test your shortcuts first!

Altogether, it could make the quality of ‘manual testing’ performed by your company itself easier to improve.

Who improves software testability?

Your developers and testers. It grows and evolves alongside the production code

This could require
Access to source control
You’ll notice some of this software testability access entail giving wider, if read-only access to Terminal logs and production code, to everyone who tests, potentially the entire team.

An all-access Platform Administrative section
That’s where you add the more easily changeable levers.

Education
This is teaching your team enough to feel comfortable setting up and navigating their local environment. You’ll probably need a fully kitted out user guide, and a more robust README.MD set-up file alongside allocating a buddy to help out when team members get stuck.

Coordination with your DevOps team
Whoever issues permissions for all the various services will probably end up with more work for a while. You can streamline this a tad by issuing a ticket to them with the exact details on how much access a new starter or team or level needs, so there is less back-and-forth setting up.

A list
Recording your progress and plans for testability will keep you on track.

Resources

Scheduling Cron job with Crontabs
Team Guide to Testability by Rob Meaney and Ash Winter
Testability Advocacy Canvas by Ash Winter
PayPal Payflow Pro Test Cards
Assembly Payments Test Data
Redash.io
Data Anonymization Techniques

Testability and the DB record

ashleygraf_ — Sun, 09 Jan 2022 11:07:52 +0000

In her Explore It? Explore It! video, Elisabeth Hendrickson, author of the awesome book Explore It! talks about observation, and how quality observation requires using all your resources from multiple sources to find out what actually happened.

She lists the below as some of the places you would look to observe changes

Access to internal storage
Application telemetry [which prints out to the Network and Console tab on Dev Tools for example. Is it appropriately detailed for the tester?]
Diagnostic tools
Consoles
Logs [for example the Terminal]
Monitoring tools (OS, Network, etc)

I want to talk about database records - or specifically, making them as easy for non-devs to view as the devs. It requires a little dev work, but you can just splash it up there first, and then make it pretty later. The easier to read, the easier to work with!

In particular, I find looking at the records of each individual entity on the administrative interface of platforms I've worked on helpful. Database records are created when a website is closed, so it's neat to be able to track the creation on a nice UI.

When I do 'manual testing', I'm using basically one user at a time, and so, that's the view I need. Combined with the Console Network tabs of the Dev Tools, the Terminal tab, and the user view(s), you start to get a holistic view of what's actually happening.

I think it's an aspect of testability, to have records (nearly) as easily viewable by testers, and product/project people as they are by developers.

Back in Level 1 Tech Support, I relied on this record view heavily, and I find it just as helpful now.

Although it only works if the relationships are recorded in an up-to-date display.

So how do you find records shown on the user view that might be missing on the admin view? Open up a user view, and check off entities as you find them that are also on the admin view. Don't forgot to check that it's linked to the most recent version of each entity, if there have been updates, as may happen if there have been major changes to the application.

This can make testing more efficient over time, while maintaining accuracy.

Avoiding false alarms

ashleygraf_ — Thu, 21 Oct 2021 06:23:00 +0000

During my first year as a tester, I had a false-alarm rate I wasn't proud of until I adopted a routine, some borrowed from my mentor Alan Giles, and some borrowed from the team at Digital Purpose, and some I found for myself. A high false-alarm rate decreases trust between yourself and the team, so it's important to get it down. This is what I do.

Don't panic. Consider all the possibilities. Are you missing something? What are you assuming?
Is your test environment set up properly?
Are you using the correct build?
Is the replicator using the correct build?
are you using the test user you think you’re using?
Check the server is refreshed. Your team might have alerts connected to your deployment pipeline of choice. This comes in handy.
Hard refresh your browser tab. That is Ctrl+F5 or Ctrl+Shift+R on Windows.
Re-test sequences that find bugs to check you saw what you think you saw. The issue here is not to re-test so many times you run out of time to fix issues. I've heard of people that record all their testing efforts to make back-tracking easier.
However, don't go too quickly. Accuracy >>>>>>> Speed. Or, it's not speedy if it's wrong and you have to do it again.
Test with accounts created before and after the change - Does the feature / bug fix need to work retrospectively? If it's for a green-field release, probably not. If it isn't, probably. It may also be best to just always test with a fresh account.
Account set-up can be speeded up with a Postman API request, or even a set of them, with the Runner hooked up to the Auth details of the new user (first step done manually to fetch this).
Confirm the requirements with the rest of the team. Does it qualify as a requirement, written or un-written?
Check your test data is set up properly. Are you testing what you think you're testing, or it is another scenario?

Test techniques mapped to types of features

ashleygraf_ — Mon, 13 Sep 2021 06:38:24 +0000

(in progress)

Test technique -> Type of feature

Boundary Value Analysis

a little too little, just enough, too much - blocking someone

Equivalence partitioning

State transition testing

if the user does this, their status changes to that.

Decision table testing

This is for forms. this is especially for complex forms with logic. if this response is made, this question or content is shown.

Graph based testing

Error guessing technique

For anything used frequently or just key features. Might be used to see all the ways a chart COULD be manipulated.