Testing is blocked? Martin Jansson No Comments
Sometimes when I read status reports or hear project managers talk about testing, I hear that “testing is blocked”. What do they mean by that? When I delve deeper in what they are talking about I sometimes see that the progress on workpackages for the testing team or testers in a team have been combined with the result of a test case such as pass/fail/blocked. But that is hardly the truth? As a tester we do a lot of things, not just running test cases (if we even do that form). If we say that one test might be blocked in the sense that it does not provide value in testing further there. That does not really mean we are blocked? Is a blocked test when you cannot get the answer to a certain question you have about a system? Does it mean that a specific test could not be done, but perhaps a similar one be done that would provide as much value?
If everything around you is blocked and there is nothing you can do as a tester, then you can perhaps say that “testing is blocked”. But what about preparing for coming tests? What about automating some of the things that you have found? Did we then mean that we were blocked in the sense that we did not come up with any ideas on things we could do? Well, perhaps you can look at various test idea triggers that can get you going?
Progress in performed test related workpackages and the result of testing are two different things. Combining them does not really give a good picture of what is happening. So, before stating that “testing is blocked” consider if you actually mean that you have no progress or that you are unable to test in a certain area. Based on what you inform and how you communicate, stakeholders will make decisions based on that.
I usually tell my stakeholders that they should not worry about me or my test team being blocked in progress, there is usually something valuable to do. Still, if I say that some tests cannot be performed or that they are blocked, where the result of those tests are important they need to assist with removing those obstacles.
During a daily stand-up meeting in a test team I was part of, nearly all testers reported that they were blocked and that they had no progress in their test cases. We had a long list scripted tests that needed to be executed according to the test project manager. I investigated what they actually meant when they said that. Interestingly, most of them had a test cases that wanted them to go in a certain direction through the system. What was blocking them was bugs. They found lots of bugs, but did not feel they had time to report them because they really needed to gain progress in those intended test cases. At that time we had to work overtime if we did not have enough progress in our test cases. I asked them to instead of trying to run the test cases they use the test case as a test mission or charter to guide them where to go. When they saw fire or smoke, they followed that trail and documented (session notes) what they saw and what happened, but also log all bugs they found. From then on not one reported being blocked in the daily stand-up meetings. The flow of reported bugs increased and the progresss was fine (in the sense that test cases were executed as far as project management thought). Stating that you are blocked might in fact be how you work that does not enable freedom and creativity.
If we were to measure something as a manager of testing, would we then see a decline in testers reporting being blocked when using methods such as SBTM instead of scripted tests that report pass/fail/blocked?
The Scripted and Exploratory Testing Continuum Henrik Emilsson 7 Comments
I have been using the Scripted – Exploratory Testing Continuum (For one source of this, see page 56 in http://www.ryber.se/wp-content/EssentialTestDesign.pdf ) in classes to explain how scripted testing and exploratory testing intertwines; and to explain that most of our testing is somewhere in the middle of the continuum.
But I have also had some issues with this description.
If this continuum exist, when can you say that you are doing more exploratory than scripted (and vice versa)?
How do you recognize when your testing is more exploratory than scripted (and vice versa)?
I do not believe that these questions can be answered because I think it has to do with the mindset of the person who sets the approach; and how far this particular mindset can be stretched.
The continuum is a simplified (yet useful) description of how your testing approach is a mix of the two approaches. And the simplification lies in that it can be seen as if our testing approach can be found in some point on the continuum. But I think that someone who has an exploratory testing approach is doing more or less exploratory testing; and someone who has a scripted testing approach is doing more or less scripted testing. Maybe this is how many of you already think of this?
However, I would like to visualize the differences in a slightly different way in order for me to understand this better.
Testing can be viewed from different point of views; and if I as a tester look at my testing approach I can have good use of the continuum. But the decision on which approach we should focus on is often driven by the management of the project or test team. So, from a managerial point of view, the testing approach is driven from a core dimension (interest, factor) which is more or less strict. I.e., I think that each approach can be seen as a set of continuums where each continuum consist of a certain dimension; and you will only modify these individual dimensions, not the entire approach.
Scripted Testing Continuums (some examples)
Exploratory Testing Continuums (some examples)
Let’s say that my main interest is Control, and I believe that a scripted testing approach is the best option in order to fulfill my interest. From this, if I now gradually become less stricter when it comes to my control demand, I won’t transit into an exploratory testing approach. My approach would instead be gradually less scripted.
In the same fashion, if my main interest is trust. My approach would gradually become less exploratory, but never transit into a scripted approach.
I think that this has to do with that the approach is a mindset; or rather an ideology. And it is hard for many people to change their mindset/ideology because your approach is derived from your school of thought.
If you have a scripted approach to testing, it is because of something you strongly believe in. If you have an exploratory approach to testing, it is because of something else you strongly believe in.
Pair testing with a flare of checking Martin Jansson No Comments
When I explain pair testing I usually say that you pair up two persons. One drives the testing while the other documents, comes with suggestions and asks for clarifications.
In this context, if we consider the use of testing and checking. What if the person driving focus on testing (that is usually the case) and the other focus on his other tasks but also that of managing checks? Try to avoid disrupting the creative, exploratory mind of the tester driving the testing. Still, at some stages asking if it is possible to check certain things. I am sure a lot of persons work this way and it is nothing new, still perhaps they have not expressed it this way.
If you set it up this way you might avoid the various biases and blindness that the driver might experience. Instead you let the passive player focus on those things. By having a dialog you will naturally affect the other, but perhaps not as much as when you focus on it yourself.
What if you in a pair testing session, switch place every now and then. Each time you switch place you also switch from Test Driven to Check Driven.
What if you did pair testing but both of you tested in the same area at the same time with collaborative note taking. One had focus on Test Driven Checking while the other had Check Driven Testing. Would you get any interesting result based on that?
Some initial thoughts on checks Martin Jansson 5 Comments
Introduction
In 2009 Michael Bolton had a talk at Agile 2009 from which he later on wrote about a distinction between testing and checking [1]. He has since then elaborated more in the area and digged deeper into it [2], and continue to do so [3]. By clarifying what we are doing at a certain time it makes it easier to determine what tools and heuristics to use among other things. By making this distinction I think it helps us as testers to do a better job and to better be able to explain to other non-testers why we need to do both testing and checking, but also what the difference is.
The idea that you can (and should) automate all testing is not interesting. It is like discussing automation of all decision making. In a discussion like this we can instead offer that checks are usually good to automate, but testing is not as easy since there is no true pass/fail result when it comes to open-ended information, that might have different value to different persons. James Bach makes an example how this is handled in BDD [4] or rather making it obvious that some checks might be easy to confirm while others are close to impossible.
When I talk about testing vs. checking and touch the subject about sapience [5], there are many who are known to promote the checking part as their main activity as a tester, who then feel uncomfortable. What is really important is to point out that they do exploratory testing as well [6], but perhaps not framing [7] or structuring it well. I’ve heard arguments that exploratory testing is less structured and formal, well perhaps for you it is. Why not change that?
Simon Morley makes some interesting observations in his post on Sapient checking [8] and I agree with him that it might confuse more if we talk about testing and checking when we are delivering information to inform decisions. Should the focus in our information be on if it was checking or testing being made? For most cases, I believe we should not.
Test Driven Checking
If we assume that testing and checking is intertwined, you might still consider that in some cases one thing drives the other or that at least one activity is in focus while the other is in the background.
Depending on how well I have organised my charters and missions for testing during a project I test differently. Sometimes when I run a test session (following principles from SBTM [9]) I do note taking on what tests and checks I have on the system, mixing the two activities. Afterwards I might go through a checklist and mark items off as done. This might be coverage related, requirements related or whatever was suitable to have in a checklist. I might have stored most of the information in my session notes, but I might still go into other areas to mark it there as well, if that was easily done. A college did exploratory testing, without note taking, and then went into an excel sheet to check off coverage areas. He then reported progress and coverage based on that.
To avoid missing some checks you might consider that after you have tested in an area and you are ready to move on, you take a peek at things to check in that area. By doing so you might avoid the confirmation bias [10] and unintentional blindness. You might also get new ideas for testing when seeing things that you should check.
When you do regression testing you might consider to first test around the area of the bug, then as a final step check if the reported bug is in fact fixed.
Check Driven Testing
This way of testing is probably the most common one by those who are unfamiliar with concepts of exploratory testing. As I see it, you might have a list of checks (or for some scripted tests) that you check, but you might test around them as well. In this case the checks are driving the testing. When you have one-liner test ideas as your main focus, I think it is most common that you let them drive you. So depending on how you define them you will have different focus. Simplifying it, are they “What if”-focused or “Verify that”-focused?
When you retest a fixed bug you might select the methd to check that the bug as it was reported is actually fixed, then you continue with what valuable testing around to see that nothing else has broken in the vicinity, as far as you know.
If you do a testing tour you might set out some designated targets (check points) and along the way do your testing reporting what you actually find out.
If you consider how kids might act when they walk, on their own, on one side of the street in one direction and all of a sudden they see their parents on the other side. The kid forgets everything they had learnt about traffic rules and risks, then just runs to the parents by crossing the street. I believe we can see this with testing and checks as well. When you get to the point where you can do the check, you forget all other things that you should really be testing, thus a case of unintentional blindness.
Conclusion
This is just one way of looking at it, this is no real conclusion. For me it helps to make a distinction between doing Test Driven Checking and Check Driven Testing. By being aware of the distinction you might plan your testing differently, you might consider a bit more what where you put your checks and where you do your testing. They will affect each other and they will be drawn to each other.
References
Humbling Experiences Rikard Edgren 2 Comments
I see humility as a very good virtue.
It is something I have failed miserably at, partly because it is easy to think something is bad just because there are many problems.
I think it’s a common fallacy for many ambitious testers – you are last in line, maybe with lower status, you want to get heard and become too persuasive…
Humility tend to give better collaboration, and thereby better results. A humble tester more often changes her mind, which is necessary.
The complexity of reality demands a humble tester.
Here is my list of humbling experiences; things I think have made me a better tester.
* write requirements
* programming something bigger than an exercise
* manage projects
* customer report of an important bug you should have caught
* experiencing too little time to test the very important stuff
* failing to explain why your testing is good
* reading something you wrote some time ago, and realizing it’s very shallow
* understanding your lack of humility
* realizing you were wrong
Which ones do I have left?
Status Reporting Questions Rikard Edgren 2 Comments
Status reporting of testing activities is extremely project-dependent. The needs of when and how and what will differ every time. Maybe that’s why there’s so little good writing about status communication; you have to make it up every time.
Templates are out of the question, and I believe examples will mislead you as well.
You’re better off by thinking around these questions (inspired by final section in Lessons Learned)
Why are you reporting?
Are you giving quality assessments, trying to prove why testing is taking place?
Is focus on the product story or the testing story?
What kind of information can change decisions that are based on your report?
What is most important?
The everlasting and most difficult question for all testing efforts.
This is what you should start your report with.
To whom are you reporting?
What kind of information is the audience interested in?
Are developers the only ones reading bug reports?
Do you have a grounded quality model?
Are you adjusting the language for the audience?
Communication includes making sure it is understood.
In which ways are you reporting?
Just one way is rarely enough; use talking, writing, long, short, formal, informal.
Are there certain times when things need to be communicated?
Project milestones and meetings might need your information.
Some things must be communicated continuously and timely; bringing up bad news on the last day is a very bad idea (but sometimes inevitable.)
Are you using status reporting as a means for new test ideas?
I often start writing an important report early, to identify things that needs more testing.
You should start thinking about the reporting before the testing starts.
Are you reporting just because you are supposed to?
Then you should read these questions again…
If status reporting is difficult, there might be something wrong with your testing.
All-Purpose Quality Status Report Rikard Edgren 1 Comment
Generic test report:
DateTime ID Author Time for writing report Time maintaining scripts for generating report Application Under Test Areas tested
For each area: Blockers Testers Tester mood Tests planned Tests executed Tests passed Tests failed Tests remaining (untested+fail) Bugs found, per priority Old bugs status (priority, severity, tester, assignee, days active) Bug resolution aggregation
Coverage (overall, or for each area) Requirements coverage Specification coverage Code coverage (statement/decision/condition) Data coverage Models' coverage Model of models coverage Serendipity For each quality characteristic: Importance Measurement scale Points Violations Gut feeling
Enhancements Noteworthy information Other artifacts Obstacles New risks Old risk status Learnings Test plan change requests
Quality assessment Information objectives fulfillment Go/No-go recommendation upcoming activities
This is of course practically useless.
You should report what is important.
Common Sense Partitioning Rikard Edgren 9 Comments
– I saw you tested “42”. How come you didn’t try “43”?
– That’s obvious. It would be the same. No need to test something that would give the same result.
– OK, so I guess you are familiar with equivalence partitioning?
– Beg your pardon?
It is quite often said that testers don’t know the basic test design techniques.
This is both true and false.
“All” testers know how to use basic equivalence partitioning, it is common sense: you put things in groups, so you don’t have to test similar things that would yield the same information.
But they don’t know the name, and they don’t know the theory behind partitioning many types of groups.
But if they are testing all important partitions, is it a problem?
Might be, but probably not. I think a tester with knowledge about the product and its usage will make a common sense selection of the partitions that are needed to be tested for that particular situation.
For cases where you need to be very thorough this will include inputs, outputs, combinations; and extensions to examples like these:
Numericals: normal, inside/outside boundaries, min/max data type, different data types, decimals, precision
Strings: A-Z, ÅÄÖ, Unicode, empty, many, extremely many, control chars, leading/trailing spaces, things that mean other things (*, SQL, links, special values)
– You know you can’t be called a real tester if you don’t know how to do equivalence partitioning.
– Be so it. I just test what is important in my project.
Notes from SWET3 Rikard Edgren 3 Comments
I spent this weekend in nice, dark, foggy Ringsjöstrand for the third Swedish Workshop on Exploratory Testing. 
Johan Jonasson, Ola Hyltén, Anders Claesson, Oscar Cosmo, Petter Mattsson, Rikard Edgren, Henrik Andersson, Robert Bergqvist, Maria Kedemo, Sigge Birgisson, Simon Morley.
The format is LAWST-style, which means a presentation is followed by a facilitated discussion, that goes on as long as it brings value.
The theme for this event was Teaching Testing, and the abstracts can be downloaded here.
I did the first presentation using my current assignment as teacher together with Henrik Emilsson on a 2-year higher vocational study for testers.
We have a lot of exercises to make the knowledge stick, and we primarily use open source applications to make it more real.
I shared how we find suitable applications and tie them to something we are teaching, and I showed some examples of applications we have used.
The discussion that followed lasted a couple of hours and spanned many areas: course literature, complexity of software, unreal situations, authorities, ISTQB, tools, how to teach so they learn etc.
I learn a lot by teaching, and also by discussing the teaching.
The second experience report came from Johan Jonasson, who is involved as instructor in BBST online courses.
The videos are available for free, but the best thing with the education is the realistic problem-based assignments and the peer reviews.
Discussions followed for a couple of hours and included cultural and online difficulties/opportunities, how to give feedback in writing, and a summary of the slides material:
“a condensed library of test information that will broaden your views.”
After a break, we had six eight-minute Lightning Talks, where you can’t go deep, but a lot of ideas are brought up:
Anders Claesson shared his model for making sure students really learn what is thought.
Sigge Birgisson talked about his efforts to create a quality vision/model/goal for deliverables.
Rikard Edgren wondered why no one (except Oscar Cosmo) test charisma when we all know it is important. (Emilsson came up with the name Charisma.)
Henrik Andersson showed that objectivity is an illusion, so we should focus on the inherent subjectivity in test selection, execution, interpretation.
Robert Bergqvist wondered if Exploratory Testing has been more accepted, or if it just is a word in fashion.
Simon Morley has a great idea on a Groopman-inspired book “How Testers Think”, although he denies he will write it.
Then we had dinner, bubble pool, a lot of conversations on many subjects (“lean is waste for testing”), perhaps a small beer, and a good night’s sleep.
The second day only had room for one topic, which was Simon Morley on “Mindset Changes: Changing the direction of the oil tanker”, about how to teach the “right” view on testing to both managers and testers.
Many suffer from test case counting, and they have to be cured in different ways.
Simon says to testers he don’t want to hear any numbers, and thereby force them to talk about what they have seen, and what they haven’t done.
He does presentations to managers about problems with number focus, and after that he can use dashboard-like reporting.
The most important dimension might be trust.
Consensus in the room was that gut feelings are better for making decisions, but that we don’t have the proper words to talk about this.
A lot of examples and ideas were shared, but we didn’t reach any solutions before the time was up…
“if communication isn’t good, it doesn’t matter what you do”
Eleven happy check-outs with a richer network and knowledge; SWET has charisma.
Funcacles or unactivated oracles? Henrik Emilsson 2 Comments
Last week I took the Rapid Test Management (RTM) class in Gothenburg with James Bach.
As always, he is such an inspiration and really makes you engage and think. Thanks for that James!
In one part of the class we were factoring an application:
Factoring: Dimensions of Interest
Factoring is the process of analyzing an object, event, or model to determine the elements that comprise it.
When we factor for the purposes of test design, we identify elements that we may need to observe or control during a test. We call those factors.
Factors may be
— intrinsic or relational
— variable or static
(quote from Bolton’s http://www.developsense.com/presentations/2010-04-ConfirmationBias.pdf)
At CAST 2011 I was taking Michael Bolton’s tutorial on Test Framing, where we discussed a lot about oracles. Michael is, like James, such an inspiration!
(The main thing for me with taking their classes is to get involved in their brains and then I know something will trigger in mine…)
But it was on the RTM class where I got to think of something that is missing here!
When we determine the elements that comprise an application we don’t really use oracles, since an oracle is a heuristic principle or mechanism by which someone recognizes a problem.
But we do use something similar to oracles when we determine the factors. It is like if you are equipped with a lot of oracles and they don’t get triggered – and this tells you that you that you have identified a function (that works).
Also, when factoring and modeling an application, I sometimes make qualified guesses on functionality that could be, or ought to be, present (indicated with a question mark in my model) but that I’m unable to identify at the moment; or it may also be implicit or unintentional functionality that I need to investigate further.
What heuristic principle or mechanism am I using to recognize a function?
What heuristic principle or mechanism am I using to recognize an ought-to-be function?
Since it cannot be an oracle (by definition), it must be something else.
My two suggestions are:
- Funcacles (merged word that stands for Functionality Oracles)
- Unactivated Oracles
Let me elaborate.
If you apply HICCUPS(F) on an application and use the consistency oracles – but they aren’t triggered, you have that heuristic principle that tells you if something works (or that might work; or that might be applicable).
Let’s have a look at a shortened description of HICCUPS(F):
Consistency is an important theme in oracles. Unless there is a compelling reason to desire otherwise, we generally tend to want a product to be consistent with:
History: That is, we expect the present version of the system to be consistent with past versions of it.
Image: We expect the system to be consistent with an image that the organization wants to project, with its brand, or with its reputation.
Comparable Products: We expect the system to be consistent with systems that are in some way comparable.
Claims: We expect the system to be consistent with what important people say about it.
Users’ Expectations: We expect the system to be consistent with some idea about what its users might want.
Product: We expect each element of the system to be consistent with comparable elements in the same system.
Purpose: We expect the system to be consistent with the explicit and implicit ways in which people might use it.
Standards and Statutes: We expect a system to be consistent with relevant standards or applicable laws.There is one more heuristic that testers commonly apply. Unlike the preceding ones, this one is an inconsistency heuristic:
Familiarity: We expect the system to be inconsistent with any patterns of familiar problems.(Shortened quotes from http://www.developsense.com/resources/Oracles.pdf)
I believe that when we are factoring and modeling we use all the consistency oracles above, but in an unactivated fashion. However, if we want to avoid using the word Oracle for this, then I suggest Funcacle in order not to confuse (sic!).
This is an important part of test analysis that I think should be elaborated further.
I’ll get back on this as soon as I have thought about it some more.
Do you have any suggestions?