Regarding comments the test eye No Comments
Due to all spam comments not being stopped by our automatic checkers, we are unfortunately forced to moderate all comments.
We will do this as quick as possible, but there might be some delay before your comment appears.
Hope that you understand this, from our point of view, drastic decision.
Henrik, Martin & Rikard
I am an Exploratory Tester Rikard Edgren 5 Comments
I am curious about how the system works
I look at details and the whole, and at many places
I use many sources to learn what is important
I am creative and see many testing possibilities
I test in many different ways, and adjust to the situation
I am good at finding important problems fast
I am aware of models enabling and limiting my testing
I can evaluate a system from many dimensions
I can communicate my thinking and findings
I am an exploratory tester
Suppose you have a risk assessment fully agreed by stakeholders, and their relatives.
Create a stable sampling across all adjacent perspectives, and let the corresponding tests be executed on best representatives platforms, theoretically covering 99.5% of potential future usage.
Now, let any opposites conglomerate, and hold their best fit tempo, until the sounds dissolve into the night.
The information is tunneled into various status report templates according to confidently standardized information objectives.
No need for the Black Viper test technique anymore!
Intertwined SFDPOT & CRUCSPIC STMP Rikard Edgren 3 Comments
I hope many of you are using SFDPOT (James Bach) and CRUCSPIC STMP (thetesteye.com) in order to investigate what to test. SFDPOT describes elements of the product, and CRUCSPIC STMP describes sought attributes of the system. They are very powerful ways to identify things to test, plus to be able to communicate it effectively.
Both are very rich, and could incorporate each other, e.g. SFDPOT could be a part of Capabilities, and CRUCSPIC STMP could be a part of Operations (often inverted as a risk) There are more connections, as seen here:
I still recommend using both as separate activities, they stimulate thinking in different dimensions (and that’s why testers are needed, right?) That they blend is not a problem, it is the thinking that matters, so just put the stuff where it makes most sense.
If this makes almost no sense to you, interpret as a hint to try both!
Lightweight Charisma Testing Rikard Edgren No Comments
One heavyweight way of testing charisma is to use dozens of potential users on dozens of alternative product solutions/prototypes.
For lightweight charisma testing, it is often fast and fruitful with an awareness of charisma violations.
This method requires an understanding of the unique charisma for your product.
Testers probably won’t be in charge of developing an explicit charisma model, but you can take part, e.g. by recommending the following process:
1. Write down what you believe are the special things about the software (gamemakers might use “playability”, but please be more specific)
2. Get inspired by thetesteye’s generic list of Charisma characteristics (provided below)
3. Document the important characteristics with as much details as possible (to make’em truly mean things in your context.)
4. Share, review, refine and gain approval for your Charisma Guidelines across the teams
When you have this list, it is fun and attention-inspiring to keep these things in the back of your head whatever type of testing you are performing.
Just remember to report appropriately, probably verbal.
Charisma: Does the product have “it”?
- Uniqueness: the product is distinguishable and has something no one else has.
- Satisfaction: how do you feel after using the product?
- Professionalism: does the product have the appropriate flair of professionalism and feel fit for purpose?
- Attractiveness: are all types of aspects of the product appealing to eyes and other senses?
- Curiosity: will users get interested and try out what they can do with the product?
- Entrancement: do users get hooked, have fun, in a flow, and fully engaged when using the product?
- Hype: should the product use the latest and greatest technologies/ideas?
- Expectancy: the product exceeds expectations and meets the needs you didn’t know you had.
- Attitude: do the product and its information have the right attitude and speak to you with the right language and style?
- Directness: are (first) impressions impressive?
- Story: are there compelling stories about the product’s inception, construction or usage?
Initial thoughts on group testing Martin Jansson 3 Comments
I want to open for a discussion on pair testing or at least widen the concept. By saying pair, we say two people, but why should we limit us to that? Depending what your objective is a different number might be more applicable?
Several of the referred authors in this article have elaborated around pair testing as something that could be wider and be used in different ways, many of the things I have expressed below has also been brought up by them. Still, I’ve added some parts and changed the scenery a bit.
Looking back at pair testing
“Pair testing is a way of approaching a test design process by having two people test the
same thing at the same time and place, continuously exchanging ideas. Even without any
special method or process, the dynamics of pairing enables the generation of more and
different ideas than either tester is likely to produce on his own. It’s an effective
complement to individual testing.” a quote from a draft article by James Bach and Sam Guckenheimer  on Pair testing.
This draft goes through the basic concept of pair testing. They discuss how it can be used in different forms and where it might be useful and in which situation.
“Two testers and (typically) one machine.
• Typically (as in XP)
– Pairs work together voluntarily. One person might pair with several others during a day.
– A given testing task is the responsibility of one person, who recruits one or more partners (one at a time) to help out.
• We’ve seen stable pairs who’ve worked together for years.
• One tester strokes the keys (but the keyboard may pass back and forth in a session) while the other suggests ideas or tests, pays attention and takes notes, listens, asks questions, grabs reference material, etc.” a quote from an article on Exploratory pair testing by Cem Kaner and James Bach .
Here James Bach and Cem Kaner elaborate around exploratory pair testing, how they suggest the pairing should be done, benefits of doing it and some drawbacks and risks with it. They also identify that it can be done in different combinations and that it is suitable in different situations.
“Working alone is working without a safety net” a quote by Brian Marick in an article  on Pair testing.
He summarized what seems to be a discussion in a workshop, where several parties were present. Several examples are listed where pair testing would be good, but he also identifies areas when it is not suitable. Furthermore he points out the importance of it not being more than two, thus naming it pair testing.
Jonathan Kohl takes on a different approach in an article about Pair testing . He gives some examples where he has worked with developers in pair testing sessions with great result. He provides some hands-on tips on how to go about this and things to consider. He is focused on working in pairs, but he also adds the concept of using this technique as a way to teach testing and to grow the relationship between different parties in the organisation.
“As in pair-programming, two people simultaneously test an application at the same computer. It can involve: testing an application from a customer’s perspective and more technical kinds of tests”, a quote from a presentation  about Pair Testing by Jonathan Kohl.
In this presentation Jonathan elaborates deeper around the usage of the Pair testing technique. He touches a range of areas and brings up some other aspects than previous writers.
One thing that I notice from the various writers is that you use pair testing or as a technique to accomplish different objectives. Each writer includes different things to the concept of pair testing. Based on that, I saw that a pair testing session could consist of different dimensions. One of these dimensions was the amount of testers involved in a session, which in turn made me question the name of pair testing. Instead I call it group testing, since I think it can be two or more based on the objective. Beside considering that there can be different amount of testers I saw that there are other dimensions worth exploring.
Dimensions of group testing
There are many dimensions that could be investigated and experimented with. Here are the initial dimensions I’ve identified and experience with:
- Amount of participants – how many are part of the group testing session?
- Role as a tester – which role do you play as a tester? Is it explorer/tester, documenter/observer, checker, coach/trainer/teacher, reviewer, manager or something else?
- The scene – Are you running a specific user scenario or perhaps combining a set of user types? What scene have you painted that you are acting in? What personas are valid in this scene? What storytelling aspects have you considered when setting up this scene?
- Mission of group test – What is your objective[s]? What do you wish to have achieved when the group test session is finished? Do you wish to find information quickly, teach, coach or train a member of the group, review testing skill, report information quickly, help trouble shooting or something else?
- Note taking style – How are you taking notes? Do you take note in one single document that all of you write in simultaneously, is one of you taking notes only or are you all writing your own material?
- Partner combo – Which other stakeholders are part of this group test session? Do you pair up with developers, project managers, line managers, CEO, sales and marketing, system managers and business analysts … who?
- Lateral thinking style – Should part of the participants take on different lateral approaches such as thinking the opposite or wearing different thinking hats?
- Personality mix – Consider how you mix personalities such as introverts, extroverts, slow thinkers and quick ones. Which combinations work better than others in your context?
- Debriefing style – Do hold a debrief after each session or is it sufficient with the ongoing discussion you have?
- Accountability focus – Do you have one driver who is accountable or do you have many driving at the same time?
- Focus areas – Does each participant focus one specific characteristic (usability, security etc), a mix of focus areas between persons or possibly everyone focus on one characteristic?
- Test Environment setup – Do you sit behind one system or test environment, one system or test environment per person or possibly a mix of systems per person?
- Session duration – Do you do a longer session or short ones? Do change driver during or after each session, either way consider how this is setup depending on the duration.
There are many dimensions to the group testing technique that need to be experimented with. My point to this is that I fail to see the point in limiting the technique to a specific setup, but instead let its content/dimensions change depending on context. There are a few major pros that you probably get by using this technique, no matter which dimension you alter. Some of these are:
- an increase in creativity
- an increase in cooperation in the team/organisation between different roles
- an ongoing training of the participants in expressing themselves both in written and verbal form
Based on my own experience, I think this is an excellent way of testing with great result. Therefore I intend to dig further in this area to see what comes up.
 Pair testing (aka Extreme Testing) by James Bach and Sam Guckenheimer – http://www.exampler.com/testing-com/test-patterns/patterns/XT-Pattern-jb.pdf
 Exploratory testing in pairs by James Bach and Cem Kaner – http://www.kaner.com/pdfs/exptest.pdf
 Pair testing by Brian Marick – http://www.exampler.com/testing-com/test-patterns/patterns/pair-testing.pdf
 Pair testing – How I brought a developer into the testlab by Jonathan Kohl – http://www.kohl.ca/articles/pairtesting.html
 Presentation on Pair testing by Jonathan Kohl – http://www.kohl.ca/presentations/pair_testing.pdf
Scenario Testing, Karlstad 2012 Rikard Edgren No Comments
I have been doing and teaching quite a lot of scenario testing lately.
I have been surprised by the ease and speed, and ability to find important problems, (also giving an embryo to a compelling bug report.)
Maybe it can be useful for you as well, probably as a complementary technique, or as a powerful activity for mature products.
“A scenario is a coherent story about how someone uses (or tries to use) the program.
A scenario test uses a scenario as a tool for evaluating a program’s behavior
The elements of the story (adapted from Carroll, 1999)
- Agents or actors
- Goals or objectives
- Motivations and emotions
- Plot (sequences of actions and events)
- Actions & events can change the goals”
(slide 301 in BBST, Test Design)
When documenting, I also add a purpose of the scenario test, so it is clear why we’re doing it.
Kaner lists five key characteristics for an ideal scenario test: story, credible, motivating, complex, easy to evaluate:
- “The test is based on a coherent story about how the program is used, including goals and emotions of people.
- The story is credible. Stakeholders will believe that something like it probably will happen.
- The story is motivating. A stakeholder with influence will advocate for fixing a program that failed this test.
- The story involves complexity: a complex use of the program or a complex environment or a complex set of data.
- Test results are easy to evaluate. This is important for scenarios because they are complex.”
(slide 302 in BBST, Test Design)
I am not rigid about Easy to evaluate, rather the contrary; I want rich scenario tests and trust our ability to find out the details when encountering problems.
To find out what to base the scenario around, Kaner suggests 17 ways.
These can help, but you get even further by understanding what’s important. To speed that understanding, 37 Sources for Test Ideas might help.
Build the scenario around something important, add complexity (sequences, data, environment) and a credible goal, often spanning outside your product.
I guess you’d like an example, but I won’t give you that, because too many of you will use it as a template and don’t make the effort to find out what matters in your context.
But make sure to not try to cover as much as possible, and don’t be afraid to add corny details (they add life.)
Try it and see if the technique can help you!
My Very First Testing Lesson Rikard Edgren 4 Comments
As everyone else, I fell into testing by chance. I wanted to work as a developer, and saw testing as a stepping stone (this quickly changed, though.)
My first day I tested a Service Pack of a big, localized product. An experienced tester guided me at the start, and I can still remember the conversation.
Rikard: So is this machine the right one to use for this test?
Anders: Yes, you can start testing. Tell me what you are thinking, so it’s easier for me to help you.
Rikard: I double-click the installer and a welcome screen appear. I click Next.
Anders: Wait! Did you read the text?
Anders: Well, read the text and tell me what you think about it.
Rikard: OK. I think the welcome screen looks good, it tells me what it’s about to do, and the Swedish is correct.
Anders: Good. You know, some people actually reads this kind of information…
Rikard: Now I follow the script and select a custom install and click Next. I select all components as the test case tells me and click Next, and now it’s time to actually perform the installation.
Anders: Stop! Why do you just follow the script and click Next all the time?
Rikard: Because it tells me to.
Anders: But do you think all users will follow a script and know what to do all the time?
Rikard: I suppose not all of them.
Anders: So then you need to do some variations, click Back and change your mind, act like a normally confused end user.
Rikard: But why isn’t that included in the test steps?
Anders: You can’t specify everything, you can’t just do only what’s required; you are a tester, and supposed to find most of the important bugs.
I did variations, I soon found that the installer would forget the selected components after Back/Next, and that it was easy to not be aware of this.
The bug was reported and fixed, not a big deal perhaps, but a very important lesson for me:
Regardless of what they say, they probably want you to find issues that will annoy end users.
The First Bug Martin Jansson 2 Comments
Last week I entered the first bug in a bug system for a new service that I just started to work on. Me and my team spent quite some time in getting it right, setting the standard for bugs to come. If the first bug is crappy, the rest can be as well. We considered the format, what was included in form of logs etc, how to reference to the build, the configuration setup, the ever so important title and the to-the-point-repro-steps. We did not add any expected result, it was too obvious. If had, it would have made some of the stakeholders go Duh! most probably. We even tried to make sure that the pinpointing task would be so much easier. As I see it, we made a good case for getting it prioritzed to get fixed.
I even set up some guidelines on how to report bugs with some nice references to BBST Advocacy course and other great materials from blogs.
Being first has its advantages. You set the level of expectation and hopefully you set your own ambition for what is to come.
Open Source Testing – RedNotebook Rikard Edgren 3 Comments
In Julian Harty’s keynote Open Sourcing Testing at Let’s Test conference, RedNotebook (together with Sikuli and CiviCRM) was suggested as an open source project where ambitious testers could collaborate and share their testing efforts and results.
I have done a small start by contacting the RedNotebook team (they are interested in bugs and enhancements, especially in the areas of usability and performance.), and creating 7 charters (or exercises) to start with.
Details can be found at https://answers.launchpad.net/rednotebook/+faq/1971
The charters are available in the question forum (we’ll start there), where methods (here’s where we’ll learn) and results (here’s where the project will learn) can be entered as answers:
1: Test changes in the latest RedNotebook version
2: Usability heuristic evaluation
3: Performance investigation
4: Evaluate automation tool applicability
5: Platform compatibility
7: Bug Advocacy
Perform these, create new or test in your own preferred ways.
If we test the same product and share the way we do it, we can improve our testing skills!