System definition and confidence in the system Martin Jansson No Comments
As a tester, part of your mission should be to inform your stakeholders about issues that might threaten the value of the system/solution. But what if you as a tester do not know the boundary of the system? What if you base your confidence of the result of your testing on a fraction of what you should be testing? What if you do not know how or when the system/solution is changed? If you lack this kind of control, how can you say that you have confidence in the result of your testing?
These questions are related to testability. If the platform we base our knowledge on is in fluctuation, then how can we know that anything of what we have learnt is correct?
An example. In a project I worked on, the end-to-end solution was extremely big, consisting of many sub systems. The solution was updated by many different actors, some doing it manually and some doing it with continuous deployment. The bigger solution was changed often and in some cases without the awareness of the other organisations. The end-to-end testers sometimes performed a test that took a fair amount of time. Quite often, they started one test and during that time the solution was updated or changed with new components or sub systems. It was difficult to get any kind of determinism in the result of testing. When writing the result of a test, you probably want to state which version of the solution you were using at the time. But how do you refer to the solution and its version in a situation like this?
When you test a system and document the result of your tests you need to be able to refer to that system in one way or another. If the system is changed continuously, you somehow need to know when it is changed, what and where the change is as well. If you do not know what and where there are changes, it will make it harder for you to plan the scope of your testing. If you do not know when, it is difficult to trust the result of your tests.
One way of identifying your system is to first identify what the system consists of. Considering the boundary of the system and what is included. Should you include configuration of the environment as part of the system? I would. Still, there are no perfect oracles. You will only be able to define the system to certain extent.
System version 1.0
System version 1.1
System version 1.2
component 1 version
component 2 version
component 3 version
As you define parts or components of the system, you can also determine when each are changed. The sum of those components are the system and its version. I am sure there are many ways to do this. Whatever method you choose, you need to be able to refer to what it is.
I think it is extremely important that you do anything you can to explore what the system is and what possible boundaries it could have. You need many and different views of the system, creating many models and abstractions. In the book “Explore IT!”, Elizabeth Hendrickson writes about identifying the eco system and performing recon missions to charter the terrain, which is an excellent way of describing it. When talking about test coverage you need to be able to connect that to a model or a map of the system. By doing that you also show what you know are coverable areas. Another way of finding out what the system is using the heuristic strategy model, by James Bach, and specifically exploring Product Elements. Something that I have experienced is that when you post and visualize the models of the system for everyone to see, you will immediately start to gain feedback about them from your co-workers. Very often, there are parts missing or dependencies not shown.
If one of your missions as a tester is to inform stakeholders to make sound decisions, then consider if you know enough of the system to be able to recommend a release to customer or not. Consider what you are referring to when you talk about test coverage and if your view of the system is enough.
- Explore It! by Elisabeth Hendrickson – https://pragprog.com/book/ehxta/explore-it
Heuristic Test Strategy Model by James Bach – http://www.satisfice.com/tools/htsm.pdf
The Oracle Problem – http://kaner.com/?p=190
A Taxonomy for Test Oracles by Douglas Hoffman – http://www.softwarequalitymethods.com/Papers/OracleTax.pdf
Scripting Your Test Data Rikard Edgren No Comments
Sometimes I wonder if testers know how easy it is to script your own variations of test data.
I prefer Ruby, and you can download this example that I will tell you about.
I was testing healthcare data and wanted to see what the performance was for larger quantities of data. We had a mock service, and the data to put there was easy to create with a script.
For the “diagnosis” area, I had an Excel sheet with the 12441 possible diagnosis codes according to ICD-10-SE. I couldn’t resist creating a test patient that had all of these diagnosis.
This will never, never happen in reality, and does not add value to the performance tests, but I did it anyway, it was fun and fast.
After the performance tests where completed I continued using the test data I had created.
It is a kind of background complexity that isn’t really necessary, but doesn’t cost a lot, and might help you discover new things. And of course it did also this time (hey, I chose the example).
When testing search functionality I saw behaviors I hadn’t seen with the more simplistic data I had elsewhere. The large variety of diagnosis names gave possibilities for the search function to go wrong.
If you aren’t already doing stuff like this, feel free to edit the Ruby script to match your needs (most data files are text in some kind and can be scripted in this way) to create more variety to your test data.
Quite often, your tests aren’t a lot better than your test data.
Five Tricky Things With Testing Rikard Edgren 5 Comments
I went to SAST Väst Gothenburg today to hold a presentation that can be translated to something like “Five Tricky Things With Testing”. It was a very nice day, and I met old and new friends. Plus an opportunity to write the first blog post in a long time, so here is a very condensed version:
1. People don’t understand testing, but still have opinions. They see it as a cost, without considering the value.
Remedy: Discuss information needs, important stuff testing can help you know.
2. Psychologically hard. The more problems you find, the longer it will take to get finished.
Remedy: Stress the long-term, for yourself and for others.
3. You are never finished. There is always more to test, but you have to stop.
Remedy: Talk more to colleagues, perform richer testing.
4. Tacit knowledge. It is extremely rare that you can write down how to test, and good testing will follow.
Remedy: More contact of the third degree.
5. There are needs, but less money.
Remedy: Talk about testing’s value with the right words, and deliver value with small effort, not only with bugs.
Summary: Make sure you provide value with your testing, also for the sake of the testing community,
There were very good questions, including one very difficult:
How do you make sure the information reaches the ones who should get it?
Answer: For people close to you, it is not so difficult; talk about which information to report and how from the beginning. I don’t like templates, so I usually make a new template for each project, and ask if it has the right information in it.
But I guess you mean people more far away, and especially if they are higher in the hierarchy this can be very difficult. It might be people you aren’t “allowed” to talk to, and you are not invited to the meetings.
One trick I have tried is to report in a spread-worthy format, meaning that it is very easy to copy and paste the essence so your words reach participants you don’t talk to.
Better answers is up to you to find for your context.
Serendipity Questions Rikard Edgren No Comments
This Tuesday I held a EuroSTAR webinar: Good Testers are Often Lucky – using serendipity in software testing (about how to increase the chances of finding valuable things we weren’t looking for)
Slide notes and recording are available.
I got many good questions, and wanted to answer a few of them here:
How can we advocate for serendipity when managers want to cut costs?
Well, the “small-scale serendipity” actually doesn’t cost anything. It just requires a tester to be ready for unexpected findings, and sometimes spend 20 seconds looking at a second place. The cost appears when investigating important problems, but in that case, I would guess it is worth it (never seeing any problems or doing no testing at all would be the lowest cost…)
I also know that many testing efforts involve running the same types of tests over and over again. When you know these tests won’t find new information, maybe it is time to skip them sometimes and do something rather different?
Do you have issues finding the root of the problem considering you are doing many variations?
If it is a product I know well, I don’t have problems reproducing and isolating. But if it is a rather new product it can be more difficult, but I would rather see these problems and communicate what I know, than not see them at all!
To take more detailed notes than normally, or to use a tool like Problem Steps Recorder (psr on Windows) can help if you expect this to happen.
Is there any common field for automated testing and serendipity?
It is easy to think that automation is a computeresque thing without a lot of manual involvement and tinkering with the product. But in my experience, you interact a lot with the product while learning and creating your tests. And I make mistakes that can discover problems with the product’s error handling.
I know this combination of coding and exploratory testing happens a lot, but it is not very elaborated in the literature (but the recent automation paper by Bach/Bolton have good examples on this.)
Another example of automation and serendipity is that you combine human observations while the tests are running. A person can notice patterns or anomalies, or maybe see what the users perception is when the software is occupied with a lot of other things.
Computers are marvellous, but they suck at serendipity.
I believe one of the most important traits of testers is that we bring new perspectives, new ideas about what is relevant.
I probably believe this from my experiences from the first development team I joined, so I will tell you about the future by telling an old story.
This was in Gothenburg, 15 years ago, and we developed a pretty cool product for interactive data analysis. Data visualization, data filtering and calculations, and we could even use the product on our own bug system. The team consisted of quite young men and women who had all gone to Chalmers, the technical high school in Gothenburg.
They had taken the same lectures, they had done the same exercises.
They collaborated well, using the modelling tools, and the thinking thinking patterns, they had learnt in school.
They weren’t exactly the same of course, they had different haircuts, personalities, specialities, but all in all, they had roughly the same ideas about how to design and develop products.
I was the first tester in their team, and I had not gone at Chalmers. At University I read philosophy, musical science, history of ideas, pratical Swedish; and rather stumbled on testing because I wanted to be a programmer. I did not think at all like the rest of the team, and that was the good part!
I saw perspectives they didn’t see, my set of mental models contained other elements than theirs.
So when they agreed a solution was perfect, I asked “but what if this box isn’t there?” or “can we really know that the data is this clean?” or “what if the user tries this?” or “isn’t this too different from this other part of the product I looked at yesterday?” or “how on earth should i test this?” or “how useful is this really?”
They were a great team, and they used my perspectives to make the product better.
I felt valuable, and maybe that’s when I started loving testing (well, maybe earlier, I have always enjoyed finding big bugs, and I will always love it. But that’s more a kind of arousal, I am talking about a deeper love, when you feel that you provide value others can’t.)
So when we get to 2030, a lot of things will be the same, and a lot of things will be different. There will definitely be a need for people carefully examining software, and bringing new perspectives, and new questions. A richer set of mental models are needed, regardless if we are called testers or something else.
But it will be new, new perspectives, and you should look out for these, and use them.
You should learn stuff, you should test software appropriately, you should embrace new situations and perspectives, and you will be ready in 2030.
I hope I will too.
Test Strategy Checklist Rikard Edgren 8 Comments
In Den Lilla Svarta om Teststrategi, I included a checklist for test strategy content. It can be used in order to get inspired about what might be included in a test strategy (regardless if it is in your head, in a document, or said in a conversation.)
I use it in two ways:
- To double-check that I have thought about the important things in my test strategy
- To draw out more information when I review/dicuss others’ test strategies
It might be useful to you, in other ways, and the English translation can be downloaded here.
Special thanks to Per Klingnäs, who told me to expand on this, and Henrik Emilsson, who continuously give me important thoughts and feedback.
Lessons learned from growing test communities Martin Jansson 3 Comments
In 2011-2012 a friend, Steve Öberg, started a local community with a few test friends. We talked about testing, sharing experiences and discussing/arguing about various test topics. It was a great initiative, but I wanted something more and bigger. I had a discussion with Emily Bache, who run a local meetup on programming. She inspired me to take the step into meetups. It seemed like a great way to organize meetings. I created Passion for Testing and started to invite friends interested in testing.
Creation of the meetup Passion for Testing
I had a few ideas on what I wanted to achieve and things I did not want to see.
I created a recruitment message about the meetup as a way to find people who had similar mindset. Here is roughly what I included in the meetup description of people who should probably join, what they probably have interest of:
- a passion for testing
- a will to learn new things
- an interest to learn more about exploratory testing
- an interest to try new techniques and methods in testing, to see what might work in your context
- an interest to practise testing as well as theorize about it
- an interest in experimenting with collaborative testing
- an interest to explore how to improve cooperation between developers, testers and other roles in a team or project
- an interest to help the open source community by providing them with valuable quality-related information about their products or services
- an interest to help research in testing by saving artifacts produced by our testing and experiments
- an interest to network with other people who are interested in testing in Gothenburg
For instance, I did not want a meeting where one person did the talk and the rest listened, then walked home. Thus without discussion or conferering. At many conferences the idea of having a discussion after the talk has been popular, something I find strange not to have.
I also wanted the meetup to be inclusive, welcoming all who wanted to learn more about testing. I wanted people who was inquisitive and interested in testing to join. It was ok to just come and listen, knowing that there were some in the audience who would debate and question what was said.
I wanted us to challenge existing practises and ideas by experimenting with them, trying the theory in practise. I welcomed new ideas on how to do things. If someone had a half-finished course, talk or idea our network was an excellent place to try it out.
I wanted to participants to ignore who their current competitors were and instead share their experience and ideas for free. Sharing material, links or other artefacts to help each other in our daily work.
I wanted to help the local startups and open source communities by helping them with testing, knowledge about testing and perhaps a handful of bug reports. Helping startups with testing, who quite often lacked testers, felt like a good thing. If that enabled them to reach a level of quality that made them succeed, then that was great.
I wanted to make the artefacts produced from these meetups to be public, so that they could be used by anyone and possibly in research. Still, this was difficult and hard to do in practise. The intent was there.
I did not want a too large group, perhaps max 15-20 people attending. More than that you have a hard time discussing.
I welcomed any non-tester to be part of our meetups. To see their perceptions added to the discussions, but also perhaps to give them some new ideas on what to expect from testers.
I wanted each meetup to have different sponsors, avoiding to sponsor the events with my own companies to avoid making them biased. I wanted the sponsor to be part of what we did and what we talked about, not just as a way for cheap exposure.
I wanted inexperienced speakers to speak up in a friendly, small audience. To practise their speaking skills. For me personally, this proved to be a great way to become better at speaking infront of an audience. As a separate, but similar effort I especially liked the initiative by Fiona Charles and Anne-Marie Charrett called Speak Easy to help speakers practise.
I welcomed topics that were difficult or that weren’t well polished, I wanted the speakers to go out on the thin ice. The heated dicussions were excellent. I have had several of these and learned a lot. Failing is a great experience.
I also wanted to get to know more testers locally, to see if they walked the walk and talked the talk. I wanted to see them solve puzzles, handle discussions on test strategy, both difficult and new areas in testing. I wanted a group of passionate testers whom I probably wanted to test with in future projects or companies. Probably people who I would someone recruit or help others recruit. This has been proved true and extremely effective. If anyone of those, who I thought were great, had a certificate or not was not important at all.
I wanted students in testing, junior testers or unemployeed people to get free lectures, workshops and courses. I also wanted them to be able to show how good they were to enable them to meet possible emplyers or recruits. This has also proved successful.
Personally, I use this meetup to become better at testing, talking about testing and finding new areas in testing that need exploring. It is my own playground.
Evolving Passion for Testing
After organizing a lot of the meetups myself, I realised that it is not possible to do this as a one-person-show. I wanted help from co-organizers to evolve what we were doing. Steve Öberg and Fredrik Almén helped me by organising Test Clinics, inspired by Michael Boltons 1 day test clinit at the end of his RST courses. The Test Clinic was held at a local pub were we as a group helped each other resolve our daily problems/obstacles in testing.
In the coming months I am trying a light version of the LAWST-style conference by letting the local meetup have an evening together. We have one subject presented by one speaker. We anticipate several hours of talk and discussion. We invite both experienced and inexperienced attendents. Everyone is expected to speak up and will be able to using the moderator form from LAWST. It will be interesting to see if we are able to make this a great, humble learning experience for all those participating.
I want more people to join but also more people to help organize events in testing. If you are interested to help out, feel free to contact me. All help is welcome!
This is my shared experience and ideas on setting up a meetup on testing. If you want to use my ideas on this, feel free to do so. If you wish to discuss the setup you can reach me on skype.
If you wish to join the meetup, you can do this here:
Not on Twitter Rikard Edgren No Comments
I don’t have a Twitter account.
I read Twitter now and then, it contains useful information, but I don’t have the time to do it properly. For me, doing it properly would mean to often write thoughtworthy things within 140 characters.
I only have one of those, so better publishing it in a blog post:
What about doing manual regression testing to free up time to make valuable automation?
So, that was one Christmaas tweet, and it did the opposite of decreasing my blogging frequency (which is the general drawback of testing tweeting in my opinion.)
Lots of test strategy Rikard Edgren No Comments
I have been doing lots on test strategy the last year. Some talks, a EuroSTAR tutorial, blog entries, a small book in Swedish, teaching at higher vocational studies, and of course test strategies in real projects.
The definition I use is more concrete than many others. I want a strategy for a project, not for a department or the future. And I must say it works. It gets more clear to people what we are up to, and discussions get better. The conversations about it might be most important, but I like to also write a test strategy document, it clears up my thinking and is a document to go back to for other reviewers.
Yesterday in the EuroSTAR TestLab I created a test strategy together with other attendants, using James Bach’s Heuristic Test Strategy Model as a mental tool. The documented strategy can be downloaded here, and the content might be less interesting than the format. In this case, I used explicit chapters for Testing Missions, Information Sources and Quality Objectives because I felt those would be easiest to comment on for the product manager and developer.
I have also put my EuroSTAR Test Strategy slides on the Publications page.
Testing Examples Rikard Edgren 8 Comments
I believe we need a lot more examples of software testing. They will be key in transferring tacit knowledge (they will not be all that is required, but an important part.) They work best when done live, so you can discuss, but that doesn’t scale very well.
So I have created a few examples in video or text format:
Exploratory Testing Session – Spotify Offline
Scenario Testing – LibreOffice Compatibility
Test Analysis – Screen Pluck
I am very interested in knowing if they are useful to you, and how.
Which other public testing examples are your favorites?