thoughts from the test eye

“Wouldn’t it be cool if we could come up with a Quality Value for our products?”  said a colleague of mine. “Yes! That would be super!” me and a couple of colleagues answered.

We had a lot of categories and data in our bug system; and perhaps most important was that we had a good data mining tool that enabled us to take the data and transform it by performing calculations and making aggregations of it.

We started out by analyzing the bug data and starting to come up with reasonable and weighted factors that would enable us to quantify the categories: Severity, Priority, Time to fix, Resolution, Bug Type,  etc. Then we constructed an algorithm that would go through all essential data and the result would be a numeric value, i.e. the Quality Value. We decided that the Quality Value should be in somewhere in the span 0-100 and scoring 100 would be the top Quality Value.
When we discovered anomalies in the result, we tuned the algorithm and the quantifiers so that the result would make more sense; and a lot of discussions were about the quantifiers and their impact on the result. After several iterations we started to get some reasonable numbers.

And by now you might have started wondering how we noticed the anomalies and why we could see that the numbers were reasonable? This happened because we already had a perceived value of the products so we were  biased by subjectivity (huh!).

Anyway, when we were satisfied with the numbers we had a marvelous Quality Value for each and all of our products! And I dare to say that we strongly believed in this Quality Value. Of course there was a constant debate on this subject, and we fought over how much certain categories should have impact on the overall value.
“My product” scored in the top interval so I was very pleased.
But pretty soon, somewhere inside of me, I heard a little voice whisper: “Ignoratio elenchi, ignoratio elenchi, …”

We were of course very naïve in that we believed that this metric would represent the quality of the product. Of course it didn’t!
A couple of observations:

• Bug data only deals with reported bugs
• Bugs are subjective
• Bug reporting is subjective
• Bug handling (management) is subjective
• Bug fixing is subjective
• All other quality criteria not caught in bug reports are not included in bug data
• Quality is value to many people that haven’t reported anything
• Bug data is only data about bugs (+ subjectivity)
• All of the above means that we really cannot compare bugs with each other

On the other hand, one conclusion we came up with that might be true was that the ability to care about the product and bugs were reflected in the Quality Value. And in some way, this meant that a high score indicated that the product was taken care of (see Broken window theory and quality). While this might have been true, we were ever so wrong with the idea on capturing the Product Quality in a single Value…

Read about Ignoratio elenchi

Also see The Quality Status Reporting Fallacy

I have written a small article for The Testing Planet about Testing & Creativity. (update: web article here)
It contains the potato from this blog, but also some new content on “cheating”.
Feel free to add comments here!

I think creativity still is too little recognized as an important aspect of our work.
We should promote it more, both because it is a vital aspect, but also to make the profession more interesting to the next generations.

There are (at least) two upcoming presentations on creativity in software testing:
Rob Lambert – Is excessive structure killing our creativity, blog, video, program.
Rikard Edgren – 5 Steg Till Kreativ Testning, SAST 15 år (a Swedish version 2 of EuroSTAR 2007 presentation Where Testing Creativity Grows)

There are a lot of other content in The Testing Planet paper, my favorite being a compelling story from Scott Barber on the publically available Black Box Software Testing course at testingeducation.org.

The Black Swan, The Impact of the Highly Improbable by Nassim Nicholas Taleb

Eyafjallajukull started to spew out tons of ashes in the sky. Air Traffic over Europe was severely hindered and stopped totally for a number of days. This is a perfect example of a Black Swan Event.

What we call here a Black Swan is an event with the following three attributes. First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable. I stop and summarize the triplet: rarity, extreme impact, and retrospective (though not prospective) predictability. A small number of Black Swans explains almost everything in our world, from the success of ideas and religions, to the dynamics of historical events, to elements of our own personal lives.

The author is a former professional trader who has been studying uncertainty, luck, risk and knowledge for many years. All models used in finance are built on the faulty assumption that it is possible to predict stock prices, oil prices and economy as a whole. We assume that we live in Mediocristan were all things are nicely described by Gaussian bell curves. While this is true for some distributions like height and weight of a human being, it is totally wrong for things like stock market movement, fortune of a person, development of technology and other things where it is often used. We need to understand that we live in Extremistan where our lives are largely dependent on random events and luck.

Taleb does not tell us how to predict Black Swans but he tries to help us make some of them grey and to anticipate that the future has many surprises for us. The purpose of the book is to help the interested reader understand why we should never believe in forecasters and stop trying to predict everything. The goal: avoid being a sucker.

Some of his statements are:

• The Ludic Fallacy: basing studies of chance on the narrow world of games and dice,
• Epistemic arrogance: our hubris for how much we know or are capable of.
• Narrative Fallacy: our need to fit a pattern to a series of connected  or disconnected facts
• Confirmation Error: looking for instances that fit your belief and find them

What can a tester learn from this information?

• Understand that estimations are very hard to make and using more advanced models like S-curves will not give us better estimations only more detailed. Understand also that it is impossible to know what critical problems will appear in what you are working on which implies that having an open mind when testing increases your chances of finding these unknown unknowns much greater.

• The beautiful word Serendipity – where you find things by chance – is behind most, if not all, human inventions. So using exploratory approaches in life as well as in testing will most likely yield a higher return than will following detailed procedures.

• Whatever models we create they will always be incomplete. So I wonder what it after agile and what is next after Exploratory testing.

• Like he says – I would rather be approximately right than precisely wrong which is why I like the low tech dashboard better than curves or tables of test cases run and failed.

There are many great insights in this book and I truly recommend it. After writing this article I seriously consider whether or not I have fallen for the Confirmation Error!

In your current assignment as a tester, test lead or test manager think about how the situation is for the test group right now.

At the stand-up morning meeting with the test group, stop and consider…

• what is that troubles the group?
• what obstacles do they see?
• what is their main concern?
• what is stopping them from performing their work?

At lunch with your co-workers, stop and listen…

• what do they complain about?
• what is making them frustrated?
• what is the work related topic that comes up every so often?

At the meeting with requirement writers, stop and consider…

• do you think the test group has better knowledge of the customer than those writing requirements?
• do the requirement writers ever ask what the test group think about new features?
• do you feel that the cooperation and collaboration is good with the requirement writers?

At the meeting with project management, stop and consider…

• what issues are brought up at almost every meeting but are usually ignored?
• what risks get lower or higher attention than others by the project manager?
• how is the relationship between the project members and the project manager?
• what risk areas are always brought up at every project and always come true?
• have you communicated your belief in the project plan?
• do you feel that cooperation and collaboration is good with the project members and the project manager?

When discussing with developers, stop and consider…

• do you provide what they are asking for?
• is there a conflict growing here? Are you nurturing the conflict?
• do you feel that the cooperation and collaboration is good with the developers?

At the line meeting with the test manager and the rest of the test group stop and consider…

• is the focus different on issues  than that the project[s] think are important?
• is the focus on getting better as a test group or is it clouded by other issues, what are these in that case?
• how many in the test group have been degraded down to a tester and are now stuck there, as they see it?
• do the internal conflict in the test group outgrow the external conflicts?
• do you feel that the cooperation and collaboration is good within the line?

When you are testing, stop and consider…

• are there any bugs you ignore and choose not to report?
• are there any areas that you know bugs won’t get fixed in that you choose to avoid?
• are there any areas that you have to little knowledge about that you choose to avoid?

Before you are about to submit your bug report, stop and consider…

• have you considered how many stake holders will look at this and how much time they will spend reviewing it? Do you ignore that fact?
• do you think the developer will be able to fix the bug with all information that you have provided?
• do you spend minimal amount of time possible on writing the report?
• do you care for and take pride in the bug report?
• do your bug reports usually get fixed or are they returned because of lack of information?

Before you send your status report about testing, stop and consider…

• have you provided truthful information, as you see it, that you think will be valuable to the reader?
• have you emphasized any area that you personally just want to get more focus?
• have you emphasized any risk that is not that risky?
• do you think stakeholders would be able to make good decisions based on the information you have provided?

Now, what is the trend you see? What picture have you painted? The next hard thing is what you intend to do about it. Remember, it is often good to start with yourself.

I stopped myself in the hallway just to ask a few quick questions:

Why do you like software testing?
– That’s not a short question! But it involves a lot of creativity, subjectivity, serendipity, critical and holistic thinking.

What’s your motivation for writing on a blog?
– I don’t know, I just can’t stop myself.

How come?
– Not sure, I think I came to a place where writing was the best way to learn things; and learning things is an absolute must.

What’s your favorite quote, right now?
– You can see a lot by just looking (Yogi Berra)

Any new paradoxes?
– No, just an old one: With check lists you do testing, with test scripts you do checking.

Who is the best tester?
– Can’t tell, and if I could, I wouldn’t. Testing is part of a team effort to develop software; it’s not a competition.

Which one of Kaner’s 101 coverages do you prefer?
– A broad version of #89: Potential Usage Coverage.

What bug would you like to find?
– A very important one that happens after very normal actions, but in a long, credible scenario. It should be on a non-released system, so it can be fixed before causing real problems.

What’s the recipe for Good Software Testing?
– 1 part preparations, 2 parts knowledge, 3 parts wide-open, trained senses, 4 parts hard work, and a big pinch each of creativity and subjectivity.

How many testers actually spend more than 5% of their time testing non-functionals?
– I think many have quality characteristics in the back of their head all the time; so for me it is 100% of the testing time.

Can you say something positive about ISTQB Certification?
– In the Advanced Syllabus there is an appendix with a lot of good recommendations regarding pitfalls in software testing.

What’s the most difficult virtue for a tester?
– Humility.

Can you think of any strange bugs?
– A crash bug when cancelling a dialog, but only if the dialog had been manually moved.

Nothing better than that?
– A crash when a value range dealt with 9.8922, 9.975, 10?

What’s your favorite accelerator?
– Windows+Break. Everybody doesn’t seem to know that this is the fastest way to bring up the System details.

If software testing is a social science, which qualitative methodology can we learn most from?
– Parts of Grounded Theory is a perfect fit.

What’s your favorite string to use when testing?
– “This is a tribute to the longest string in the world.”

What is the danger of solving problems?
– It wasn’t the important thing, and now you have stopped looking.

Do you have a lesson for software testing?
– No, but the last one in the book Lessons Learned in Software Testing is excellent, the one on Test Strategy Heuristics.

Do you want to be Deming or Fleming?
– Ah, you’re thinking serendipity-wise. Fleming and the penicillin, of course.

Do you have an alternative to SMART goals?
– I want my goals to be Vague, Ongoing, Motivating, Important and Trustworthy.

Where would you like to have an extra pair of eyes?
– I just have to steal this answer: I would give them to a blind.

How long does it take to become a really good tester?
– Don’t know. I might be there, and if so it happened gradually over many years. But it takes about one year to master sourdough baking, so it could be something similar for testing, if you have the basic skills, knowledge and motivation that are needed (don’t ask me about that!)

What’s your biggest weakness?
– Could be that I am easy to misunderstand. I often try to be serious and humuorous, at the same time; which can be very difficult to evaluate.

What development process are you using at your current assignment?
– Same as everybody else, a mix of Waterfall and Agile.

What do you have against Agile?
– Nothing really. It’s just so over-used, and doesn’t seem to mean much more than it’s something good (just like “professional”)

Don’t you agree that testing is a service?
– Can’t say it’s not true (and it’s extra important for testers to be service-minded), but I would talk about long-term team ownership several times before using service vocabulary.

If you were a young, aspiring software tester with a lot of spare time, what would you do?
– I would find an open-source tool I like (maybe a test tool?), and test it thoroughly, learning everything I can about the test approaches while using them.

Are you a certified tester?
– No, but I have a diploma from a basic software testing course.

What’s in your Software Testing Dystopia?
– Extreme specialization, same work all the time, no system overview, and cumbersome software with just some value.

What questions would you not want to be asked?
– What’s the most important aspect of software testing?

Anything else you would like to add?
– I am always on the testers’ side!

Thank you very much! Have a good day, and a nice vacation.

Last week, me and Martin won the competition “The EuroSTAR Test Lab Apprentices”!
Read more at: http://www.eurostarconferences.com/delegates/the-test-lab-apprentice.aspx

See you in the Test Lab in Copenhagen!

Cheers,
Henrik & Martin

There are many good articles and reviews on how Session-based test management (SBTM) is used out in the field, see some of the references below. I’m going to try to analyse what I’ve experienced so far with Session-based test management. If you are new to these concepts please see reference [1] and [2] below to get started.

Learning & Education

When I try explain to others what is special about SBTM I emphasis learning and education. As a test trainer, test lead or an experienced tester it is a very powerful tool in teaching test design and communicating test ideas. The session report is one of the keys, but also the way you debrief. For instance, if you pair an experienced tester with a novice tester, you will be able to train both at the same time while at the same time getting a good result. You start out with a mission with a set of ideas that drive the group, you discuss what you intend to do and identify possible test ideas. As you start working your way towards the mission, you sometimes identify new things or change your original test scope. This collaboration teaches the novice how the expert works her way through the system, what heuristics that are applied and how new test ideas pop up. It also teaches the expert to better express and organise her thoughts. I’ve seen many times that the testing becomes a lot more structured when you need to express to someone, sitting next to you, what you are doing and why.

Speed

Since you are not bound to do just one test case in a specific amount of time, you will instead know that you have a certain amount of time to do testing on. A mission that you might follow if you think it adds value. If you find something else that is of higher value, you should investigate that instead. This gives the testers more freedom, but it also means that during a session (a time box) you are focused on testing that you think is valuable, no matter what. At the debriefing you need to explain why you didn’t follow the mission and what made you prioritize the other area. You will in a natural way try to focus on things that adds the most value, as you see it as a tester. This in combination with having prioritized missions will let you have a bigger chance in focusing on the right things. Comparing this way of working with how you sometimes do test cases, where you see problems in other areas as things that stop you from executing the “real” tests and where you report “no progress” when not being able to finish the test case. In those situations, you can sometimes be mentally blocked from doing what is expected of you. This might be an issue with the mindset of some testers and test leads, and not applicable to the whole test scripts vs. exploratory test discussion.

The indication that I see is that you will do a whole lot more tests using session-based testing than when you are focused on running specific test cases with predefined steps you should take. In some cases you might do duplicate tests that others also do, but a few extra pairs of eyes is most often good.

Accountability

Ad hoc and exploratory testing can sometimes be seen as unstructured and lacking accountability. Michael Bolton has discussed this in his blog in a good way here. One of the main ideas behind SBTM is that you instead of showing test cases that you have run, where you might or might not have followed the actual steps, is that you show in your session reports what was actually being tested and what was guiding you. The metrics produced is then based on what was covered and how.

Comparing SBTM with running a test case that is lacking the full repro steps, but where you give the freedom to the tester to test in the area. You use the full tool support so that you keep track on what revision, what environment and what context you think was interesting for a specific test. SBTM is lacking in the collection of metrics data based on that you are not able to state what areas each one of your tests in a session touched. Instead you enter what areas the whole session touched and might give an inaccurate truth. The error rate is still low since you have short sessions ranging from 30 minutes to a few hours, but the metrics on area coverage will be erroneous.

If it is important that you show what specific functions you have covered or what requirements you have covered, the actual Areas in SBTM might not be the too solution to this. Instead you should probably have tracability matrixes or something similar, as I see it. I’ve not explored this fully yet though.

Test areas

If you use SBTM, according to James and Jon Bach, you enter the areas that you are able to report time on in the coverage.ini file. It is good to spend some time in finding out the core areas and their hierarchical structure. The important thing is that you consider what you want to measure on and what is interesting in reporting, based on the metrics produced.

I think that it is good to setup the root areas as something along these lines:

feature | <area> | <sub area> | <sub area>

revision | <product> | <revision #>

environment | <hardware> | <hardware revision>

You will be able to add new areas later on, just keep track that you use the same kind of structure and that new areas are inspected every now and then. You can change the coverage.ini file without any severe issues other than having to change the dependencies afterwards.

Focus/Defocus Strategies

I’ve tried various setups in how to do the focus and defocus when running sessions. In one case I used a time slot for the whole group between a start time to a finish time where everyone where to round up and debrief. This strategy pushed the testers into delivering and being focused during this time. It also made some of the testers who had a harder time to start at a specific time to actually be part of these sessions. I used this strategy when in training and it worked well for that. If someone walked into the test lab to interupt it was easy to just tell them that we had training session and that they would be ready in about X minutes.

Another strategy was to ask each tester how many sessions he/she thought were doable during one week, then try to hold them to that commitment. This gives the freedom to the tester to run sessions at the hours that fits them. Still, it made it harder for controlled debriefs.

My recommendation is to try different strategies and see what fits your team best, perhaps even have a mix of them to fit each individual.

Satisfaction

Grumpyness is usually a common state of being, as a tester. I’ve noticed that the satisfaction level is different depending on what test technique you use (this is just a minor observation).

In one case we had a long list of test cases that were to be executed on a specific test environment. To be able to run these test cases you need to do certain setups and configurations that took quite some time. The test cases did not have anything to do with these preparations. The interesting thing was that most bugs, at that time, was in the area where we prepared to test. What happend was that at each stand-up meeting every tester complained about not getting anywhere and that they had no progress at all. The project managers thought we were doing a loozy job because we gained no progress. They thought we were slow.

So, instead of executing those test cases we ignored them all and ran test sessions in those areas where we explored what was causing these preparations to halt. A large number of bugs where reported and at the next stand-up meeting we instead said that we had made lots of progress in those areas where we had reported many Must bugs and that we were blocked from running those specific test cases. The satisfaction level was now high and we talked differently about things that stopped us, we found issues that stopped the release which was good progress.

Speaking for myself and some in my current team, I believe that you become more satisfied in as a tester when you use the session-based test management instead of running hardcore test scripts. You know that the important thing is that you test and focus on areas that that are important. So, hours spent testing or a number of sessions per week is something that you can talk about.

Test focused

The amount of time spent on testing vs. how much time you spend on planning is a lot different from the regular test scripts approach. This is the core of Exploratory testing and this is where SBTM shines. As a test lead you view your testers as intelligent enough to make their own decisions. The missions will guide them, but they will form their own decisions.

For instance, lets assume you have 10 weeks of testing. Do you spend a better half of that with planning what you should do and how to execute each test case? Or do you do just-in-time planning and setting up missions so that you always have a steering. Naturally I think the later is best. Overplanning is a curse for testing since we know that all plans change, requirements are just a whisper-game and testing is a social science. I’d rather spend time testing than discussing or planning what would be possible to test or what we cannot test.

Tools for SBTM

SBTM package by James and Jon Bach

James has a session.exe available from his web site. The documentation is good and you will be able to get started quickly there. Those who are fond of using simple text editors will like this approach. Included in the session.exe is various compilers written in perl. Paul Carvalho has done his own lessons learned here and updated the compiler scripts in Ruby.

I think the main idea behind this tool setup is that each user should own his/her processes and tools. So that you should make them fit your own company and way of working.

Test Explorer

This tool is recommended by James Bach. I test complex systems that only has a small part that has a GUI, this tool did not suite my way of working at all.

James Lyndsays tools

James L. has made an excelsheet that supports his way of working with SBTM. He has introduced a concept called Test Points that I find interesting. He has ideas on how to address regression testing and how to report progress. I recommend reading his paper [5] and trying out his tools.

Session Tester

This open source project [6] is run by Jonathan Kohl [7] and goes a somewhat different path than what Bach staked out from the start. Jonathan has been driving this project to steer in the direction that the tester community wants it.

I tried out the 0.21 release and noticed that lots of the ideas that I had gotten from using Bachs tools were implemented here, but with some changes or adaptions. I am not fond of using a text editor for all things, but want instead a powerful tool that is specialized to my needs as a session-based tester. There was no tool that fulfilled this requirement as I saw it. Therefore I offered my services to Jonathan Kohl to assist in the project.

We are now driving for a 0.3 release. Go to the website mentioned above to see what is included. If you want to be part of this notify me or Jonathan.

Courses

Rapid Software Testing

Some believed that this course would teach how to use this technique, but it does not. Instead it empowers you to become a great tester, which as I see it is even better.

SBTM Courses

I think there are individual consultants who offer courses in this, but it seems quite rare.

Documenting in a session

The skill of journal keeping is practised efficiently in SBTM. I see this skill as essential for a good tester and SBTM brings it out in a natural way.

I think each tester should have some personal freedom in how they document, but it is good to agree on some standard way of expressing yourselves. Bring up the discussion after a while that you have used SBTM and see if you can agree on something.

Organising missions and charters

There is a way of using the TODO lists in order to create template sessions, but I did not use this fully. Instead we had missions in a separate excelsheet together with other tasks. I figure missions are easy to include in regular Sprint logs or similar planning.

We let testers add new missions that they thought were necessary. Organising this in a good way will make it a lot easier to measure progress and to see trends. I’ve not explored fully where you could go with this, but I think you can use other tools such as Trac to do your planning in.

Summary

I’ve tried to cover a part of the areas where I thought previous reviewers had missed or not focused on. In order to make the full picture for yourself, you as well need to dig deeper.

I recommend exploring this way of testing. It will change the way you perceive testing and the way you plan. I think it is a lot more efficient that traditional test approach, but there are several holes to fill to make it even better.

If you want to own your tool by building on what James Bach created, if you want to buy a tool such as Test Explorer or if you want to participate in creating a tool for SBTM, such as Session Tester, is your choice.

References

[1] http://www.satisfice.com/sbtm/

[2] http://www.satisfice.com/articles/sbtm.pdf

[3] http://www.staqs.com/sbtm/

[4] http://www.developsense.com/blog/2010/01/exploratory-testing-is-accountable/

[5] http://www.workroom-productions.com/papers/AiSBTv1.2.pdf

[6] http://sessiontester.openqa.org/

[7] http://www.kohl.ca/blog/

When I meet testers I often ask what test design techniques they use on a regular basis. I am really curious which of the techniques that I know of that they use and I am always looking for new material. I have taught test design classes to maybe a thousand people over the years and I keep asking for THEIR examples of applying techniques after the class is over. So far I cannot remember having received any specifics at all. So I am starting to wonder..

There is so much talk about test ideas, heuristics and lists and although these are wonderful new ideas, I am afraid that we are forgetting a core skill that can be very rewarding. One of my favorite techniques is state modeling. The ISTQB foundation class shows you a table or a graph and you need to identify what series of events are possible. This is much like the puzzles for children where they have to help the dog find the bone in the maze. Well, no surprise, this is just their way of simplifying something beautiful to total pointlessness. So let me tell you how it can be done better.

A project I am currently is an application for administration of scientific reports. A researcher sends their report to an editorial board which then assigns an editor and a numbers of referees. Then the evaluation process starts and reports come in. It all ends by a report getting refused or accepted. Now this is a school book case for doing state graph modeling. I would not find it strange of the developer had already created a number of such models for their own benefit, but as usual, there were none. Now I started to get excited. A really cool and useful model was to be created and I would be the father of it! Days like this are what make me happy!

Step 1) I read all documents I could find. I questioned the business users and got a pretty good picture of how they worked. Then I went to the developers and started asking questions about what statuses the manuscripts had at different points in the process.

Step 2) I created a state-chart and went back to the developers. The response I got was that they wished they had done that themselves before and they instantly saw a numbers of changes that would make their programming easier.

Image: The state graph as it is right now

Step 3) Creating a detailed state table. One thing I really like with this technique is that is very close to actual design of the code. The manuscript is in one state and then something happens that put it in another state and also a number of parameters should be updated. The manuscript is new, state 0, the admin selects editor and referee and presses button send mail. Now status should be changed to 1, e-mails sent to all selected referees and editor and a notification sent to the author the evaluation has started. When in state 1 and additional referees are selected, send mail notifies only the new people etc. This turned out to be one of the most important parts of my test design and the group solved a lot of issues together. So before I had executed any test cases at all I have already helped making design changes and found some serious omissions. I keep on asking questions like “What if this happens when I am in that state”.

Image: small part of the state table.

Step 4) Test execution. I listed a number of test ideas and ran each idea in a session inspired by SBTM. Logging was done in Word since this supports adding screenshots. I did start with the SBTM notepad format that can be analyzed but soon realized that there were so few logs and only one tester, that the benefit of logging text and pictures in the same place was better for me. So my form of exploratory testing is to have test ideas and test design models. The only scripted test cases we created were for the demo where the goal was to show that it worked. Needless to say we got some questions during the demo and deviated from the script…yeah it would have been good to test it some more before the demo.

Step 5) Continuing the project we realized that some additional functionality needed to be added to make it work smoothly. More specifically we added a new status 6 – published. This helped us a lot in selecting what manuscripts to show on different screens. The state model made it very easy to see the effects of our changes.

To summarize this: state models help finding bugs before executing test cases, they help clarifying the design, they guide me during test execution with overview and detail on a couple of pages and they are a great support when evaluating changes to the application. C’mon, how can you not be Excited by this Hot Super Model!

If you just do what’s on the list of things to do, I think you can accomplish decent things, but nothing great.
I don’t dare saying this is a general truth for everybody creating something new, so let’s focus on software development.
There are many management models, and many of them boils down to something like “do the things on this prioritized list”, and the list might be items in a detailed project plan, a Backlog, test cases or sticky notes.
As an attack, and a defense, of these systems I’m saying:

to accomplish something great you got to do a lot more than what is on the list

The reason for this is that you cannot in advance know about all important things and possibilities that are associated with what your team is trying to accomplish.
Great software has even more than the right capabilities, impeccable reliability, super-performance and perfect usability.
There are no Gods in software development that can specify the details in advance.
And if there was an almighty capable of doing this, it would take too much time to read and understand all items on the list.

If the input for the software is taken from the actual demands of the customers, you can give them what they want, but not necessarily what they need, and probably not something that will be very valuable for many customers. And the item on the list will probably focus on functionality, without all implicit details around quality characteristics that can sum up to a great product.

So the solution is to take height for this, for example by
* not performing (detailed) time estimation and planning
* allocate time for “creative slack”
* pad all estimates with an appropriate percentage
* add extra time for polishing (that must be spread out from start to finish)
* rotate free, unscheduled roles
* encourage everybody to pursue interesting things that emerge
* perform one unscheduled “good” thing for every other item on the list
* schedule vague items that can mean almost anything
* but most importantly: have skilled people doing their best to create valuable software

Usability is an important part of any product (if it’s too difficult to use, it  doesn’t matter how great the functionality is) and thereby an important characteristic for the testing team.

But when reading about usability testing, it often involves an outside person trying to use a feature for the first time. Now this is a good thing, but there is a lot more to Usability than Learnability.
There also exist systematic inspection methods, e.g. Heuristic Evaluation by Jacob Nielsen, that should be performed by super-experts on usability.
He recommends alternating between user testing and evaluation, but I think a manual software tester can do both at the same time, if she has (some) knowledge about the domain, (some) knowledge about the product, and (some) knowledge about usability problems.

Here is a very cheap way to do usability testing; ask the following questions to someone with many hours of using the product (probably a real user or a manual system tester)

General usability: What’s your most important usability issues?
Affordance: Would you say that the products invites to discovering possibilities, or are there features you learned too late?
Intuitiveness: Is it easy to understand and explain what the product can do?
Minimalism:Have you seen redundant information anywhere? Any important information missing?
Learnability: Are there any areas you feel are difficult to understand and learn?
Memorability: Do you forget how to perform some actions?
Discoverability: Can you discover all available operations by (systematic) exploration of the user interface?
Operability: Are any common operations cumbersome?
Interactivity: Are there situations where you aren’t sure of which kinds of interactivity is supported?
Errors: Have you encountered strange error messages, or situations that were difficult to recover from?
Consistency: Is the user interface annoyingly different anywhere, or hard to understand?
Tailorability: Are there any (default) settings you would like to change?
Accessibility: Did you notice any problems while using High DPI and mostly the keyboard? Color-blindness?
Localization: Any experiences of running translated product or data?
Documentation: Are there any information you would like to see in the Help?

You might already know about many of the issues in the answers, but now you know more about their importance.
You won’t get any quantitative measurements, but hopefully a lot of valuable information.

Even better might be to have questions like these in the back of your head when executing manual tests, you’ll probably get valuable information for free. There is no need to memorize the categories, the important ones will emerge.