In my past, I have seen different sorts of test documentation.
From no test documentation at all, to very detailed test documentation with test cases with single steps and the expected behavior. On this page I will explain the extent of test documentation I consider useful. Of course, this always depends on the context in which I move. And you always have to consider, if for example legal requirements for test documentation have to be met. Or in a customer project, the customer may require a specific test documentation.
What have I seen?
Detailed test documentation
A test plan contains the high level overview about the tests and the strategy/approach to run the tests for the component.
One or more test cases exists for every feature of the component. For the good cases, as well as for error cases. A test case usually looks like this: Test case title, needed preconditions such accounts or test data and the post condition. And for each test step: the action and the expected behavior. The same steps often appear in multiple test cases when testing different cases for one functionality. In a good set-up the test cases are management with a test case management system.
In my opinion this requires a lot of effort to describe the individual test cases including the steps. And to maintain the maintenance effort must be considered as well.
I discovered that the documented tests become obsolete after a certain time, or sometimes even after a short time. One reason is that they are not executed regularly and one other reason is to keep them up to date.
From my point of view this kind of documentation has only one advantage, if persons who are not familiar with the domain have to execute the tests frequently and manually.
No test documentation
Of course, the initial effort to have no documentation is zero. But in my opinion, the effort arises later. If you need to test the component after a while, you first need to re-learn and understand what is important for the component and what is not. This could decrease the quality of the test and thus result in errors.
My preferred scope of test documentation
I think it is important to have a test documentation. A good documentation makes it easier for new colleagues to familiarize themselves with the components. And it helps when working on the component after a while and to share knowledge in the company. From my point of view, the following structure has proven itself.
Test scenarios
Overview of the different test scenarios. I suggest to omit a detailed description of the individual test cases. What I prefer is to invest the effort in the automation of the individual steps so that they can be carried out continuously in a pipeline.
Risks
What risks are to be minimized or completely excluded by the test? And you should also look at the point in this way: what would be the worst thing that could happen? For example login, a login without a correct password is possible. This must, of course, not happen under any circumstances.
Blast Radius
If the component is unavailable or causing errors, which other components are affected by it? This blast radius should of course be as low as possible.
Required components
Overview about all components needed to execute end-to-end tests of the component.
Specialties
If any, what are the components specialties. Are there any special features that need to be taken into account.
Quality characteristics
Overview about quality characteristics and how important each is for the component. For example the ISO 25010 lists various quality features.
The overview should also contain the test types and the tools (if used) used to verify the different quality characteristics.
Test Accounts / Test Data
This section should give an overview about:
- The needed test accounts.
- Needed test data or how to prepare the test data.
Test documentation final thoughts
Component documentation could embed the test documentation. The component documentation should contain a brief description of the component, use cases, architecture description and so on. This way you can find all the important information about a component on a single page or on a page with sub-pages.
The effort for this kind of test documentation is manageable and the maintenance of the documentation is low. This gives the team a good overview of the component, including the test approach, and of the important test scenarios.
I would put the effort of creating and maintaining detailed test cases into test automation. That way, tests can be triggered automatically when a new version of the software has been deployed. And they usually provide faster feedback to the developer.