There are many ideas that contradict one another in software. In one case, people may say that the DRY Principal – “Don’t Repeat Yourself” – should be adhered to without compromise. In other threads, people draw a line on what context that principal should be applied, and push very strongly for duplication in tests, as well as not sharing code between services or even bounded contexts of a modular application. Testing culture, similarly, has a lot of people believing that everything should be automated and that manual testing is dead. Here we’ll discuss the opposing idea: that manual testing still has a place in code culture, and discuss how and why it does, and where it fits into automated testing practices.
Types of Testing
Let’s highlight a few different layers and types of testing – while there are many typs of tests, we’ll focus only on a few:
- Unit Testing: Automated testing of small bits of code. Can also include automated tests within a module or service that test integration between modules and components in the code (eg a slightly higher level test, but still isolated from other services.) True unit testing does not have an external database or any reliance on external services. As such, if built optimally, they should be extremely fast to run and have the benefit of being up-to-date documentation of the code by coupling to the behaviour.
- Integration Testing: Combines modules or services together to test subsets of a system. Doesn’t need to run against a full environment. Can still be inexpensive to run as it doesn’t require a full environment and can be run from a module produced to combine modules/services.
- System Testing: Automated testing that spans multiple modules or services, checking that they behave as expected when hooked up. Generally integration testing is run from “outside” of any particular service against a production-like environment. The tests makes requests against the public or internal APIs of the application to ensure the system as a whole, or subsets of components, behaves as expected. They are slow to run and expensive to maintain relative to unit testing.
- Manual Testing: Similar to Integration Testing, manual testing will test against the entirety of the system but differs in that it is executed by people using the UI (or APIs as needed) to validate the system and its behaviour.
Manual Versus Automated Testing
Manual testing is often executed by dedicated QA teams in older organizations. Many modern technology organizations do not have explicit QA teams but may have engineering teams that aid in automated testing (The Software Engineer In Test role, for example, which joins teams to improve tooling and processes around automated testing.) The trend is to replace manual testing approaches with automated approaches and teams to support those goals and efforts. The major heuristic is that manual execution is wasteful as it doesn’t provide value going forward, and so building automation of all tests is logically better as an artifact is built in testing and maintains that the behaviour works forevermore.
There is some danger with automated testing. Of course, we know that software changes, so there is an overhead there in maintenance of the tests, and if they aren’t covering anything relevant, and aren’t easy to maintain and break frequently, then the cost of a test may outweigh the value it provides in its life, especially if the test isn’t covering important behaviour. It’s worse if the tests take a very long time to run.
I would suggest, then, that there is manual testing required to discover issues with a change, and that unit testing should be preferred over system tests, while system tests should be reserved for ensuring behaviour of a system is correct, and tests added to the system test suite be chosen with clear intention.
Manual testing can be entirely replaced by automated testing. But if it’s done indiscriminately, the cost of running and maintaining the automated testing may become too high. Likewise, if all test scripts ever created need to executed by humans, the cost will be far too high as a system grows in complexity and scope. I would argue that manual testing should be considered a piece of the development cycle apart from system testing. A human is better at “experimenting” than a script and can find blindspots that automated system testing can. As bugs are discovered by a human, and then fixed by a human, their fixes can generally include the faster, cheaper, more maintainable unit tests, rather than building system tests for some particular condition.
Summary
System test isn’t intended to cover a module or service’s particular behaviour, but only the behaviour of the components that touch. System tests and integration tests may only be concerned with testing behaviours that a unit test can not cover, and unit tests should test the specifics of a module/service that would be extraneous to cover in system tests. Manual testing should be done as a part of the development lifecycle, and issues identified/fixed be turned into unit tests wherever possible, while system testing be reserved for testing that cannot be covered with unit tests, especially confirming that components of the system are talking to one other correctly.
multiplexed musings 