Code coverage is a commonly used metric in software engineering, especially in white-box unit testing. It can provide valuable information about how much of the code is tested, especially what is not tested. 100% coverage means that the tests cover 100% of the code. This sounds very good in theory, however, it might not always be the best objective for the project.

What Is Code Coverage?

Code coverage is a percentage indicator of how much of a program’s source code is run during a specific test suite. In comparison to a program with lower coverage, one with high code coverage has more of its source code run during testing, which can be perceived to have lower chances of undetected bugs.

Teams can measure code coverage based on different coverage criteria:

• Function coverage: the percentage of functions (or methods) covered by tests. If 700 out of 1000 functions are covered, then you have 70% coverage.
• Statement coverage: the percentage of executable statements that the tests cover.
• Branch coverage: how many of the branches are covered in the tests (branches refer to each path from a decision point, such as if-else statements).
• Condition coverage: the percentage of Boolean expressions that have been tested for both the true and false values.
• Line coverage: the percentage of lines tested.

The coverage percentage changes with the addition or deletion of tests, but also with the overall number of code lines.

The Benefits of Good Code Coverage

Code coverage is a good starting point for estimating the quality of a software project. A higher coverage can mean a reduced risk. If large parts of the code are untested, the probability of bugs and defects in those parts is higher.

With good coverage, there can be fewer regression issues. As the codebase grows, so do the chances of existing parts being impacted. However, if something is broken with a code change, existing tests can catch it, and the team can fix it quickly.

The codebase is constantly updated with new features and fixes, and so should the test code. You can use code coverage to confirm that the testing objectives defined at the beginning of the project are kept throughout the SDLC.

It can also help when refactoring. Knowing that a part of the code is covered by tests can increase the team’s confidence to refactor that part, knowing that any breaking changes will be discovered by the tests.

Lastly, if the tests are high-quality, then having good coverage can also increase confidence in the project’s quality.

Why Achieving 100% Should Not Be the Goal

With that being said, it sounds like the higher the percentage, the better, right? Not necessarily. There are a few downsides when working towards 100% coverage.

• More tests don’t always mean better tests. You can have all the lines of the code covered by tests, but if those tests are not high-quality, then they will not uncover bugs in the code. They will also give a false sense of confidence.
• Reaching from 80% to 100% coverage means a lot of work. It usually takes complex, difficult-to-maintain tests to test all edge cases and unusual flows across the code. The closer you are to 100%, the less valuable the return of work will be. That time is probably better spent doing other types of work.
• Quality of the code may suffer. When trying to test every single line of code, tests become bloated with redundancy, tightly coupled to implementation details, and challenging to read. Adding sleeps and waits only to cover a complicated portion of code is an anti-pattern. The pursuit of the 100% coverage goal comes at the expense of test quality and maintainability.

What To Do Instead

A generally accepted rule is that striving for 80% coverage is a good objective. However, obsessing over the numbers is not useful unless the tests are high-quality and bring true value.

• Quality intelligence helps you tailor your test strategy: combining quality intelligence with test automation allows your team to focus on running only the tests that matter, based on exactly what changed in your codebase. The integration between Tricentis Testim & SeaLights (learn more here) allows you to create a targeted test list focused on real coverage gaps, reducing blind spots by ensuring the right tests run at the right time.
• Choose a realistic goal: As already stated, 80% coverage can give the confidence boost needed, especially if this percentage covers the most critical parts of the code, and it’s a good goal for most projects.
• Don’t compromise quality for quantity: The logic might be that if 80% is good enough, then 90% is great, and 95% is even better. But this is not the case if the focus is only on writing more tests without focusing on quality. It’s better to have fewer well-written tests than to spend the same amount of time on more tests, but with poor quality.
• Use tools and reports to measure your efforts: Evaluate coverage data periodically to find untested code and adjust testing priorities accordingly. There are a lot of tools that can be used to measure the percentage of code covered by tests.
• Don’t rely on code coverage as the single quality metric: Coverage should not be the only metric used to measure quality. Other useful indicators of quality can be defect density (i.e., the number of defects per lines of code), average time to find and fix bugs, and code quality (usually measured by static analysis tools).

Takeaways

Code coverage highlights the percentage of code executed during tests and can be a useful metric in software engineering for unit testing. However, striving for 100% coverage can be misleading, as it may lead to lower-quality tests and a false sense of security. Instead, aiming for around 80% coverage, particularly on critical code, is often more effective. Balancing coverage with meaningful quality assessment is essential; fewer high-quality tests are more valuable than many low-quality ones. Regularly analyze coverage data to identify gaps and consider other quality indicators, such as defect density and bug-fixing time, to ensure a comprehensive evaluation of code quality.

The post Code Coverage: Why 100% Isn’t the Holy Grail appeared first on AI-driven E2E automation with code-like flexibility for your most resilient tests.

AI-driven E2E automation with code-like flexibility for your most resilient tests