Code Coverage – Part I: An Introduction

This post is part of a four-part series on code coverage:

So, you’ve written some automation tests. Congratulations!

Maybe you’ve been writing your test cases treating the system under test as a black box, or maybe you know a little about how it works.

Ever wondered what your test code actually looks like from the developer code’s perspective?

Ever wondered if there’s a way to measure how much your code actually tests the system?

What is code coverage and why should I care?

Testing is all about driving confidence to make changes. If your tests cover every functionality required of the system, you can be rest assured that it works the way it is supposed to no matter how extensive a refactoring has been performed. If your tests only cover a small fraction of code, you probably shouldn’t rely on it fully for regression testing. If the developer just wants to make a small change, and the tests cover the lines modified, there’s a good chance the rest of the code will still work as intended after the change if the tests pass.

Testing is all about driving confidence to make changes. Code coverage gives us a way to measure this “confidence”.

The simplest measure of code coverage is Statement Coverage. It’s the proportion of lines of executable source code that is run during testing. That is, it should ignore comments, blank lines and other “non-code”, but include anything executable like functions, conditionals and the like.

Statement Coverage is the proportion of lines of executable source code that is run during testing.

Before we move on, you might be wondering: Is Statement Coverage actually helpful for assessing our tests? How do I use it?

  • Can I use it on any program? (Is it language dependent?)
  • Does it always yield the same result given the same tests? (Is it consistent/reliable?)
  • Does x% coverage means I’ve tested enough? (Is it sufficient as a metric?)
  • How much statement coverage is enough? Do I need to achieve 100% coverage?
  • How should I use code coverage in my SDLC?
  • Are there any other coverage metrics I should know about?

This series of blog posts should give you some context to answering each of these questions and give you some ideas as to how to assess and strengthen your testing framework.

First, let’s go through a concrete example of how to get code coverage in Golang. See you in the next post!

Published by quiche