In quality engineering, we want to know how “good” or comprehensive our testing is. We want to know, if our test coverage in any area is sufficient. This can help to decide if we want to move on to the next area or invest more time and effort before doing so.

Challenges with testing of big software projects

When you write your own small application or service, it is rather easy to see if test coverage is good enough. Often, we can test every important part manually and be confident enough. If projects get more and more complex, you cannot guess anymore if your test is “good” or if you need to do more. If you want to test a single file of the kernel tree, you will need to take all the other parts of the kernel into account. You simply will have a hard time to see this.

GCOV and the Linux kernel

There are a lot of commercial tools out there, as well as a lot of frameworks for many languages etc. The Linux kernel supports GCOV, and this is what comes to help us for analyzing code coverage.

Please note, even if we are able to achieve a good test coverage, it won’t be useful for argumentation with regards to safety certification, as our test cases won’t be derived from requirements (so called black-box tests). While we would prefer this, this is not feasible, so we want to use code coverage information to be confident enough about our code.

Installing GCOV, the Linux kernel, LTP and rapido

We can install gcov[1] and gcovr[2], a report generation tool for gcov plus some dependencies for rapido. The following is by example for openSUSE, but should not be much different on other distributions.

zypper in gcov gcovr qemu

We also need a linux kernel installed`. You’ll need a couple of build dependencies again, but we’ll skip this here. We need to work on the same location on the host as we do on the guest later, so we need to place the kernel in /host for simplicity.

sudo mkdir /host
sudo chown <user> /host
cd /host<F2>
git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
make x86_64_defconfig
make kvm_guest.config

scripts/config --enable DEBUG_FS
scripts/config --enable GCOV_KERNEL
scripts/config --enable GCOV_PROFILE_ALL

make -j8
INSTALL_MOD_PATH=./mods make modules_install

Next, we need LTP[4], asusming you have a toolchain and autotools installed:

git clone https://github.com/linux-test-project/ltp.git
cd ltp
make autotools
./configure
make
sudo make install

Next we can proceed and install rapido[5]:

git clone https://github.com/rapido-linux/rapido.git
cd rapido
cp rapido.conf.example rapido.conf

Find the lines below and uncomment/set them accordingly in rapido.conf:

KERNEL_SRC="/host"
VIRTFS_SHARE_PATH="/host"
LTP_DIR="/opt/ltp"

Plus, we need to change the following in rapido’s cut/ltp.sh

 cat lsmod ip ping tc \
to
 cat lsmod ip ping tc gcov gzip \

This could be added upstream, but for now you have to add the binaries to the list manually.

Now we can boot the system and verify it is working:

./rapido cut ltp

This will start your kernel and allow you to run LTP tests. Stop the VM with

## Prepare Coverage scripts
Check the kernel documentation on gcov[6], then create a file "gather_on_test.sh"
in /host with the following content:
``` bash
#!/bin/bash -e

DEST=$1
GCDA=/sys/kernel/debug/gcov

if [ -z "$DEST" ] ; then
  echo "Usage: $0 <output.tar.gz>" >&2
  exit 1
fi

TEMPDIR=$(mktemp -d)
echo Collecting data..
find $GCDA -type d -exec mkdir -p $TEMPDIR/\{\} \;
find $GCDA -name '*.gcda' -exec sh -c 'cat < $0 > '$TEMPDIR'/$0' {} \;
find $GCDA -name '*.gcno' -exec sh -c 'cp -d $0 '$TEMPDIR'/$0' {} \;
tar czf $DEST -C $TEMPDIR sys
rm -rf $TEMPDIR

echo "$DEST successfully created, copy to build system and unpack with:"
echo "  tar xfz $DEST"

Running the tests and collect coverage data

Run rapido again, exactly as you did to verify it is working. Then, when the VM is finished booting, run LTP with the syscalls group:

cd /opt/ltp
./runltp syscalls

And then grab a cup of tea and put on a record. This will take a while. Once it is finished, you can collect coverage data:

cd /host
./gather_on_test.sh coverage.tar.gz
shutdown

Generate Coverage report

Now we are able to generate a coverage report:

cd /host
tar xzf coverage.tar.gz
gcovr -j 8 -s -o cov.txt -r .

This will take quite a while again. After it is complete, you can look at the report in the file cov.txt.

Conclusion

As you see, it is a rather lengthy process to get coverage data for a kernel and a set of tests. And we also receive rather low coverage numbers, and those only for a minimal configuration of the kernel. Anyway, having coverage information may be helpful, so I’ll continue to look into this topic. In the next post, I will cover how to read the report plus how to obtain coverage information for a small part of interest. Anyways it is always fun to explore available tools and see what they can do for you.