Coverage report changes depending on the current directory

[AlexSkrypnyk]

Summary

kcov is installed as global binary, but the produced report changes the coverage based on the current directory.

kcov version: v42.

Please note that while I am using BATS below, this is only for demonstration purposes of how one script (BATS) calls another script (SUT script).

This issues in NOT BATS-related (I have tested this throughly).

Details

I'm running BATS tests with kcov.

There are 2 directories:
SCRIPTS_DIR=/workspace/scripts - a directory with scripts that are SUT
TEST_DIR=/workspace/tests - a directory with BATS tests.

I want to have coverage for .sh and .bash files in both SUT and BATS test helpers (not actual .bats test files).

This is as working command that produces correct coverage:

kcov --include-pattern=.sh,.bash --bash-parse-files-in-dir="${SCRIPTS_DIR}","${TEST_DIR}" "/tmp/coverage" bats "${TEST_DIR}/test1.bats" 

And the contents of test1.bats is:

#!/usr/bin/env bats

# Helpers that I want to have coverage for as well.
load "_helpers.bash"

@test "Mytest {
  pushd "/tmp/random_test_dir" >/dev/null || exit 1

  # 'run' is a BATS internal command that creates a sub-shell.
  run ./scripts/myscript.sh
  assert_success

  popd >/dev/null || exit 1
}

Now, the problem is the following:

When I run kcov ... command above from /workspace dir - the coverage correctly includes tests from both SCRIPTS_DIR and TEST_DIR.

But if I run the kcov ... command above in another dir, say, /otherworkspace, the coverage for the SCRIPTS_DIR is not included.

Please note that all paths are absolute.

I have tried different variations with and without --include-path, --bash-parse-files-in-dir, relative and absolute paths to coverage report, dirs etc etc. I also tried additional options with --strip-path, --path-strip-level=

The only time it worked is when the kcov... command was run from the same dir where both SCRIPTS_DIR and TEST_DIR files were residing (/workspace).

I have tested this both using kcov Docker image and standalone binary and I have observed the same outcome.

What I am looking from this issue

I have a feeling that I'm missing something obvious.

From the list of options, I do not see anything like --working-dir. I was not able to find anything about working directory in the documentation either.

It would be great to either have this behaviour documented or provide actual --working-dir options that could be used to resolve paths.

In addition, reading the current documentation does not explain if the paths (any paths) should be relative or absolute, what happens if some of the relative (like --include-path) and some absolute (like out-dir arg) and how these are resolved. So, documenting this would help a lot to understand how to use kcov.

[SimonKagstrom]

Thanks for the detailed report!

I guess ${SCRIPTS_DIR} etc are absolute paths, as in your example? If you only have one of them, e.g., ${SCRIPTS_DIR}, set in --bash-parse-files-in-dir, does that make any difference if you run from /otherworkspace?

[SimonKagstrom]

I've tested, but I'm not able to reproduce this. I have this directory structure:

/tmp/kalle/
/tmp/kalle/b.sh
/tmp/kalle/a.sh
/tmp/manne/
/tmp/manne/d.sh
/tmp/manne/c.sh

and I get coverage for all files if I do

 kcov --bash-parse-files-in-dir=/tmp/kalle,../manne /tmp/kcov/ ./a.sh

(also tested with absolute paths).