Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reporting skipped test cases #23766

Closed
wants to merge 10 commits into from
Closed

Reporting skipped test cases #23766

wants to merge 10 commits into from

Conversation

linzhp
Copy link
Contributor

@linzhp linzhp commented Sep 25, 2024 •
edited
Loading

<skipped> is a valid and common test case status supported by JUnit XML. Reporting skipped test cases as "passed" hides potential broken test cases. This PR reports them when --test_summary=detailed.

Before the fix:

INFO: Found 1 target and 1 test target...
INFO: Elapsed time: 103.095s, Critical Path: 43.23s
INFO: 476 processes: 36 internal, 440 darwin-sandbox.
INFO: Build completed successfully, 476 total actions
//foo/collect:go_default_test    PASSED in 0.6s
    PASSED  collect.TestCollect (0.0s)
    PASSED  collect.TestCollectBuildFailure (0.0s)
    PASSED  collect.TestFindComponentTestPaths (0.0s)
    PASSED  collect.TestFindComponentTestPaths/component_tests_dir_does_not_exist_in_directory_tree (0.0s)
    PASSED  collect.TestFindComponentTestPaths/component_tests_dir_does_not_exist_in_provided_directory (0.0s)
    PASSED  collect.TestFindComponentTestPaths/component_tests_dir_exists_in_directory_tree (0.0s)
    PASSED  collect.TestFindComponentTestPaths/component_tests_dir_exists_in_provided_directory (0.0s)
    PASSED  collect.TestFindComponentTestPaths/component_tests_dir_should_be_the_same (0.0s)
    PASSED  collect.TestFindComponentTestPaths/directory_above_service_root (0.0s)
    PASSED  collect.TestFindComponentTestPaths/in_component_tests_directory (0.0s)
    PASSED  collect.TestFindComponentTestPaths/multiple_component_tests_dirs (0.0s)
    PASSED  collect.TestFindPythonTestPath (0.0s)
    PASSED  collect.TestFindPythonTestPath/not_a_python_package (0.0s)
    PASSED  collect.TestFindPythonTestPath/python_tests_path_does_not_exist (0.0s)
    PASSED  collect.TestFindPythonTestPath/python_tests_path_exists (0.0s)
    PASSED  collect.TestRunComponentTests (0.0s)
    PASSED  collect.TestSetTest (0.0s)
    PASSED  collect.TestSetTest/component (0.0s)
    PASSED  collect.TestSetTest/fail (0.0s)
    PASSED  collect.TestSetTest/unit (0.0s)
    PASSED  collect.TestString (0.0s)
    PASSED  collect.TestString/should_be_component (0.0s)
    PASSED  collect.TestString/should_be_unit (0.0s)
Test cases: finished with 23 passing and 0 failing out of 23 test cases

Executed 1 out of 1 test: 1 test passes.

After the fix:

INFO: Found 1 target and 1 test target...
INFO: Elapsed time: 147.787s, Critical Path: 42.03s
INFO: 476 processes: 36 internal, 440 darwin-sandbox.
INFO: Build completed successfully, 476 total actions
//foo/collect:go_default_test    PASSED in 0.7s
    PASSED  collect.TestCollectBuildFailure (0.0s)
    PASSED  collect.TestFindComponentTestPaths (0.0s)
    PASSED  collect.TestFindComponentTestPaths/component_tests_dir_does_not_exist_in_directory_tree (0.0s)
    PASSED  collect.TestFindComponentTestPaths/component_tests_dir_does_not_exist_in_provided_directory (0.0s)
    PASSED  collect.TestFindComponentTestPaths/component_tests_dir_exists_in_directory_tree (0.0s)
    PASSED  collect.TestFindComponentTestPaths/component_tests_dir_exists_in_provided_directory (0.0s)
    PASSED  collect.TestFindComponentTestPaths/component_tests_dir_should_be_the_same (0.0s)
    PASSED  collect.TestFindComponentTestPaths/directory_above_service_root (0.0s)
    PASSED  collect.TestFindComponentTestPaths/in_component_tests_directory (0.0s)
    PASSED  collect.TestFindComponentTestPaths/multiple_component_tests_dirs (0.0s)
    PASSED  collect.TestFindPythonTestPath (0.0s)
    PASSED  collect.TestFindPythonTestPath/not_a_python_package (0.0s)
    PASSED  collect.TestFindPythonTestPath/python_tests_path_does_not_exist (0.0s)
    PASSED  collect.TestFindPythonTestPath/python_tests_path_exists (0.0s)
    PASSED  collect.TestRunComponentTests (0.0s)
    PASSED  collect.TestSetTest (0.0s)
    PASSED  collect.TestSetTest/component (0.0s)
    PASSED  collect.TestSetTest/fail (0.0s)
    PASSED  collect.TestSetTest/unit (0.0s)
    PASSED  collect.TestString (0.0s)
    PASSED  collect.TestString/should_be_component (0.0s)
    PASSED  collect.TestString/should_be_unit (0.0s)
    SKIPPED collect.TestCollect (0.0s)
Test cases: finished with 22 passing, 1 skipped and 0 failing out of 23 test cases

Executed 1 out of 1 test: 1 test passes.

@github-actions github-actions bot added the awaiting-review PR is awaiting review from an assigned reviewer label Sep 25, 2024
@iancha1992 iancha1992 added the team-Rules-Java Issues for Java rules label Sep 26, 2024
@fmeum fmeum added team-Core Skyframe, bazel query, BEP, options parsing, bazelrc and removed team-Rules-Java Issues for Java rules labels Sep 30, 2024
fmeum
fmeum reviewed Oct 9, 2024
View reviewed changes
src/main/java/com/google/devtools/build/lib/exec/TestXmlOutputParser.java Outdated Show resolved Hide resolved
src/test/py/bazel/BUILD Outdated Show resolved Hide resolved
src/test/shell/bazel/BUILD Outdated Show resolved Hide resolved
fmeum
fmeum approved these changes Oct 20, 2024
View reviewed changes
Copy link
Collaborator

@fmeum fmeum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@meisterT Could you review this?

@meisterT meisterT added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-review PR is awaiting review from an assigned reviewer labels Oct 21, 2024
@github-actions github-actions bot removed the awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally label Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@meisterT meisterT meisterT approved these changes

@fmeum fmeum fmeum approved these changes

Assignees
No one assigned
Labels
team-Core Skyframe, bazel query, BEP, options parsing, bazelrc
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@linzhp @meisterT @fmeum @iancha1992