Changing way "failing" tests are implemented, run: test for working condition, except non-pass

[cowtowncoder]

Currently we have a set of "failing" tests for many/most repos: tests under src/test/java/com/fasterxml/.../failing/. These tests:

1. Are written like any other tests, expecting to pass,but
2. Due to issue not (yet) fixed, fail
3. Are configured in Maven pom.xml to be skipped for junit runs

Design is this way so that we do have reproduction of a problem (so when fix applied, tests start passing), but will not block test suite (and especially CI system, Github actions).

But unfortunately this means that the tests are not automatically run at all and need to be manually run every now and then.
And when not doing that, tests may have unexpectedly start to pass, without anyone noticing it.
In fact this just happened recently with jackson-databind "master" branch (3.0.0-SNAPSHOT) -- due to change in ObjectMapper defaults, some tests started passing. In this particular case it meant test setup needed to be changed, but it could also happen when an actual problem gets fixed.

Ideally, tho, we would have a test setup where these failing cases were run, as-is, but with inverted logic: build would fail if one or more of these expected-to-fail tests would pass. This would let us know immediately when formerly failing test started to pass.

But how to do that?

One approach some projects take -- and I think jackson-module-kotlin is an example -- is to change test itself to test for expected failure. While this does fulfill goals, I don't like the idea of "testing the wrong thing" -- especially when failure modes vary a lot. It seems better to have just one tests, checking for expected correct behavior.

I am ok with some minor changes to tests: for example, adding annotation(s). But I don't know if any test framework (well, JUnit / Maven surefire to run) allows such definition of "Expect non-passing; fail build if pass"? Is anyone familiar with such features, extensions?

EDIT: reference to Python's @unittest.expectedFailure which seems to be about what I'd like:

https://stackoverflow.com/questions/8017514/tdd-practice-distinguishing-between-genuine-failures-and-unimplemented-features

[cowtowncoder]

/cc @pjfanning @JooHyukKim @k163377 @yawkat @carterkozak

[JooHyukKim]

Now we have done jackson-databind module via FasterXML/jackson-databind#4696.
Do we need this new feature in some other module also?

[cowtowncoder]

We could consider it, but none of other modules have as many. So not sure.

[cowtowncoder]

No good solution via SO:

• https://stackoverflow.com/questions/4055022/mark-unit-test-as-an-expected-failure-in-junit (refers to Python's @expectedFailure)

And ChatGPT pointed to JUnit "assertThrows" while admitting it's not quite the same (solid answer, assuming it didn't miss anything)

[cowtowncoder]

One immediate problem is that getting an uncaught exception is not the only way for test to fail -- and not necessarily the most common even: failing assertions are. I guess one could catch assertion errors... but fundamentally it all comes down to this:

1. Test can pass in just one way: no uncaught exceptions, failing assertions
2. There's unbound number of ways a test can fail

so it really would be best to just say "fail if test passes" and do nothing else. Adding code to check exact way test happens to fail now is not great.

[JooHyukKim]

Theres actually JUnit life cycle to do exactly that, Make tests fail when tests pass

[cowtowncoder]

@JooHyukKim that sounds good -- how easy would that be to use?

[JooHyukKim]

Will try to write sample code and share here then see how it feels 👌🏼

[JooHyukKim]

FasterXML/jackson-databind#4696

[JooHyukKim]

Wrote about this work in Medium post. I was thinking sharing with other maintainers from other projects can benefit from this.

Anyway, WDYT of closing this now @cowtowncoder ? Improvement can start in a new discussion

[cowtowncoder]

Great idea to write a blog post!

And yes, will close now.