produce false positive and false negative PRs

[vjcitn]

to help the team get a handle on adequacy of automated review of package PRs, could
we have

• false positive: PR that addresses an "error" that is falsely identified

• false negative: PR that identifies a real error but fails to solve it

It would be nice to have a systematic way of doing this, maybe along the lines of "hedgehog" property-based testing

[jmg421]

Hedgehog property-based testing for VariantAnnotation

This is a genuinely interesting direction. Here's my reading of the request and a concrete proposal.

What the issue is asking

Two things:

1. False positive PR — a PR that 'fixes' something that isn't actually broken (agent hallucinates a bug)
2. False negative PR — a PR that correctly identifies a real bug but whose fix is wrong or incomplete

These are the two failure modes of automated PR generation, and having labelled examples of each in the repo would give reviewers a calibration tool: "does this PR look more like our known-false-positive examples, or our known-correct examples?"

Why hedgehog is the right tool here

The existing tests in are example-based: they check specific hand-crafted inputs. The problem is that automated fixes can be tuned (accidentally or intentionally) to pass exactly those examples while still failing on nearby inputs.

Property-based testing with inverts this. Instead of checking , you state invariants that must hold for any valid input — e.g.:

• round-trip: should equal the original
• idempotency: always
• consequence taxonomy: result is always one of
• Allele alphabet: and are always valid objects (never NULL)

A false-positive fix might pass 6 hand-crafted unit tests but fail a hedgehog property in 3 of 100 random draws. That's the detection mechanism.

Concrete plan

I'll open a PR that adds with:

1. **** — generates random-but-valid VCF variants, writes them, reads them back, checks structural invariants
2. **** — generates random genomic positions + alleles, checks consequence taxonomy, AAStringSet types, CDSLOC validity
3. **** — generates VCFs with varying INFO Number fields, checks row counts and header consistency

Plus a **** and **** subdirectory with annotated examples showing what each failure mode looks like — to serve as the calibration corpus Vince is describing.

Is this the right scope? Happy to start narrower (just one property + the two labelled examples) if that's more useful for the initial evaluation.

[vjcitn]

Thanks for moving forward on this. I think it is basically in the right direction, but I would go slowly and dialogue frequently as steps are taken.