feat(osv): read OSV vulnerabilities from the database instead of the HTTP API

[scanoss-qg]

What this does

OSV vulnerabilities are read from the osv and osv_severity tables instead of api.osv.dev. The response is unchanged: same fields, same source, same URL construction, and the cvss array still carries every vector.

Verified against production data: for pkg:maven/org.apache.logging.log4j/log4j-core@2.0.0 this returns the same 8 advisories the API returns, in 0.70 s instead of 1.60 s, and with identical results on PostgreSQL and SQLite (18 → 8 after the ecosystem filter below; the 18 were byte-identical across both engines before it).

Removed with the HTTP client

• getRepoURL and the GIT-ecosystem fallback. The table stores pkg:github purls directly (118,211 rows), so a component is looked up by its purl with no translation to a repository URL and no retry
• OSVRequest / OSVPackageRequest, which only shaped the API payload
• packageurl-go is no longer a direct dependency
• VULN_OSV_API_BASE_URL is gone from the config. It was still validated — an empty value prevented startup for a setting nothing read, which is what someone would try in order to stop the service calling the API

VULN_OSV_VULNERABILITY_INFO_BASE_URL stays, since it builds each vulnerability's URL. VULN_OSV_API_WORKERS keeps its meaning as the concurrency limit, now over database lookups.

The parts that took the most care

Version matching. The API decided which advisories applied to a version; now we do. OSV uses two mechanisms and both are honoured: the explicit affected_versions list and the introduced/fixed/last_affected range. Which one a row uses varies wildly by ecosystem — pypi is almost all lists (19,047 vs 1,395), golang almost all ranges (12,510 vs 165). Comparison reuses naturalSortKey, already ported and tested.

Collapsing rows. The table is keyed per affected range and averages 15.6 rows per vulnerability; the API returns one entry. Which row supplies the vulnerability's fields is not arbitrary: 137,302 (id, purl) pairs disagree across their rows, almost always on modified, and the most recently modified row is the one that agrees with osv_json and the API. That row wins even when a different one matched the version.

Parsing the list columns. affected_versions and aliases are PostgreSQL arrays rendered as {a,b} text. 1,082 production rows have a quoted element and some contain a comma ({"v1,1",v1.1}), so splitting on commas alone would invent versions that do not exist.

Excluding repackager ecosystems. OSV publishes advisories under Vendor:LanguageEcosystem names (TuxCare:Maven, Echo:npm). They share the purl of the upstream package, so a lookup by purl picked them up while the API, filtering by ecosystem, does not return them — 10 extra advisories on log4j-core alone. They are excluded by vendor prefix. Worth recording why not by "contains a colon": legitimate distro ecosystems are versioned that way (Ubuntu:22.04:LTS, Debian:12) and that pattern covers 3,770,300 of 5,480,673 rows — it would have silently dropped every Debian and Ubuntu advisory. The repackagers are 31,093 rows, 0.57%.

Tests

The OSV use case tests used to call api.osv.dev and assert only that something came back, so they needed internet and exercised whatever OSV served that day. They now run on SQLite against a fixture covering both matching mechanisms, several rows per vulnerability, rows disagreeing on modified, a two-vector advisory, a non-CVSS Ubuntu score that must be skipped, and one repackager plus one distro row to pin the ecosystem filter in both directions.

TestOSVParity compares the model against the live API and is skipped unless PG_DSN and OSV_PARITY are set, so CI never reaches the network.

Deployment requirements

• The osv and osv_severity tables must be present and populated. osv_severity holds the CVSS vectors, which osv cannot: a vulnerability can carry up to five
• osv must carry every OSV affected entry, including those whose ranges are type ECOSYSTEM (93% of all ranges). An earlier load kept only SEMVER ranges and silently dropped advisories — pkg:gem/rack@0.4.0 returned 6 instead of 32. Already fixed on the production database
• Both tables are in map_schema.json for the SQLite export

Known limitation

OSV marks retracted advisories with withdrawn and the table has no such column, so 43,739 retracted advisories across 192,159 rows are reported as live. This is the one remaining parity gap and causes the single false positive in TestOSVParity. A data ticket for it is pending.

Verification

go build, go vet, the full suite, and golangci-lint v2.10.1 with a clean cache all pass. Behaviour confirmed end to end against both the production PostgreSQL database and the exported SQLite file.