Deduplicate the calibration reference by peptidoform (v4.2.0)

CHANGELOG.md

@@ -6,6 +6,37 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [4.2.0] - 2026-08-28
+
+### Changed
+
+- Calibration and fine-tuning now use **one PSM per peptidoform**, the first observation in
+  the reference. A reference built from a search result repeats a peptidoform once per
+  spectrum it was identified in, each time with a different observed retention time, so the
+  fit was given conflicting targets and weighed peptidoforms by how often they happened to be
+  identified. On a reported MS2Rescore case (201,593 PSMs, one run) the reference selected by
+  auto-calibration held 6,331 PSMs but only 2,623 peptidoforms; the repeats of one peptidoform
+  disagreed on the observed retention time by 236 s, two thirds of the 354 s range covered by
+  the identifications. Fitting on the first observations improved the calibration on those
+  2,623 peptidoforms from 9.69 to 4.51 s mean absolute error (median 5.98 to 2.09 s; within
+  5 s 44.9 % to 81.4 %). Retention times are in whatever unit the input uses; DeepLC does not
+  convert them.
+
+  Charge states of one peptidoform count as repeats, since retention time does not depend on
+  precursor charge. When repeats disagree by a large fraction of the observed range, DeepLC
+  now says so: that means the reference mixes runs or contains low-confidence PSMs, which
+  deduplication hides rather than fixes.
+
+  There is no option for it: the public functions keep the signature they had. A caller who
+  really wants every reference PSM to count fits a `Calibration` on its own targets and passes
+  it to `predict_and_calibrate`, which uses an already fitted calibration as given; `train`
+  remains available for full control over a training set.
+
+### Added
+
+- `deeplc._reference_selection.deduplicate_psms`, which returns the first PSM of every
+  peptidoform in a `PSMList`.
+
 ## [4.1.1] - 2026-08-26
 
 ### Changed

deeplc/__main__.py

@@ -135,7 +135,14 @@ def _validate_finetune(ctx, param, value):
     help="Path to a model file. Uses the built-in default model if not provided.",
 )
 def predict(
-    psms, psm_filetype, reference, reference_filetype, auto_calibrate, finetune, output, model
+    psms,
+    psm_filetype,
+    reference,
+    reference_filetype,
+    auto_calibrate,
+    finetune,
+    output,
+    model,
 ):
     """Predict retention times for a list of peptide-spectrum matches."""
     if auto_calibrate and reference:

deeplc/_reference_selection.py

@@ -128,3 +128,108 @@ def _select_by_score(candidates: PSMList) -> PSMList:
 
     top_indices = np.argsort(scores)[::-1][:n_select]
     return candidates[top_indices]
+
+
+def deduplicate_psms(psm_list: PSMList, ignore_charge: bool = True) -> PSMList:
+    """
+    Keep one PSM per peptidoform, the first occurrence in the list.
+
+    A reference set built from a search result usually contains the same peptidoform
+    identified in many spectra, with a different observed retention time each time. Those
+    repeats do not add information about the gradient: they give the calibration one x value
+    with several conflicting y values, and their number is what a spline fit weighs, so a
+    peptidoform seen 261 times counts 261 times while one seen once counts once. On a
+    reported MS2Rescore case, 6,331 reference PSMs collapsed to 2,623 peptidoforms, and the
+    observed retention times of one repeated peptidoform differed by two thirds of the whole
+    observed range.
+
+    Only the first observation is kept, which is what a caller who has already sorted or
+    filtered its PSMs expects, and it makes the result independent of how many times a
+    peptidoform happened to be identified.
+
+    Parameters
+    ----------
+    psm_list
+        PSMs to deduplicate.
+    ignore_charge
+        Treat charge states of the same peptidoform as duplicates (default). Retention time
+        does not depend on precursor charge, so the charge states of one peptidoform are
+        repeats of the same measurement. Set to False to keep one PSM per peptidoform *and*
+        charge.
+
+    Returns
+    -------
+    PSMList
+        The first PSM of every peptidoform, in the original order.
+
+    """
+    seen: set[str] = set()
+    keep = np.zeros(len(psm_list), dtype=bool)
+    for i, psm in enumerate(psm_list.psm_list):
+        # modified_sequence is the ProForma string without the charge state, so it keeps the
+        # terminal and global modifications that distinguish two peptidoforms while ignoring
+        # charge. Taking it from psm_utils also avoids cutting the string ourselves, which a
+        # modification label containing a slash would break.
+        key = psm.peptidoform.modified_sequence
+        if not ignore_charge:
+            key = f"{key}/{psm.peptidoform.precursor_charge}"
+        if key not in seen:
+            seen.add(key)
+            keep[i] = True
+
+    n_dropped = int((~keep).sum())
+    if n_dropped:
+        LOGGER.info(
+            "Deduplicated the reference: %d of %d PSMs are repeats of a peptidoform already "
+            "in the set and were dropped, leaving %d. Pass deduplicate_reference=False to "
+            "keep them.",
+            n_dropped,
+            len(psm_list),
+            int(keep.sum()),
+        )
+        _warn_on_conflicting_retention_times(psm_list, keep, ignore_charge)
+    return psm_list[keep]
+
+
+def _warn_on_conflicting_retention_times(
+    psm_list: PSMList, keep: np.ndarray, ignore_charge: bool
+) -> None:
+    """
+    Report the largest retention-time disagreement among the dropped repeats.
+
+    A small spread is ordinary chromatographic jitter; a spread of the order of the gradient
+    means the repeats are not the same elution event, so the reference was built from PSMs
+    that a search would normally not put in one calibration set (decoys, low-scoring hits, or
+    several runs pooled into one file). Worth saying out loud, because deduplication then
+    hides a data problem rather than solving it.
+    """
+    by_key: dict[str, list[float]] = {}
+    for psm in psm_list.psm_list:
+        rt = psm.retention_time
+        if rt is None or np.isnan(rt):
+            continue
+        key = psm.peptidoform.modified_sequence
+        if not ignore_charge:
+            key = f"{key}/{psm.peptidoform.precursor_charge}"
+        by_key.setdefault(key, []).append(float(rt))
+
+    spreads = [max(v) - min(v) for v in by_key.values() if len(v) > 1]
+    if not spreads:
+        return
+    worst = max(spreads)
+    observed = [
+        float(psm.retention_time)
+        for psm in psm_list.psm_list
+        if psm.retention_time is not None and not np.isnan(psm.retention_time)
+    ]
+    span = (max(observed) - min(observed)) if observed else 0.0
+    LOGGER.log(
+        logging.WARNING if span and worst > 0.25 * span else logging.INFO,
+        "Repeated peptidoforms disagreed on the observed retention time by up to %.2f "
+        "(median %.2f) against an observed range of %.2f. A large disagreement means the "
+        "repeats are not the same elution event: check whether the reference mixes runs or "
+        "includes low-confidence PSMs.",
+        worst,
+        float(np.median(spreads)),
+        span,
+    )

deeplc/core.py

@@ -12,7 +12,7 @@
 from torch.utils.data import DataLoader
 
 from deeplc import _model_ops
-from deeplc._reference_selection import select_reference_psms
+from deeplc._reference_selection import deduplicate_psms, select_reference_psms
 from deeplc.calibration import (
     Calibration,
     SplineTransformerCalibration,
@@ -164,6 +164,13 @@ def calibrate(
         Fitted calibration instance.
 
     """
+    # One point per peptidoform: a reference taken from a search result repeats a
+    # peptidoform once per spectrum it was identified in, each time with a different observed
+    # retention time, which gives the fit conflicting targets and weighs peptidoforms by how
+    # often they happened to be identified. A caller who wants the repeats to count fits a
+    # Calibration itself and passes it in already fitted.
+    psm_list_reference = deduplicate_psms(psm_list_reference)
+
     # Get calibration
     if calibration is None:
         LOGGER.debug("No calibration provided, using SplineTransformerCalibration by default.")
@@ -232,6 +239,7 @@ def predict_and_calibrate(
     predict_kwargs
         Additional keyword arguments to pass to the prediction function.
 
+
     Returns
     -------
     np.ndarray
@@ -313,6 +321,7 @@ def finetune_and_predict(
     predict_kwargs
         Additional keyword arguments to pass to the prediction function.
 
+
     Returns
     -------
     np.ndarray
@@ -443,6 +452,11 @@ def finetune(
     """
     LOGGER.info("Fine-tuning model...")
 
+    # One point per peptidoform, as in calibrate(): training on the same peptidoform several
+    # times with contradictory retention times teaches the model the average of a
+    # disagreement. Use deeplc.train() for full control over the training set.
+    psm_list_reference = deduplicate_psms(psm_list_reference)
+
     # Fine-tuning needs enough reference data to both fit and validate on. The
     # default validation split leaves too few PSMs to early-stop against on a small
     # reference set, which is how a fit ends up worse than the model it started

docs/source/usage.rst

@@ -56,6 +56,19 @@ For calibration, pass a reference file with observed retention times using the
 ``--psm-file-reference`` option. If no reference file is provided, DeepLC
 attempts automatic calibration from high-confidence PSMs in the input file.
 
+The reference is deduplicated before it is used: only the first PSM of each
+peptidoform is kept, charge states included. A search result usually contains
+the same peptidoform from many spectra with a different observed retention time
+each time, and those repeats give the calibration conflicting targets while
+weighing peptidoforms by how often they were identified. This is unconditional;
+a caller that really wants every reference PSM to count fits a
+:class:`~deeplc.calibration.Calibration` on its own targets and passes it to
+:func:`~deeplc.core.predict_and_calibrate`, which uses an already fitted
+calibration as given. When the repeats of a peptidoform disagree by a large
+fraction of the observed retention-time range, DeepLC reports it: the reference
+then mixes runs or contains low-confidence PSMs, and deduplication only hides
+that.
+
 For a full list of options:
 
 .. code-block:: sh

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "deeplc"
-version = "4.1.1"
+version = "4.2.0"
 description = "DeepLC: Retention time prediction for (modified) peptides using Deep Learning."
 readme = "README.md"
 license = { file = "LICENSE" }