Apparent sequence-model contribution depends strongly on negative-set construction
Across 94 ENCODE eCLIP datasets, how much a sequence model appears to add beyond nucleotide composition shifts by a factor of nearly five depending solely on how the negative examples are constructed, with the model, source peaks, and fold design held fixed.
Where this came from
The original goal was to see whether a model trained to predict RNA-binding protein binding sites could also flag disease-causing variants. For a while it looked promising: SpliceBERT separated pathogenic from benign non-coding variants at 0.83 AUROC without ever seeing a disease label.
Then I built the control. Evolutionary conservation alone, with no sequence model at all, reached 0.91. Scoring each variant with a different protein's model still reached 0.68. The model was contributing something, but far less than the AUROC suggested, and most of what looked protein-specific was not.
While checking that, what kept getting in the way was the negatives. Every AUROC I computed depended on which windows I had decided to call “not bound”, and I could not find a paper in the literature that reported what a plain nucleotide counter scores on its own negative set. So I stopped chasing a better model number and measured the thing all the numbers rest on.
Key numbers
The finding
With the model class, source peaks, chromosome-blocked fold design, and estimator held fixed, the estimated incremental contribution of a 4-mer model beyond composition spans 4.84-fold (95% CI 3.98 to 5.81) across three negative-set protocols. The apparent AUROC moves in the opposite direction: stricter composition matching lowers it in 94 of 94 datasets while the estimated contribution rises.
The movement is not an AUROC artifact. It survives five estimands including unbounded deviance. It holds for a 7089-parameter CNN and a fine-tuned SpliceBERT. And it replicates directionally on 135 held-out datasets from Horlacher et al. 2023, a benchmark I did not build, covering 108 proteins.
The practical implication is narrow but firm: report a composition-only AUROC under the same negative-set protocol as the sequence-model AUROC. Incremental contributions computed under different protocols are not comparable.
Three negative-set protocols
Across all three arms the positives, model architecture, source peaks, and fold design stay fixed. Protocols A and B vary the composition-matching constraint on randomly sampled windows. Protocol C changes the candidate pool entirely, drawing from other proteins' binding sites rather than varying a matching threshold.
Negative windows are matched to the positives on dinucleotide frequencies. This is the tightest composition constraint: it achieves a lower median GC gap than GC-matching and cuts dinucleotide mismatch 2.27-fold, producing a composition baseline of 0.627 and the most headroom for a sequence model to contribute.
+0.0663 incremental AUROCNegatives are matched on overall GC content. GC is a single summary statistic, so dinucleotide patterns can remain mismatched — a looser constraint than Protocol A. The composition baseline rises to 0.783 and the measured incremental contribution falls accordingly.
+0.0265 incremental AUROCNegatives are drawn from other proteins' binding sites rather than randomly sampled windows. This arm is not a third setting of the same composition-matching knob: only 20% of pairs end up composition-matched, and not by design. The high composition baseline (0.825) leaves little headroom for a model to add anything.
+0.0122 incremental AUROCEstimator choice matters too
The standard two-stage estimator used in the literature returns a non-zero incremental contribution even when the model contains no information the composition baseline does not. Applied to that null case, it returns values between +0.011 and +0.014, which is 90% of the smallest arm's reported increment.
Cross-fitting cuts this floor by at least 95% and lands within 0.0005 of the known zero. The protocol span shrinks from 5.42-fold to 4.84-fold under cross-fitting, which is why the cross-fitted estimate is reported as primary throughout the paper.
Verification and reproducibility
Every number in the paper is committed to the repository alongside the code that produces it. A standalone verifier checks 1156 numeric assertions against a golden-value file, fails loudly on any disagreement, and works offline with no cloud credentials or GPU.
285 AUROCs are recomputed from committed per-window scores rather than compared against a stored value, and the headline contrast is recomputed from raw sequence. Two of the neural sweeps are committed rather than rebuilt by the default path; the provenance file names which tables belong to which category.
Infrastructure
The CPU-heavy preprocessing and fold scoring ran on GCP Batch with results written to GCS. GPU fine-tuning used Modal, which handled container provisioning and returned scores locally rather than back to GCS. Final analysis and figure generation ran locally against the downloaded tables.