RUIN: A Language-Model-Assisted Screen for Formalism, Citation Practice, and Artifact Availability in Computer Science Publications

Valer Bocan · Valentina E. Balas

Research evaluation in computer science relies on citation counts, which measure attention rather than quality and are silent on reproducibility, formalism, and citation relevance. We introduce RUIN (Rigor, Utility, Integrity, Necessity), an experimental screen in which a language model records judgments on four content-level dimensions: formalism proportionality, citation integrity, structural soundness, and artifact availability. Its central construct, proportionality between formal and conceptual complexity, is operationalized through a five-level classification and a three-question Necessity Test, and every aggregate score is computed deterministically from the recorded judgments. Applied to 404 papers from the Romanian Journal of Information Science and Technology, 2010–2025, RUIN finds that only 18.1% rest on obtainable code or data, much of it public benchmarks, and 6.4% release their own code. Formalism theater and excessive self-citation are each flagged in about one eligible paper in five, and with related flags they cap 36.6% of the scores. Scores were stable across repeated runs (intraclass correlation 0.92), and a second analysis of every capped paper reproduced 95% of the caps. Classifications were less stable: alternative scoring rules move up to 113 papers to another band, while the ranking of papers barely changes. Final scores are not distinguishably associated with Web of Science citations (within-year Spearman ρ = −0.09), so the screen is not a citation proxy; whether it measures quality remains untested. We therefore present RUIN as an instrument for assisted assessment, not as a validated measure of integrity. The framework, runs, and data are openly available.