2026-10-18T05:36:14Z https://lifeofresearch.com/oai2

oai:lifeofresearch.com:11 2026-09-22T17:33:51Z type:article openaire

Catastrophic Forgetting in Continual RLHF: A Measurement Framework for Round-Over-Round Capability Degradation Pranay Mahendrakar Reinforcement learning from human feedback (RLHF) is widely understood to incur an alignment tax: aligning a language model with human preferences can degrade capabilities the base model possessed. This phenomenon is well documented in single-round comparisons. What is not well documented, despite being the actual production setting, is the cumulative degradation across multiple rounds of RLHF — the iterated case in which preference data is collected, a reward model is retrained, and the policy is updated repeatedly. This paper argues that the existing alignment-tax literature, while valuable, leaves five distinct measurement gaps unaddressed: round-over-round longitudinal dynamics, capability-stratified rather than aggregate degradation, systematic comparison across RLHF algorithms, long-tail and rare-capability decay, and mechanistic understanding of why specific components forget. We propose a measurement framework targeting each of these gaps and a concrete experimental protocol — a multi-round RLHF study on an open base model with capabilitydecomposed evaluation — that academic teams could execute today. We argue that this is one of the higher-leverage open problems in alignment evaluation: the relevant techniques exist, the cost is moderate, the production relevance is high, and the empirical baseline is genuinely thin. Life of Research 2026-04-28 info:eu-repo/semantics/article https://lifeofresearch.com/articles/11 https://doi.org/10.5281/zenodo.19853746 eng https://lifeofresearch.com/articles/11/files/11/Catastrophic_Forgetting_Continual_RLHF_Mahendrakar.pdf application/pdf https://creativecommons.org/licenses/by/4.0/ info:eu-repo/semantics/openAccess