{"id":11,"identifier":"LOR-00011","url":"https://lifeofresearch.com/articles/11","title":"Catastrophic Forgetting in Continual RLHF: A Measurement Framework for Round-Over-Round Capability Degradation","abstract":"Reinforcement learning from human feedback (RLHF) is widely understood to incur an alignment tax: aligning a language\nmodel with human preferences can degrade capabilities the base model possessed. This phenomenon is well documented in\nsingle-round comparisons. What is not well documented, despite being the actual production setting, is the cumulative\ndegradation across multiple rounds of RLHF — the iterated case in which preference data is collected, a reward model is\nretrained, and the policy is updated repeatedly. This paper argues that the existing alignment-tax literature, while valuable,\nleaves five distinct measurement gaps unaddressed: round-over-round longitudinal dynamics, capability-stratified rather\nthan aggregate degradation, systematic comparison across RLHF algorithms, long-tail and rare-capability decay, and\nmechanistic understanding of why specific components forget. We propose a measurement framework targeting each of\nthese gaps and a concrete experimental protocol — a multi-round RLHF study on an open base model with capabilitydecomposed evaluation — that academic teams could execute today. We argue that this is one of the higher-leverage open\nproblems in alignment evaluation: the relevant techniques exist, the cost is moderate, the production relevance is high, and\nthe empirical baseline is genuinely thin.","authors":[{"name":"Pranay Mahendrakar","affiliation":"","orcid":"0009-0003-7224-029X"}],"type":"article","license":"CC-BY-4.0","keywords":[],"publication_date":"2026-04-28","version":1,"views":6,"downloads":5,"created_at":"2026-09-22 17:33:51","doi":"10.5281/zenodo.19853746","source":"Zenodo","source_url":"https://zenodo.org/records/19853746","files":[{"id":11,"name":"Catastrophic_Forgetting_Continual_RLHF_Mahendrakar.pdf","size":91790,"mime":"application/pdf","downloads":5,"download_url":"https://lifeofresearch.com/articles/11/files/11/Catastrophic_Forgetting_Continual_RLHF_Mahendrakar.pdf"}]}