论文：《Inference-Time Scaling for Generalist Reward Modeling》

**核心：the GRM generates principles and critiques based on the input query and responses **

之前deepseek在SFT的基础上加上GRPO强化学习后训练发现效果奇好，模型通过“思考”后的答案在推理方面精度明显提升，比如：代码、数学等，现在他们又提出一种新的强化学习训练方法，要不换个数据标注格式试试？不是让模型像思维链一样连续思考，而是换个方式“思考” ，让它自己对自己的答案自己生成规则和评价，因为作者认为这样的原则是模型根据输入的问题及其答案动态生成的，不是人工预定义的死板原则，因此可以使奖励的生成过程更加自适应，从而提升精度，本质上相当于另一种推理格式 ，仍然是人类价值对齐的一种方式 ，虽然GRPO还是曾经那个GRPO，但换汤不换药的效果竟然确实更好！而且作者说还可以减少数据标注量，只是论文中那个训练数据量对于一般公司还是很大啊！整体方案如下：

首先，对预训练的大模型用带有Specific Criteria 、Analysis 和Scores的RM标注数据进行SFT训练，取名为拒绝式微调（RFT）， 主要作用是规范大模型GRM的回答格式，使其答案中带有上述这种规则和评价，同时可兼容很多种问法，而不是之前R1回答里面带有一系列think那种回答格式，或者普通聊天模型那种最终答案，作者说预训练数据是自有的，但这些强化学习数据来自互联网经过自己修改而成，不过没看到开源，重要的东西他们竟然不开源，嗯，比较恼火！然后，从上述同一RM数据集中取些数据进行GRPO强化学习（RL ）训练，这个目的是使模型的回答更加符合人类偏好，同时加强模型自己生成正确规则和评价的精度，作者实测不用RFT直接RL训练也可以提升精度 ，以上两部分都只训练了900步，说明这个训练方法的精度上限应该还没有达到。训练完之后，作者想通过大量采样（多次生成规则+评论）来进一步提升回答精度，对于每个查询及其对应的回答研究者执行了N_RFT次采样，比如：k=32，却发现推理很慢，即提升了精度却显著增加了推理阶段的计算量，于是作者只能加入并行采样来缓解推理延迟，进一步还用RM数据训练了个分类模型**Meta**** RM** 去除质量低的回答样本，由于RM数据是同源的，作者说精度又提升了，所以放到现实中的未训练数据上测试很可能效果有所下降，最后就可以根据投票机制和最大值函数等后处理输出精度最高的答案了。不过，作者也讲了，此训练方法在通用数据上效果不错，但专业数据和特殊数据上效果就有待提升了，可能是这方面的训练数据缺少导致的，当然，即便排除这格因素，模型还是有可能回答错误，因为有些训练数据的标注本身就是错误的，另外，数学推理方面的精度、长文本方面的精度等目前都还有所欠缺，他们还要继续优化。上面这个图看起来并不太方便，根据以上文字说明，再看下面这个图就清晰些了，上方为微调+RL训练，下方为推理：

训练阶段，研究者让GRM自行生成原则，原则引导的奖励生成函数为：

推理阶段，Meta RM去除质量低的回答样本之后，采用逐点投票获得样本的最终奖励：

实际上，里面的微调SFT、GRPO和Meta RM都是以前提出的已有算法，重点是它的数据标注格式 ，作者却把它放到了非论文主体部分，这让看此算法论文的难度明显增加了。。。默认的训练数据标注格式如下（其它格式可以从论文中查找参考）：

一、Q&R：

# Query
Please help me review for my behavioral neuroscience exam by giving
me the core information that is essential to this course. Please give an
expansive review and include explanations in analogy or metaphors

同一Query采样出多个回答

# Response 1
Behavioral neuroscience, also known as biological psychology, explores
the relationship between the brain and behavior. It examines how the
nervous system influences cognitive processes, emotions, and motor func
tions. Here’s a comprehensive review with analogies and metaphors:\n
1. **Neuroanatomy**: Imagine the brain as a complex city with differ
ent neighborhoods (lobes) and roads (neuronal pathways). The cerebral
cortex is the outer layer, like the skin of the city, responsible for higher
functions like thinking and perception. The cerebellum is like a chore
ographer, ensuring smooth and coordinated movements. The brainstem
acts as the city’s backbone, connecting the brain with the spinal cord, and
controlling basic life functions.
2. **Neurons**: These are the building blocks of the nervous system.
Think of them as the city’s messengers, transmitting information via
electrical and chemical signals. The dendrites receive messages, the cell
body processes them, and the axon sends the signal to other neurons at
the synapse.
3. **Synapses and neurotransmitters**: The synapse is like a tiny bridge
between neurons. Neurotransmitters are the messengers that cross this
bridge, transmitting signals from one neuron to another. Analogous to
different languages, there are various neurotransmitters (e.g., dopamine,
serotonin, glutamate) that influence distinct aspects of behavior and
mood.
4. **Sensory Systems**: These are the city’s surveillance cameras, detect
ing external stimuli. For example, the visual system (retina) captures
light, and the auditory system (cochlea) picks up sound waves. These
systems transmit information to the brain for processing.
5. **Motor Systems**: The motor systems act like the city’s transportation
network, sending signals from the brain to muscles, enabling movement.
The somatic motor system controls voluntary movements, while the au
tonomic nervous system is like an unseen hand managing involuntary
functions like heart rate and digestion.
6. ** Plasticity**: Brain plasticity is like the city’s ability to adapt and
change over time. It refers to the brain’s capacity to reorganize itself by
forming new neural connections in response to experiences, learning, or
injury.
7. **Memory**: Memory can be thought of as a vast library where in
formation is stored and retrieved. There are three main types: sensory
memory (like a quick glance at a book), short-term memory (reading a
paragraph), and long-term memory (remembering the entire book).
8. **Learning**: Learning is the process of acquiring new knowledge,
skills, or behaviors. It’s like adding new books to the library or updat
ing old information. There are different types, such as classical condi
tioning (learning through association, like Pavlov’s dogs) and operant
conditioning (learning through consequences, like reinforcements and
punishments).
9. **Emotions and Motivation**: Emotions are like the city’s weather, in
fluencing how we perceive and respond to events. Different brain regions,
like the amygdala (the emotional storm center), contribute to emotional
experiences. Motivation can be seen as the city’s energy, driving us to act
in pursuit of goals, with systems like the mesolimbic dopamine pathway
playing a key role.
10. **Hormones and the Endocrine System**: These are the city’s internal
postal service, sending chemical messages throughout the body. Hor
mones, like cortisol and oxytocin, can influence behavior and mood by
interacting with the brain and nervous system.\n
Understanding these core concepts will help you navigate the complex
world of behavioral neuroscience and prepare for your exam. Remember
that this field is constantly evolving, so stay curious and engaged with
the latest research.

# Response 2
Certainly! Behavioral neuroscience is a fascinating field that explores the
intricate relationship between the brain and behavior. It’s like unraveling
the mysteries of a complex machine, where the brain is the central pro
cessing unit, and behavior is the output. Let’s dive into the core concepts
that form the foundation of this course.\n
1. Neuroanatomy: Understanding the Brain’s Architecture\n The brain is
a remarkable structure, and understanding its anatomy is crucial. Think
of it as a vast city, with different regions serving as specialized districts.
The cerebral cortex is the bustling metropolis, where higher cognitive
functions like decision-making and problem-solving take place. The lim
bic system, on the other hand, is the emotional heart, regulating feelings
and memories. And deep within, the brainstem acts as the city’s control
center, managing essential functions like breathing and heartbeat.\n
2. Neurophysiology: The Language of the Brain\n Just as a city commu
nicates through intricate networks of roads and signals, the brain uses
electrical and chemical signals to transmit information. Neurons, the
brain’s messengers, fire action potentials like lightning bolts, carrying
vital information. Neurotransmitters, on the other hand, are the chemical
couriers, facilitating communication between neurons at synapses, akin
to the handoff of a letter from one postal worker to another.\n
3. Sensory and Motor Systems: The Brain’s Gateways\n The brain is not
an isolated entity; it constantly interacts with the external world. The
sensory systems act as the brain’s eyes, ears, and touch receptors, gath
ering information from the environment. Imagine them as surveillance
cameras and sensors, feeding data into the brain’s processing centers.
Conversely, the motor systems are the brain’s actuators, executing com
mands and translating neural signals into physical movements, like a
robotic arm carrying out instructions.\n
4. Learning and Memory: The Brain’s Database\n Just like a computer
stores and retrieves data, the brain has its own mechanisms for learn
ing and memory. Long-term potentiation (LTP) is like upgrading the
bandwidth between neurons, strengthening connections and facilitating
information storage. Consolidation is akin to backing up data, transfer
ring memories from short-term to long-term storage. And retrieval is like
accessing a file, recalling stored information when needed.\n
5. Motivation and Emotion: The Brain’s Driving Forces\n Behavior is
not just a product of cognitive processes; it’s also influenced by motiva
tion and emotion. The reward system, centered around structures like
the nucleus accumbens, acts as the brain’s carrot-and-stick, reinforcing
desirable behaviors with pleasurable sensations. The amygdala, on the
other hand, is the brain’s security guard, processing emotional cues and
triggering appropriate responses, like the fight-or-flight reflex.\n
6. Neuroplasticity: The Brain’s Adaptability\n The brain is not a static
entity; it’s a dynamic and ever-changing landscape. Neuroplasticity is
the brain’s ability to reorganize and rewire itself in response to experi
ence and environmental changes. It’s like a city constantly undergoing
renovations, adapting to new demands and challenges. This plasticity
allows the brain to recover from injuries, acquire new skills, and adapt to
changing circumstances.\n
7. Neurological Disorders: When the Brain Malfunctions\n Just as a
city can experience breakdowns and malfunctions, the brain is suscepti
ble to various disorders and dysfunctions. Neurodegenerative diseases
like Alzheimer’s and Parkinson’s are akin to the gradual deterioration
of a city’s infrastructure, leading to cognitive and motor impairments.
Psychiatric disorders, such as depression and anxiety, can be viewed
as imbalances in the brain’s chemical signaling system, disrupting emo
tional regulation and behavior.\n
By understanding these core concepts and embracing the analogies and
metaphors, you’ll gain a deeper appreciation for the intricate workings
of the brain and its profound influence on behavior. Remember, the brain
is a complex and fascinating realm, and behavioral neuroscience is the
key to unlocking its secrets.

二、DeepSeek-GRM-27B Result:

# DeepSeek-GRM-27B Result
Specific Criteria:
1. **Depth of Analogies and Metaphors** (Weight: 30%): The extent
to which the analogies and metaphors used are deep, creative, and
effectively convey complex concepts.
2. **Comprehensiveness of Core Information** (Weight: 30%): The
thoroughness with which the core information essential to behavioral
neuroscience is covered.
3. **Clarity and Organization** (Weight: 20%): The clarity of explanations
and the logical organization of the information.
4. **Engagement and Appeal** (Weight: 20%): The level of engagement
and appeal the response has, making the information interesting and
memorable.
Analysis:
- **Response 1**: This response excels in comprehensiveness and clarity,
covering all major aspects of behavioral neuroscience with detailed
and well-organized explanations. The analogies and metaphors are
well-crafted and effectively enhance understanding. However, some
metaphors could be more creative to enhance engagement further.
- **Response 2**: This response is also comprehensive and clear, but
it stands out more in the depth and creativity of its analogies and
metaphors. The organization is logical, and the explanations are clear.
The engagement level is high, making the information appealing and
memorable.
Scores: \boxed{8, 9}

# Ground Truth
Response 2 is better than Response 1.

Specific Criteria为生成的规则，Analysis为生成的评论，都是文本。

而之前普通的RM标注结果为：

# DeepSeek-BTRM-27B Result
Response 1: 0.4665; Response 2: 0.3209.

作者使用的训练数据说明：

训练集包括1250K RFT数据，包括1070K通用指令数据和186K拒绝采样数据和237K RL数据，一般指令数据来自
内部数据集，拒绝采样数据和RL数据来自相同的RM数据集，涵盖单个、成对和多个响应这三种数据格式，
来源于内部数据和开源数据集。

论文同时提出了一些未来的改进方向：

• 整合工具 ：将代码解释器、搜索引擎等工具融入 RM 中，可以提高批判的准确性，尤其是在需要精确步骤或广泛知识的任务中。
• 解耦****原则与批判生成 ：将原则生成和批判生成分解为独立的阶段，可以提高效率并为集成到 RL 流程中带来便利。
• 用于 ** LLM 离线评估** ：利用 DeepSeek-GRM 生成的原则，可以分析特定 LLM 的弱点，为模型评估提供更具解释性的协议。
• 探索长程推理 ：将 DeepSeek-GRM 与长程推理技术相结合，可能进一步提升其在复杂推理任务上的性能。

另外，由于本算法出自deepseek，自然也用了可以大量节省计算量的MOE ：

MoE的核心：它并非让所有专家同时工作，而是通过一个路由器（moe的大脑：softmax+topk）为每个输入（例如文本中的token）动态选择一个或多个专家（多个小网络）进行处理，从而解决硬件性能瓶颈。