Credit assignment in agent reinforcement learning

Inspired by recent work on credit assignment ( Pignatelli et al., 2023 ) for pure text reasoning tasks ( Shao et al., 2024 ; Cui et al., 2025 ; Cheng et al., 2025 ) , in this paper, we introduce a fine-grained turn-level credit assignment strategy for multi-turn LLM agent training. Compared with textual reasoning tasks like mathematical problem solving, multi-turn agent interactive tasks present a more intuitive setting to highlight the importance of fine-grained credit assignment. The key contributions are as follows: • We propose modeling multi-turn long-horizon reasoning tasks in LLM agents as Markov Decision Processes (MDPs), which naturally capture the sequential decision-making structure of such problems. To train multi-turn LLM agents effectively within the MDP framework, we present a fine-grained turn-level advantage estimation strategy using both outcome and turn-level rewards. In this work, we instantiate our approach within the GRPO algorithm. Notably, our strategy is general and can be compatible with a wide range of RL methods. • To highlight the importance of credit assignment mechanisms in multi-turn reasoning, we construct an agent that performs question answering using a Wikipedia search tool. The agent operates in multiple steps: reasoning, search, and answer summarization. It learns to leverage the Wikipedia search engine to retrieve relevant information in support of its final answer through RL training. Figure 1 illustrates the multi-turn agent workflow and compares baselines of trajectory-level advantage estimation with our proposed GRPO-based variant. • Experimental results on multi-turn reasoning and search tasks show that compared with baselines using trajectory-level advantage estimation, our MDP formulation and fine-grained turn-level credit assignment significantly improve the multi-turn reasoning performance of LLM agents in complex decision-making tasks. In particular, our method achieves 100% success in tool invocation and 50% accuracy in exact answer matching, significantly outperforming baselines, which fail to invoke tools and achieve only 20–30% exact match accuracy. Additionally, we find that our method promotes more stable and consistent tool use during training, whereas baselines with coarse-grained trajectory-level credit assignment often forget to call tools and exhibit higher variance. These findings further highlight the critical role of precise credit assignment in effective multi-turn agent training.

Group-based reinforcement learning. Unlike PPO ( Schulman et al., 2017 ) , which estimates advantages using an additional value function, group-based reinforcement learning (RL) algorithms such as GRPO ( Shao et al., 2024 ) compute advantages directly from the statistics of a sampled group of trajectories $G_{\tau}$ . Specifically, GRPO was originally designed for single-turn tasks under a trajectory-wise policy optimization framework. To extend it to long-horizon tasks, we adapt it to the stepwise setting and calculate the trajectory-level advantage as: $\displaystyle A^{T}(\tau_{i})=\left(R({\tau_{i}})-1/|G_{\tau}|\sum\nolimits_{j\in G_{\tau}}R({\tau_{j}})\right)/\sigma_{G_{\tau}},$ (1) where $\sigma_{G_{\tau}}$ denotes the standard deviation of rewards within the group $G_{\tau}$ . This trajectory-level computation assigns the same advantage value to every step in trajectory $\tau_{i}$ , thereby overlooking the finer credit assignment required within a trajectory. To address this limitation, one can instead adopt a step-level group relative advantage estimator ( Feng et al., 2025b ) . Here, steps with identical current states $\tilde{\bm{s}_{i}}$ across all group trajectories are clustered into step-level groups $G_{\tilde{\bm{s}_{i}}}$ , and their advantages are computed as: $\displaystyle A^{S}(\tilde{\bm{s}_{i}})=\left(R(\tilde{\bm{s}_{i}})-1/|G_{\tilde{\bm{s}_{i}}}|\sum\nolimits_{j\in G_{{\tilde{\bm{s}_{i}}}}}R(\tilde{\bm{s}_{j}})\right)/\sigma_{G_{{\tilde{\bm{s}_{i}}}}}.$ (2) Compared to Eq. ( 1 ), the step-level estimator in Eq. ( 2 ) provides more fine-grained and effective credit assignment across steps within the same trajectory.

Standard RL fine-tuning for large language models distributes credit at token granularity within a single generation [ 36 , 43 , 45 , 69 , 3 ] , while credit assignment across multi-turn environment interaction remains less settled [ 70 , 46 , 66 , 29 , 56 , 79 , 22 ] .