Understanding how language models generate text often starts with the output, a token appearing one after another, but rarely examines the probability structure underneath. A recent post by Sithankanna breaks down the probability of an LLM trajectory from first principles, starting with the chain rule and working toward the simplified product form that underpins algorithms like REINFORCE. The derivation is short, precise, and worth walking through for anyone who wants to understand what the model is actually computing when it generates a sequence.
What constitutes a trajectory
A trajectory is the full sequence of states and actions from generation start to finish. For a short example, the author defines it as a sequence where you begin in state s0, take action a0, arrive at state s1, take action a1, and end at state s2. In the context of a language model, the state is the full prompt plus any tokens generated so far, and the action is the next token sampled from the model's output distribution.
The question is straightforward: what is the probability of this entire trajectory under a policy with parameters θ? The policy is the model itself, and the parameters are its weights. For each possible next token, the model outputs a probability distribution. That distribution is the policy.
The chain rule gives you the starting point
The derivation begins with the chain rule for three random variables: the joint probability of x, y, and z equals p(x) times p(y given x) times p(z given x and y). Applied to a trajectory with states and actions, this expands into a product where each term conditions on everything that came before it.
At this stage, no assumptions have been made about the policy or the environment. The expansion is pure probability, and it applies to any sequence of dependent random variables. What follows is a series of simplifications that exploit the specific structure of how language models generate text.
The initial state doesn't depend on the model
The first simplification is trivial but important. The policy doesn't choose the initial state. The prompt is given to the model before generation begins. Therefore, the probability of the initial state given the policy parameters reduces to the probability of the initial state alone. This removes θ from the first term.
For anyone coming from supervised learning, this maps directly: the training data exists independently of the model. The model doesn't generate its own input.
The environment transition is independent of the policy
The next state in the trajectory is produced by the environment, not the policy. The policy chooses an action, and the environment responds with the next state. Crucially, once the current state and action are given, the environment uses the same transition rule regardless of the policy parameters. This means the environment transition terms don't depend on θ either.
The author then applies the Markov assumption to the environment: given the current state and action, earlier states and actions are irrelevant for determining the next state. This requires that the state contain all information from the past needed to predict what comes next. In a language model, the state is the entire sequence of tokens generated so far, which satisfies this requirement by construction.
The policy simplifies under the Markov assumption
The policy's action probabilities also simplify. If the current state contains the relevant history, then the probability of taking an action depends only on the current state, not the full trajectory. The author introduces the notation πθ for the policy, making it visually distinct from environment probabilities. Under the Markov assumption, the action probability at each time step reduces to πθ(at given st).
This is where the factorization starts to look clean. Each time step contributes two terms: the policy probability for the action taken, and the environment probability for the resulting state. The full trajectory probability is a product of these paired terms across all time steps.
Token generation makes the environment terms disappear
This is the key insight that connects the general derivation to language models specifically. When a language model generates a token, the next state is obtained by appending that token to the current state. Once the current state and the chosen token are known, there is exactly one possible next state: the concatenation of the two. The transition probability is therefore equal to one for every valid token trajectory.
This isn't a simplification or an approximation. It's a mathematical consequence of how text generation works. The environment doesn't introduce randomness beyond what the policy already determines. Every environment transition term evaluates to one, and they vanish from the product.
The result is that the trajectory probability reduces to a product of policy probabilities alone. For a given prompt, the probability of generating a specific sequence of tokens is the product of the model's probability for each token given the prompt and all preceding tokens.
Why this matters beyond the math
The derivation ends here, and the author notes that the policy gradient itself is derived elsewhere. But the factorization is the foundation. When you understand that trajectory probability is just a product of conditional token probabilities, concepts like log probability, sampling strategies, and reinforcement learning from human feedback start to make more concrete sense. The model isn't doing anything mysterious at the sequence level. It's multiplying probabilities, one token at a time.
For developers working with language models, this framework explains why log-probability is a natural measure of sequence likelihood, why beam search tries to maximize the product of token probabilities, and why RLHF needs to account for the full trajectory distribution rather than individual tokens in isolation. The chain rule factorization is simple, but it's the right place to start understanding everything that builds on top of it.