ORPO, KTO, SimPO & the alignment zoo

1

Concept

After DPO showed that preference alignment could look like supervised pairwise training, many objectives explored different assumptions. The names multiply quickly, but the useful questions stay stable: What feedback format is available? What anchors the policy? How is a response scored? Is length normalized? Are chosen examples also trained with ordinary likelihood? Does the method require a reference model?

ORPO, or Odds Ratio Preference Optimization, combines two signals in one training stage. The supervised term raises likelihood of the chosen response. A preference term compares odds associated with chosen and rejected responses under the policy, penalizing the rejected response relative to the chosen one. It is described as reference-model-free: there is no frozen second model in the objective. “Monolithic” refers to combining instruction learning and preference alignment rather than requiring a separate SFT-then-alignment pipeline.

This combination has an intuitive advantage: preference loss alone can push the rejected response down without ensuring the chosen response remains a strong language-model target. The supervised component explicitly rehearses the desired output. Yet the weighting between terms becomes part of the behavioral specification. Too much imitation can blunt preferences; too much rejection pressure can distort likelihood.

KTO, or Kahneman-Tversky Optimization, changes the feedback interface. It can consume examples labeled desirable or undesirable without requiring two responses to the same prompt. That matters when product feedback arrives as individual thumbs-up and thumbs-down events, where constructing matched pairs would discard data or introduce arbitrary pairing. KTO is motivated by prospect theory: gains and losses are evaluated relative to a reference point and can be weighted asymmetrically.

Unpaired data is not automatically unbiased. Users choose which interactions to rate, positive and negative events may come from different prompt populations, and one person’s dislike may concern style while another’s concerns correctness. Propensity, source, locale, task, model version, and feedback UI should be recorded. Otherwise an objective can learn the selection mechanism rather than the intended preference.

SimPO removes the explicit reference model and uses the policy’s length-normalized log-probability as an implicit reward. For a response $y$ of length $|y|$, the score is based on

Training makes the chosen score exceed the rejected score by a target margin. Length normalization addresses the fact that summed log-probabilities become more negative as sequences grow. It also changes the inductive bias, so the interaction with response truncation and desired verbosity must be evaluated rather than assumed.

These methods are not interchangeable labels on the same implementation. Reference-free training can reduce memory, but the missing reference changes regularization. Pairwise objectives exploit within-prompt comparisons, while unpaired objectives can use more naturally collected feedback. Combined SFT terms can preserve response likelihood, while separate stages allow independent data mixtures and schedules.

Choosing a method begins with the data contract. If reliable pairs exist and a reference is affordable, DPO is a strong baseline. If only binary feedback exists, KTO’s interface may fit. If memory for a reference model is limiting, ORPO or SimPO may be attractive. Then compare against SFT and task-specific baselines using identical data where possible.

The durable lesson is not to memorize acronyms. Map every objective onto four axes: feedback unit, response score, anchor or regularizer, and optimization target. Once those are explicit, the zoo becomes a collection of testable design choices rather than a parade of fashionable names.