Learning from Preferences · Active learning · Query synthesis

Active Query Synthesis for Learning from Preferences

Generate pairwise comparisons that are both informative and easier to answer, without exhaustively scoring a large query pool.

Namrata Nadagouda · Nauman Ahad · Maegan Tucker · Mark A. Davenport

Read paper

Info-Synth active query synthesis and approximation framework — Info-Synth generates an optimal continuous query; Pair M-dist and Pair Opt-dist map it to allowable dataset pairs when needed.

01 / Motivation

A useful query must be informative and answerable.

Standard pairwise comparison response models can treat every comparison as equally reliable. In practice, near identical options are difficult to distinguish, while highly dissimilar options may lack a meaningful common ground for comparison.

(a) Ideal query — The confidence aware response model favors comparisons at an intermediate, perceptible level of similarity.

(b) Points are too close — The confidence aware response model favors comparisons at an intermediate, perceptible level of similarity.

02 / Method

From response reliability to closed-form query geometry.

The framework localizes an unknown preference vector with Bayesian updates and chooses the next pair by maximizing mutual information.

Confidence aware response model

Reliability depends on the geometry of the pair.

The model uses heteroskedastic noise tied to distances between the user and each candidate, naturally discouraging comparisons that are too close or too far apart.

P(Y|w) = Φ (

‖w − q‖² − ‖w − p‖² σ₀ √‖w − q‖⁴ + ‖w − p‖⁴

)

Approximation methods

Lower dimensional datasets

Pair M-dist

A Hessian induced Mahalanobis metric filters candidate pairs before mutual information refinement.

Higher dimensional datasets

Pair Opt-dist

An optimality based surrogate filters candidate pairs by midpoint, direction and distance deviations, avoiding the expensive Hessian.

Query Synthesis method: Info-Synth

Center at the posterior mean.

The optimal query midpoint lies at the current posterior mean, ensuring equi-probability of the response outcomes.

Align with maximum uncertainty.

The pair is oriented along the principal eigenvector of the posterior covariance; its distance follows from a one-dimensional optimization.

03 / Evaluation

One framework, four evaluation settings.

The evaluation spans a user study, synthetic preference learning, a fixed text summary dataset, and continuous controller tuning.

Human color estimation

A color estimation study tests whether the confidence aware model better captures how people answer pairwise comparison queries.

Add user-study figure 1assets/fig-user-study-1.png

Add user-study figure 2assets/fig-user-study-2.png

Reddit TL;DR summaries

Pair approximation is evaluated in a high dimensional embedding space under a fixed pool of allowable summary comparisons.

Text-summary preference-learning results 1

Add text results 1assets/fig-text-1.png

Text-summary preference-learning results 2

Add text results 2assets/fig-text-2.png

Synthetic preference learning

Continuous synthesis and approximation is evaluated in a low dimensional setting.

Add synthetic resultsassets/fig-synthetic.png

Robot controller tuning

Continuous query synthesis selects pairs of controller gains for subjective trajectory-quality comparisons.

Simulated robot controller-gain tuning results

Add robot resultsassets/fig-robot.png

04 / Takeaway

Effective preference queries should be informative about the user while avoiding comparisons that are too similar or too dissimilar to answer reliably.

05 / Citation

Cite this work

@article{nadagouda2026active,
  title   = {Active Query Synthesis for Preference Learning},
  author  = {Nadagouda, Namrata and Ahad, Nauman and
             Tucker, Maegan and Davenport, Mark A.},
  journal = {arXiv preprint arXiv:2605.26072},
  year    = {2026}
}