PersonalQuery- Benchmark

Personalized retrieval usually asks what a user wants. Preferences, interaction histories, and query intent all shape the answer. PQB asks a narrower, harder question: when the need is fixed, does the wording alone move the ranking?

Existing benchmarks rarely make this test possible. Personalized datasets vary the need itself; natural query logs lack stable identities and long-term expression histories; and intent-matched query pairs are almost never available. As a result, a system’s sensitivity to how users write is rarely separated from its sensitivity to what users want.

When users express the same retrieval need differently, how robust are retrieval systems to personalized expression differences?

Central question · Section 1

PQB isolates expression as an independent variable. It models two complementary dimensions from each user’s real review history — how they organize a sentence, and which writing errors they characteristically make — then constructs queries that hold the product, its core attributes, the five core retrieval terms, and their frequencies fixed.

Existing personalized information retrieval methods mainly model differences in user needs through user preferences and query intent, while paying little attention to how different users express the same retrieval need. Real query data typically lack stable user identities, long-term expression histories, and intent-matched counterparts, which impedes systematic evaluation of the impact of personalized expression differences on retrieval outcomes. To this end, we propose PersonalQuery-Benchmark (PQB). PQB models personalized syntactic expression and personalized writing errors from users’ historical texts, constructs personalized syntactic expression queries and personalized writing-error queries, and validates benchmark quality through LLM assessment and independent human evaluation. PQB covers three Amazon product domains with 6,516 query instances, and is evaluated on five retrieval models representing lexical, learned sparse, dense bi-encoder, late-interaction, and hybrid architectures, plus two LLM-based ranking models. Results show that even when the information need is held fixed, personalized expression differences induce measurable ranking variation; different retrieval models exhibit clearly different sensitivities to syntactic variation and writing-error variation, while higher overall retrieval effectiveness does not necessarily correspond to stronger robustness against expression variation. PQB provides a new benchmark and evaluation framework for systematically studying personalized expression robustness. Code and dataset are available at github.com/cpxjazz123/PersonalQuery-Benchmark.

PQB samples target products from three Amazon domains and links each to users with enough review history to model expression. The product side fixes the retrieval target; the user side supplies the syntax and the error habits.

Each benchmark row contains a target product a, a user u, and a query q_u,a. When a valid writing-error variant survives generation, the row also carries the paired query q̃_u,a. Across the three domains this yields 3,654 syntactic queries and 2,862 writing-error queries.

Every PQB example starts from a real product and a real user. Generation is constrained so that the retrieval target stays fixed; only the expression is allowed to vary.

From review history to a paired query

The pipeline separates product-side semantics from user-side expression evidence. Candidate queries are generated under product attributes, filtered for lexical and semantic validity, then matched to users through their fitted distributions.

Five first-stage retrievers and two LLM rerankers are evaluated on the same paired queries. All figures below are equal-weight averages over Baby Products, Musical Instruments, and Video Games; per-domain tables are in the paper.

• 01

  Every system moves. Flip@1 ranges from 26.57% for SPLADE to 52.19% for ColBERTv2, even though the target product, five core terms, and term frequencies are identical.

• 02

  The most stable system is not the strongest retriever. SPLADE has the lowest mean rank shift (3.10%); ColBERTv2 has the highest disagreement at depth (34.46% at K=20).

• 03

  Specialized reranking is not a cure. Qwen3-Reranker-8B has a larger mean reciprocal-rank shift (6.63%) than general-purpose Llama-3.1-8B (6.33%).

• 01

  Every system degrades. All seven retrievers show positive Deg@K at every depth and positive MRR degradation; local character changes are enough to reduce retrieval effectiveness.

• 02

  Robustness is architecture-specific. BGE-M3 Hybrid is least affected (4.92% at K=20, 3.24% MRR_deg). BM25 loses the most hit rate at depth (12.33%); Qwen3-Reranker-8B loses the most reciprocal rank (6.64%).

• 03

  Deeper cutoffs do not recover the loss. For every system, Deg@20 exceeds Deg@1: affected targets are pushed down, not merely moved just outside the top ranks.

DeepSeek-V4-Pro assesses the benchmark automatically; five trained annotators independently label 360 stratified queries. Annotators see neither the generation condition nor the model’s judgment, and a third annotator adjudicates disagreements.

• 01

  Queries remain language-like. Human raters give 4.21/5 for naturalness and 4.16/5 for syntactic quality; the LLM rates them 4.38 and 4.31.

• 02

  The personal signal is identifiable. Decisive syntactic attribution reaches 66.3% for humans against a 50% random baseline, with only 8.7% no-preference judgments.

• 03

  Error profiles are faithful. 84.6% of human judgments pass the consistency check strictly, and 94.8% are acceptable. Agreement is substantial: α = 0.69–0.77 and κ = 0.64–0.71.

If PQB is useful for your work, please cite the manuscript. The benchmark construction code and generated queries are available in the project repository.

@misc{liang2026pqb,
  title  = {PersonalQuery-Benchmark: Evaluating Retrieval
            Robustness to Personalized User Expression Differences},
  author = {Liang, Wenyu and Yin, Hongzhi and Wu, Tongtong and
            Liu, Fan and Dai, Kexin and Wang, Weiqing},
  year   = {2026},
  note   = {Manuscript submitted to ACM Transactions on
            Information Systems (TOIS)},
  url    = {https://github.com/cpxjazz123/PersonalQuery-Benchmark}
}