The challenge of translating a user's information need into an effective search query is fundamental in modern cross-modal retrieval systems, which allow users to search image collections with natural language but often lack interactive refinement tools. Our paper addresses the gap by investigating Cross-modal Query Suggestion, the task of dynamically generating targeted query alternatives that verbalize the semantic content of visually coherent image groups within an initial retrieval result set. Building upon our prior work establishing a dedicated benchmark, this extended study systematically compares three distinct methodological paradigms for generating these suggestions: i) latent-conditioned generation, which operates on visual embeddings; ii) caption-based generation, relying on textual abstractions of the result set; and iii) image-conditioned generation, which directly utilizes representative images from the result set as input for Large Multimodal Models (LMMs). We further expand the evaluation framework by introducing new metrics for semantic fidelity against human annotations and computational efficiency. While our evaluation assumes a controlled setup with disjoint image clusters, we validate its robustness against automated partitioning and noise. This work establishes a comprehensive evaluation framework to map the structural trade-offs of these paradigms, and validates the effectiveness of the proposed protocol with a tailored user preference study. Our experiments demonstrate that approaches utilizing direct visual grounding via LMMs produce more image-specific suggestions, Large Language Models (LLMs) reach the best tradeoff between the assessed properties, while latent-conditioned methods have the lowest computational cost. These findings offer critical insights for developing dynamic, visually informed interactive search systems. The dataset, the methods implementations and the code to replicate our experiments are available at: github.com/Ruggero1912/CroQS-benchmark
Query Suggestion for Text-to-Image Retrieval
Giacomo Pacini;Fabio Carrara;Nicola Messina;Nicola Tonellotto;Giuseppe Amato;Fabrizio Falchi
2026
Abstract
The challenge of translating a user's information need into an effective search query is fundamental in modern cross-modal retrieval systems, which allow users to search image collections with natural language but often lack interactive refinement tools. Our paper addresses the gap by investigating Cross-modal Query Suggestion, the task of dynamically generating targeted query alternatives that verbalize the semantic content of visually coherent image groups within an initial retrieval result set. Building upon our prior work establishing a dedicated benchmark, this extended study systematically compares three distinct methodological paradigms for generating these suggestions: i) latent-conditioned generation, which operates on visual embeddings; ii) caption-based generation, relying on textual abstractions of the result set; and iii) image-conditioned generation, which directly utilizes representative images from the result set as input for Large Multimodal Models (LMMs). We further expand the evaluation framework by introducing new metrics for semantic fidelity against human annotations and computational efficiency. While our evaluation assumes a controlled setup with disjoint image clusters, we validate its robustness against automated partitioning and noise. This work establishes a comprehensive evaluation framework to map the structural trade-offs of these paradigms, and validates the effectiveness of the proposed protocol with a tailored user preference study. Our experiments demonstrate that approaches utilizing direct visual grounding via LMMs produce more image-specific suggestions, Large Language Models (LLMs) reach the best tradeoff between the assessed properties, while latent-conditioned methods have the lowest computational cost. These findings offer critical insights for developing dynamic, visually informed interactive search systems. The dataset, the methods implementations and the code to replicate our experiments are available at: github.com/Ruggero1912/CroQS-benchmarkI documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
