An intuitive way to search for images is to use queries composed of an example image and a complementary text. While the first provides rich and implicit context for the search, the latter explicitly calls for new traits, or specifies how some elements of the example image should be changed to retrieve the desired target image. This is the problem of image search with free-form text modifiers, which consists in ranking a collection of images by relevance with respect to a bi-modal query.
Publications home
ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity
at
| Tenth International Conference on Learning Representations (ICLR), virtual event, 25-29 April, 2022 |
Overview
Current approaches typically combine the features of each of the two elements of the query into a single representation, which can then be compared to the ones of the potential target image [1,2,3].
Our work aims at shedding new light on the task by looking at it through the prism of two familiar and related frameworks: text-to-image and image-to-image retrieval. Taking inspiration from them, we exploit the specific relation of the query element with targeted images and derive lightweight text-guided attention mechanisms which enable to mediate between the two complementary modalities. We validate our approach on several retrieval benchmarks [4,5,6], querying with images and their associated free-form text modifiers. Our method obtains state-of-the-art results without resorting to side information, multi-level features, heavy pre-training nor large architectures as in previous works.
Our method: 2 independent modules
- Our Explicit Matching (EM) module measures the compatibility of potential target images with the textual requirements.
- Our Implicit Similarity (IS) module considers the relevance of the target images with respect to the properties of the reference image implied by the textual modifier.
- AIS, AEM: text-guided attention mechanisms (MLP) that select the visual cues which should be emphasized during matching.
- Tr: linear transformation (fully-connected layer) to project in the visual space.
- Both modules are trained jointly with the batch-based classification loss.
We provide some Grad-CAM [7] visualisations of image parts contributing the most to the EM and IS scores.
The EM module addresses the image parts that are the most related to the caption (the slits on the side).
The IS module attends to visual cues that are shared between the reference and the target images (sole shape & heel).
Ablative results
We evaluate ARTEMIS on 3 datasets.
- Visual search and cross-modal search results reveal that these datasets can be either text-centric (as FashionIQ [4]) or image-centric (as Shoes [5,8] and CIRR [6]).
- An ablation of our modules shows that adding our attention mechanisms helps our modules to improve upon their respective baselines, cross-modal & visual search
- Combining our two modules brings the best performance with respect to using either one of them alone, for both image-centric and text-centric datasets. This shows our modules’ complementarity.
The resulting ARTEMIS model outperforms its attention-free version, considered here as “late fusion”, and which combines scores of visual and cross-modal search. This is especially the case for the CIRR dataset. This shows the importance of the attention mechanisms in making our two modules compatible.
Take-home messages
- ARTEMIS combines cross-modal and visual search scoring strategies, making them compatible for Image search with text modifiers.
- ARTEMIS models all pairwise interactions including with the target image, without large extra-cost.
- ARTEMIS is versatile: it works with different visual and textual encoders, on different domains.
References
- Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. CVPR 2019.
- Yanbei Chen, Shaogang Gong, and Loris Bazzani. Image search with text feedback by visiolinguistic attention learning. CVPR 2020.
- Seungmin Lee, Dongwan Kim, and Bohyung Han. Cosmo: Content-style modulation for image retrieval with text feedback. CVPR 2021.
- Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. Fashion IQ: A new dataset towards retrieving images by natural language feedback. CVPR 2021.
- Xiaoxiao Guo, Hui Wu, Yu Cheng, Steven Rennie, Gerald Tesauro, and Rogerio Feris. Dialog-based interactive image retrieval. NeurIPS 2018.
- Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-trained vision-and-language models. ICCV 2021.
- Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. ICCV 2017.
- Tamara L. Berg, Alexander C. Berg, Jonathan Shih. Automatic Attribute Discovery and Characterization from Noisy Web Data. ECCV 2010.
@inproceedings{delmas2022artemis,
title={ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity},
author={Delmas, Ginger and Rezende, Rafael S and Csurka, Gabriela and Larlus, Diane},
booktitle={International Conference on Learning Representations},
year={2022}
}
To make robots autonomous in real-world everyday spaces, they should be able to learn from their interactions within these spaces, how to best execute tasks specified by non-expert users in a safe and reliable way. To do so requires sequential decision-making skills that combine machine learning, adaptive planning and control in uncertain environments as well as solving hard combinatorial optimisation problems. Our research combines expertise in reinforcement learning, computer vision, robotic control, sim2real transfer, large multimodal foundation models and neural combinatorial optimisation to build AI-based architectures and algorithms to improve robot autonomy and robustness when completing everyday complex tasks in constantly changing environments.
VISION
The research we conduct on expressive visual representations is applicable to visual search, object detection, image classification and the automatic extraction of 3D human poses and shapes that can be used for human behavior understanding and prediction, human-robot interaction or even avatar animation. We also extract 3D information from images that can be used for intelligent robot navigation, augmented reality and the 3D reconstruction of objects, buildings or even entire cities.
Our work covers the spectrum from unsupervised to supervised approaches, and from very deep architectures to very compact ones. We’re excited about the promise of big data to bring big performance gains to our algorithms but also passionate about the challenge of working in data-scarce and low-power scenarios.
Furthermore, we believe that a modern computer vision system needs to be able to continuously adapt itself to its environment and to improve itself via lifelong learning. Our driving goal is to use our research to deliver embodied intelligence to our users in robotics, autonomous driving, via phone cameras and any other visual means to reach people wherever they may be.
This web site uses cookies for the site search, to display videos and for aggregate site analytics.
Learn more about these cookies in our privacy notice.
Cookie settings
You may choose which kind of cookies you allow when visiting this website. Click on "Save cookie settings" to apply your choice.
This content is currently blocked. To view the content please either 'Accept social media cookies' or 'Accept all cookies'.
For more information on cookies see our privacy notice.