Vermouth

Abstract

Diffusion models are trained to generate images, but the features they compute along the way carry a lot of information about what is in an image. Vermouth reuses those generative priors for tasks that are usually solved with discriminative models: retrieval, classification, and segmentation.

The framework keeps a pre-trained Stable Diffusion model frozen as a feature source, adds a unified head that merges its hierarchical features into one representation, and pairs it with an adapted expert that supplies the discriminative signal. One framework then covers zero-shot sketch-based image retrieval, few-shot classification, and open-vocabulary semantic segmentation.

What the pieces are

Short definitions of the components, for readers who do not work with diffusion models daily.

Stable Diffusion prior

A text-to-image latent diffusion model. While it denoises an image it builds internal features that already encode objects, parts, and layout — those features are what we borrow, instead of the generated picture.

Unified head

The diffusion U-Net produces features at several resolutions. The unified head aggregates this hierarchy into a single representation that every downstream task reads from.

Adapted expert

A pre-trained discriminative model, lightly adapted, that contributes the class-separating signal generative features are weaker at on their own.

Method

One prior, one shared head, three different views of visual perception.

Tasks it is evaluated on

Zero-shot SBIR

Sketch-based image retrieval: given a hand drawing, find photos of the same category — where the categories were never seen during training.

Few-shot classification

Learning a new class from a handful of labelled examples.

Open-vocabulary segmentation

Labelling every pixel with a class described in free-form text, rather than a fixed label set.

Results

Values transcribed from the project README.

BibTeX

Dong, S., Zhu, M., Cheng, K., Wang, N., & Gao, X. (2024). Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors. In IJCAI 2024, pp. 740–748.

@inproceedings{DBLP:conf/ijcai/DongZC0024,
  author       = {Shiyin Dong and
                  Mingrui Zhu and
                  Kun Cheng and
                  Nannan Wang and
                  Xinbo Gao},
  title        = {Bridging Generative and Discriminative Models for Unified Visual Perception
                  with Diffusion Priors},
  booktitle    = {Proceedings of the Thirty-Third International Joint Conference on
                  Artificial Intelligence, {IJCAI} 2024, Jeju, South Korea, August 3-9,
                  2024},
  pages        = {740--748},
  publisher    = {ijcai.org},
  year         = {2024},
  url          = {https://www.ijcai.org/proceedings/2024/82},
  timestamp    = {Fri, 18 Oct 2024 20:53:11 +0200},
  biburl       = {https://dblp.org/rec/conf/ijcai/DongZC0024.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}