Transient Attributes for High-Level Understanding
and Editing of Outdoor Scenes

People

Pierre-Yves Laffont
Zhile Ren
Xiaofeng Tao
Chao Qian
James Hays

Abstract

We live in a dynamic visual world where the appearance of scenes changes dramatically from hour to hour or season to season. In this work we study “transient scene attributes” – high level properties which affect scene appearance, such as “snow”, “autumn”, “dusk”, “fog”. We define 40 transient attributes and use crowdsourcing to annotate thousands of images from 101 webcams. We use this “transient attribute database” to train regressors that can predict the presence of attributes in novel images. We demonstrate a photo organization method based on predicted attributes. Finally we propose a high-level image editing method which allows a user to adjust the attributes of a scene, e.g. change a scene to be “snowy” or “sunset”. To support attribute manipulation we introduce a novel appearance transfer technique which is simple and fast yet competitive with the state-of-the-art. We show that we can convincingly modify many transient attributes in outdoor scenes.

SIGGRAPH 2014 paper

Paper preprint
(PDF, 16MB)

Supplementary document
(PDF, 2.9MB)

Supplementary video
(MP4, 140MB)

The paper can be downloaded from ACM Digital Library. We provide supplementary materials in addition to the paper:

BibTeX citation

@article {Laffont14,
    title = {Transient Attributes for High-Level Understanding and Editing of Outdoor Scenes},
    author = {Pierre-Yves Laffont and Zhile Ren and Xiaofeng Tao and Chao Qian and James Hays},
    journal = {ACM Transactions on Graphics (proceedings of SIGGRAPH)},
    volume = {33},
    number = {4},
    year = {2014}
}

SIGGRAPH slides

We presented our work at SIGGRAPH 2014 in Vancouver. We share our slides (.PPTX, 70MB) for scientific dissemination, along with our SIGGRAPH "fast-forward" video (.WMV, 18MB).

Web-based user interfaces

We show examples of user interfaces that leverage our transient attributes on the following pages:

for visualizing annotated images from two webcams of our database, in a color-coded attribute space
for exploring a photo collection and progressively get closer to the desired image by providing feedback, such as “more clouds” then “less winter”
for searching a photo collection by providing an acceptable range of attribute labels, which can be inferred from a high-level query string such as “some snow, no clouds”.

These pages have been tested with Chrome v35 (recommended), Firefox v29, and Internet Explorer v10.

Transient Attributes Database

Our dataset contains 8571 images from 101 webcams, all annotated with 40 attribute labels. We provide download links below:

An archive which contains the aligned images (downsampled to <480px high) – this is what we used for the image manipulation part of our SIGGRAPH 2014 paper
The same images prior to manual alignment (i.e., with some camera motion between frames of a particular webcam) – this is what we used for training our attribute regressors
The per-image aggregated attribute labels (a README file and the list of attributes are included)

Please contact Zhile Ren (jrenzhile@gmail.com) if you have questions about the database.

Code for our Transient Attributes regressors

We release our trained regressors that can predict transient attribute labels for an outdoor input image. This code was used to estimate the attribute labels for photo collections in the two web interfaces above.

Transient Attributes regressors code v1.2 (.TAR, 5.4GB), Matlab code tested on Linux
Testing/training splits (.TAR, 440KB) for evaluating your own regressors - requires downloading our annotated database

Funding

This research is supported by:

NSF CAREER Award 1149853 to James Hays,
the Intelligence Advanced Research Projects Activity (IARPA) via Air Force Research Laboratory,
contract FA8650-12-C-7212.

License

This dataset is licensed under a Creative Commons Attribution 4.0 International License.