Multi-modal data collections, such as corpora of paired images and text
snippets, require analysis methods beyond single-view component and topic
models. For continuous observations the current dominant approach is based on
extensions of canonical correlation analysis, factorizing the variation into
components shared by the different modalities and those private to each of
them. For count data, multiple variants of topic models attempting to tie the
modalities together have been presented. All of these, however, lack the
ability to learn components private to one modality, and consequently will try
to force dependencies even between minimally correlating modalities. In this
work we combine the two approaches by presenting a novel HDP-based topic model
that automatically learns both shared and private topics. The model is shown to
be especially useful for querying the contents of one domain given samples of
the other.