The study of irreducible higher-order interactions has become a core topic of study in complex systems, as they provide a formal scaffold around which to build a quantitative understanding of emergence and emergent properties. Two of the most well-developed frameworks, topological data analysis and multivariate information theory, aim to provide formal tools for identifying higher-order interactions in empirical data. Despite similar aims, however, these two approaches are built on markedly different mathematical foundations and have been developed largely in parallel - with limited interdisciplinary cross-talk between them. In this study, we present a head-to-head comparison of topological data analysis and information-theoretic approaches to describing higher-order interactions in multivariate data; with the goal of assessing the similarities, and differences, between how the frameworks define “higher-order structures.” We begin with toy examples with known topologies (spheres, toroids, planes, and knots), before turning to more complex, naturalistic data: fMRI signals collected from the human brain. We find that intrinsic, higher-order synergistic information is associated with three-dimensional cavities in an embedded point cloud: shapes such as spheres and hollow toroids are synergy-dominated, regardless of how the data is rotated. In fMRI data, we find strong correlations between synergistic information and both the number and size of three-dimensional cavities. Furthermore, we find that dimensionality reduction techniques such as PCA preferentially represent higher-order redundancies, and largely fail to preserve both higher-order information and topological structure, suggesting that common manifold-based approaches to studying high-dimensional data are systematically failing to identify important features of the data. These results point towards the possibility of developing a rich theory of higher-order interactions that spans topological and information-theoretic approaches while simultaneously highlighting the profound limitations of more conventional methods.