An Information-Geometric View of Learning in High Dimensions

Abstract: We consider the problem of data feature selection prior to inference task specification, which is central to high-dimensional learning. Introducing natural notions of universality for such problems, we show a local equivalence among them. Our analysis is naturally expressed via information geometry, and represents a conceptually and practically useful learning methodology. The development reveals the key roles of the singular value decomposition, Hirschfeld-Gebelein-Renyi maximal correlation, canonical correlation and principle component analyses, Tishby’s information bottleneck, Wyner’s common information, Ky Fan k-norms, and Brieman and Friedman’s alternating conditional expectation algorithm. As we’ll discuss, this framework provides a basis for understanding and optimizing aspects of learning systems, including neural network architectures, matrix factorization methods for collaborative filtering, rank-constrained multivariate linear regression, and semi-supervised learning, among others.

Joint work with Shao-Lun Huang, Anuran Makur, and Lizhong Zheng

Biography: Gregory W. Wornell received the B.A.Sc. degree (with honors) from the University of British Columbia, Canada, and the S.M. and Ph.D. degrees from the Massachusetts Institute of Technology, all in Electrical Engineering and Computer Science, in 1985, 1987 and 1991, respectively.
His research interests and publications span the areas of signal processing, information theory, statistical inference, digital communication, and information security, and include architectures for sensing, learning, computing, communication, and storage; systems for computational imaging, vision, and perception; aspects of computational biology and neuroscience; and the design of wireless networks. He has been involved in the Information Theory and Signal Processing societies of the IEEE in a variety of capacities, and maintains a number of close industrial relationships and activities. He has won a number of awards for both his research and teaching, including the IEEE Leon K. Kirchmayer Graduate Teaching Award, and is a Fellow of the IEEE.