Hi all,
Our next Information Theory and Applications seminar for the academic year
will take place on Monday, November 18 at 10:00, in room A500.
The speaker this week is Shachar Shayovitz, who will tell us about the role
of information theory in active learning. See title and abstract below.
See you there,
Or, Oron, Yuval and Alex
---------------------------------------------
Title: Information Theoretic Active Learning
Abstract: Active learning is a machine learning paradigm aimed at improving
model efficiency by strategically selecting the most informative data
points for labeling, thereby reducing reliance on large annotated datasets.
This is particularly relevant in privacy-sensitive applications where user
data cannot be easily or fully annotated. Unlike traditional active
learning approaches, which assume that the training and test sets share the
same distribution, our research introduces novel methodologies that
function effectively without this assumption, both in stochastic
(probabilistic) and individual (non-probabilistic) settings.
In the first part of our research, we address active learning within the
stochastic setting, where data is governed by a probability distribution
from a known hypothesis class. We propose a novel information-theoretic
criterion for active learning, grounded in the Redundancy-Capacity theorem
from universal source coding. This criterion naturally balances the
exploration-exploitation trade-off in feature selection and offers a more
robust alternative to heuristic-based methods. Both theoretical analysis
and empirical validation demonstrate that our approach outperforms
conventional methods in various tasks.
The second part of our research focuses on active learning in the
individual setting, where no probabilistic relationship between the
training and test sets is assumed. Drawing on universal source coding
principles, we introduce a new criterion for selecting data points that
minimizes the min-max regret on the test set. For tasks such as binary
classification and linear regression, our criterion coincides with
established active learning strategies, offering a unified framework for
general hypothesis classes.
However, applying active learning to deep learning models presents a unique
challenge due to the complexity and high-dimensionality of modern neural
networks. In deep learning scenarios, existing active learning techniques
often struggle with computational efficiency and accuracy, as neural
networks require a large amount of labeled data to generalize effectively.
To address these challenges, we extended our criterion by incorporating
variational inference to approximate the posterior distribution of model
parameters. This approach allowed us to develop a highly efficient,
low-complexity algorithm tailored for deep learning applications. Our
algorithm not only simplifies the task of selecting informative data points
but also enhances the performance of neural networks in scenarios where
labeled data is scarce. Experimental results demonstrate that our approach
outperforms state-of-the-art active learning methods for deep learning. In
particular, we achieved a reduction of 15.4%, 11%, and 35.1% in the
required labeled data for CIFAR10, EMNIST, and MNIST datasets,
respectively, even in the presence of out-of-distribution data. These
results highlight the robustness and practical applicability of our method
in reducing the labeling effort for deep learning tasks, making it a
promising solution for real-world applications where large-scale data
annotation is prohibitive.
Joint work with Meir Feder
