In this work, we characterize the outputs of individual neurons in a trained
feed-forward neural network by entropy, mutual information with the class
variable, and a class selectivity measure based on Kullback-Leibler divergence.
By cumulatively ablating neurons in the network, we connect these
information-theoretic measures to the impact their removal has on
classification performance on the test set. We observe that, looking at the
neural network as a whole, none of these measures is a good indicator for
classification performance, thus confirming recent results by Morcos et al.
However, looking at specific layers separately, both mutual information and
class selectivity are positively correlated with classification performance. We
thus conclude that it is ill-advised to compare these measures across layers,
and that different layers may be most appropriately characterized by different
We then discuss pruning neurons from neural networks to reduce computational
complexity of inference. Drawing from our results, we perform pruning based on
information-theoretic measures on a fully connected feed-forward neural network
with two hidden layers trained on MNIST dataset and compare the results to a
recently proposed pruning method. We furthermore show that the common practice
of re-training after pruning can partly be obviated by a surgery step called
bias balancing, without incurring significant performance degradation.