VectorQuantization using Information Theoretic Concepts 
Tue LehnSchiĝler, Anant Hegde, Deniz Erdogmus, Jose C. Principe

Abstract  The process of representing a large data set with a smaller number of vectors in the best possible way, also known as vector quantization, has been intensively studied in the recent years. Very efficient algorithms like the Kohonen Self Organizing Map (SOM) and the Linde Buzo Gray (LBG) algorithm have been devised. In this paper a physical approach to the problem is taken, and it is shown that by considering the processing elements as points moving in a potential field an algorithm equally efficient as the before mentioned can be derived. Unlike SOM and LBG this algorithm has a clear physical interpretation and relies on minimization of a well defined costfunction.
It is also shown how the potential field approach can be linked to information theory by use of the Parzen density estimator. In the light of information theory it becomes clear that minimizing the free energy of the system is in fact equivalent to minimizing a divergence measure between the distribution of the data and the distribution of the processing element, hence, the algorithm can be seen as a density matching method. 
Keywords  Information particles, Information theoretic learning, Selforganizing map, VectorQuantization 
Type  Journal paper [With referee] 
Journal  Natural Computing 
Year  2005 Month January Vol. 4 pp. 3951 
Publisher  Kluwer Academic Publishers 
ISBN / ISSN  15677818 
Electronic version(s)  [pdf] 
Publication link  http://dx.doi.org/10.1007/s1104700496198 
BibTeX data  [bibtex] 
IMM Group(s)  Intelligent Signal Processing 