Analysis and convergence of weighted dimensionality reduction methods
DOI:
https://doi.org/10.17533/udea.redin.14674Keywords:
PCA, PPCA, WPCA, WRDA, dimensionality reductionAbstract
We propose to use a Fisher type discriminant objective function addressed to weighted principal component analysis (WPCA) and weighted regularized discriminant analysis (WRDA) for dimensionality reduction. Additionally, two different proofs for the convergence of the method are obtained. First one analytically, by using the completeness theorem, and second one algebraically, employing spectral decomposition. The objective function depends on two parameters U matrix being the rotation and D diagonal matrix weight of relevant features, respectively. These parameters are computed iteratively, in order to maximize the reduction. Relevant features were obtained by determining the eigenvector associated to the most weighted eigenvalue onthe maximum value in U. Performance evaluation of the reduction methods was carried out on 70 benchmark databases. Results showed that weighted reduction methods presented the best behavior, PCA and PPCA lower than 17% while WPCA and WRDA higher than 45%. Particularly, WRDA method had the best performance in the 75% of the cases compared with the others studied here.
Downloads
References
B. A. Olshausen, D. J. Field. “Emergence of simplecell receptive field properties by learning a sparse code for natural images”. Nature. Vol. 381. 1996. pp. 607- 609. DOI: https://doi.org/10.1038/381607a0
C. M. Bishop. Pattern recognition and machine learning. Ed. Springer. New York. 2006. pp. 1-738.
M. E. Tipping, C. M. Bishop. “Mixtures of probabilistic principal component analyzers”. Neural Computation. Vol. 11. 1999. pp. 443-482. DOI: https://doi.org/10.1162/089976699300016728
A. Sharma, K. K. Paliwal, G. C. Onwubolu. “Pattern Classification: An Improvement Using Combination of VQ and PCA Based Techniques”. American Journal of Applied Sciences. Vol. 2. 2005. pp. 1445-1455. DOI: https://doi.org/10.3844/ajassp.2005.1445.1455
J. C. Burges. “Geometric Methods for Feature Selection and Dimensional Reduction” in: O. Maimon, L. Rokach. (editors). Data Mining and Knowledge Discovery Handbook: A Complete Guide for Practitioners and Researchers.. Ed. Springer. New York. 2006. pp. 59-91. DOI: https://doi.org/10.1007/0-387-25465-X_4
L. K. Saul, K. Q. Weinberger, J. H. Ham, F. Sha, D. D. Lee. “Spectral methods for dimensionality reduction”. In: O Chapelle, B. Schölkopf, A. Zien. (editores). Semisupervised Learning. Ed. MIT Press. Cambridge. Massachusetts. USA. 2006. pp. 279-294. DOI: https://doi.org/10.7551/mitpress/6173.003.0022
J. Venna. Dimensionality reduction for visual exploration of similarity structures. PhD thesis. Helsinki University of Technology. Helsinki. 2007. pp. 11-32.
L. V. Maaten, E. Postma, J. V. Herik. “Dimensionality Reduction: A Comparative Review”. Elsevier Journal of Machine Learning Research. Vol. 10. 2009. pp. 1-41.
L. Wolf, A. Shashua. “Feature Selection for Unsupervised and Supervised Inference: The Emergence of Sparsity in a Weight-Based Approach”. Journal of Machine Learning Research. Vol. 6. 2005. pp. 1855-1887.
A. Blum, P. Langley. “Selection of relevant features and examples in machine learning”. Artificial Intelligence. Vol. 97. 1997. pp. 245-271. DOI: https://doi.org/10.1016/S0004-3702(97)00063-5
M. A. Turk, A. P. Pentland. “Face Recognition Using Eigenfaces”. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Maui. Hawaii. USA. Vol. 1. 1991. pp. 586-591.
G. Balcerowska, R. Siuda. “Inelastic background subtraction from a set of angle-dependent XPS spectra using PCA and polynomial approximation”. Vacuum. Vol. 54. 1999. pp.195-199. DOI: https://doi.org/10.1016/S0042-207X(98)00459-X
W. Deng, J. Hu, J. Guo, W. Cai, D. Feng. “Robust, accurate and efficient face recognition froma single training image: A uniform pursuit approach”. Pattern Recognition. Vol. 43. 2010. pp. 1748–1762. DOI: https://doi.org/10.1016/j.patcog.2009.12.004
Q. Zhao, H. Lu, D. Zhang. “A fast evolutionary pursuit algorithm based on linearly combining vectors”. Pattern Recognition. Vol. 39. 2006. pp. 310-312. DOI: https://doi.org/10.1016/j.patcog.2005.09.001
L. G. Sánchez-Giraldo, F. Martínez-Tabares, G. Castellanos-Domínguez. “Functional Feature Selection by Weighted Projections in Pathological Voice Detection”. Lecture Notes in Computer Science. Vol. 5856. 2009. pp. 329-336. DOI: https://doi.org/10.1007/978-3-642-10268-4_39
L. G. Sánchez-Giraldo, G. Castellanos-Domínguez. “Weighted feature extraction with a functional data extension”. Neurocomputing. Vol. 73. 2010. pp. 1760-1773. DOI: https://doi.org/10.1016/j.neucom.2009.12.030
D. Skocaj, A. Leonardis. “Weighted and robust incremental method for subspace learning”.Proceedings of the Ninth IEEE International Conference on Computer Vision. Vol. 2. 2003.pp. 1494-1501. DOI: https://doi.org/10.1109/ICCV.2003.1238667
M. Tipping, C. Bishop. “Probabilistic principal component analysis”. Journal of the Royal Statistical Society. Series B. Vol. 61. 1999. pp. 611-622. DOI: https://doi.org/10.1111/1467-9868.00196
J. H. Friedman. “Regularized discriminant analysis”. Journal of the American Statistical Association. Vol. 84. 1989. pp. 165-175. DOI: https://doi.org/10.1080/01621459.1989.10478752
A. R. Webb. Statistical Pattern Recognition. 2a. ed. Ed. John Willey and Sons. London. 2002. pp. 305-360. DOI: https://doi.org/10.1002/0470854774
C. Lai. W. J. Lee. M. Loog. P. Paclik. D. Tax. URL: http://ict.ewi.tudelft.nl/~davidt/occ/index.html. Consultada el 1 de Julio de 2009.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2018 Revista Facultad de Ingeniería
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Revista Facultad de Ingeniería, Universidad de Antioquia is licensed under the Creative Commons Attribution BY-NC-SA 4.0 license. https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en
You are free to:
Share — copy and redistribute the material in any medium or format
Adapt — remix, transform, and build upon the material
Under the following terms:
Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
NonCommercial — You may not use the material for commercial purposes.
ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
The material published in the journal can be distributed, copied and exhibited by third parties if the respective credits are given to the journal. No commercial benefit can be obtained and derivative works must be under the same license terms as the original work.
Twitter