RESEARCH OF THE JOINT USE OF MATHEMATICAL MORPHOLOGY AND CONVOLUTIONAL NEURAL NETWORKS FOR THE SOLUTION OF THE PRICE TAG RECOGNITION PROBLEM

Authors

DOI:

https://doi.org/10.20998/2079-0023.2020.01.05

Keywords:

image recognition, object detection, morphology, normalization of geometric transformations, convolutional neural network, recurrent neural networks, neural network training, retail chains, price tags, barcode, software application

Abstract

The work is devoted to solving the problem of recognizing images containing symbolic type information, barcodes, logos and other signs. Example of such images are price tags in shopping centers, flyers, invitations, tickets to various events. The information on such images is of a different type and its recognition requires various approaches. The work addressed the recognition of price tags in retail chains. The accuracy of object detection has a significant role for their recognition. A significant role for recognition of image elements has the accuracy of their detection. The combination of classical methods of image analysis and the neural network approach were investigated. Particular attention was paid to the study in the comparative aspect of the object detection by morphological method and by processing a convolutional neural network. Studies have shown that morphology yields a significantly lower detection quality than a neural network, but is several times faster than it. Since speed has a great importance for the implementation of algorithms on mobile devices, post-processing with additional filters and normalization of geometric distortions were added to the morphology, that significantly improved the accuracy of detection and subsequent recognition. Based on the research results of detection and recognition of barcodes and symbolic information presented on price tags, conclusions are drawn about choosing approaches and technologies for solving these problems, an algorithm has been developed and, on its basis, an application for recognizing price tags of various retail chains. A mobile version of the application has been developed also. The algorithm is constructed in such a way that the first step is the detection of the supporting element, for example, a barcode, then other price tag elements are detected relative to this supporting element. The barcode is detected with the methods based on mathematical morphology and mathematical statistics which were used to improve the accuracy of the algorithm, or convolutional neural networks. To detect prices and product names, the convolutional neural network CRAFT is exploited, which can process low-quality images. The found name and price are normalized to eliminate geometric distortions and transferred to the Tesseract library for recognition. This library works with many languages and is in the public access. The price tag recognition application was created in C ++ using the OpenCV, ZXing, Libtorch, Tesseract libraries.

Author Biographies

Andrii Kovtunenko, Kharkiv National University of Radio Electronics

Kharkiv National University of Radio Electronics, bachelor, SYTOSS R&D Engineer; Kharkiv, Ukraine

Olena Yakovleva, Kharkiv National University of Radio Electronics

Candidate of Technical Sciences, Docent, Kharkiv National University of Radio Electronics, Associate Professor of Informatics Department, Kharkiv, Ukraine

Valentyn Liubchenko, Kharkiv National University of Radio Electronics

Candidate of Technical Sciences, Docent, Kharkiv National University of Radio Electronics, Associate Professor of Informatics Department, SYTOSS R&D Senior Developer, Kharkiv, Ukraine

Olha Yanholenko, National Technical University «Kharkiv Polytechnic Institute»

Candidate of Technical Sciences, National Technical University «Kharkiv Polytechnic Institute», Associate Professor of Software Engineering and Management Information Technology Department, Kharkiv, Ukraine

References

Serra J. Image Analysis, Mathematical Morphology. Academic Press, 1982. 621 p.

Putjatin E. P., Jakovleva E. V., Ljubchenko V. A. Razlozhenie matricy centroaffinnogo preobrazovanija dlja normalizacii izobrazhenij [Centroaffine transformation matrix decomposition for image normalization]. Radiojelektronika i informatika [Radioelectronics and informatics]. 1996, no. 4 (05), pp. 91–94.

Artificial Intelligence Development Services. Available at: https://www.sytoss.com/data-science-and-neural-network (accessed 10.06.2020).

Image Polygonal Annotation with Python (polygon, rectangle, circle, line, point and image-level flag annotation). Available at: https://github.com/wkentaro/labelme (accessed 15.04.2020).

Common Objects in Context. Available at: http://cocodataset.org/#home (accessed 01.04.2020).

Zharkov A., Zagaynov I. Universal Barcode Detector via Semantic Segmentation. Available at: https://arxiv.org/abs/1906.06281 (accessed 10.02.2020).

Ioffe S., Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Available at: https://arxiv.org/abs/1502.03167 (accessed 01.03.2020).

Zhou X. et. al. EAST: an efficient and accurate scene text detector. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2017. pp. 5551–5560.

Simonyan K., Zisserman A. Very deep convolutional networks for large-scale image recognition. Available at: https://arxiv.org/abs/1409.1556 (accessed 15.05.2020).

Ronneberger O., Fischer P., Brox T. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. Springer, Cham. 2015. pp. 234–241.

ZXing (“Zebra Crossing”) barcode scanning library for Java, Android. Available at: https://github.com/zxing/zxing (accessed 10.02.2020).

Hochreiter S., Schmidhuber J. Long short-term memory. Neural computation. 1997, vol. 9, no. 8. pp. 1735–1780.

Understanding LSTM Networks. Available at: https://colah.github.io/posts/2015-08-Understanding-LSTMs (accessed 10.03.2020).

Tan M., Le Q. V. Efficientnet: Rethinking model scaling for convolutional neural networks. Available at: https://arxiv.org/abs/1905.11946 (accessed 20.04.2020).

Downloads

How to Cite

Kovtunenko, A., Yakovleva, O., Liubchenko, V., & Yanholenko, O. (2020). RESEARCH OF THE JOINT USE OF MATHEMATICAL MORPHOLOGY AND CONVOLUTIONAL NEURAL NETWORKS FOR THE SOLUTION OF THE PRICE TAG RECOGNITION PROBLEM. Bulletin of National Technical University "KhPI". Series: System Analysis, Control and Information Technologies, (1 (3), 24–31. https://doi.org/10.20998/2079-0023.2020.01.05

Issue

No. 1 (3) (2020)

Section

SYSTEM ANALYSIS AND DECISION-MAKING THEORY

License

Copyright (c) 2020 Bulletin of National Technical University "KhPI". Series: System Analysis, Control and Information Technologies

Authors who publish with this journal agree to the following terms:

  • Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
  • Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
  • Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).