Control Over Employee Actions And Analysis Of Customer Demand

Abstract

The article presents the method recognition of sales consul- tants on the basis of a neural network to determine the posture and clarifying algorithms, also methods of monitoring the behavior of the seller-consultant and analysis of its interaction with the buyer. A brief review of the analog systems is given. The description of the proposed method is presented, the obtained results and ways of improvement are shown.

Introduction

The modern era is characterized by a transition from the economy of producers to the economy of consumers. In the conditions of toughening competition in the sphere of trade and rendering services, client-oriented services acquire special importance. The main problem of introducing such services is the human factor, control of which is problematic due to the lack of ready-made software products. Ensuring the proper quality of service delivery becomes the main objective of the market strategy for business development. To improve the quality of service, it is proposed to develop and implement a software product to monitor the activities of consultant salesmen through the analysis of their work with the use of equipment for video fixing. The work of the software product is based on the algorithm Pose Estimator, which allows you to determine the position of a person, clarifying algorithms, auxiliary neural networks that help to identify the seller-consultant, and also determine the quality of services provided to them.

The Proposed Methodology

To solve this problem, we suggest using a cascade of two neural networks, as well as a number of methods and algorithms:

• Fast convolutional neural network FastPoseEstimator, trained on mobilenet architecture.
• Algorithm of stabilization of “key” points, allowing determine those points of the body, which could not recognize the neural network.
• Neural network for determine the behavior of the sales assistant and store employee.
• Algorithms for tracking people in the frame.
• Library “Face Recognition” to identify buyers and sellers.

The first stage, the use of the neural network ”Fast pose estimator” The main task of the neural network is to establish a persons pose through a nonparametric representation called the Part Affinity Fields (PAFs) by developers, to further determine the location of the sellers uniform of the consultant (branded T-shirt, cap, etc.). The main advantage of the neural network is the high speed of the work, 1 frame in 1-2 seconds on GPU and 5-8 seconds on CPU. The main drawback is the de- crease in the quality of work when compared with the classic version of Pose Estimtor. Input data for the algorithm “PoseEstimator” is a graphic image of the sales consultant, on the output - an image with the selected parts of the human body.

The second phase, a stabilization algorithm ”key” points of the a person body The skeleton is built from the starting point located under the throat, then the eyes, shoulders and pelvis are recognized. From the shoulders completed the hands, from the points of the pelvis of the foot, from the eyes ears. Thus, the key point for the construction is the point under the neck, then we call it the Control over employee actions and analysis of customer demand III initial one. It is necessary to determine the dominant color of the t-shirt seller. To improve the quality of work, the following algorithm is proposed for finding the missing ”key” points, based on information about the structure of the human structure. Consider one of the options for finding the missing “key” point of the body. If only one shoulder of a person is recognized, the second shoulder can be completed by depositing an equal segment from the projection of the starting point on the normal of the point of the found shoulder to the side of the not found shoulder equal to the distance from the projection of the point to the normal of the point of the found shoulder to the point of the found shoulder. Thus, using the information about the structure of the human structure it is possible to determine the approximate location of the missing points of the shoulders, pelvis points. If there is only a starting point, it is proposed to deter- mine some area of a fixed size on the humans body by depositing the area below the key point by N pixels.

The third stage, the determination of the dominant colour in the uniform section. The main task of this stage is to establish a dominant colour in the area of the uniform of a person to determine it in the group of sales consultants. Within the framework of this algorithm, an image from the “Pose Estimator” with the tops of the human body parts is input. On the basis of which there is a selection of the necessary clothing of a person. To implement the definition of dominant colour in an established area, there are several methods: determining the ratio of a pixel to a given set of colours and clustering by the k-means method. In the first method, the image is converted to HSV colour space, after which all pixels of the image are analyzed and based on the Hue, Saturation, Value data, the colour is set. The idea of the k-means method is to minimize the total quadratic deviation of the cluster points from the centers. At the first stage, you select points (three-dimensional RGB space) and determine whether each point belongs to this or that center. Then at each stage, the centers are redefined until a single center is found.

The fourth stage, training the appearance of sales consultants. To determine only the colour of a persons clothing is not enough to classify him as a sales consultant group. It is necessary to take into account the conditions of the difference in the illu- mination of the room at different times of the day, as well as the likelihood that there may be clarified and dark areas in the room. Thus, the recognized colour of the shape can vary. To solve this problem, it is necessary to teach the system all possible colors that can be “read” from the clothing of the seller-consultant. The operator of the software product at the beginning of work with the program should start the training mode, with by means of which prevailing color uniforms randomly moving around the store sales assistant. On the basis of the prevailing colors of the clothes is formed an “average dom- inant color”, which is the central point in the formation of the color range for referring a person to a sales consultant.

Analysis of the quality of communication with the seller by buyers

After the classification of people in the frame to groups of buyers and sellers, the program automatically controls the quality of services rendered by sellers- consultants. To assess the quality of the seller’s communication with the client, it is proposed to use a set of algorithms and a cascade of neural networks. The main task of the algorithms is to determine the location of the seller next to the buyer, as well as control over the personalization of the employees appeal to the buyer. In order to exercise control over the personalization of the employees appeal to the buyer, it is proposed to determine the field of view of people in the frame. The seller must always interact with the buyer, be in the review area of the buyer and talk about the benefits of the goods. Thus, to accomplish the task, it is necessary to build a field of view based on the location of the eyes and ears obtained from Pose Estimator, after which the sector of intersection of these Control over employee actions and analysis of customer demand V areas and the angle of interaction between the buyer and the seller should be determined. If the areas do not intersect or the interaction angle is less than the angle set by the operator of the software product, it is considered that the seller is near the buyer, but does not interact with it.

To determine the interest and emotionality of the store employee when inter- acting with the buyer, it is proposed to use neural networks that define the above parameters by the store employee behavior: hand speeds, moving around the store to accompany the buyer, movements to show goods, etc. In addition, it is planned to use a separate neural network, which, by facial expression and behavior, will determine customer satisfaction with the services provided by the sellers. The conclusion about the work sellers of the store is made on the basis of all the factors that determine the interaction with the buyer, after which at the end of the month an estimate is calculated for each store employee.

Interchamber Tracking

To control the movement of employees and customers in the premises, it is proposed to use a number of algorithms and methods for inter-camera tracking. The specified module will allow to lead a person between frames, as well as to recognize a person if he has left the control zone of one camera and entered the control zone of another camera. Tracking is based on the data returned from the PoseEstimator algorithm in the first stage and the data returned from the stabilization algorithms of the key points returned in the second stage. Thus, we do not use third-party neural networks, do not develop new ones, but use the data that was obtained earlier. This helps us to significantly accelerate the speed of the program. It is proposed to divide the body area obtained from the key points stabilization algorithm into five equal parts, set the dominant color for each part and form the color range for each part (these algorithms are described in stages 3 and 4, they are used again). In addition, PoseEstimator returns the coordinates of points and other parts of the body. But since the legs and arms do not have a solid contour and the area cannot be formed on the basis of two points (left upper and lower right corners of the area), it is proposed to form a small area around a point of N pixels. We used this approach in the stabilization algorithm of key points at the second stage. For the formed areas, the dominant color is also set and the color range is calculated. After that, an array with the found color ranges is assigned a key that identifies the person in the frame. At the stage of tracking the area are the above-mentioned algorithm, one now checks the occurrence of the dominant color of the area in the previously found color range. In the case of such a border, the color range is refined to improve the quality of work. If there is more than 50% in the color area, the found person is recognized by the color and the key generated at the first recognition stage is assigned to it.

Identification of Employees and Customers

The task of identifying sellers and buyers is quite difficult in its decision due to the low quality of images from video cameras. We offer a number of modules and hardware to solve the problem. It is proposed at the entrance to the store to install a camera with a resolution of at least 720px, with which customers will be identified by face using the python library ”Face Recognition”. This library is easy to connect and use, it includes modules for searching a face in a photo and recognizing a face from an existing base. Thus, after the first call, the client will get into the database and will be Control over employee actions and analysis of customer demand VII identified with each new visit to the store. And the control of its movement will be provided by the interchamber tracking module. For sellers, it is proposed to equip a “starting” point in the hall (for example, when leaving the service room), where the 720px camera will also be located, which allows identifying an employee. Further movement of the employee will be monitored by the inter-camera tracking module.

Overview of Analogues

It should be noted that the finished software products that allow to solve the problem discussed in this article are not present. Similar software products perform only part of tasks. The simplest example of intelligent video surveillance is motion detection. One detector can replace several video surveillance operators. And in the 2000s, the first video analytics systems began to appear, capable of recognizing objects and events in the frame. Most of the solutions work with face recognition technolo- gies. Solutions in this area include Apple, Facebook, Google, Intel, Microsoft and other technology giants. Surveillance systems with automatic passenger identi- fication are installed in 22 US airports. In Australia, they are developing a bio- metric system of face recognition and fingerprinting within a program designed to automate passport and customs control. An interesting project of NTech-Lab Company showed a system capable of real-time recognition of sex, age and emotions using the image from a video camera. The system is able to evaluate the audience’s reaction in real time, so you can identify the emotions that visitors experience during presentations or broadcasts of advertising messages. All NTechLab projects are built on self-learning neural networks. In our system, we do not yet use data on a person’s face. We plan to process this information at the next stages of the project development. In other systems, the object tracking function is used - tracking. The operation of the tracking modules is related to the operation of the motion detector. To construct the trajectories of the movement, a sequential analysis of each frame is carried out, on which moving objects are present. In the general case, several moving objects can be present in one frame, so the program needs not only to construct trajectories, but also to distinguish objects and their movements.

The simplest implementation of tracking considers two frames and builds trajecto- ries along them. First, the movements on the current and previous frame are marked, then, by analyzing the speed, the direction of movement of objects, and also their sizes, the probabilities of the transition of objects from one point of the trajectory of the previous frame to another point of the current are calculated. The most probable movements are assigned to each object and added to the trajectory. Objects in the frame can move in different ways: their trajectories may intersect, they can disappear and arise again. To improve the accuracy of tracking, some manufacturers use the technology of sequence analysis and con- tinuous post-processing of the results obtained. The program builds graphs - it analyzes the transitions of objects from one state to another. In order to under- stand which object the movement corresponds to, the speeds and directions of motion, position, color characteristics are also analyzed. As a result, a set of the most probable displacements of the object is formed, forming a trajectory. We have planned to use this approach in our system.

Another analogue of our system - GPS-trackers. These systems work based on the definition of geolocation. To implement this solution, each employee must be equipped with a separate GPS tracker, the data from which will be sent to the server at some interval. However, this solution has a number of drawbacks:

1. The solution is not cost-effective, since it is necessary to purchase GPS trackers for all personnel.
2. We can’t exclude the situation in which the seller can give his GPS-tracker to a partner to deceive the system.
3. Such a solution is not universal.

When identifying sales consultants through the camera, it is possible to expand the functionality, determine the level and time of interaction of the seller with the buyer, and much more. Also, analogs include systems for counting the number of visitors on a video stream. These systems also have a number of shortcomings, the main one of which is the impossibility of identifying sales consultants and the quality of their services.

Conclusion

In order to improve the technological process of detecting the seller’s consultant, it is possible to develop additional functionality. To more accurately determine the seller’s consultant, it is possible to analyze several elements of the uniform at once (for example, a yellow T-shirt and black Control over employee actions and analysis of customer demand IX pants). In addition, it is possible to search for the company logo on the uniform, the location of which will allow us to identify with confidence the person as the seller-consultant. Another factor that allows to detect the seller, can serve as a definition of be- havior, characteristic for the seller-consultant. To solve this problem, you will need to create another neural network. It is also possible to identify additional factors that determine the quality of the sellers work in the store, such as the presentation goods to customers who have not stopped at the rack with the goods, but passing by. In addition, it is planned to implement the ability to install “dead zones” in the program. This function allows you to set the non-human type for those objects that the Fast Pose Estimator neural network defines as people. Modules for customer identification and tracking allow you to delve into the marketing segment. We expect to identify the necessary goods and goods that are popular based on past visits to the store by the buyer. For example, when buying a short when visiting the store in the new, we will recommend the employee to advertise a T-shirt to the shorts already purchased. Thus, the developed software will make it possible to qualitatively improve the work of the sales assistant and, as a result, will lead to an improvement in the customer focus of the business.

References

1. M.D.Khorunsjiy The method of quantitative estimation of color differences in the perception of digital images. Scientific and technical herald of information technologies, mechanics and optics, 2008
2. O. Ulyanova Psychological features of network marketing sales consultants. 2013, Retrieved from https://cyberleninka.ru/article/n/psihologicheskie-osobennosti-prodavtsov-konsultantov-setevogo-marketinga
3. Cao Z., Simon T., Wei S.-E., Sheikh Y. Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. 2016, Retrieved from http://arxiv.org/abs/1611.08050
4. E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, B. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. 2016, Retrieved from https://arxiv.org/pdf/1605.03170.pdf
5. Y. Osipove, D. Application of cluster analysis by k-means method for classification of scientific texts. 2017, Retrievedfrom https://cyberleninka.ru/article/n/primenenie-klasternogo-analiza-metodom-k-srednih-dlya-klassifikatsii-tekstov-nauchnoy-napravlennosti.