Parainfia-An Infotainment System For Semi-Paralyzed Using Electroencephalogram

Abstract

With the advent of social media the variety of digital media is growing exponentially. The sole purpose of media is to introduce a sense of joy among users. Multimedia customization and recommendations depend on emotional traits of the content a user is watching. The levels of media enjoyment of a normal person are usually extracted using various surveys or feedback gathering. Enjoyment level varies from an individual to another, the presence of techniques for automatically extracting this type of information from paralyzed will make possible the advancement of media content analysis of such people. In this paper, we have made use of EEG data acquired from Neurosky Mindwave headset. The data is then transformed to frequency domain using Fourier transform and various feature extraction steps are performed. Enjoyment level is divided into four levels based on the attention of the user. Finally, attempt is made to draw a conclusion from the achieved result that this enjoyment level detection mechanism can be integrated with a web application which can help paralyzed people customize their choices of media they are interested in watching.

Keywords-Multimedia, Social media, EEG, Neurosky Mindwave, Enjoyment level, web application

Introduction

Paralysis is a disorder in which a person loses voluntary control over some parts of the body. Today around 1. 9% of the world population is suffering from some or other type of paralysis. Paralysis is broadly classified into two categories Hemiplegia and Quadriplegia. In Hemiplegia voluntary control over one half of body is lost whereas in Quadriplegia control over all the limbs is lost. All these forms of paralysis render a person with very less or no means of access to movies, videos, news or songs. Moreover, even if they get access to these, they might face difficulty in changing the content as they wish or like to be. In most of the cases, such people rely or depend on someone or the other. To make them independent and overcome difficulties of getting access to multimedia content we propose an infotainment system which can be used by a normal person or paralyzed person alike. The only limitation we need to mention (take care) is that the users should be able to perform head and eye movements. The core of our system lies in the quantitative analysis of various levels of enjoyment a person experiences while going through the media content being shown to him in our interface. Often enjoyment is related to the emotional experiences of a person. The definition of enjoyment varies from one person to another. Phrases like “I enjoyed it”, “I loved it”, “I did not enjoy” are used to express the level of joy that a person feels.

All these are relative measures of enjoyment of how a person perceives the content shown to him or watched by him. Every person has his/her own definition of things that induce a sense of joy for him. To better understand all these relative terms we need to dive deep into the working of our brain. By working of our brain we intend to focus on frequency analysis of various waves our brain is known to produce namely delta, theta, alpha, beta and gamma. Since we are dealing with emotions, the frontal lobe is the area of our concern as various studies have found out that the frontal lobe is responsible for processes related to the emotional state in the human brain. For quantitative analysis of enjoyment, we use EEG using single channel dry electrode which is a non-invasive methodology.

An electrode is placed over the frontal lobe [1] and various values in the power spectrum of a delta, theta, alpha, beta and gamma waves can be evaluated. In our system, the Neurosky EEG headset is used. The advantage of using this particular headset is it comes with a proprietary software which provides us with attention and meditation values. Various feature extraction methods are carried out on the EEG data. The processed data is then fed to the random forest classifier for obtaining decision boundaries so as to decide whether a person is enjoying the content or not. Then depending on decision boundaries if a person is not enjoying the content, he will be prompted or asked for consent to change the content. Each signal corresponds to frequency measures as well as various states of patients being like relaxing, ranging over 8-14 Hz; whereas concentration ranging over 13-30 Hz; deep sleep, from 0-4 Hz; Meditating from 4-8 Hz.

The frequency band of brain signals like alpha is 8-12 Hz, beta is 12-25 Hz, theta is 4-8 Hz, the delta is 1-4 Hz and gamma is above 25 Hz [1]. These mentioned output signals are used for decision-making process computer operations such as action triggering for mouse clicks. Our system uses voluntary eye blink detection as an event to trigger mouse click and X, Y axis of accelerometer to map the movement of the neck with the mouse. Z axis will not be useful because of limited neck movements. Zhen Liang et al. [2] characterizes the enjoyment level into 4 levels using single channel dry electrode. Extraction of time-frequency components was carried out from the obtained EEG signals using STFT. Correlation between predicted and actual enjoyment levels was established and a statistical multivariate regression model was used for prediction of enjoyment levels. Gunawana et al. [3] talks about the K nearest neighbor algorithm to classify attention into two levels strong and weak. Here the author has represented a way of early determination of lack of attention of a person who is given a task which requires high attention. Fourier transform is applied on obtained EEG data to convert the data from time domain to frequency domain. And in the process, PSD i. e. power spectral density of the signal was determined.

The attention is evaluated as a function of response time. The paper Detecting Meditation using a Dry Mono-Electrode EEG Sensor [4] concludes that attention and meditation can be analyzed better with frequency and frequency-time analysis of EEG. The author also compares conventional methods and relatively recent methods of EEG, the later one which uses dry single channel electrodes being easier to use and very cost effective. Also, a person trained for meditation can meditate effectively than a normal person. Edla et al. [5] explores the ways of building a classifier for determining whether the given EEG data represents concentration or meditation. A random forest classifier is deployed using ensemble learning approach. At every stage, a random number of trees are generated and each tree makes a decision whether the user is concentrating or not and the majority of votes is considered answer of that particular stage.

Also, various statistical ways of feature extraction such as mean, standard deviation, maximum, minimum were mentioned. Signals generated by eye blinks and mouth clenching [6] Fig. 1(a) Single Eyeblink Fig. 1(b) Double Eyeblinks Fig. 1(c) Mouth Clenching From figure 1(a), (b), (c) we can clearly distinguish between a single eye blink, double eye blink and mouth clenching. We can employ these characteristics of the signals generated for triggering particular actions in the BCI. Eye blink detection can also be carried out using face recognition [7]. Also, a tree-like structure constituting of words can be navigated using left and right eye blink thus forming sentences. But this methodology requires heavy processing and efficient algorithms to detect blinks. Bansal et al. [8] devices a way to calculate the area of interest in a web page using EEG signals. The procedure is to divide the page into various parts and monitor the attention of user at real time using Neurosky Mindwave headset and also taking into consideration the rate at which the user scrolls over the content. And hence the area of interest is a function of the user’s attention and scrolling.

Proposed Architecture

The architecture of the proposed EEG-Based BCI System for recognizing the attention level as well as other signals required for our system is shown in Fig 2. In the proposed system, the first task conducted is to collect the EEG Headset data along with the Accelerometer sensor data, then for second stage the acquired data undergo feature extraction and characterization of EEG attention for each user and finally in the signals recognition phase, users EEG signals have classified into four levels and appropriate action is triggered depending upon the enjoyment classes.

• Neurosky Mindwave EEG headset The Neurosky Mindwave mobile [9] is a single dry electrode EEG device which has the capability to capture the Brain electrical activity from the external side of the frontal lobe of humans. The device has an inbuilt Bluetooth module to transmit raw data from a brain. It has got two electrode, one is for capturing the frontal signals where the strength of the signal is maximum and another has a reference electrode which clips to the ear. Hardware of Mindwave mobile has got 2 chips one of them is responsible for processing raw EEG data obtained from brain and produces the digital format EEG power unit data which are of higher frequencies (31-40) Hz Fig 6 shows the processed values. Another chip transmits the data in a serial manner
• Hc-05 Bluetooth Module: HC‐05 module is an easy to use Bluetooth SPP (Serial Port Protocol) module, designed for transparent wireless serial connection setup. The HC-05 Bluetooth Module can be used in a Master or Slave configuration, making it a great solution for wireless communication. This serial port Bluetooth module is fully qualified Bluetooth V2. 0+EDR (Enhanced Data Rate) 3Mbps Modulation with complete 2. 4GHz radio transceiver and baseband [10].
• Arduino Leonardo: The Arduino Leonardo is a microcontroller board based on the ATmega32u4. The Leonardo differs from all preceding boards in that the ATmega32u4 has built-in USB communication, eliminating the need for a secondary processor. This allows the Leonardo to appear to a connected computer as a mouse and keyboard, in addition to a virtual (CDC) serial / COM port. B. Description of software
• Arduino Platform: Arduino project provides the Integrated Development Environment (IDE). Which is the open-source electronics platform based on easy-to-use hardware and software. Arduino board reads the input from sensor data and output the serial values or actions as per the instructions provided to the controller board and perform the controlling task.
• Python platform: As python is the high-level programming or scripting language used for writing the serial data from the Arduino board to the CSV files and also used for creating the classifier for signal classification.
• Matlab working environment [11]: Matlab software is an interactive tool for creating and analyzing data in the form of the matrix and plotting them, it also provides the feature of readymade function for creating feature extractor. It includes facilities for managing the variables in your workspace and importing or exporting data. It also includes tools for developing, managing, debugging, and profiling M-files, MATLAB applications. C. System flow: The basic flow of the proposed system is as shown in the Fig 3. Working with brain Signals for general BCI consist of five different phases. Brain activity measurement, preprocessing of signals, feature extraction, Classification of signals using classifier and final phase is controlling the BCI interface with required classified signals for a proper functioning of the system. Depending upon the certain condition on signals value, the action will be triggered as values are lower compared to a preset threshold value. The system design is shown in Fig 4. The Neurosky EEG headset is a single dry electrode capable for monitoring the attention level, mediation level as well as other brain signals. The range of attention and meditation level value produces by EEG headset is 0-100. HC-05 Bluetooth module is used to transmit the EEG headset data to the Arduino board for further processing.

Conclusion

The system developed is useful for paralyzed to watch the media content they would like to see based on their attention levels. The enjoyment level of media is categorized into 4 levels according to which the contents can be changed. We have achieved an accuracy of around 80% for our system based on the correlation between user input and the EEG data acquired. The accuracy of the proposed system can be increased using various machine learning techniques. Also incorporation of a virtually optimized keyboard will further add to the ease of usage for paralyzed people.

Future Work

The future work will focus on collecting EEG data of paralyzed people and working on that particular dataset. In order to increase accuracy of the system we would build a random forest classifier [5] [15] which uses ensemble learning approach. Other methods, such as the wavelet transform and feature extraction could be applied to analyze the signal for the simple feature. More experiments with the group of paralyzed patients can be done along with correlation and validating the result. This research paper talks about the feedback system form semi-paralysis patients and triggering the action as per his/her brain signals but for the research, we took the data of Normal persons and tried to acquire attention and other brains artifact along with generating their feedback for media content. For actual paralysis patients, it can be done in the near future and the EEG raw data can be classified with classifiers to get the better result.