Journal of Biomedical Engineering and Biosciences (JBEB)

ISSN: 2564-4998

1. Introduction

In recent years, research comparing emotions and autonomic nervous system (ANS) indicators and establishing correlations between them has become increasingly active. In the healthcare field, research is dedicated to identifying emotions through ANS indicators [1]. In this domain, two models, the Discrete Emotional Model (DEM) and Affective Dimensional Model (ADM), are employed. DEM discretizes emotions, such as happiness, sadness, anger, etc. ADM captures emotions through Valence and Arousal. The advantages of these methods lie in evaluating and analyzing emotions through a unified approach, facilitating further progress in existing research, and enabling emotion analysis in diverse contexts. However, research analyzing the temporal relationship between the autonomic nervous system and emotions at a high resolution is still relatively scarce. Heart Rate Variability (HRV) as a Key Measure, derived from ECG recordings, serves as a significant physiological measure reflecting the regulatory capacity of the cardiac autonomic nervous system [2]. Kwang Ho Choi et al. [3] used the International Affective Picture System (IAPS) [4] to investigate the effectiveness of HRV as a tool for evaluating emotions. Emotional assessment employed the Self-Assessment Manikin (SAM). They argue that it is only when visual stimuli elicit high levels of emotion that the assessment based on HRV might be applicable, their measurements are post-hoc, but with a high temporal resolution. However, the correlation between the autonomic nervous system and emotions, the order of reactions, and mechanisms remain unclear. Additionally, in the aforementioned studies, the timing of emotional experiences and the temporal changes in autonomic nervous indicators were not considered. Therefore, compared to previous studies where stimuli were administered before measuring emotional indices, we have developed a system capable of concurrently analyzing cECG, respiration, and emotional indices with high temporal resolution. This enables the exploration of the relationship between emotional indices and autonomic nervous system indices. In the experiment, respiratory measurements were taken simultaneously with the recording of facial expressions and ECG. As breathing is closely related to the autonomic nervous system, focusing on it allows us to understand its impact on changes in ANS indicators and the sequential relationship between emotions and the autonomic nervous system. In addition, this study focuses on the emotion of laughter, which is relatively easy to trigger. From a health perspective, laughter is a feasible health practice at the individual level. Literature [5] reports the contribution of pleasurable emotions, such as happiness, to longevity. Therefore, understanding the neural mechanisms of laughter associated with pleasurable emotions and their relationship with the autonomic nervous system is considered beneficial for society. We have developed a system for visualizing the response of emotions triggered in facial expressions, ECG, and respiratory data, aiming to clarify the relationship between the autonomic nervous system, emotions, and respiration.

2. Materials and Methods

The system consists of four main components. Firstly, there are the system circuits and testing equipment. Secondly, there is the setup of the environment for testing emotion data, cECG signals, and respiratory signals. Finally, there is the preprocessing of the acquired data and the ultimate data analysis. As illustrated in Figure 1, we will provide a more detailed explanation in this chapter.

2. 1. cECG Measurement Equipment and Respiratory Measurement Tape

Figure 1 shows the cECG measurement circuit and detection belt in this work. The circuit board includes the following four parts, A:the front-end detection circuit. B: Pulse generator, C: AD8232 Fast-recovering circuit. D: Bluetooth transmission circuit. E: Amplifier gain adjustment. The capacitive coupling electrode includes two parts, F:active guard side and G: measurement side. This circuit is equipped with a fast recovery function to reduce the possibility of QRS wave loss caused by oversaturation caused by motion artifacts during measurement [6].

Figure 1 also delineates the respiratory belt employed in this study. The section highlighted with a purple circle features stretchable conductive rubber, while the remainder comprises non-stretchable materials. Previous investigations [7] have documented a positive correlation between chest range of motion and ventilation during deep breathing. Consequently, utilizing this respiratory measurement belt allows for a relatively predictive estimation of lung ventilation volume. During inhalation, the chest area expands, causing the conductive rubber to stretch, thereby increasing resistance and decreasing the measured partial pressure in series with it. Conversely, during exhalation, the chest area contracts, leading to the contraction of the conductive rubber and an increase in the measured partial pressure. Therefore, by analyzing voltage changes, we can discern the breathing patterns of the human body. Additionally, considering the potential impact of body movement on amplitude fluctuations, this article only supports measurements in a stationary sitting state.

2. 2. Facial Expression Analysis Module

The Facial Action Coding System (FACS) theory, developed by Ekman & Friesen, enables the description of visible facial movements in humans [8]. It is a system that allows for the mechanical identification of human expressions based on the movements of specific muscle groups called Action Units (AU). Each AU represents the movement of a specific part of the face, such as the eyebrows, eyes, or mouth, and is assigned a unique number. A single expression is often composed of combinations of multiple AUs, allowing for a detailed analysis of the complexity of human expressions. In this study, we utilized this theory to analyze facial expressions.

Py-Feat (Python Facial Expression Analysis Toolbox) [9] is a facial information parsing library used in Python, based on the Facial Action Coding System (FACS). As mentioned earlier, FACS is a system that encodes subtle facial movements to represent specific emotions. Py-Feat utilizes this system to detect emotions from facial images and videos. It features facial landmark detection, classification, and emotion estimation, employing machine learning models to analyze various facial expression data and identify emotions such as joy, sadness, surprise, etc. Widely used in fields like psychology, marketing, user experience research, Py-Feat contributes to understanding human emotions. Py-Feat includes seven categories: anger, disgust, fear, happiness, sadness, surprise, and neutral, with all cumulative values summing to 1.0. Figure 2 presents the results of analyzing sample images using Py-Feat, happiness has a value close to 1.0. According to reference [10], instantaneous heart rate (IHR) typically decreases during periods of negative emotions and increases during periods of positive emotions. In this study, we utilized this library to primarily analyze the subjects' happy expressions from facial images.

2. 3. Analysis Methods

We performed heart rate variability analysis on the recorded ECG, calculating RRI, SDNN, and LF/HF. Specifically, RRI, SDNN, LF/HF were computed every second, while facial expression values were calculated approximately every 0.2 seconds, ensuring each indicator could be tracked at least once per second. Furthermore, we referred to the cECG, instantaneous heart rate (IHR), and respiratory data to observe the response of respiratory and IHR data when a smiling expression occurred. This approach facilitated the visualization and assessment of IHR, smiling expressions, and respiratory data on a shared timeline.

Figure 1 depicts a schematic representation of the measurement setup. A monitor was positioned in front of the subject, accompanied by a network camera mounted above it. To induce laughter, we utilized videos with a duration of approximately 3 to 5 minutes, including sketches, comedian conversations, anime, and amusing animal behavior. Simultaneously, the participant wore a cECG measurement belt and a respiratory measurement belt for synchronous detection of cECG, respiratory signals, and facial images. The cECG was sampled at 250 sps, while facial images were captured at 5 fps. Following data acquisition, we applied FACS theory to analyze facial expressions in the images, obtaining happiness values. Data processing is divided into two main stages: preprocessing and data analysis. Preprocessing requires separating the cECG signal and respiratory signal from the received signal for HRV analysis. The SciPy library's find_peaks function [11] identified QRS waves, and we calculated the time difference between adjacent QRS waves. SDNN and LF/HF utilized RRI from a 1-minute ECG for computation. The ECG range was shifted by 1 second increments from the beginning to the end of the measurement, allowing for the calculation of SDNN and LF/HF based on data from 30 seconds before and after each moment. This approach facilitated the comparison of happiness values at each moment with SDNN and LF/HF values. Data analysis requires analyzing facial expression and aligning the time axes of cECG signal, respiratory signal, and emotion analysis data.

3. Results and Discussion

The results of the preliminary experiments are illustrated in Figure 3. According to Figure 3 (a), In the time range of 75 to 95 seconds, a Happiness index below 0.2 is observed, followed by a surge to values exceeding 0.8 between 95 and 105 seconds. Additionally, LF/HF starts to rise from 75 seconds and increases to approximately 5.0 over 40 seconds, indicating sympathetic nervous system activation (when parasympathetic activity is suppressed or sympathetic activity is stimulated, values of 4.0 or higher are considered as indicative thresholds [12]). While previous reports have discussed the activation of the sympathetic nervous system through pleasurable emotions such as laughter [13], in this experiment, the Happiness index calculated from facial expressions momentarily surged from below 0.1 to around 0.8, while LF/HF, serving as an autonomic nervous system indicator, increased relatively slowly over approximately 40 seconds. Given that the LF/HF value is an average of the 30 seconds of ECG data before and after the time point, we consider the gradual rise in LF/HF to be reasonable. On the other hand, in the Figure 3 (b), the Happiness index exhibits multiple discontinuous surges, each of short duration, without a clear upward trend in LF/HF. Based on this result, it can be inferred that shorter durations of elevated Happiness do not lead to significant changes in LF/HF and sympathetic nervous system activation.

Furthermore, According to Figure 3 (b), although LF/HF does not show an upward trend, it appears to influence the fluctuation pattern of RRI (approximately 225-250 seconds and 300 seconds). As mentioned earlier, we will focus on IHR , which represents a more immediate autonomic nervous system response, and compare it with the Happiness level for further discussion.

(a) Long term continuous laughter

(b) Short and repeated laughter

3. 1. Relationship between Happiness Index and IHR during Emotion-Inducing Video Stimuli

Figure 4 illustrates the graph of Figure 3 (a), where cECG, IHR, and Happiness are plotted on the same timeline. The horizontal line represented in burgundy indicates the Resting Heart Rate (RHR) during rest, serving as a reference baseline for IHR variations.

(a) Respiratory sinus arrhythmia (RSA) without laughter

(b) Changes in IHR during laughter occurrence and the index of happiness detected at different levels of laughter

During the time period shown in Figure 4 (a), no laughter was observed (Happiness index: 0.00), and periodic increases and decreases in IHR were observed. As mentioned in section 3.2, respiratory measurements confirmed that the IHR increases during inhalation and decreases shortly thereafter. These fluctuations are therefore considered to be Respiratory Sinus Arrhythmia (RSA). RSA refers to the phenomenon where heart rate increases during inhalation and decreases during exhalation, which is a common characteristic in healthy individuals. The respiratory cycle is about 3 to 5 seconds, and the rise and fall of IHR also occur within this cycle. Additionally, fluctuations in the baseline of the ECG were observed, and these fluctuations are similar to the respiratory cycle. This is because the capacitive electrodes used for ECG measurement are placed on the chest, and the periodic changes in pressure on the electrodes are caused by breathing.

In Figure 4 (b), we have added the results of facial expression analysis at different time points as a reference for the happiness index. During the period from 76 to 95 seconds, the Happiness index generally remains below 0.2, and RSA can be confirmed. Between 96 to 106 seconds and 114 to 116 seconds, the Happiness index rises above 0.4 (reaching above 0.8 between 102 to 105 seconds), and during this period, RSA cannot be confirmed. Furthermore, with the increase in the Happiness index, IHR takes approximately 3 seconds to rise by about 15 beats per minute (bpm) from the RHR value. Subsequently, with the decrease in Happiness, IHR takes around 3 seconds to drop by approximately 15 bpm. From these results, it can be observed that when the Happiness index reaches a certain value, RSA becomes less pronounced and affects heart rate variability. This result corresponds one-to-one with the Facial Expression Analysis results of Py-Feat, confirming the feasibility of the experiment.

Figure 5 depicts a graph where the cECG, IHR, Happiness, and RHR of Figure 3 (b) are plotted on the same timeline. During the periods of 120 to 126 seconds, 140 to 152 seconds, and 160 to 170 seconds, the Happiness index remains mostly below 0.1, and RSA can be clearly confirmed. On the other hand, during the intervals marked as (1), (2), and (3), there is a slight increase in the Happiness index, and the characteristics of RSA (the rise and fall of IHR) are subtle. Additionally, during periods (2) to (3), changes in the baseline of the ECG occur compared to other periods. Therefore, during this interval, respiratory may become shallower with the occurrence of laughter. During the interval marked as (4), the Happiness index reaches a maximum of around 0.6 and lasts for approximately 1 second. In the interval marked as (5), the Happiness index peaks at 1.0 and lasts for more than 4 seconds. Judging from the Happiness index and its occurrence frequency, it can be said that period (5) evokes stronger laughter than period (4). Moreover, the increase in IHR is also greater during this period.

(a) cECG, IHR, happiness, and RHR with brief multiple laughter on the same timeline.

(b) Changes in happiness index and IHR during fake smiles

As shown in Figure 5 (b), for comparison, participants were instructed to fake a smile while their ECG and facial images were measured, mirroring the procedure of the previous experiment. Participants commenced pretending to smile at 100 seconds and continued for 30 seconds. Between 60 and 100 seconds and after 100 seconds, there were no significant changes in IHR, despite the increase in the Happiness index caused by the fake smile. RSA remained observable. The distinct differences in RSA and IHR induced by genuine laughter and fake smiling through visual stimuli were clearly demonstrated. The significant presence of RSA indicates an increase in activity of the vagus nerve (a component of the parasympathetic nervous system) [14]. Additionally, as mentioned in literature [15], compared to fake smiling, spontaneous laughter is associated with more pronounced activation of the sympathetic nervous system, consistent with the findings of this study. Furthermore, we must consider another potential factor. During the occurrence of happiness, changes in human respiration may lead to the increase or decrease of IHR, thus rendering the features of RSA subtle. In the next section, we will utilize visualization of the respiratory state during the rise of happiness through respiratory waveform to further validate this.

3. 2. Experiment Triggering Emotions with Video and Respiratory Data Integration

Firstly, Figure 6 presents the graphs of the electrocardiogram, respiration, and IHR during a 20-second rest period (for explanatory purposes, the amplitude of the respiration data has been enlarged). The respiratory data is plotted in orange, and in this example, the period is approximately 4 to 5 seconds. Additionally, when the conductive rubber is stretched, its resistance increases, causing the voltage across the voltage divider resistor in the circuit to decrease. The areas circled in gray represent a relative increase in waveform amplitude, indicating the chest is undergoing exhalation. The black arrow points to a downward convex shape in the respiratory waveform, which can be interpreted as the transition period from inhalation to exhalation. By simultaneously monitoring the respiratory waveform and IHR, it can be observed that the IHR (red) temporarily increases by approximately 4 bpm at this point. Furthermore, as described in section 3.1, based on the respiratory data, this temporary increase or decrease in IHR can be understood as RSA.

As illustrated in Figure 7, in addition to recording the ECG and facial images of the subjects while watching videos that induce laughter, we also collected respiratory data at 250 sps. During the period from 60 to 96 seconds, Happiness is relatively low (<0.1), and compared to the resting state, there is almost no disturbance in the respiratory waveform. On the other hand, focusing on the IHR, the characteristics of RSA confirmed in Figure 6 have become subtle. From 109 to 119 seconds, the IHR increases by about 5 bpm from the RHR. It is observed that the Happiness index steadily increases during periods (1), (2), (3), and (4), while the IHR increases by approximately 8 to 10 beats bpm throughout the entire duration compared to the RHR. The data indicates a lengthening of the breathing time. Particularly during periods (3) and (4), each breath takes about 7 seconds (breathing waveform from 147 to 154 seconds and 162 to 169 seconds). From 200 to 236 seconds, the duration of Happiness is longer and the values are higher compared to the preceding periods. The IHR increases by approximately 10 bpm overall compared to RHR, reaching a peak of 23 bpm around 233 seconds (the maximum IHR in this experiment). Based on the amplitude variations in the breathing waveform from 220 to 234 seconds, it can be inferred that longer breathing cycles, with irregular rhythms, are induced compared to the periods of rest and shorter durations of Happiness. This is because during intense laughter, there is typically a series of short, shallow exhalations following a deep inhalation [16].

On the other hand, we conducted an inhalation test. As shown in Figure 8, even without visual stimuli, inhaling alone can cause an increase in IHR. After inhaling around 30 seconds, the IHR temporarily increases by 10 bpm near 33 seconds, followed by an immediate decrease. Therefore, the phenomenon of the maximum IHR increasing by 23 bpm during the period from 220 to 234 seconds may also be related to the deep inhalation at that time.

(a)

3. 3. Experiment Triggering Emotions with Video and Holding Breath

In section 3.2, it was mentioned that the increase in IHR might be related to deep breathing. In Figure 7, the IHR increased by about 10 bpm from the RHR between 149 and 151 seconds. Compared to the resting state and the previous period, the interval between breaths became longer, but the amplitude of the respiratory waveform hardly changed. Therefore, the increase in IHR might be influenced by both breathing and emotional arousal. According to reference [17], ceasing to breathe leads to a decrease in heart rate. Therefore, we assume that the relationship between the increase in IHR and emotion is stronger when happiness increases. After taking a deep breath and holding it, when emotions are aroused (when happiness increases), there may be three possible changes in IHR: the first IHR will briefly increase while happiness increases, the second IHR will decrease but not be lower than RHR, and the third IHR will not decrease.

To further explore which relationship is more significant, between inhalation-induced increases in IHR and those induced by emotional arousal, we conducted an experiment where participants consciously held their breath while watching videos that elicited laughter.As depicted in Figure 9 (a) (b), participants inhaled at 40 seconds and 113 seconds respectively, holding their breath for 30 seconds each time. The difference lies in the absence of laughter-inducing stimuli in the former and its presence in the latter. In the absence of visual stimuli, at 40 seconds, after inhaling and holding their breath, IHR temporarily increased by approximately 20 bpm from RHR, then decreased after about 3 seconds. During the 46 to 60-second interval, IHR was similar to RHR, while in the 60 to 80-second interval, IHR decreased by a maximum of approximately 9 bpm compared to RHR. On the other hand, after inhaling and holding their breath at 113 seconds, IHR temporarily increased by approximately 20 bpm, then decreased within 3 seconds. However, during the 125 to 133-second period that triggers laughter 10 seconds later, the Happiness index increased (emotional arousal occurred), and IHR increased accordingly. At this point, IHR exceeded RHR by approximately 10 bpm. This is consistent with the previously assumed first scenario.

(a) Inhale and hold breath without trigger laughter
(b) Inhale and hold breath, trigger laughter 10 seconds later
(c) Inhale and hold breath, always trigger laughter

In addition, the experiment in Figure 9 (c) is similar to the previous one in Figure 9 (b) (deep inhalation and breath holding at 40 seconds), but the difference is that after deep inhalation and breath holding, the stimulation that causes laughter does not stop. It can be observed that IHR gradually decreases from 42 to 49-second period, but it is not lower than RHR. Then, from 49 to 52-second period and from56 to 59-second period, the IHR rises twice to 80 bpm. This is consistent with the previously assumed second scenario. Finally, after 66 seconds, although the happiness index was weak, a certain level of happiness index could still be observed, and IHR did not continue to decline below RHR, which is different from the situation of after 46 seconds in Figure 9 (a) and 144 to 149-second period in Figure 9 (b). This is consistent with the previously assumed third scenario.

Therefore, it can be concluded that although breathing affects IHR variation, the relationship between emotional arousal during an increase in the Happiness index and IHR increase is more pronounced.

4. Discussion

The RSA effect refers to the increase in heart rate during inhalation leading to increased pulmonary blood flow and the decrease in heart rate during exhalation resulting in reduced pulmonary blood flow. The physiological reason why organisms have RSA is to improve gas exchange efficiency [18]. In other words, RSA is believed to regulate pulmonary blood flow based on lung capacity, thus conserving respiratory and circulatory energy during gas exchange. Additionally, it is known that RSA is more pronounced during reduced inhalation or decreased breathing frequency. Therefore, in our experiment, we observed an increase in IHR and a gradual weakening of RSA as the Happiness index rose. Regarding the interpretation of this result, firstly, with the rise in the Happiness index, emotional arousal (sympathetic nervous system activation) leads to an increase in heart rate (IHR). Physiologically, sympathetic nervous system activation causes the body to become tense, increasing cerebral blood flow, resulting in an elevated heart rate. Furthermore, the disrupted rhythm and reduction in amplitude of RSA during the increase in the Happiness index may be associated with irregular breathing. In an excited state, energy-saving mechanisms are less dominant, thus weakening the characteristics of RSA aimed at enhancing energy efficiency during gas exchange. Additionally, when the effect of increased heart rate during inhalation was eliminated in the breath-holding experiment, the IHR exhibited three characteristics in relation to the intensity of the Happiness index: first, as the Happiness index increased, the IHR also increased; second, after inhalation, the IHR did not continue to decrease below the RHR; and third, when the Happiness index was weak but did not disappear, the IHR tended to stabilize and no longer continued to decrease. Therefore, we believe there is a strong association between the rise in the Happiness index (sympathetic nervous system activation) and the increase in heart rate.

5. Conclusion

In this study, we developed a system capable of high temporal resolution data measurement. The cECG and respiratory data were sampled at 250 sps, while facial images were captured at 5 fps. This system was designed to investigate the relationship between physiological indicators obtained from cECG and facial expressions. During the experiment, participants watched videos designed to induce laughter, while their facial images, cECG, and respiratory signals were simultaneously recorded. Based on the collected data, we discussed the relationship between the autonomic nervous system and facial expressions. Additionally, control experiments were conducted to explore the impact of breathing on the autonomic nervous system. Initially, we observed that visual stimuli inducing an increase in the Happiness index also led to an increase in the IHR, accompanied by disruptions in RSA rhythm. Furthermore, inhalation also contributed to the increase in IHR. To further elucidate the potentially stronger association between the increase in Happiness index and the activation of the sympathetic nervous system leading to IHR elevation, we consciously stopped breathing during the viewing of video stimuli and conducted measurements. The results revealed that during breath-holding, when video stimuli caused an increase in the Happiness index, the IHR also increased simultaneously, exceeding the resting heart rate by approximately 10 bpm. Therefore, this reveals a positive correlation between the increase in Happiness index (sympathetic nervous system activation) and the rise in IHR.

Acknowledgements

Financial support from China Scholarships Council (No.202208050103). We would like to express our gratitude to MAXELL Corporation for their support of this research.

References