Conveying Emotions through Shape-changing to Children with and without Visual Impairment

Touch is the first human sense as we start feeling our touch inside our mother’s uterus [22, 25]. In social interactions, we use touch to express our emotions towards others [62]. People display different emotions by the way they touch (e.g., pat, push, scratch) [25], the gesture’s pressure, the body’s location contact [26], and their culture [13, 14]. Our responses to touch are hard to ignore, so it plays a crucial role in our interpretation of others’ intentions [25]. and our well-being [1, 22, 70]. For example, affective touch, such as holding hands, hugging, or patting on the back, can be a passive intervention to emotional and intellectual development [22] and can lower our anxiety and stress, helping our emotional self-regulation and support [22].

Although touch can smoothly express different emotions in social interactions, tactile information is less explored than visual and audio information [26, 27]. In human-computer interaction, various forms and materials can display emotional communication [33, 34, 38, 71]. They can use different tactile information such as vibration [45, 59, 69], temperature [66], pressure [56], 3D objects [43, 44], and shape-changing [21, 29, 32]. For example, short and rapid vibrations relate with higher arousal levels [45, 59, 69], warm stimuli represented more pleasant emotions than cold [66], pressurized squeeze relates to unpleasant and aroused emotions as opposed to finger touch that conveys pleasant and relaxed emotions [56]. And sharper objects are associated with high arousal levels while rounded shapes elicit positive valence [21, 43, 44].

Robots’ skin is also an interactive surface that can act as a tactile sensing platform to perceive human emotions [15, 62], as an expressive medium using its artificial skin to convey different emotions [29, 30, 31, 32], or as a comforting agent to study the influence of touch and hugging in human-robot interactions [1, 28, 64, 68, 70]. Soft materials have the potential to be used in robot skin; similar to human skin, they have no predefined format and can be adapted to different robots and different shape-change forms [30]. These materials can cover the sensors, creating more robust and less expensive robots while increasing their contact zone expressiveness, pleasantness, and sensitiveness [62].

Adding shape-changing communication to social robots’ skin can enrich the design space of a robot’s expressive spectrum, silently informing the interaction through visual and tactile/haptic feedback [30]. Results with adults showed that a shape-changing robot, with spikes and goosebumps, could express emotions, in which texture-shape relates to valence, while shape-changing frequency influences arousal [29]. Spikes mapped with anger, while goosebumps with pleasure and excitement [29]. Other shapes can convey other emotions; wrinkles-inspired shape represents relaxed and nervous emotions under resting and vacuumed states; tentacle-like shapes potentially tell liveliness; adhesive suction cups express attachment; and skin pores indicate interaction exposure, etc. [30].

The power of touch in human social interaction suggests that it can be a socially expressive medium when interacting with technology, namely robots. However, tactile information in robotics is mainly limited to how people manipulate and touch the robot [1, 63], not how its skin elicits emotions. The present work expands this line of research by investigating how new shape-changing skins evoke emotions and how children perceive them.

Sighted and people with blindness can equally identify materials and their features using only touch [4, 65]; however, sighted increase their performance whenever they also see the material [4]. Also, age influences users’ material-mapping skills, regardless of their visual capacity, having 33% accuracy at age five and reaching 69% at thirteen and 86.5% in adults [65].

Since birth, the user’s visual ability influences tactile exploration and mapping strategies [4]. While children with blindness progressively learn to process haptic information directly, increasing their haptic-mapping skills with training and age [65]. Age-matched sighted children have similar skills; however, they convert haptic information into a visual image and semantic information, even when they do not see the objects [65]. Another factor that increases children’s haptic-mapping skills, regardless of their visual ability, is the context. In a storytelling activity, children who know the story context, title, and characters during book exploration recognize more haptic elements in the storybook [65].

The capacity of sighted children and children with blindness to translate haptic information into semantic and object features suggests that shape-changing materials have the potential to map other concepts. However, it is still untapped how other ideas, such as emotions, can be expressed using shape-changing materials and if the emotional-mapping performance and strategies are similar among children with blindness, low vision, and sighted.

Emotion recognition, in human interactions, combines visual and auditory signals [17, 55]. When there is a sensory impairment, the remaining senses can partly compensate for the impaired sense [17, 19, 36, 55]. To enrich their emotion recognition ability, people with visual impairment can use speech information (e.g., tone, tempo, volume, silence moments, and voice energy) and body touch whenever they are familiar and close enough to touch their partner [55]. Those compensatory strategies may allow them to recognize the emotional state of the conversational partner. Still, they may not be enough to overcome the subtle emotional changes of facial expressions (e.g., smile) [55]. Additionally, people with visual impairment may have difficulty perceiving and expressing emotions due to their visual acuity [41]. Especially children with blindness must rely entirely on other senses to recognize others’ emotions without having a reference to their sighted partner’s emotional behaviors. At the same time, children with low vision can use their existing visual capacity to perceive and mimic others’ emotions, which may help them develop their social-emotional competencies [41].

Previous research on human-computer interaction explored different modalities to develop social-emotional functions of people with visual impairment. For example, a vibrotactile belt informed the participant with VI of the affective state of their conversational partners based on facial expressions [6, 8, 9]. Another study suggested an actuator-mediated "body touch" platform to mimic emotions or intentions from conversation partners without actual body contact [55].

Additionally, sound can elicit emotions in an accessible but more intrusive way. Hear beacons can enact digital emoji emotions [11], and an audio movie description positively enhances presence and emotion elicitation to the experience [23, 24]. Although music is universally used to elicit emotions [18, 39, 67], recent studies showed that the emotional variables (valence, arousal, intensity, and preference) associated with music differed significantly between adults with and without VI [52]. For sadness, adults with VI associated lower valence, higher arousal, higher intensity, and preference than the sighted group. sighted adults associate higher valence, arousal, and intensity for happiness compared to their peers. For anger and fear, only arousal and intensity significantly differed between groups and lower in groups with VI.

In storytelling activities, multi-sensory platforms explored how children with visual impairment, individually [12] and in mixed-visual ability groups [2, 31], perceive and express emotions using tangible, audio, and lights modalities. Those platforms used the story audio, tangibles (e.g., a calm sheep, Rose-a happy robot, and a spiky-angry robot), and emotional character sounds (e.g., laughing, crying). Children with and without visual impairment could play, perceive, and adapt the story to the emotions of the different characters.

Various assistive systems and modalities help people with and without visual impairment to perceive emotions in interactions; however, it is still untapped how children can use them. Our study presents a first step to exploring touch as a non-intrusive emotional medium for children with and without visual impairment.

We used ten tactile stimuli based on previous work [29, 30], as shown in Fig. 1. The stimuli were bio-inspired and designed to express internal states: tentacles (expressing liveliness), goosebumps (excitement or fear), wrinkles (nervousness), concaves (sadness), and spikes (representing anger). Each stimulus had one of the five shapes unit, i.e. tentacles, goosebumps, wrinkles, concaves, and spikes, and could be expressed in static or dynamic mode, resulting in the ten stimuli: Tentacles_S, Tentacles_D, Goosebumps_S, Goosebumps_D, Wrinkles_S, Wrinkles_D, Concaves_S, Concaves_D and Spikes_S, Spikes_D (where _S means static and _D means dynamic). Shape units are fabricated with soft silicone using 3D-printed molds and can be pneumatically controlled to change their surface properties. As the manufacturing process can change the physical characteristics of the shapes assembled, we decided to replicate the previous fabrication process [29, 30]. The authors shared with us their preexistent 3D models. And we made 3 different shape skins: Spikes, Goosebumps, and Concaves shared the same skin as the previous study [29]; Tentacles and Wrinkles shapes units used two different skins [30]. In the shared skin, there were small hard tips. Spikes had them inflated, while in Goosebumps and Concaves, tips were vacuumed underneath the skin. We used a FlowIO [61] device with a maximum power of 255 pwm for inflation and vacuum motors to control dynamic shape movements. The device allowed the control of five channels to perform inflation and vacuum. The shape-changing frequency measures the number of shape rises per minute (rpm). The design of the shape movements and their correspondent emotional mapping was based on an iterative process with one teenager with blindness, two educators from inclusive schools, and four psychologists. Through the iterative process, we converged on the following movement properties for each of the shapes unit: Tentacles, Wrinkles and Splikes in dynamic mode had 20rpm, Goosebumps and Concaves had 40rpm. For Tentacles, Wrinkles, and Spikes, the movement behavior repeatedly went from neutral to inflation, followed by a vacuum. For Goosebumps, the repeated behavior went from neutral to inflation; and Concaves went from neutral to vacuum.

In the supplementary files, we included videos for each tactile-stimulus [48].

Our participants were children in an age range still learning to identify emotions [20]. To help them in the mapping process, we adopted a strategy suggested by the school’s psychologists and educators: providing context to each tactile stimuli-mapping task. This context-driven activity helps participants relate to the feelings and physical sensations. It facilitates the mapping process while keeping children engaged in a stimuli-mapping activity [65].

Four school psychologists and researchers co-created the stories, inspired by children’s books and school activities [7, 40, 46, 49, 53, 57]. We built ten characters, and each character had five possible one-sentence stories, mapping to one of the five emotions: anger, calm, happy, sad, or fear. One example of a character is Peter that is in his living room and could be: 1) sad because he misses his grandmother who lives far away; 2) happy watching television; 3) anger because his mother ordered him to clean his bedroom; 4) calm because he was listening to some relaxing music; 5) scared because there was a party of his parents’ friends and he is afraid of talking to people he does not know well; or 6) none of those. All the characters and stories/emotions were presented randomly to each child. All stories are available in the supplementary files [48].

The study had three independent variables in a mixed-design approach. Shape unit and movement were combined as a within-subjects factor in ten tactile stimuli that each child interacted with: Tentacles_D and Tentacles_S, Wrinkles_D and Wrinkles_S, Goosebumps_D and Goosebumps_S, Concaves_D and Concaves_S, Spikes_D and Spikes_S. Shapes skins were hidden in a box (Figure 3 b) to evaluate and compare the tactile-emotional mapping of children with and without VI. To compare identical stimuli, we hide the shapes inside a box, testing haptic-only stimuli and avoiding visual cue interference. All Children touched each of the ten tactile stimuli for 25 seconds without seeing them. The order of all stimuli was randomly assigned. The third independent variable was children’s visual acuity with two between-subjects levels, sighted (S) and visually impaired (VI).

From the 500 stimuli evaluations, 97% of the stimuli were attributed to an emotional label. In thirteen cases, children select the no_emotion option (NoEmotion=13, N_VI=5, N_S=6); A second emotional label only occurred in four instances (SecondEmotion=4, N_VI=5, N_S=6) and only one child, C32LV, explicitly said they feared the shapes after experiencing the Spikes_D stimuli.

After converting the chosen emotions into values of valence and arousal, as described in Section 3.3.4, we analyzed each of these continuous variables with three-way mixed ANOVA. We tested for main effects and interaction effects between our independent variables: shape unit and movement as the two within-subjects factors, and children’s visual acuity as the between-subjects factor. Each shape had different valence and arousal levels, and the movement condition had a different effect in children with and without VI.

In this section, we present the qualitative results of children’s reported strategies for associating tactile stimuli with emotions and their ideas of shape’s potential applications. We categorized their strategies in two broad themes: 1) tactile stimuli features driven where it is linked to the tactile features (roughness, movement, shape, frequency) and different-mapping (sensations or concepts); and 2) story context situations, in which participants leveraged the story context and familiar situations to associate with emotions.

Not all the shapes reliably map to a specific emotion. For this tactile vocabulary, tentacles and happiness were the most frequent combination, followed by spikes linked to anger/fear. Sadness was challenging to express using shape-changing stimuli, which is aligned with previous research [29]. Although distinct shape units can communicate a wide range of valence levels, arousal was harder to express; introducing movement to a shape unit did not significantly increase perceived arousal.

Children’s justifications support a high variance in their associations, which suggests that tactile stimuli’ emotional relations are highly user-dependent and can be mediated by multiple individual factors. As seen in figure 4, stimuli can be perceived as pleasant by some and unpleasant by others; for example, Spikes_D is mapped to anger (N=16) and happy (N=12) "I feel many spikes like they are angry" (C35S, angry) and "it is a thrilled feeling like tickles." (C29LV, happy).

Nevertheless, there is a tendency to associate dynamic/static Tentacles with happiness, dynamic/static Spikes with anger and fear, and static Wrinkles and Concaves with calmness. In Goosebumps and Concaves, the emotional mappings have high variance. This result can be a side effect of using the same shape skin for Goosebumps, Concaves, and Spikes. Some children noticed that hardy points of the Spikes shape unit when exploring other shapes, which could have influenced their responses - e.g., "angry peaks and fearful mountains" (C21LV, Goosebumps), "It seems a nervous heartbeat with needles" (C10S, Concaves). Therefore, goosebumps and concaves emotional mapping should be reassessed in new shapes (without contracted spikes).

Finally, the proposed shape-changing designs did not reliably convey sadness, which should be the focus of future research.

Children with and without visual impairment have different emotional-tactile mapping. Visual features for sadness influenced children’s mapping. A more reliable tactile-emotional association exists in sighted children as seen in Figure 4. Five stimuli mapped with an emotion: Tentacles_S for happiness, Wrinkles_S or Concaves_S to calm, Spikes_D and Spikes_S to fear and anger and Concaves_D to sad. In children with visual impairment, the association is less reliable as only three stimuli mapped with emotions: Tentacles_D with happiness, Wrinkles_D with calm and Spikes_D with fear and anger; all the other shape-changing stimuli did not have a reliable association.

Spikes_D consistently mapped anger and fear regardless of children’s visual ability. Interestingly, sadness was rarely selected by children with VI, but sighted children perceived sadness through the Concaves_D stimuli. The difference can potentially be explained by the visual nature of the bio-inspired metaphor; it was a "looking inside, self-absorbed feeling". The analogy can eventually become too "graphic" to be interpreted by children with VI and have a more natural mapping to sadness for sighted children. Further research is needed to understand whether visual metaphors play a role in tactile emotional precepts. One promising research avenue is the study of crossmodal correspondences [21, 43] and its differences dependent on users’ visual ability levels [4].

Movement and visual acuity influenced valence and arousal levels. The valence levels of emotions rated by sighted children ranged from lower negative (for dynamic) to lower positive for static shapes. While children with visual impairment chose similar lower positive valences for static and dynamic shapes. We also found opposite attributions of arousal by sighted and children with VI for three shape units when these were dynamic and static. These differences also support the distinct emotional-tactile mappings for children with and without VI. Additionally, one child with VI explicitly stated that they were more comfortable with dynamic shapes, as they knew the expected behaviour (e.g. "it is a thrilled feeling like tickles. Touching a moving stimulus is less stressful, we know what to expect"(C29Lv, happy)). Although we do not know whether sighted children feel the same way, this fear of unexpected behaviours should be considered in the future.

In Spikes stimuli, the emotional mapping was different between visual impairment levels. An additional result worth reflecting on is the tendency for different mappings in Spikes stimuli between children with blindness and low vision, as seen in figure 7. Children with low vision perceive spikes shapes similar to sighted children, with high arousal and negative valence. However, children with blindness showed a more uniform distribution covering all valence and arousal quadrants. The other shapes did not reflect the same tendency. Future research should aim to understand the meaning and reasons for this unexpected effect.

The study sheds light on the impact of visual acuity on tactile-emotional associations. Like music and material-mapping, tactile stimuli showed different emotional associations between children with and without visual impairment [52]. Those differences have three main implications for tactile communication. First, our findings suggest that sighted children leverage their visual experiences to make sense of shape-changing feedback, aligned with previous research [4]. Thus, designers can explore the visual features of tactile feedback to develop new shape-changing interactive systems. Second, the unexpected result of Spikes emotional mapping raised awareness of the potential differences in tactile perceptions for children with low vision compared to children with blindness. Third, designers should consider movement features to reduce children’s stress from unexpected events, which contrasts with prior work that focuses on expressing higher arousal levels [29].

Children did not have difficulty mapping emotions and describing their associations. The most preferred strategy disclosed was tactile-related; however, children with VI used it less than their sighted peers. To mitigate the order bias, we presented emotions randomly.

Self-reported association strategies revealed links to tactile-stimuli features (form and movement) and story context. Aligned with previous research, when using tactile-driven strategies, children used their sensations (e.g. fluffy, relaxing, aggressive) more than predefined concepts (e.g. urchin, tears, waves, nails). Children with VI, especially children with blindness, used more concept-driven justifications and familiar touchable concepts to map the emotions (e.g., tears, wrinkles, plant spines). This finding supports previous research [65] in which children with blindness extensively train tactile classification and can reach the same skill level as their sighted peers [4].

Moreover, children also leveraged living experiences to tactile and context associations. They can relate to familiar shapes (e.g. human skin, grass or hair) and rhythms (e.g. heartbeat, breathing rate) or story context using self-disclosure (e.g. family vacations, waves, fear) or empathize with the story’s character (e.g. "It is normal to be scared when you speak to others" (C10S, fear, Peter)). Occasionally, they used a combined strategy, using the story context to justify the character’s tactile behaviour (e.g. "John’s skin had some ups and downs showing insecurity, fear. He was nervous because of the cat." (C12S, fear, Spikes_S))

The emotion-tactile strategies justifications have two main highlights: 1) shape movement did not affect children’s emotional mapping; 2. Wrinkles_S and Concaves_D were harder to justify by children with VI; that effect did not apply to sighted children. Results suggest that Concaves are unfamiliar shapes for children with VI. We hypothesize that Concaves are not commonly encountered on everyday surfaces. Also, Wrinkles_S need more tactile expressiveness, maybe by increasing its rugosity.

Our findings exhibit similarities and differences compared to a previous study on mapping shape-changing stimuli to emotions by adults  [29] (N=40 in laboratory experiments). Six tactile stimuli were used in the previous work, including two shape units, i.e., spikes, goosebumps, two movement levels (fast speed vs. slow speed), and two amplitude levels. Similar to our findings, participants associated Spikes stimuli with emotions of high-negative valence and high-positive arousal, such as fear and anger.

In contrast to the previous findings that mapped the Goosebumps to positive-valence emotions, our results indicate that children associate negative-valence emotions with these stimuli. In addition, prior work highlighted the impact of changes in shape’s speed on the perception of emotional arousal; however, we did not observe a significant difference in emotional perception based on variations in movement. Various reasons could explain these differences.

First, the modality users interacted with differs between the two studies. In our study, shape units were placed inside a box, and participants could only rely on their sense of touch to interpret the stimuli. However, in previous research, shape units were not concealed; thus, participants may have relied on visual and tactile features to attribute emotions. This difference in emotional mapping, due to the absence or presence of visual cues, is supported by previous research [4], in which sighted users had a higher categorization accuracy using visual exploration compared to haptic exploration.

Second, participants’ demographics varied in age and background between user studies. Previous work focused on adult participants, while our study focused on children. Participants also differed in terms of visual acuity and cultural backgrounds, which may affect emotional perceptions.

Our work yielded a set of reflections and guidelines for emotional, tactile expressions, and affective computing studies, especially in mixed-visual ability environments. Therefore, first, we reflect on the benefits and compromises of using stories as a methodological technique to conduct affective computing studies with children. Second, we reflect on tactile-emotional mappings. Lastly, we recommend design guidelines for shapes as an emotional medium for children with and without visual impairment and the trade-offs needed in mixed-visual ability groups.

Our results are based on qualitative and quantitative data among fifty children, categorized into two groups: 26 children with visual impairment and 24 sighted. Our qualitative exploration extended to all children showed mapping strategies and children’s classifications of shapes into emotional variables such as valence and arousal in the two distinct groups. Our quantitative analysis explored the different emotional categories of shapes and compared the perception differences among children with and without visual impairment. The richness of our sample size, permits the use of our quantitative and qualitative insights into future studies and the differences between children with and without visual impairment on emotional and tactile perception. However, it is crucial to note that when we segmented the qualitative analysis of the twenty-six children with visual impairment into their visual acuity levels, the distinct responses between children with blindness and those with low vision may not be universally applicable. This limitation arises from the insufficient representation of children across the different visual impairment levels.Consequently, while our results catalyze awareness about the emotional-tactile perception that can be applied when exploring the differences between children with visual impairment and sighted, our findings in the subgroup of visual impairment, showing the differences between children with blindness and low vision, cannot be universally applied due to the limited number of children in each subgroup (blind, low-vision).

Secondly, this study included children from 8 schools in a single country. Although results guarantee some consistency regarding educational background, it needs to explore a broader spectrum of cultural profiles. Also, the number of children with visual impairment was insufficient to quantitatively confirm the differences in tactile-emotion associations and mapping strategies between blind and low-vision perceptions. Further research can replicate the reported study with more children from different cultures, countries, and visual abilities. Additionally the presence of the spikes, concaves and goosebumps on the same shape skin could have impacted their children’s mappings, as our system may not always completely extract and conceal the spikes inside the cavity. Further research should explore novel shapes, isolated in separate skins, or multisensory feedback to express sadness and distinguish between fear and anger precepts.

Additionally, emotions have subjective and individual factors that our predefined values of arousal and valence did not express; also, our work did not evaluate emotional dominance. Further research with children should include a children’s adapted version of Self-Assessment Manikin (SAM) [10] into a tactile questionnaire allowing children with and without visual impairment to classify valence, arousal, and dominance [35]. Lastly, in future work, EEG or GSR sensors can be explored to easier the emotional mapping process.