Augmented reality and autism

The most effective interventions in people with autism are based on a personal work by specialized professionals and these therapies, when well designed and well done, achieve a constant improvement in different areas such as social communication, the presence of stereotypes and other atypical behaviors characteristic of autism. However, person-to-person therapies are by their very nature limited in their duration and frequency and it is interesting to explore what else we can do and what other tools can be helpful. That is where new technologies can enter, with some novel possibilities. In addition, computer technologies can give us feedback and provide us with quantitative data on symptomatology, the severity of the condition or the evolution of the participant based on the therapies and education received.

Children with ASD seem to be especially receptive to interventions with digital devices ¹. They have a clear preference for electronic media, game-like elements and programs or computer-generated speech. The rapid development of innovative technologies and the low cost of sensors can help to provide new tools and new paradigms for the understanding of social communication. These data can help to better identify the levels of each child and to select the therapies that are being most effective with him.

Among the aspects that have been seen to improve in people with ASD after using digital systems are social and emotional skills, recognition of facial expression, adaptive behaviors, academic skills, communication skills, vocational and pre-employment training and the control of inappropriate behaviors. Training social interaction allows one of the most commented situations by parents and teachers, the feeling of disconnection of the child during a conversation. This impression is most likely due to deficits in social communication manifested in low social and emotional reciprocity, a little use of non-verbal communication and difficulties in initiating, developing, maintaining and understanding social relationships.

An interdisciplinary group of Brain Power, MIT and Harvard Medical School have carried out a study on the Brain Power System. It is a system of augmented reality glasses that allows using different gamified applications – that is, they are used as if they were games – and that takes advantage of the power of artificial intelligence.

The BPS is a combination of hardware and software that can be applied to different types of virtual reality glasses. It includes a series of applications-games and has sensors that capture in real time movement data, physiology, the operation of the app, video and audio. The analysis of the gyroscope data and the accelerometers allow following the direction of the user’s head and movements. The presence of a small screen and a small speaker provide visual and auditory information to the user.

People with ASD spend less time looking at the faces of others and can do so in an atypical way, for example, focusing less on the eyes and mouth. There are two theories: that it is a process of aversion or indifference. The aversion of the gaze explains that the person with ASD avoids eye contact because they experience it as threatening, causing anxiety or generating overstimulation. The indifference of the gaze is based on the fact that the reduction of visual contact in ASD is a passive phenomenon where the eyes of others are not seen as something relevant or a stimulus that engages. It is possible that the situation is varied in different individuals.

The applications serve to train important aspects in the daily life of people with ASD such as the recognition of emotions, the direct gaze to the face, the eye contact and the self-regulation of the behavior. Addressing deficits in the processing of other people’s face information is important because it is often the key to the transmission of social information, such as emotions, expressions, gaze direction and others.

The authors published a pilot study ² with two cases of children diagnosed with ASD of 8 and 9 years. The intervention was well tolerated by the children and they described it as fun and engaging. Both quickly learned to use the system and there were no technical problems. This system also helps children and adults with ASD learn on their own skills for daily life that help them on the way to self-sufficiency. Their therapists reported that throughout the intervention they improved in nonverbal communication, eye contact and social interaction. One day after the intervention, an evaluation of the children was made using the “aberrant behavior checklist” and improvements were detected in irritability, lethargy, stereotypies, hyperactivity and inappropriate language control.

One of the apps is called Face Game. It uses vision algorithms and detects human faces in the user’s field of vision. Above them, it superimposes a kind of animated cartoon that attracts the boy’s attention. When he stares at it, the cartoon gradually fades, revealing the human face below. At that time, the program rewards the user with points. While keeping his eyes fixed on the interlocutor, the points increase until they slow down and stop. This prevents the gaze from becoming uncomfortable. The program gives more points if you look directly into the eyes and the program can be customized in both the difficulty and the reward.

Another program is Emotion Game. A limited understanding of the emotions implies a non-flexible behavior and problems of adaptation. The game seeks to assess and identify different emotions and reinforce the identification of human facial expressions trained by therapists. The system presents two emoticons to choose the one that most resembles the emotional expression presented by the interlocutor. The user chooses one or the other with a slight tilt of the head and receives points if successful.

The current diagnoses rarely have a quantitative follow-up and the digital tools could help to avoid this problem. Through the so-called machine learning, we can take advantage of this data to identify the best educational and therapeutic strategy for each individual user. Even so, it must be remembered that the study has only been done in two children and that the population with ASD is very demographically and clinically heterogeneous. Another caution is that it will also have to be tested on girls or on women because there are differences according to sex in social communication, memory and cognitive flexibility ³ and this may affect the results.