Smart Machines enabling All Learners

A conversation with Prof. Markus Gross.

Gross is Vice President of the Global R&D and Director of Disney Research. He is a Professor of Computer Science at the Swiss Federal Institute of Technology Zürich (ETH), where he is also the head of the Computer Graphics Laboratory. Gross is passionate about using emerging technologies to enhance educational possibilities, leading new research as well as entrepreneurial educational solutions.

An interesting thing happened when Markus Gross started treating his son for dyslexia: As he learned about the neural networks that cause the condition, he recognized their similarities to machine learning.

He took a creative leap, and voila! He was able to develop learning exercises for his son that were built on machine learning principles. The results were so good that the program was adopted by Switzerland’s school system.  And those results are proving so good that today – twenty years later — Markus is applying his expertise in Augmented Reality and other advanced technologies to the education of all children.

“The learning program is basically a machine learning system that tries to model the internal state of learning and knowledge by the student at each and every moment,” he explains.  “This allows us to adjust the training to the individual’s capability, strengths and weaknesses; this is what we actually do.”

 Markus is at the forefront of advanced technologies in his dual roles as Professor of Computer Science at the Swiss Federal Institute of Technology in Zurich and Vice President of Global R&D at Disney Company.

His original program, still very much in use, concerns dyslexia and the related condition of dyscalculia. Dyslexia, which affects 5%-10% of the western world population, is the inability to develop reading and writing comprehension while otherwise having an average or above average intelligence. Dyscalculia, which affects about 5% , is the inability to perform simple mathematical calculations.

“The conditions are very severe because they are basically spread-out over all aspects of learning in school and in professional life, and it often leads to demotivation and frustration,” he notes.

As he began working with neuroscientists, Markus recognized that dyslexics are unable to develop proper mapping and automation functions. The former prevents them from constructing the building blocks of spoken and written language, while the latter prevents them from completing the “parietal shift” of cognitive power moving from the frontal cortex to the parietal lobes.

He built a team with experts from the University of Zurich that combined neuroscience, computer science and machine intelligence into a learning system that would overcome the mapping and automation roadblocks to cognitive processing.

The result was a computer-based, personalized and interactive training program based on the concept of multimodality – the ability of the brain to process information through a number of different perceptual channels simultaneously. Thus, a child experiences a new word through numerous sensory cues, such as color, motion and music. The combination of cues breaks the roadblocks.

“The fundamental idea is to basically rewrite, to transcode, the initial information that is in a word; we call this a string in computer science, the sequence of letters,” he explains. We transcode it to a variety of perceptual codes. Then, to measure the amount of processing, we use a concept which is well known from information theory, entropy. The interesting aspect is that we prove this multimodal transcoding helps to trigger the parietal shift that otherwise would be delayed or not happening, and  this is essential for automation.”

There is now almost twenty years of data from over a hundred thousand Swiss students, whose every keystroke has been captured. The results show 30% less in pencil and paper dictation errors in the dyslexia training and up to 45% more correct calculations for dyscalculia. This is after six weeks of training, four days a week, twenty minutes a day. There is also a significant decrease in self-reported anxiety about school and learning, with more than 25% of students reporting less anxiety.

While he began as an accidental educator, Markus continued to be interested and involved as EdTech mainstreamed. Now he’s in a prime position to explore and implement new concepts.

“Moving forward, I want to combine our concepts with two equally powerful concepts I learned at Disney,” he says. “One relates to the notion of storytelling. I really want to make the learning experience a journey through a theme park with individual rides, shows, and a very personalized story, if you will. The second is the advent of Augmented Reality with a new generation of devices looming on the horizon.”

He offers the creation of an AR music band as an example, showing how a child places trading cards with photos of band members under an iPhone and watches them come alive on the screen, singing and playing their music. The child’s manipulation of the cards changes the music.

“Creativity can best be taught through play and interaction; this is our core concept,” he says. “It includes both the digital and haptic — the physical — experience. This is why we believe that Augmented Reality is so important,  because it allows us to seamlessly combine the real and the virtual world. It gives a child a haptic experience enriched with all the capabilities of virtual and digital.”

 “Reading and writing is an artifact of our society,” he says matter-of-factly. “Evolution has taught us to communicate through spoken language. My personal area of expertise is visual computing — conveying information through visual or audio-visual — and this is actually the concept I applied in our therapy for dyslexia.”