Getting to the doctor can be complicated. And the task can be particularly difficult for parents of children with motor disorders such as cerebral palsy, because a clinician must evaluate the child in person regularly, often for an hour at a time. Attending these frequent assessments can be costly, time-consuming and emotionally taxing.
MIT engineers hope to alleviate some of that stress with a new method that remotely assesses patients’ motor function. By combining computer vision and machine learning techniques, the method analyzes patient videos in real time and calculates a clinical motor function score based on certain pose patterns it detects in the video images.
The researchers tested the method on videos of more than 1,000 children with cerebral palsy. They found that the method could process each video and assign a corresponding clinical score with more than 70 percent accuracy to what a clinician had previously determined during an in-person visit.
Video analytics can be run on a range of mobile devices. The team envisions that patients could be assessed on their progress simply by setting their phone or tablet to take a video as they move around their own home. They could then load the video into a program that would quickly analyze the video frames and assign a clinical score or level of progress. The video and score could then be sent to a doctor for review.
The team is currently adapting the approach to evaluate children with metachromatic leukodystrophy – a rare genetic disorder that affects the central and peripheral nervous system. They also hope to adapt the method to assess patients who have suffered a stroke.
“We want to reduce some of the stress on patients by not having to go to the hospital for every evaluation,” says Hermano Krebs, a principal investigator in MIT’s department of mechanical engineering. “We believe this technology could potentially be used to remotely assess any conditions affecting motor behavior.”
Krebs and his colleagues will present their new approach at the IEEE Body Sensor Networks Conference in October. The study’s authors at MIT are first author Peijun Zhao, co-principal investigator Moises Alencastre-Miranda, Zhan Shen and Ciaran O’Neill, and David Whiteman and Javier Gervas-Arruga of the Takeda Development Center Americas, Inc.
Network training
At MIT, Krebs is developing robotic systems that physically work with patients to help them regain or strengthen their motor functions. He also adapted the systems to assess patients’ progress and predict which therapies might be most effective for them. Although these technologies work well, their accessibility is significantly limited: patients must go to a hospital or facility where the robots are in place.
“We asked ourselves: how could we extend the good results obtained with rehabilitation robots to a ubiquitous device? » Krebs remembers. “As smartphones are ubiquitous, our goal was to leverage their capabilities to remotely assess people with motor disabilities, so they can be assessed anywhere.”
Image: Dataset created by the Stanford Neuromuscular Biomechanics Lab in collaboration with Gillette Children’s Specialty Healthcare
Researchers were first interested in computer vision and algorithms that estimate human movements. In recent years, scientists have developed pose estimation algorithms designed to take a video – for example, of a girl kicking a soccer ball – and translate her movements into a corresponding series of skeletal poses, in real time . The resulting sequence of lines and points can be mapped with coordinates that scientists can analyze in more detail.
Krebs and his colleagues aimed to develop a method to analyze skeletal pose data from patients with cerebral palsy – a disorder traditionally assessed according to the Gross Motor Function Classification System (GMFCS), a five-level scale that represents the general motor function of a child. . (The lower the number, the greater the child’s mobility.)
The team worked with a publicly available dataset of skeletal pose, produced by the Neuromuscular Biomechanics Laboratory at Stanford University. This dataset included videos of over 1,000 children with cerebral palsy. Each video showed a child performing a series of exercises in a clinical setting, and each video was labeled with a GMFCS score that a clinician assigned to the child after the in-person assessment. The Stanford group ran the videos through a pose estimation algorithm to generate skeletal pose data, which the MIT group then used as a starting point for their study.
The researchers then looked for ways to automatically decipher patterns in cerebral palsy data that are characteristic of each clinical level of motor function. They started with a space-time graph convolutional neural network – a machine learning process that trains a computer to process spatial data that changes over time, such as a sequence of skeletal poses, and assign a classification.
Before the team applied the neural network to cerebral palsy, they used a pre-trained model on a more general dataset, containing videos of healthy adults performing various daily activities like walking, running, sit down and shake hands. They took the basis of this pre-trained model and added a new layer of classification, specific to clinical scores related to cerebral palsy. They refined the network to recognize distinctive movement patterns of children with cerebral palsy and accurately classify them into key levels of clinical assessment.
They found that the pre-trained network learned to correctly classify the children’s mobility levels, and did so more accurately than if it was trained on cerebral palsy data alone.
“As the network is trained on a very large data set of more general movements, it has some ideas about how to extract features from a sequence of human poses,” says Zhao. “While the larger dataset and the cerebral palsy dataset may be different, they share some common patterns of human actions and how those actions can be encoded.”
The team tested their method on a number of mobile devices, including various smartphones, tablets and laptops, and found that most devices could run the program successfully and generate a clinical score from videos, virtually in real time.
Researchers are currently developing an app that they say could one day be used by parents and patients to automatically analyze patient videos, taken in the comfort of their own environment. The results could then be sent to a doctor for further evaluation. The team also plans to adapt the method to assess other neurological disorders.
“This approach could be easily extended to other disabilities such as stroke or Parkinson’s disease once tested in this population using settings appropriate for adults,” says Alberto Esquenazi, chief medical officer at Moss Rehabilitation Hospital of Philadelphia, who did not participate in the study. study. “It could improve care and reduce the overall cost of health care and the need for families to lose productive work time, and I hope (it could) increase adherence.”
“In the future, this could also help us predict how patients would respond more quickly to interventions,” says Krebs. “Because we could evaluate them more often, to see if an intervention has an impact. »
This research was funded by Takeda Development Center Americas, Inc.
