Microsoft Corp.'s researchers teamed up with pharma giant Novartis AG to develop a system called AssessMS, which uses the company's Kinect motion camera and machine learning software to track movements, analyzing things like gait and ability to touch one's nose with a finger.

Multiple sclerosis, which affects 2.3 million people worldwide, is an incurable and unpredictable disease of the central nervous system. For some it is fatal within a few years, while many others suffer physical deterioration over several decades. Most patients have what is called relapsing-remitting MS, where symptoms flare up and then improve, making it critical for doctors to track the severity of the disease over a period of time.

That makes it challenging for physicians, who have to rely on memory and eyesight to measure changes in a MS patient's symptoms—clinicians typically use a series of physical movement tests and then repeat them at the next appointment, often months later.

"Neurologists are very good at understanding what patients need and finding the right treatments for them, but what they are not very good at is being consistent in the way they quantify symptoms," said Cecily Morrison, a Microsoft researcher at the Human Experience + Design research group in Cambridge, U.K.

"Patients often see different doctors each time they are assessed and the question is—did I change or was there a change in the way the doctor scores it?"

Cecily Morrison of Microsoft Research in Cambridge, Jan. 11, 2016.
Cecily Morrison of Microsoft Research in Cambridge, Jan. 11, 2016.
Source: Microsoft Corp.
The disease causes the immune system to attack myelin, a fatty substance that coats and protects nerve fibers. That, in turn, can interrupt nerve impulses traveling to and from the brain and spinal cord, leading to pain, fatigue, vision loss and impaired coordination.  Some patients lose the ability to speak and walk.

In January 2015, Google announced a deal with Biogen Inc. to study multiple sclerosis using sensors, software and data analysis. Biogen has also developed an iPad app to measure the disease's progress. 

Imprecise measurements of progression make it harder for patients and doctors to make choices about care and when to start or stop certain treatments, some of which carry side effects. 

It also makes new drug trials difficult and costly for pharmaceutical companies like Novartis; because there there is so much noise in the test results, they have to try treatments on more patients and for longer times to make sure they work.
AssessMS system prototype in action.
AssessMS system prototype in action.
Source: Microsoft Corp.
So Novartis approached Microsoft. Most multiple sclerosis patients are screened by a doctor every three to six months, and scored on their movements. Microsoft, working with three of the top MS clinics in Europe, set up a system to do the same evaluation.

Microsoft's prototype mounts a Kinect motion-sensing camera (originally designed to accompany the company's Xbox video-game consoles) on a screen, asking patients to do things like extend their arms and hold still, or touch their extended hand to the tip of their nose and then repeat with eyes closed. The camera collects precise data on the patient's movement, indicating the degree of impairment.


Microsoft got help from several neurology experts in multiple sclerosis, who scored the video clips to teach the software's algorithm how to recognize the degree of impairment. In total, the software has analyzed 150 to 300 videos for each movement, completing phase one of the project in December. This year they will expand to five new clinics and hospitals to get more patient videos to perfect the algorithms.

"The beauty of computers is they don't get tired, can be used in different settings and use the same criteria—unlike neurologists," said Paul Matthews, head of the brain sciences division at London's Imperial College. Still, Microsoft will need to prove that its camera is as effective in picking up small movements like tremors as the human eye, he said.

If the tracking systems shows promise, Novartis plans to take it to the clinical validation process and seek regulatory approval.

"We are approaching parity with humans on accuracy, but perhaps better than humans with consistency," said Antonio Criminisi, a researcher at Microsoft. "But before feeling confident that this really works all the time, we need to be able to look at many more patients." Criminisi is a 16-year veteran of Microsoft Research who has used machine learning to analyze medical data such as CT and MRI scans. While capturing and working with motion data is much harder than working with static images, the effort can make a huge difference for MS patients.

"This is really super-interesting work," said Tim Coetzee, chief advocacy, services and research officer at the National MS Society in the U.S.  "The problem we are trying to solve in MS cries out for tools like this one where it is about being able to give the physician some consistent approach to measure the evolution of the disease."

Coetzee foresees a day when the technology can be used more frequently—in an in-home system, or a wearable device that gives constant feedback. That would empower patients with better information about their condition, giving them back some of the control that the disease takes away from them.
"One of the scariest things about MS is the unknown," Microsoft's Morrison said, "And measurement is one way to help."