How does artificial intelligence help develop the field of medical image analysis?

Medical images, such as MRIs and X-rays, are essential tools for diagnosing disease and assessing health status, but these images are often complex and difficult to understand to the untrained eye. They look like a mysterious collection of black and white spots, and it can be extremely difficult to decipher where one anatomical structure (such as a tumor) ends and another begins.

When AI systems are trained to understand the boundaries of anatomical structures, they can automatically and accurately identify pathological areas in medical images, rather than wasting time manually tracing anatomy across multiple images, saving doctors valuable time and helping them make more accurate decisions.

The problem is that training AI systems to analyze medical images requires a lot of human effort. Researchers and doctors have to manually mark a huge number of images to identify areas of interest. For example, training a model to recognize the cerebral cortex in MRI images requires marking this cortex in a large number of images. This is a time-consuming and labor-intensive process, which affects the development of these systems and limits their spread.

What is the solution?

To overcome these challenges, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory, Massachusetts General Hospital, and Harvard Medical School have developed a new interactive framework called ScribblePrompt, an innovative tool that can quickly and accurately analyze any medical image, even images that have not been previously trained on.

How does ScribblePrompt work?

Instead of relying on manually labeling millions of images, the researchers devised a new way to mimic how users analyze medical images. They analyzed more than 50,000 different medical scans, including MRIs, ultrasounds, photographs, and microscope images, to see how specialists label important parts of these images across structures in the eyes, cells, brains, bones, skin, and more.

They then used advanced algorithms to simulate these signals, such as scribbling and clicking on different areas of medical images. Additionally, they used superpixel algorithms to discover new areas of interest to medical researchers, which helped them generate massive amounts of training data. This data helped train ScribblePrompt to understand and segment medical images with high accuracy, even those it had never seen in training data before.

“AI has great potential for analyzing images and other high-dimensional data, enabling doctors to make more accurate decisions quickly,” says Haley Wong , a Ph.D. student at MIT and lead author of a new paper on ScribblePrompt. “So we developed ScribblePrompt, which helps doctors identify important parts of medical images with extreme accuracy and speed. With this tool, doctors can save about 28 percent of their time compared to the latest available models, such as the Segment Anything Model (SAM), developed by Meta . ”

ScribblePrompt Features:

ScribblePrompt has a simple interface. Users, whether doctors or nurses, can handwrite or click on the part of the image they want to highlight, and the tool will automatically highlight the exact area, whether it's small veins in a retinal image or any other shape. Users can also highlight areas in images by drawing a box around them, making it an ideal tool for a variety of needs.

The tool can then make corrections based on user feedback. It can increase the accuracy of the selection by adding additional details to the area the user has selected. For example, if the tool has missed some edges, the user can complete their drawing manually. The user can also delete unwanted parts using the Negative Scribble tool.

These interactive, self-correcting capabilities have made ScribblePrompt the tool of choice among researchers at Massachusetts General Hospital’s Department of Neuroimaging, with 93.8% of researchers preferring ScribblePrompt over Meta’s SAM model to improve their clips based on scribble corrections, and 87.5% of researchers preferring ScribblePrompt to make click-based edits.

ScribblePrompt was trained on simulated drawings and clicks on 54,000 images across 65 datasets, including scans of the eyes, chest, spine, cells, skin, abdominal muscles, neck, brain, bones, and teeth. It recognized 16 types of medical images, including CT scans, X-rays, MRIs, ultrasounds, and photographs.

What makes ScribblePrompt different?

Ease of use: Using ScribblePrompt does not require any special technical skills.
Natural Interaction: ScribblePrompt provides a simple user interface that allows users to interact with images in a natural way, as if they were drawing on a piece of paper. The user can simply draw lines or boxes on the image to define the area they want to analyze, and the tool will accurately understand the instructions and highlight the desired area.
Learning from Big Data: The tool was trained on a massive amount of data, which helped it develop a deep understanding of medical images, and it can recognize different types of medical images, such as MRI, X-ray, and ultrasound.
Generalization to new types of images: ScribblePrompt can handle new types of images that it has not been previously trained on, making it a flexible tool that can be used in a wide range of medical applications.

What impact does ScribblePrompt have in healthcare?

ScribblePrompt represents a quantum leap in the healthcare field , thanks to its ability to analyze medical images with extreme accuracy, speed, and ease of use without the need for any technical skills. Here are some of the expected effects of this development:

Improved diagnostic accuracy: ScribblePrompt helps doctors pinpoint diseased areas with greater accuracy in medical images, leading to more reliable diagnoses. It also reduces the time required to analyze images, allowing doctors to diagnose conditions faster.
Personalization of treatment: It helps in determining treatment goals with high accuracy, and can be used to evaluate the effectiveness of treatment and monitor the development of the patient’s health condition over time, allowing treatment to be personalized for each individual patient.
Increased productivity: Reduces the time and effort required to analyze medical images, allowing doctors to focus on other aspects of healthcare.
Developing medical research: It can be used to analyze huge amounts of medical data, contributing to the discovery of new and more effective treatments.
Expanding healthcare access: ScribblePrompt can be used for remote diagnostics, giving people living in remote areas access to healthcare.