No matter how improved the algorithms are, they will be useless unless a doctor finds the data sensible. So how doctors interact with information can be crucial determinants in the utility of ML technology.
Traditionally pathologists take reference of their colleagues and other sources like textbooks to examine the problem at hand. It can be an image of a cancer scan or of that of blunt trauma. Modern-day technology enables the computers to skim through thousands of images and retrieve information which is the closest to the query. For example, the reverse image search option in Google images.
Now, there is widespread adoption of ‘digital pathology’. The clinicians can now examine the data received as images on a computer. And, can draw insights from it making the whole process of diagnosis relatively easy.
On these lines, a group of researchers at Google artificial intelligence department have teamed up to introduce machine learning tools and methods to propel the adoption of digital pathology:
- Similar image search for histopathology – SMILY
- Human-centered tools for coping with imperfect algorithms during medical decision-making
Similar Medical Images Like Yours (SMILY)
The first step in developing SMILY was to apply a deep learning model, trained using 5 billion natural, non-pathology images (for example, dogs, trees, man-made objects, etc.) During the training, the network learned to distinguish between similar and dissimilar images by comparing the embeddings.
Embedding is a summarization of the numerical vector which is a compression of all the images fed into the deep learning models during training.
SMILY allows a user to select a region of interest, and obtain visually-similar matches. As can be seen in the example below where selecting a small region in a slide will enable SMILY to efficiently search a database of billions of cropped images in a few seconds.
Because pathology images can be viewed at different magnifications (zoom levels), SMILY automatically searches images at the same magnification as the input image. Though these machine learning tools for similar image retrieval drastically reduce the time taken through traditional methods, they fall short on understanding the intent of the user or searcher.
A doctor can look at the same report for different things and it varies from case to case. To make SMILY more interactive, the researchers refined the tool by enabling end-users to express what similarity means on-the-fly:
- Refine-by-region allows pathologists to crop a region of interest within the image, limiting the search to just that region
- Refine-by-example gives users the ability to pick a subset of the search results and retrieve more results like those
- Refine-by-concept sliders can be used to specify that more or less of a clinical concept be present in the search results (e.g., fused glands)
The above concepts need not be built into a machine learning model. Instead, this method allows end-users to create new concepts post-hoc, customising the search algorithm towards concepts they find important for each specific use case.
The above illustration depicts how interactive the tool is when augmented with refinement.
SMILY: A Clinical Approach
The study conducted by the researchers was published in Nature journal and the results show that the pathologists have found this new tool useful in decision making like keeping a track of the likelihood of a hypothesis.
Tools such as these make the doctors more inclusive in the digital process and eliminate the passivity that the existing methods bring with them. SMILY strikes the right balance by equipping the doctors with domain expertise while allowing them to customise the tool for their own benefit.
Know more about the tool.