Medical Imaging is one of the popular fields where the researchers are widely exploring deep learning. But the research may not translate easily into a practical or production-ready tech. In an engaging session by Abdul Jilani at the Computer Vision Developer Conference 2020, Abdul Jilani who is the lead data scientist at DataRobot explained the various challenges that applied machine learning face and how these can be overcome in real-time environments.
“Building good training datasets and the wrong practices which lead to data leakage were the most commonly faced challenges that I investigated in a famous COVID X-rays prediction paper,” he said. He further shared examples where Google’s medical AI was super accurate in the lab, but the real-life story turned out to be very different, as it failed to give results at all when applied to real-life situations.
Most pre-trained models fail on medical images, and traditional feature engineering and image augmentation doesn’t seem to work on medical images as on other datasets. While deep learning in medical imaging should yield powerful and human-like results in terms of efficiency, it often faces challenges in terms of — data, expertise and production. All of these are interconnected, and a shortfall in any of these may lead to subsequent failure of the model.
Challenges In Terms Of Data
Some of the challenges that practitioners face in terms of data while dealing with medical imaging are:
The rarity of medical imaging datasets: To find a dataset that deals with medical imaging is challenging when compared to other domains. It is usually hard to find because of the proprietary values, the data being personally identifiable information, IP, the monetary value associated with it and more. There is also a challenge in terms of lesser devices being in the field such as MRIs than other domains such as self-driving cars where cameras can capture images.
Jilani shared that one of the ways to address this challenge is crowdsourcing images, where people can be asked to share images based on their encounter. For instance, COVID-19 X-ray images evolved from similar crowdsourcing where COVID positive patients were asked to share their X-ray images. Another way to deal with it is the democratisation of data where organisations make the datasets available for researchers to carry their research. Some of the publicly available datasets are CheXphoto, MURA, RSNA, among others.
Generalisation: The lab images may differ significantly when compared to the real-time images which are more clear, high-res compared to latter. These differences in training and test data can lead to discrepancies in actual models.
Jilani shares that it is therefore essential to have a library that has datasets that has consistent data, for instance, ChexPhoto. Other ways to deal with these differences are data augmentation where images can mimic the actual data so that training and production data are the same. It can do so by adding noise, blur, sharpness to images to mimic the real data.
Data collection and sampling bias: Improper sampling and data stitching can lead to sampling bias which can lead to improper model generation. For instance, in the crowdsourcing, as mentioned above, where COVID positive patients shared X-rays, it became a challenge as there were no actual X-ray images available of the same patient to compare with. This led to comparisons being done to already available X-ray images such as pediatric X-rays which were not in tandem with the collected sample. Merging these heterogeneous datasets can cause data leakage, which is a big challenge. Improper sampling can lead to patterns that are not relevant.
Jilani suggests that this can be avoided by adhering to one class of population to avoid data leakage. Robust sampling, i.e., considering sampling of same age buckets, same annotation, the same geography can overcome this challenge. He also suggested SME validation or conducting a proper clinical study as one of the crucial ways to address this challenge.
Multi-modal nature of use cases: Most medical diagnoses require multi-modal data to understand symptoms of a medical condition from different angles properly.
As Jilani shares, this can be dealt with using multi-modal algorithms and approaches.
Challenges In Terms Of Expertise
Some of the challenges that practitioners face in terms of expertise while dealing with medical imaging are:
The rarity of experts: Radiologist experts are rare that have a familiarity with deep learning techniques. Automating X-ray classification requires a lot of time and effort and expertise, which is currently lagging. A lot of times there is pseudo labelling, lack of annotations, activation maps, which require subject matter experts to point these out. As Jilani says, knowing where to look is the key in medical tasks rather than just bucketing, which may lead to algorithms learning wrong patterns.
The way to deal with this is to have access to a group who are experts in the field. Hiring interns is also a good idea who can conduct tasks such as labelling, learn to use tools to build models, among others.
Robust feature imaging: Jilani shared that not all data augmentation is useful. For instance, brain CT scan, retina screening, and others will require different data for new feature engineering, which again requires subject matter experts.
To deal with this, It is important to prioritise to reduce time and compute. Artifact removal, such as cropping images to remove artifacts can avoid data leakage.
Challenges In Terms Of Production
Some of the challenges that practitioners face in terms of production while dealing with medical imaging are:
Standards: There need to be specific standards of uniformity when it comes to medical data. This is where DICOM and PACS come into picture which are international standards for the communication and management of medical imaging information and related data. It stores metadata along with images pixel and can save multiple images per patient ID.
Architectures: There needs to be proper architecture in place to carry deep learning in medical imaging. For instance, SOTA architectures are a good starting point, which can be done using RESNET.
