Turning AI onto itself: AI algorithm detects when medical images will be difficult for radiologists or AI to make an effective diagnosis

A new AI algorithm can identify when medical images are likely to be difficult for either a radiologist or AI to make an effective diagnosis. The algorithm can potentially be used to triage medical scans and highlight cases that warrant further in-depth clinical evaluation or additional tests to support a definitive clinical diagnosis.

The algorithm, called UDC by AI healthcare company Presagen, was originally designed to automatically detect errors in medical data, particularly data that cannot be reliably verified by experts. 

When applied to images of x-rays to detect pneumonia, errors by radiologists were rare when the x-ray images had clear features. However, UDC found the diagnosis (or label) for several x-ray images to be neither correct nor an error. Verification of these images by an independent radiologist also agreed that they were indeed difficult images to diagnose, with their independent assessment often disagreeing with the original diagnosis provided in the public dataset. Similarly, AI that was trained to diagnose pneumonia also found the assessment difficult for these images.

Removal of poor-quality (difficult) images identified by UDC from the training dataset improved AI accuracy for diagnosing pneumonia in x-rays images by over 10% as measured on a hold out blind test set, and the AI was shown to be more scalable (generalizable). The accuracy also exceeded benchmarks set by the current literature for that public dataset. 

The AI Scientist that led the project, Dr Milad Dakka, said “Our results suggest these poor-quality images are uninformative, counter-productive or confusing when used in training AI. The ability to identify when new images are poor-quality is important to prevent an inaccurate AI clinical assessment, but also to alert the radiologist when the scan is likely to be difficult to diagnose or when a new scan should be taken.”

Presagen Co-Founder and Chief Strategy Officer, Dr Don Perugini said “Many AI practitioners believe that AI performance and scalability can be solved with more data. This is not true, and we call it the AI data fallacy. Even 1% poor-quality data can impact the performance of the AI. Building accurate and scalable AI is about using the right data.”

Presagen has recently developed a range of patent-pending AI technologies that drive a fundamental paradigm shift in developing commercially scalable AI products for real-world problems, which apply beyond healthcare and to AI more generally.

Dr Michelle Perugini said “We are excited to present to the world the suite of technologies, which we believe advance the field of AI. These technologies will allow Presagen to build scalable ‘out of the box’ AI products that are more commercially viable and technically superior, and thus market dominating. This is vital in Presagen’s journey to become world-leaders in AI Enhanced Healthcare and a dominant player in the AI-in-Femtech market globally. More importantly, we see it as an opportunity to change, lead, and dominate the AI industry.”