The FDA’s new list of AI and machine learning compatible devices highlights opportunities for improvement.
Craig Coombs and Qiang Kou, Nyquist data
FDA released a list of authorized or approved artificial intelligence and machine learning devices in September, documenting much of the agency’s work in the innovative field of AI / ML.
Extracting this information from the FDA’s decades-old database is laborious at best and often impossible. Despite the time the FDA has spent creating this new list, a lack of even text-based search capability makes the list itself cumbersome and time consuming to review.
Wouldn’t it be better if there was an AI resource that could quickly compile a list of FDA AI / ML authorizations and approvals, allowing searches in seconds rather than hours?
Like many databases, the FDA database uses text matching to find relevant entries. The weakness of text matching is that it doesn’t match the search term as a whole. For example, if you search for “pain”, all records containing “pain” will be flagged, as well as records containing “Spain” and “painting”.
And text matching often misses relevant results. When users search for “pediatric”, the FDA database does not search for other spellings such as “pediatric” or related terms such as “newborn,” “newborn,” “infant,” ” children â€and so on.
The strength of modern AI-based search algorithms is that they can understand content and help users find what they’re looking for. A JAMA article published in July used a type of AI research algorithm known as natural language processing (NLP) to identify an additional 23% of reports from the FDA database with patient deaths that were not classified like deaths using the text match search. Many of these misclassified reports never mentioned “deathâ€, but rather “patient expired†or “could not be resuscitatedâ€. This demonstrates the limitations of text matching and why the medical device industry should shift to modern AI-based methods, not only for searching databases, but also for querying adverse event reports. , reminders and review details of applications.
The authors used Nyquist Data’s commercially available AI-based search engine to rate the FDA database with the keywords “machine learning,†“artificial intelligence,†“deep learning,†or “Neural network”. We only used the publicly available FDA database, which includes summary files disclosed by the FDA. We quickly generated a list of 222 devices and compared it to the FDA’s list, finding three key differences:
1. Text search usually omits relevant citations.
NLP methods have discovered more than 20 AI / ML devices that were not on the FDA list but were approved before June 2021. For example, the RapidScreen RS-2000 System approved in 2001 clearly indicated that it used an artificial neural network for classification, and the Pathwork Diagnostics Original Tissue Test approved in 2008, also made it clear that it used a machine learning approach based on the selection of markers to build a predictive model.
2. A text match search will always be out of date compared to an AI-enabled search due to the speed of the AI.
The latest device on the FDA’s list is the Philips precise position, approved on June 17, 2021. According to our research, the FDA has approved at least 24 AI / ML devices after the list deadline. The FDA says it will update the list periodically, but an AI / ML compatible search engine using NLP algorithms can update itself automatically in milliseconds.
3. AI / ML search engines are more flexible, but have clear search and exclusion criteria.
The FDA said it created its list “by researching the FDA’s public information, as well as reviewing information in publicly available resources cited below and other publicly available documents published by specific manufacturers.” . How they searched for the information and with what keywords is unclear.
Gili Pro BioSensor is on the list, but the publicly available reclassification order does not mention anything about ‘machine learning’ or ‘artificial intelligence’, only that it uses an optical sensor system and software algorithms to obtain and analyze video signals and estimate vital signs.
The RX-1 Rhythm Express Remote Heart Monitoring System is also included but does not explicitly say “machine learning†or “artificial intelligenceâ€. He mentioned that a built-in algorithm processes the acquired ECG to detect arrhythmias, compress the ECG, and suppress most in-band noise without distorting the morphology of the ECG. There are at least 10 other ECG analysis devices included in the FDA’s list, but none have explicitly mentioned “machine learning” or “artificial intelligence” in their public information. Assuming these examples are on the list, that means the FDA may have used personal knowledge of the devices in a tedious search, pointing out gaps in text-match searches.
Filtering for more intelligence
Besides these differences between text matching and AI algorithms, the AI ​​results could be further interrogated to find interesting results. What if you wanted to know which AI / ML devices required a clinical trial for market authorization? The FDA listing requires that you individually call up each device, one at a time, to review the content of the submission. Using Nyquist Data’s commercially available AI search engine, the authors immediately developed a list of AI / ML devices and simultaneously filtered the clinical trial descriptions.
Most of the AI ​​/ ML devices have radiological clearances from the FDA that did not require clinical trials to establish substantial equivalence with the predicate devices. However, five AI / ML devices submitted clinical trial data to support their claims of substantial equivalence or their claims of safety and efficacy: one in ophthalmology (K200667), one in microbiology (K142677) and three in radiology (P200003, DEN170073 and K183019). This information is essential in determining the criteria for potential testing of innovations in the same fields. This additional research takes milliseconds using a commercial AI search engine, but can take several hours using the FDA database search engine. In addition to the amount of work required to use non-AI search algorithms, one cannot ignore the loss of valuable information that is easily missed.
We are excited about the innovation, flexibility and intelligence of the FDA demonstrated in the approval / authorization of medical devices associated with AI and ML. Their list improves regulatory monitoring for everyone.
Nonetheless, as regulatory professionals, we should use AI / ML to unlock FDA databases to improve regulatory watch. The better regulatory intelligence that comes from the use of AI algorithms can drive better business practices in regulatory and clinical affairs, quality management, and the development of business strategies.
Craig Coombs is President of Coombs Medical Device Consulting, Medical Device Submission Instructor at the University of California, Santa Cruz, and a member of the Nyquist Data Advisory Board.
Qiang Kou is the technology co-founder of Nyquist Data. He holds a doctorate in bioinformatics from Indiana University.
The opinions expressed in this blog post are those of the author alone and do not necessarily reflect those of MassDevice.com or its employees.
