MLCommons has created MedPerf, a new testing platform to benchmark and evaluate medical AI models in healthcare. MedPerf relies on a wide variety of real-world medical data and protects patient privacy, according to the engineering consortium. MedPerf allows healthcare organizations, rather than vendors, to evaluate their AI models on demand. - The platform, developed with feedback from more than 40 companies and academic institutions, aims to improve the effectiveness of AI models, reduce bias, and support regulatory compliance.
- MedPerf is compatible with popular machine learning libraries and can also evaluate private models and those accessible through APIs.
- While currently focused on radiology scan-analyzing models, MedPerf is seen by MLCommons as a foundational step in advancing medical AI through open and scientific methods.
According to TechCrunch, medical AI models often show biases toward specific patient populations, including minorities, as they are trained using data from limited clinical settings. - In a Pew Research Center survey, 60% of U.S. adults said they would feel uncomfortable if their healthcare provider relied on AI for tasks such as diagnosing diseases and recommending treatments, while only 39% would feel comfortable.
|
