Title: Audit Uncovers Data Challenges in AI-Enhanced Drug Discovery Benchmarks<\/p>\n
Within the field of drug discovery, artificial intelligence (AI) presents an opportunity to transform the process of finding potential therapeutics. Integral to this operation are benchmark datasets that act as testing arenas for machine-learning models. These benchmarks are essential in evaluating models and establishing their qualification as leading solutions. Nonetheless, a recent audit of 51 datasets uncovers considerable data challenges that could distort the perceived performance of these AI models.<\/p>\n
At the core of these challenges is the phenomenon known as train-test leakage. This occurs when the same molecule or highly similar ones are present in both the training and test sets of a dataset, which artificially enhances the performance metrics of a model. Additional issues include inconsistent annotations and improperly parsed structures within the data.<\/p>\n
A study conducted by Maximilian Schuh from the Technical University of Munich examined the frequency of these challenges across 51 benchmark configurations from widely used datasets like Polaris, Therapeutics Data Commons (TDC), MoleculeNet, and drug-target interaction (DTI) benchmarks. Their results suggest that a majority of datasets displayed issues, particularly train-test leakage and conflicting labels. DTI benchmarks were notably susceptible to overlap, showing high similarities between training and test samples.<\/p>\n
The repercussions of these data challenges were further explored by intentionally incorporating them into datasets and retraining various machine-learning models. Findings indicated that the existence of similar molecules in test sets could boost model performance, while contradictory labels typically led to diminished performance. Re-assessing benchmark leaderboards with alternative test sets frequently changed the ranking of models, emphasizing the influence of data integrity on perceived efficacy.<\/p>\n
Yang Zhang from the National University of Singapore noted the prevalence of data leakage and its capacity to misrepresent model performance. Pedro Ballester from Imperial College London pointed out the core issue of benchmarks lacking explicit connections to specific application contexts, impacting their relevance at different phases of drug discovery.<\/p>\n
In response, Schuh and his team advocate for enhanced transparency and systematic auditing of benchmark datasets. To aid in this effort, they created BenchAudit, an open-source toolkit intended to detect challenges like train-test contamination and label inconsistencies. Their goal is to raise awareness and promote truthful reporting of test set contents, rather than eliminating all dataset imperfections.<\/p>\n
This audit highlights the importance of stringent data management practices in AI-enhanced drug discovery. By addressing these benchmark challenges, researchers can ensure that the advancement of machine-learning models relies on precise, trustworthy data, ultimately speeding up the discovery of new therapeutics.<\/p>\n","protected":false},"excerpt":{"rendered":"
Title: Audit Uncovers Data Challenges in AI-Enhanced Drug Discovery Benchmarks Within the field of drug discovery, artificial intelligence (AI) presents an opportunity to transform the process of finding potential therapeutics. Integral to this operation are benchmark datasets that act as testing arenas for machine-learning models. These benchmarks are essential in evaluating models and establishing their […]<\/p>\n","protected":false},"author":1,"featured_media":375275,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"Default","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[174],"class_list":["post-375274","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-source-chemistryworld-com"],"_links":{"self":[{"href":"https:\/\/wolfscientific.com\/index.php?rest_route=\/wp\/v2\/posts\/375274","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wolfscientific.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wolfscientific.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wolfscientific.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wolfscientific.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=375274"}],"version-history":[{"count":0,"href":"https:\/\/wolfscientific.com\/index.php?rest_route=\/wp\/v2\/posts\/375274\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/wolfscientific.com\/index.php?rest_route=\/wp\/v2\/media\/375275"}],"wp:attachment":[{"href":"https:\/\/wolfscientific.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=375274"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wolfscientific.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=375274"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wolfscientific.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=375274"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}