56 - The Use of Artificial Intelligence in IRB Administration

Background:  With the advent of artificial intelligence (AI) tools, there arose interest in utilizing them for IRB administrative processes, particularly those internal processes that have little to no impact on investigators but could improve internal IRB operations.

Program

Description:  At the University of Texas at Arlington (UTA), there are several possible review categories, including a “flex review” pathway for non-federally funded and/or non-FDA regulated protocols. The review category is determined by the IRB specialist upon their review of the electronic submission based on certain conditions, like funding and the involvement of investigational drugs or devices. In partnership with an artificial intelligence commercial development team, we conducted a proof-of-concept project in which the AI scanned the data to predict the review category of the IRB submission (e.g. full board, expedited, exempt, or flex review). The conditions and rules were given to the AI model to test whether it could accurately predict the review category.

Program

Assessment:  In collaboration with the commercial development team, we conducted accuracy testing for the AI predictions using the model. Across three runs, we assessed the model’s reasoning behind its final answers for review categories. Our testing included new protocols, modifications, and continuing review submissions. To ensure a representative range of typical submissions to the IRB, we used historical data and created new submissions specifically for testing. Through iterative refinement of conditions, rules, and the API provided to the model, we achieved improved accuracy—from an initial 45% to 69%—in predicting review categories (e.g., full board, expedited, exempt, or flex review).

Limitations: Creating rules and conditions for an AI model involves expertise in both AI and human subject protections. Identifying team members to write these conditions and allocate time for feedback can be challenging. For limited scope AI projects, this approach works well, but ongoing training and development costs should be considered. Having the technical expertise for development, implementation, and maintenance can also be limiting for some institutions.

Discussion:  Our project served as a proof-of-concept, and we are actively refining the model while building expertise within our institution. Our ongoing efforts involve collecting additional data and exploring diverse applications, such as generating consent forms and streamlining the protocol review process. Notably, this program could enable IRB staff to focus on specific protocols aligned with their skills and availability, while also quickening the identification of submissions requiring full board review.