68 - Preliminary Results from the Development of a Quantification System for Reporting of Non-Compliance: An Impact Scoring System for Quality Assurance Report

Background:  Quality Improvement and Assurance units conduct not-for-cause audits of human research studies, but triggers for sending the results of compliance reviews to the IRB were not defined at our institution. To increase transparency and consistency, we developed a scoring system to assess the need to report findings to the IRB.

Program

Description:  Data from 104 internal not-for-cause audits conducted since 2022 that resulted in non-compliance findings was leveraged to create a composite scoring system. Three experienced Quality Improvement and Assurance experts reviewed findings from and assigned a subjective impact score to each study’s audit report: low impact, medium impact, and high impact. To create an objective scoring equation, the group interpreted federal reporting requirements to assign weights to each non-compliance finding category (e.g., major informed consent findings, minor regulatory findings, etc.) to generate quantifiable impact scores for each reviewed study. The subjective ratings and whether the audit reports were shared with the IRB were compared to the equation-generated scores, and discrepancies were discussed. When consensus was reached on the weights for each finding type, the generated equation was iteratively tested and refined using the same dataset. A subjectivity multiplier with pre-determined criteria was piloted after consultation with a statistician. Final categorical scores and their subsequent reporting requirements were determined by consensus, and further testing and piloting is ongoing.

Program

Assessment:  The impact score system required 3 rounds of edits to generate scores that consistently aligned with existing policies. The resultant categories were low impact (≤5 points), medium impact (5.1–20 points), and high impact (>20 points). The final equation has been preliminarily found to be accurate in indicating the urgency and severity of reporting non-compliance findings to the IRB. Specifically, we found that this scoring system aligned with existing practices for IRB reporting, in that all reports from studies assigned a “high” impact score were shared with the IRB. A risk map was created to identify which areas of study management regularly result in riskier and/or more severe non-compliance findings.

Limitations: Our data set is limited to the findings from 104 not-for-cause audits solely within our institution. Further, our results are preliminary and the scoring system will require full implementation to determine its applicability and accuracy. Additionally, the scoring system is still subject to a level of subjectivity on behalf of the auditor, which may vary between different auditors. Further fine-tuning of the subjectivity criteria will be necessary.

Discussion:  The QAR scoring system enhances transparency between QIA Specialists, auditors, the IRB, and investigators when determining the course of action to take in response to findings made during an audit. This standardized equation can be used nationwide, regardless of an institution's size or resources. However, future data collection is needed to assess the system's impact on training, education, and non-compliance findings. This could increase the transparency and efficiency of reviews, and help institutions customize their HRP education program to minimize risks to human research subjects.