68 - Preliminary Results from the Development of a Quantification System for Reporting of Non-Compliance: An Impact Scoring System for Quality Assurance Report

Background:  Quality Improvement and Assurance units conduct not-for-cause audits of human research studies, but triggers for sending the results of compliance reviews to the IRB were not defined at our institution. To increase transparency and consistency, we developed a scoring system to assess the need to report findings to the IRB.

Program

Description:  We created an objective scoring system based on federal regulations to reduce subjectivity in reporting non-compliance findings in Quality Assurance Reports (QAR). This system quantifies their impact on safety and data integrity. Hosted in REDCap, the QAR Scoring System for Non-Compliance Reporting indicates the urgency of reporting findings to the IRB and offers clear guidance for decisions. Data from 119 internal not-for-cause audits conducted since 2022 that resulted in non-compliance findings was leveraged to create a composite scoring system. Three experienced Quality Improvement and Assurance experts reviewed findings from and assigned a subjective impact score to each study’s audit report: minor impact, moderate impact, and severe impact. By comparing impact and findings across reviews, we developed an objective scoring equation to indicate severity of impact on safety and urgency of reporting.

The following steps were taken:
1. Assigning Value to Type of FindingValues were assigned to critical, major, and minor findings based on urgency per federal regulations, with critical findings having the highest impact value.
2. Determining Weights of CategoriesWe ranked categories by risk to subject safety and data integrity: Informed Consent, Regulatory, Study Documentation, Study Procedures, Eligibility, Investigational Products, Adverse Events & Toxicities. This ranking provided numerical weights for each category.
3. Composite ScoringMultiplying the sum of critical, major, and minor findings in each category by their respective weights gives an impact sub-score. The total of these sub-scores results in a final impact score that dictates the next steps.

We tested, refined, and validated the equation using our REDCap dataset. We compared the final impact score with the initial determination to identify outliers. We found additional factors influencing IRB reporting. These were compiled into a rubric to adjust the final impact score by 50% based on harm to subjects, data integrity, and other considerations. Examples include:
1. Multiply the final score by 1.5 when 50% or more of the patient population is affected
2. Multiply the final score by 0.5 when <10% of subjects affected (in studies that have enrolled greater than 100 subjects)

Program

Assessment:  The impact score system required 3 rounds of edits to generate scores that consistently aligned with existing policies. The resultant categories were low impact (≤5 points), medium impact (5.1–19.9 points), and high impact (>20 points). The final equation has been preliminarily found to be accurate in indicating the urgency and severity of reporting non-compliance findings to the IRB. Specifically, we found that this scoring system aligned with existing practices for IRB reporting, in that all reports from studies assigned a “high” impact score were shared with the IRB. A risk map was created to identify which areas of study management regularly result in riskier and/or more severe non-compliance findings.

Limitations: Our data set is limited to the findings from 119 not-for-cause audits solely within our institution. Further, our results are preliminary and the scoring system will require full implementation to determine its applicability and accuracy. Additionally, the scoring system is still subject to a level of subjectivity on behalf of the auditor, which may vary between different auditors. Further fine-tuning of the subjectivity criteria will be necessary.


Discussion:  The QAR scoring system enhances transparency between QIA Specialists, auditors, the IRB, and investigators when determining the course of action to take in response to findings made during an audit. This standardized equation can be used nationwide, regardless of an institution's size or resources. However, future data collection is needed to assess the system's impact on training, education, and non-compliance findings. This could increase the transparency and efficiency of reviews, and help institutions customize their HRP education program to minimize risks to human research subjects.