Evaluation of a prospective adverse event reporting system in interventional radiology

https://doi.org/10.1016/j.crad.2021.04.009 0009-9260/ 2021 The Authors. Published by Elsevi creativecommons.org/licenses/by/4.0/). AIM: To assess the performance of a prospective adverse event (AE) reporting system. MATERIALS AND METHODS: Four hundred and seventy-one consecutive arterial procedures were performed in 465 patients (median age, 65 years; interquartile range, 54e77; 276 men) over 2 years by four interventional radiologists at a single centre where clinical follow-up was not performed routinely by interventional radiology (IR). AEs were reported prospectively using a radiology information system or in interventional radiologists’ electronic records and combined in a departmental listing of adverse events (DLAE). A retrospective medical record review was performed to identify a reference standard list of AEs for this observational cohort study. AEs were graded according to the Society of Interventional Radiology AE classification system. Descriptive statistics were calculated for the performance of the DLAE. A model comparing the rate of reporting of AEs with and without integration of clinical follow-up was tested for significance. RESULTS: Thirty-eight of the 471 (8%) IR procedures had an AE according to the reference standard. The DLAE identified 20/38 (53%) of AEs (K1⁄40.67 [good agreement], 95% confidence interval [CI] agreement1⁄40.53e0.81; p1⁄40.0001; sensitivity 52.6% [95% CI, 36e69%], specificity 100% [95% CI, 99e100%], positive predictive value [PPV] 100%, negative predictive value [NPV] 96 [95% CI, 94.5e97%], accuracy 96% [95% CI, 94e97%]). The performance of the AE reporting system will improve with integration of clinical follow-up (p1⁄40.0015). CONCLUSION: A prospective AE reporting system without clinical integration will not detect all procedure complications. 2021 The Authors. Published by Elsevier Ltd on behalf of The Royal College of Radiologists. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/ 4.0/).


Introduction
Adverse events (AEs) are common in medical practice and are detrimental to patient outcomes. AEs occur in one tenth of medical inpatient, surgical inpatient, and intensive care episodes and half of these AEs are preventable. 1 Major AE can lead to delayed discharge, patient disability, and death. 1 Reporting AEs is a fundamental process in a qualityassurance programme. An optimal reporting system allows comprehensive review of all detected AEs in an interventional radiology (IR) department quality meeting. It provides an opportunity for healthcare workers to q This work was presented at RSNA 2020.

Contents lists available at ScienceDirect
Clinical Radiology evaluate preventable AEs and adjust clinical care processes to improve patient safety and patient outcome. An effective AE reporting system provides patients, regulators, and referrers with confidence that procedures are safely delivered. Procedure-specific major AEs, number as a percentage of procedure number, is a key performance indicator that can be compared to indicative qualityimprovement thresholds published by IR societies. 2e4 Many IR departments use a hybrid model of prospective AE reporting including a computer-based reporting system of AE entered by IR staff supplemented by physician recording of AEs. The performance of this prospective method of AE reporting in IR is not known.
Improving systems to reduce AEs is important, and is consistent with the American College of Radiologists (ACR)'s Imaging 3.0 focus on quality and safety. 5 The Society of Interventional Radiology also recognises the importance of AEs and cites complication rates as one of the primary indicators of procedural success. 6 Ideally, this system of AE reporting should document all AEs. The purpose of this study was to assess the sensitivity of detection of AEs by an IR prospective reporting system.

Materials and methods
Institutional review board approval was received for this retrospective study and the requirement for patient consent was waived. Adult patients who underwent a fluoroscopyguided procedure in a university hospital performed by one of four fellowship-trained interventional radiologists (with 15,9,9, and 5 years of experience) from 1 May 2017 to 30 April 2019 were identified using a search of the radiology information system (RIS). All arterial fluoroscopy-guided procedures were included. Endovascular stent graft insertion for abdominal and iliac aneurysm repair performed in the operating room were excluded. The rationale for selecting arterial procedures was that on prior experience, AEs were expected more commonly in this procedural type than in venous or non-vascular IR procedures. An adequate sample size was calculated (95% confidence level) to contain data from at least 243 arterial procedures.

Data collection for reference standard list
A list of all fluoroscopy-guided IR procedure was generated using a RIS search. In the 2-year study period 11,731 consecutive fluoroscopy-guided IR procedures were performed in 9,598 patients. Five hundred and twenty-one arterial IR procedures were performed in 515 patients. Fifty endovascular abdominal and iliac aneurysm repair procedures in 50 patients were performed in the operating room with open surgical arterial access for the interventional radiologist and/or vascular surgeon and were excluded. Four hundred and seventy-one arterial IR procedures were performed in 465 patients (median age, 65 years; interquartile range, 54e77 years; 276 men) and were included in the study (Fig 1).
The medical record review was a two-stage process. In the primary review, the patient radiology record procedure notes, clinical notes and electronic medical record were reviewed by a trained medical student. Clinical records were reviewed from 1-year pre-procedure to 1-year postprocedure date. The reviewer was blind to the AE status of the patient. When a possible AE was identified, the demographics, procedure type, AE description, AE grade (according to the Society of Interventional Radiology AE classification system), 7 (Electronic Supplmentary Material Appendix A), AE subcategory (minor or major), timing of AE (early or late), and length of hospital admission were reported using a standardised computer-based abstraction form (Excel, Microsoft, Redmond, WA, USA) that was developed during the pilot study. Assessment of causation of the AE was not performed as part of this study. When patients had two or more procedures, each procedure was entered as a unique entry in the database. To ensure data quality throughout the study, the computer database underwent periodic quality assurance monitoring.
An AE was reported as early if it occurred in the immediate post-procedure period in the IR recovery area. A late AE occurred in the period after the patient had departed from the IR recovery area.
When using the Society of Interventional Radiology AE classification system, the delay in discharge was calculated using an expected admission duration for example, within 24 h for uterine artery embolisation (UFE) and arterial angioplasty, <18 h for yttrium-90 radio-embolisation. Embolisation procedures for acute haemorrhage were excluded from delayed discharge calculations because expected admission duration was complex and confounded by comorbidities.
In the secondary review, an interventional radiologist reviewed the identified possible AEs (n¼53) in the database. A medical record review was performed again to verify that the event was an AE and confirm the assigned grade.

Data collection for the departmental list of adverse events
During the study, patients were admitted and clinically followed by their primary referrer's clinical service with the exception of UFE patients who were admitted primarily under IR. Under the model of care, all interventional radiologists worked as diagnostic and interventional radiologists and did not perform inpatient or outpatient follow-up except for UFE patients. All unscheduled outpatient followup was performed by the referring clinical service.
The departmental radiology quality programme (which has been in effect since 2012) directed that radiology consultants and IR trainees are responsible for identifying and recording AEs. Trainees and IR fellows were educated at IR departmental induction that AEs were to be reported according to the national and departmental quality-assurance policy. 8 For ease of reference, the policy was available through the trainee and fellow radiology department guidebook. When an AE is identified, it is recorded using a RIS code "adverse event" generated by the radiographer. The interventional radiologist reported the AE details and outcome when known on the RIS. When an interventional radiologist became aware of an AE through direct communication from clinical staff, patient follow-up, or through review of imaging tests, the details were reported in the RIS as an "adverse event" entry or in the physicians' AE registry. AE reporting is mandatory in the institution. The physicians are actively encouraged by the healthcare organisation, team environment, and radiology management to perform open disclosure and AE reporting.
The departmental list of adverse events (DLAE) was generated for quality-improvement meetings by the IR quality lead and the fellow in IR. The DLAE report was a combination of an "adverse event" RIS enquiry and adverse event reports from the four IR consultants.
Monthly quality-improvement meetings, biannual audit days, and outcome meetings were used to reinforce education of staff regarding adverse event reporting. The departmental quality-improvement conference lists were used to identify the list of IR AEs that occurred during procedures performed in the study period. The same standardised computer-based abstraction form as above was used for recording AE details. The DLAE was assessed for completeness by (1) cross-reference to the original electronic records of AE submitted by the consultant interventional radiologist and (2) by a search performed of AE code entries on the RIS for the study period. Interventional radiologists were asked to review undetected major missed AE cases and confirm if they were aware of the event and had failed to report it.

Statistical analysis
Categorical variables are provided as absolute numbers and percentages. AE were categorised into minor and major complications of embolisation, transarterial chemoembolisation, angioplasty, yttrium-90 radio-embolisation, and UFE procedures for calculation of procedure-related AE rates. These rates were compared to the societal threshold AE rates for major complications (Table 1).
Continuous variables are provided as medians with interquartile ranges (IQRs). AEs identified in the DLAE (n¼20) were compared to the reference standard AE list (n¼38). Sensitivity and specificity values were calculated for detection of AEs by the quality-improvement system on a per-procedure basis using the reference standard list. Agreement values were calculated between the DLAE and reference standard AE list. A confidence interval was calculated with a p-value of 0.0001. A root-cause analysis was performed by the authors of AEs not included in the DLAE in order to identify factors that led to them not being recorded.

Modelling of the effect of late clinical follow-up
A model of AE reporting that included clinical followup of patients after departure from the IR area performed by the interventional radiologists was generated. The model assumed that all late AEs would be detected and recorded in the model DLAE. The rate of detection of model AEs was calculated and compared to the performance of the DLAE. The improvement in detection of AEs was tested for statistical significance using McNemar's chi-square test.

Reference standard complication rate
There were 38 AEs (including one death), which occurred in 38 patients (Electronic Supplementary Material Table S1). An AE occurred in 8% of all procedures. There were 20 minor and 18 major AEs. One patient death occurred within 24 h of embolisation for upper gastrointestinal haemorrhage due to small bowel ischaemia and multi-organ failure. A breakdown by procedure-specific AE rate is listed in Table 1. No procedure-specific quality-improvement threshold for AEs was exceeded.   Table 2).

Comparison of reference standard and DLAE rates of major and minor AEs
Fifty-three per cent of AEs were recorded by the DLAE. The DLAE recorded 72% and 35% of major and minor AEs, respectively. The additional AEs detected in the reference standard AE list lead to a twofold increase in the overall complication rate (4% versus 8%, p<0.0001).

Unreported early AEs
Review of the six unreported early AEs demonstrated that they were all arterial access related minor complications. In all six cases, the AE was detected by the interventional radiologist and recorded in the radiology reports but were not reported in the RIS or DLAE. Review suggests that the causes may be errors of omission (classified as an active error) and multiple latent errors. The error of omission occurred during the process of interventional radiologist communicating with the IR technologist and the IR technologist registering the adverse event RIS entry. The latent errors in the reporting of the AEs include an ineffective AE reporting organisational structure that is dependent on an IR technologist entering a code on a radiology specific RIS in the IR environment at the time of the procedure.
Root cause analysis of the six unrecorded early AEs suggested in all cases that the recording of the detected AE was an error of omission by IR.

Unreported late AEs
Review of the 12 unreported late AEs demonstrated that there were errors. Twelve late AE cases were detected by the clinical team and recorded in the clinical notes but not reported in the DLAE. There was an ineffective organisational structure; there was no process for reporting of an AE directly to the DLAE by the primary clinical team and there was no IR clinical follow-up.

Modelling of the effect of late clinical follow-up
The model suggested that the addition of late clinical follow-up to the model of care would record 32/38 AEs. By comparison to the DLAE reporting rate (20/38 AEs), integration of late clinical follow-up would significantly increase AE reporting (p¼0.0015).

Discussion
Prospective reporting of AEs by interventional radiologists using a RIS and personal record system significantly under-reports AEs by comparison to the reference standard. The system used in the present study detected 55% of all AEs, which is 11 times more than prior reports of similar prospective incident-based reporting using AE electronic registries. 9,10 The reference standard method for detecting AEs is a manual medical record review, but this is time-consuming, resource-intensive, and expensive. Manual review of medical records performed in this study took 280 h to generate the reference standard database. As a consequence, it is an impractical means for the routine detection and monitoring of AEs in IR practice. AE identification may be performed prospectively or retrospectively. Retrospective search methods include medical record review by non-medical trained staff or automated methods or discharge diagnostic code review by trained staff. Automated retrospective methods for detecting AEs are Table 1 Overall, minor and major adverse event number (percentage in parenthesis are a percentage of total procedure number) for arterial interventional radiology procedures using the reference standard report, departmental listing of adverse events and comparison to the Society of Interventional Radiology major complication threshold.  possible using natural language processing of electronic health records and threshold value triggers for diagnostic tests to detect AEs such as hospital-acquired pneumonia.
There are no commercially available automated methods to process IR AEs. Health record review can be triggered by incident and accident reports (studies have found that compared to manual chart review they underestimate the true incidence of AEs by a factor of about 20 9,10 ) or after a specific discharge diagnostic code has been detected (this has a low to moderate sensitivity and PPV for AEs 11,12 ). Artificial intelligence may offer better methods for detection of AEs in IR. Automated methods for encoding and classifying electronic clinical narratives in an electronic health record (EHR) and digital capture of progress notes and other clinical results will provide data for trigger tools such as the Harvard Medical Practice Study trigger system to identify AEs from the patient record for review. 13 Discharge coding of patient medical records can also be used to identify AEs. Among patients undergoing inpatient surgical procedures at Department of Veterans Affairs (DVA) medical centres, natural language processing analysis of electronic medical records to identify postoperative complications had higher sensitivity and lower specificity compared with patient safety indicators based on discharge coding. 14 The root cause analysis of the missed early AEs suggested system errors that could be targeted for improvement. Organisational barriers to AE reporting included the AE RIS entry system that was only accessible by IR technologists within the radiology department.
An optimal AE reporting system would be a searchable registry integrated with the radiology workflow that would allow direct reporting of AEs by all clinical staff on multiple platforms. Integration with internal systems, such as radiology dictation software and electronic medical records, was found to improve AE reporting compliance. 15 The system of AE collection needs to integrate clinical follow-up detected AE. Potential solutions include development of a system for educating clinical teams regarding the system for AE submission to IR, reducing the organisational barriers to AE submission, primary admission by the IR or including clinical follow-up by IR in the model of care. Clinical IR follow-up could be resourced by increased IR team staffing, inclusion of inpatient follow-up rounds and outpatient clinic time in the interventional radiologists' job plan or delegation to an IR service nurse practitioner.
Previous studies have suggested organisational interventions to increase AE reporting. Auditing and an escalation process with automated reminder email, text, and physician contact can improve online self-reported IR quality-assurance database compliance with record entry and improve the final AE reporting rates. 16 AE reporting can also be increased using a structured monthly mortality and morbidity conference to review AE confirming whether each complication warranted institutional AE reporting and whether timely reporting had occurred. 17 Other organisational interventions proven to improve safety and lead to quality improvement, such as introduction of a procedural checklist, have not been evaluated in AE reporting. Checklists such as the Cardiovascular and Interventional Radiological Society of Europe (CIRSE) IR patient safety checklist and World Health Organization Surgical Safety Checklist have been introduced to improve patient safety. 18,19 Use of a procedural checklist can result in a significant reduction in the rate of death and complications during surgery. 18,20 The patient safety checklist can be modified locally to include an item at "sign out" focusing on AE reporting and quality assurance.
Retrospective medical record review may be biased as AE interpretation is subjective and may not be externally reproducible. The study cannot correct for systematic underreporting due to concerns about possible punitive action regarding errors if they are reported, how submitting errors could affect relationships with colleagues, and whether the content of reviews will be peer-review protected. The model evaluating the effect of integration of late clinical follow-up in the AE system assumes that all additionally detected AE would be recorded by in the DLAE.
In conclusion, a prospective AE reporting system without clinical integration will not detect all procedure complications.