How best to measure surgical quality? comparison of the Agency for Healthcare Research and Quality Patient Safety Indicators (AHRQ-PSI) and the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) postoperative adverse events at a single institution

Background. Evaluating surgical outcomes is an important tool to compare providers and institutions and to drive process improvements. Differing methodologies, however; may provide conflicting measurements of similar clinical outcomes making comparisons difficult. ACS-NSQIP is a validated, risk-adjusted, clinically derived data methodology to compare observed to expected outcomes after a wide variety of operations. The AHRQ-PSI are a set of computer algorithms to identify potential adverse in-patient events using secondary ICD-9-CM diagnosis and procedure codes from hospital discharge abstracts. Methods. We compared the ACS-NSQIP and AHRQ-PSI methods for hospital general surgical (n = 6565) or vascular surgical inpatients procedures (n = 1041) at a tertiary-care academic institution from April 2006 to June 2009 on 7 adverse event types. Results. ACS-NSQIP inpatient adverse events were identified in 564 (7.4%) patients. AHRQ-PSIs were identified in 268 (3.5 %) patients. Only 159 (2.1%) patients had inpatient events identified by both methods. Using ACS-NSQIP as the clinically based standard the sensitivity of the specific AHRQ-PSI ranged from 0.030 for infections to 0.535 for PE/DVT Positive predictive values of AHRQ-PSI ranged from 18% for hemorrhage/hentatoma to 89% for renal failure. Greater agreement at greater ASA class and wound classification was observed. Conclusion. AHRQ-PSI algorithms identified less than a third of the ACS-NSQIP clinically important adverse events. Furthermore, the AHRQ-PSI identified a large number of events with no corresponding clinically important adverse outcomes. The sensitivity of the AHRQ-PSI for detecting clinically relevant adverse events identified by the ACS-NSQIP varied widely. The AHRQ-PSI as applied to postoperative patients is a poor measure of quality performance. (Surgery 2011;150:943-9.)