Chapter 7. Assessment of Cardiac Operations to Improve Performance

Historical—Hunter, Nightingale, Codman, and Cochrane

It may seem a strange principle to enunciate as the very first requirement in a Hospital that it should do the sick no harm. It is quite necessary, nevertheless, to lay down such a principle, because the actual mortality in hospitals…is very much higher than…the mortality of the same class of diseases among patients treated out of hospital…

Florence Nightingale, 1863

Surgeons often assume that performance improvement is a new concept arising from the complexities of modern medicine and surgery. Assessing risk to improve outcomes is not a new principle. The formal assessment of patient care had its beginnings in the mid-1800s.One of the earliest advocates of analyzing outcome data was Florence Nightingale, who was troubled by observations that hospitalized patients died at higher rates than those treated out of hospital.1 She also noted a vast difference in mortality rates among different English hospitals, with London hospitals having as high as a 90% mortality rate, whereas smaller rural hospitals had a much lower mortality rate (12 to15%).Although England tracked hospital mortality rates since the 1600s, the analysis of these rates was in its infancy. Nightingale made the important observation that raw mortality rates were not an accurate reflection of outcome, because some patients were sicker when they presented to the hospital and, therefore, would be expected to have a higher mortality. Nightingale translated these observations into a highly successful improvement plan by suggesting simple measures such as better sanitation, less crowding, and location of hospitals distant from crowded urban areas. Her observations and action plans likely represent the beginning of risk adjustment to implement performance improvement.

Earlier contributions to performance improvement came from John Hunter in the 1700s. Hunter was relatively uneducated by the standards of the time but he had two qualities that allowed him to become the pre-eminent surgeon of the time and the father of modern surgery.2 He had great technical skill and a belief that disease was caused by anatomic abnormalities. More importantly, he was unwilling to accept hypothetical abstract explanations of illness like “humors” or “spirits.” He required that he verified causes of illness himself and that he could explain them on the basis of anatomy. For example, treatment of venereal disease was an important part of a physician's practice in the mid-1700s.Hunter was outspoken about the failure of the majority of medications claiming to cure gonorrhea. He said that “gonorrhea could be cured by the most ignorant, because gonorrhea mostly cures itself.” He even performed an ingenious test by treating some of his patients with pills made of bread. He recorded the results and almost all of the patients had resolution of gonorrhea. This was one of the first trials documenting the placebo effect. John Hunter's friends and patients included Benjamin Franklin, Edward Jenner, Lord Byron, Casanova, and Adam Smith. His list of enemies was probably equally as distinguished. What is clear is that the assessment of outcomes of illness could not advance without understanding the basic anatomic disease substrate that John Hunter proposed.

Ernest Amory Codman, a Boston surgeon, was an early advocate of outcome analysis and scrutiny of results. Codman was a classmate of Harvey Cushing and became interested in the issues of outcome analysis after a friendly bet with Cushing about who had the lowest complication rate with the delivery of anesthesia. This effort not only represented the first intraoperative patient records, but also served as a foundation for Codman's later interest (almost passion) for documentation of outcomes. Codman actually paid a publisher to disseminate the results obtained in his privately owned Boston hospital.3 Codman was perhaps the first advocate of searching for a cause of all complications. He felt that most bad outcomes were ultimately the result of physician errors or omissions and completely ignored any contribution from hospital-based and process-related factors. His efforts were not well received by his peers, and eventually his private hospital closed because of lack of referrals.

Further definition of outcome assessment occurred in the mid-1900s.As more and more therapeutic options became available to treat the diseases that predominated in the early twentieth century, a need arose to determine the best alternative of multiple therapies. Randomized trials were first used in the 1940sto test hypotheses. One of the earliest randomized trials was conducted to determine whether streptomycin was effective against tuberculosis.4 Although the trial proved the effectiveness of streptomycin against tuberculosis, it stimulated a great deal of controversy. After World War II, several clinicians advocated the use of randomized, controlled trials (RCTs) to better identify the optimal treatment to provide the best outcome. Foremost among these was Archie Cochrane (Fig. 7-1).Cochrane lived during an exciting time. He lobbied for a national health service in Great Britain but this advocacy was tempered by 4 years as a prisoner of war in multiple WWII German POW camps. He saw soldiers die from tuberculosis and he was never sure what the best treatment was. He could choose between collapse therapy, bed rest, supplemental nutrition, or even high-dose vitamin therapy. A quote from his book sums up his frustration.5

I had considerable freedom of clinical choice of therapy: my trouble was that I did not know which to use and when. I would gladly have sacrificed my freedom for a little knowledge.

Cochrane, 1971

His experience with the uncertainty about the best treatment for TB and other chest diseases continued after the war, when he became a patron of randomized controlled trials (or RCTs as he liked to refer to them) to test important medical hypotheses. In 1987, the year before Cochrane died, he referred to a systematic review of RCTs of care during pregnancy and childbirth as “a real milestone in the history of randomized trials and in the evaluation of care. “He suggested that other specialties should copy the methods used. This led to the opening of the first Cochrane center (in Oxford, UK) in 1992 and the founding of the Cochrane Collaboration in 1993.Today the Cochrane Collaboration is a repository of RCTs, both disease-specific and specialty-specific (http://www.cochrane.org).The ascendancy of RCTs represents a major step forward in outcome assessment to improve surgical operations.

Measures of Successful Operations

Performance Measures—Outcomes, Structure, and Process

In the early 1960s Donabedian suggested that quality in health care is defined as improvement in patient status after accounting for the patient's severity of illness, presence of comorbidity, and the medical services received.6 He further proposed that quality could best be measured by considering three domains: structure, process, and outcome. Only recently has the notion of measuring healthcare quality using this Donabedian framework been accepted and implemented. In 2000, the Institute of Medicine (IOM) issued a report that was highly critical of the U.S. health-care system, suggesting that between 50,000 and 90,000 unnecessary deaths occur yearly because of errors in the health-care system.7 The Institute of Medicine reports created a heightened awareness of more global aspects of quality. For most of the history of cardiac surgery, quality was equated with operative mortality (ie, outcome measure) .After the IOM report appeared, a distinct change in the landscape of quality assessment occurred, and other aspects of Donabedian's framework were utilized. The narrow focus on operative mortality gave way to a broader analysis that also included additional outcome measures falling under the general category of operative morbidity. The emphasis on such outcomes measures expanded to include the processes of surgical care delivery. Process measures monitor provider compliance with desirable, often evidence-based, processes and systems of care. These measures typically include the choice of medication, timing of antibiotic administration, use of surgical techniques such as the internal thoracic artery graft, and other interventions considered appropriate for optimal care. Finally, structural measures such as the use of health information technology, physical plant design, participation in systematic clinical registries, and procedural volume are also considered important elements in this more global model of medical quality and performance. All these measures fall under the general rubric of performance measures. Birkmeyer and coworkers enumerated certain advantages and disadvantages associated with each specific type of performance measure.8 For example, the fact that structural measures can be readily tabulated in an inexpensive manner using administrative data is a distinct advantage. On the other hand, many structural measures do not lend themselves to alteration. Particularly in smaller hospitals, there may simply be no way to increase procedural volume or to introduce costly design changes in an attempt to improve their performance on structural measures. Attempts to alter structure might even have adverse consequences (eg, unnecessary operations, costly and unnecessary new beds).Process measures are often linked to health-care quality and they are usually actionable on a practical level. Their major disadvantage lies in the fact that process measures may not be generally applicable to all patients undergoing a given procedure, and their linkage to outcomes may be weak. Outcomes measures are the most important endpoint for patients, but accurate assessment of outcomes is often limited by inadequate sample size and lack of appropriate risk adjustment.

The following definitions are adapted from those proposed by the Joint Commission on Accreditation of Healthcare Organizations (formerly JCAHO now officially called The Joint Commission):

• Performance measure: A quantitative entity that provides an indication of an organization's performance in relation to a specified process or outcome
• Outcome measure: A measure that indicates the results of process measures. Examples are operative mortality, the frequency of postoperative mediastinitis, renal failure, myocardial infarction, etc.
• Process measure: A measure that focuses on a process leading to a certain outcome. Intrinsic in this definition is a scientific basis for believing that the process will increase the probability of achieving a desired outcome. Examples include the rate of IMA (CABG) patients or the fraction of CABG patients placed on beta-blocking agents postoperatively.
• Structural measure: A measure that assesses whether an appropriate number, type, and distribution of medical personnel, equipment, and/or facilities are in place to deliver optimal health care. Examples include enrollment in a national database or procedural volume.

Although there are numerous proposed criteria for an ideal performance measure, the following are generally accepted attributes:

• The measure must be clearly linked to quality of care.
• The measure must be objective, evidence-based, and risk-adjusted if possible.
• The physician must have the ability to influence the measure in a clinical setting.
• The measure must be sufficiently well defined to permit practical measurement.

Several national organizations are specifically devoted to the rigorous development and evaluation of performance measures. Perhaps the most visible of these is the National Quality Forum (NQF), a public-private collaborative organization that uses a process of exhaustive, evidence-based scrutiny of candidate measures to determine their relevance to both patients and health-care providers. The NQF considers whether candidate measures can be measured accurately and whether actionable interventions can improve performance for candidate measures. Having gone through this “trial by fire,” NQF-endorsed measures have a high level of national credibility not otherwise possible.

Patient Satisfaction

Other outcomes following cardiac procedures, such as patient satisfaction and health-related quality of life, are less well studied but extremely important in the assessment of performance. Meeting or exceeding patients' expectations is a major goal of the health-care system. The growing importance of patient-reported outcomes also reflects the increasing prevalence of chronic disease in our aging population. The goal of therapeutic interventions is often to relieve symptoms and improve quality of life, rather than cure a disease and prolong survival. This is especially important in selecting elderly patients for cardiac operations. One early report from the United Kingdom suggested that as many as one third of patients over the age of 70 had no improvement in their disability and overall sense of well-being after cardiac surgery.9 More recent studies suggest that CABG results in excellent health-related quality of life 10 to 15 years after surgery in most patients,10 and that this benefit extends to those patients older than 80 years of age.11 Future research into patient-perceived performance assessment is inevitable given the aging of populations in the developed countries.

Risk Stratification and Risk Models

Essential for the assessment of the success of cardiac operations is the ability to arrange patients according to their severity of illness. Viewed simplistically, patients who are sicker before operation have a greater chance of a poor outcome. Various risk-adjustment systems assess and adjust for the incremental risk associated with specific preoperative factors variously known as risk factors, risk predictors, comorbidities, or covariates. These are discussed in detail in Chapter 8. Table 7-1 is a partial listing of some of the severity measures commonly used in risk assessment of patients undergoing cardiac procedures. The risk stratification systems listed in Table 7-1 are in constant evolution, and the descriptions in the table may not reflect current or future versions of these systems. For example, in 2009 the Society of Thoracic Surgeons published the most extensive, comprehensive set of cardiac surgery risk models yet available. Twenty-seven risk models encompassed nine endpoints for each of three major groups of cardiac procedures (isolated CABG, isolated valve, and valve + CABG).12-14

Most risk-adjustment algorithms share several common features. First, the risk factors or covariates in the model are associated with a specific outcome. Second, if the goal is to measure provider performance, the risk factors include only patient characteristics (not hospital, physician, or regional characteristics) present prior to surgery, not factors that might be influenced by the health-care provider (eg, what type of cardioplegia was used).15 Third, a sufficient number of patients must have the risk factor, and a sufficient number must experience the adverse outcome, in order to construct the risk models. Finally, it is necessary to define the period of observation for the outcomes of interest (eg, in-hospital or 30-day mortality).

The severity indices listed in Table 7-1 define severity predominantly based on clinical measures (eg, risk of death or other adverse clinical outcome).At least, two of the severity measures shown in Table 7-1 (MedisGroups used in the Pennsylvania Cardiac Surgery Reporting System and the Canadian Provincial Adult Cardiac Care Network of Ontario) assess severity based on resource use (eg, hospital length-of-stay, cost) as well as on clinical measures.16,17 Of the nine severity measures listed in Table 7-1, only one, the APACHE III system, computes a risk score independent of patient diagnosis.18 All of the others in the table are diagnosis-specific systems that use only patients with particular diagnoses in computing severity scores.

Once developed from a reference population, each of the risk stratification measures shown in Table 7-1 are validated against a different set of patients to assure that they are adequate predictors of the population risk of operative mortality or other outcomes. Although useful, these risk models are imperfect.19 There are too many individual patient and procedural differences, many of them unknown or unmeasured, to allow completely accurate preoperative risk assessment. Dupuis and coworkers suggest that risk stratification and risk models are good predictors of population risk but not of individual patient risk.19 The most important reason that risk-adjustment methods fail to completely predict individual outcomes is that the data set used to derive the risk score comes from retrospective, observational data that contain inherent selection bias (ie, patients were given a certain treatment that resulted in a particular outcome because a clinician had a selection bias about what treatment that particular patient should receive.)In observational datasets, patients are not allocated to a given treatment in a randomized manner. In addition, clinician bias is not always founded in evidence-based data. Methods are available that attempt to overcome some of these limitations of observational data. These methods include use of instrumental variables and propensity scores.20,21 Observational datasets are much more readily available and represent “real-world” treatment and outcomes compared with RCTs.An excellent review of the subtleties of evaluating the performance of risk-adjustment methods is given in the book by Iezzoni, and this reference is recommended to the interested reader.22

Ideally, differences in risk-adjusted outcomes result from differences in quality of care, but this is not necessarily the case. One study simulated the mortality experience for a hypothetical set of hospitals assuming perfect risk adjustment and with prior perfect knowledge of poor quality providers.23 These authors used various simulation models, including Monte Carlo simulation, and found that under all reasonable assumptions, sensitivity for determining poor quality was less than 20% and the predictive error for determining high outliers was greater than 50%.Much of the observed mortality rate differences between high outliers and nonoutliers were attributable to random variation. Park and coauthors suggest that providers identified as high outliers using conventional risk adjustment methods do not provide lower quality care than do nonoutliers.24

Comorbidities as Risk Factors

Comorbidities are coexisting diagnoses that are indirectly related to the principal surgical diagnosis but that may alter the outcome of operations. These are vitally important because physicians or hospitals that care for patients with a higher prevalence of serious comorbid conditions are at significant disadvantage in unadjusted comparisons. Comorbid illness in patients with cardiac disease is common. In one series of patients with myocardial infarction, 26% also had diabetes, 30% had arthritis, 6%had chronic lung problems and 12% had gastrointestinal disorders.25

Several generic indices of comorbidity are available for general medical and surgical diagnoses, although none are as specific or detailed as the risk models used in cardiac surgery. Table 7-2 compares 13 commonly used comorbidity measures. The Charlson Index, the CIRS, the ICED, and the Kaplan Index are valid and reliable measures of comorbidity as measured in certain specific patient populations, but not in patients undergoing cardiac operations.26 The other nine comorbidity measures in Table 7-2 do not have sufficient data to assess their validity and reliability and are probably less useful than the four validated measures. There are many limitations of comorbidity indices, and they are not applied widely in studies of efficacy or medical effectiveness for cardiac operations.

Perhaps the most serious drawback of comorbidity scoring systems is the imprecision of the data used to form the indices. Most of the data used to construct the indices comes from two sources:(1) administrative databases in the form of computerized discharge abstract data and (2) out-of-hospital follow-up reports including questionnaires and medical chart reviews. Administrative data in the form of discharge abstracts include clinical diagnoses that are often assigned by nonphysicians not involved in the care of the patient. Inaccuracies in discharge coding, and in administrative databases in general, are common.27,28 Less serious diagnoses are often not coded for in the most seriously ill patients. Additionally, the accuracy of out-of-hospital follow-up studies is hard to validate and these studies may contain significant inaccuracies. Despite their shortcomings, analyses that compare physician or hospital outcomes and that do not provide some adjustment, albeit imperfect, for patient comorbidity will likely discriminate against providers or hospitals that treat disproportionate numbers of elderly patients with multiple comorbid conditions.

The Performance of Risk Models

Many risk models for cardiac operations are used to assess surgical performance (Table 7-3).Before a risk model and its component risk factors are used to evaluate provider performance, these models are tested for accuracy. Many patient variables are candidate risk factors for operative mortality following coronary revascularization. Examples include serum BUN, cachexia, oxygen delivery, HIV, case volume, low hematocrit on bypass, the diameter of the coronary artery, and resident involvement in the operation. On the surface, these variables seem like valid risk factors, but many are not. All putative risk factors should be subjected to rigorous scrutiny like those variables listed in Table 7-3. The regression diagnostics (eg, receiver operating characteristics (ROC) curves and cross-validation studies) performed on the models included in Tables 7-1 and 7-3 suggest that the models are good, but not perfect, at predicting outcomes. In statistical terms this means that all of the variability in operative mortality is not explained by the set of risk factors included in the regression models. Hence, it is possible that inclusion of new putative risk factors in the regression equations may improve the validity and precision of the models. New regression models, and new risk factors, must be scrutinized and tested using cross-validation methods and other regression diagnostics before acceptance.

It is uncertain whether inclusion of many more risk factors will significantly improve the quality and predictive ability of regression models, and there is an ongoing tension between parsimonious models and those that contain many variables.For example, the Society for Thoracic Surgeons (STS) risk stratification model, described in Tables 7-1 and 7-3, includes many predictor variables, whereas the Toronto risk adjustment scheme includes only five predictor variables (see Table 7-3).Yet the regression diagnostics for these two models are similar, suggesting that both models have equal precision and predictive capabilities. Studies show that much of the predictive ability of risk models is contained in a relatively few number of risk factors.29,30 Other studies suggest that the limiting factor in the accuracy of current risk models may be failure to understand and account for all the important factors related to risk.31 Additionally, risk models are useful for predicting the average outcome for a population of patients with specific risk factors, but not necessarily accurate for predicting the outcome for a specific patient. Further work needs to be done, both to explain the differences in risk factors seen among the various risk stratification models and to determine which models are best suited for studies of quality improvement and performance assessment.

Occasionally negative studies appear that suggest that a particular patient variable is not a risk factor for a particular patient outcome. Care must be exercised in interpreting negative results.32 Many putative risk factors labeled as “no different from control” in studies using inadequate samples have not received a fair test. For example, Burns and associates studied preoperative template bleeding times in 43 patients undergoing elective CABG.33 They found no increased postoperative blood loss in patients with prolonged skin bleeding times. Their study reports five patients whose bleeding times were prolonged greater than 8 minutes. In this small sample size there was a trend toward more units of blood transfused, but differences between high and low bleeding time groups were reported as “not significant” by the authors at the α= 5% level (ie, p = 0.05).Using the author's data, it is possible to compute a β error for this negative observation of less than 0.5.This means that there is as much as a 50% chance that the negative finding is really a false-negative result. This high false-negative rate occurs because of the small sample size and the wide variation in the bleeding time values. We have found elevated bleeding time (>10 minutes) to be a significant multivariate risk factor for excessive blood transfusion after CABG in two different studies.34,35 Although there is controversy about the value of bleeding time as a screening test,36 it is possible that disregarding the bleeding time after an inconclusive negative trial, such as that of Burns and coworkers, may ignore a potentially important risk factor. Care must be taken in the interpretation of a negative finding, especially in a small study group. Freiman and coauthors sounded a similar cautionary note after reviewing the medical literature over a ten-year period.37 These authors found that 50 of 71 “negative” randomized controlled trials could have missed a 50% therapeutic improvement from intervention because the sample size studied was too small.37 Their conclusions were that many therapies discarded as ineffective after inconclusive negative trials may still have a clinically important effect. Today journal editors and authors are much more aware of the relationships among sample size, statistical power, and negative test results compared with the earlier report of Freiman and coauthors. Nonetheless, negative findings in a literature report require scrutiny and an understanding of type II statistical errors.38 Finally, some risk factors may not appear to improve the discrimination of a particular model, but they may still be very important predictors of outcome if present in a particular patient.39

Assessing Quality of Care—Underuse, Misuse, and Overuse

Assessing the quality of cardiac care is a worthy goal of measuring performance. However, this goal is elusive and a little bit like trying to define “pornography”—everyone knows it when they see it, but it is very hard to define. A major problem arises in attaining this goal because uniform definitions of quality of care are not available. Performance measures are a means of assessing cardiac surgical performance. Providers who do not meet the performance standards outlined by these measures are guilty of misuse of health care resources. But there are other indices of health-care quality not covered by these measures, including appropriateness of care and disparities in care (eg, women and minorities receiving worse care).Inappropriate use of procedures is often referred to as overuse, and failure to provide indicated care as underuse. Both are found in treatment of cardiovascular diseases. For example, there is substantial geographic variation in the rates at which patients with cardiovascular diseases undergo diagnostic procedures, with little, if any, evidence that these variations are related to survival or improved outcome. In one study, coronary angiography was performed in 45% of patients after acute myocardial infarction for patients in Texas compared with 30 for patients in New York State (p < 0.001 for comparison between states).40 Another study showed large variations in care delivered to patients having cardiac operations.41 Among six institutions that treated very similar patients (Veteran's Administration Medical Centers), there were large differences in the percentage of elective, urgent, and emergent cases at each institution, ranging from 58to 96% elective, 3 to 31% urgent, and 1 to 8% emergent.41 There was also a tenfold difference in the preoperative use of intra-aortic balloon counterpulsation for control of unstable angina, varying from 0.8 to 10.6%.41 Similar variations in physician-specific practices exist for mitral valve procedures, carotid end arterectomy, and blood transfusion during cardiac procedures.42-44 This variation in clinical practice may reflect uncertainty about the efficacy of available interventions, or differences in practitioners' clinical judgment.

Although wide differences in use of cardiac interventions initially fueled charges of overuse in certain areas,45 recent evaluations suggest that underuse of indicated cardiac interventions (either PTCA or CABG) may be a cause of this variation.46,47 Whether caused by underuse or overuse of cardiovascular services, regional variations in resource utilization suggest that a rigorous definition of the “correct” treatment of acute myocardial infarction, as in other cardiovascular disease states, is elusive and the definition of quality of care for such patients is imperfect. Regional variations in cardiovascular care delivery are only a few of the examples of unclear best practices. Age, gender, race, community size, and hospital characteristics influence utilization of diagnostic and operative interventions without much evidence that these factors should direct the appropriate treatment for various cardiac disorders.48,49 Although performance measures that define performance assessment may be a way to judge quality of care among providers, much more work needs to be done to define best practices and to limit practice variations before performance measures reflect quality of care.

Improving Quality of Care (Expert Guidelines)

Recognizing the difficulties in defining “best practices” for a given illness, professional organizations opted to promote practice guidelines or “suggested therapy” for given diseases.50,51 These practice guidelines represent a compilation of available published evidence, including randomized trials and risk-adjusted observational studies, as well as consensus among panels of experts proficient at treating a given disease. For example, the practice guideline for coronary artery bypass grafting is available for both practitioners and the lay public on the Internet (http://www.cardiosource.org/Science-And-Quality/Practice-Guidelines-and-Quality-Standards/2011/Coronary-Artery-Bypass-Graft-Surgery-CABG-Guideline-Update-for.aspx). Guidelines are a list of recommendations that have varying support in the literature. The strength of a guideline recommendation is often graded by class and by level of evidence used to support the class of recommendation. A typical rating scheme used by the Society of Thoracic Surgeons Evidence Based Workforce and by the Joint American College of Cardiology/American Heart Association Task Force on Practice Guidelines has three classes of recommendations as follows:

• Class I—Conditions for which there is evidence and/or general agreement that a given procedure or treatment is beneficial, useful, and effective.
• Class II—Conditions for which there is conflicting evidence and/or a divergence of opinion about the usefulness/efficacy of a procedure or treatment.
  • Class IIa—Weight of evidence/opinion is in favor of usefulness/efficacy.
  • Class IIb—Usefulness/efficacy is less well established by evidence/opinion.
• Class III—Conditions for which there is evidence and/or general agreement that a procedure/treatment is not useful/effective and in some cases may be harmful.

Evidence supporting the various classes of recommendations ranges from high quality randomized controlled trials to consensus opinion of experts. Three categories describe the level of evidence used to arrive at the class of recommendation:

• Level A—Data derived from multiple randomized clinical trials or meta-analyses
• Level B—Data derived from a single randomized trial, or nonrandomized studies
• Level C—Consensus opinion of experts, case studies, or standard-of-care

Guidelines provide surgeons with accepted evidence-based standards of care that most would agree on, with an ultimate goal of limiting deviations from accepted standards. Guideline development represents a work in progress, and the best approach has yet to be determined. Many recommendations that appear in professional guidelines are developed from lower levels of evidence including expert consensus.52 Many published guidelines do not adhere to currently accepted standards for developing guidelines,53 and barriers to acceptance of practice guidelines exist.54 For example, guidelines for the management of bleeding during cardiac procedures are available but are relatively unsuccessful in limiting nonautologous blood transfusion.55 There is still marked variability in transfusion practices and blood conservation interventions.56 These limitations of the effectiveness of guidelines in altering physician practices are a drawback of the guideline efforts, and the most important aspect of practice guideline development is their implementation in real-world practice.

Efficacy Studies (RCTs) versus Effectiveness (Observational) Studies

There are many efficacy studies relating to cardiothoracic surgery. These studies attempt to isolate one procedure or device as beneficial for patient outcome. The study population in efficacy studies is specifically chosen to contain as uniform a group as possible. Typical examples of efficacy studies include randomized, prospective, clinical trials (RCTs) comparing use of a procedure or device in a well-defined population compared with an equally well-defined control population. Efficacy studies are different than effectiveness studies.5 The latter deal with whole populations and attempt to determine the best treatment option that provides optimal outcome in a population that would typically be treated by a practicing surgeon. An example of an effectiveness study is a retrospective study of outcome in a large population treated with a particular intervention (eg, heart valve or blood conservation measure).Risk stratification is capable of isolating associations between outcome and risk factors. Methodological enhancements in risk adjustment, such as propensity matching,57 are capable of reducing biases inherent in population-based, retrospective studies but can never eliminate all confounding biases in observational studies.

One reasonable strategy for using risk stratification to improve patient care is to isolate high-risk subsets from population-based, retrospective studies (ie, effectiveness studies) and then to test interventions to improve outcome in high-risk subsets using RCTs. This is a strategy that should ultimately lead to the desired goal of improved patient care. For example, a population-based study on postoperative blood transfusion revealed that the following factors were significantly associated with excessive blood transfusion (defined as more than 4 units of blood products after CABG):(1) template bleeding time, (2) red blood cell volume, (3) cardiopulmonary bypass time, and (4) advanced age.35 Based on these retrospective effectiveness studies, investigators hypothesized that interventions aimed at reducing blood transfusion after CABG would most likely benefit patients with prolonged bleeding time and low red blood cell volume. A prospective clinical trial tested this hypothesis using two blood conservation techniques, platelet-rich plasma saving and normovolemic hemodilution of whole blood, in patients undergoing CABG. Results showed that blood conservation interventions reduced bleeding and blood transfusion only in the high-risk subset of patients.34 These studies imply that more costly interventions such as use of platelet-rich plasma savers are more efficacious in high-risk patients, with the high-risk subset defined by risk stratification methodologies.

Other Goals of Outcome Analysis: Cost Containment and Altering Physician Practices

Financial factors are a major force behind health care reform. America's health care costs amount to 15 to 20% of the gross national product and this figure is rising at an unsustainable rate of 6% annually. Institutions that pay for health care are demanding change, and these demands are fueled by studies that suggest that 20 to 30% of care is inappropriate with services both underused and overused compared with evidence-based practice standards.58 This resulted in a shift in emphasis, with health-care costs being emphasized on equal footing with clinical outcomes of care. Sometimes the two are combined into one number, value, which is quality divided by cost.

As noted previously, variations in physician practice distort the allocation of health-care funds in an inappropriate way. Research suggests that there is a 17-year lag time between medical discovery and when most patients benefit from the discovery.59 The failure on the part of some surgeons to implement innovation has a huge cost in terms of morbidity and mortality.60 Solutions to this problem involve altering physician practice patterns to be consistent with best available evidence, something that has been extremely difficult to do.61 How can physician practice patterns be changed in order to improve outcome? Evidence suggests that the typical process of outcome assessment, the case-by-case review (traditionally done in the morbidity and mortality [M and M] conference format), may not be cost-effective, may not change surgeon practice patterns, and may not improve quality.62 Health care experts suggest that the M and M conference should be replaced by external accountability using profiles of practice patterns at institutional, regional, or national levels. One alternative model for quality improvement involves oversight that emphasizes the appropriate balance between internal mechanisms for quality assessment and improvement and external accountability.62 Risk adjustment is an essential element particularly for the latter approach so that surgeons who operate on high-risk patients are not penalized for their potentially higher rate of adverse outcomes. Accurate risk assessment is essential for fairness and to assure acceptance by all stakeholders.

Rewarding High Performers (“Pay for Performance”)

Performance measures are regarded as the cornerstone of quality assessment. Identifying performance measures and undertaking steps to comply with these measures involves a management philosophy that seeks to integrate organizational functions (eg, technical aspects of care delivery and customer service)in order to meet customer needs. Beginning a quality initiative of this sort typically employs the following steps:

• A patient population is specified.
• Process measures are developed to serve as quality metrics for this population.
• The process measures are collected.
• Compliance with performance measures is tracked for participating centers.
• Aggregate results are used to establish norms and benchmarks.
• Participating centers receive data feedback compared with benchmarks.
• Participating centers are evaluated for their adherence to performance measures.

It becomes immediately apparent that the validity of this approach is primarily dependent on the validity of the performance measures. This recognition led to the emergence of several national organizations specifically devoted to the rigorous development of performance measures, most notably the National Quality Forum (NQF).

The NQF, with the assistance of the Society of Thoracic Surgeons, played a central role in developing national consensus standards for adult cardiac surgery. This process yielded 20 NQF-endorsed measures (Table 7-4).These measures constitute a unique set of nationally accepted tools that can be used to determine the quality of care in cardiac surgical programs. Prior to their development, there was no general agreement on the specific parameters linked to “quality” in cardiac surgery. NQF measures provide the “toolbox” for quality measurement in cardiac surgery and are used by payors to determine quality and guide resources to high performers.

The STS Database provides benchmark values for all of the performance measures in Table 7-4. In the most direct application of NQF measures, an institution simply compares its values against the benchmark values. This process allows providers to pinpoint opportunities for improvement so that resources can be specifically directed to areas of greatest need with the expectation that payors will provide increased resources to those institutions that meet the standards.

More sophisticated approaches to performance assessment involve combining process, structural, and outcome measures into a composite measure,63 one of the most important recent trends in performance measurement.Composite models are strongly endorsed by the Institute of Medicine. They provide a much more comprehensive assessment of provider quality than any single measure, and they demonstrate relative quality comparisons between an institution and a national benchmark. Composite models are preferred, because it is possible to excel in one specific area of performance but not in another. For example, it is possible to have low operative mortality but have infrequent use of internal mammary artery bypass in patients undergoing CABG, and the latter adversely impacts a patient's long-term outcome. A composite measure accounts for both of these domains of performance.

Many believe that additional incentives for quality improvement can be obtained by linking quality scores to reimbursement.64 This concept, commonly called pay-for-performance (P4P) or value-based purchasing, is supported by a variety of organizations. Effective performance-based payments show positive results in private industrial applications and, in spite of the absence of convincing evidence, there is widespread belief that similar results can be obtained in medicine.P4P is particularly popular amongst third-party payers. Historically payment was based on the number and the complexity of services provided to patients, but with P4P, some portion of payment is determined by the quality rather than the quantity of services. It remains to be seen whether reimbursement incentives will lead to meaningful improvements in quality of care.

There are several reimbursement models associated with P4P, the most common of which is the tournament model. In the tournament approach, there are unequivocal winners and losers. Top performers get bonuses which come from reduced payments to the lower performers. Although popular because of its simplicity, this budget-neutral approach in which one “robs Peter to pay Paul,” penalizes precisely the group which most needs financial resources for improvement.65 Regardless of the mode of implementation, it is obvious that performance measures are destined to be an important and intrinsic part of the surgical milieu in upcoming years.

Assessment Using Operative Mortality

By far, the bulk of available experience with outcome assessment in cardiothoracic surgery deals with operative mortality, particularly in patients undergoing coronary revascularization. Table 7-3 is a list of risk models used to assess operative mortality in patients undergoing coronary revascularization. Many of the risk stratification analyses shown in Table 7-1 and Table 7-3 are used to evaluate mortality outcomes in CABG patients, because mortality is such an unequivocal endpoint of greatest interest to patients and is recorded with high accuracy. For the diagnosis of ischemic heart disease, Table 7-3 lists the significant risk factors found to be important for a spectrum of the risk stratification systems. The definition of operative mortality varies among the different systems (either 30-day mortality and/or in-hospital mortality), but the risk factors identified by each of the stratification schemes in Table 7-3 show many similarities. Some variables are risk factors in almost all stratification systems; some variables are only rarely significant risk factors. Separate data sets validated each of the models; hence, there is some justification in using any of the risk stratification methods both in preoperative assessment of patients undergoing coronary artery bypass grafting (CABG) and in assessing provider performance (either physicians or hospitals).Validation and assessment of surgeons and hospitals using the risk models in Table 7-3 must be done with caution, because, as described previously, risk models are imperfect. Results indicating that a provider is a statistical outlier should always be corroborated by clinical review.

There are many critical features of any risk-adjustment algorithm that must be considered when determining its suitability for profiling provider performance. Daley provides a summary of the key features that are necessary to validate any risk adjustment model.66 Differences in risk-adjusted mortalities across providers may not necessarily reflect differences in the process and structure of care,67 an issue that needs further study.

Assessment Using Postoperative Morbidity and Resource Utilization

Patients with nonfatal outcomes following operations for ischemic heart disease make up more than 95% of the pool of patients undergoing operation. Obviously all non-fatal operative results are not equivalent. Patients who experience renal failure requiring lifelong dialysis, or a serious sternal wound infection, have not had the same result as a patient who leaves the hospital with no major complications, as occurs in about 85% of patients entered in the STS Database. The complications occurring in surviving patients range from serious organ system dysfunction to minor limitation or dissatisfaction with lifestyle, and account for a significant fraction of the cost of the procedures. We estimate that as much as 40% of the yearly hospital costs for CABG are consumed by 10 to 15% of the patients who have serious complications after operation.68 This is an example of a statistical principle called the Pareto principle and also suggests that reducing morbidity in high risk cardiac surgical patients has significant impact on cost reduction.

A great deal of information exists on nonfatal complications after cardiac operations. Several large databases identify risk factors for both nonfatal morbidity and increased resource utilization. Table 7-5 is a summary of some of the risk factors identified by available risk stratification models that are associated with either serious postoperative morbidity or increased resource utilization as measures of undesirable outcomes.

STS Risk Models for Postoperative Morbidity

For many years, operative mortality was the sole criterion for a successful CABG procedure. This concept gave way to a broader focus on the entire hospitalization associated with CABG. There is universal agreement that nonfatal complications play a central role in the assessment of CABG quality, but many morbidity outcomes are relatively difficult to define and track. Risk adjustment is particularly difficult because of the fact that risk factors for most complications are not well established. The low frequency of some complications also creates statistical challenges.

Shroyer and coworkers used part of the large national experience captured in the Society of Thoracic Surgeons Database to examine five important postoperative CABG complications: stroke, renal failure, reoperation within 24 hours after CABG, prolonged (>24 hours) postoperative ventilation, and mediastinitis.69 Revised morbidity models using contemporary statistical approaches followed this landmark study by Shroyer and coworkers.12-14 In 2009, the STS morbidity risk models were updated using data from 2002 to2006, with specific models for isolated CABG, isolated valve, and combined CABG + valve procedures. Given the contemporary data and large reference populations (eg, 774,881 patients for isolated CABG model), these are the most comprehensive, nationally representative, and thoroughly documented morbidity risk models currently available. These risk models will undoubtedly play an important role in future attempts at performance assessment.

Patient Satisfaction as an Outcome

Patients' assessment of surgical outcome is an alternate means of judging performance.There are several difficulties with measurement of patient-reported outcomes, and consequently cardiothoracic surgeons are not deeply involved with systematic measurements of patient satisfaction after operation. One problem is that patient-reported outcomes may be dependent on the type of patient who is reporting them and not on the type of care received. For example, younger Caucasian patients with better education and higher income are more likely to give less favorable ratings of physician care.70

Considerable research deals with instruments that are available to measure patient satisfaction. At least two of these instruments, the Short-Form Health Survey or SF-3671 and the San Jose Medical Group's Patient Satisfaction Measure,72 are used to monitor patient satisfaction over time. The current status of these and other measures of patient satisfaction does not allow accurate comparisons among providers, because the quality of the data generated by these measures is poor. These instruments are characterized by low response rates, inadequate sampling, infrequent use, and unavailability of satisfactory benchmarks. Nonetheless, available evidence indicates that feedback on patient satisfaction data to physicians may impact physician practices.73 It is likely that managed care organizations and hospitals will use patient-reported outcome measures to make comparisons between institutions and between individual providers.

Risk stratification methodology can identify patients who are optimal candidates for revascularization based on quality of life and functional status considerations .Multivariate risk factors associated with unimproved postoperative quality of life after CABG include female gender,74 patients with anxiety and depressive disorders,75,76 and operations complicated by sternal wound infection.77 Interestingly, patients who have CABG are likely to have equivalent mid-term health-related quality of life compared with patients undergoing percutaneous coronary interventions.78 One comparative study found no difference between patients older than 65 years and those younger than or equal to 65 with regard to quality of life outcomes (symptoms, cardiac functional class, activities of daily living and emotional and social functioning).79 This study identified a direct relationship between clinical severity and quality of life indicators, because patients with less comorbid conditions and better preoperative functional status had better quality of life indicators six months after operation than those with significant comorbidities.In contrast, Rumsfeld and coworkers found that improvement in the self-reported quality of life (from Form SF-36) was more likely in patients who had relatively poor health status before CABG compared with those who had relatively good preoperative health status.80 Interestingly, these same authors found that poor preoperative self-reported quality of life indicator was an independent predictor of operative mortality following CABG.81 These findings suggest that the risks of patient dissatisfaction after CABG are poorly understood but may be dependent on preoperative comorbid factors as well as on the indications for, and technical complexities of, the operation itself. At present there is no well-established risk model to identify patients who are likely to report dissatisfaction with operative intervention following CABG.

Composite Performance Measures for CABG

Recently, performance assessment expanded to include nonfatal adverse outcomes. In developing a set of performance measures for cardiac surgery, the NQF included five major complications (stroke, reoperation, prolonged ventilation, renal failure, and sternal infection) as important outcomes to measure. They also added a number of process measures, including use of the internal thoracic artery and a number of evidence-based perioperative medications (preoperative beta-blockade; discharge beta-blockade, lipid-lowering agents, and aspirin). Participation in a systematic data registry and procedural volume were included as structural measures of quality. In 2005, the STS embarked on development of a composite measure of CABG quality that incorporated many of these NQF individual measures. The Institute of Medicine and the NQF advocate composite measures of healthcare quality because composite scores provide a much more comprehensive view of quality compared with a single measure alone. Furthermore, the additional endpoints provide an increased number of observations and allow increased discrimination among providers. In a series of 2007 publications, the STS described the development of a composite measure that consists of 11 individual outcomes and process endpoints within four quality domains (risk-adjusted operative mortality, risk-adjusted major morbidity, use of the internal mammary artery for revascularization and use of NQF-endorsed cardiac medications).63,82 Using this framework and based on their composite score, providers are rated one, two, or three stars depending on whether there is a 99% Bayesian probability that their performance is below, at, or superior to the national STS average performance. This composite scoring methodology is gaining acceptance by health care providers and is the most widely used national system for assessing CABG provider quality. Furthermore, the feedback reports to providers from this composite scoring system are designed to show areas of strength and weakness to guide their internal performance improvement activities.

STS Database and Quality Improvement

The Society of Thoracic Surgeons recognized a compelling need for a national standard in cardiac surgery as early as 1986 when a formal STS committee was formed to develop a national database of cardiac surgery. This committee gathered and analyzed in order to establish a national standard of care in cardiac surgery. The STS Database is a voluntary registry that currently collects perioperative patient data from more than 90 % of cardiac centers in the United States. Individual participant sites enter extensive clinical data on each patient undergoing cardiac surgery. This information is harvested quarterly and aggregated at the Duke Clinical Research Institute (DCRI).The data are analyzed and reports, which include benchmark data and risk-adjusted outcomes, are provided to each site. This reporting process allows sites to pinpoint areas in need of improvement so that tailored quality assessment and performance improvement programs can be developed. The database has numerous important practical applications that allow performance assessment and document workload.

The STS database allows accurate determination of thoracic surgeons' workload.To paraphrase Winston Churchill, never have so many physicians and other health care professionals owed so much to so few. The “few” in this case are the 29 members of the American Medical Association/Specialty Society Relative Value Scale Update Committee, or RUC (rhymes with “truck”) for short. The RUC's recommendations to the Centers for Medicare and Medicaid Services (CMS) significantly influence the relative values assigned to physician services and, as a result, how much physicians are paid. STS data allows monitoring of trends in the patient profile of cardiac surgery patients over the years. This kind of information impacts negotiations with RUC. Deliberations with the RUC were traditionally based on small surveys but the use of STS data allowed a more accurate presentation of objective information that provides a truly fair and meaningful workload analysis.83

The STS database reporting and feedback process to individual sites produced impressive performance improvements in surgical outcomes. Database information showed a progressive increase in CABG operative risk from 1993 to 2008.In spite of that risk, however, the observed operative mortality steadily declined from over 4% to approximately 2% during this period.

The database plays a central role in the current national quality initiatives. Information from the database allows one to monitor and analyze all of the National Quality Forum (NQF) performance measures specified in Table 7-4. Specific NQF-based reports allow STS database participants to focus directly on the national measures chosen to reflect cardiac surgical quality. In addition, composite measures are available from the STS database. These composite measures (see previous) determine the net impact of all CABG performance measures (structural, process, and outcome measures) generated from the STS database.13 Composite scores are likely better indicators of performance than any single performance measure.

There is a groundswell of public demand for provider report cards and other hospital/surgeon ranking systems. The advantages and disadvantages of such reporting systems are addressed elsewhere in this chapter, but it should be mentioned that numerous third party payers are in the process of developing these scorecard protocols. In order to pre-empt the development of ill-advised and simplistic scorecards, STS experts developed a fair and meaningful quality rating system based on objective information from the STS Database.13

The STS Database presently contains records of approximately three million patients, thereby making it the largest cardiac surgery database in the world. This clinical database is an exceptionally valuable resource for cardiac surgeons and its risk assessment algorithms are generally accepted as standard quality assessment tools in cardiac surgery. The database has numerous important practical applications including performance assessment, workload documentation, provider benchmarking and documentation of national performance measures. These applications serve to promote quality improvement and ensure that the specialty is well represented in national regulatory programs.

Management Philosophy and Performance Assessment

American health care made almost unbelievable strides in the last 100 years. We are at the brink of being able to treat disease at the genotypic molecular level. Further, cardiac surgeons treat patients considered inoperable as recently as a decade ago. Yet almost no one is happy with the health-care system. It costs too much, excludes many, is inefficient, and is ignorant about its own effectiveness. A similar state of confusion existed with Japanese industry after World War II. Out of the confusion and crisis of post-World War II, Japan became a monolith of efficiency. Two major architects of this transformation were an American statistician, W. Edwards Deming, and a Romanian-American theoretician, J.M. Juran. They led the way in establishing and implementing certain principles of management and efficiency based on quality. Their efforts are recognized in Japan by the annual awarding of the Deming Prizes in recognition of achievements in attaining high quality. Deming's and Juran's books are some of the classics of quality management in industry.84,85

Deming's and Juran's management philosophy are sometimes referred to as total quality management or TQM. The amazing turnaround in Japanese industry led many organizations to embrace and modify the principles of TQM, including organizations involved in delivery and assessment of health care. Table 7-6 outlines the key features of TQM. An example of the application of the principles of TQM to health care delivery is described in the book by Berwick and coworkers.86 These authors recount the experiences of the National Demonstration Project on Quality Improvement in Health Care in applying principles of TQM to solve a broad spectrum of health care problems. Twenty-one health care organizations throughout the United States participated in an experiment to learn and implement TQM principles at every level within an institution. This effort produced improvement in customer-supplier throughput (improvement in the patient care process) to a variable extent.

The development and success of TQM stimulated creation of other similar quality management philosophies. Some examples include the Six-Sigma approach developed by Motorola, the Toyota Production System, Lean Manufacturing, the Disney Way Management System, Change Management systems, and the Theory of Constraints. These management programs are characterized by a desire to reduce waste and error through careful analysis of processes and bottlenecks in the throughput. They often use statistical quality control whenever applicable. They emphasize standardization, reduction of output variance, and focus on systems approaches to problems rather than blaming individuals. Some, but not all, of these management principles have applications in health care, and they have been extensively and successfully applied at the Virginia Mason Clinic in Seattle.

Total Quality Management is the management philosophy often used for healthcare applications. A TQM project starts from critical observations. For example, excessive blood transfusion after operation may result in increased morbidity, including disease transmission, increased infection risk, and increased cost. Tools such as flow diagrams that document all of the steps in the process are used in a TQM project (eg, steps involved in the blood transfusion process after CABG).A logical starting point for efforts to improve the quality of the blood transfusion process would be to focus on a high-risk subset of patients who consume a disproportionate amount of blood resources. An Italian economist, named Pareto, made the observation that a few factors account for the majority of the outcomes of a complex process, and this has been termed the Pareto principle. Juran was one of the first to apply this principle to manufacturing in the United States and Japan.85 A graphical method of identifying the spectrum of outcomes in a process is included in most statistics programs, and is termed a Pareto diagram. Figure 7-2 is an example of a Pareto diagram for blood product transfusion. Figure 7-2 suggests that about 20% of the patients consume 80% of the blood products transfused following cardiac procedures. Substantial savings in cost and possibly other morbidity will result by decreasing the amount of blood transfusion in these 20% of “high-end” users. For TQM purposes, strategies can be devised and tested to decrease blood product consumption in the high-risk subset, and ultimately, monitors are set up to measure the effectiveness of the new strategies. Other tools of TQM such as data sampling strategies and use of control charts play an important role in the process.

Northern New England TQM Project

A superb example of a TQM-based approach to improving cardiac surgery quality is the Northern New England Cardiovascular Study Group (NNECVDSG).Founded in 1987, this voluntary consortium of clinicians, scientists, and administrators represents cardiac surgery programs in Northern New England. Its mission is to study and improve the quality of cardiovascular care provided to patients through the use of systematic data collection and feedback. Shortly after its formation, this group developed and validated a logistic risk model to account for case mix differences across its member institutions,87 Using this model, the group analyzed CABG outcomes for 3055 patients operated on at five medical centers in Maine, New Hampshire, and Vermont between July 1987 and April 1989.88 Overall unadjusted CABG mortality was 4.3% but this varied substantially among centers (3.1 to6.3%).Even after case-mix adjustment, significant variability persisted among medical centers (p = 0.021) and surgeons (p = 0.025).In 1990, the NNECVDSG initiated a regional intervention aimed at reducing both absolute CABG mortality and inter-institutional variability.89 The three major components of this TQM approach included feedback of outcomes data, training in continuous quality improvement techniques, and site visits to each program. During the latter, visitors from each discipline focused on the practice of their counterparts at the host institutions. Numerous changes were implemented as a result of these site visits including technical aspects, processes of care, personnel organization and training, decision-making, and methods of evaluating care. Following these interventions, observed mortality declined to less than expected in all categories of patient acuity.

Subsequent to these landmark papers, the consortium continued to grow in size, and the registry forms the basis for numerous publications aimed at improving the care of cardiac surgical patients. Their publications cover a wide range of topics including the impact of preoperative variables on hospital and long-term mortality, the optimal conduct of cardiopulmonary perfusion, the prevention of specific postoperative complications, on-pump vs. off-pump CABG surgery, and modes of death following CABG.90-92 Over two decades since its inception, the NNECVDSG remains at the forefront of efforts to improve cardiac surgery quality through voluntary, confidential, and collaborative TQM.

Dangers of Outcome Assessment— Risk Aversion

After the introduction of provider report cards in NY and Pennsylvania in the early 1990s, studies emerged suggesting that providers were changing their practice patterns in response. The release of risk-adjusted data may alienate providers and result in the sickest patients having less accessibility to care. This may have already happened in New York State93–95 and in other regions where risk-adjusted mortality and cost data were released to the public. Of even more concern is the selection bias that may exist in managed care HMO enrollment. Morgan and coworkers suggest that Medicare HMOs benefit from the selective enrollment of healthier Medicare recipients and the dis-enrollment or outright rejection of sicker beneficiaries.96 This form of separation of patients into unfavorable or favorable risk categories is a direct result of risk assessment and may be an unintended consequence of performance assessment methodology. This type of discrimination undermines the effectiveness and appropriateness of care. Omoigui and colleagues addressed this issue in a report about the effect of publication of surgeon-specific report cards in NY state.94 These authors concluded that surgeons in that state were less willing to operate on high risk patients. Patients in NY state were subsequently transferred in disproportionately large numbers to the Cleveland Clinic, where both their expected and observed adverse outcomes exceeded those of other referral areas without report cards. Although this “outmigration” phenomenon was challenged by some, additional studies in NY and Pennsylvania suggest that the concept of risk aversion may have some validity.97–100 With the increasing use of percutaneous coronary interventions, a similar phenomenon was observed, with evidence of risk aversion in states with public reporting.101–104 Clearly there is some degree of risk aversion in public reporting environments, and this could result in denial of care to the high-risk patients that might benefit most from intervention. Others suggest that it may redirect such patients to the most experienced providers, which could be a more positive result.100,105 To mitigate the problem of risk aversion, some suggested not reporting or separately reporting the results of the most seriously ill patients, some of whom have risk factors that are difficult to adequately model with available risk models like those in Table 7-3. It is also important to review patients turned down for high-risk CABG or PCI to be certain that report cards are not having the unintended negative consequence of denial of appropriate care.

Validity and Reliability of Assessment Methods

Accuracy of Databases

Perhaps the most important tool of any outcome assessment endeavor is a database that is made up of a representative sample of the study group of interest. The accuracy of the data elements in any such database cannot be overemphasized.106 Factors such as the source of data, the outcome of interest, the methods used for data collection, standardized definitions of the data elements, data reliability checking, and the time frame of data collection are essential features that must be considered when either constructing a new database or deciding about using an existing database.106

Data obtained from claims or administrative databases are less reliable than those obtained from clinical databases. Because claims data are generated for the collection of bills, their clinical accuracy is inadequate and it is likely that these databases overestimate complications for billing purposes. They may incorrectly classify some surgical procedures, and this in turn may result in erroneous and misleading outcomes results.107 Furthermore, claims data underestimate the effects of comorbid illness and contain major deficiencies in important prognostic variables for CABG, such as left ventricular function and number of diseased vessels. The Duke Databank for Cardiovascular Disease found major discrepancies between clinical and claims databases, with claims data failing to identify more than half of the patients with important comorbid conditions such as CHF, cerebrovascular disease and angina.22,108 The Health Care Financing Administration (HCFA) used claims data to evaluate variations in the mortality rates in hospitals treating Medicare patients. After an initially disastrous effort at risk adjustment from claims data,109,110 HCFA developed new search and analysis algorithms. Despite these advances, the HCFA administration halted release of the 1993 Medicare hospital mortality report because of concerns about the database and fears that the figures would unfairly punish inner-city public facilities.111 The quality of databases used to generate comparisons cannot be overemphasized.

Logistic Regression and Hierarchical Regression Models

One of the most common yet controversial applications of logistic regression models is provider profiling, sometimes mandated by governmental organizations,112,113 in which case the results are usually published as report cards and made available to the public. The statistical methodology previously used to develop most such report cards is straightforward. The probability of mortality for each of a provider's patients during a given time period is estimated using logistic regression or some other multivariate method based on a large database containing multiple surgeons' patients. These probabilities are aggregated to determine a particular provider's expected mortality, or E. The observed mortality, O, is simply the counted number of operative deaths. An O/E ratio has a value close to one if the performance is what would be predicted from the model. Ratios significantly greater than one imply worse than expected performance, and ratios significantly less than one suggest better than expected performance. Often, the O/E ratios are multiplied by the population unadjusted mortality rate to obtain the risk-adjusted mortality ratio (RAMR) also called a standardized mortality ratio (SMR).

Statisticians realize that this intuitively appealing approach, aggregating patient-level data to make inferences about providers using logistic regression, is not ideal.114 Assessing operative mortality among hospitals and among surgeons is inherently multilevel. Multiple levels exist that may alter operative mortality, including surgeons, hospitals, referring physicians, etc. In such situations simply aggregating between levels may lead to erroneous conclusions. This was long ago recognized in education, where the analogous situation would be the inappropriate aggregation of pupil test scores to evaluate teacher performance. Multilevel or hierarchical models are available for such situations, and these models address most of the major concerns regarding the use of standard models that assess multilevel processes involving multiple surgeons and multiple hospitals. For example, the sample sizes from various providers often differ substantially. In the latter situation, the observed estimates of observed mortality are a much less accurate reflection of their true but unobserved future mortality. Hierarchical models “shrink” the observed mortality rates of lower volume providers toward the mean of the overall population of providers, a way of borrowing strength or pooling the data from multiple levels of hospitals and providers. The resulting estimates are more accurate and stable. Standard logistic models do not accurately partition the multiple levels of variability (between and within providers), which is one of the central questions to be answered by profiling. Hierarchical models correctly partition this variability and account for sample size variation and compensate for multiple comparisons. Numerous studies investigated the difference in the results of provider profiling obtained from traditional logistic regression versus hierarchical modeling. For example, Goldstein and Spiegelhalter, using a hierarchical model of operative mortality of surgeons in New York state, reduced the number of surgeon outliers from three to one.115 A logical objection to the use of hierarchical models is that, by reducing the chance of false outlier identification, it may also reduce the sensitivity to detect true outliers. Ultimately, this tradeoff is a health policy and regulatory decision.112 Hierarchical models are complex and require not only extensive computer resources but also close planning and oversight by a statistician experienced in these methods. Most investigators regard them as the best model for multilevel comparisons, and hierarchical modeling is used both by the state of Massachusetts and by the Society of Thoracic Surgeons for the development of provider profiles.

The Downside of Performance Assessment

Cost and Imperfection

Collecting risk-adjusted data and implementing performance enhancement adds to the administrative costs of the health care system. It is estimated that 20% of health care costs ($150 billion to $180 billion/year) are spent on the administration of health care.116 The logistical costs of implementing a risk-adjustment system are substantial. Additional costs are incurred in implementing quality measures that are suggested by risk-stratification methodology. A disturbing notion is that the costs of performance assessment may outweigh the payers' willingness to pay for these benefits. For example, Iowa hospitals estimated that they spent $2.5 million annually to gather MedisGroups severity data that was mandated by the state. Because of the cost, the state abandoned this mandate and concluded that neither consumers nor purchasers used the data anyway.117 Similarly, one report suggests that public release of quality indicators did not significantly improve composite hospital performance.118 It is possible that quality improvement may cost rather than save money; although one of the principles of TQM (often quoted by Deming) is that the least expensive means to accomplish a task (eg, deliver health care) is the means that employs the highest quality in the process. Ultimately, improved quality will be cost-efficient, but start-up costs may be daunting. In order to be cost-effective, any cost savings realized from performance assessment must be factored into the total costs of gathering risk-adjusted data and implementing quality improvement. Further, given the substantial costs of a single serious complication such as stroke, dialysis-dependent renal failure, or a sternal infection, the cost savings of performance assessment and quality improvement programs may be substantial.

Interpretation of Risk Adjusted Outcomes

One of the least well understood aspects of performance report cards is the correct interpretation of risk-adjusted or risk-standardized outcomes, which are derived by comparing observed outcomes with those predicted by statistical risk models. There is a tendency by the public, by insurance companies and by government officials to regard Risk-Adjusted Mortality Ratios (eg, O/E ratios) and Risk-adjusted or Standardized Mortality Rates as the ultimate metric of provider performance. Compared with unadjusted rates these are certainly superior approaches, but their limitations must be recognized. First, all risk models are only approximations of reality, and they cannot adjust for all possible combinations of risk factors. These models are useful for predicting population average outcomes given a particular patient mix, but they are less useful for predicting the outcome for specific patients. Second, even if there were perfect risk adjustment, the results for a particular provider must be correctly interpreted. The risk-standardized or risk-adjusted mortality for a hospital reflects its performance for its specific patient case mix, compared with what would have been expected had these same patients been cared for by an average provider in the reference population. Because indirect rather than direct standardization is used for virtually all risk models used in profiling, it is not appropriate to directly compare the risk-adjusted results of one institution with those of another. The risk-adjusted mortality of a small community hospital is based largely on a low-risk population. Even though it is adjusted, it cannot be compared directly to the risk-adjusted rate of a quaternary referral center, which is based largely on a population of patients that the community program rarely if ever sees. In this extreme example, these two hospitals may have virtually no types of patients in common, and even though their rates are adjusted they should not be directly compared with one another.

Ranking Providers—League Tables versus Funnel Plots

There is a problem with report cards of provider results that appear in the lay press and on the Internet. Most report cards rank providers in the form of league tables, similar to tables used in sports to rank teams or individuals. League tables always have someone on top and someone on the bottom. In general, the public does not understand that there is no meaningful difference for the vast majority of names published in league tables of cardiac surgeons' performance.119 The limited sample size of any individual surgeon or hospital leads to wide fluctuation in outcomes over time. Reporting of one surgeon as being better (ie, higher in the league table) than another is inaccurate and probably unethical. Spiegelhalter addressed this concern and suggested better options for reporting surgeons' and hospitals' results.120 He advocated funnel plots as a far better alternative to league tables to report provider outcomes. A funnel plot is a plot of individual surgeons' volume (x-axis) versus risk-adjusted mortality (y-axis) with population confidence intervals. The plot allows immediate identification of outliers (ie, providers outside the confidence intervals) and gives the viewer an estimate of the larger uncertainties (ie, increased confidence intervals) of risk-adjusted mortality of low-volume providers compared with high-volume providers. The limits of uncertainty (ie, control limits) form a funnel around the provider outcome. The United Kingdom Central Cardiac Audit Database (http://www.ic.nhs.uk/services/national-clinical-audit-support-programme-ncasp/heart-disease/adult-cardiac-surgery) and the STS Congenital Heart Surgery Database Report use funnel plots to identify outliers.119

Effectiveness and Appropriateness of Care

Since the Institute of Medicine report on medical errors that appeared in 2000,7 reducing medical errors both by physicians and by hospitals garnered significant resources. Since then, several authors suggested that the culture of safety in hospitals is unchanged.121,122 Brennan and coauthors point out that the IOM distinguishes safety from effectiveness. Effectiveness is defined as an intervention based on available evidence that improves quality, whereas safety encompasses a much narrower definition of limiting accidental injury.121 These authors suggest redirection of health care goals toward effectiveness interventions and away from accident reduction interventions. An important advantage of focus on effectiveness is the ease with which effectiveness outcomes can be measured compared with safety outcomes. There is some evidence that focus on evidence-based interventions (eg, providing aspirin to cardiac patients when they leave the hospital) improves the effectiveness of treatment with secondary benefit of reducing errors.123 Furthermore, there is still a reluctance to deal transparently with medical mistakes. Health care providers did not spend significantly increased resources on safety and error management largely because the return on this type of investment is very hard to measure. On the other hand, quality improvement efforts based on evidence of effectiveness are likely to be more readily embraced and may save more lives than will safety-related interventions that lack an evidence base.121

Practice guidelines present evidence-based recommendations regarding cardiovascular interventions.50 It is reasonable to ask how successful these practice guidelines are in altering practice. Most studies suggest that implementation of practice guidelines is difficult and changes in practice related to guideline recommendations are rare.54,124 Professional societies addressed this difficulty by introducing appropriateness criteria. Appropriateness criteria are lists of appropriate indications for interventions in common clinical scenarios. They document indications for drug or device intervention with a scale metric. For example, the AHA/ACC/STS Joint Task Force generated appropriateness criteria for coronary revascularization that used a scale of 1 to 10.125 Values of 7 through 10 indicated that coronary revascularization is appropriate for patients with a particular scenario. Scores of 1 to 3 indicate revascularization is inappropriate and unlikely to improve health outcomes or survival. The mid range scores (4 to 6) suggests that improvement in survival or other health care outcomes with coronary revascularization is uncertain. The aim of appropriateness criteria is to guide physician decision making toward use of evidence based interventions. It is likely that expansion of the appropriateness criteria concept will occur.

Decision Analysis, Machine Learning, and Neural Networks

Development of risk models to predict outcomes from cardiac surgical interventions (eg, Tables 7-1 and 7-3) consumes a great deal of effort. Although the models are fairly accurate at predicting outcomes for populations of patients undergoing operations, they still have distinct limitations. The biggest drawback of these risk models is their mediocre performance at predicting outcomes for individual patients.19 Consequently, risk-adjustment to predict individual outcomes is difficult to apply at the bedside and is often inaccurate. For patient-specific needs, risk stratification and outcomes assessment are in their infancy.

Better patient-specific predictors of outcome are needed for clinical decision making. The decision to operate for coronary artery diseases is extremely complex because there are more than two alternative treatments, more than two outcomes, and many intervening events that may occur to alter outcomes. As mentioned above, there are enormous variations in the way that surgeons practice. These variations can increase cost, cause harm, and confuse patients. Other global enhancements are available to direct surgeons toward evidence based care. These include decision analysis systems, artificial intelligence, and computer intensive risk modeling. The tools of decision analysis, similar to how airline pilots make decisions about complicated flight problems, were applied to the area of physician decision making in an effort to eliminate practice variations and to provide accurate and effective decisions at the patient bedside.126,127 Decision models generated to address surgical outcomes typically employ the familiar decision tree. An important part of creating the decision tree is to estimate the probabilities of each of the various outcomes for a given set of interventions. This part of the decision analysis tree relies heavily on the results of risk stratification and regression modeling, especially computer-intensive methods such as Bayesian models, to arrive at a probability of risk for a particular outcome. All of the diverse pieces of evidence used to form consensus guidelines, including meta-analyses, expert opinions, unpublished sources, randomized trials, and observational studies are employed to arrive at probabilities of outcome and intervening events. For example, Gage et al. performed a cost-effectiveness analysis of aspirin and warfarin for stroke prophylaxis in patients with nonvalvular atrial fibrillation.128 They used information from published meta-analyses of individual-level data from five randomized trials of antithrombotic therapy in atrial fibrillation to estimate the rates of stroke without therapy, the percentage reduction in the risk of stroke in users of warfarin, and the percentage of mild, moderate, or severe intracranial hemorrhages following anticoagulation. Their decision tree suggests that in 65-year-old patients with nonvalvular atrial fibrillation but no other risk factors for stroke, prescribing warfarin instead of aspirin would affect quality-adjusted survival minimally but increase costs significantly. Application of decision analysis methods to clinical decision making can standardize care and decrease risks of therapy but these methods are in the early developmental phase and much more work needs to be done before ready acceptance by surgeons.

The quantum increase in computational power that evolved over the past decade led to an entirely new “culture” of modeling called machine learning or algorithmic models.20,129 Rather than using classical statistical models with their specific assumptions and limitations, algorithmic modeling is focused more on predictive accuracy and less on trying to constrain nature to a particular parametric equation set of input variables. Examples of such algorithmic techniques include neural networks, decision and regression trees, and support vector machines. Because these models often do not provide an easily understood relationship between predictor and outcome variables, some have pejoratively referred to them as “black boxes,”129 However, despite this criticism, such models may, in many instances, provide superior predictive accuracy, even though the mechanism behind the relationship may be less transparent. Furthermore, in machine learning techniques, the numbers of predictors that can enter the model are not limited as they are in standard regression.

It was postulated over a decade ago that machine learning or artificial intelligence methods such as neural networks would provide the next major advance in predictive modeling for cardiac surgery. However, a large study to test this postulate was conducted by Lippmann and Shahian at the MIT Lincoln Laboratory using data from the 1993 Society of Thoracic Surgeons database of 80,606 patients who underwent CABG.31 A two-layer neural network did not provide any substantial improvement compared with standard logistic regression, and this neural network also underestimated risk in the highest risk groups. Somewhat improved performance was obtained by combining the results of the neural network with those of logistic regression; however, the absolute improvement in ROC curve area was relatively small. The failure of neural networks to improve predictive accuracy is likely a reflection of the imperfect and limited information presented to these models, which is the same information used to develop and evaluate traditional statistical models. No modeling technique is better than the information presented to it for training, and neural networks are no exception. It may well be that if presented with a more comprehensive set of predictor variables, some of which are not even recognized today as being significant, these machine learning techniques may prove to be more accurate. More work in this area is likely to proceed in the future.

Volume/Outcome Relationship and Targeted Regionalization

At least 10 large studies addressed the notion that hospitals performing small numbers of CABG operations have higher operative mortality (http://www.nap.edu/catalog/10005.html). Seven of these ten studies found increased operative mortality in low volume providers. In three other large studies there was no such association. In general, the volume-outcome association for CABG is weaker compared with that of other complex and much less frequently performed procedures such as esophagectomy and pancreatectomy.130–132 The Institute of Medicine summarized the relationship between higher-volume and better outcome and concluded that procedure or patient volume is an imprecise indicator of quality even though a majority of the studies reviewed showed some association between higher procedure volume and better outcome. Many other risk factors are more important determinants of outcome than provider volume in the STS database.133,134 Furthermore, the volume-outcome association applies only to average performance for certain provider volume categories. Some individual low-volume programs have superb performance, whereas some high-volume programs have inferior performance. The real problem with low-volume programs is that it is very hard to accurately measure their performance in a reasonably short time-frame because their sample sizes are too small. Strategies that improve the performance measurement and the actual outcomes of low-volume providers may prove helpful.132 Nallamothu and coauthors advocate a role for “selective regionalization” of higher risk patients, because they found that low risk patients did equally well in high volume or low volume hospitals.135 Other authors point out that a policy of regionalized referrals for CABG might have unintended adverse effects on health care including increased cost, decreased patient satisfaction, overcrowding of referral centers, and reduced availability of surgical services in remote or rural locations.136 Furthermore, there is no clear cut proof that a policy of regionalization would have the desired effects. Peterson and co-authors reviewed the STS National Cardiac Database to evaluate the relationship of CABG volume to outcome.133 They found almost no potential benefit in closing low-volume centers. The wide variability in risk-adjusted mortality among hospitals with similar volume precluded the ability of hospital volume to discriminate those centers with significantly better or worse mortality. They suggested that procedural volume is not an adequate marker of quality for CABG surgery. Other authors reached similar conclusions regarding the limited ability of regionalization to impact outcomes.137,138 Other more important process and structural variables are principle determinants of outcomes in CABG patients.

Human Factors Research and Performance Assessment

Surgeons make errors in the operating room. The causes of these errors and their ultimate impact on outcomes are an important measure of performance. Structured observation of surgeons by experts in human factor analysis can provide performance assessment and improve outcomes.139 The airline industry had success in limiting errors by instituting human factor analysis of pilots during simulated flight. This industry is used as a model of successful implementation of error avoidance behavior and process improvement.140 de Leval and co-authors applied these same principles of human factors analysis to pediatric cardiac surgery with some success.141 They employed self-assessment questionnaires and human factors researchers who observed behavior in the operating room, an approach similar to the quality improvement steps used in the airline industry. Their study highlighted the important role of human factors in adverse surgical outcomes. More importantly, they found that appropriate behavioral responses in the operating room can mitigate potentially harmful events during operation. Such studies emphasize that human factors are associated with outcomes, both good and bad. Behavior modification and process improvement that involves human factor analysis hold great promise for future error reduction in cardiac surgery.

Public Reporting and Provider Accountability

Today there is an unprecedented call for accountability and public reporting. That call comes from consumer groups and insurers as it has in the past, but today it is coming from governmental organizations including the U.S. Congress as well. There is now federal legislative force mandating the collection and the release of this information. The Centers for Medicare and Medicaid Services (CMS), for example, made it quite clear that their upcoming “pay for performance” programs will include mandated public reporting of data.

The World Wide Web provides ready access to a wide variety of medical facts, particularly concerning cardiothoracic surgery. Simple Internet searches provide the public with literature reviews of particular procedures, the results of randomized trials, new innovations, and surgeon and hospital-specific outcomes. This ready public access will undoubtedly increase. There is limited external scrutiny or validation of many of the information sources. Most information available on these sites is accepted at face value by the public and quality control of the information sources is limited to self-imposed efforts on the part of the website authors. The Agency for Healthcare Research and Quality (AHRQ) attempted to empower the public to critically evaluate the various web-based sources of health care information in order to limit the spread of misinformation. The success of the AHRQ efforts is uncertain but becomes extremely critical as the amount of healthcare information available on the web skyrockets.

Despite shortcomings associated with web-based public information sources, the national cry for public reporting continues unabated. Recognizing the inevitability of this national movement, the Society of Thoracic Surgeons initiated a project to develop a public reporting format that contains clinical data that is meaningful, audited, totally transparent and clinically relevant both for physicians and for the public. In collaboration with Consumers Union, the well respected consumer rating organization, the STS began publishing the composite CABG results for those program participants that agreed to do so. This type of partnership may well become the paradigm for other professional organizations as they manage public reporting.

Information Management: Electronic Medical Records

Medical records are an invaluable source of information about patient risk factors and outcomes. Inevitably computer applications are applied to medical records. Pilot studies assessed the importance of computerized medical records in a variety of clinical situations. Perhaps the most important example of successful implementation of computerized medical records lies within the Veteran's Health Affairs (VHA) medical system in the United States.142 Over a 20-year period the VHA system of hospitals completed a dramatic conversion from sub-standard care to a nationally recognized superb health care delivery system.142 This transformation was aided, if not completely caused by, the implementation of a costly but highly successful computerized medical record system. Iezzoni pointed out the difficulties with computerized medical records and suggests that they may not adequately reflect the importance of chronic disability while at the same time prolonging the time that physicians spend documenting care into the computer system.143 Yet the advantages of reduced medical errors, improved efficiency, and expanded access to medical information all overshadow most objections to implementation of an electronic information system.

Legislation likely to be passed by Congress allocates health care reforms that include an investment of $50 billion to promote health information technology.144 Further, in its economic recovery package, the Obama administration plans to spend $19 billion to accelerate the use of computerized medical records in doctors' offices (http://www.nytimes.com/2009/03/01/business/01unbox.html).Medical experts agree that electronic patient records, when used wisely, can help curb costs and improve care. Pending legislation indicates that physicians will be paid a bonus only for the “meaningful use” of digital records, although the government has not yet defined that term precisely. The new legislation also calls for creation of “regional health I.T. extension centers” to help doctors in small office practices use electronic records.It is apparent that the need for data about large groups of patients exists, especially for managed care and capitation initiatives. It is reasonable to expect that efforts to computerize medical records will expand. Applications of electronic medical records that may be available in the future for cardiothoracic surgeons include monitoring of patient outcomes, supporting clinical decision making, and real-time tracking of resource utilization.

Computer applications were applied to the electronic medical record in hopes of minimizing physician errors in ordering. Computerized physician order entry (CPOE) is one of these applications that monitors and offers suggestions when physicians' orders do not meet a pre-designed computer algorithm. CPOE is viewed as a quality indicator and private employer-based organizations used the presence of CPOE to judge whether hospitals should be part of their preferred network (http://www.leapfroggroup.org/).One of these private groups is the Leapfrog Group and an initial survey by this Group in 2001 found that only 3.3% of responding hospitals currently had CPOE systems in place (http://www.ctsnet.org/reuters/reutersarticle.cfm?article=19325). In New York State, several large corporations and health-care insurers agreed to pay hospitals that meet the CPOE standards a bonus on all healthcare billings submitted. Other computer-based safety initiatives that involve the electronic medical record are likely to surface in the future. The impact of these innovations on the quality of health care is untested and any benefit remains to be proven.

The success of information technology used to reduce medical errors is mixed. Innovations that employ monitoring of electronic medical records may reduce errors.145 However, with the increasing implementation of commercial CPOE systems in various settings of care evidence suggests that some implementation approaches may not achieve previously published results or may actually cause new errors or even harm.146 Much work needs to be done before computer-aided methods lead to medical error reduction but the future will see more efforts of this type made. The increasing role of CPOE systems in health care invites much more scrutiny about the effectiveness of these systems in actual practice.

Performance assessment and public reporting are here to stay, and they are likely to revolutionize the practice of medicine. Adequate risk adjustment is essential for outcomes analysis so that providers are not penalized when they care for severely ill patients. Composite performance measures, encompassing a broad spectrum of quality metrics, will be increasingly used. Cardiothoracic surgeons have been at the forefront of all these activities and innovations, but much work remains to be done. The ultimate goal of performance assessment is to improve patient care and to maintain the highest possible professional standards.