Chapter 6. Assessment of Severity of Illness

• In the last three decades, intensive care units (ICUs) and critical care researchers have amassed a body of pathophysiologic knowledge that has advanced the care of critically ill patients. Severity of illness scoring systems are tools that have been designed both to predict and to evaluate, from multiple perspectives, the outcomes of critically ill patients.
• Most of these scoring systems have evolved from multivariate regression analysis applied to large clinical databases to identify the most relevant factors for prediction of mortality. Scoring systems are then validated by prospective application to other patient populations.
• The ideal components of a scoring system are data collected during the course of routine patient management that are easily measured, objective, and reproducible.
• The most widely applied scoring systems in adults at the present time are the Acute Physiology and Chronic Health Evaluation (APACHE), the Mortality Probability Models (MPM), Simplified Acute Physiology Score (SAPS), and Sequential Organ Failure Assessment (SOFA).
• The potential uses of severity-of-illness scoring systems as applied to patient groups include clinical investigation (to standardize or compare study groups), ICU administration (to guide resource allocation and budget), and assessment of ICU performance (to compare performance over time or between health care settings).
• The use of scores in the delivery of care to individual patients is controversial; in some studies the accuracy of prediction of outcomes of scoring systems is not greater than that of the individual clinician's judgment.

Severity-of-illness scoring systems were developed to evaluate the delivery of care and provide prediction of outcome of groups of critically ill patients who are admitted to intensive care units (ICUs). The purpose of this chapter is to review the scientific basis for these scoring systems and to make recommendations for their use. While there is a growing recognition that when properly administered, these tools are useful in assessing and comparing patient populations with diverse critical illnesses, their use for predicting individual patient outcome is controversial and unresolved.

There are five major purposes of severity-of-illness scoring systems (Table 6-1). First, scoring systems have been used in randomized controlled trials (RCTs) and other clinical investigations.1–5 The second purpose of severity-of-illness scoring systems is to quantify severity of illness for hospital and health care system administrative decisions such as resource allocation. The third purpose of these scoring systems is to assess ICU performance and compare the quality of care of different ICUs and within the same ICU over time. For example, severity-of-illness scoring systems could be used to assess the impact on patient outcomes of planned changes in the ICU, such as changes in bed number, staffing ratios, and medical coverage.6 The fourth purpose of these scoring systems is to assess the prognosis of individual patients in order to assist families and caregivers in making decisions about ICU care. Finally, scoring systems are now being used to evaluate suitability of patients for novel therapy (e.g., the use of the APACHE II assessment for prescription of recombinant human activated protein C [drotrecogin alfa]).

The general hypothesis underlying the use of severity-of-illness scoring systems is that clinical variables that can be assessed on ICU admission predict survival and other outcomes of critically ill patients. This hypothesis is based on observations that increasing age, presence of underlying chronic disease, and increasingly severe abnormalities of the physiology of critically ill patients are associated with increased mortality. Scoring systems that have evolved combine relevant clinical variables to yield prediction of risk of mortality. Early in this evolution, severity-of-illness scores calculated at ICU admission or in the 24 hours following ICU admission predicted hospital mortality. More recently, scores have been calculated over the course of the ICU stay to provide updated prediction of hospital mortality. This dynamic approach uses change in organ dysfunction over time to add further accuracy and precision to outcome prediction.7–11

It is important to note that scoring systems have been developed using databases from patients already admitted to ICUs and not from the pool of patients outside the ICU, where the triage decision to admit a patient to the ICU is made. While severity-of-illness scoring systems in theory could be used to increase the accuracy of triage decisions regarding appropriateness of ICU admission, reformulation of the scoring methods would likely be necessary to better reflect the patient population outside the ICU, in whom triage occurs. Obviously, ICU resources should be focused on patients who are most able to benefit from ICU care. However, to date there are no reports regarding the use of scoring systems to assist in decisions regarding appropriateness of ICU admission.

The major scoring systems that are the focus of this chapter were designed specifically to predict outcome of critical illness. Initially, clinical and physiologic variable selection was based on subjective judgment of clinicians, review of the literature, and development of consensus. Subsequently, logistic regression modeling techniques were used to select predictive variables from a derivation data set. Ideal variables are simple, well-defined, reproducible, and widely available measurements or data that are collected routinely in the course of patient care. A large number of clinical and physiologic variables were collected on many patients, and their survival status at ICU and hospital discharge was recorded. Multiple logistical regression identifies the specific variables that best predicted survival and assigns relative weights to each variable. This set of variables is then tested prospectively for accuracy of prediction in another sample of patients to validate the selection process and appropriate weighting of variables.12

The sampling frequency and the period of measurements of physiologic variables are additional methodologic considerations that are important in the development of scoring systems. Most scoring systems use the most abnormal measurement of a physiologic variable in the 24 hours prior to ICU admission. More recently, scoring systems have used the most abnormal value of a physiologic variable for each successive 24-hour period while a patient is in the ICU, and have then correlated physiologic variables during this period with subsequent ICU outcome. Therefore prediction prognosis could be adjusted daily based on the patient's course and response to treatment. Furthermore, the change in organ dysfunction could be used to improve outcome prediction.7–11 Studies have shown that the change in organ dysfunction from day 0 to day 1, from day 0 to day 3,9 and indeed from day to day,13 can be used to accurately predict outcome of the critically ill.

Another important consideration in the development of severity-of-illness scoring systems is the patient sample used to derive the scoring system. For example, it is relevant to know whether scoring systems were derived in medical, surgical, or medical-surgical ICUs, whether community or tertiary care teaching hospital ICUs were used, whether ICUs were selected from one country or from many countries, and how many different ICUs were used to establish the scoring system. Furthermore, scoring systems derived from the sample of patients involved in a clinical trial may be biased and may not represent a general population of critically ill patients.

Methodologic Considerations

Critical appraisal of severity-of-illness scoring systems involves the measurement of accuracy (calibration and discrimination), reliability, content validity, and methodological rigor.14

Discrimination describes the ability of a model to distinguish between a patient who will live and one who will die. If discrimination is perfect, there is no overlap in probability estimates between patients who live and those who die.15 Discrimination is described by the area under the receiver operating characteristic (ROC) curve15,16 (Fig. 6-1). The ROC curve shows the relation between the true-positive rate (sensitivity) and the false-positive rate (100%-specificity). Because sensitivity and specificity are computed from independent columns in the decision matrix, and are therefore independent of sample mortality, the area under the ROC curve represents the proportion of patients who not only died, but who also had a higher probability of death than the patients who lived.14

The area under the ROC curve ranges from the lower limit of 0.5 for chance performance to 1.0 for perfect prediction. A model is considered to discriminate well when this area is greater than 0.8. An area of 0.9 means that a randomly selected actual nonsurvivor will have a more severe score than a randomly selected survivor 90% of the time.12 It does not mean that a prediction of nonsurvival occurs with probability 0.9, nor does it mean that a prediction of death is associated with observed nonsurvival 90% of the time. The area under the ROC curve illustrates the discriminating ability over the entire range of prediction scores.15

Calibration compares the observed mortality with that predicted by the model within the severity strata. Patients are placed into subgroups according to predicted risk. Typically, 10 strata are formed, called deciles of risk.15 Calibration is evaluated using goodness-of-fit tests; the most commonly used is the Hosmer-Lemeshow χ2 statistic 17. Typically, a 2 × 10χ2 table is created, with 10 strata of probabilities. The lower the overall χ2, the better the fit. The calibration test must be interpreted with care, as it is very sensitive to sample size.

Reliability refers to inter- (between) and intra- (within) observer agreement in the use of any severity of illness score, and represents the agreement in the data collection.14 The greater the subjectivity involved in the scoring system (i.e., choosing a primary diagnosis or assessing the level of consciousness in a sedated, intubated patient), the poorer the reliability of the system. Intraobserver reliability can be measured using a variety of techniques, and is expressed on a range between 0 (measurement involves nothing but error) and 1 (no variable error). A reliability coefficient of greater than 0.7 (suggesting that no more than 30% of the score is due to error) has been used as a statistical standard.14 The kappa statistic is a measure of interobserver reliability.

Content validity reflects the comprehensiveness of the model.14 Mortality is dependent not only on measured physiologic derangements and underlying health status, but may also be influenced by factors that are difficult to quantify, such as duration of organ system failure before treatment was instituted, and whether the admission was planned or unplanned, among others. However, as the number of variables increase in a scoring system, the reliability and ease of capturing the data decrease. The inclusion of many variables (overfitting) may actually reduce the performance of the model because some of these variables will be correlated with the outcome by chance alone. It has been proposed that stepwise regression should not be used unless there are at least 10 outcome events for each potential predictor.

Methodologic rigor refers to the avoidance of bias in development of a model. It is important that any severity-of-illness scoring system is based on a large database of all consecutive eligible patients in order to avoid bias.14 Several ICUs should be involved in data collection to avoid local bias in interpretation of coding or scoring rules. Clinical and laboratory variables should be those that are routinely collected, because collection of unusual data (such as serum ammonia) may bias treatment (treatment effect). Rigor must be applied in the consistency of data collection, and rules for dealing with missing data need to be uniformly applied. Validation using a second independent patient population is important in assessing the reliability of the model. Finally, the usefulness of a rigorously developed and validated scoring system can be degraded by poor application.

Scores Established at Admission

The scoring systems most commonly used in critically ill adults are APACHE II,18 APACHE III,19 MPM II,20 SAPS II,21 and SOFA.7,22 The variables included in each of these scoring systems are summarized in Table 6-2. The Pediatric Risk of Mortality (PRISM) score23 is the most widely used scoring system in pediatric critical care.

Some clinical variables are common to APACHE II, APACHE III, MPM II, SAPS II, and SOFA, probably because these variables measure specific clinical and physiologic functions that are major determinants of mortality. Specifically, each of these scoring systems uses age, type of admission, heart rate, blood pressure, assessment of renal function (blood urea nitrogen, creatinine, and/or urine output), assessment of neurologic function (Glasgow Coma Scale or presence of coma), assessment of respiratory function (mechanical ventilation, Pao2/FIo2, or alveolar-arterial oxygen gradient), and assessment of chronic health status. In contrast, other variables are not uniformly shared: serum potassium in APACHE II, glucose and albumin in APACHE III, and serum bicarbonate in SAPS II. These unique variables exist because of differences in the derivation of each scoring system, such as patient sample size, types of patients included, and statistical methods used to derive each score. An important difference between severity of illness scoring systems is how the predictor variables were chosen.24 For instance, in the APACHE II model, the developers selected those variables they thought relevant to patient outcome and then arbitrarily weighted each variable. In the development of MPM II, SAPS II, and APACHE III, statistical techniques were applied to identify those variables that were independently associated with death. These variables were then further refined by use of linear discriminant function and stepwise logistic regression analysis, and the final set of variables were then weighted by statistical methods and analyzed and presented as a cumulative score to predict mortality.

The Acute Physiology and Chronic Health Evaluation II (APACHE II) system18 is the most commonly used clinical severity-of-illness scoring system in North America. APACHE II is a disease-specific scoring system. It uses age, type of admission, chronic health evaluation, and 12 physiologic variables (acute physiology score or APS) to predict hospital mortality (see Table 6-2). The 12 physiologic variables are defined as the most abnormal values during the 24 hours after ICU admission.

The predicted hospital death rate is computed from the weighted sum of APACHE II score, a variable determined by whether the patient had emergency surgery, and the specific diagnostic category weight. APACHE II score was validated in 5815 ICU admissions from 13 hospitals. The correct classification rate for a 50% predicted risk of death was 85%. APACHE III19 extended APACHE II by improving calibration and discrimination through the use of a much larger derivation and validation patient sample. However, at this time, APACHE III is a proprietary commercial product.

The main disadvantages of the APACHE II system are its failure to compensate for lead-time bias,25 the requirement to select only one clinical diagnosis, and inaccuracies in clinical subsets, which produce poor interobserver reliability. In spite of these shortcomings, APACHE II remains the most well known and most widely used severity of illness scoring system.24

APACHE III is a disease-specific score that was developed from 17,440 admissions in 40 U.S. hospitals. Eighteen variables (see Table 6-2) were included, and their respective weights were derived by logistic regression modeling. To improve the accuracy of assessment of neurologic function, the Glasgow Coma Scale (GCS) score was changed, because reliability testing suggested the need to eliminate similar GCS scores that could occur in patients who had different neurologic presentations. The APACHE III score is a sum of physiology, age, and data from seven potential comorbid conditions. The final APACHE III score can vary between 0 and 300. Risk estimate equations for hospital mortality are calculated from the weighted sum of disease category (78 diagnostic categories are included), a coefficient related to prior treatment location, and the APACHE III score. In the original derivation sample, estimates of mortality for the first day in the ICU had an area under the ROC curve of 0.90, and the correct classification at 50% mortality risk level was 88%. Although APACHE III scores can be calculated from published information, weights to convert the score to probability of death are proprietary, therefore the APACHE III system has not been widely accepted or used.

The Mortality Probability Model (MPM II)20 was developed from 19,124 ICU admissions in 12 countries. MPM II is not disease specific. MPM0 is the only severity-of-illness scoring system that was derived at ICU admission and can therefore be used at ICU admission. MPM II does not yield a score, but rather a direct probability of survival. Burn, coronary care, and cardiac surgery patients are excluded. MPM0 includes three physiologic variables, three chronic diagnoses, five acute diagnoses, and three other variables: cardiopulmonary resuscitation prior to admission, mechanical ventilation, and medical or unscheduled surgery admission (see Table 6-2). Each variable is scored as absent or present and is allocated a coefficient. The sum of these coefficients constitutes the logit that is used to calculate the probability of hospital mortality.

The MPM2420 was designed to be calculated for patients who remained in the ICU for 24 hours or longer. MPM24 includes 13 variables, 5 of which are used in the MPM0. In the validation data set, the area under the ROC curve was 0.82 and 0.84 for the MPM0 and MPM24, respectively.

The most recent version of the Simplified Acute Physiology Score II (SAPS II)21 was developed from a sample of 13,152 admissions from 12 countries, based on a European/North American multicenter database. SAPS II is not disease specific. SAPS II uses 17 variables (see Table 6-2) that were selected by logistic regression: 12 physiology variables, age, type of admission (scheduled surgical, unscheduled surgical, or medical), and three underlying disease variables (acquired immunodeficiency syndrome, metastatic cancer, and hematologic malignancy). The area under the ROC curve was 0.86 in the validation sample. The probability of hospital mortality is calculated from the score.

The Sequential Organ Failure Assessment (SOFA) was originally developed as a descriptor of a continuum of organ dysfunction in critically ill patients over the course of their ICU stay.22 The SOFA score is composed of scores from six organ systems, graded from 0 to 4 according to the degree of dysfunction/failure. The score was primarily designed to describe morbidity; however, a retrospective analysis of the relationship between the SOFA score and mortality was developed using the European/North American Study of Severity System database.7,21 Subsequently, SOFA was evaluated as a predictor of outcome in a prospective Belgium study.13 SOFA score on admission was not a good predictor of mortality (area under the ROC curve 0.79); however, mean SOFA score and highest SOFA score had better discrimination (area under the ROC curve 0.88 and 0.90, respectively). Independent of the initial value, an increase in the SOFA score during the first 48 hours of ICU admission predicts a mortality rate of at least 50%.

Dynamic Severity of Illness Scoring Systems

All severity-of-illness scoring systems at ICU admission have a relatively high rate of misclassification of survivors and nonsurvivors. Misclassifications may be caused by the following: (1) exclusion of strong outcome risk factors that cannot be measured or have not been measured at ICU admission, (2) exclusion of complications that occur during ICU stay26, and/or (3) exclusion of treatment effects that modify outcome. Scoring systems applied over the course of the ICU stay can diminish the impact of these factors. However, discrimination of scoring systems applied during the ICU course is lower than discrimination of scoring systems evaluating outcome at the time of initial admission to the ICU.

MPM48 and MPM7227 were developed to estimate the probability of hospital mortality at 48 and 72 hours in the ICU. MPM48 and MPM72 have the same 13 variables and coefficients that are used in MPM24, but the models differ in the constant terms, which reflect the increasing probability of mortality with increasing length of ICU stay, even if physiologic parameters are constant. In the validation group, the areas under the ROC curves of MPM48 and MPM72 were 0.80 and 0.75, respectively.

APACHE III also can be used to calculate a daily risk of hospital mortality.28 A series of multiple logistic regression equations was developed for ICU days 2 to 7. The APACHE III daily risk estimate of mortality includes the acute physiology score (APS) on day 1, APS on current day, change in APS since the previous day, the indication for ICU admission, the location and length of treatment before ICU admission, whether the patient was an ICU readmission, age, and chronic health status.

The SOFA score has been used to describe increasing accuracy of outcome prediction when used over the first 7 days of the ICU course.13 More recently, the changes in SOFA score in cardiovascular, renal, and respiratory dysfunction from day 0 to day 1 of sepsis were significantly correlated with 28-day mortality in two large cohorts of patients who had severe sepsis.

Comparison of the Different Scoring Systems

Comparing the accuracy of the different scoring systems is difficult because of differences in populations used to derive these scores and different statistical methods. Thus there have been few head-to-head comparisons of different scoring systems. A multinational study29 compared different generations of the three main severity-of-illness scoring systems in 4685 ICU patients. APACHE III, APS II, and MPM II all showed good discrimination and calibration in this international database and performed better than did APACHE II, SAPS, and MPM. APACHE II and APACHE III have been compared in 1144 patients from the United Kingdom.30 APACHE II showed better calibration, but discrimination was better with APACHE III. Both scoring systems underestimated hospital mortality, and APACHE III underestimated mortality by a greater degree.

Comparison of Clinical Assessment with Scoring Systems

Clinical judgment to predict outcome has been criticized because it is not very reproducible, it has a tendency to overestimate mortality risk, and bias is introduced by the ability to recall particularly memorable, rare, and recent events.15 Three studies compared APACHE II with physicians' mortality predictions in the first 24 hours of ICU admission,31–33 and one study evaluated physicians' predictions only.34 Discrimination by physicians had ROC curve areas ranging between 0.85 and 0.89, which were similar to32,34 and even significantly better than those of APACHE II.31,33 In contrast to ability to discriminate, calibration rate of physicians' predictions of mortality versus APACHE II differed. For high-risk patients, APACHE II and physicians had similarly correct predictions for mortality, ranging from 71% to 85%. However, for estimated mortality risks below 30%, rates of correct classification of physicians' predictions were 39% to 69%, compared with 51% and 67% for APACHE II.31

Customization of Scoring Systems for Specific Diseases

Severity-of-illness scoring systems have been developed, derived, and validated for specific diseases to improve the accuracy of general scoring systems. APACHE III uses 74 disease classifications and derives a unique mortality risk prediction for each of these disease classifications. New scoring systems have been introduced to better predict mortality for patients with multiple organ failure and sepsis. The original models of SAPS II and MPM II did not perform well in patients who had severe sepsis, because mortality in severe sepsis was higher than mortality in patients with other diagnoses. Both models subsequently were customized5 for sepsis by using the original data to derive coefficients unique for sepsis to calculate predicted mortality. Furthermore, severity-of-illness scoring systems specifically designed for sepsis have been developed.

Prediction of mortality in sepsis will likely benefit from a dynamic approach that is based on evolution of multiple organ dysfunction. Commonly used organ failure–based systems that have been studied include the Sequential Organ Failure Assessment (SOFA) score,22 the Multiple Organ Dysfunction Score (MODS),35 and the Logistic Organ Dysfunction System (LODS).36

All three systems attribute points for organ dysfunction in six different organ systems. MODS,35 which applies to surgical patients, differs from SOFA and LODS in the cardiovascular assessment. MODS scores the cardiovascular system based on the “pressure-adjusted heart rate,” defined as the product of the heart rate multiplied by the ratio of the right atrial pressure to the mean arterial pressure. LODS and MODS have excellent discrimination, with ROC curve areas of 0.85 and 0.93, respectively.35,36

APACHE II, MODS, and SOFA were recently used to compare outcome prediction in and prospective study of 949 ICU patients.37 There were no significant differences between MODS and SOFA in terms of mortality prediction. The area under the ROC curves for APACHE II, SOFA, and MODS were 0.880, 0.872, and 0.856, respectively. In patients with shock, the MODS and SOFA scores were slightly better mortality predictors than APACHE II score (area under ROC curve 0.852 and 0.869 vs. 0.825).

Some have suggested that organ failure–based scoring systems could provide an outcome measure to be used as a surrogate for the end point of mortality.38 Thus, for large (and expensive) randomized clinical trials such as those recently conducted in the treatment of sepsis or acute lung injury, could a reduction in some score of organ failure be taken as a measure of reduced morbidity and hence high drug efficacy?

Many randomized controlled trials in critical care have successfully evaluated organ dysfunction as secondary outcome variables by using scoring systems. Important recent examples include the ARDS Network study of 6 mL/kg vs. 12 mL/kg of ideal body weight tidal volume in patients who had acute lung injury.39 The use of a protocol of 6 mL/kg ideal body weight, positive end-expiratory pressure (PEEP), and guidelines for respiratory rate and minute ventilation decreased mortality from 40% (with 12 mL/kg tidal volume) to 30%. In addition, the 6 mL/kg tidal volume strategy significantly increased the number of days patients were alive and free of respiratory, hepatic, cardiovascular, coagulation, and renal dysfunction39 as assessed using the Brussels scoring system.9 The randomized controlled trial of recombinant human activated protein C (rhAPC; drotrecogin alfa) showed that rhAPC decreased mortality of severe sepsis from 31% to 25% compared to placebo.40 The SOFA score was used in this study to evaluate organ dysfunction and rhAPC improved markers of organ dysfunction.

Scoring Systems Specific for Trauma Patients

Scoring systems have been developed to improve triage of trauma patients and to predict their mortality (see Chap. 92). Trauma scoring systems were developed using general trauma patient samples, not specifically critically ill trauma patients. The initial scores were either anatomic (Injury Severity Score or ISS1,41) or physiologic (Trauma Score or TS42 and Revised Trauma Score or RTS43). Recently, trauma scoring systems have been expanded to include age, anatomy, and physiology, including the Trauma and the Injury Severity Score or TRISS methodology,2 and A Severity Characterization of Trauma or ASCOT.44 Large trauma registries facilitated implementation and validation of trauma scoring systems in large samples of patients. Table 6-3 summarizes the main trauma scoring systems.

The accuracy of TRISS and APACHE II have been compared in critically ill trauma patients.45 APACHE II classifies trauma patients under only four diagnostic categories: postoperative multiple trauma, postoperative head trauma, nonoperative multiple trauma, and nonoperative head trauma. In APACHE II, patients with combined head and other injuries were assigned to multiple trauma, which was given a lower weight than the isolated head trauma category in predicting mortality.46 The number of derivation patient samples of APACHE II were much smaller than the samples used for the trauma scores. TRISS tends to perform better than APACHE II. APACHE II significantly overestimates the risk of mortality in the lower ranges of predicted risk and underestimates the risk of mortality in the higher ranges. APACHE III attempted to improve prediction of mortality for head-injured patients by revising the definition for head trauma, allowing assignment of patients with isolated head trauma as well as head trauma and other injuries to the head trauma category. This resulted in a higher predicted mortality that more closely reflected the actual mortality.

Scoring Systems in Randomized Controlled Trials and Other Clinical Research

Clinical research in critical care often includes heterogeneous samples of critically ill patients. Severity-of-illness scores are commonly used to describe the acuity of illness so that readers can compare different studies and so that studies can be compared with a clinician's practice. Scoring systems are used in randomized controlled trials (RCTs) to describe severity of illness, to assess comparability at baseline of control and treatment groups, to assess the expected mortality, to determine sample size, and to perform stratified randomization. The success of randomization is often assessed by using scoring systems to confirm that the baseline characteristics of control and treatment groups were not significantly different.

In a randomized controlled trial, if the randomization is not balanced, then outcomes may be altered by the imbalance in baseline characteristics. In that instance, severity of illness scoring systems may be used to do an adjusted analysis (i.e., by adjusting groups for differing severity of illness and then calculating adjusted mortality). For example, very recently a large multicenter RCT of two different PEEP regimens in patients who have acute lung injury was reported. There was no difference in mortality between groups, but unfortunately age was significantly higher in the high PEEP group. Therefore, an adjusted analysis using age and severity of illness was done to adjust for differences in baseline characteristics and found no difference in adjusted mortality. Scoring systems are also used to determine the effect of the therapeutic intervention across different disease severity and mortality risk strata. In a study with no positive drug effect, finding efficacy in a subgroup of patients (e.g., in the sickest patients47) can be hypothesis-generating for new studies involving these sicker patients only. Finally, the clinicians reading the results of an RCT can compare the severity of disease of patients in the study with the severity of disease of patients in their practices to decide whether to use the new treatment.

Scoring Systems for Administrative Purposes

The major purposes of scoring systems in administration are to describe utilization of ICU beds and resources, to describe acuity of illness, and to relate resource utilization (e.g., funding, drug utilization, and/or personnel) to acuity of care in an ICU. Resource utilization can be described, for example, by the Therapeutic Intervention Scoring System (TISS) score,48,49 developed at the Massachusetts General Hospital in 1974. The purpose of TISS was to provide quantitative data to determine the severity of illness in individual patients, in order to determine appropriate utilization of intensive care facilities. TISS quantifies the amount of critical care provided to patients by measuring 76 nursing activities, monitoring techniques, resuscitation procedures, and technology. Each intervention is given 1 to 4 points. Therefore, TISS assesses severity of illness indirectly by the level of services provided to the patient. TISS was designed as a descriptor of the intensity of care, and was not designed specifically to predict outcome.

TISS scores have been used to categorize the level of care that patients require.49,50 Beck and coworkers used TISS scores at ICU discharge as an objective assessment of the risk of premature discharge, and investigated the relationships of discharge time, TISS scores, and discharge destination on post-ICU mortality.51 There was a significant association between increasing TISS scores and post-ICU mortality at ICU discharge (χ2 for trend = 0.90, p = 0.028). Patients with high TISS scores (>30) who were treated in hospital wards had significantly increased severity-adjusted mortality risks compared with a comparable group of patients who were discharged to high-dependency units.

In addition, acuity of care can be correlated with indices of resource utilization.52 Furthermore, reimbursement can be guided by assessment of severity of illness. For example, planning for ICU bed allocation, staffing, and budget can be aided by measures of admission numbers, diagnoses (e.g., diagnosis-related groups [DRGs] and case-mix groups [CMGs]), and severity of illness.

Scoring Systems to Assess Intensive Care Unit Performance

Scoring systems can be used by ICUs to evaluate quality of care (quality assurance; see Chap. 3), to assess performance of an ICU over time, to assess performance of different intensivists, and to assess performances of different ICUs (see Table 6-1). The scoring systems provide a tool to normalize for differences in severity of illness of different samples of patients. Although quality assurance has largely been supplanted by newer approaches such as continuous quality improvement, severity-of-illness scoring systems nonetheless can be used to assess predicted and actual mortality. ICUs can review the outcomes of patients in general, or for specific disease categories, and compare the actual outcomes with predicted mortality. The performance of an ICU also can be followed over time. Evaluation of new technologies or new treatment modalities in an ICU also can be the object of continuous quality improvement evaluations.

There are potential problems associated with the use of scoring systems to compare actual with expected mortality in an ICU. For example, biases in the regression techniques used to calculate the risks of mortality can lead to situations in which hospitals providing care to more severely ill patients will tend to have actual mortality rates above predicted, and thus will appear to be giving suboptimal care. This occurs because most scoring systems underestimate mortality of high-risk patients. Also, medical and nursing interventions can improve physiologic data, leading to a lower estimated risk of mortality for the same patient.53 The outcomes of individual intensivists can be adjusted for severity of illness to better assess performance. This is controversial for several reasons. First, patient sample size of the intensivist may be insufficient to draw legitimate conclusions regarding performance.54 Second, ICU care is team care, including house officers and other caregivers, so outcomes are less influenced by the behavior of individual physicians.

Scoring systems can be used to compare ICUs in different hospital settings (tertiary care, community, academic, etc) and to compare ICUs of different countries. A comparison of New Zealand and U.S. hospitals demonstrated different patient selection and fewer admissions to ICUs in New Zealand, and yet hospital mortality rates were comparable.55 A similar comparison between hospitals in Canada and in the United States revealed similar results.56 However, important differences in mortality have been observed between pediatric ICUs in the United Kingdom and Australia. For comparable severity of illness, the mortality rates of critically ill children were higher in the United Kingdom than in Australia.57

Severity-of-illness scoring systems also can be used to assess ICU performance in different models of organization. For example, Carson and coworkers58 evaluated the effects of changing from an “open” to a “closed” model of ICU care by dedicated intensivists by using a “before/after” study design. Patient severity of illness as assessed by APACHE II was greater, yet care costs were similar, and the ratio of actual to predicted mortality was lower after converting a medical ICU from open to closed care. Similar studies involving patients with sepsis demonstrated that changing ICU staffing to include physicians formally trained in critical care medicine reduced mortality.59,60 Other examples of the use of scoring systems to assess ICU performance include studies of availability of ICU technology and studies of organizational practices and outcomes.61

Rapoport and coworkers62 described a method to assess cost effectiveness of ICUs. A clinical performance index was defined as the difference between actual and MPM II predicted mortality. The economic performance (resource use) used a surrogate for costs: the “weighted hospital days,” a length-of-stay index that weights ICU days more heavily than non-ICU days. Predicted resource use was calculated by a regression including severity of illness and percentage of surgical patients. The actual and predicted survival and actual and predicted resource use of hospitals were compared with the mean. A scatterplot illustrated which units were more than one standard deviation off for clinical and economic performance.

The cost effectiveness of ICUs should include nonmortality measures of effectiveness such as quality of life, return to independent living, and patient/family satisfaction.63 These nonmortality measures of outcome need to be adjusted for ICU severity of illness by using severity-of-illness scoring systems.

Scoring Systems to Assess Individual Patient Prognosis and to Guide Care

The assessment of individual patient prognosis is complex. Moreover, the use of severity-of-illness scoring systems for assessment and prediction of individual patient prognosis is controversial. We believe that management decisions cannot be based solely on prognosis as evaluated by the scoring systems. Assessment of individual patient prognosis influences decisions regarding triage of patients (i.e., ICU admission), decisions regarding intensity of care, and decisions to withhold and withdraw care.

Theoretically, a very accurate estimate of patient prognosis could be used to triage patients who have such a good prognosis that ICU admission would be unnecessary and inappropriate, and to identify patients who are so hopelessly ill that ICU admission would be futile and inappropriate. Scoring systems may complement physician judgment regarding appropriateness of ICU admission. However, it is important to emphasize that most scoring systems were derived from patients already admitted to an ICU using data from the first 24 hours of ICU admission. The Mortality Probability Model (MPM II) might be more accurate and appropriate because MPM0 used variables available immediately at ICU admission rather than the worst values of variables over the first 24 hours in the ICU. However, none of the commonly used scoring systems were validated for the purpose of triage of ICU patients.

Scoring systems have been used to assist in triage of patients to intermediate care (monitoring) or to intensive care (life support). Recently, APACHE III was modified to estimate the probability of need for life support of patients admitted for ICU monitoring.64 Among 8040 ICU admissions for monitoring, 79% were predicted to have a low probability (<10%) for active treatment during their ICU stay. These patients were admitted to an intermediate care unit and 96% received no subsequent active treatment. The predictive equation had a ROC curve area of 0.74. There are scoring systems designed specifically for triage of trauma patients. The Triage Index65 for trauma patients assesses injury severity and predicts an outcome using physiologic variables available before admission. The Trauma Score42 and the Revised Trauma Score43 are derived from the Triage Index. In the part of the Revised Trauma Score used for triage, the T-RTS, specific decision rules are proposed to indicate appropriate transfer to a trauma center.43 These rules are based on the score and the GCS score. There are at least two caveats regarding use of scoring systems to guide ICU triage decisions. First, a patient who could be admitted to the ICU who has a very low probability of mortality estimated by the MPM0, might in fact have a higher actual probability of mortality if ICU admission were denied,20 because outcome could be adversely affected by ward admission and the associated lower intensity of monitoring and treatment. Second, physicians tend to underestimate mortality in low-risk patients. Thus scoring systems can be more accurate than clinician judgment for risk estimate of low-risk patients.31

A novel use of severity of illness scoring systems is in patient selection for specific therapies. The advent of a new therapy for severe sepsis, recombinant human activated protein C (rhAPC; drotrecogin alfa), is an example. In a multicenter RCT (the PROWESS trial), rhAPC significantly decreased mortality compared to placebo in treatment of severe sepsis.40 Treatment with rhAPC activated was associated with a reduction in the relative risk of death of 19.4% (95% confidence interval [CI], 6.6 to 30.5) and an absolute reduction in the risk of death of 6.1% (31% vs. 25%, p = 0.005). The number needed to treat (NNT) with rhAPC was 16 to save 1 life. A post-hoc analysis of the study data performedby the FDA reported a differential benefit according to APACHEII score (Fig. 6-2); among patients with an APACHE II score of 25 or more, the relative risk of death among patients treated with rhAPC, as compared with those given placebo, was 0.71 (95% CI, 0.59 to 0.85), whereas among those witha score of 24 or less, the relative risk of death was 0.99 (95% CI, 0.75 to 1.30). Consequently, the FDA concluded, “efficacy of drotrecogin alfa has not been established in patients with lower risk of death (e.g., APACHE II scores <25),”66 and the drug has not been approved for use in patients with lower severity of illness. Therefore, in the U.S. and in some other countries, regulatory and payer groups permit use of rhAPC only in patients who have severe sepsis and a high risk of death as defined by an APACHE II score greater than 25. In contrast, in Europe rhAPC was approved for use in patients who have severe sepsis and two or more organ dysfunctions or an APACHE II score greater than 25. These differing regulatory approvals and clinical practices likely reflect differences in how APACHE II scores can be used to make individual patient therapeutic decisions.

In support of this approach (i.e., use of APACHE II to identify high-risk patients), a cost-effectiveness analysis by Manns and coworkers67 found that rhAPC is relatively cost effective when targeted to patients with severe sepsis, greater severity of illness (an APACHE II score of 25 or more), and a reasonable life expectancy if they survive the episode of sepsis. In a second cost-effectiveness analysis by Angus and coworkers, rhAPC cost $27,400 per quality-adjusted life-year when limited to patients with an APACHE II score ≥25, and was cost-ineffective when limited to patients with a score <25.68 Clearly, the results of this subgroup analysis are dependent on the validityof the post hoc reanalysis performed by the FDA. Manns and coworkers concluded, “giventhe discrepancy between the published study results and thereanalysis, it would be reasonable to restrict the use of activatedprotein C to patients with an APACHE II score of 25 or moreuntil convincing evidence of effectiveness and cost effectiveness in patients with less severe illness becomes available.”67

There are several criticisms of using baseline APACHE II scores in individual patients to guide therapy.69 First, the PROWESS trial was not powered to determine an efficacy difference among APACHE II subgroups. Even more importantly, the APACHE II disease severity scoring system was not designed and has not been validated for use to discriminate any parameter in the individual patient. Furthermore, the interobserver and intraobserver variability in the determination of APACHE II scores among experienced intensive care physicians may be as high as 10% to 15%.70 To our knowledge, this is the only example of a proven therapy in critical care that is allocated based on an individual patient's APACHE II score.

Use of severity-of-illness scoring systems to assist in decision making regarding withholding and withdrawal of care is controversial for several reasons. First, scoring systems are designed to describe severity of illness and probability of death in groups of patients, not individual patients. Second, even in groups of patients, no system is perfectly calibrated and such systems cannot perfectly discriminate survivors from nonsurvivors. Third, scoring systems can guide care decisions only in the context of appropriate understanding of the ethical principles relevant to withholding and withdrawal of care.71 Nonetheless, scoring systems could assist in deciding that ICU care is futile. Schneiderman and coworkers72 proposed that “when physicians conclude (either through personal experience, experiences shared with colleagues, or considerations of published empirical data) that in the last 100 cases a treatment has been useless, they should regard that treatment as futile.” Using this definition of futility, let us consider use of scoring systems and physician judgment to help predict futility. Calibration of APACHE III found that for an estimated mortality rate above 90%, the rate of correct classification was 85%, with a specificity of 99.8%. By comparison, for the same estimated mortality rate above 90% strata, physicians' predictions yield a correct classification of 70% to 76% and a specificity of 97% to 99%.31,32 Thus APACHE III may be more accurate than physicians in predicting that a group of patients have a 90% chance of mortality. However, at a quantitative threshold of futility of less than 1% chance of survival, scoring systems are not precise enough. The highest precision of any scoring system to date was a 95% probability of death, meaning that 5% of patients with that score would survive28 (Fig. 6-3). Therefore, severity-of-illness scoring systems may not accurately identify patients in whom ICU care is futile if futility is defined as less than 1% chance of survival. The SUPPORT study73 (Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments) is important because it was designed to determine whether providing physicians with accurate predictions of death would change physician behavior, patient satisfaction, and decisions regarding care. SUPPORT was designed to estimate survival of seriously ill hospitalized patients who were not necessarily in an ICU. The SUPPORT73 prognosis model includes nine diagnostic groups and the following 15 prognostic factors: disease group, 11 physiologic variables, age, history of malignancy, and the number of days the patient was hospitalized before study entry. In phase I of the study, the investigators noted shortcomings in communication, variability in frequency of aggressive treatment, and variability in care at the time of death (CPR, comfort care, pain management, etc). In phase II of the study74 physicians in the intervention group received probability estimates of 6-month survival, outcome of cardiopulmonary resuscitation, and incidence of functional disability at 2 months. Specifically trained nurses made multiple contacts with the patients, families, physicians, and hospital staff to elicit preferences, improve understanding of outcomes, encourage attention to pain control, and facilitate advance care planning and patient-physician communication. Importantly, the phase II intervention did not improve care or patient outcomes. Patients experienced no improvement in patientphysician communication. Also, there was no change in the incidence or timing of written DNR (do not resuscitate) orders, physicians' knowledge of their patients' preferences not to be resuscitated, number of days spent in the ICU before death, or use of hospital resources. Thus the SUPPORT study showed that providing physicians with objective outcome predictions did not change physicians' attitudes and behavior.

Several observations suggest that there is a gap between scoring system predicted outcome and decisions to withhold and withdraw ICU care. Patients in whom care was withdrawn in a medical ICU had APACHE II predicted mortality on the day of ICU admission of only 61% ± 22%.75 Furthermore, patients with prolonged multiorgan system failure who continue to require life support generally do not have very abnormal physiologic parameters54 and thus have relatively low APS scores. Finally, an increasing proportion of critically ill patients in ICUs die without CPR (cardiopulmonary resuscitation),76 and many die after withholding or withdrawal of care.

A major portion of ICU resources is spent on patients who have minimal chances of survival. However, until a public consensus is reached about dealing with these very difficult issues,77 broad ethical principles of beneficence, nonmalfeasance, and autonomy are likely to be more important components of end-of-life decisions than quantitative data provided by scoring systems. Broader social and economic policy issues should be separate concerns.

Severity-of-illness scoring systems are not perfect, partially because of error and bias. Error and bias limit the reproducibility of scoring systems outside the original sample of patients, and thus limit the applicability of scoring systems to different clinical situations. Specifically, bias of scoring systems can be related to the selection of included variables, to the collection of data, to the lead time before the onset of the acute disease and admission of the patient to the ICU, to the imprecision in choosing a principal admission diagnosis, to the inaccuracy of certain scoring systems for specific disease categories, and finally to the use of scoring systems for purposes they were not meant to accomplish.

Bias Related to the Selection of Variables and to the Collection of Data

Variables can be included in a severity-of-illness score by a multivariate analysis that shows that each variable is a statistically independent predictor of mortality. Alternatively, variables can be selected by consensus of experts. Consensus panel selection of variables is subjective, and variables can be interrelated.15 The problem with interrelated variables is that two such variables are not independent of each other as predictors of mortality. Noncontinuous variables increase error in the computation of risk of mortality. Noncontinuous variables are classified as present or absent, so a single misclassification results in a large error in outcome prediction.15

Detection bias is another cause of bias of the included variables. Detection bias means that variables are only detected if measured. However, because scoring systems use variables measured in clinical practice, not all variables will be measured on all patients on all days. Therefore, in several scoring systems, unmeasured (undetected) variables are assigned a normal value. The assumption that unmeasured physiologic variables are normal can underestimate the risk of mortality. APACHE II, APACHE III, and SAPS II contain some variables that are not used routinely in daily care, such as albumin and bilirubin levels.

Use of the worst value of a variable in 24 hours also causes errors. Most scoring systems use the worst value of a variable in a 24-hour period. However, selection of the worst value can be subjective. There are other errors associated with collection of data, including temperature conversion from Fahrenheit to Celsius, creatinine conversion to the international system, use of the Glasgow Coma Scale on deeply sedated patients,54 transcription errors, and errors in analysis of data. Direct computer data entry may decrease transcription error.

Bias Related to Poor Calibration

Statistical regression in scoring systems has a propensity for poor calibration. Regression techniques tend to underpredict the likelihood of death of more severely ill patients, and tend to overpredict the likelihood of death of patients with less severe illness (Fig. 6-3). These errors can create a pernicious bias. For example, hospitals providing care to more severely ill patients will tend to have actual mortality rates above predicted, and thus will appear to be giving poor care. On the other hand, hospitals with less severely ill patients will tend to have actual mortality rates lower than predicted, and will appear to be giving better than average care.78

Lead-Time Bias

Lead-time bias refers to the different lengths of time that patients are ill prior to ICU care and scoring. Lead-time differences also influence mortality. Acute physiology scores do not assess previous treatments. Thus, for the same score, a patient hypoxemic in the emergency room can improve rapidly and have a better outcome than a patient referred from another hospital for persistent hypoxemia. Because of lead-time bias, APACHE II underestimates the mortality of patients referred from other ICUs,25 other hospitals, or even within other parts of the same hospital.15 APACHE III contains a variable to assess patient location and treatment prior to ICU admission in an attempt to minimize lead-time bias.

Therapies provided prior to and immediately after ICU admission change physiologic variables and thus influence physiologic scores. Rapid and successful resuscitation in the emergency department prior to ICU admission or early in the ICU will hide abnormally low values of variables that would have been recorded as the worst over 24 hours. Theoretically, poor care would increase physiologic scores and increase predicted mortality rate, whereas good care would decrease scores and reduce predicted mortality rate.78 The effects of treatment can be minimized and mortality prediction might be enhanced by using hospital presentation data.

Imprecise Principal Diagnosis

Inaccurate diagnosis is another source of error in many scoring systems. Some scoring systems (e.g., APACHE III) adjust for the differing prognoses of patients who have different diseases but similar physiologic abnormalities. Accurate diagnosis can be difficult in the critically ill for several reasons. First, patients in the ICU often suffer from several conditions. APACHE II and APACHE III require identification of one diagnosis or organ failure that prompted ICU admission. Consider a patient with bacterial pneumonia and sepsis who is admitted to the ICU. In APACHE III, patients who have bacterial pneumonia and patients who have sepsis who have similar physiologic scores have different predicted mortality (see Fig. 6-4). Thus the assignment of appropriate diagnosis is very important in making an accurate mortality prediction.15 The principal diagnosis could differ between prospective identification versus retrospective chart review.54

Severity‐of-Illness Scoring Systems for Specific Disease Categories

For patients with specific disease processes, it is debated whether specific severity scoring systems are better than general ones. Because inaccuracy of diagnosis can cause error, we recommend that scoring systems be tested for different diagnostic categories. Both disease-specific systems (APACHE II and APACHE III) and non–disease-specific systems (SAPS II and MPM II) need to be compared in external validating patient samples.15

APACHE II, APACHE III, and SAPS II have performed well in several disease-specific categories, including liver failure,79,80 malignancy,81–83 cardiac bypass surgery,84–86 sepsis,87 peritonitis,88–90 pancreatitis,91–97 acute myocardial infarction,98–101 HIV patients,102,103 obstetric patients,104 and stroke.105 APACHE III performed well in head-injured patients.106,107 In general, the performance of APACHE II, APACHE III, and SAPS II for these disease processes was similar to that reported for heterogeneous ICU patients.

APACHE II and III have consistently performed poorly in trauma,46,108,109 in postoperative patients,110,111 and in women with eclampsia.112

Inaccuracy of Scoring Systems for Certain Types of Intensive Care Units or Different Geographic Regions

The patient sample used to derive and validate a scoring system can influence the mortality predictions. There are potentially important differences in outcomes of comparable patients in community versus teaching hospitals, in different regions of a country, and in different countries because of the influence of health care funding and policy. For example, a comparison of New Zealand and U.S. hospitals demonstrated different patient selection and fewer ICU admissions in New Zealand, and yet found similar hospital mortality.55

Some investigators have used scoring systems to compare critical care in different countries but came to sharply different conclusions. For example, Sirio and coworkers113 used APACHE II to compare Japan and the United States. Despite an ROC curve area of only 0.78 in the Japanese patient sample, they concluded that APACHE II performs well in Japan. In contrast, the Intensive Care Society APACHE II study114 examined 8724 critically ill patients and reported that crude death rates in hospital varied more than twofold between intensive care units in Britain and Ireland. Application of the APACHE II equation produced an ROC value of 0.83 and failed to explain outcome in four intensive care units. They concluded that the American APACHE II equation did not fit their data uniformly, and cited systematic differences in medical definitions and diagnostic labeling, diagnostic mix, measurement of physiologic variables, effectiveness of treatment, and differences in age-specific health status between the two countries.

The performance of APACHE III has been assessed in several countries including Brazil,115 the United Kingdom,116 Korea,117 and Australia.118 In most countries, the observed hospital mortality was significantly higher than the APACHE III predicted mortality rate. In the Australian study, when the model was corrected for hospital characteristics, the observed hospital mortality rate was not different. The area under the ROC curve was 0.92. The APACHE III mortality model, when adjusted for hospital characteristics had good discrimination and calibration in the Australian adult ICU population.

There has been rapid growth in the number and types of severity-of-illness scoring systems in critical care, and they are increasingly used for clinical research, administrative tasks, quality assurance, and individual patient prognosis. Therefore, physicians and administrators must understand the principles underlying development and testing of these systems, as well as the sources of errors in their development, to interpret the literature and to decide how and which system to use. We recommend four uses of severity-of-illness scoring systems. First, scoring systems are useful in clinical trials and in clinical research. The scoring system used must be validated and published in peer-reviewed literature. When researchers and clinicians have a common language for description of severity of illness, clinicians can compare the patients in studies with the patients in their own practices to decide how the results of the studies influence their own practices.

Second, scoring systems may be used for administrative purposes, to describe resource utilization relative to acuity of illness, and to assist with resource-allocation decisions.

The third potential use of scoring systems is to assess ICU performance. However, several biases limit this application because very little is known about accuracy of generalizations of scoring systems to different categories of hospitals, hospitals from different countries with different health care systems, or even different ICUs in the same hospital. Therefore, we believe that use of scoring systems to compare ICU performance is limited and requires further evaluation.

The fourth potential use of scoring systems is to assess individual patient prognosis and to guide care. We believe that scoring systems have limited use for individual patient prognosis and care decisions. At best, they can guide physicians, families, and patients in difficult decisions. Patients' preferences and patients' quality of life prior to ICU admission cannot be integrated into mathematical models. Severity-of-illness scoring systems predict probability of mortality, but they are not helpful in assessing probability of death in the 6 months following discharge from the ICU. Finally, scoring systems do not predict quality of life or return to independent living of patients.