Risk assessment tools for predicting surgical outcomes of patients who undergo elective abdominal aortic aneurysm repair

Risk assessment tools for predicting surgical outcomes of patients who undergo elective abdominal aortic aneurysm repair

Abdominal aortic aneurysm: diagnosis and management

Evidence review H

NICE Guideline, No. 156

London: National Institute for Health and Care Excellence (NICE); 2020 Mar.

ISBN-13: 978-1-4731-3452-2

Risk assessment tools for predicting surgical outcomes of patients who undergo elective abdominal aortic aneurysm repair

Review question

What is the accuracy of available risk assessment tools in predicting poor and good surgical outcomes in people with unruptured abdominal aortic aneurysms?

Introduction

Various multifactorial risk models have been developed that aim to facilitate decision making before abdominal aortic aneurysm (AAA) repair; however, there is no consensus as to which tools should be used and when they should be used. This review question aims to determine which assessment tools are accurate in predicting surgical outcomes after elective AAA repair and might therefore inform patients in their decision to undergo surgery for an unruptured AAAs.

PICO table

Methods and process

This evidence review was developed using the methods and process described in Developing NICE guidelines: the manual. Methods specific to this review question are described in the review protocol in Appendix A.

Declarations of interest were recorded according to NICE’s 2014 conflicts of interest policy.

A single broad search was used to identify all studies that examine the diagnosis, surveillance or monitoring of AAAs. This was a ‘bulk’ search that covered multiple review questions. The database was sifted to identify all studies that met the criteria detailed in Table 1. The relevant review protocol can be found in Appendix A.

Table 1

Inclusion criteria.

Cohort studies in which multivariate models were used to assess the accuracy of risk assessment tools (risk prediction models) for predicting peri- and postoperative outcomes of patients undergoing EVAR or open repair procedures were considered for inclusion. Prospective and retrospective cohort studies with sample sizes greater than 500 participants were included.

The included studies all reported the area under the curve (AUC) of receiver operating characteristic (ROC) curves for each model. A ROC curve plots the sensitivity of a model against its specificity across the full range of possible thresholds scores. Accuracy, in terms of being able to discriminate between cases and non-cases, is then measured by the AUC. The committee interpreted AUCs in accordance with thresholds suggested by Hosmer and Lemeshow (2000). An area under the curve (AUC) of 1 represents a perfect prediction; an area less than of 0.6 represents a worthless prediction (equivalent to ‘chance’). An AUC value between 0.6 and 0.69 indicates poor model discrimination. Values of 0.7 to 0.79 indicates acceptable model discrimination; values of 0.8 to 0.89 indicate excellent discrimination, and values greater than 0.9 indicate outstanding discrimination.

It was not appropriate to pool AUCs from identified studies due to dissimilar definitions of outcome, factors, and mix of confounders between studies. Where a model was examined in two or more studies, we have reported the individual AUC with 95% CIs reported by each study, and a summary median and range of AUCs for the study sample. Where a model was examined in a single study we have reported the AUC with the reported 95% CIs.

Studies were excluded if they:

were case-control or cross-sectional studies
were not in English
were not full reports of the study (for example, published only as an abstract)
were not peer-reviewed.

Clinical evidence

Included studies

From an initial database of 16,274 abstracts, 66 were identified as being potentially relevant. Following full-text review of these articles, 10 studies were included. These included 4 prospective cohort studies and 6 retrospective cohort studies.

An update literature search was performed and provided by Cochrane, in December 2017. The search found a total of 2,180 abstracts; of which, 5 full manuscripts were ordered. Upon review of the full manuscripts, none of the studies met the inclusion criteria for this review question.

Excluded studies

The list of papers excluded at full-text review, with reasons, is given in Appendix G.

Summary of clinical studies included in the evidence review

Table 2

Included studies.

See Appendix D for full evidence tables.

Quality assessment of clinical studies included in the evidence review

The GRADE working group has not published criteria for assessing imprecision in relation to AUC statistics. For the current review, the AUC classification categories referred to above were used. Minimal important difference (MID) levels of 0.7 and 0.8 were chosen for the assessment of imprecision, to be applied to the range of AUCs reported across contributing studies (or to the 95% confidence interval where a model was evaluated by a single study). When evidence on the prognostic utility of a risk assessment tool was obtained from a single study, the evidence was downgraded one level if the 95% CI around an AUC crossed one MID (0.7 or 0.8), or two levels if the 95% CI crossed both MIDs. When evidence on the prognostic utility of a risk assessment tool was obtained from more than one study, the evidence was downgraded one level if the AUC range crossed one MID (0.7 or 0.8), or two levels if the AUC range crossed both MIDs.

See Appendix E for full modified GRADE tables.

Economic evidence

Included studies

A literature search was conducted jointly for all review questions by applying standard health economic filters to a clinical search for AAA. This search returned a total of 5,173 citations. Following review of all titles and abstracts, no studies were identified as being potentially relevant to risk factors associated with AAA expansion or rupture. No full texts were retrieved, and no studies were included as economic evidence.

An update search was conducted in December 2017, to identify any relevant health economic analyses published during guideline development. The search found 814 abstracts; all of which were not considered relevant to this review question. As a result no additional studies were included.

Excluded studies

No studies were retrieved for full-text review.

Economic model

This review question does not lend itself to economic evaluation, and was not prioritised by the committee for economic modelling. As such, no economic model was developed for this review question.

Evidence statements

An area under the curve (AUC) of 1 represents a perfect prediction; an area less than of 0.6 represents a worthless prediction (equivalent to ‘chance’). An AUC value between 0.6 and 0.69 indicates poor model discrimination. Values of 0.7 to 0.79 indicate acceptable model discrimination; values of 0.8 to 0.89 indicate excellent discrimination, and values greater than 0.9 indicate outstanding discrimination.

30-day mortality

People undergoing EVAR or open repair

Very low- to low-quality evidence from 4 cohort studies, including up to 8,271 people with unruptured AAA, indicated that the Comorbidity Severity Score (CSS), Glasgow Aneurysm Scale (GAS), modified Leiden score and the Vascular Governance North West (VGNW) risk model had acceptable discriminatory power at predicting 30-day mortality after EVAR or open surgical repair.

People undergoing EVAR

Very low-quality evidence from 1 cohort study, including 862 people with unruptured AAA, indicated that the modified Leiden score had acceptable discriminatory power at predicting 30-day mortality after EVAR. Very low-quality evidence from 2 cohort studies, including up to 6,360 people with unruptured AAA, indicated that the CSS and the GAS had poor discriminatory power at predicting 30-day mortality after EVAR.

People undergoing open repair

Very low-quality evidence from 1 cohort study, including up to 862 people with unruptured AAA, indicated that the CSS and the modified Leiden score had acceptable discriminatory power at predicting 30-day mortality after open surgical repair. Very low-quality evidence from 2 cohort studies, including 2,773 people with unruptured AAA, indicated that the GAS had poor discriminatory power at predicting 30-day mortality after open surgical repair.

In-hospital mortality

People undergoing EVAR or open repair Moderate-quality evidence from 1, including up to 1,124 people with unruptured AAA, indicated that the British Aneurysm Repair (BAR) score had excellent discriminatory power at predicting in-hospital mortality after EVAR or open surgical repair.

Very low- to moderate-quality evidence from 4 cohort studies, including up to 19,140 people with unruptured AAA, indicated that the Medicare tool, Physiological and Operative Severity Score for enUmeration of Mortality (POSSUM tool) and the VGNW risk model had acceptable discriminatory power at predicting in-hospital mortality after EVAR or open surgical repair.

Very low- to moderate-quality evidence from 3 cohort studies, including up to 15,322 people with unruptured AAA, indicated that the GAS, Vascular-POSSUM tool and the Vascular Biochemical and Haematological Outcome Model (VBHOM) had poor discriminatory power at predicting in-hospital mortality after EVAR or open surgical repair.

People undergoing EVAR

Low-quality evidence from 1, including up to 1,124 people with unruptured AAA, indicated that the British Aneurysm Repair (BAR) score had acceptable discriminatory power and the Medicare tool had poor discriminatory power at predicting in-hospital mortality after EVAR.

Low-quality evidence from the same study indicated that the VGNW had a discriminatory power no better than chance at predicting in-hospital mortality after EVAR.

People undergoing open repair

Moderate-quality evidence from 1, including up to 1,124 people with unruptured AAA, indicated that the British Aneurysm Repair (BAR) score had acceptable discriminatory power while the Medicare tool and the VGNW risk model had poor discriminatory power at predicting in-hospital mortality after EVAR.

Mortality after 1 year in people undergoing EVAR or open repair

Very low-quality evidence from 1 retrospective cohort study, including 1,096 patients with unruptured AAA, indicated that the Carlisle calculator had acceptable discriminatory power at predicting mortality at 1 and 2 years. Very-low quality evidence from the same study indicated that the Carlisle calculator had poor discriminatory power at predicting mortality at 3, 4 and 5 years.

Postoperative morbidity

Low-quality evidence from 1 retrospective cohort study, including 1,911 patients with unruptured AAA, indicated that the GAS had poor discriminatory power at predicting cardiac complications (type of complications were not specified) after open surgical repair. Moderate-quality evidence from the same study indicated that the GAS had poor discriminatory power at predicting severe postoperative complications (including cardiac, cerebrovascular, renal, pulmonary venous, and peripheral arterial complications, as well as sepsis) after open surgical repair.

Length of stay

Moderate-quality evidence from 1 retrospective cohort study, including 1,911 patients who underwent with unruptured AAA, indicated that the GAS had poor discriminatory power at predicting prolonged length of stay (longer than 5 days) in intensive care after open surgical repair.

The committee’s discussion of the evidence

Interpreting the evidence

The outcomes that matter most

The committee agreed that the outcomes which matter most are mortality and complications that occur within 30 days of surgery. The committee considered that these outcomes were more important than long-term outcomes because their clinical experience highlighted that patients undergoing AAA surgery are at risk of experiencing more serious complications soon after surgery.

The quality of the evidence

The committee only considered studies where a pre-existing risk assessment tool was tested on a validation cohort. Studies in which risk assessment tools were developed using a derivation cohort and tested on the same cohort were not considered in this review. This was because these types of studies only assessed the internal validity of risk models (the degree to which errors have been minimised within a study). The committee believed that it was more important to evaluate the external validity (the degree to which a study’s findings are generalisable to wider populations and other settings) of risk models as it enabled them to determine the prognostic utility of the tools.

The committee noted that investigators from the majority of included studies collected data from national or international disease registries. It was considered that this type of approach to data collection may have introduced bias to findings due to an inability to accurately record and assess confounding. One study in particular (Giles et al., 2009) was considered to be at high risk of bias because investigators assessed codes from a health insurance provider database to ascertain the presence of risk factors, and subsequently used the data to calculate risk scores.

Benefits and harms

The committee concluded that the majority of assessed risk assessment tools had poor-to-acceptable discriminatory power as pooled estimates of AUCs across included studies ranged from 0.65 to 0.75. They contrasted this with equivalent predictive statistics, such as QRISK2, which is recommended by NICE for predicting cardiovascular disease (CG181), on the basis of AUCs between 0.77 and 0.84, which would be classified as acceptable-to-excellent discrimination using the rules of thumb adopted here. The committee noted that one study by Grant et al. (2014) suggested that the British Aneurysm Repair Score (BAR) had excellent discriminatory power at predicting in-hospital mortality in a heterogeneous group of patients who underwent endovascular or open surgical repair (AUC of 0.83). Upon examination of a treatment-specific subgroup analysis, the BAR score had acceptable discriminatory power at predicting in-hospital mortality in patients who only underwent endovascular repair (AUC of 0.75). The same was observed for patients who only underwent open repair (AUC of 0.70). In light of the variation between the overall and treatment-specific AUCs, the committee had little confidence in the discriminatory power of the BAR score at predicting in-hospital mortality. Overall, the committee considered the AAA tools assessed in this review to have insufficient discriminatory power for predicting postoperative outcomes of patients undergoing elective AAA surgery. There was little confidence about the clinical utility of the assessment tools as the committee could not see how using tools with AUCs of around 0.70 would lead to appropriate decisions about patient management and prognostic outcomes.

The committee considered that use of risk assessment tools with insufficient discriminatory power could have potentially harmful effects on patient care. This is because such tools could result in the decision to operate on a patient who shouldn’t be operated on, or vice versa. The committee discussed decision-making without the use of risk assessment tools. They noted that most of the clinical data used to derive risk assessment tools are commonly collected and are already available before surgery. They agreed that individual variables (as opposed to risk models) can be still useful for making judgments of an individual’s risk of postoperative morbidity and mortality.

Cost effectiveness and resource use

The committee considered that the recommendations were unlikely to have an impact on costs or resource use within the NHS as risk assessment tools are not routinely used outside the context of research.

Other factors the committee took into account

The committee did not want to preclude development of tools for assessing postoperative outcomes of AAA surgery. Thus, the committee chose to specify individual risk assessment that should not be used rather than state that all risk assessment tools should not be used.

The committee decided against making a research recommendation because extensive research into risk assessment tools for AAA surgery has already been performed over recent decades and further research in this area is unlikely to be viewed as a priority.

Appendices

Appendix A. Review protocols

Review protocol for risk assessment tools for predicting surgical outcomes of patients who undergo elective AAA repair

Table

i) Prospective observational studies using multivariate analysis; population >500 ii) Prospective observational studies using smaller populations (>200) will be considered if insufficient evidence is identified

Appendix B. Literature search strategies

Clinical search literature search strategy

Main searches

Bibliographic databases searched for the guideline

Cumulative Index to Nursing and Allied Health Literature - CINAHL (EBSCO)
Cochrane Database of Systematic Reviews – CDSR (Wiley)
Cochrane Central Register of Controlled Trials – CENTRAL (Wiley)
Database of Abstracts of Reviews of Effects – DARE (Wiley)
Health Technology Assessment Database – HTA (Wiley)
EMBASE (Ovid)
MEDLINE (Ovid)
MEDLINE Epub Ahead of Print (Ovid)
MEDLINE In-Process (Ovid)

Identification of evidence for review questions

The searches were conducted between November 2015 and October 2017 for 31 review questions (RQ). In collaboration with Cochrane, the evidence for several review questions was identified by an update of an existing Cochrane review. Review questions in this category are indicated below. Where review questions had a broader scope, supplement searches were undertaken by NICE.

Searches were re-run in December 2017.

Where appropriate, study design filters (either designed in-house or by McMaster) were used to limit the retrieval to, for example, randomised controlled trials. Details of the study design filters used can be found in section 4.

Search strategy review question 9

Table

Medline Strategy, searched 29th September 2016 Database: 1946 to September Week 3 2016

Health Economics literature search strategy

Sources searched to identify economic evaluations

NHS Economic Evaluation Database – NHS EED (Wiley) last updated Dec 2014
Health Technology Assessment Database – HTA (Wiley) last updated Oct 2016
Embase (Ovid)
MEDLINE (Ovid)
MEDLINE In-Process (Ovid)

Search filters to retrieve economic evaluations and quality of life papers were appended to the population and intervention terms to identify relevant evidence. Searches were not undertaken for qualitative RQs. For social care topic questions additional terms were added. Searches were re-run in September 2017 where the filters were added to the population terms.

Health economics search strategy

Table

Appendix C. Clinical evidence study selection

Appendix D. Clinical evidence tables

Download PDF (328K)

Appendix E. GRADE tables

An area under the curve (AUC) of 1 represents a perfect prediction; an area less than of 0.6 represents a worthless prediction (equivalent to ‘chance’). An AUC value between 0.6 and 0.69 indicates poor model discrimination. Values of 0.7 to 0.79 indicates acceptable model discrimination; values of 0.8 to 0.89 indicate excellent discrimination, and values greater than 0.9 indicate outstanding discrimination.

30-day mortality

Table

0.69 (Not reported) 0.74 (Not reported)

In-hospital mortality

Table

0.60 (0.56, 0.63) 0.69 (Not reported)

Mortality after 1 year

Table

Postoperative morbidity

Table

Length of stay

Table

Appendix F. Economic evidence study selection

Appendix G. Excluded studies

Clinical studies

Table

Economic studies

No full text papers were retrieved. All studies were excluded at review of titles and abstracts.

Appendix H. Glossary

Abdominal Aortic Aneurysm (AAA)

A localised bulge in the abdominal aorta (the major blood vessel that supplies blood to the lower half of the body including the abdomen, pelvis and lower limbs) caused by weakening of the aortic wall. It is defined as an aortic diameter greater than 3 cm or a diameter more than 50% larger than the normal width of a healthy aorta. The clinical relevance of AAA is that the condition may lead to a life-threatening rupture of the affected artery. Abdominal aortic aneurysms are generally characterised by their shape, size and cause:

Infrarenal AAA: an aneurysm located in the lower segment of the abdominal aorta below the kidneys.
Juxtarenal AAA: a type of infrarenal aneurysm that extends to, and sometimes, includes the lower margin of renal artery origins.
Suprarenal AAA: an aneurysm involving the aorta below the diaphragm and above the renal arteries involving some or all of the visceral aortic segment and hence the origins of the renal, superior mesenteric, and celiac arteries, it may extend down to the aortic bifurcation.

Abdominal compartment syndrome

Abdominal compartment syndrome occurs when the pressure within the abdominal cavity increases above 20 mm Hg (intra-abdominal hypertension). In the context of a ruptured AAA this is due to the mass effect of a volume of blood within or behind the abdominal cavity. The increased abdominal pressure reduces blood flow to abdominal organs and impairs pulmonary, cardiovascular, renal, and gastro-intestinal function. This can cause multiple organ dysfunction and eventually lead to death.

Cardiopulmonary exercise testing

Cardiopulmonary Exercise Testing (CPET, sometimes also called CPX testing) is a non-invasive approach used to assess how the body performs before and during exercise. During CPET, the patient performs exercise on a stationary bicycle while breathing through a mouthpiece. Each breath is measured to assess the performance of the lungs and cardiovascular system. A heart tracing device (Electrocardiogram) will also record the hearts electrical activity before, during and after exercise.

Device migration

Migration can occur after device implantation when there is any movement or displacement of a stent-graft from its original position relative to the aorta or renal arteries. The risk of migration increases with time and can result in the loss of device fixation. Device migration may not need further treatment but should be monitored as it can lead to complications such as aneurysm rupture or endoleak.

Endoleak

An endoleak is the persistence of blood flow outside an endovascular stent - graft but within the aneurysm sac in which the graft is placed.

Type I – Perigraft (at the proximal or distal seal zones): This form of endoleak is caused by blood flowing into the aneurysm because of an incomplete or ineffective seal at either end of an endograft. The blood flow creates pressure within the sac and significantly increases the risk of sac enlargement and rupture. As a result, Type I endoleaks typically require urgent attention.
Type II – Retrograde or collateral (mesenteric, lumbar, renal accessory): These endoleaks are the most common type of endoleak. They occur when blood bleeds into the sac from small side branches of the aorta. They are generally considered benign because they are usually at low pressure and tend to resolve spontaneously over time without any need for intervention. Treatment of the endoleak is indicated if the aneurysm sac continues to expand.
Type III – Midgraft (fabric tear, graft dislocation, graft disintegration): These endoleaks occur when blood flows into the aneurysm sac through defects in the endograft (such as graft fractures, misaligned graft joints and holes in the graft fabric). Similarly to Type I endoleak, a Type III endoleak results in systemic blood pressure within the aneurysm sac that increases the risk of rupture. Therefore, Type III endoleaks typically require urgent attention.
Type IV– Graft porosity: These endoleaks often occur soon after AAA repair and are associated with the porosity of certain graft materials. They are caused by blood flowing through the graft fabric into the aneurysm sac. They do not usually require treatment and tend to resolve within a few days of graft placement.
Type V – Endotension: A Type V endoleak is a phenomenon in which there is continued sac expansion without radiographic evidence of a leak site. It is a poorly understood abnormality. One theory that it is caused by pulsation of the graft wall, with transmission of the pulse wave through the aneurysm sac to the native aneurysm wall. Alternatively it may be due to intermittent leaks which are not apparent at imaging. It can be difficult to identify and treat any cause.

Endovascular aneurysm repair

Endovascular aneurysm repair (EVAR) is a technique that involves placing a stent –graft prosthesis within an aneurysm. The stent-graft is inserted through a small incision in the femoral artery in the groin, then delivered to the site of the aneurysm using catheters and guidewires and placed in position under X-ray guidance.

Conventional EVAR refers to placement of an endovascular stent graft in an AAA where the anatomy of the aneurysm is such that the ‘instructions for use’ of that particular device are adhered to. Instructions for use define tolerances for AAA anatomy that the device manufacturer considers appropriate for that device. Common limitations on AAA anatomy are infrarenal neck length (usually >10mm), diameter (usually ≤30mm) and neck angle relative to the main body of the AAA
Complex EVAR refers to a number of endovascular strategies that have been developed to address the challenges of aortic proximal neck fixation associated with complicated aneurysm anatomies like those seen in juxtarenal and suprarenal AAAs. These strategies include using conventional infrarenal aortic stent grafts outside their ‘instructions for use’, using physician-modified endografts, utilisation of customised fenestrated endografts, and employing snorkel or chimney approaches with parallel covered stents.

Goal directed therapy

Goal directed therapy refers to a method of fluid administration that relies on minimally invasive cardiac output monitoring to tailor fluid administration to a maximal cardiac output or other reliable markers of cardiac function such as stroke volume variation or pulse pressure variation.

Post processing technique

For the purpose of this review, a post-processing technique refers to a software package that is used to augment imaging obtained from CT scans, (which are conventionally presented as axial images), to provide additional 2- or 3-dimensional imaging and data relating to an aneurysm’s, size, position and anatomy.

Permissive hypotension

Permissive hypotension (also known as hypotensive resuscitation and restrictive volume resuscitation) is a method of fluid administration commonly used in people with haemorrhage after trauma. The basic principle of the technique is to maintain haemostasis (the stopping of blood flow) by keeping a person’s blood pressure within a lower than normal range. In theory, a lower blood pressure means that blood loss will be slower, and more easily controlled by the pressure of internal self-tamponade and clot formation.

Remote ischemic preconditioning

Remote ischemic preconditioning is a procedure that aims to reduce damage (ischaemic injury) that may occur from a restriction in the blood supply to tissues during surgery. The technique aims to trigger the body’s natural protective functions. It is sometimes performed before surgery and involves repeated, temporary cessation of blood flow to a limb to create ischemia (lack of oxygen and glucose) in the tissue. In theory, this “conditioning” activates physiological pathways that render the heart muscle resistant to subsequent prolonged periods of ischaemia.

Tranexamic acid

Tranexamic acid is an antifibrinolytic agent (medication that promotes blood clotting) that can be used to prevent, stop or reduce unwanted bleeding. It is often used to reduce the need for blood transfusion in adults having surgery, in trauma and in massive obstetric haemorrhage.

Final

Methods, evidence and recommendations

This evidence review was developed by the NICE Guideline Updates Team

Disclaimer: The recommendations in this guideline represent the view of NICE, arrived at after careful consideration of the evidence available. When exercising their judgement, professionals are expected to take this guideline fully into account, alongside the individual needs, preferences and values of their patients or service users. The recommendations in this guideline are not mandatory and the guideline does not override the responsibility of healthcare professionals to make decisions appropriate to the circumstances of the individual patient, in consultation with the patient and/or their carer or guardian.

Local commissioners and/or providers have a responsibility to enable the guideline to be applied when individual health professionals and their patients or service users wish to use it. They should do so in the context of local and national priorities for funding and developing services, and in light of their duties to have due regard to the need to eliminate unlawful discrimination, to advance equality of opportunity and to reduce health inequalities. Nothing in this guideline should be interpreted in a way that would be inconsistent with compliance with those duties.

NICE guidelines cover health and care in England. Decisions on how they apply in other UK countries are made by ministers in the Welsh Government, Scottish Government, and Northern Ireland Executive. All NICE guidance is subject to regular review and may be updated or withdrawn.

Bookshelf ID: NBK556919PMID: 32407035

Table 1Inclusion criteria

Parameter	Inclusion criteria
Population	People who are being assessed for surgery for a confirmed unruptured AAA Subgroups: by type of surgery
Assessment tools	Vascular Study Group of New England Cardiac Risk Index (VSG-CRI) Revised Cardiac Risk Index British Aneurysm Repair (BAR) Score Vascular Governance North West P Logistic risk model for mortality following elective AAA repair Medicare risk prediction for perioperative mortality during AAA repair Glasgow Aneurysm Scale (GAS) POSSUM prediction models, including V-POSSUM score Modified Customised Probability Index (m-CPI) Customised Probability Index (CPI) Surgical Risk Scale Vascular Biochemistry and Haematology Outcome Models (VBHOM) Estimation of Physiologic Ability and Surgical Stress (E-PASS) EVAR Risk Assessment (ERA) model, also known as the ‘Australasian model’ Co-morbidity Severity Score of the Society for Vascular Surgery and the American Association for Vascular Surgery (SVS/AAVS co-morbidity score) Leiden/modified Leiden score (John) Carlisle Calculator Eagle score Vanzetto score George’s score (aneurysm risk score)
Endpoints	Mortality Peri- and post-operative complications Successful exclusion of the aneurysm, aneurysm rupture, or further aneurysm growth Need for reintervention Quality of life Resource use, including length of hospital or intensive care stay, and costs

Table 2Included studies

Study	Details
Ambler Graeme K, Gohel Manjit S, Mitchell David C, Loftus Ian M, Boyle Jonathan R, Audit , Quality Improvement Committee of the Vascular Society of Great, Britain , and Ireland (2015) The Abdominal Aortic Aneurysm Statistically Corrected Operative Risk Evaluation (AAA SCORE) for predicting mortality after open and endovascular interventions. Journal of vascular surgery 61(1), 35–43 [PubMed: 24985535]	Study design: retrospective cohort study Location(s): UK Population: patients who underwent elective EVAR or open repair Sample size: 2,694 Outcome: in-hospital mortality Risk assessment tool(s): Vascular Biochemical and Haematological Outcome Model (VBHOM), Physiological and Operative Severity Score for enUmeration of Mortality (POSSUM), Vascular Governance North West (VGNW) risk model for mortality, and the Medicare risk prediction tool
Biancari F, Heikkinen M, Lepantalo M, and Salenius J P (2003) Glasgow Aneurysm Score in patients undergoing elective open repair of abdominal aortic aneurysm: A Finnvasc study. European Journal of Vascular and Endovascular Surgery 26(6), 612–617 [PubMed: 14603420]	Study design: retrospective cohort study Location(s): Finland, Netherlands, and UK Population: patients who underwent elective open repair of AAA Sample size: 1,911 Outcome: 30-day mortality, severe complications, cardiac complications ICU stay longer than 5 days Risk assessment tool(s): GAS
Biancari F, Hobo R, and Juvonen T (2006) Glasgow Aneurysm Score predicts survival after endovascular stenting of abdominal aortic aneurysm in patients from the EUROSTAR registry. The British journal of surgery 93(2), 191–4 [PubMed: 16392108]	Study design: retrospective cohort study Location(s): Finland, Netherlands, and UK Population: patients who underwent elective EVAR of aneurysms greater than 4.0 cm in diameter Sample size: 5,498 Outcome: 30-day mortality Risk assessment tool(s): GAS
Carlisle JB, Danjoux G, Kerr K. et al. (2015) Validation of long-term survival prediction for scheduled abdominal aortic aneurysm repair with an independent calculator using only pre-operative variables. Anaesthesia. 70 (6):654–65. doi: 10.1111/anae.13061. [PubMed: 25959175] [CrossRef]	Study design: retrospective cohort study Location(s): UK Population: patients undergoing elective EVAR or open repair Sample size: 1,096 Outcome: Mortality after 1 year Risk assessment tool(s): (John) Carlisle Calculator
Choke E, Lee K, McCarthy M, Nasim A, Naylor A R, Bown M, and Sayers R (2012) Risk models for mortality following elective open and endovascular abdominal aortic aneurysm repair: A single institution experience. European Journal of Vascular and Endovascular Surgery 44(6), 549–554 [PubMed: 22981409]	Study design: prospective cohort study Location(s): UK Population: patients undergoing elective EVAR or open repair Sample size: 1,153 Outcome: The primary end point was the composite of 30-day and in-hospital death Risk assessment tool(s): Medicare risk prediction tool, VGNW risk model and GAS
Eslami Mohammad H, Rybin Denis, Doros Gheorghe, Kalish Jeffrey A, Farber Alik, Vascular Study Group of New, and England (2015) Comparison of a Vascular Study Group of New England risk prediction model with established risk prediction models of in-hospital mortality after elective abdominal aortic aneurysm repair. Journal of vascular surgery 62(5), 1125–33.e2 [PubMed: 26187291]	Study design: retrospective cohort study Location(s): UK Population: patients who underwent elective EVAR or open repair Sample size: 4,431 Outcome: in-hospital mortality Risk assessment tool(s): Medicare risk assessment tool, VGNW risk model and GAS
Faizer Rumi, DeRose Guy, Lawlor D Kirk, Harris Kenneth A, and Forbes Thomas L (2007) Objective scoring systems of medical risk: a clinical tool for selecting patients for open or endovascular abdominal aortic aneurysm repair. Journal of vascular surgery 45(6), 1102–1108 [PubMed: 17543670]	Study design: retrospective cohort study Location(s): Canada Population: patients who underwent elective EVAR or open repair Sample size: 862 Outcome: 30-day mortality Risk assessment tool(s): GAS, Leiden Scoring System, and Comorbidity Severity Score (CSS)
Giles Kristina A, Schermerhorn Marc L, O’Malley A James, Cotterill Philip, Jhaveri Ami, Pomposelli Frank B, and Landon Bruce E (2009) Risk prediction for perioperative mortality of endovascular vs open repair of abdominal aortic aneurysms using the Medicare population. Journal of vascular surgery 50(2), 256–62 [PMC free article: PMC2785461] [PubMed: 19249184]	Study design: retrospective cohort study Location(s): USA Population: patients older than 67 years, with at least 2 years of prior medical claims, who underwent elective EVAR or open repair Sample size: 22,860 Outcome: the primary endpoint was the composite of 30-day and in-hospital mortality Risk assessment tool(s): Medicare risk prediction tool
Grant S W, Grayson A D, Purkayastha D, Wilson S D, McCollum C, participants in the Vascular Governance North West, and Programme (2011) Logistic risk model for mortality following elective abdominal aortic aneurysm repair. The British journal of surgery 98(5), 652–8 [PubMed: 21412997]	Study design: prospective cohort study Location(s): UK Population: patients undergoing elective EVAR or open repair of AAA Sample size: 1,936 Outcome: 30-day mortality Risk assessment tool(s): VGNW risk model
Grant S W, Grayson A D, Mitchell D C, and McCollum C N (2012) Evaluation of five risk prediction models for elective abdominal aortic aneurysm repair using the UK National Vascular Database. The British journal of surgery 99(5), 673–9 [PubMed: 22415901]	Study design: prospective cohort study Location(s): UK Population: patients undergoing elective EVAR or open repair Sample size: 10,891 Outcome: in-hospital mortality Risk assessment tool(s): Medicare risk prediction tool, VGNW risk model, GAS, VBHOM, Vascular-POSSUM (V-POSSUM)
Grant S W, Hickey G L, Carlson E D, and McCollum C N (2014) Comparison of three contemporary risk scores for mortality following elective abdominal aortic aneurysm repair. European Journal of Vascular and Endovascular Surgery 48(1), 38–44 [PMC free article: PMC4082141] [PubMed: 24837173]	Study design: prospective cohort study Location(s): UK Population: patients undergoing elective EVAR or open repair Sample size: 1,124 Outcome: in-hospital mortality Risk assessment tool(s): British Aneurysm repair (BAR) score, Medicare risk prediction tool and VGNW risk model for mortality

Review question 9	What is the accuracy of available risk assessment tools in predicting poor and good surgical outcomes in people with unruptured abdominal aortic aneurysms?
Objectives	To determine which assessment tools are accurate in predicting surgical outcome and might therefore inform patients in their decision to undergo surgery for an unruptured abdominal aortic aneurysms
Type of review	Prognostic
Language	English only
Study design	i) Prospective observational studies using multivariate analysis; population >500 ii) Prospective observational studies using smaller populations (>200) will be considered if insufficient evidence is identified
Status	i) Published papers only (full text) No date restrictions
Population	People who are being assessed for surgery for a confirmed unruptured abdominal aortic aneurysm Subgroups: by type of surgery
Assessment tools	Vascular Study Group of New England Cardiac Risk Index (VSG-CRI) Revised Cardiac Risk Index British Aneurysm Repair (BAR) score Vascular Governance North West P Logistic risk model for mortality following elective AAA repair Medicare risk prediction for perioperative mortality during AAA repair Glasgow Aneurysm Scale (GAS) POSSUM prediction models, including V-POSSUM score Modified Customised Probability Index (m-CPI) Customised Probability Index (CPI) Surgical Risk Scale Vascular Biochemistry and Haematology Outcome Models (VBHOM) Estimation of Physiologic Ability and Surgical Stress (E-PASS) EVAR Risk Assessment (ERA) model, also known as the ‘Australasian model’ Co-morbidity Severity Score of the Society for Vascular Surgery and the American Association for Vascular Surgery (SVS/AAVS co-morbidity score) Leiden/modified Leiden score (John) Carlisle Calculator Eagle score Vanzetto score George’s score (aneurysm risk score)
Endpoint	Mortality Peri- and post-operative complications Successful exclusion of the aneurysm, aneurysm rupture, or further aneurysm growth Need for re-intervention Quality of life Resource use, including length of hospital or intensive care stay, and costs
Other criteria for inclusion / exclusion of studies	Exclusion: Non-English language Abstract/non-published (i only)
Baseline characteristics to be extracted in evidence tables	Age Sex Size of aneurysm Comorbidities
Search strategies	See Appendix B
Review strategies	Double-sifting of randomly selected 20%. Appropriate NICE Methodology Checklists, depending on study designs, will be used as a guide to appraise the quality of individual studies. 20% will be appraised by a second reviewer. Data on all included studies will be extracted into evidence tables. Where statistically possible, a meta-analytic approach will be used to give an overall summary effect. All key findings from evidence will be presented in GRADE profiles and further summarised in evidence statements.
Key papers	Patterson BO, Holt PJ, Hinchliffe R, Loftus IM, Thompson MM. Predicting risk in elective abdominal aortic aneurysm repair: a systematic review of current evidence. Eur J Vasc Endovasc Surg. 2008 Dec;36(6):637–45 [PubMed: 18922709] – SYSTEMATIC REVIEW Bohm N, Wales L, Dunckley M, Morgan R, Loftus I, Thompson M. Objective risk-scoring systems for repair of abdominal aortic aneurysms: applicability in endovascular repair? Eur J Vasc Endovasc Surg. 2008 Aug;36(2):172–7 [PubMed: 18485762] Grant SW, Hickey GL, Carlson ED, McCollum CN. Comparison of three contemporary risk scores for mortality following elective abdominal aortic aneurysm repair. Eur J Vasc Endovasc Surg. 2014 Jul;48(1):38–44 [PMC free article: PMC4082141] [PubMed: 24837173] Grant SW, Sperrin M, Carlson E, Chinai N, Ntais D, Hamilton M, Dunn G, Buchan I, Davies L, McCollum CN, 2015. Calculating when elective abdominal aortic aneurysm repair improves survival for individual patients: development of the Aneurysm Repair Decision Aid and economic evaluation. Health Technology Assessment 19(32) [PMC free article: PMC4781543] [PubMed: 25924187] Tang TY, Walsh SR, Fanshawe TR, Seppi V, Sadat U, Hayes PD, Varty K, Gaunt ME, Boyle JR. Comparison of risk-scoring methods in predicting the immediate outcome after elective open abdominal aortic aneurysm surgery. Eur J Vasc Endovasc Surg. 2007 Nov;34(5):505–13 [PubMed: 17869138]

Medline Strategy, searched 29th September 2016 Database: 1946 to September Week 3 2016 Search Strategy:
1 Aortic Aneurysm, Abdominal/
2 Aortic Rupture/
3 (aneurysm* adj4 (abdom* or thoracoabdom* or thoraco-abdom* or aort* or spontan* or juxtarenal* or juxta-renal* or juxta renal* or paraerenal* or para-renal* or para renal* or suprarenal* or supra renal* or supra-renal* or short neck* or short-neck* or shortneck* or visceral aortic segment*)).tw.
4 or/1–3
5 prognosis.sh.
6 diagnosed.tw.
7 cohort.mp.
8 predictor:.tw.
9 death.tw.
10 exp models, statistical/
11 or/5–10
12 (sensitiv: or predictive value:).mp. or accurac:.tw.
13 11 or 12
14 “signs and symptoms”/
15 ((sign or signs) adj5 symptom*).tw.
16 Risk Factors/
17 factor*.tw.
18 predict*.tw.
19 or/14–18
20 13 or 19
21 4 and 20
22 animals/ not humans/
23 21 not 22 (12444)
24 limit 23 to english language

Medline Strategy
Economic evaluations
1 Economics/
2 exp “Costs and Cost Analysis”/
3 Economics, Dental/
4 exp Economics, Hospital/
5 exp Economics, Medical/
6 Economics, Nursing/
7 Economics, Pharmaceutical/
8 Budgets/
9 exp Models, Economic/
10 Markov Chains/
11 Monte Carlo Method/
12 Decision Trees/
13 econom*.tw.
14 cba.tw.
15 cea.tw.
16 cua.tw.
17 markov*.tw.
18 (monte adj carlo).tw.
19 (decision adj3 (tree* or analys*)).tw.
20 (cost or costs or costing* or costly or costed).tw.
21 (price* or pricing*).tw.
22 budget*.tw.
23 expenditure*.tw.
24 (value adj3 (money or monetary)).tw.
25 (pharmacoeconomic* or (pharmaco adj economic*)).tw.
26 or/1–25
Quality of life
1 “Quality of Life”/
2 quality of life.tw.
3 “Value of Life”/
4 Quality-Adjusted Life Years/
5 quality adjusted life.tw.
6 (qaly* or qald* or qale* or qtime*).tw.
7 disability adjusted life.tw.
8 daly*.tw.
9 Health Status Indicators/
10 (sf36 or sf 36 or short form 36 or shortform 36 or sf thirtysix or sf thirty six or shortform thirtysix or shortform thirty six or short form thirtysix or short form thirty six).tw.
11 (sf6 or sf 6 or short form 6 or shortform 6 or sf six or sfsix or shortform six or short form six).tw.
12 (sf12 or sf 12 or short form 12 or shortform 12 or sf twelve or sftwelve or shortform twelve or short form twelve).tw.
13 (sf16 or sf 16 or short form 16 or shortform 16 or sf sixteen or sfsixteen or shortform sixteen or short form sixteen).tw.
14 (sf20 or sf 20 or short form 20 or shortform 20 or sf twenty or sftwenty or shortform twenty or short form twenty).tw.
15 (euroqol or euro qol or eq5d or eq 5d).tw.
16 (qol or hql or hqol or hrqol).tw.
17 (hye or hyes).tw.
18 health* year* equivalent*.tw.
19 utilit*.tw.
20 (hui or hui1 or hui2 or hui3).tw.
21 disutili*.tw.
22 rosser.tw.
23 quality of wellbeing.tw.
24 quality of well-being.tw.
25 qwb.tw.
26 willingness to pay.tw.
27 standard gamble*.tw.
28 time trade off.tw.
29 time tradeoff.tw.
30 tto.tw.
31 or/1–30

Predictor	No of studies	Design	Risk of bias	Inconsistency	Indirectness	Imprecision	No. of participants	Area under the ROC curve (95% CI)	Quality
30-day mortality in patients undergoing EVAR or open repair
CSS	1 Faizer (2007)	Retrospective cohort study	Serious¹	Serious²	Not serious	Serious³	862	0.69^a (Not reported) 0.74^b (Not reported)	Very low
CSS	1 Faizer (2007)	Retrospective cohort study	Serious¹	Serious²	Not serious	Serious³	862	Median: 0.715 Range: 0.69 to 0.74	Very low
GAS	3 (Faizer 2007, Biancari 2006, Biancari 2003)	Retrospective cohort studies	Serious¹	Very serious⁴	Not serious	Serious³	8,271	0.47^a (Not reported) 0.72 ^b (Not reported) 0.70 (0.66, 0.74) 0.67 (0.61, 0.72)	Very low
GAS	3 (Faizer 2007, Biancari 2006, Biancari 2003)	Retrospective cohort studies	Serious¹	Very serious⁴	Not serious	Serious³	8,271	Median: 0.71 Range: 0.47 to 0.72	Very low
Modified Leiden score	1 Faizer (2007)	Retrospective cohort study	Serious¹	Not serious	Not serious	Serious³	862	0.70 ^a (Not reported) 0.71 ^b (Not reported)	Low
Modified Leiden score	1 Faizer (2007)	Retrospective cohort study	Serious¹	Not serious	Not serious	Serious³	862	Median: 0.705 Range: 0.70 to 0.71	Low
VGNW risk model	1 Grant (2011)	Prospective cohort study	Not serious	N/A	Not serious	Very serious⁵	829	0.70 (Not reported)	Low
30-day mortality in patients undergoing EVAR
CSS	1 Faizer (2007)	Retrospective cohort study	Serious¹	N/A	Not serious	Very serious⁵	862	0.69 (Not reported)	Very low
GAS	2 (Faizer 2007, Biancari 2006)	Retrospective cohort study	Serious¹	Very serious⁴	Not serious	Serious³	6,360	0.47 (Not reported) 0.70 (0.66, 0.74)	Very low
GAS	2 (Faizer 2007, Biancari 2006)	Retrospective cohort study	Serious¹	Very serious⁴	Not serious	Serious³	6,360	Median: 0.585 Range: 0.47 to 0.70	Very low
Modified Leiden score	1 Faizer (2007)	Retrospective cohort study	Serious¹	N/A	Not serious	Very serious⁵	862	0.70 (Not reported)	Very low
30-day mortality in patients undergoing open repair
CSS	1 Faizer (2007)	Retrospective cohort study	Serious¹	N/A	Not serious	Very serious⁵	862	0.74 (Not reported)	Very low
GAS	2 (Biancari 2003, Faizer 2007)	Retrospective cohort studies	Serious¹	Serious²	Not serious	Serious³	1,911 862	0.67 (0.61, 0.72) 0.72 (Not reported)	Very low
GAS	2 (Biancari 2003, Faizer 2007)	Retrospective cohort studies	Serious¹	Serious²	Not serious	Serious³	1,911 862	Median: 0.695 Range: 0.67 to 0.72	Very low
Modified Leiden score	1 Faizer (2007)	Retrospective cohort study	Serious¹	N/A	Not serious	Very serious⁵	862	0.71 (Not reported)	Very low

a: Data from the EVAR group of the Faizer (2007) trial

b: Data from the open repair group of the Faizer 2007 trial

1: Retrospective cohort study in which investigators retrospectively reviewed data from surgical registries, medical records or healthcare insurance provider databases to establish the presence or absence of risk factors, downgrade 1 level.

2: AUC range spans across 2 c-statistic classification categories, downgrade 1 level

3: AUC range crosses one minimal important difference (0.7 or 0.8), downgrade 1 level.

4: AUC range spans across 3 or more c-statistic classification categories, downgrade 2 levels. 3. 95% CI not reported, downgrade 2 levels.

5: 95% CI not reported, downgrade 2 levels.

Predictor	No of studies	Design	Risk of bias	Inconsistency	Indirectness	Imprecision	No. of participants	Area under the ROC curve (95% CI)	Quality
In-hospital mortality in patients undergoing EVAR or open repair
BAR	1 Grant (2014)	Prospective cohort study	Not serious	N/A	Not serious	Serious¹	1,124	0.83 (0.76, 0.89)	Moderate
GAS	2 (Grant 2012, Eslami 2015)	Retrospective cohort studies	Serious²	Not serious	Not serious	Not serious	15, 322	0.60 (0.56, 0.63) 0.69 (Not reported)	Moderate
GAS	2 (Grant 2012, Eslami 2015)	Retrospective cohort studies	Serious²	Not serious	Not serious	Not serious	15, 322	Median: 0.645 Range: 0.60 to 0.69	Moderate
Medicare tool	4 (Grant 2012, Ambler 2015, Eslami 2015, Grant 2014)	Prospective and retrospective cohort studies	Serious²	Not serious	Not serious	Not serious	19,140	0.71 (0.69, 0.74) 0.72 (Not reported) 0.77 (Not reported) 0.78 (0.70, 0.86)	Moderate
Medicare tool	4 (Grant 2012, Ambler 2015, Eslami 2015, Grant 2014)	Prospective and retrospective cohort studies	Serious²	Not serious	Not serious	Not serious	19,140	Median: 0.745 Range: 0.71 to 0.78	Moderate
POSSUM	1 Ambler (2015)	Retrospective cohort study	Serious²	N/A	Not serious	Very serious³	2,694	0.73 (Not reported)	Very low
V-POSSUM	1 Grant (2012)	Retrospective cohort study	Serious²	N/A	Not serious	Not serious	10,891	0.62 (0.59, 0.65)	Moderate
VBHOM	2 (Grant 2012, Ambler 2015)	Retrospective cohort studies	Serious²	Serious⁴	Not serious	Serious²	13,585	0.61 (0.58, 0.64) 0.74 (Not reported)	Very low
VBHOM	2 (Grant 2012, Ambler 2015)	Retrospective cohort studies	Serious²	Serious⁴	Not serious	Serious²	13,585	Median: 0.675 Range: 0.61 to 0.74	Very low
VGNW risk model	4 (Ambler 2015, Grant 2012, Grant 2014, Eslami 2015)	Prospective and retrospective cohort studies	Serious²	Not serious	Not serious	Serious²	19,140	0.70 (Not reported) 0.71 (0.68, 0.74) 0.75 (0.65, 0.84) 0.77 (Not reported)	Low
VGNW risk model	4 (Ambler 2015, Grant 2012, Grant 2014, Eslami 2015)	Prospective and retrospective cohort studies	Serious²	Not serious	Not serious	Serious²	19,140	Median: 0.73 Range: 0.70 to 0.77	Low
In-hospital mortality in patients undergoing EVAR
BAR	1 Grant (2014)	Prospective cohort study	Not serious	N/A	Not serious	Very serious³	1,124	0.75 (0.55, 0.95)	Low
Medicare tool	1 Grant (2014)	Prospective cohort study	Not serious	N/A	Not serious	Very serious³	1,124	0.66 (0.47, 0.85)	Low
VGNW risk model	1 Grant (2014)	Prospective cohort study	Not serious	N/A	Not serious	Very serious³	1,124	0.56 (0.31, 0.81)	Low
In-hospital mortality in patients undergoing open repair
BAR	1 Grant (2014)	Prospective cohort study	Not serious	N/A	Not serious	Serious¹	1,124	0.70 (0.61, 0.78)	Moderate
Medicare tool	1 Grant (2014)	Prospective cohort study	Not serious	N/A	Not serious	Serious¹	1,124	0.68 (0.58, 0.78)	Moderate
VGNW risk model	1 Grant (2014)	Prospective cohort study	Not serious	N/A	Not serious	Serious¹	1,124	0.64 (0.53, 0.75)	Moderate

1: AUC range (or confidence interval) crosses one minimal important difference (0.7 or 0.8), downgrade 1 level.

2: The majority of evidence was obtained from retrospective cohort studies in which investigators retrospectively reviewed data from surgical registries, medical records or healthcare insurance provider databases to establish the presence or absence of risk factors, downgrade 1 level.

3: 95% CI not reported, downgrade 2 levels.

4: AUC range spans across 2 c-statistic classification categories, downgrade 1 level.

Predictor	No of studies	Design	Risk of bias	Inconsistency	Indirectness	Imprecision	No. of participants	Area under the ROC curve (95% CI)	Quality
1 year mortality in patients undergoing EVAR or open repair
Carlisle calculator	1 Carlisle (2015)	Retrospective cohort study	Serious¹	N/A	Not serious	Very serious²	1,096	0.73 (Not reported)	Very low
2 year mortality in patients undergoing EVAR or open repair
Carlisle calculator	1 Carlisle (2015)	Retrospective cohort study	Serious¹	N/A	Not serious	Very serious²	1,096	0.71 (Not reported)	Very low
3 year mortality in patients undergoing EVAR or open repair
Carlisle calculator	1 Carlisle (2015)	Retrospective cohort study	Serious¹	N/A	Not serious	Very serious²	1,096	0.68 (Not reported)	Very low
4 year mortality in patients undergoing EVAR or open repair
Carlisle calculator	1 Carlisle (2015)	Retrospective cohort study	Serious¹	N/A	Not serious	Very serious²	1,096	0.67 (Not reported)	Very low
5 year mortality in patients undergoing EVAR or open repair
Carlisle calculator	1 Carlisle (2015)	Retrospective cohort study	Serious¹	N/A	Not serious	Very serious²	1,096	0.66 (Not reported)	Very low

1: Retrospective cohort study in which investigators retrospectively reviewed data from medical records to establish the presence or absence of risk factors, downgrade 1 level.

2: 95% CI not reported, downgrade 2 levels.

Predictor	No of studies	Design	Risk of bias	Inconsistency	Indirectness	Imprecision	No. of participants	Area under the ROC curve (95% CI)	Quality
Cardiac complications (not specified) in patients undergoing open repair
GAS	1 Biancari (2003)	Retrospective cohort study	Serious¹	N/A	Not serious	Serious²	1,911	0.69 (0.65, 0.73)	Low
Severe complications in patients undergoing open repair (complications included cardiac, cerebrovascular, renal, pulmonary venous, and peripheral arterial complications, as well as sepsis)
GAS	1 Biancari (2003)	Retrospective cohort study	Serious¹	N/A	Not serious	Not serious	1,911	0.65 (0.62, 0.68)	Moderate

1: Retrospective cohort study in which investigators retrospectively reviewed data from surgical registries, medical records or healthcare insurance provider databases to establish the presence or absence of risk factors, downgrade 1 level.

2: AUC 95% CI crosses one minimal important difference (0.7 or 0.8), downgrade 1 level.

Predictor	No of studies	Design	Risk of bias	Inconsistency	Indirectness	Imprecision	No. of participants	Area under the ROC curve (95% CI)	Quality
ICU length of stay longer than 5 days in patients undergoing open repair
GAS	1 Biancari (2003)	Retrospective cohort study	Serious¹	N/A	Not serious	Not serious	1,911	0.63 (0.59, 0.68)	Moderate

1: Retrospective cohort study in which investigators retrospectively reviewed data from surgical registries, medical records or healthcare insurance provider databases to establish the presence or absence of risk factors, downgrade 1 level.

No.	Study	Reason for exclusion
1	Amaranto Daniel J, Wang Edward C, Eskandari Mark K, Morasch Mark D, Rodriguez Heron E, Pearce William H, and Kibbe Melina R (2011) Normal preoperative white blood cell count is predictive of outcomes for endovascular procedures. Journal of vascular surgery 54(5), 1395–1403.e2 [PubMed: 21802244]	Not specific to AAA: Study included a mixed population of patients with carotid stenosis, peripheral arterial disease and aortic aneurysms (location unspecified).
2	Arya Shipra, Kim Sung In, Duwayri Yazan, Brewster Luke P, Veeraswamy Ravi, Salam Atef, and Dodson Thomas F (2015) Frailty increases the risk of 30-day mortality, morbidity, and failure to rescue after elective abdominal aortic aneurysm repair independent of age and comorbidities. Journal of vascular surgery 61(2), 324–31 [PubMed: 25312534]	Risk assessment tool (modified frailty index) is not included in the review protocol.
3	Baas Annette F, Janssen Kristel J. M, Prinssen Monique, Buskens Eric, and Blankensteijn Jan D (2008) The Glasgow Aneurysm Score as a tool to predict 30-day and 2-year mortality in the patients from the Dutch Randomized Endovascular Aneurysm Management trial. Journal of vascular surgery 47(2), 277–81 [PubMed: 18241749]	Sample size less than 500 participants.
4	Bertges Daniel J, Goodney Philip P, Zhao Yuanyuan, Schanzer Andres, Nolan Brian W, Likosky Donald S, Eldrup-Jorgensen Jens, Cronenwett Jack L, Vascular Study Group of New, and England (2010) The Vascular Study Group of New England Cardiac Risk Index (VSG-CRI) predicts cardiac complications more accurately than the Revised Cardiac Risk Index in vascular surgery patients. Journal of vascular surgery 52(3), 674–683.e3 [PubMed: 20570467]	Sample size less than 500 participants.
5	Bohm N, Wales L, Dunckley M, Morgan R, Loftus I, and Thompson M (2008) Objective Risk-scoring Systems for Repair of Abdominal Aortic Aneurysms: Applicability in Endovascular Repair?. European Journal of Vascular and Endovascular Surgery 36(2), 172–177 [PubMed: 18485762]	Sample size less than 500 participants.
6	Ballotta E, Da Giau, G, Gruppo M, Mazzalai F, Spirch S, and Terranova O (2009) Elective abdominal aortic aneurysm repair in the very elderly: a systematic review. Minerva medica 100(1), 95–103 [PubMed: 19277007]	This systematic review did not assess the accuracy of risk assessment tools for predicting postoperative outcomes. Instead investigators compared death and complication rates between patients who received EVAR and those who received open repair.
7	Bang J Y, Lee J B, Yoon Y, Seo H S, Song J G, and Hwang G S (2014) Acute kidney injury after infrarenal abdominal aortic aneurysm surgery: a comparison of AKIN and RIFLE criteria for risk prediction. British journal of anaesthesia 113(6), 993–1000 [PubMed: 25256546]	This retrospective study compared the ability of Acute Kidney Injury Network (AKIN) criteria and Risk, Injury, Failure, Loss, and end-stage (RIFLE) criteria in predicting mortality in patients undergoing infrarenal AAA surgery. This is out of scope of this review question.
8	Beck Adam W, Goodney Philip P, Nolan Brian W, Likosky Donald S, Eldrup-Jorgensen Jens, Cronenwett Jack L, Vascular Study Group of Northern New, and England (2009) Predicting 1-year mortality after elective abdominal aortic aneurysm repair. Journal of vascular surgery 49(4), 838–4 [PubMed: 19341875]	No risk assessment tools were assessed. This study explored risk factors which could be used to develop a risk prediction model.
9	Carlisle J B, Danjoux G, Kerr K, Snowden C, and Swart M (2015) Validation of long-term survival prediction for scheduled abdominal aortic aneurysm repair with an independent calculator using only pre-operative variables. Anaesthesia 70(6), 654–65 [PubMed: 25959175]	No multivariate analysis was performed. Investigators used univariate analysis (Kaplan-meier curves) to establish the discrimination and calibration of a bespoke survival calculator (not specified in the review protocol).
10	de Bruin , Jorg Lucas, Karthikesalingam Alan, Holt Peter J, Prinssen Monique, Thompson Matt M, Blankensteijn Jan D, Dutch Randomised Endovascular Aneurysm Management Study, and Group (2016) Predicting reinterventions after open and endovascular aneurysm repair using the St George’s Vascular Institute score. Journal of vascular surgery 63(6), 1428–1433.e1 [PubMed: 27005591]	No multivariate analysis was performed. Investigators used Kaplan meier curves (univariate analysis) to assess whether St Georges Vascular Institute Scores could be used to predict the need for re-intervention.
11	De Martino , Randall R, Nolan Brian W, Goodney Philip P, Chang Catherine K, Schanzer Andres, Cambria Robert, Bertges Daniel J, Cronenwett Jack L, Vascular Study Group of Northern New, and England (2010) Outcomes of symptomatic abdominal aortic aneurysm repair. Journal of vascular surgery 52(1), 5–12.e1 [PMC free article: PMC5240813] [PubMed: 20471771]	Study does not assess that predictive capacity of risk assessment tools. Instead, investigators report descriptive statistics of outcomes of patients who underwent surgical repair of symptomatic AAAs.
12	DeMartino R R, Huang Y, Mandrekar J, Goodney P P, Oderich G S, Kalra M, Bower T C, Cronenwett J L, and Gloviczki P (2017) External validation of a 5-year survival prediction model after elective abdominal aortic aneurysm repair. Journal of Vascular Surgery , [PMC free article: PMC6114131] [PubMed: 28807385]	Risk assessment tool is not included in the review protocol. Investigators used data from the VSGNE database but the tool assessed is not the same as the Vascular Study Group of New England Cardiac Risk Index (VSG-CRI) specified in the review protocol. Furthermore, it is unclear whether multivariate analysis was performed.
13	Deery Sarah E, Lancaster Robert T, Baril Donald T, Indes Jeffrey E, Bertges Daniel J, Conrad Mark F, Cambria Richard P, and Patel Virendra I (2016) Contemporary outcomes of open complex abdominal aortic aneurysm repair. Journal of vascular surgery 63(5), 1195–200 [PubMed: 27109792]	No risk assessment tools were assessed.
14	Dijkstra M L, Van Sterkenburg , S M M, Lardenoye J W, Zeebregts C J, and Reijnen M M. P. J (2016) One-year outcomes of endovascular aneurysm repair in high-risk patients using the endurant stent-graft: Comparison of the ASA classification and SVS/AAVS medical comorbidity grading system for the prediction of mortality and adverse events. Journal of Endovascular Therapy 23(4), 574–582 [PubMed: 27170150]	No multivariate analysis was used to assess the predictive capacity of the risk assessment tool. Instead, descriptive statistics were used to highlight mortality rates of patients categorised as low, medium or high risk according to the risk assessment tool.
15	Egorova Natalia, Giacovelli Jeannine K, Gelijns Annetine, Greco Giampaolo, Moskowitz Alan, McKinsey James, and Kent K Craig (2009) Defining high-risk patients for endovascular aneurysm repair. Journal of vascular surgery 50(6), 1271–9.e1 [PMC free article: PMC3081634] [PubMed: 19782526]	Risk assessment tool development: logistic regression was performed to identify independent risk factors which could be used for developing a bespoke risk assessment tool (not specified in the review protocol). The tool was not tested against a validation cohort.
16	Eslami Mohammad H, Rybin Denis V, Doros Gheorghe, and Farber Alik (2017) Description of a risk predictive model of 30-day postoperative mortality after elective abdominal aortic aneurysm repair. Journal of vascular surgery 65(1), 65–74.e2 [PubMed: 27720320]	Risk assessment tool development: logistic regression was performed to identify independent risk factors which could be used for developing a risk assessment tool (not specified in the review protocol). The tool was not tested against a validation cohort.
17	Eslami Mohammad H, Rybin Denis V, Doros Gheorghe, Siracuse Jeffrey J, and Farber Alik (2017) External validation of Vascular Study Group of New England risk predictive model of mortality after elective abdominal aorta aneurysm repair in the Vascular Quality Initiative and comparison against established models. Journal of vascular surgery, [PubMed: 28807384]	Conference abstract
18	Forbes Thomas L, Steiner Stefan H, Lawlor D Kirk, DeRose Guy, and Harris Kenneth A (2005) Risk-adjusted analysis of outcomes following elective open abdominal aortic aneurysm repair. Annals of vascular surgery 19(2), 142–8 [PubMed: 15782273]	Study did not assess the predictive capacity of risk assessment tools. Instead authors describe a method of categorising patients at risk of mortality following elective open AAA repair.
19	Fowkes F G. R, Greenhalgh R M, Powell J T, et al. (1998) Length of hospital stay following elective abdominal aortic aneurysm repair. European Journal of Vascular and Endovascular Surgery 16(3), 185–191 [PubMed: 9787298]	No risk assessment tools were assessed. Instead, investigators assessed which patient-related factors were associated with increased length of stay.
20	Goncalves F B, Ultee K H. J, Hoeks S E, Stolker R J, and Verhagen H J. M (2016) Life expectancy and causes of death after repair of intact and ruptured abdominal aortic aneurysms Presented in the Plenary Rapid Pace Session at the 2015 Vascular Annual Meeting of the Society for Vascular Surgery, Chicago, Ill, June 17–20, 2015. Journal of vascular surgery 63(3), 610–6	No risk assessment tool was assessed. Instead, authors assessed risk factors independently associated with mortality.
21	Goueffic Yann, Becquemin Jean-Pierre, Desgranges Pascal, and Kobeiter Hisham (2005) Midterm survival after endovascular versus open repair of infrarenal aortic aneurysms. Journal of endovascular therapy: an official journal of the International Society of Endovascular Specialists 12(1), 47–57 [PubMed: 15683271]	Study does not assess that predictive capacity of risk assessment tools. Instead, investigators report descriptive statistics of outcomes of patients who underwent surgical repair of unruptured AAAs.
22	Grant S W, Hickey G L, Grayson A D, Mitchell D C, and McCollum C N (2013) National risk prediction model for elective abdominal aortic aneurysm repair. The British journal of surgery 100(5), 645–53 [PubMed: 23338659]	Risk assessment tool development: logistic regression was performed to identify independent risk factors which could be used for developing a bespoke risk assessment tool (not specified in the review protocol). The tool was not tested against a validation cohort.
23	Hadjianastassiou V G, Tekkis P P, Goldhill D R, and Hands L J (2005) Quantification of mortality risk after abdominal aortic aneurysm repair. The British journal of surgery 92(9), 1092–8 [PubMed: 15997450]	Risk assessment tool development: logistic regression was performed to identify independent risk factors which could be used for developing a bespoke risk assessment tool which is not mentioned in the review protocol. Additionally, the study sample included patients who underwent elective and emergency aneurysm surgery.
24	Hadjianastassiou V G, Franco L, Jerez J M, Evangelou I E, Goldhill D R, Tekkis P P, and Hands L J (2006) Optimal prediction of mortality after abdominal aortic aneurysm repair with statistical models. Journal of Vascular Surgery 43(3), 467–473 [PubMed: 16520157]	Risk assessment tool development: logistic regression was performed to identify independent risk factors which could be used for developing a bespoke risk assessment tool (not specified in the review protocol). The tool was not tested against a validation cohort.
25	Hicks Caitlin W, Black James H, 3rd , Arhuidese Isibor, Asanova Luda, Qazi Umair, Perler Bruce A, Freischlag Julie A, and Malas Mahmoud B (2015) Mortality variability after endovascular versus open abdominal aortic aneurysm repair in a large tertiary vascular center using a Medicare-derived risk prediction model. Journal of vascular surgery 61(2), 291–7 [PubMed: 25154686]	No multivariate analysis was used to assess the predictive capacity of the risk assessment tool. Instead, descriptive statistics were used to highlight mortality rates of patients categorised as low, medium or high risk according to the risk assessment tool.
26	Hirzalla Osama, Emous Marloes, Ubbink Dirk Th, and Legemate Dink (2006) External validation of the Glasgow Aneurysm Score to predict outcome in elective open abdominal aortic aneurysm repair. Journal of vascular surgery 44(4), 712–717 [PubMed: 16930929]	Sample size less than 500 participants.
27	Lalys Florent, Durrmann Vincent, Dumenil Aurelien, Goksu Cemil, Cardon Alain, Clochard Elodie, Lucas Antoine, and Kaladji Adrien (2017) Systematic Review and Meta-Analysis of Preoperative Risk Factors of Type II Endoleaks after Endovascular Aneurysm Repair. Annals of vascular surgery 41, 284–293 [PubMed: 27903482]	Study explores risk factors associated with endoleaks after EVAR; however it is not clear whether this is after elective or emergency EVAR. Individual studies were reviewed to ascertain I they were relevant to this review question.
28	Kertai Miklos D, Steyerberg Ewout W, Boersma Eric, Bax Jeroen J, Vergouwe Yvonne, van Urk , Hero , Habbema J Dik F, Roelandt Jos R. T. C, and Poldermans Don (2003) Validation of two risk models for perioperative mortality in patients undergoing elective abdominal aortic aneurysm surgery. Vascular and endovascular surgery 37(1), 13–21 [PubMed: 12577134]	Sample size less than 500 participants.
29	Kodama A, Narita H, Kobayashi M, Yamamoto K, and Komori K (2011) Usefulness of POSSUM physiological score for the estimation of morbidity and mortality risk after elective abdominal aortic aneurysm repair in Japan. Circulation Journal 75(3), 550–556 [PubMed: 21282877]	Sample size less than 500 participants.
30	Khashram M, Williman J A, Hider P N, Jones G T, and Roake J A (2016) Systematic review and meta-analysis of factors influencing survival following abdominal aortic aneurysm repair. European Journal of Vascular and Endovascular Surgery 51(2), 203–215 [PubMed: 26602162]	Risk assessment tool (American Society of Anaesthesiologist score) not included in the review protocol.
31	Kim Jihoon T, Kim Min-Ju, Han Youngjin, Choi Ji Yoon, Ko Gi-Young, Kwon Tae-Won, and Cho Yong-Pil (2016) A new risk-scoring model for predicting 30-day mortality after repair of abdominal aortic aneurysms in the era of endovascular procedures. Annals of surgical treatment and research 90(2), 95–100 [PMC free article: PMC4751151] [PubMed: 26878017]	Risk assessment tool development: logistic regression was performed to identify independent risk factors which could be used for developing a bespoke risk assessment tool which is not mentioned in the review protocol.
32	Mani K, Venermo M, Beiles B, Menyhei G, Altreuther M, Loftus I, and Bjorck M (2015) Regional differences in case mix and peri-operative outcome after elective abdominal aortic aneurysm repair in the vascunet database. European Journal of Vascular and Endovascular Surgery 49(6), 646–652 [PubMed: 25752419]	Study did not assess the predictive capacity of risk assessment tools. Instead authors reported descriptive statistics. Additionally multivariate analysis was only performed to assess which risk factors were associated with mortality
33	Markar Sheraz R, Walsh Stewart R, Griffin Kathryn, Khandanpour Nader, Tang Tjun Y, and Boyle Jonathan R (2009) Assessment of a multifactorial risk index for predicting postoperative pneumonia after open abdominal aortic aneurysm repair. Vascular 17(1), 36–9 [PubMed: 19344581]	Risk assessment tool (Postoperative Pneumonia Risk Index) not specified in the review protocol. Furthermore, it is unclear whether multivariate analysis was performed.
34	Mastracci Tara M, Greenberg Roy K, Hernandez Adrian V, and Morales Catherine (2010) Defining high risk in endovascular aneurysm repair. Journal of vascular surgery 51(5), 1088–1095.e1 [PubMed: 20420976]	Risk assessment tool development: logistic regression was performed to identify independent risk factors which could be used for developing a bespoke risk assessment tool (not specified in the review protocol).
35	Matsumura Jon S, Katzen Barry T, Sullivan Timothy M, Dake Michael D, Naftel David C, Excluder Bifurcated Endoprosthesis, and Investigators (2009) Predictors of survival following open and endovascular repair of abdominal aortic aneurysms. Annals of vascular surgery 23(2), 153–8 [PubMed: 18774682]	No risk assessment tools were assessed. Instead investigators assessed which individual factors were independently associated with survival following EVAR.
36	Nesi F, Leo E, Biancari F, Bartolucci R, Rainio P, Satta J, Rabitti G, and Juvonen T (2004) Preoperative risk stratification in patients undergoing elective infrarenal aortic aneurysm surgery: Evaluation of five risk scoring methods. European Journal of Vascular and Endovascular Surgery 28(1), 52–58 [PubMed: 15177232]	Sample size less than 500 participants.
37	Mousa Albeir Y, Bozzay Joseph, Broce Mike, Yacoub Michael, Stone Patrick A, Najundappa Aravinda, Bates Mark C, and AbuRahma Ali F (2016) Novel Risk Score Model for Prediction of Survival Following Elective Endovascular Abdominal Aortic Aneurysm Repair. Vascular and endovascular surgery 50(4), 261–9 [PubMed: 27114446]	Risk assessment tool development: logistic regression was performed to identify independent risk factors which could be used for developing a bespoke risk assessment tool (not specified in the review protocol). It is unclear whether an external validation cohort was used.
38	Patterson B O, Holt P J. E, Hinchliffe R, Loftus I M, and Thompson M M (2008) Predicting Risk in Elective Abdominal Aortic Aneurysm Repair: A Systematic Review of Current Evidence. European Journal of Vascular and Endovascular Surgery 36(6), 637–645 [PubMed: 18922709]	Systematic review including prospective and retrospective observational studies. Additionally, some studies had sample sizes of less than 500 patients. Individual studies were assessed to determine whether they met the protocol’s inclusion criteria.
39	Patterson Benjamin Oliver, Karthikesalingam Alan, Hinchliffe Robert J, Loftus Ian M, Thompson Matt M, and Holt Peter J. E (2011) The Glasgow Aneurysm Score does not predict mortality after open abdominal aortic aneurysm in the era of endovascular aneurysm repair. Journal of vascular surgery 54(2), 353–7 [PubMed: 21458200]	No multivariate regression was performed to assess the predictive capacity of the Glasgow Aneurysm Score (GAS). Instead, investigators used univariate analysis to assess which components of the GAS were independently associated with death.
40	Pisimisis George T, Bechara Carlos F, Barshes Neal R, Lin Peter H, Lai Win S, and Kougias Panagiotis (2013) Risk factors and impact of proximal fixation on acute and chronic renal dysfunction after endovascular aortic aneurysm repair using glomerular filtration rate criteria. Annals of vascular surgery 27(1), 16–22 [PubMed: 23088805]	No risk assessment tools were assessed. Instead, investigators used multivariate regression to identify which factors were associated with acute kidney injury and chronic kidney disease.
41	Png Chien Yi M, Tadros Rami O, Beckerman William E, Han Daniel K, Tardiff Melissa L, Torres Marielle R, Marin Michael L, and Faries Peter L (2017) An anatomic risk model to screen post endovascular aneurysm repair patients for aneurysm sac enlargement. The Journal of surgical research 217, 29–35.e1 [PubMed: 28095987]	Risk assessment tool development: logistic regression was performed to identify independent risk factors which could be used for developing a bespoke risk assessment tool (not specified in the review protocol). Furthermore, the study included less than 500 participants.
42	Prytherch D R, Ridler B M. F, Ashley S, Audit Research Committee of the Vascular Society of Great, Britain, and Ireland (2005) Risk-adjusted predictive models of mortality after index arterial operations using a minimal data set. The British journal of surgery 92(6), 714–8 [PubMed: 15810045]	No multivariate analysis was performed: instead the descriptive statistics (chi-square test) was used to test the predictive power of the risk assessment tool.
43	Prytherch D R, Sutton G L, and Boyle J R (2001) Portsmouth POSSUM models for abdominal aortic aneurysm surgery. The British journal of surgery 88(7), 958–63 [PubMed: 11442527]	No multivariate analysis was performed. Instead, the descriptive statistics (chi-square test) was used to test the predictive power of the risk assessment tool.
44	Ramanan Bala, Gupta Prateek K, Sundaram Abhishek, Gupta Himani, Johanning Jason M, Lynch Thomas G, MacTaggart Jason N, and Pipinos Iraklis I (2013) Development of a risk index for prediction of mortality after open aortic aneurysm repair. Journal of vascular surgery 58(4), 871–8 [PMC free article: PMC4547535] [PubMed: 23676190]	Risk assessment tool development: logistic regression was performed to identify independent risk factors which could be used for developing bespoke a risk assessment tool (not specified in the review protocol). It is unclear whether an external validation cohort was used.
45	Samy A K, Murray G, and MacBain G (1994) Glasgow aneurysm score. Cardiovascular Surgery 2(1), 41–44 [PubMed: 8049922]	Risk assessment tool development: logistic regression was performed to identify independent risk factors which could be used for developing a risk assessment tool. No external validation cohort was used.
46	Samy A K, Murray G, and MacBain G (1996) Prospective evaluation of the Glasgow Aneurysm Score. Journal of the Royal College of Surgeons of Edinburgh 41(2), 105–107 [PubMed: 8632380]	No multivariate regression was performed to assess the predictive capacity of the Glasgow Aneurysm Score (GAS). Instead, investigators reported proportions of patients who survived, according to different GAS ranges.
47	Scali Salvatore T, Beck Adam W, Chang Catherine K, Neal Dan, Feezor Robert J, Stone David H, Berceli Scott A, and Huber Thomas S (2016) Defining risk and identifying predictors of mortality for open conversion after endovascular aortic aneurysm repair. Journal of vascular surgery 63(4), 873–81.e1 [PMC free article: PMC4808623] [PubMed: 26613868]	No risk assessment tools were assessed. Instead investigators assessed independent risk factors associated with conversion to open surgical repair.
48	Schlosser Felix J. V, Vaartjes Ilonca, van der Heijden , Geert J M. G, Moll Frans L, Verhagen Hence J. M, Muhs Bart E, de Borst, Gert J, Tiel Groenestege, Andreas T, Kardaun Jan W. P. F, de Bruin, and Agnes (2010) Mortality after elective abdominal aortic aneurysm repair. Annals of surgery 251(1), 158–64 [PubMed: 19838103]	No risk assessment tools were evaluated. Investigators assessed which risk factors were associated with mortality after elective AAA repair.
49	Sconfienza Luca Maria, Santagostino Ilaria, Di Leo, Giovanni, Piazza Raffaella, Gozzi Gino, Trimarchi Santi, and Sardanelli Francesco (2013) When the diameter of the abdominal aorta should be considered as abnormal? A new ultrasonographic index using the wrist circumference as a body build reference. European journal of radiology 82(10), e532–6 [PubMed: 23849990]	No risk assessment tools were evaluated. Additionally, multivariate analysis was not performed.
50	Setacci F, Sirignano P, Galzerano G, De Donato , G , Ceriello D, Paroni G, Cappelli A, and Setacci C (2012) Siena EVAR Score. The Journal of cardiovascular surgery 53(2), 229–34 [PubMed: 22456646]	Study did not assess the predictive value of a risk assessment tool. Instead, logistic regression was performed to identify independent risk factors which could be used to develop a bespoke risk assessment tool. It is unclear whether an external validation cohort was used
51	Steyerberg E W, Kievit J, de Mol Van Otterloo, J C, van Bockel, J H, Eijkemans M J, and Habbema J D (1995) Perioperative mortality of elective abdominal aortic aneurysm surgery. A clinical prediction rule based on literature and individual patient data. Archives of internal medicine 155(18), 1998–2004 [PubMed: 7575054]	Not an observational study (RQ9). This study combined results from literature data with individual patient data to assess risk factors which could be used to produce a clinical prediction rule. No external validation cohort was used.
52	Stone David H, Goodney Philip P, Kalish Jeffrey, Schanzer Andres, Indes Jeffrey, Walsh Daniel B, Cronenwett Jack L, Nolan Brian W, Vascular Study Group of New, and England (2013) Severity of chronic obstructive pulmonary disease is associated with adverse outcomes in patients undergoing elective abdominal aortic aneurysm repair. Journal of vascular surgery 57(6), 1531–6 [PMC free article: PMC3930461] [PubMed: 23466183]	No risk assessment tool was assessed. Instead, authors assessed risk factors independently associated with in-hospital and long-term mortality.
53	Tang T Y, Walsh S R, Prytherch D R, Wijewardena C, Gaunt M E, Varty K, and Boyle J R (2007) POSSUM models in open abdominal aortic aneurysm surgery. European journal of vascular and endovascular surgery : the official journal of the European Society for Vascular Surgery 34(5), 499–504 [PubMed: 17572117]	Wrong population: the study sample included patients with ruptured aneurysm who underwent emergency open repair (for ruptured aneurysms). This is out of scope of the review question.
54	Tang T Y, Walsh S R, Fanshawe T R, Seppi V, Sadat U, Hayes P D, Varty K, Gaunt M E, and Boyle J R (2007) Comparison of Risk-scoring Methods in Predicting the Immediate Outcome after Elective Open Abdominal Aortic Aneurysm Surgery. European Journal of Vascular and Endovascular Surgery 34(5), 505–513 [PubMed: 17869138]	Sample size less than 500 participants.
55	Tang Tjun, Walsh Stewart R, Fanshawe Thomas R, Gillard Jonathan H, Sadat Umar, Varty Kevin, Gaunt Michael E, and Boyle Jonathan R (2007) Estimation of physiologic ability and surgical stress (E-PASS) as a predictor of immediate outcome after elective abdominal aortic aneurysm surgery. American journal of surgery 194(2), 176–82 [PubMed: 17618800]	Unclear if multivariate analysis was performed.
56	Ultee Klaas H. J, Soden Peter A, Zettervall Sara L, Darling Jeremy, Verhagen Hence J. M, and Schermerhorn Marc L (2016) Conversion from endovascular to open abdominal aortic aneurysm repair. Journal of vascular surgery 64(1), 76–82 [PMC free article: PMC4926647] [PubMed: 27345505]	No risk assessment tool was assessed. Instead, authors assessed risk factors associated with conversion to open surgical repair during EVAR.
57	van Beek , Sytse C, Blankensteijn Jan D, Balm Ron, Dutch Randomised Endovascular Aneurysm Management trial, and collaborators (2013) Validation of three models predicting in-hospital death in patients with an abdominal aortic aneurysm eligible for both endovascular and open repair. Journal of vascular surgery 58(6), 1452–1457.e1 [PubMed: 23880548]	Sample size less than 500 participants.
58	Vande Geest, Jonathan P, Di Martino, Elena S, Bohra Ajay, Makaroun Michel S, and Vorp David A (2006) A biomechanics-based rupture potential index for abdominal aortic aneurysm risk assessment: demonstrative application. Annals of the New York Academy of Sciences 1085, 11–21 [PubMed: 17182918]	Wrong study design. This was a study which aimed to predict postoperative aneurysm rupture through evaluation of 13 three-dimensional computer simulations.
59	Vogel Todd R, Dombrovskiy Viktor Y, and Graham Alan M (2009) Elective abdominal aortic aneurysm repair: relationship of hospital teaching status to repair type, resource use, and outcomes. Journal of the American College of Surgeons 209(3), 356–63 [PubMed: 19717040]	No risk assessment tools were assessed. Instead, investigators assessed the relationship between type of hospital and type of AAA surgery performed.
60	Wisniowski Brendan, Barnes Mary, Jenkins Jason, Boyne Nicholas, Kruger Allan, and Walker Philip J (2011) Predictors of outcome after elective endovascular abdominal aortic aneurysm repair and external validation of a risk prediction model. Journal of vascular surgery 54(3), 644–53 [PubMed: 21788113]	Sample size less than 500 participants.
61	Yuo Theodore H, Sidaoui Joseph, Marone Luke K, Avgerinos Efthymios D, Makaroun Michel S, and Chaer Rabih A (2014) Limited survival in dialysis patients undergoing intact abdominal aortic aneurysm repair. Journal of vascular surgery 60(4), 908–13.e1 [PubMed: 24854417]	No risk assessment tool was assessed. Instead, authors assessed risk factors independently associated with mortality.