RESEARCH METHODS IN HEALTH Investigating health and health services Second edition Ann Bowling
Open University Press Buckingham • Philadelphia
Open University Press Celtic Court 22 Ballmoor Buckingham MK18 1XW email:
[email protected] world wide web: www.openup.co.uk and 325 Chestnut Street Philadelphia, PA 19106, USA First edition published 1997 Reprinted 1998, 1999, 2000 First published in this second edition 2002 Copyright © A. Bowling 2002 All rights reserved. Except for the quotation of short passages for the purpose of criticism and review, no part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of the publisher or a licence from the Copyright Licensing Agency Limited. Details of such licences (for reprographic reproduction) may be obtained from the Copyright Licensing Agency Ltd of 90 Tottenham Court Road, London, W1P 0LP. A catalogue record of this book is available from the British Library ISBN
0 335 20643 3 (pb) 0 335 20644 1 (hb)
Library of Congress Cataloging-in-Publication Data Bowling, Ann. Research methods in health: investigating health and health services / Ann Bowling.– 2nd ed. p. cm. Includes bibliographical references and index. ISBN 0-335-20644-1 (hb) – ISBN 0-335-20643-3 (pbk.) 1. Public health–Research–Methodology. 2. Community health services– Research–Methodology. I. Title. RA440.85 .B69 2001 362.1′07′2–dc21 2001045769
Typeset by Graphicraft Limited, Hong Kong Printed in Great Britain by Biddles Limited, Guildford and King’s Lynn
Contents
Preface Acknowledgements
Section 1 Investigating health services and health: the scope of research 1
2
Evaluating health services: multidisciplinary collaboration
xv xvi
1
5
Introduction Health services research The assessment of quality Audit Medical audit, clinical audit and quality assurance Evaluation Structure, process and outcome Appropriateness and inappropriateness Outcome
6 6 7 8 8 9 10 11 12
Summary of main points Key questions Key terms Recommended reading
15 15 16 16
Social research on health: sociological and psychological concepts and approaches
17
Introduction Sociological and psychological research on health
18 18
1 Health and illness The bio-medical model The social model of health Lay definitions of health
19 19 20 20
vi
Contents
3
Lay theories of illness Variations in medical and lay perspectives
21 22
2 Social factors in illness and responses to illness Social variations in health: structural inequalities Psycho-social stress and responses to stress Stigma, normalisation and adjustment The Sick Role and illness behaviour
23 23 24 27 29
3 Models of health behaviour Health lifestyles Health behaviour Models of health-related actions
33 33 34 35
4 Health-related quality of life Theoretical influences on measurement
38 39
5 Interactions between health professionals and patients Communication Patients’ evaluations of health care
41 41 42
Summary of main points Key questions Key terms Recommended reading
51 53 53 53
Health needs and their assessment: demography and epidemiology with Ian Rees Jones
54
Introduction
55
1 The assessment of health needs Health needs The need for health and the need for health care Methods of assessing health needs The role of epidemiological and demographic research
55 55 56 59 62
2 Epidemiology The role of epidemiology Epidemiological research Methods of epidemiology Assessing morbidity, mortality, incidence and prevalence
63 63 64 66 73
3 The role of demography Demographical methods in relation to assessing need Rates: births and deaths The need to standardise Analyses of survival
77 78 79 80 83
Summary of main points Key questions
86 87
Contents
4
Key terms Recommended reading
87 88
Costing health services: health economics with Ian Rees Jones
89
Introduction Health economics Demand, utility and supply Economic appraisal Cost minimisation Cost-effectiveness Cost–benefit analysis Marginal cost Complete costs Event pathways Opportunity cost Discounting Cost–utility analysis Cost–utility analysis and economic valuations of health Costing health services Study methods used for costings
90 90 92 93 94 94 95 96 96 97 98 100 100 101 107 111
Summary of main points Key questions Key terms Recommended reading
112 113 113 113
Section II The philosophy, theory and practice of research 5
vii
115
The philosophical framework of measurement
117
Introduction The philosophy of science Paradigms Objectivity and value freedom Deductive and inductive approaches The survival of hypotheses and paradigm shifts Theoretical influences on social research methods Social science and grounded theory Positivism Phenomenology Choice of methods
118 118 119 119 120 124 125 125 126 128 130
Summary of main points Key questions Key terms Recommended reading
131 132 132 132
viii
Contents 6
The principles of research
133
Introduction Searching the literature Literature reviews Meta-analyses Rigour Aims, objectives and hypotheses Concepts and theories Research proposals Research design and research methods Selection of measurement instruments Level of data and statistical techniques Reliability and validity Threats to reliability and validity Ethics and ethical committees Dissemination
134 134 136 136 138 138 139 141 143 144 144 147 153 156 159
Summary of main points Key questions Key terms Recommended reading
160 161 162 162
Section III Quantitative research: sampling and research methods 7
163
Sample size and sampling for quantitative research
165
Introduction
166
1 Calculation of sample size, statistical significance and sampling The sampling unit Calculation of sample size and statistical power Testing hypotheses, statistical significance, the null hypothesis Type I and type II errors One- or two-sided hypothesis testing Statistical, social and clinical significance Sampling frames Sampling Confidence intervals and the normal distribution External validity of the sample results 2 Methods of sampling Random sampling Non-random sampling: quota sampling Sampling for qualitative research Sampling for telephone interviews
166 166 167 169 170 172 173 174 176 177 182 183 183 186 187 188
Contents
8
9
ix
Summary of main points Key questions Key terms Notes Recommended reading
189 191 191 191 192
Quantitative research: surveys
193
Introduction
194
1 Survey methods The survey Descriptive and analytic surveys Retrospective (ex post facto), cross-sectional surveys Prospective, longitudinal surveys Cross-sectional and longitudinal cohort studies Triangulated research methods and surveys
194 194 195 197 197 200 201
2 Methods of analysing change in longitudinal surveys Analysing change Sample attrition and analysing change Stopping rules and analysis of interim results
202 202 209 212
Summary of main points Key questions Key terms Recommended reading
212 213 213 214
Quantitative research: experiments and other analytic methods of investigation
215
Introduction The experimental method Internal and external validity Reducing bias in participants and the investigating team Blind experiments The randomised controlled trial in health care evaluation Other analytic methods of investigation Before–after study with non-randomised control group After-only study with non-randomised control group Time series studies using different samples (historical controls) Geographical comparisons People acting as own controls Within-person, controlled site study Threats to the validity of causal inferences in other analytic studies
216 216 219 223 224 225 228 231 231
Summary of main points Key questions
234 235
232 232 233 233 233
x
Contents
10
Key terms Recommended reading
235 236
Sample selection and group assignment methods in experiments and other analytic methods
237
Introduction Random sampling Convenience and purposive sampling Volunteers Type of investigation and type of sampling frame Response rates: experiments and other analytical studies Ensuring similarity in group characteristics: random allocation Other allocation methods: cross-over methods Methods of group design for improving the basic RCT Common methods of controlling to obtain equivalence in non-randomised studies Summary of main points Key questions Key terms Recommended reading
Section IV
The tools of quantitative research 11
12
238 238 238 239 239 240 241 247 248 250 252 253 253 253
255
Data collection methods in quantitative research: questionnaires, interviews and their response rates
257
Introduction Structured and semi-structured questionnaires Postal questionnaires and self-administration Structured and semi-structured interviews Non-response Item non-response
258 258 259 260 263 269
Summary of main points Key questions Key terms Recommended reading
271 271 272 272
Questionnaire design
273
Introduction Planning Piloting Questionnaire layout The covering letter Question form, order and wording
274 274 275 275 277 278
Contents
13
14
xi
Rules for form Scores Constructing additional items and scales Attitude measurement scales Rules for order and wording Rules for questions by type of topic Checking the accuracy of responses Translating an instrument and cultural equivalence
278 286 287 288 294 300 306 307
Summary of main points Key questions Key terms Recommended reading
308 309 309 309
Techniques of survey interviewing
310
Introduction Types of interview The interviewer Interviewer handbooks Sampling by interviewers Interviewer training Interviewer bias Persistence in contacting respondents Approaching respondents Motivating people to respond Third parties and distractions Beginning the interview Rules for structured interviewing Interviewing techniques The end of the interview Recording responses Debriefing Quality control
311 311 311 313 315 316 317 318 319 320 322 323 324 326 330 331 332 332
Summary of main points Key questions Key terms Recommended reading
333 333 334 334
Preparation of quantitative data for coding and analysis
335
Introduction Coding Coding transfer sheets The code book Numerical values for codes Coding open questions Coding closed questions
336 336 338 339 340 341 343
xii
Contents Checking returned questionnaires Data entry on to computer Cleaning the data Checking for bias in the analyses Missing values and data checks Computer packages for the analysis of quantitative data The analysis
343 344 345 346 346 347 348
Summary of main points Key questions Key terms Recommended reading
349 350 350 350
Section V Qualitative and combined research methods, and their analysis 15
16
351
Unstructured and structured observational studies
355
Introduction Observation Participant observation Gaining access Hardware: video- and audiotapes Establishing validity and reliability Observation and triangulated methods Structured observations: what to record Time sampling Recording observed (non-verbal) body language Unstructured observations Combining structured and unstructured recordings Theoretical analysis of observational data Categorisation of observational data Narratives Audio-observation: conversation sampling Recording and analysing verbal communication
356 358 359 360 361 362 364 364 366 366 367 368 369 371 373 373 374
Summary of main points Key questions Key terms Recommended reading
375 376 376 376
Unstructured interviewing and focus groups
377
Introduction
378
1 Unstructured interviews Types of unstructured, in-depth interview In-depth interviewing: sample selection and size The process of the interview
378 378 379 381
Contents
17
xiii
Techniques of in-depth interviewing Analysis and presentation of in-depth interview data Categorising qualitative data: content analysis Rules for coding Computer programs for analysing qualitative data Narrative format
382 385 387 389 391 392
2 Focus groups Focus group interviews Group composition Appropriate topics for focus groups Methods of analysis
394 394 395 395 398
Summary of main points Key questions Key terms Recommended reading
399 400 400 401
Other methods using both qualitative and quantitative approaches: case studies, consensus methods, action research and document research
402
Introduction
403
1 Case studies The study of single or small series of cases Examples of case studies The analysis
403 403 404 406
2 Consensus methods Methods for establishing and developing consensus Delphi technique Consensus development panels Nominal group process The analysis
406 406 407 407 407 409
3 Action research and rapid appraisal techniques Action research Stages of action research Rapid appraisal
410 410 412 414
4 Document research Documents as sources of, or for, research Types of documents Authenticity, bias, error and interpretation Types of document research Analysis of documents Diary methods Analysis of diaries
415 415 416 418 419 421 424 427
Summary of main points Key questions
427 428
xiv
Contents Key terms Recommended reading
429 429
Glossary References Index
430 439 479
Preface to the second edition
This book is more than a text on research methods. It is an introduction to the theoretical concepts, as well as the descriptive and analytic research methods, that are used by the main disciplines engaged in research on health and health services. In order to understand why the various research methods are used, it is important to be aware of the conceptual backgrounds and scientific philosophies of those involved in research and evaluation, in particular in demography, epidemiology, health economics, psychology and sociology. The second edition, while essentially similar to the first edition, includes updated references and reference to key methodological developments (for example, cluster randomisation, utility analyses, patients’ preferences and perception of risk). The book is aimed at students and researchers of health and health services, health professionals and the policy-makers who have the responsibility for applying research findings, and who need to know how to judge the soundness of that research. The idea for the book, and its structure, is grounded in my career as a researcher on health and health service issues, and the valuable experience this has provided in meeting the challenges of research on people and organisations in real life settings. The varying terminology used by members of different disciplines in relation to the same research methods is often confusing. This variation simply reflects the multidisciplinary nature of this whole area, and the specialised languages of each discipline. While no descriptor can be labelled as incorrect, the multitude of them, especially when not clearly defined, can easily lead to confusion. Therefore, I have tried to justify the terminology used where it differs from that in other disciplines. Towards the end of the book I have included a Glossary. The first time each entry appears in the text it is highlighted in bold.
Section 1 Investigating health services and health: the scope of research ‘Would you tell me, please, which way I ought to go from here?’, asked Alice. ‘That depends a good deal on where you want to get to,’ said the cat. Lewis Carroll (1865) Alice’s Adventures in Worderland
INTRODUCTION Research is the systematic and rigorous process of enquiry which aims to describe phenonema and to develop and test explanatory concepts and theories. Ultimately it aims to contribute to a scientific body of knowledge. More specifically, in relation to the focus of this book, it aims to improve health, health outcomes and health services. This book aims to provide an overview of the range of research methods that are used in investigations of health and health services. Ultimately it is to guide the reader into choosing an appropriate research method and design in order to address a particular research question. However, it is not possible to place research methods in a hierarchy of excellence, as different research methods are appropriate for addressing different research questions. If the research question is descriptive, for example ‘what is the health status of population X?’, then a cross-sectional survey of a sample of that population is required to provide population estimates. The survey method will also enable the answers to secondary questions to be estimated for that population (e.g. ‘Are men more likely than women to report poor health status?’) and certain (non-causal) types of hypotheses to be tested (e.g. ‘Men will be X times more likely than women to report good health status’). If the research question is ‘Do women have worse health outcomes than men following acute myocardial infarction (AMI)?’ then a prospective, longitudinal survey of identified men and women who had suffered an AMI would be undertaken in order to be able to compare their health outcomes over time in the future.
2
Investigating health services and health: the scope of research If the research aims to find out information on a topic about which little is known, or is too complex or sensitive for the development of standardised instruments, then qualitative methods (e.g. observational methods, in-depth interviews and/or focus groups) may be more appropriate (e.g. ‘Is there quality of life on long-stay psycho-geriatric wards?’ ‘Are there dehumanising care practices in long stay institutions?’ ‘How do doctors prioritise their patient caseload?’). And if the research aims to investigate cause-and-effect issues then an experimental design is, in theory, required (e.g. ‘Do women aged 75+ have worse health outcomes than men aged 75+ following thrombolysis therapy for acute myocardial infarction?’ ‘Do patients with osteoarthritis of the knee benefit from physiotherapy?’ ‘Are specialists’ outreach clinics held in general practitioners’ surgeries as cost-effective as specialists’ out-patient clinics in hospitals?’). While the double-blind, randomised controlled trial is the true experimental design, and most appropriate for addressing these types of questions, there are also situations in which this method is unrealistic, impractical or inappropriate and other well-designed analytic (as opposed to descriptive) methods have to be employed instead (see Chapter 9). For some cause-and-effect questions, the RCT may be the most appropriate research design but it would be unethical to randomise people to interventions that are unacceptable, and the issue must therefore be addressed using other methods, such as a prospective, longitudinal survey of a population (e.g. ‘Does drinking spirits increase the risk of heart disease?’). Finally, research methods should not be seen in isolation from each other. A triangulated or combined methodological approach to addressing different facets of a research issue, using different methods which complement each other, is increasingly recommended as a means of establishing the external validity of the research. In the same way in which prospective, longitudinal surveys can inform the results from randomised controlled trials, so qualitative research findings can enhance quantitative survey data by placing the latter into real social contexts and enhancing understanding of relevant social processes. The importance of using triangulated research methods is enhanced by the multifaceted nature of health, and the multidisciplinary character of research on health and health services. This includes investigations by anthropologists, demographers, epidemiologists, health economists, health geographers, health policy analysts, health psychologists, historians, medical sociologists, statisticians and health professionals (clinicians, nurses, physiotherapists and so on). Specialists in public health medicine play a key role in health services research, as they are equipped with a range of research skills, including epidemiology. In Britain and in some other countries, they also have responsibility for assessing needs for health services in specific geographical areas, and advising purchasers on effective health care. There is a close working relationship between researchers investigating health and health services and health professionals, particularly in relation to the development of measures of clinical outcomes and the appropriateness of health care interventions.
Evaluating health Investigating health services: servicesmultidisciplinary and health: thecollaboration scope of research
3
One consequence of this multidisciplinary activity is that a wide range of qualitative and quantitative, descriptive and analytical research methods is available. This diversity should enrich the approach to research design, although there has been a tendency in research on health services to focus mainly on the experimental method. All methods have their problems and limitations, and the over-reliance on any one method, at the expense of using multiple research methods, to investigate the phenomenon of interest can lead to ‘a very limited tool box’ (Pope and Mays 1993), sometimes with questionable validity (Webb et al. 1966), and consequently to a limited understanding of the phenomena of interest. It is necessary at this point to distinguish between the terms health research and health services research.
Health research Health research has been defined in relation to health generally. As well as having an emphasis on health services, it has an important role in informing the planning and operation of services aiming to achieve health (Hunter and Long 1993): the process for obtaining systematic knowledge and technology which can be used for the improvement of the health of individual groups. It provides the basic information on the state of health and disease of the population; it aims to develop tools to prevent and cure illness and mitigate its effects and it attempts to devise better approaches to health care for the individual and the community. (Davies 1991) The broader aspects of health research are described in Chapters 2 and 3 (e.g. in relation to health needs and sociological and psychological aspects of health).
Health systems and health services research Health systems research has been defined fairly broadly as: ‘ultimately concerned with improving the health of a community, by enhancing the efficiency and effectiveness of the health system as an integrated part of the overall process of socio-economic development’ (Varkevisser et al. 1991). In Britain and the USA the general focus is on health services research, rather than on health systems research. Health services research is defined more narrowly in relation to the relationship between health service delivery and the health needs of the population: for example, as ‘the identification of the health care needs of communities and the study of the provision, effectiveness and use of health services’ (Medical Research Council, see Clarke and Kurinczuk 1992). While there is an overlap with health research, health services research needs to be translated into action to be of value and should ‘transcend the R (acquiring knowledge) and the D (translating that knowledge into action) divide’ (Hunter and Long 1993).
4
Investigating health services and health: the scope of research Each of these definitions emphasises the multidisciplinary nature of health research, health systems research and health services research. Health services research, for example, has been described as ‘a space within which disciplines can meet’ (Pope 1992), and as an area of applied research, rather than a discipline (Hunter and Long 1993). Within these definitions, the topics covered in Chapters 1, 3 and 4, on evaluating health services, health needs and their assessment (the latter also comes within the definition of broader health research) and the costing of health services, are encompassed by health services research. Chapter 2, on social research on health, also falls within both health research and health services research. Not everyone would agree with these definitions and distinctions. For example, some might categorise the assessment of needs as health research rather than health services research. What is important is not the distinctions and overlaps between these branches of research, but a respect for each discipline in relation to its contribution to a multidisciplinary body of knowledge about health and disease, health systems as a whole and health services. Finally, it should be pointed out that research on health services is not insulated from the society within which it is placed. It is often responsive to current policy and political issues (see Cartwright 1992), and is thus dependent upon decisions taken by others in relation to research topics and research funding. While it is common for researchers to initiate new research ideas, much of the funding for this research comes from government bodies, who tend to prioritise research and development on a local or national basis. The research topics are rarely value free. The research findings are also disseminated to members of a wide range of professional and management groups. In relation to this multidisciplinary nature, the agenda for research and the consumers of the research findings, it contrasts starkly with the traditional biomedical model of research.
Evaluating health services: multidisciplinary collaboration
1
5
Evaluating health services: multidisciplinary collaboration
Introduction Health services research The assessment of quality Audit Medical audit, clinical audit and quality assurance Evaluation Structure, process and outcome Appropriateness and inappropriateness Outcome
6 6 7 8 8 9 10 11 12
Summary of main points Key questions Key terms Recommended reading
15 15 16 16
6
Investigating health services and health: the scope of research
Introduction Research on health and health services ranges from descriptive investigations of the experience of illness and people’s perceptions of health and ill health (known as research on health, or health research) to evaluations of health services in relation to their appropriateness, effectiveness and costs (health services research). However, these two areas overlap and should not be rigidly divided, as it is essential to include the perspective of the lay person in health service evaluation and decision-making. Other related fields of investigation include audit, quality assurance, and the assessment of needs for health services (usually defined in terms of the need for effective services), which comes within the umbrella of health research but also has a crucial link with health services research. Audit and quality assurance are not strictly research in the sense of contributing to a body of scientific knowledge and adherence to rigorous methods of conducting research (quantitative or qualitative). Instead they are concerned with monitoring in order to ensure that predefined standards of care are met. They are increasingly important activities with the emphasis on clinical governance in health care (Lugon and Secker-Walker 1999). They are described briefly below with the other main areas of research activity.
Health services research It was explained in the introduction to Section I that health services research is concerned with the relationship between the provision, effectiveness and efficient use of health services and the health needs of the population. It is narrower than health research. More specifically, health services research aims to produce reliable and valid research data on which to base appropriate, effective, cost-effective, efficient and acceptable health services at the primary and secondary care levels. Thus, the research knowledge acquired needs to be developed into action if the discipline is to be of value; hence the emphasis throughout industry and service organisations on ‘research and development’. The focus is generally on: • the relationships between the population’s need and demand for health services, and the supply, use and acceptability of health services; • the processes and structures, including the quality and efficiency, of health services; • the appropriateness and effectiveness of health service interventions, in relation to effectiveness and cost-effectiveness, including patients’ perceptions of outcome in relation to the effects on their health, healthrelated quality of life and their satisfaction with the outcome.
Evaluating health services: multidisciplinary collaboration
7
These areas of research are addressed in more detail in this chapter and in the other chapters included in Section I. Health services research is distinct from audit and quality assurance, although they share the same concepts in relation to the evaluation of structure, process and outcome. Audit and quality assessment aim to monitor whether predefined and agreed standards have been met. Health services research has evaluation – rather than monitoring – as its aim. Health services research is also broader than traditional clinical research, which directly focuses on patients in relation to their treatment and care. Clinical research has traditionally focused on biochemical indicators, and more recently, and in selected specialties only, on the measurement of the broader quality of life of the patients. Health services research investigates the outcome of medical interventions from social, psychological, physical and economic perspectives. It has also been cogently argued that health services research should be concerned with the evaluation of the health sector in the broadest sense, and not limited to health services alone (Hunter and Long 1993). Quality assessment and audit will be described next, followed by the concepts central to the latter and to health services research: the evaluation of the structure, process and outcome, including appropriateness, of health services.
The assessment of quality The quality of care can be defined in relation to its effectiveness with regard to improving the patient’s health status, and how well it meets professionals’ and the public’s standards about how the care should be provided (Donabedian 1980). Higginson (1994) stated that quality of care needs to include effectiveness, acceptability and humanity, equity, accessibility and efficiency. Building on work by Shaw (1989) and Black (1990), she defined quality of health care in broad terms: • effectiveness (achieving the intended benefits in the population, under usual conditions of care); • acceptability and humanity (to the consumer and provider); • equity and accessibility (the provision and availability of services to everyone likely to benefit (in ‘need’)); • efficiency (greatest benefit for least cost). Higginson adds that patient empowerment might also be included, in order that they may increase their control over the services received, and each patient should be offered care that is appropriate. Quality is clearly relevant to health services research. Quality assurance and medical and clinical audit are all initiatives to establish and maintain quality in health care, and also involve the evaluation of structure, process and outcome in relation to quality.
8
Investigating health services and health: the scope of research
Audit Audit is directed at the maintenance and achievement of quality in health care. Audit aims to improve patient outcome, to develop a more costeffective use of resources and to have an educational function for health professionals. In theory, it should lead to change in clinical practice by encouraging a reflective culture of reviewing current practice, and by inducing changes which lead to better patient outcomes and satisfaction. Suggested criteria for undertaking an audit include: the issue addressed should be a common, significant or serious problem; any changes following audit should be likely to benefit patients and to lead to greater effectiveness; the issue is relevant to professional practice or development; there is realistic potential for improvement; and the end result is likely to justify the investment of the time and effort involved (Clinical Resource and Audit Group 1994). Investigators of audit have reported that most audit has focused on process, rather than structure or outcomes (e.g. Packwood 1995).
Medical audit, clinical audit and quality assurance Audit consists of reviewing and monitoring current practice, and evaluation (comparison of performance) against agreed predefined standards (Standing Committee on Postgraduate Medical Education 1989). It is divided into medical and clinical audit, and is related to quality assurance. These have become commonplace in the British National Health Services (NHS) and are now built into the structure of provider units (e.g. hospitals and, increasingly, general practice). These three concepts have been clarified by Higginson (1994) as follows. • Medical audit is the systematic critical analysis of the quality of medical care, including a review of diagnosis, and the procedures used for diagnosis, clinical decisions about the treatment, use of resources and patient outcome (Secretaries of State for Health, Wales, Northern Ireland and Scotland 1989a). Examples of medical audit include analyses of avoidable deaths, and the assessment of medical decision-making, resources and procedures used in relation to patient outcome. • Clinical audit is conducted by doctors (medical audit) and other health care professionals (e.g. nurses, physiotherapists, occupational and speech therapists), and is the systematic critical analysis of the quality of clinical care. It includes collecting information to review diagnosis and the procedures used for diagnosis, clinical decisions about the treatment, use of resources and patient outcome (Secretaries of State for Health, Wales, Northern Ireland and Scotland 1989a). • Quality assurance is a clinical and management approach which involves the systematic monitoring and evaluation of predefined and agreed levels of service provision. Quality assurance is the definition
Evaluating health services: multidisciplinary collaboration
9
of standards, the measurement of their achievement and the mechanisms employed to improve performance (Shaw 1989). Medical and clinical audit is usually one part of a quality assurance programme. Quality assurance usually implies a planned programme involving the whole of a particular health service. Audit can be carried out internally by organisations, members of a discipline (peer review), individuals who systematically review their work or that of their teams, or external bodies (e.g. purchasers for contract monitoring, or professional bodies). Certain criteria need to be met for conducting successful audit: for example, effective clinical leadership; strategic direction (vision, strategy, objectives and planning); audit staff and support (e.g. high calibre, right skill mix, reward, staff development); basic structures and systems (e.g. business planning); training and education; understanding and involvement (e.g. communication, leadership and so on); organisational environment (e.g. structure, relationships) (Walshe 1995). The process of audit involves multiple methods, such as document searching and analysis (e.g. analysis of complaints files, random or systematic selection of nursing and medical records for routine reviews), analysis of routine data, clinical case reviews and presentations in team meetings (see Hopkins 1990 for review). It can also include the collection of information by focus groups of patients or by questionnaire, e.g. patient satisfaction, patient assessed outcome (see Riordan and Mockler 1996 for an example of this in an audit of a psychogeriatric assessment unit). While quantitative research methodology is most appropriate for audit, much can also be gained by supplementing this with qualitative methods such as observation (e.g. visits to wards and clinics to assess quality by observation). The design of audits should also aim to be scientifically and methodologically rigorous (Russell and Wilson 1992; Department of Health 1993b).
Evaluation Evaluation is the use of the scientific method, and the rigorous and systematic collection of research data to assess the effectiveness of organisations, services and programmes (e.g. health service interventions) in achieving predefined objectives (Shaw 1980). Evaluation is central to health services research and audit. Evaluation is more than audit because it aims to record not only what changes occur, but also what led to those changes. Evaluation can be divided into two types: formative and summative. Formative evaluation involves the collection of data while the organisation or programme is active, with the aim of developing or improving it. Summative evaluation involves collecting data about the active (or terminated) organisation or programme with the aim of deciding whether it should be continued or repeated (a health promotion activity or screening programme) (Kemm and Booth 1992).
10
Investigating health services and health: the scope of research
Structure, process and outcome The evaluation of health services is usually based on the collection of data about the structure, inputs, process, outputs and outcomes of the service (Donabedian 1980). Structure refers to the organisational framework for the activities; process refers to the activities themselves; and outcome refers to the impact (effectiveness) of the activities of interest (e.g. health services and interventions) in relation to individuals (e.g. patients) and communities. Health outcome relates to the impact of the service on the patient (effectiveness). The structure and process of services can influence their effectiveness. These concepts have been clearly described in relation to the evaluation of health services by St Leger et al. (1992). Thus, it is often necessary to measure structure and process in order to interpret the outcome of the care. For example, the collection of qualitative and quantitative descriptive data about process and structure is essential if the investigator wishes to address the question of whether – and how – the outcome was caused by the activity itself, and/or by variations in the structure, or the way it was organised or delivered (process). These data can enhance the influence of the research results. These concepts, and their operationalisation, are described below.
Structure and inputs The structure of an organisation refers to the buildings, equipment, staff, beds and so on needed to meet defined standards. The assessment of quality will be in relation to their numbers, type and suitability. It is represented in economic terms by its fixed costs (see Chapter 4). The operationalisation of this concept requires measurement of the raw materials forming the inputs. These can be operationalised in relation to the distribution of staff, their mix in relation to level of training, grade and skill, availability, siting and type of buildings (e.g. hospitals, clinics and types), facilities and equipment, numbers and types of services, consumables (e.g. medication) used and other types of capital and financial resources. Data on structure and inputs can be obtained by questionnaire and document analysis. The study design might be a descriptive survey or the data might be collected within an experimental design comparing organisations in relation to outcome.
Process and outputs The process refers to how the service is organised, delivered and used. It is assessed in medical audit in relation to deviation from predefined and agreed standards. It includes accessibility (e.g. proximity to public transport,
Evaluating health services: multidisciplinary collaboration
11
waiting lists), the way in which personnel and activities interact, and interaction between personnel and patients. In other words, it is the documentation and analysis of dynamic events and interactions. Data on processes are essential for the evaluation of whether scarce health service resources are used efficiently. The types of data to be collected include outputs (e.g. the activities that occur through the use of the resources in the system). These can be operationalised in relation to rates of hospital discharge, number and type of supplies given (e.g. medication, equipment), the number of patient–professional contacts and their type, the number of home visits, average lengths of hospital stay, length of consultation, medical and surgical intervention rates, waiting lists and waiting times. Donabedian (1980) included accessibility as a process indicator (e.g. levels of use by different population groups, adequacy and appropriateness of services provided). The analysis of process also involves the collection of data about the quality of the relationship, and communications, between professional and professional, and professional and patient (e.g. timely provision of information to general practitioners (GPs) about their patients’ treatment/discharge, provision of information to patients), plans or procedures followed and documentation. Some of the information can be extracted from records and, increasingly, computer databases, combined with checks with patients and professionals in relation to its accuracy and completeness. Alternatively, it can be collected by asking patients to provide the information. Appropriate methods include questionnaire surveys and document analyses.
Appropriateness and inappropriateness
Appropriateness Appropriateness is relevant to outcome. Appropriateness of health care interventions has been variously defined. Investigators at Rand in the USA defined it in terms of whether the expected health benefit of the procedure exceeds its expected negative health consequences by a sufficiently wide margin to justify performing the procedure, excluding considerations of financial cost (Chassin 1989). The view of the British NHS Executive is that appropriateness of care refers to the selection, on the basis of the evidence, of interventions of demonstrable effectiveness that are most likely to lead to the outcome desired by the individual patient (Hopkins 1993). The definition used in Britain often includes consideration of resources (Chantler et al. 1989; Maxwell 1989), and of the individuality of the patient. There is no consensus internationally on a definition of appropriateness. The emphasis in health services research is on the measurement of the appropriateness of, as well as the effectiveness of, interventions in the
12
Investigating health services and health: the scope of research broadest sense. Policy-makers, purchasers and providers of health services aim, in theory, to identify the most appropriate treatments and services to deliver and purchase (outcome assessment) and the level of need in the population for the interventions, and to monitor their provision and mode of delivery (measurement of processes and structure). Patients themselves also want to know whether the treatment will work and whether they will recover – as well as where to go for their treatment. The difficulties at policy level stem from the relative dearth of research data on appropriateness and effectiveness. Appropriateness is not limited to interventions, but also applies to organisational factors. For example, there is an increasing literature on the appropriateness of length of hospital inpatient stays (Houghton et al. 1997).
Inappropriateness All medical treatments aim to save or prolong life, to relieve symptoms, to provide care and/or to improve health-related quality of life. However, the assessment of health outcomes and appropriateness of treatments has been given impetus by the increasing evidence about high rates of inappropriate treatments. For example, in the USA, relatively high levels of inappropriateness rates have been found in relation to surgical interventions for coronary heart disease (Chassin et al. 1987; Winslow et al. 1988; Smith 1990). High levels of inappropriate care and wide variations in practice (e.g. intervention rates) have been documented in the UK in relation to various procedures (Brook et al. 1988; Anderson and Mooney 1990; Coulter et al. 1993). While Brook (1994) argued that there is too much literature on medical practice for doctors to assimilate routinely, it is also the case that there is insufficient research evidence on the appropriateness of many medical interventions. Methods for developing consensus on appropriateness criteria are described in Chapter 17.
Outcome Health service outcomes are the effects of health services on patients’ health (e.g. their health gain) as well as patients’ evaluations of their health care. Reliable and valid information on outcomes of health services is essential for audit, as well as for purchasing policies. Donabedian (1980) defined health outcome as a change as a result of antecedent health care. This is a narrow definition, although widely used, and excludes the maintenance of patients in a stable condition, which can also be a valid aim of treatment. It also excludes many health promotion and prevention activities. Outcome refers to the effectiveness of the activities in relation to the achievement of the intended goal. Purchasing debates in health care have focused on health care costs in relation to broader ‘health
Evaluating health services: multidisciplinary collaboration
13
gains’ or ‘benefits’ from the treatments and interventions that are being contracted for. There is similar debate about the definition and measurement of outcome in relation to social care and input from social services. Outcome is more complex in the context of social care, and also in the case of long-term health care, than it is with specific, time-limited treatments and interventions. In relation to social care, and long-term health care, the objective is to measure what difference this made to the recipient’s life in the broadest sense (Qureshi et al. 1994). Health outcome measurement has traditionally focused on survival periods, toxicity, biochemical indicators and symptom rates, relapses, various indicators of physical and psychological morbidity, and easily measured social variables (e.g. days off work or school, number of bed days, hospital readmission rates, other indicators of health service use). Lohr (1988) defined outcome in relation to death, disease, disability, discomfort and dissatisfaction (‘the five Ds’), and argued that measurement instruments should focus on each of these concepts. However, the trend now is to incorporate positive indicators (e.g. degrees of wellbeing, ability, comfort, satisfaction), rather than to focus entirely on negative aspects of outcome.
Broader measures of outcome In health and social services research, more positive criteria of quality of life are increasingly being incorporated into the broader assessment of outcome. Treatment and care need to be evaluated in terms of whether they are more likely to lead to an outcome of a life worth living in social, psychological and physical terms. Health and ill health is a consequence of the interaction of social, psychological and biological events (sometimes called the biopsychosocial model of ill health). Thus each of these elements requires measurement in relation to: patients’ perceived health status and health-related quality of life (physical, psychological and social); reduced symptoms and toxicity; and patients’ (and carers’ where appropriate) satisfaction with the treatment and outcome (see Chapter 2). Thus, the assessment of outcome needs to incorporate both the medical model and the patient’s perspective.
Health and health-related quality of life Health status and health-related quality of life are two distinct conceptual terms which are often used interchangeably. Health status is one domain of health-related quality of life. The definition of health status traditionally focused on physical morbidity and mental health, and was negative in its operationalisation. Because the current usage of health status implies a multifaceted concept, it overlaps with the broader concept
14
Investigating health services and health: the scope of research of health-related quality of life. Both can encompass physical health (e.g. fitness, symptoms, signs of disease and wellness), physical functioning (ability to perform daily activities and physical roles), social functioning and social health (relationships, social support and activities), psychological well-being (depression, anxiety), emotional well-being (life satisfaction, morale, control, coping and adjustment) and perceptions. It is increasingly accepted that an instrument which encompasses the above domains is measuring health-related quality of life, rather than a narrower aspect of physical or mental health status (see WHOQOL Group 1993; Bowling 1995b, 1997). In addition, the concepts of perceived health status, quality of life and health-related quality of life can be complex to analyse as they might be mediated by several interrelated variables, including self-related constructs (e.g. self-efficacy, self-esteem, perceived control over life) and subjective evaluations could be influenced, in theory, by cognitive mechanisms (e.g. expectations of life, level of optimism or pessimism, social and cultural values, aspirations, standards for social comparisons of one’s circumstances in life). Few investigators have taken these variables, and their interplay, into account, although associations between expectations of treatment and patient outcome, and between level of optimism and patients’ coping strategies, have been reported (Higginson 2000; Koller et al. 2000). Health-related quality of life as an outcome measure broadens outcome towards considering the impact of the condition and its treatment on the person’s emotional, physical and social functioning and lifestyle. It addresses the question of whether the treatment leads to a life worth living, and it provides a more subjective, patient-led baseline against which the effects of interventions can be evaluated. It can only do this, however, if the measurement scale reflecting its components is valid, reliable, precise, specific, responsive to change and sensitive. A universal questionnaire to elicit the relevant information for a number of conditions would need to be of enormous length. Disease-specific quality of life scales are needed, not simply for greater brevity, but to ensure sensitivity to sometimes small, but clinically significant, changes in health status and levels of disease severity. A quality of life measure used in research on health and health care should be able to inform the investigator of the effects of the condition or treatment on the patient’s daily, as well as long-term, life. It should also be capable of providing information on whether, and to what extent, any gains in survival time among patients with life-threatening conditions are at the expense of reductions in quality of life during the period of the treatment and in the long term. A disease-specific, or condition-specific instrument will have a narrower focus generally, but contain more details of relevance to the area of interest. If the investigator is interested in a single disease or condition, then a disease-specific indicator is appropriate, although if the respondent has multiple health problems it may be worth combining it with a generic measure. If the research topic covers more than one condition, or general health, then generic measures might be more appropriate. It is not possible in this short space to recommend specific
Evaluating health services: multidisciplinary collaboration
15
measures; generic and disease-specific measures have been reviewed by the author elsewhere (Bowling 1995b, 1997). The theoretical influences which shaped the development of health status and health-related quality of life scales are described briefly in Chapter 2.
Summary of main points
• Research is the systematic and rigorous process of enquiry that aims to describe processes and develop explanatory concepts and theories, in order to contribute to a scientific body of knowledge. • Health services research aims to produce reliable and valid research data on which to base appropriate, effective, cost-effective, efficient and acceptable health services in the broadest sense. • The quality of care relates to its effectiveness at improving patients’ health status and how well it meets predefined and agreed standards about how the care should be provided. • Audit is directed at the maintenance and achievement of quality in health care. It consists of review and monitoring of current practice, and evaluation against standards. • Medical audit is the systematic critical analysis of the quality of medical care; clinical audit is the systematic critical analysis of the quality of clinical care by all health care professionals. • Quality assurance is a clinical and management approach which is the systematic monitoring and evaluation of predefined and agreed levels of service provision. • Evaluation is the use of the scientific method, and the rigorous and systematic collection of research data to assess the effectiveness of organisations, services and programmes (e.g. health service interventions) in achieving predefined objectives. • Evaluation is more than audit because it aims to record not only what changes occur, but also what led to those changes. • The evaluation of health services is usually based on collecting data about the structure, process and outcomes of services, as well as the appropriateness of the services. • Outcome should usually include measurement of the impact of the condition and the service (i.e. health care intervention) on the broader health-related quality of life of the patient.
Key questions
Define research. Distinguish between health research, health systems research and health services research.
16
Investigating health services and health: the scope of research What are the key components of health services research? Distinguish between evaluation and audit. What is the difference between audit and quality assurance? Distinguish between the structure, process and outcome of health services. What are health service inputs and outputs? What are the main domains of health-related quality of life which should be included in the measurement of health outcomes?
Key terms
appropriateness audit clinical audit disease-specific quality of life evaluation health-related quality of life health research health services research health status
inputs medical audit outcome outputs process quality of life quality assurance structure
Recommended reading Black, N., Brazier, J., Fitzpatrick, R. and Reeve, B. (eds) (1998) Health Services Research Methods. A Guide to Best Practice. London: BMJ Publishing. Bowling, A. (1995) Measuring Disease. A Review of Disease-specific Quality of Life Measurement Scales. Buckingham: Open University Press. Donabedian, A. (1980) Explorations in Quality Assessment and Monitoring. Vol. 1. The Definition of Quality and Approaches to its Assessment. Ann Arbor, MI: Health Administration Press. Higginson, I. (1994) Quality of care and evaluating services. International Review of Psychiatry, 6, 5–14. Hunter, D.J. and Long, A.F. (1993) Health research. In W. Sykes, M. Bulmer and M. Schwerzel (eds) Directory of Social Research Organizations in the UK. London: Mansell. Jenkinson, C. (ed.) (1997) Assessment and Evaluation of Health and Medical Care. Buckingham: Open University Press. Long, A. (1994) Assessing health and social outcomes. In J. Popay and G. Williams (eds) Researching the People’s Health. London: Routledge. St Leger, A.S., Schnieden, H. and Wadsworth-Bell, J.P. (1992) Evaluating Health Services’ Effectiveness. Buckingham: Open University Press.