Mix of methods is needed to identify adverse events in general practice: A prospective observational study
BMC Family Practice volume 9, Article number: 35 (2008)
The validity and usefulness of incident reporting and other methods for identifying adverse events remains unclear. This study aimed to compare five methods in general practice.
In a prospective observational study, with five general practitioners, five methods were applied and compared. The five methods were physician reported adverse events, pharmacist reported adverse events, patients' experiences of adverse events, assessment of a random sample of medical records, and assessment of all deceased patients.
A total of 68 events were identified using these methods. The patient survey accounted for the highest number of events and the pharmacist reports for the lowest number. No overlap between the methods was detected. The patient survey accounted for the highest number of events and the pharmacist reports for the lowest number.
A mix of methods is needed to identify adverse events in general practice.
Patient safety is important in primary care, as most patients and most of their health problems are treated in this setting . Adverse events in primary care occur between five and 80 times per 100 000 consultations . General practitioners (GPs) were positive about reporting adverse events , but the validity and usefulness of incident reporting systems remains unclear [, page 3]. A range of methods is available for identifying adverse events, such as review of medical records and case reviews of deceased patients [5–7]. The aim of the presented study was to compare five different methods for identifying adverse events in general practice with respect to the number and type of identified events, the patient subgroups affected, and the agreement of events across the methods.
Study design and setting
A prospective observational study was performed focused on five GPs in two practices in a period of five months (May to October 2006). A total of approximately 8250 patients were registered with the two practices. The ethical committee of the Radboud University Nijmegen Medical Centre approved the study.
We defined an adverse event as an unintentional event with actual or potential harm to the patients' health status. This broad definition is consistent with the International Patient Safety Event Classification of the World Health Organization: "a process or act of omission or commission that resulted in hazardous health care conditions and/or unintended harm to the patient" . The five methods were: GPs reported adverse events, pharmacist reported adverse events, patients' experiences of adverse events, assessment of a random sample of medical records, and assessment of all deceased patients.
GP reported adverse events
The GPs recorded all events during a five-month period. On the basis of an existing international taxonomy for errors in general practice  a simplified computerised registration form was made. The GPs registered event date, birth date of patient, gender, event category (practice administration (archive; medical record; appointment; other), diagnostic (wrong diagnosis; delayed diagnosis; missed diagnosis; other), therapeutic (wrong, incomplete; delayed; none, though it should be; other), communication (with patients; with caregivers; other)), and additional remarks and/or context.
Pharmacist reported adverse events
The pharmacist recorded events from her point of view in the same period. An adjusted form was developed for this purpose. Event date, birth date of patient, gender, practice, event category (prescribing error (wrong prescription, wrong administration; wrong dose; other), adverse reaction (adverse reaction; allergic reaction; overdose; interaction; contra-indication; other), dispensing error (too late; wrong medicine; wrong dose; other)), and additional remarks or context were recorded.
Patient reported adverse events
In the waiting room of the two practices samples of 50 patients, consecutively visiting the practice, were invited to complete a questionnaire on experienced problems with safety of their health care in the previous six months. A drop box was used to collect the completed questionnaires. Questions were derived from items of the Medical Harvard Study,  and from questions of two survey studies. [11, 12] Questions had to be answered with yes or no; and asked whether something had gone wrong in the care they had received from their GP during the past half year (see table 1 for the complete questionnaire). The questionnaire guaranteed anonymity of participating patients.
Assessment of medical records
Thirty medical records per GP (in total 150 medical records) were randomly selected (using a list of random numbers) from patients who had visited their GP in the observation period. Anonymous medical records, containing the information from this period were printed out. Two clinical researchers examined the information independently [RWo, RWe]. They scrutinised the records for indications of events and, when found, categorised the event (errors in office administration, diagnosis, treatment or communication with their subcategories); and added demographic data of the patient (birth date, gender). Subsequently the GPs discussed their findings and reached consensus.
Assessment of all deceased patients
Both practices had a registration of deceased patients from which the medical records were retrieved of all patients who had died in the period May-October 2006. One GP examined the medical records of these patients for events. The same registration form and analysis procedure as for the audits of medical records was used.
Numbers of events per category per method and overall were added up and percentages were calculated. Age range, mean age, age category (<50 and > = 50 years old) and percentage of women were calculated per method and overall.
A qualitative analysis of events will be published elsewhere. 
Table 2 presents the events reported by the five methods. A total of 68 events were identified using these methods. Each of the methods provided events that were not found with other methods; no overlap between the methods was detected. Mostly women were affected, in almost two-third of the registered events. Patients over 50 years and patients under 50 year were almost equally affected, but the age group below 18 was rarely involved. An exception to this general trend was the subgroup of events based patient registration, which involved mostly patients under 50 years of age.
GP reported events
GPs registered 20 events in five months; in these months there were 4095 patients who had visited the practice, resulting in almost 5 events per 1000 patients visiting the practice. Most of these events concerned women aged 50 years or older. GPs registered little events that concerned treatment.
Pharmacist reported events
The pharmacists registered six events per 16320 prescriptions (including repeated prescriptions), all related to female patients. Of these there were five prescribing errors and one known allergic reaction.
Patient reported events
Twenty-seven patients answered positive to one or more items of the patient questionnaire. In total, 78 positive answers were given, it is unclear whether these answers referred to one or more events. Patients answering positive to questions were slightly younger compared to all responders (43,8 vs. 45,5 years old). By far most often (16/78) mentioned was a breach of confidentially of their medical record during treatment by their GP. Next, a lack of respect by GP or practice assistant was reported (8/78), followed by a delay in diagnosis (5/78), inappropriate drug prescribed (5/78) or inappropriate advice given (5/78) and a wrong appointment at the practice (5/78).
Assessment of medical records
There had been 4,095 patients consulting the practices during the study period, from which 150 medical records were randomly selected. Analysis resulted in the finding of 11 events, all errors of treatment and communication.
Assessment of deceased patients
During the study period 28 patients had died. One medical record was not available for examination, as this patient had not given permission. This method generated between one or two events per ten deceased patients.
All five methods proved to identify a number of adverse events. The patient survey accounted for the highest number of events and the pharmacist reports for the lowest number. All methods resulted in a variety of events, except for the pharmacist reports, which only referred to pharmaceutical treatment. The identified events referred to adult male and female patients of all ages, but events on children were very seldom reported. Events based on patient registration mostly involved individuals aged 50 years or younger. There was no overlap between the methods regarding the identified events.
A systematic review of methods to identify adverse events in health care concluded that "the available methods have widely differing purposes, strengths and weaknesses and must be considered as complementing each other by providing different levels of qualitative and quantitave information" [, page 4]. Incident reporting systems have the advantage of being not as time-consuming as formal studies, but they are likely to underestimate the number of adverse events (numerator) while the number of opportunities for incidents (denominator) remains unknown. A recent study in hospitals also concluded that different methods should be used in order to provide an adequate assessment of clinical adverse events .
Our study in general practice suggested that GPs did not report all adverse events, as compared to other sources. It was difficult to assess the validity of the GP reported events because of the lack of overlap with other methods. Event reporting by GPs is probably important for raising awareness and a safety culture, but it is unlikely to be a comprehensive method for identifying adverse events. None of the GPs had a feeling of suspicion with registration. They did not think the registered events would be used against them. They all felt it was difficult to remember to register events due to daily routine working procedures and time pressure.
Almost three in ten patients reported on health safety issues, and patients as a group reported a substantial number of adverse events. It was unclear whether the checked items of the patient questionnaire referred to one or more health safety issues; we assumed these were from 27 events (thus one event per patient), in order not to overestimate number of events. In this study almost 60% of patient reported events concerned psychological harm or harm in trust or confidence. Another study showed that less than a third of reported harm was physical . Patients registering events may reveal blind spots in care provided by the GP. The questionnaire seemed quite easy to complete and its analysis asked for a relatively low time investment. However, the validity and usefulness of patient reported incidents needs further research [16, 17].
Inaccuracies of medical record are well documented . This method generated different types of adverse events, although the number in diagnostic events was rather low. It took a large time-investment to go through the medical records.
The reporting of adverse events by a pharmacist generated relatively few events. A study showed that community pharmacists correct 1% of the prescriptions , which obviously influences the number of remaining events. The low numbers may explained by the computerised systems, which including monitoring for potential contra-indications and interaction of drugs.
The audit of medical records of deceased patients was feasible, but the number of identified adverse events was low, and therefore its yield was restricted.
The limitations of this study should be recognized. The number and type of adverse events seemed to depend on the individual who registered. One could imagine asking other (para)medical caregivers (physiotherapists, midwifes, etc) to register as well. Also, in the communication with hospitals and referrals to specialists, things may go wrong. Specialists could therefore be a registration source of adverse events for GPs and this probably goes the other way around as well. Future studies should take this into account. Registration of events stays difficult as one kept in doubt whether an event is an adverse event or something that is generally accepted as inherent to GP care. This might have negatively influenced the number of registered events. Patients may also experience not every incident in their care as an adverse event. Whether this was the case may be influenced by the response of the doctor towards the event .
Patient safety has attracted "a level of public interest that the rest of the quality-improvement field in health care has failed to excite" . This may be explained by a fascination for the accidental death, despite the prolonged uncertainty about the "true" estimates of mortality resulting from medical error . In our study, none of the identified adverse events was associated with mortality.
The public interest for patient safety has led to a number of potentially valuable initiatives to identify and prevent adverse events. None of the methods to identify adverse events proved to be superior. We suggest that more research is needed on the various approaches in general practice before their wide scale implementation can be recommended.
Green LA, Fryer GE, Yawn BP, Lanier D, Dovey SM: The ecology of medical care revisited. N Engl J Med. 2001, 344: 2021-2025. 10.1056/NEJM200106283442611.
Sandars J, Esmail A: The frequency and nature of medical error in primary care: understanding the diversity across studies. Fam Pract. 2003, 20: 231-236. 10.1093/fampra/cmg301.
Mikkelsen TH, Sokolowski I, Olesen F: General practitioners' attitudes toward reporting and learning from adverse events: results from a survey. Scand J Prim Health Care. 2006, 24: 27-32. 10.1080/02813430500508330.
WHO: Strengths and weaknesses of available methods for assessing the nature and scale of harm caused by the health system: a systematic literature review. Geneva. 2003
Britt H, Miller GC, Steven ID, Howarth GC, Nicholson PA, Bhasale AL, Norton KJ: Collecting data on adverse events: a method for monitoring incidents in general practice. Fam Pract. 1997, 14: 101-6. 10.1093/fampra/14.2.101.
Bhasale A: The wrong diagnosis: identifying causes of potentially adverse events in general practice using incident monitoring. Fam Pract. 1998, 15: 308-18. 10.1093/fampra/15.4.308.
Holden J, O'Donnell S, Brindley J, Miles L: Analysis of 1263 deaths in four general practices. Br J Gen Pract. 1998, 48: 1409-12.
World Health Organization, World Alliance for Patient Safety: The Conceptual Framework of an International Patient Safety Event Classsification. 2008, Copenhagen: WHO report, (Executive Summary)
Makeham MA, Dovey SM, County M, Kidd MR: An international taxonomy for errors in general practice: a pilot study. MJA. 2002, 177: 68-72.
Leape LL, Brennan TA, Laird N, Lawthers AG, Localio AR, Barnes BA, Hebert L, Newhouse JP, Weiler PC, Hiatt H: The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med. 1991, 324: 377-384.
Blendon RJ, DesRoches CM, Brodie M, Benson JM, Rosen AB, Schneider E, Altman DE, Zapert K, Herrmann MJ, Steffenson AE: Views of practicing physicians and the public on medical errors. N Engl J Med. 2002, 347: 1933-40. 10.1056/NEJMsa022151.
Agoritsas T, Bovier PA, Perneger TV: Patient reports of undesirable events during hospitalization. J Gen Intern Med. 2005, 20: 922-28. 10.1111/j.1525-1497.2005.0225.x.
Wetzels R, Wolters R, Van Weel C, Wensing M: Harm caused by adverse events in primary care: a clinical observational study. J Clin Eval Pract.
Olsen S, Neale G, Schwab K, Psaila B, Patel T, Chapman EJ, Vincent C: Hospital staff should use more than one method to detect adverse events and potential adverse events: incident reporting, pharmacist surveillance and local real-time record review may all have a place. Qual Saf Health Care. 2007, 16: 40-4. 10.1136/qshc.2005.017616.
Kuzel AJ, Woolf SH, Gilchrist VJ, Engel JD, LaVeist TA, Vincent C, Frankel RM: Patient reports of preventable problems and harms in primary health care. Fam Med. 2004, 2: 333-40. 10.1370/afm.220.
Davis RE, Jacklin R, Sevdalis N, Vincent CA: Patient involvement in patient safety: what factors influence patient participation and engagement?. Health Expec. 2007, 10: 259-267. 10.1111/j.1369-7625.2007.00450.x.
Buetow S, Elwyn G: Patient safety and patient error. Lancet. 2007, 369: 158-159. 10.1016/S0140-6736(07)60077-4.
Stange KC, Zyzanski SJ, Smith TF, Kelly R, Langa DM, Flocke SA, Jaén CR: How valid are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patients visits. Med Care. 1998, 36: 851-67. 10.1097/00005650-199806000-00009.
Elder NC, Meulen von der MB, Cassedy A: The identification of medical errors by family physicians during outpatient visits. Ann Fam Med. 2004, 2: 125-129. 10.1370/afm.16.
Van Vorst RF, Araya-Guerra R, Felzien M, Fernald D, Elder N, Duclos C, Westfall JM: Rural community members' perceptions of harm from medical mistakes: a High Plains Research Network (HPRN) Study. J Am Board Fam Med. 2007, 20: 135-43. 10.3122/jabfm.2007.02.060147.
Buurma H, de Smet PA, Hoff van den OP, Egberts AC: Nature, frequency and determinants of prescription modifications in Dutch community pharmacies. Br J Clin Pharmacol. 2001, 52: 85-91. 10.1046/j.0306-5251.2001.01406.x.
Brennan TA, Gawande A, Thomas E, Studdert D: Accidental deaths, saved lives, and improved quality. N Engl J Med. 2005, 353: 1405-1409. 10.1056/NEJMsb051157.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2296/9/35/prepub
The participating general practitioners, patients, pharmacist are gratefully acknowledged. The Netherlands Organisation for Health Research and Development (ZonMW) provided the financial resources for this study.
The authors declare that they have no competing interests.
RWe performed the study according to protocol and wrote the manuscript. RWe and RWo collected and analysed data. MW and CvW initiated and supervised the study, and developed the study protocol. All authors read and approved the final version of the manuscript.
About this article
Cite this article
Wetzels, R., Wolters, R., van Weel, C. et al. Mix of methods is needed to identify adverse events in general practice: A prospective observational study. BMC Fam Pract 9, 35 (2008). https://doi.org/10.1186/1471-2296-9-35