Specialist wait time reporting using family physicians’ electronic medical record data: a mixed method study of feasibility and clinical utility

Background More than 50% of Canadian adult patients wait longer than four weeks to see a specialist after referral from primary care. Access to accurate wait time information may help primary care physicians choose the timeliest specialist to address a patient’s specific needs. We conducted a mixed-methods study to assess if primary to specialist care wait times can be extracted from electronic medical records (EMR), analyzed the wait time information, and used focus groups and interviews to assess the potential clinical utility of the wait time information. Methods Two family practices were recruited to examine primary care physician to specialist wait times between January 2016 and December 2017, using EMR data. The primary outcome was the median wait time from physician referral to specialist appointment for each specialty service. Secondary outcomes included the physician and patient characteristics associated with wait times as well as qualitative analyses of physician interviews about the resulting wait time reports. Results Wait time data can be extracted from the primary care EMR and converted to a report format for family physicians and specialists to review. After data cleaning, there were 7141 referrals included from 4967 unique patients. The 5 most common specialties referred to were Dermatology, Gastroenterology, Ear Nose and Throat, Obstetrics and Gynecology and Urology. Half of the patients were seen by a specialist within 42 days, 75% seen within 80 days and all patients within 760 days. There were significant differences in wait times by specialty, for younger patients, and those with urgently labelled medical situations. Overall, wait time reports were perceived by clinicians to be important since they could help family physicians decide how to triage referrals and might lead to system improvements. Conclusions Wait time information from primary to specialist care can aid in decision-making around specialist referrals, identify bottlenecks, and help with system planning. This mixed method study is a starting point to review the importance of providing wait time data for both family physicians, specialists and local health systems. Future work can be directed towards developing wait time reporting functionality and evaluating if wait time information will help increase system efficiency and/or improve provider and patient satisfaction. Supplementary Information The online version contains supplementary material available at 10.1186/s12875-022-01679-x.


Introduction
Canada has the longest wait times to see a specialist physician according to a recent Commonwealth Fund survey of eleven countries [1]. More than 50% of adult patients wait longer than four weeks to see a specialist, compared to the international average of 36% [1]. Median wait times to see specialists in Ontario range from 39 to 76 days for medical specialists and 33 to 66 days for surgical specialists [2]. Wait times from primary to specialty care in Canada have been increasing and are 155% longer than in 1993 [3]. Prolonged waiting for specialists may be associated with adverse consequences such as increased pain and suffering, inferior medical outcomes, and economic costs from lost wages and productivity [4,5].
In 2007, Willcox et al. conducted a cross-national comparison of strategies used by countries to measure and reduce wait times [6]. Five countries used supply-side strategies, such as targeting funding toward increased hospital capacity and staff. Some countries used more complex initiatives that addressed demand side techniques, such as using explicit criteria to prioritize access to surgery. The authors recommended that policymakers consider extending the measurement of waiting times to include the point of referral to treatment in order to reflect patients' actual experience. To date, there are few benchmarks in place targeting wait times from primary to specialty care [5,7]. In addition, there are limited interventions found in the literature that have shown impact on reducing wait times from primary care to specialty care. Jaakkimainen et al. published a study that outlines patient and provider characteristics of wait times from primary to specialty care [2]. They found that patient factors and most physician factors do not seem to be consistently associated with wait times, except for family physician practice location and practice size.
In Canada, most referrals are directed to specific specialist physicians by primary care providers. In some cases, referrals are made to specialist clinics and the specialist is then selected for the referral by the specialty clinic. Specialist visits are covered by Canada's universal health insurance plan, and primary care providers can refer patients to any specialist of their choosing. Primary care providers typically select the specialist, and fax or email a referral letter to the specialist's office using a paper-based system or an electronic health record of their choosing. Historically, there has been little information available to primary care physicians about which specialists have long waits or are available in different regions. Physicians rely on an ad hoc system based on personal experience, word of mouth, electronic medical record (EMR) or internet searches to identify specialists. At times, physicians may refer a patient and find out months later the referral was never received or additional information is needed [8]. Neimanis et al. analyzed 770 referrals and found that 36.4% of referrals received no response within a 5 to 7 week period [8]. Access to accurate wait time information may help family physicians choose the most suitable specialist to address a patient's specific needs.

Objective
This mixed methods study assessed the feasibility of creating primary to specialist care wait reports from primary care electronic medical records (EMRs). The primary outcome was the median wait time from physician referral to specialist appointment for each specialty service. The clinical utility of the resulting wait time data was evaluated through focus groups with family physicians and specialist physicians [9].

Study setting
A convenience sample of two primary care clinics from the University of Toronto Practice-based Research Network (UTOPIAN) [10] were recruited for this study: an academic family health team in downtown Toronto, Ontario (caring for approximately 12,000 patients of all ages), and a community primary care clinic in Vaughan, Ontario (caring for approximately 10,000 patients of all ages). Both are affiliated with Sinai Health and utilized the same EMR (Nightingale), at the time of the study. We included both sites to study community and downtown academic family practice perspectives and to include data from diverse referral networks. Ethics approval was obtained from the Sinai Health Research Ethics Board.

Data collection
Patients from the two study sites were included if they had one or more referral records between January 1 st , 2016 and December 31 st , 2017, and if they had at least 2 visits in 3 years prior to December 31 st , 2017. Only patients who were not deceased by the end of the study period were included. Age was calculated at January 1 st 2016. The unit of analysis is a "visit", where multiple visits can be nested for a patient. For example, one patient might have referrals to both cardiology and endocrinology, possibly over different visits during the study window. The data set was created by entering the back end Keywords: Wait time, Specialist referrals, Primary care of the electronic health record and extracting fields from patient referral. Variables in the final dataset included: family practice location, age, sex, urgency of visit coded to 4 categories (as per availability, within 1 month, within 1 week and urgent), specialist name, specialty, specialist department or clinic, specialist address, referral date, referral status, patient postal code and appointment date. Wait times were defined as the number of days between referral date and appointment date. Routine referrals were considered those indicated by the referring physician "as per availability" and "within 1 month". Urgent referrals included those marked "urgent" and "within 1 week". Referral data was further cleaned by manual review by the lead study author (MN) to remove referrals with an unknown site, missing department/speciality or to a non-medical or surgical specialty (for example, referrals to physiotherapy or optometry) and referrals where the wait time was missing. Where referrals to a specific specialty were fewer than 50 patients, specialities were grouped into an "other" category. Wait times were calculated by taking the appointment date and subtracting the number of days from the referral date. To validate the accuracy and completeness of the wait times calculated, a random sample of 150 referrals was selected and a research assistant reviewed the charts to assess the agreement between wait times calculated through EMR data that was extracted, cleaned and coded, and wait times calculated through manual chart review (the gold standard).

Creation of wait time reports
The format of the wait time report was developed with input from the study investigators. A sample report is included in Appendix 1 and has been anonymized for purposes of publication.

Statistical analysis Quantitative analysis
Wait times did not have a normal distribution therefore the analysis included the calculation of median wait times and inter-quartile ranges [2]. Median wait times were calculated and stratified by specialist/department and by practice location. Exploratory, bivariate analyses were conducted examining the median wait time including completeness of wait time data and patient or physician characteristics that may have contributed to wait times such as patient age or sex, patient income level before taxes [11], combined material and social deprivation index [12] or physician age, sex and location of practice. Area-level indices such as income quintiles, and the material and social deprivation index were derived using patients' 6-digit postal codes. Non-parametric Wilcoxon rank-sum tests and Kruskal Wallis tests were used to assess the association between these patient/provider characteristics and median wait-times. We calculated the proportion of referrals that were seen within a benchmark of 18 weeks (a benchmark used in the United Kingdom where 90-95% of referrals are targeted to be booked within 18 weeks) [13]. All statistical analyses were conducted at an alpha of 0.05, using SAS version 9.4.

Qualitative analysis
A qualitative approach informed by grounded theory was used [14]. A review of the literature helped to inform the design of the qualitative aspect of the study [15,16]. Focus groups were used to capture the subjective meaning and utility of the wait time reports that were created. Specialists from the top 7 specialties referred to by family physicians at both practice sites were invited to participate in a focus group. The aim was to have 6-8 participants in the specialist focus group, and in each of the Toronto and Vaughan family physician focus groups. Individual interviews were scheduled with participants who were unable to attend a focus group. Focus groups and interviews were conducted in-person by the qualitative research assistant and the Principal Investigator. At the start of the session, physicians received a paper wait time report to review that included median wait times of specialists (anonymized) by practice address for 7 specialty areas. Specialists received a similar paper report plus a more detailed confidential report with their personal wait times listed for each referral made to them specifically. A semi-structured interview guide was used for family physicians (Appendix 2) and specialists (Appendix 3) that asked about the relevance, clinical utility and acceptability of the wait time reports. Focus groups and interviews were conducted over 2 months from October to November, 2018. Interviews lasted 60 min and were audio-recorded and transcribed verbatim, anonymized and analyzed using techniques informed by grounded theory [14] including coding, interpretations of data patterns and constant comparison method [17].
Research team members performed line by line open coding of interview transcripts [18] using an inductive approach with principles of constant comparison [19] for analyzing data. Co-investigators (M.N. and T.M.) read the same transcripts independently and met with one another to develop a coding manual. The transcripts were subsequently coded line by line by the qualitative researcher (T.M.). Team members met to review and refine codes and any discrepancies were resolved through discussion. Inductive analysis was appropriate for use in this research since there were no previous studies dealing with the phenomenon under investigation [20]. NVivo 11 (NVivo qualitative data analysis Software; QSR International Pty Ltd. Version 11, 2016) was used for data management. Figure 1 summarizes the wait time data extraction and cleaning process from the EMR. The data cleaning process required some manual review of spreadsheets to remove some referrals as noted in the Figure. It took approximately 2-3 h of time to identify and remove missing referrals, and specialties that were not applicable. Validation of wait time calculations from 150 charts demonstrated a wait time concordance rate of 100%. Referrals analyzed included referrals where appointments dates were pending or not recorded in the EMR. The final sample of specialist referrals where wait time information was available included 7141 referrals (4967 unique patients). Table 1 outlines characteristics of the referrals. Wait times differed across age groups. We observed differences in median wait times in those > 65 years (46.0 days) versus those 0-19 years (40.0 days) (p = 0.0006). We also observed differences in median wait times in those > 65 years (46.0 days) versus those 20-44 years (42.0 days) (p = 0.0002). There was no difference in wait times by sex. Wait times for referrals marked as urgent had a shorter wait time compared to routine referrals: 13 days versus 43 days respectively (p < 0.01). There seems to be some cyclicality to wait times based on season referred, with a longer wait in the summer season, however, more robust data using more referrals and practice sites would be needed to examine this trend further. Analysis of income level and material and social deprivation level for patients with an available postal code did not show any meaningful difference in wait times by income quintile before taxes, or by material and social deprivation.

Quantitative analysis
Of the 4967 unique patients, 69% had one referral, 22% had 2 referrals, 6% had 3 referrals and 3% of patients had 4 or more referrals. There were 1357 unique specialist names in the database and 596 unique departments. The list was reduced to 33 unique specialities by grouping similar specialty services/departments ( Table 2). The top 10 specialties consulted were Dermatology, Gastroenterology, Ear Nose and Throat, Obstetrics and Gynecology, Urology, Ophthalmology, The median wait time of all referrals was 42 days, 75% of patients were seen within 80 days and all patients were seen within 760 days. Almost 90% of patients saw the specialist within an 18 week benchmark, and 99.53% of referrals were seen within the year (Fig. 2).

Qualitative analysis
We conducted three focus group sessions, two family physician-centred focus groups and one specialist-centered focus group, along with two one-on-one specialist interviews. In total, 6 specialists and 14 family physicians participated in the focus groups. Socio-demographic surveys were completed by 17 of the 20 (85%) participants ( Table 3). Thirteen of the 17 respondents Table 1 Comparison of wait times by patient and referral characteristics (N = 7141) a FHT Family Health Team, b Denominators may differ between referral characteristics due to missing data, c 25 th Pctl 25 th percentile, d minimum: minimum wait time, e 75 th Pctl 75 th percentile, f maximum maximum wait time, g Routine referrals include referrals indicated by the physician to be "as per availability" and "within 1 month, while urgent referrals include those marked "urgent" and "within 1 week", h P-value based on Wilcoxon rank-sum test for variables with 2 levels, or P-value based on Kruskal-Wallis test if characteristic has > 2 levels  Table 4 summarizes the information obtained from the focus groups and interviews organized by topic area discussed in the semi-structured interviews. General impressions of wait time reports by family physicians were that the data was viewed positively and seen to have value. Family physicians perceived that having wait time information had significant clinical utility when deciding upon which specialist to refer. Some physicians reported that in non-urgent cases, they would refer to their preferred specialist-this was often based on familiarity or a prior good working relationship. However, referring a patient to a specialist with a shorter wait, particularly when patient quality of life was of concern was seen as important by most, for example if the patient was in pain or had elevated anxiety. The following comment exemplifies this:   Wait time information could also provide a means for family physicians to learn of new specialists and manage patient expectations around waiting for the referral. Having easy access to data at the point of care was identified as important to family physicians (versus paper reports or an email with wait time information).
Wait time information was viewed as an important piece of information by both family physicians and specialists for local, regional and provincial planning.
"I think it's amazing. It seems like this would be really useful to be out there both for like a [specialty named]

Quality or cannot be discerned 2. Data provides no contextual information which may account for outliers
Wait time data may not change referral patterns-some PCPs report having "favourites" who they already refer to and accessing wait time data may be There may be unintended consequence of making data available  General impressions of wait time reports by specialists were that seeing the wait time data was interesting. Some specialists had never seen their own wait time data or that of their colleagues. Specialists had variable perceptions about the clinical utility of wait time data. The data could reveal system inefficiencies and allow for redistribution of referrals for some specialties. However, even if aware, specialists acknowledge they may not be able to change their practice or appointment booking processes.

Alienation of some specialists, particularly those with longer wait times
"When you're looking at a process, it's either inefficient or it's a capacity problem. And at least from the [specialty named] side of things, there are some reasonable evidence based on a glance of the distribution that it's an inefficiency problem. So certain people are holding up the line, and certain people are not. And they're not being redistributed that way. But that's the way referrals have been made in Ontario for the last 200 years. And so that's why I think there's a lot more push now to create programs like a rapid assessment clinic. A lot of groups now are sort of first-come, first-serve, depending on individual practitioner wait times. [

…]So I guess the way I look at it is that it just confirms a lot of people's suspicion that within this is probably not a capacity issue right now, it's probably an inefficiency of distribution issue. " (Specialist)
Focus groups and interviews furthermore highlighted that specialists may be unaware of their own actual wait times and levels of triaging referrals by specialist offices can vary widely. Specialists stated that sub-specialist wait times would be another key piece of data for additional analysis to help determine where system bottlenecks may be. A perceived limitation of the wait time report was that the data does not reflect variance with regards to referrals. For example, factors influencing referrals and wait times include patient preference, primary care provider preference, type of problem or diagnosis and appropriateness of the referral.
"It could be perceived that Dr. X with a wait time of 190 days is bad. But it might be that, you know, they are the only person who does that. Or b) they have… they're doing a really good job, and a lot of people want to go to them, and they don't want to go anywhere else, and they want to wait. " Having access to additional, more robust data was of interest to both specialists and family physicians.
There were mixed views about establishing wait time benchmarks. While benchmarks are seen by some specialists as arbitrary because they depend on sub-specialty or diagnosis, they were also seen as important for setting standards and improving healthcare system. Family physicians thought benchmarks may be unrealistic because of geography and location of specialists, and may set up unreasonable expectations. Benchmarks may also inherently place judgement, blame and stress on specialists. It may be better to look at similar healthcare systems doing better and find ways of mirroring or learning from them.

Discussion
We conducted a mixed-methods study to evaluate the feasibility of producing primary to specialist care wait time reports from the EMR and conducted focus groups and interviews to determine the clinical utility of the wait time information. Primary care EMRs house valuable specialist wait time data, which has been an untapped resource in most jurisdictions. Interviews with family physicians and specialists demonstrated the relevance and clinical utility of wait-time related information, especially for family physicians.
The 2 year wait time data extracted from the health records of patients in both clinics closely resembles broader trends in specialist referral patterns in the Province. Our top 10 list of specialist referrals varied slightly from another Ontario study published in 2017 that examined specialist referrals in Ontario [21]. Our list includes rheumatology in the top 10, whereas the other study includes cardiology and plastic surgery. Our top 10 list of specialties was used to form the list of specialists that were invited to the specialist physicians focus groups/interviews. On further evaluation of our wait time data, there was no meaningful difference detected in wait times when comparing low income to high income populations, or by least material and social deprivation to greatest material and social deprivation level. There was no association between patient sex and referral wait times, however there was an association between patient age and wait times, with older patients having shorter wait times compared to younger patients. A possible difference was detected by season referred, with summer months resulting in longer wait times, although more robust data involving more years and referrals would be needed to definitively conclude these trends. These results are similar to information found in a 2014 study examining specialist wait times in Ontario. Although slightly different measures of socioeconomic status were used, specialist wait times in this study did not vary by patient income or social and material deprivation scales [2]. It was reassuring to note that wait times were not impacted by a patient's income level in Canada's single payer system. In terms of the wait times themselves, the overall median wait time in this study was 42 days and the 75 th percentile was 82 days. Our range is 34-75 days for the median wait time compared to the median range 33-76 days in 2014, and 75 th percentile of 62 to 112 days compared to 63 to 231.5 days in the 2014 study [2]. Interestingly, triaging does seem to happen on a system level with a median wait time that is 30 days less for urgent versus routine appointments. With approximately 94% of referrals being marked as "routine", family physicians in these practices do not seem to be "overcalling" the urgency of the nature of the problem when making referrals to specialists, or do not use the option of marking the urgency of a referral.
Specialists and family physicians in this study highlighted that wait time reporting would enable primary care physicians to more comprehensively consider the factors when creating a referral. Such factors may include: (1) having the shortest wait time is most important to reduce morbidity experienced by patients while waiting; (2) that geography is more important to a patient and some may prefer a longer wait time if it meant they could see a specialist closer to where they live; or (3) it may be that seeing a specific specialist is most important, and a patient would be willing to wait longer for their consultation to see a particular specialist. Wait time data may also be useful for physicians newer to practice or those without a preferred specialist in mind to learn about specialists in the region. Additionally, combining referral data across practices/regions could lead to better local, regional and provincial specialist access information for human resource planning purposes. This could complement other initiatives that may contribute to shortening specialist wait times including telehealth consults [22] (electronic asynchronous consultations obviating the need for face to face appointments between patient and specialist) and e-referral systems (i.e. province wide electronic health referral system where wait times could be viewed and referral status could be tracked), pre-assessment in specialized clinics [23] (a model of triage and appointment allocation to reduce wait times), and central intake [24] (instead of having multiple-queues and multiple-servers to manage referrals, specific specialists in a given jurisdiction would have a single queue allowing each patient to see the first available specialist) [25,26]. In 2008, the Edmonton North Primary Care Network (PCN) developed a provincial e-referral system, which includes a specialist database with information on specialist referral requirements, forms and protocols, and tracks wait times [27]. A trend analysis of the referral wait time (defined as the time from referral by a family physician to an appointment date with a specialist) from 2009 to 2011 using the program database (n= 33,281 referrals) for 22 specialties showed a decrease in the overall wait time year over year [25]. The province of Ontario is currently introducing eReferral which is an electronic platform, integrated within a growing number of electronic health management systems but also available as a webbased platform, which allows secure referrals to be sent and received between family physicians and specialists [28]. eReferral enhances communication between primary care providers and specialists, eliminates the need for fax-based methods of correspondence, and displays specialist wait time information [29].
The United Kingdom's National Health Service (NHS) tracks wait times from the time a patient is referred to a specialist by the family doctor to the time the patient receives medical treatment from a specialist. The NHS has set a benchmark: 90-95% of patients should wait no longer than 18 weeks from the time at which they are referred to the time when they are treated [11]. In comparing our median wait time data with the NHS 18 week benchmarks it was interesting to note that about 90% of patients had a wait time within the 18 week target. Applying benchmarks could be another way to shift the needle on wait times. A first step is to be able to have access to the wait time information, which is presently not easily extractable and publicly available by most EMR vendors.
This study demonstrated that wait time information, valued by family physicians and specialists, lives within electronic health records, but is not easily extractable without manual manipulation. Aside from time required to create the wait time reports manually, limitations of the current study include a high degree of missing data such as postal codes and urgency of the referral, which were used to derive income quintiles, material deprivation indices, and urgency of referral. The degree of missing data highlights the need to have these fields entered in a codifiable format in the EMR. The study also employed a small sample size (both in terms of unique patients and numbers of referrals) and only a single EMR. The same level of detailed referral data may not be available in all EMRs. The study setting focused on the Greater Toronto Area, and is likely not representative of wait times for a broader geography. The study also did not consider the time to make the referral in the first place (wait time 0), before it was sent to the specialist and the appointment was booked. However, the primary objective of the study was not to analyze the wait times themselves, but to explore the feasibility of extracting wait time data from the EMR and the clinical utility of the information for family physicians and specialists. The heterogeneity with which referral information is captured in primary care EMRs is a hurdle to scaling up wait time reporting to provincial/national levels. However, this study highlights important potential uses of wait time information for providers, patients and health regulators, and next steps would include the development and expansion of systems, such as eReferral, to more accurately collect and report wait time data. Finally, the descriptive qualitative aspect of this research never intended to reach saturation of all ideas related to wait time reporting, but rather act as a starting point to obtain information about how family physicians and specialists' view and value wait time information.

Conclusion
Wait time information is perceived as valuable information to family physicians and specialists. While there are challenges with scaling up the functionality of various EMRs to provide wait time data, this study demonstrates that having access to specialist wait time information could aid provider and patient decisionmaking regarding specialist referrals, and potentially help with human resource planning and reduction of system bottlenecks. The use of technologies such as machine learning to code existing EMR data, or eReferral systems that readily report wait times, may improve the efficiency of creating wait time reports or sharing wait time information. Future work can be directed towards expanding primary care to specialist wait time reporting functionality through referral modalities such as e-referral systems. This way, family physicians and patients can make a more informed choice when considering the multitude of factors that go into the decision of which specialist to refer to.