Skip to main content
  • Research article
  • Open access
  • Published:

Printed educational materials directed at Ontario family physicians do not improve adherence to guideline recommendations for diabetes management: a pragmatic, factorial, cluster randomized controlled trial [ISRCTN72772651]



Printed educational materials (PEMs) have long been used to inform clinicians on evidence-based practices. However, the evidence for their effects on patient care and outcomes is unclear. In Ontario, despite widely available clinical practice guidelines recommending antihypertensives and cholesterol-lowering agents for patients with diabetes, prescriptions remain low. We aimed to determine whether PEMs can influence physicians to intensify prescribing of these medications.


A pragmatic, 2 × 2 factorial, cluster randomized controlled trial was designed to ascertain the effect of two PEM formats on physician prescribing: a postcard-sized message (“outsert”) or a longer narrative article (“insert”). Ontario family physician practices (clusters) were randomly allocated to receive the insert, outsert, both or neither. Physicians were eligible if they were in active practice and their patients were included if they were over 65 years with a diabetes diagnosis; both were unaware of the trial. Administrative databases at ICES (formerly the Institute for Clinical Evaluative Sciences) were used to link patients to their physician and to analyse prescribing patterns at baseline and 1 year following PEM mailout. The primary outcome was intensification defined as the addition of a new antihypertensive or cholesterol-lowering agent, or dose increase of a current drug, measured at the patient level. Analyses were by intention-to-treat and accounted for the clustering of patients to physicians.


We randomly assigned 4231 practices (39% of Ontario family physicians) with a total population of 185,526 patients (20% of patients with diabetes in Ontario primary care) to receive the insert, outsert, both, and neither; among these, 4118 practices were analysed (n = 1025, n = 1037, n = 1031, n = 1025, respectively). No significant treatment effect was found for the outsert (odds ratio (OR) 1.01, 95% confidence interval (CI) 0.98 to 1.04) or the insert (OR 0.99, 95% CI 0.96 to 1.02). Percent of intensification in the four arms was similar (approximately 46%). Adjustment for physician characteristics (e.g., age, sex, practice location) had no impact on these findings.


PEMs have no effect on physician’s adherence to recommendations for the management of diabetes-related complications in Ontario. Further research should investigate the effect of other strategies to narrow this evidence-to-practice gap.

Trial registration

ISRCTN72772651. Retrospectively registered 21 July 2005.

Peer Review reports


Researchers have estimated that it takes 17 years for research findings to enter usual care [1]. Thus, clinicians may deliver outdated, unnecessary, and even harmful care many years after evidence emerges [2]. This disconnect between research and routine clinical practice is known as the evidence-to-practice gap, or the second translational gap, and prevents patients from receiving the best-known care [3].

Evidence-to-practice gaps are prominent in primary care [3]. For example, in Ontario, despite a universal health insurance system, medications proven to prevent diabetes-related complications are underprescribed [4]. Cardiovascular disease remains the leading cause of death and disability among individuals with diabetes [5], highlighting the need for knowledge translation (KT) strategies to improve the clinical management of its risk factors, such as hypertension. Antihypertensives and cholesterol-lowering agents have been shown to reduce morbidity in individuals with diabetes more effectively than glucose-lowering agents alone [6, 7]. Thus, a multi-faceted treatment approach, managing all risk factors, is vital. This body of evidence has been translated into numerous clinical practice guidelines [8,9,10,11]; nevertheless, in 2007, the prescribing of relevant medications in Ontario fell short of guideline recommendations by as much as 30% [4].

Many KT strategies have been proposed to help close such evidence-to-practice gaps. Printed educational materials (PEMs) are recommendations for clinical care delivered in print format to the recipient, and have long been favoured to address evidence-to-practice gaps due to their low costs and potential for wide reach [12]. Bero et al. were the first to review evaluations of this KT strategy, finding that PEMs have little to no effect on changing provider behaviour, and therefore recommended that more intensive interventions be used to alter practice [13]. However, in a 2004 systematic review, PEMs were found to result in modest absolute improvements in care, ranging from 3.6 to 17%, with a median of 8.1%; thus, the authors concluded that PEMs should not be discarded, especially as costs are low and, in many situations, resources for behaviour change interventions are scarce [14]. Clarifying this, a 2008 systematic review concluded that PEMs were ineffective when compared to other KT interventions, but when compared to no intervention, led to statistically significant improvements in care [15]. However, a 2015 systematic review again concluded that PEMs do not improve outcomes, neither at the physician level, nor at the patient level [16]. These authors suggested that the encouraging results identified in previous reviews may be due to the inclusion of specialist physicians, who might respond differently to PEMs than do primary care physicians [16]. Most recently, the evidence has again reversed, with a 2020 systematic review finding that PEMs have “modest, but potentially important, improvements in professional practice” [17]. The conclusions from this review were influenced by the findings from the first two published OPEM (Ontario Printed Educational Messages) trials [18, 19]. The third and last OPEM trial is reported here, providing more real-world randomized evidence on the effectiveness of PEMs.


The aim of the OPEM programme was to test if PEMs improve care in a larger scale within usual primary care clinical practice. The underlying purpose was to help determine whether PEMs should be routinely used by local decision makers (i.e., the Ontario Ministry of Health and Long-Term Care) aiming to address evidence-to-practice gaps in primary care. We report here the third of three pragmatic [20], multicenter, factorial, cluster randomized controlled trials (cRCTs) amongst Ontario family physicians (FPs). A cluster design was chosen to minimize contamination between physicians working together, and between patients cared for by the same physician. Further, a factorial design allowed us to simultaneously test the independent effects of two different versions of a PEM: an insert and an outsert (described in detail in methods). In this randomized controlled trial (RCT), we aimed to establish whether PEMs urging intensification of antihypertensives and cholesterol-lowering medications had an impact on physician prescribing of medications for diabetes-related complications, measured at the individual patient level.


Study design and randomization

A pragmatic [20], multicenter, 2 × 2 factorial, cRCT design was chosen with physician practices as the unit of randomization and outcomes measured at the individual patient level. To prevent contamination, physicians who work in group or shared practices were identified by common address and were randomized to receive the same intervention [4]. Group practices allow for patients to be seen by more than one physician; therefore, patients who received a prescription written by an Ontario FP qualified for inclusion, even if the physician wasn’t their primary care provider [4].

Randomization was carried out at the cluster level by the study statistician using computer-generated random numbers, omitting stratification. Practices were allocated in a 1:1:1:1 ratio to receive a longer or shorter PEM, both, or neither. Allocation was concealed from physicians and patients who were both blinded as to the existence of the trial; physicians were obviously aware of the intervention they received but it is unlikely that patients were made aware of this by their physician. All patients who qualified for inclusion within the practices were automatically included in the study and their prescriptions analysed; no recruitment efforts were required.

The PEMs were mailed alongside the January edition of informed (see intervention section) to all Ontario FP practices on January 15, 2005. Pre- and post-intervention prescribing were measured from January 15, 2004 to January 14, 2005, and January 16, 2005 to January 15, 2006, respectively.

Additional details regarding the study design are available in the published protocol [4].

Study setting

The study was carried out in Ontario, Canada. Ontario residents are eligible to access the majority of health care services at no cost as a result of a publicly funded health care system, the Ontario Health Insurance Program (OHIP) [21]. Among the list of qualifying services is visits to a family doctor [21]. Further, the majority of prescriptions are covered for Ontario residents aged 65 and older through the Ontario Drug Benefit (ODB) program [22]. This includes medications to treat diabetes [22].

Data sources

All study data were obtained from administrative databases at ICES (formerly the Institute for Clinical Evaluative Sciences); no data collection or real time follow-up of participants was required. The following seven databases were linked to ascertain patient and physician characteristics, treatment regimens, and the effectiveness of the intervention: 1) Ontario Drug Benefit Claims (ODB, which collects prescription information for individuals 65 and older in Ontario) [23]; 2) Ontario Health Insurance Plan Claims Database (OHIP, which contains information on the inpatient and outpatient services that are publicly funded for Ontario residents); 3) Corporate Provider Database (CPDB, which contains demographic, specialty, eligibility, and practice location information for all physicians funded by the Ministry of Health); 4) ICES Physician Database (IPDB, which contains information on both demographic and professional characteristics of all Ontario physicians); 5) Registered Persons Database (RPDB, which houses information on basic patient demographics, received from the Ministry of Health, for all Ontarians with a health card number); 6) Drugs List (which contains the 8-digit drug identification number (DIN) for prescription drugs approved for use in Canada); 7) Ontario Diabetes Dataset (ODD, an ICES-derived database that contains information on incident and prevalent cases of diabetes in Ontario) [24]. These datasets were linked using unique, encoded identifiers and analyzed at ICES.

Study population


Eligible FPs were registered for reimbursement with OHIP between August 1, 2003 and July 31, 2004 (lookback period) [4]. At the time of the study, the vast majority of Ontario physicians billed through OHIP [25]. FPs who were not considered to be in “active” practice in 2004 were excluded. This included those who accumulated less than $50,000 in fee-for-service billings in 2004, those who prescribed medications to fewer than 100 patients aged 66 and above in 2004, and those who did not prescribe medication to an individual 66 or older in at least 10 months in 2004 [14]. The latter two exclusion criteria were used to ensure that the physicians had adequate experience with seniors. Furthermore, physicians who submitted a claim under a speciality other than FP during the lookback period were excluded. Physicians were excluded during the post-intervention period for the following reasons: submitting a claim under a speciality other than FP, no longer practicing, failed to be randomized, not matched to at least one patient with diabetes, missing information on personal identifiers (due to ICES constraints), and working in shared practices that were mistakenly sent multiple versions of the PEM.


Patients were considered for inclusion if they had a diabetes diagnosis (type 1 or type 2) on or before January 15, 2004. Individuals who were younger than 66 years as of January 15, 2005, had an invalid personal identifier at ICES, or were non-Ontario residents were excluded. Individuals who did not see an OPEM physician during the pre-intervention period, or received an equal number of billable services from more than one physician, were also excluded. Individuals who did not see an OPEM physician during the post-intervention period, and those who filled a prescription for one of the study drugs that was prescribed by a non-OPEM physician (i.e., by an Endocrinologist), were excluded. Individuals who died before the intervention was delivered were also excluded. Lastly, individuals who were matched to a physician with missing personal identifiers were excluded.

Individuals who received a prescription for an ACE inhibitor, “other” antihypertensive agent (described below), or a cholesterol-lowering agent during the pre-intervention period were identified in ODB. The prescription(s) closest to January 15, 2005 (PEM mailout) were identified and the OPEM physician/physician group that prescribed these drugs was flagged. The patient was then linked to this physician/physician group. If a patient was not prescribed a study drug during the pre-intervention period, this patient was linked to the OPEM physician/physician group that provided the majority of visits for that patient in the pre-intervention year.


Informed was a peer-reviewed practice synopsis introduced in 1994 that provided an overview of the latest research findings to promote evidence-based medicine, developed using expertise from both clinical and research staff at ICES [4]. Subscription to informed was free and voluntary for all FPs in Ontario, and approximately 15,000 FPs subscribed throughout its 13-year life [18, 19].

The OPEM programme utilized the reach of informed, and its reputation, to deliver the intervention to the eligible FPs. The PEMs came in two different forms: a short, directive message (outsert), and a long, detailed message (insert) (see Figs. 1 and 2, respectively). The outsert was a postcard-sized paper attached to the front page of informed in the bottom left corner [4]. Bright colors and large font sizes were used to attract the eye of the reader. The main recommendations to prescribe more than one antihypertensive, one of which is an ACE inhibitor, and a cholesterol-lowering agent are clearly highlighted on front side of the outsert, while the back side provides a brief explanation for the recommendations, as well as a link to obtain more information [4]. The alternative intervention against which the outsert was compared was a two-page, more traditional narrative review article (insert) that was designed to look like the rest of the articles in that edition of the newsletter, but discussed a topic that was not covered elsewhere in the edition [4]. FPs received one of the two messages, both, or neither; all four groups received the same issue of informed. PEMs were only delivered once.

Fig. 1
figure 1

Front and back sides of the outsert PEM

Fig. 2
figure 2

First page of the insert PEM


The goal of the trial was to determine whether PEMs are effective at prompting physicians to intensify their patient’s treatment regimen. A standard definition for the intensification of diabetes treatment does not exist. However, the addition of a new (i.e., not previously prescribed) agent to the patient’s treatment regimen and an increase in dose of an existing medication are often used as indicators [26,27,28,29].


The primary outcome is therefore defined as the intensification of a patient’s treatment regimen by adding a new drug, either an angiotensin-converting enzyme (ACE) inhibitor, “other” antihypertensive (e.g., angiotensin II receptor blockers, beta-blockers, calcium channel blockers, diuretics), or cholesterol-lowering agent (e.g., statins, fibrates, antilipemic and atherosclerosis inhibiting agents) and/or increasing the dose of a current ACE inhibitor, “other” antihypertensive, or cholesterol-lowering agent (see Additional file 1 Table 2 for a detailed description of outcome programming).


Drug switches occur for a multitude of reasons, including patient preference, physician preference, pharmaceutical influence, and patient side effects. Physicians may have switched their patient from one drug to another, perhaps a stronger drug, in response to the intervention. Thus, medication switches were included in the secondary outcome measurement to capture physicians who intensified their patient’s treatment regimen by making a drug switch. The secondary outcome is therefore defined as the addition of an ACE inhibitor, “other” antihypertensive agent, or cholesterol-lowering agent, the increase in dose of a current ACE inhibitor, “other” antihypertensive agent, or cholesterol-lowering agent, or the switch from one drug to another across all drug classes.


We were interested in detecting improvements in the proportion of individuals experiencing treatment intensification as small as 5% [4]. Previous data revealed that 36% of individuals with diabetes in Ontario were prescribed an ACE inhibitor at baseline [4]. With an assumption that each FP sees ten individuals with diabetes, and an intracluster correlation coefficient of 0.1, 1250 practices per arm provides 97% power to detect a 5% improvement in prescribing [4].

Statistical analysis

Logistic regression models estimated using generalized estimating equations with robust standard errors were fit to ascertain the effectiveness of PEMs while taking into account the clustering of patients to their physician groups. Analyses followed the recommendations set forth by the CONSORT extension for cRCTs [30]. Further, following standard practice for the analysis of pragmatic RCTs, the primary and secondary analyses were planned as intention-to-treat analyses. Three logistic regression models were fit: two for the primary outcome (unadjusted, and adjusted for physician characteristics), and one for the secondary outcome (unadjusted). We deemed two-tailed p-values less than or equal to 0.05 to be statistically significant. All statistical analyses were conducted using SAS version 9.4 (SAS Institute, Cary, NC).


Physician and patient selection

The number of physicians and patients included in the trial is shown in Fig. 3. Over half of FPs were deemed ineligible due to not being in “active” practice or for submitting a claim under another speciality during the lookback period. A further 221 were excluded after randomization for reasons such as submitting a claim under a different specialty, no longer practicing, and working in a practice that received multiple PEM formats during the post-intervention period; thus, 4957 physicians were ultimately included, working in 4118 practices. Approximately 20% of individuals with a diabetes diagnosis on or before January 15, 2004 met the eligibility criteria. Seventy-two patients were excluded after randomization because they were matched to a physician with missing information (ICES constraints); accordingly, 185,454 patients were included in the final analysis.

Fig. 3
figure 3

Physician and patient selection

Descriptive statistics

1025, 1037, 1031, and 1025 physician practices (clusters) received the insert, outsert, both, and neither, respectively, and were analysed. There were 44,845, 47,602, 45,508, and 47,499 patients in physician practices that received the insert, outsert, both, and neither, respectively. Characteristics of the physicians and the patients at baseline were well balanced (Tables 1 and 2). The number of physicians working in solo and shared practices was 3548 (72%) and 1409 (28%), respectively. The median number of eligible diabetes patients per physician group was 36 (interquartile range (IQR) = 38), while the median number of patients per individual physician was 32 (IQR = 31).

Table 1 Baseline characteristics of physicians
Table 2 Baseline characteristics of patients

Analysis of intervention effects: primary outcome

The percent of patients experiencing treatment intensification was similar across trial arms, with approximately 46% receiving an additional drug to their regimen or an increase in dose of a current drug (Table 3). In the main effects model, the odds ratio (OR) for the outsert effect was 1.01 (95% confidence interval (CI) 0.98 to 1.04), and the OR for the insert effect was 0.99 (95% CI 0.96 to 1.02) (Table 4). A model was also fit to include the interaction between the outsert and the insert. The OR for the interaction was 1.01 (95% CI 0.98 to 1.11), indicating that the effect of the outsert was unchanged by the presence of the insert (p = 0.17). Further, neither a statistically significant outsert effect, nor an insert effect, were observed after adjusting for the following physician characteristics: physician age, sex, country of medical training (Canadian vs. other), practice location (Northern vs. Southern, rural vs. non-rural), total general patient visits per year, total billings per year, and years in practice (Table 4).

Table 3 Percent of patients who experienced treatment intensification
Table 4 Unadjusted and adjusted intervention effects

The intracluster correlation coefficient was 0.023, and the variance inflation factor was 2.01.

Analysis of intervention effects: secondary outcome

The percent of individuals experiencing a treatment intensification based on additions, dose increases, and switches was comparable across treatment groups and was, on average, slightly higher (approximately 2%) than those based on additions and dose increases alone (Table 3). The ORs for the outsert and insert effects were 1.01 (95% CI 0.98 to 1.04) and 0.99 (95% CI 0.96 to 1.02), respectively (Table 4).


The aim of the OPEM trial was to determine whether PEMs can successfully influence FPs to improve adherence to guideline recommendations for diabetes care through treatment intensification. There were no changes (neither statistically, nor clinically) in intensification of medications (with or without including drug switches) in response to either version of the PEM, with no interaction effect. Thus, while the trial was primarily designed to test the independent effects of both the outsert and the insert, we have little reason to believe that FPs who received the combination of both PEMs would have any difference in outcome. The PEMs are mostly duplicates of each other; FPs, who already face demands on their time, are unlikely to read the insert in full after reading the brief, post-card sized summary.

Over all arms, regardless of the intervention received, we found that female physicians, those who received their medical training outside of Canada, and those who had been in practice for fewer years were more likely to engage in treatment intensification (Table 4). Nevertheless, these three effect sizes were very close to one, suggesting that, while statistically significant, the effect of the aforementioned physician characteristics on treatment intensification was minimal and is unlikely to translate into clinically meaningful effects.

Our findings are consistent with previous research [13, 16, 31, 32], including our own previously published OPEM trials [18, 19]. The failure of PEMs to bring about behavior change among physicians may be an indication that this KT strategy is largely ineffective and could be safely abandoned. PEM-based interventions are centered around the idea that evidence, once shared with an individual, prompts change. The negative findings from this and many other studies suggests that other barriers may exist. For example, physicians often struggle to “unlearn” old behaviors [33, 34]. To change clinical practice, physicians must replace routine, outdated operations with new, evidence-based practices. While adding to their knowledge base can be a simpler undertaking, dismissing current practices presents a greater challenge [33, 34]. The process of change disrupts the status quo equilibrium, causing physicians to be uncertain about practices they considered to be “certain” [34]. Physicians included in the study had been practicing as an FP for 22 years on average (Table 1). As a result, it is possible that they may have adopted standard prescribing practices for individuals with diabetes over the years and were thus hesitant to implement a change to multiple drug classes when presented with evidence in the PEMs. Furthermore, the patients themselves may act as a barrier to PEM effectiveness. A qualitative study by Parsons et al. [35] investigated why printed educational toolkits mailed to Ontario FPs failed to improve cardiovascular disease management among individuals with diabetes. The FPs believed that patient motivation and willingness to change behavior was the primary obstacle preventing them from adhering to the PEM recommendations [35].


Our trial was randomized, and, as consent was not required, also blinded with high internal validity. It was also very large, offering a precise estimate of effect, and was designed with mostly pragmatic features to align with our pragmatic (decision-making) intention, increasing its relevance to real-world decision making in Ontario and similar settings (see Additional file 1 Table 1 and Fig. 1 for PRECIS-2 table and wheel, respectively) [36]. The PEMs were delivered in a way that is likely similar to that of a routine large scale programme: by mail to physicians alongside a journal many of them routinely received, targeting the providers who usually care for patients with diabetes in Ontario. Outcomes were measured using routinely collected electronic health information, avoiding any distortions to usual care that might arise from data collection from physicians or patients. Lastly, while prescribing recommendations were outlined in the PEMs, the choice to prescribe was ultimately left to the physician – a naturalistic approach with no restrictions on behaviour, identical to a PEM program as might be implemented in routine practice.


Our study design cannot distinguish the point at which PEMs failed. Failure may be due to non-delivery, or the messages not being opened or read by the physicians. A 1997 survey revealed that Ontario physicians considered informed to be a useful source of clinical information (71% of physicians reporting received informed, and of these, 89% reported that the information was useful) [37]. But receptiveness may not be the same as clinical influence, and 8 years later, at the time of our trial, its influence may have declined. Further, while Canadian postal delivery is reliable, physicians may not be opening their own mail, and even if they do, may not be reading informed.

Our study is relevant only to print based educational materials delivered by mail. It is possible that physicians would have been more influenced by the PEM had it been delivered by an alternative route (e.g., fax, email). Following the shift towards electronic information as a basis for clinical decision making, electronically-delivered recommendations may have been more widely received by physicians [38, 39]. However, these are seldom reported in the literature. A 2020 systematic review of PEMs identified only two studies comparing printed versus electronically delivered guidelines and found little to no difference in effectiveness between the two delivery strategies [17]. That said, the quality of evidence from these studies was graded as low [17]; thus, additional studies investigating electronically delivered guidelines may be warranted.

While the trial was designed with a pragmatic intention, it focused on higher volume clinicians treating older persons with diabetes. Only 5179 of the 10,863 eligible FPs practicing in Ontario were included, and although the excluded physicians were mostly inactive or saw few patients over 65 years, we cannot claim complete representativeness. The results are applicable to Ontario FPs who billed at least $50,000 in 2004, and who prescribed medications to at least 100 seniors (≥66 years old) in at least 10 months in 2004. It is possible that physicians who treat fewer individuals 66 and above and have less experience with diabetes care might have a different outcome.

In addition, the ODB claims database only contains prescriptions that have been reimbursed, which depends on whether they have been dispensed. The intervention effect might have been diluted by patients who received an intensified prescription but failed to get it dispensed. These individuals are likely to be balanced among the four groups as a consequence of randomization; thus, our estimate of the relative effect is unlikely to biased.

A discussion of the limitations would be incomplete without acknowledging the timeframe in which the study was conducted. The PEMs were mailed in January 2005, and the outcome measurement took place between 2005 and 2006. Estimates at the time of the trial suggested that prescriptions for all of ACE inhibitors, “other” antihypertensives, and cholesterol-lowering agents were at least 30% below guideline recommendations [4]. One may wonder whether the results from the present study are relevant to clinical practice today. In 2015, it was found that the percent of patients aged 65 and above with diabetes in Ontario who filled a prescription for an ACE inhibitor or angiotensin II receptor blocker and a statin was approximately 60 and 72%, respectively [40]. Further, the 2018 Diabetes Clinical Practice Guidelines still declare that diabetes-related cardiovascular complications are responsible for the greatest burden of death among individuals with diabetes and recommend treatment with appropriate medications to manage these risks, including antihypertensives and cholesterol-lowering agents [11]. Thus, the evidence-to-practice gap continues to persist. It is, however, worth noting that, while the recommended target drug classes remain the same, some of the medications that were included as part of treatment intensification in the trial, for example atherosclerosis inhibiting agents, are not recommended today. Therefore, one must interpret the results within the timeframe of the study.


Overall, in this large, randomized trial, we found that PEMs alone, whether long and narrative or brief and action focused are not effective at intensifying prescriptions for individuals with diabetes aged 66 and above in Ontario primary care. The implications for local decision makers are clear: despite their low costs and wide reach, PEMs, in the formats we tested, are not effective at bridging diabetes-related evidence-to-practice gaps in primary care in Ontario. Funding agencies should be cautious in supporting additional evaluations of these PEMs without first developing a deep and theoretically coherent understanding of the facilitators and barriers to their success. Researchers may consider conducting qualitative studies to gain perspective from the physicians on why this particular KT strategy has been repeatedly found to be ineffective, and to determine whether modifications to the design of the PEMs and/or the method of delivery may result in greater uptake, or, whether this KT strategy should be safely abandoned. In this modern era of electronic health records, one might look towards electronic interventions such as point of care reminders, on screen prompts, and electronic audit and feedback [41, 42].

Availability of data and materials

The dataset from this study is held securely in coded form at ICES. While data sharing agreements prohibit ICES from making the dataset publicly available, access may be granted to those who meet pre-specified criteria for confidential access, available at The full dataset creation plan and underlying analytic code are available from the authors upon request, understanding that the computer programs may rely upon coding templates or macros that are unique to ICES and are therefore either inaccessible or may require modification.


ACE inhibitor:

Angiotensin-converting enzyme inhibitor


Corporate Provider Database


Cluster randomized controlled trial


Drug identification number


Family physician


ICES Physician Database


Interquartile range


Knowledge translation


Ontario Drug Benefit Claims


Ontario Diabetes Dataset


Ontario Health Insurance Plan Claims Database


Ontario Printed Educational Message programme


Odds ratio


Printed educational material


Randomized controlled trial


Registered Persons Database


Standard deviation


  1. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. 2011;104:510–20.

    Article  Google Scholar 

  2. Grimshaw J, Eccles M, Lavis J, Hill S, Squires J. Knowledge translation of research findings. Implement Sci. 2012;7:50.

    Article  Google Scholar 

  3. Lau R, Stevenson F, Ong B, et al. Addressing the evidence to practice gap for complex interventions in primary care: a systematic review of reviews protocol. BMJ Open. 2014;4:e005548.

    Article  Google Scholar 

  4. Zwarenstein M, Hux JE, Kelsall D, et al. The Ontario printed educational message (OPEM) trial to narrow the evidence-practice gap with respect to prescribing practices of general and family physicians: a cluster randomized controlled trial, targeting the care of individuals with diabetes and hypertension in Ontario, Canada. Implement Sci. 2007;2:37.

    Article  Google Scholar 

  5. Leon B, Maddox TM. Diabetes and cardiovascular disease: epidemiology, biological mechanisms, treatment recommendations and future research. World J Diabetes. 2015;6:1246.

    Article  Google Scholar 

  6. UK Prospective Diabetes Study Group. Tight blood pressure control and risk of macrovascular and microvascular complications in type 2 diabetes: UKPDS 38. BMJ. 1998;317:703–13.

    Article  Google Scholar 

  7. Heart Protection Study Collaborative Group. MRC/BHF heart protection study of cholesterol-lowering with simvastatin in 5963 people with diabetes: a randomised placebo-controlled trial. Lancet. 2003;361:2005–16.

    Article  Google Scholar 

  8. Canadian Diabetes Association Clinical Practice Guidelines Expert Committee. Canadian Diabetes Association 2003 clinical practice guidelines for the prevention and Management of Diabetes in Canada. Can J Diabetes. 2003;27(suppl 2):S1–S152.

    Google Scholar 

  9. Canadian Diabetes Association Clinical Practice Guidelines Expert Committee. Canadian Diabetes Association 2008 clinical practice guidelines for the prevention and management of diabetes in Canada. Can J Diabetes. 2008;32(suppl 1):S1–S201.

    Google Scholar 

  10. Canadian Diabetes Association Clinical Practice Guidelines Expert Committee. Canadian Diabetes Association 2013 clinical practice guidelines for the prevention and Management of Diabetes in Canada. Can J Diabetes. 2013;37(suppl 1):S1–S212.

    Google Scholar 

  11. Diabetes Canada. Diabetes Canada 2018 clinical practice guidelines for the prevention and Management of Diabetes in Canada. Can J Diabetes. 2018;42:S1–S325.

    Article  Google Scholar 

  12. Giguère A, Légaré F, Grimshaw J. Printed educational materials: effects on professional practice and healthcare outcomes (review). Cochrane Database Syst Rev. 2012;10:CD004398.

    PubMed  Google Scholar 

  13. Bero L, Grilli R, Grimshaw J, Harvey E, Oxman A, Thomson M. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. BMJ. 1998;317:465.

    Article  CAS  Google Scholar 

  14. Grimshaw JM, Thomas RE, MacLennan G, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004;8:1–72.

    Article  Google Scholar 

  15. Farmer A, Légaré F, Turcot L, et al. Printed educational materials: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2008;3:CD004398.

    Google Scholar 

  16. Grudniewicz A, Kealy R, Rodseth R, Hamid J, Rudoler D, Straus S. What is the effectiveness of printed educational materials on primary care physician knowledge, behaviour, and patient outcomes: a systematic review and meta-analyses. Implement Sci. 2015;10:164.

    Article  Google Scholar 

  17. Giguère A, Zomahoun HTV, Carmichael PH, et al. Printed educational materials: effects on professional practice and healthcare outcomes (review). Cochrane Database Syst Rev. 2020;8:CD004398.

    PubMed  Google Scholar 

  18. Zwarenstein M, Shiller SK, Croxford R, et al. Printed educational messages aimed at family practitioners fail to increase retinal screening among their patients with diabetes: a pragmatic cluster randomized controlled trial [ISRCTN72772651]. Implement Sci. 2014;9:87.

    Article  Google Scholar 

  19. Zwarenstein M, Grimshaw JM, Presseau J, et al. Printed educational messages fail to increase use of thiazides as first-line medication for hypertension in primary care: a cluster randomized controlled trial [ISRCTN72772651]. Implement Sci. 2016;11:124.

    Article  Google Scholar 

  20. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20:637–48.

    Article  CAS  Google Scholar 

  21. Health care in Ontario. Accessed 12 Jun 2021.

  22. Get coverage for prescription drugs. Accessed 12 Jun 2021.

  23. Levy AR, O’Brien BJ, Sellors C, Grootendorst P, Willison D. Coding accuracy of administrative drug claims in the Ontario drug benefit database. Can J Clin Pharmacol. 2003;10:67–71.

    PubMed  Google Scholar 

  24. Hux JE, Flintoft V, Ivis F, Bica A. Determination of prevalence and incidence using a validated administrative data algorithm. Diabetes Care. 2002;25:512–6.

    Article  Google Scholar 

  25. Chan BTB, Schultz SE. Supply and utilization of general practitioner and family physician Services in Ontario. ICES investigative report. Toronto: Institute for Clinical Evaluative Sciences; 2005.

    Google Scholar 

  26. Fu A, Sheehan J. Treatment intensification for patients with type 2 diabetes and poor glycaemic control. Diabetes Obes Metab. 2016;18:892–8.

    Article  CAS  Google Scholar 

  27. Desai U, Kirson N, Kim J, et al. Time to treatment intensification after Monotherapy failure and its association with subsequent glycemic control among 93,515 patients with type 2 diabetes. Diabetes Care. 2018;41:2096–104.

    Article  CAS  Google Scholar 

  28. Arnold R, Yang S, Gold E, Farahbakhshian S, Sheehan J. Assessment of the relationship between diabetes treatment intensification and quality measure performance using electronic medical records. PLoS One. 2018;13:e0199011.

    Article  Google Scholar 

  29. Canivell S, Mata-Cases M, Real J, et al. Glycaemic control after treatment intensification in patients with type 2 diabetes uncontrolled on two or more non-insulin antidiabetic drugs in a real-world setting. Diabetes Obes Metab. 2019;21:1373–80.

    Article  CAS  Google Scholar 

  30. Campbell MK, Piaggio G, Elbourne DR, Altman DG, for the CONSORT Group. Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012;345:e5661.

    Article  Google Scholar 

  31. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362:1225–30.

    Article  Google Scholar 

  32. Shah BR, Bhattacharyya O, Yu CHY, Mamdani MM, Parsons JA, et al. Effect of an educational toolkit on quality of care: a pragmatic cluster randomized trial. PLoS Med. 2014;11(2):e1001588.

    Article  Google Scholar 

  33. Rushmer R, Davies HTO. Unlearning in health care. Qual Saf Health Care. 2004;13:10–5.

    Article  Google Scholar 

  34. Gupta DM, Boland RJ, Aron DC. The physician’s experience of changing clinical practice: a struggle to unlearn. Implement Sci. 2017;12:28.

    Article  Google Scholar 

  35. Parsons JA, Yu CHY, Baker NA, Mamdani MM, Bhattacharyya O, et al. Practice Doesn’t always make perfect: a qualitative study explaining why a trial of an educational toolkit did not improve quality of care. PLoS One. 2016;11(12):e0167878.

    Article  Google Scholar 

  36. Loudon K, Treweek S, Sullivan F, et al. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147.

    Article  Google Scholar 

  37. Zwarenstein M, Grimshaw JM, Presseau J, et al. Printed educational messages fail to increase use of thiazides as first-line medication for hypertension in primary care: a cluster randomized controlled trial [ISRCTN72772651]. Implement Sci. 2016;11:124.

  38. Alper BS, Stevermer JJ, White DS, Ewigman BG. Answering family physicians' clinical questions using electronic medical databases. J Fam Pract. 2001;50(11):960–5.

    CAS  PubMed  Google Scholar 

  39. Davies K. The information-seeking behaviour of doctors: a review of the evidence. Health Inf Libr J. 2007;24:78–94.

    Article  Google Scholar 

  40. Bamimore MA, Devlin RA, Zaric GS, Garg AX, Sarma S. Quality of diabetes Care in Blended fee-for-service and blended capitation payment systems. Can J Diabetes. 2021;45:261–8.

    Article  Google Scholar 

  41. Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles MP, Grimshaw J. The effects of on-screen, point of care computer reminders on processes and outcomes of care. Cochrane Database Syst Rev. 2009;3:CD001096.

    Google Scholar 

  42. Tuti T, Nzinga J, Njoroge M, et al. A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory. Implement Sci. 2017;12:61.

    Article  Google Scholar 

Download references


Parts of this material are based on data and/or information compiled and provided by CIHI. However, the analyses, conclusions, opinions and statements expressed in the material are those of the authors, and not necessarily those of CIHI. We thank IQVIA Solutions Canada Inc. for use of their Drug Information Database. We gratefully acknowledge Dr. Sonja Reichert’s contributions in terms of sharing her clinical diabetes expertise.


This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). This study also received funding from the Canadian Institutes of Health Research (grant 724180703). The analyses, conclusions, opinions and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred.

Author information

Authors and Affiliations



MZ conceived the study, wrote the protocol and oversaw the implementation of the intervention. AH, MZ, NK, DN, and JR developed the study plan. AH conducted all analyses with assistance from NK and JR. AH wrote the first draft of the manuscript. All authors reviewed and revised the manuscript and approved the final version for submission.

Corresponding author

Correspondence to Alison H. Howie.

Ethics declarations

Ethics approval and consent to participate

The OPEM programme was approved by the Research Ethics Board at Sunnybrook Health Sciences Centre and the Women’s College Health Sciences Centre in Toronto, Ontario.4 Because all study outcomes were measured using ICES data, informed consent from the physicians and the patients was not required. We sought and received approval from the College of Physicians and Surgeons of Ontario to link their data on physician identifiers to the administrative databases held at ICES.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

PRECIS-2 table, wheel, and outcome programming.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Howie, A.H., Klar, N., Nash, D.M. et al. Printed educational materials directed at Ontario family physicians do not improve adherence to guideline recommendations for diabetes management: a pragmatic, factorial, cluster randomized controlled trial [ISRCTN72772651]. BMC Fam Pract 22, 243 (2021).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: