- Research article
- Open Access
Evaluation of electronic recruitment efforts of primary care providers as research subjects during the COVID-19 pandemic
BMC Primary Care volume 23, Article number: 95 (2022)
Recruiting healthcare providers as research subjects often rely on in-person recruitment strategies. Little is known about recruiting provider participants via electronic recruitment methods. In this study, conducted during the COVID-19 pandemic, we describe and evaluate a primarily electronic approach to recruiting primary care providers (PCPs) as subjects in a pragmatic randomized controlled trial (RCT) of a decision support intervention.
We adapted an existing framework for healthcare provider research recruitment, employing an electronic consent form and a mix of brief synchronous video presentations, email, and phone calls to recruit PCPs into the RCT. To evaluate the success of each electronic strategy, we estimated the number of consented PCPs associated with each strategy, the number of days to recruit each PCP and recruitment costs.
We recruited 45 of 63 eligible PCPs practicing at ten primary care clinic locations over 55 days. On average, it took 17 business days to recruit a PCP (range 0–48) and required three attempts (range 1–7). Email communication from the clinic leaders led to the most successful recruitments, followed by brief synchronous video presentations at regularly scheduled clinic meetings. We spent approximately $89 per recruited PCP. We faced challenges of low email responsiveness and limited opportunities to forge relationships.
PCPs can be efficiently recruited at low costs as research subjects using primarily electronic communications, even during a time of high workload and stress. Electronic peer leader outreach and synchronous video presentations may be particularly useful recruitment strategies.
Recruiting individual healthcare providers, such as physicians, nurse practitioners, and other medical professionals, as subjects in research studies is often a challenging task . Healthcare providers report numerous barriers to participation, including time constraints, low interest in research topics, concerns about relationships with patients, loss of professional autonomy, and reluctance to modify existing clinical workflows [2,3,4,5]. Furthermore, researchers often face additional barriers to recruiting healthcare providers, including difficulty gaining permission to enter healthcare facilities, obtaining accurate eligibility and contact information, persuading eligible providers to participate, and scheduling data collection . Consequently, many studies using healthcare providers as subjects, even randomized controlled trials (RCTs) taking place in academic medical centers, fail to achieve recruitment targets, leading to extensions of timelines, reductions in statistical power, or wasted resources [7,8,9].
Current best practices for recruiting healthcare providers include using clinical champions to contact eligible providers and establishing relationships with clinic personnel [5, 10, 11]. These best practices resulted in development of the 7R framework that provides guidance on recruitment strategies pertaining to: relationships, reputation, requirements, rewards, reciprocity, resolution, and respect . The 7R framework has been effective in recruiting healthcare providers and medical groups [5, 11], but several strategies rely on in-person interactions between the research team and eligible providers. To date, there is little evidence on how best to implement the 7R framework via electronic methods. Even less information about healthcare provider recruitment is available during times of particularly high workload and stress. The COVID-19 pandemic resulted in significant increases in providers’ workload, stress, and burnout, which may have further hindered their participation in research [12, 13]. Thus, researchers trying to recruit healthcare providers as subjects likely experienced additional logistical barriers due to healthcare facility closures, visitor restrictions, and other physical distancing policies . In addition, existing literature minimally quantifies the effect of electronic recruitment strategies on healthcare providers recruitment success and costs of recruitment [2, 15, 16].
The purpose of the current study is to describe and evaluate the implementation of a primarily electronic provider recruitment strategy that was based on the 7R framework. In the fall of 2020, we recruited primary care providers (PCPs) for a pragmatic RCT of an electronic health record (EHR)-based decision support intervention. Due to COVID-19 restrictions, we modified our recruitment approaches to use exclusively electronic strategies. We report detailed metrics capturing the electronic strategies’ effectiveness and a cost estimate of our recruitment approach. We also describe challenges faced when adopting the 7R framework for predominantly electronic provider recruitment. Our findings are relevant for future researchers aiming to recruit providers as research subjects by estimating the time and resources required, especially during times of high workload and stress.
The goal of the RCT was to assess a clinician-facing electronic health record (EHR)-based decision support tool called The Chronic Pain OneSheet (i.e., OneSheet) . OneSheet is a patient-level dashboard that aggregates information relevant to the CDC Guideline for Prescribing Opioids for Chronic Pain into a single view . OneSheet was designed to help PCPs quickly access the clinical information needed to take evidence-based action. OneSheet allows PCPs to collect and review patients’ chronic pain history, treatment plan, treatment-related risks, outcomes, and goals more efficiently. The study’s primary outcome, constructed using EHR data, is the proportion of patients with chronic pain for whom a PCP conducted guideline-recommended goal setting, pain and function assessment, and opioid-related risk assessment. The study received institutional board approval at Indiana University and is registered with ClinicalTrials.gov (NCT04295135).
Setting & eligibility criteria
Eligible PCPs included physicians, nurse practitioners, and physician assistants from general internal medicine or family medicine practices that provide primary care services to adult patients with chronic pain. PCPs were recruited from two separate academic health center sites across two states, including one of the largest safety-net health systems in the Midwest. Due to the pandemic, our recruitment of PCPs at the Midwest site was conducted electronically and is thus the focus of this study. The Midwest site is part of a health system that includes ten federally qualified community health centers that provide primary care services to about 85,000 patients per year, with about 290,000 visits in 2020 .
Original recruitment strategy
The 7R framework guided our original pre-pandemic recruitment strategy [5, 11]. Briefly, the 7R framework consists of seven strategies: 1) Relationship: recruiters need to be known for their involvement in the local medical community and for doing practical research, 2) Reputation: recruiters need to be known for doing research and participants need to believe that the relationship and information will not be abused, 3) Requirements: resource demands for participants in study-related activities need to be minimized, 4) Rewards: nominal recognition for participating and the reward of learning new knowledge are important in recognizing the participant’s effort, 5) Reciprocity: mutual obligation should be negotiated for what is to be provided by recruiters and what is to be expected from participants, 6) Resolution: recruitment persistence and a willingness to repeatedly make contact until agreement to participate is eventually reached, and 7) Respect: recruiters need to genuinely respect participants, their work, and their constraints. For our research study, several 7R framework strategies were identical for all eligible PCPs. Specifically, eligible PCPs received identical information on anticipated rewards, requirements, and reciprocity expectations.
Subsequently, we planned to rely heavily on 1) the research team’s reputation and relationships that stemmed from prior studies to design OneSheet; 2) clinical championship from a PCP research team member, and – crucially – 3) in-person presentations and individual interactions with eligible PCPs (see Fig. 1). Following a mass email from the primary care service line leader to eligible PCPs, the study team planned to follow up with in-person recruitment presentations to PCPs and drop-in consultations (e.g., during lunch hours) at each clinic. We then planned to distribute paper-based informed consent forms that eligible PCPs could complete during these interactions. Finally, we planned for a PCP champion, also a member of the research team, to promote the study among colleagues, including announcements and reminders of recruitment during in-person meetings.
Modified recruitment strategy due to COVID-19 restrictions
Due to the emergent need for physical distancing, the participating health system requested that our study team cancel planned in-person recruitment activities. As originally planned, the primary care service line leader, who is a physician, sent introductory emails connecting the research coordinator to clinic leaders (i.e., clinic manager and chief physician) at each primary care clinic site (see Fig. 1). Next, a research team member with a clinical background gave an 8-min synchronous video presentation at each clinic’s virtual monthly provider meeting. After all presentations occurred, we conducted three iterations of extensive email follow-up. First, the primary care service line leader sent additional emails to each clinic’s leaders asking them to assist in recruiting eligible PCPs at each clinic. Second, each clinic’s leaders contacted eligible PCPs through mass emails, individual emails, or, in rare cases, in-person prompting. Finally, a research coordinator sent a “last chance” email to all eligible PCPs who had not yet consented to participate in the study.
To quantify recruitment effectiveness, the research coordinator maintained a detailed spreadsheet of each communication between relevant stakeholders (including the primary care service line leader, clinic leaders, the study team) and eligible PCPs. To minimize measurement bias, the spreadsheet data were entered during our recruitment efforts. From this spreadsheet, we calculated metrics to evaluate the recruitment process, including the number of business days and recruitment contacts needed to enroll a PCP as a study participant. We defined a successful recruitment outcome as receiving a signed informed consent form and a completed demographic questionnaire from an eligible PCP. We also adopted a previously used approach calculating the costs of recruitment . Specifically, we calculated costs based on the role of the primary recruitment personnel (i.e., the study research coordinator, the co-principal investigators, and a co-investigator), percentage of time spent on recruitment, total recruitment hours, and the hourly rate.
The originally-planned March 2020 start was delayed due to COVID-19 (see Fig. 1). In total, we recruited 45 of 63 eligible PCPs practicing at ten primary care clinic locations. The majority of PCPs were medical or osteopathic physicians (70%), female (73%), white (68%), and not of Hispanic or Latino ethnicity (82%) (see Table 1). On average, recruited PCPs had spent 14 years practicing medicine. The clinical credentials and gender of PCPs who were not successfully enrolled in the study were similar to those recruited.
Overall, PCP recruitment lasted 55 business days. The synchronous video presentations at the ten clinics’ virtual provider meetings occurred over 31 business days. We spent an average of three business days and sent an average of three emails to clinic leaders before successfully being scheduled to attend a virtual provider meeting (see Table 2). Once the video presentations were completed, the research coordinator exchanged, on average, ten emails with clinic leaders to discuss recruitment-related topics. We also engaged in extensive email follow-up with eligible PCPs. Counting from the date of first contact, which was the video presentation, to the date of consent, we spent on average 17 business days (range: 0–48) and three attempts (range: 1–7) to recruit an eligible PCP. We counted contact attempts as video presentations, emails from the primary care service line leader, clinic leaders, and research team.
In Table 3, we report recruitment rates by recruitment strategy. Prompting from clinic leaders were most effective, leading to 15 successfully recruited PCPs. The researcher-led synchronous video recruitment presentations at provider meetings resulted in 11 recruited PCPs. Research coordinator emails to eligible PCPs shortly after the synchronous video recruitment presentation resulted in seven additional recruited PCPs. Other recruitment strategies collectively, including primary care service line leader emails, led to the remaining 12 recruited PCPs. The research coordinator spent 154 h on the recruitment activities with a total cost of $4004 or $89 per recruited PCP. Additionally, the research coordinator was supported by three team members involved in developing the recruitment materials, attending presentations, and communicating with health system leadership. We estimated that these three team members spent a total of 80 h on recruitment related efforts. Two-thirds of 80 h effort was devoted to recruitment during the preparation period (February–July) and one-third during the recruitment period (August–October). Including the additional effort from these team members, the total recruitment cost was $9092 or $202 per recruited PCP (See Table 4).
Several challenges encountered in recruiting PCPs are described in Table 5. Specifically, given the lack of evidence on how to operationalize the 7R framework via predominantly electronic recruitment, we may not have fully leveraged the Relationships and Reputation factors to their fullest extent. For instance, research team members could not interact in-person with eligible PCPs nor forge strong relationships with clinic leaders to facilitate recruitment. Furthermore, our electronic recruitment messages, compared to in-person methods, may have diminished Reciprocity, Resolution, and Respect. For instance, eligible PCPs often attended synchronous video presentations via phone instead of using video conferencing software, limiting their ability to see the slides and the demonstration video. By attending via phone, PCPs may have been more likely to be distracted, engaging in multitasking behaviors, or reluctant to ask clarifying questions.
Our study demonstrates that PCPs can be efficiently recruited as research subjects at low cost using predominantly electronic approaches, even during the COVID-19 pandemic. Our success can be attributed to the effective modification of recruitment strategies for enrolling healthcare providers described in the 7R framework [5, 11] to electronic modes of recruitment. First, we leveraged existing relationships from prior research projects to connect with key stakeholders. Specifically, a primary care service line leader sent introductory emails to clinic leaders (Relationship & Reputation). As with in-person recruitment [5, 16, 20,21,22], email introductions facilitated connections with each clinic and recruitment activity rollout. However, as with in-person recruitment, successful recruitment required substantial email follow-up with clinic leaders and eligible PCPs. Nevertheless, our recruitment metrics are comparable or better in terms of the number of contacts and days to enroll a PCP relative to previous studies [6, 16]. Thus, despite not having frequent in-person check-ins at the clinics, we were able to establish a good relationship with clinic leaders, which was borne out in our recruitment numbers.
Second, we relied on clinic leaders at each clinic location to assist with recruiting eligible PCPs (Relationship & Reputation). As with in-person recruitment [5, 10, 21,22,23], using such “in-house” clinic leaders, who were either physicians or managers, was an effective strategy in our study. Thus, “in-house” clinic leaders had comparative efficacy in recruiting PCPs even with predominantly electronic modes. Eligible PCPs were more responsive to emails from clinic leaders, resulting in higher recruitment rates relative to emails from the research team. To ease the burden, we shared draft emails to be used by the clinic leaders to approach eligible PCPs. Thus, future studies should explore what specific aspects of electronic communication between clinician champions and eligible PCPs are most effective.
Third, while in-person recruitment methods are generally thought to be most effective [5, 10, 11, 21], we found a number of PCPs enrolled after attending a synchronous video presentation. Our video presentation articulated anticipated requirements, expectations, rewards, and demonstrated respect for PCP’s time (Requirements, Rewards, Reciprocity, and Respect). Thus, assuming future meetings of eligible healthcare providers are available virtually, video conferencing software holds a strong promise for effectively increasing recruitment efforts. This is notable, especially for studies that span institutions and geographies for which in-person recruitment may be cost prohibitive. However, we faced several unanticipated challenges with recruiting PCPs during synchronous video presentations, including PCPs connecting via phone. This prevented some PCPs from viewing our video presentation and may have reduced their engagement. Given the flexibility, relatively low cost, and effectiveness of the synchronous video presentations, additional research is needed to identify best practices for gaining higher PCP engagement and buy-in during synchronous virtual presentations.
Finally, effective adaptation of the 7R framework allowed us to conduct our recruitment at a relatively low costs, an estimated $9092 or $202 per recruited PCP. Our recruitment costs are substantially lower than one study estimating recruitment costs published to date . Of note, Fagnan et al., reported costs associated with recruiting an entire practice, used multiple recruiters with clinical training and thus higher hourly rates, and had contacted substantially greater number of eligible subjects (N = 3669 practices) relative to our study. The cost difference could also be explained by a substantial difference in the scales of the studies. Fagnan’s study enrolled eligible practices from six states across the country relative to only one in our study. Unlike Fagnan’s recruitment costs, our costs calculations also did not include travel time to recruit eligible PCPs, since there were no in-person interactions, potentially contributing to cost-savings. Finally, our research team recruited eligible PCPs from a health system with which the team had an ongoing academic partnership, which may have allowed us to incur lower recruitment costs.
Our study has several strengths. First, to our knowledge, we are the first to report experiences of recruiting PCPs as research subjects in an RCT during a pandemic. Second, we demonstrated that recruiting PCPs in an RCT can be done electronically, within a reasonable timeframe, and relatively low cost. Nevertheless, our study is not without limitations. Due to the co-occurring COVID-19 pandemic resulting in high workload and stress among eligible PCPs, we were unable to gather feedback from eligible PCPs on how they perceived our predominantly electronic recruitment strategy. Relatedly, we did not conduct a baseline assessment of several 7R framework strategies, such as PCPs perception of the research team’s reputation for conducting rigorous research, which may have shed more light on the effectiveness of our recruitment efforts. Second, recruited PCPs came from a single health system in a Midwestern state, thus the generalizability of our findings may be limited. Third, we calculated costs per recruited PCPs using only the research coordinator’s salary and effort. We used this approach, consistent with other studies, because of difficulty accurately allocating percent effort of other team members to recruitment. Importantly, the research coordinator’s primary responsibility was recruitment, thus the coordinator’s efforts should adequately capture anticipated costs per recruited PCP. Furthermore, because our recruitment strategies were rolled out sequentially, some strategies may have led to fewer recruited PCPs because fewer eligible PCPs remained at the time of rollout. Finally, as our research team relied heavily on established relationships with primary care leadership and individual clinics, other teams without such connections may experience additional challenges recruiting PCPs electronically.
PCPs can be efficiently recruited as research subjects using primarily electronic communications, even during time of high workload and stress. Electronic recruitment was shown to be more efficient, both in time and costs. Clinic leader outreach and synchronous video presentations may be particularly useful recruitment strategies.
Availability of data and materials
The datasets generated and analyzed for the current study are available from the corresponding author on reasonable request.
Primary care provider
Randomized controlled trial
Electronic health record
Centers for Disease Control and Prevention
Wu J, Lewis ET, Barnett PG, Nevedal AL. Instant messaging: an innovative way to recruit primary care providers for qualitative research. J Gen Intern Med. 2020;35(5):1612–4.
Asch S, Connor SE, Hamilton EG, Fox SA. Problems in recruiting community-based physicians for health services research. J Gen Intern Med. 2000;15(8):591–9.
Johnston S, Liddy C, Hogg W, Donskov M, Russell G, Gyorfi-Dyke E. Barriers and facilitators to recruitment of physicians and practices for primary care health services research at one Centre. BMC Med Res Methodol. 2010;10(1):109.
Sahin D, Yaffe MJ, Sussman T, McCusker J. A mixed studies literature review of family physicians' participation in research. Fam Med. 2014;46(7):503–14.
Riis A, Jensen CE, Maindal HT, Bro F, Jensen MB. Recruitment of general practices: is a standardised approach helpful in the involvement of healthcare professionals in research? SAGE Open Med. 2016;4:2050312116662802.
Hysong SJ, Smitham KB, Knox M, Johnson K-E, SoRelle R, Haidet P. Recruiting clinical personnel as research participants: a framework for assessing feasibility. Implement Sci. 2013;8:125.
Hummers-Pradier E, Scheidt-Nave C, Martin H, Heinemann S, Kochen MM, Himmel W. Simply no time? Barriers to GPs’ participation in primary health care research. Fam Pract. 2008;25(2):105–12.
Rothwell PM. Subgroup analysis in randomised controlled trials: importance, indications, and interpretation. Lancet. 2005;365(9454):176–86.
McDonald AM, Knight RC, Campbell MK, et al. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7(1):9.
Fulda KG, Hahn KA, Young RA, et al. Recruiting practice-based research network (PBRN) physicians to be research participants: lessons learned from the North Texas (NorTex) needs assessment study. J Am Board Fam Med. 2011;24(5):610–5.
Solberg LI. Recruiting medical groups for research: relationships, reputation, requirements, rewards, reciprocity, resolution, and respect. Implement Sci. 2006;1:25.
Morgantini LA, Naha U, Wang H, et al. Factors contributing to healthcare professional burnout during the COVID-19 pandemic: a rapid turnaround global survey. PLoS One. 2020;15(9):e0238217.
Sharma M, Creutzfeldt CJ, Lewis A, et al. Health-care professionals' perceptions of critical care resource availability and factors associated with mental well-being during coronavirus disease 2019 (COVID-19): results from a US survey. Clin Infect Dis. 2021;72(10):e566–76.
Sathian B, Asim M, Banerjee I, et al. Impact of COVID-19 on clinical trials and clinical research: a systematic review. Nepal J Epidemiol. 2020;10(3):878–87.
Broyles LM, Rodriguez KL, Price PA, Bayliss NK, Sevick MA. Overcoming barriers to the recruitment of nurses as participants in health care research. Qual Health Res. 2011;21(12):1705–18.
Fagnan LJ, Walunas TL, Parchman ML, et al. Engaging primary care practices in studies of improvement: did you budget enough for practice recruitment? Ann Fam Med. 2018;16(Suppl 1):S72–9.
Harle CA, Apathy NC, Cook RL, et al. Information needs and requirements for decision support in primary care: an analysis of chronic pain care. AMIA Annu Symp Proc. 2018;2018:527–34.
Dowell D, Haegerich TM, Chou R. CDC guideline for prescribing opioids for chronic pain—United States, 2016. Jama; 2016;315(15):1624–45.
Indiana Primary Health Care Association. IPHCA Dashboard for Health Centers in Indiana. 2021. https://www.indianapca.org/resource/resource-link/dashboard-for-health-centers-in-indiana/. Accessed 21 May 2021.
Bower P, Brueton V, Gamble C, et al. Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014;15(1):399.
Chaudhari N, Ravi R, Gogtay NJ, Thatte UM. Recruitment and retention of the participants in clinical trials: challenges and solutions. Perspect Clin Res. 2020;11(2):64–9.
Goldman V, Dushkin A, Wexler DJ, et al. Effective recruitment for practice-based research: lessons from the REAL HEALTH-diabetes study. Contemp Clin Trials Commun. 2019;15:100374.
Ngune I, Jiwa M, Dadich A, Lotriet J, Sriram D. Effective recruitment strategies in primary care research: a systematic review. Qual Prim Care. 2012;20(2):115–23.
Research reported in this publication was entirely funded by the National Institute on Drug Abuse (NIDA) of the National Institutes of Health under Award Number R33DA046085. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. NIDA was not involved in the study design, data collection, analysis, interpretation of data, and manuscript development.
Ethics approval and consent to participate
All participating PCPs provided written informed consent at the time of trial enrollment. The study received institutional board approval at Indiana University and is registered with ClinicalTrials.gov (NCT04295135).
Consent for publication
Dr. Harle discloses past research grant funding for studies of information technology to support pain care to his institution from Security Risk Solutions, Inc. and the Agency for Healthcare Research and Quality. Dr. Harle also discloses personal fees from Indiana Health Information Exchange, personal fees from New York eHealth Collaborative, personal fees from RTI International, outside the submitted work. Dr. Mazurenko discloses past and current research grant funding for studies of information technology to support pain care to her institution from Security Risk Solutions, Inc. and the Agency for Healthcare Research and Quality. Dr. Adams is supported by National Institutes of Health funding through the NIH K08 EB022631, U24 NS115708, and R33 DA046085. The other authors have no conflicts to report.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Mazurenko, O., Sanner, L., Apathy, N.C. et al. Evaluation of electronic recruitment efforts of primary care providers as research subjects during the COVID-19 pandemic. BMC Prim. Care 23, 95 (2022). https://doi.org/10.1186/s12875-022-01705-y
- Primary care provider
- Randomized clinical trial
- Clinical decision support