Skip to main content
Download PDF

A 360 degree mixed-methods evaluation of a specialized COVID-19 outpatient clinic and remote patient monitoring program

BMC Primary Care volume 23, Article number: 151 (2022) Cite this article

Abstract

Background

Our goals are to quantify the impact on acute care utilization of a specialized COVID-19 clinic with an integrated remote patient monitoring program in an academic medical center and further examine these data with stakeholder perceptions of clinic effectiveness and acceptability.

Methods

A retrospective cohort was drawn from enrolled and unenrolled ambulatory patients who tested positive in May through September 2020 matched on age, presence of comorbidities and other factors. Qualitative semi-structured interviews with patients, frontline clinician, and administrators were analyzed in an inductive-deductive approach to identify key themes.

Results

Enrolled patients were more likely to be hospitalized than unenrolled patients (N = 11/137 in enrolled vs 2/126 unenrolled, p = .02), reflecting a higher admittance rate following emergency department (ED) events among the enrolled vs unenrolled, though this was not a significant difference (46% vs 25%, respectively, p = .32). Thirty-eight qualitative interviews conducted June to October 2020 revealed broad stakeholder belief in the clinic’s support of appropriate care escalation. Contrary to beliefs the clinic reduced inappropriate care utilization, no difference was seen between enrolled and unenrolled patients who presented to the ED and were not admitted (N = 10/137 in enrolled vs 8/126 unenrolled, p = .76). Administrators and providers described the clinic’s integral role in allowing health services to resume in other areas of the health system following an initial lockdown.

Conclusions

Acute care utilization and multi-stakeholder interviews suggest heightened outpatient observation through a specialized COVID-19 clinic and remote patient monitoring program may have contributed to an increase in appropriate acute care utilization. The clinic’s role securing safe reopening of health services systemwide was endorsed as a primary, if unmeasured, benefit.

Peer Review reports

Background

The COVID-19 pandemic dramatically increased the demand for telemedicine and telehealth services [1,2,3], including in outpatient settings to support patients with confirmed or suspected COVID-19 [4]. Several health systems have introduced remote patient monitoring (RPM) programs as a vital adjunct to in-person COVID-19 clinical care. Such programs seek to guide patients to the appropriate level of care, whilst minimizing unnecessary pathogen exposure to clinical staff and non-COVID-19 patients [5,6,7,8,9,10,11,12].

RPM directs patient data collected in non-clinical settings to clinical teams providing care oversight [13, 14]. It has traditionally been used to guide care for patients with chronic disease [14,15,16,17,18,19,20,21], and its impact may depend on the disease and monitoring approach [15, 22]. Outside of chronic disease, the use of RPM to manage short-term conditions has been primarily limited to post-operative care [14, 23]. Prior to COVID-19, its use in acute infectious illness was virtually non-existent.

COVID-19 expanded RPM possibilities from both the clinical [5,6,7,8,9, 24] and technical [25, 26] perspectives. Recent RPM efforts have combined digital and/or sensor technology (i.e., pulse oximetry) with clinical oversight to determine the need for care escalation. Evaluations of these efforts thus far are limited. Most notably, RPM enrollment reduced 30 day hospital readmissions in COVID-19 patients discharged from the hospital [11, 12]; similar findings were also noted for cancer patients with COVID-19 [27]. Another evaluation found an association between RPM enrollment and a reduced admittance rate following an ED visit [5]. Patient engagement and satisfaction with programs have also been documented [5,6,7].

This nascent body of work suggests that RPM programs are acceptable to patients and may decrease undesirable healthcare utilization, though additional evaluation is needed to understand how perceived benefits amongst stakeholders compare with actual care utilization and patient outcomes. Such analysis can inform the future direction of such programs, particularly given the non-trivial resources they require. We evaluated a specialized COVID-19 clinic with an integrated RPM program in an academic medical center using a mixed-methods approach. This approach included quantifying the program’s impact on acute care utilization and patient outcomes as well as qualitatively analyzing patient, provider, and administrator perspectives of the program.

Methods

Design

We conducted a convergent mixed methods [28] evaluation of a dedicated COVID-19 outpatient clinic at a large academic medical center, Stanford Health Care (Palo Alto, CA, USA). A quantitative retrospective cohort analysis matched patients enrolled and unenrolled in the clinic to understand the clinic’s impact on downstream emergency department (ED) utilization and hospitalization rates. Semi-structured interviews with patients, frontline clinicians, and administrative stakeholders and subsequent analysis were guided by Proctor et al. (2011) Outcomes for Implementation Research [29]. The Stanford Institutional Review Board approved the present retrospective evaluation and determined it did not meet the definition of human subjects research (Protocol #56054). Informed consent was obtained from all interview participants.

Specialized COVID-19 “CROWN” clinic with RPM

In April 2020, Stanford Health Care (SHC) launched a specialized COVID-19 clinic called CROWN (standing for Care and Respiratory Observation of patients With Novel coronavirus) to provide RPM to adult outpatients recently diagnosed with COVID-19 [30]; this clinic combined with RPM services entail the intervention that is the focus of this evaluation. Any positive COVID-19 laboratory test throughout the health system went into a central pool monitored by ED nurses. Patients with a positive COVID-19 test and fewer than 14 days of symptoms were offered enrollment in the program that included periodic check-ins by phone and/or video. The frequency and method of communication (phone or video) was based on a risk stratification tool developed by lead clinicians, which incorporated age, pre-existing conditions, clinical severity of illness and the clinical course of disease to place patients into low, medium and high risk categories (Additional file 1: Appendix A1). These tiers were used to guide the frequency of outreach through phone calls and video visits as well as the distribution of pulse oximeters to medium and high-risk patients (Additional file 1: Appendix A2). Clinicians at any point could designate a patient receive a higher level of care (e.g. shorter intervals between outreach), including escalation to in-person care at CROWN or the ED if needed. CROWN clinicians followed patients from the date of their enrollment to 14–21 days from first symptoms and provided patients a dedicated phone number to contact the clinic if concerns or questions arose. Biometric data was shared verbally by the patient in the course of each encounter and was not electronically transmitted. Further details regarding the risk stratification tool and RPM outreach protocol have been previously described [30].

Quantitative evaluation - creation of the matched retrospective cohort

We extracted electronic health record demographic and acute care utilization data of all patients who received a positive test result from a Stanford testing facility between May 1, 2020 and September 30, 2020. The time between the COVID-19 test result date and enrollment varied, with a median of 2 days. To ensure a similar duration of disease progression for the purposes of matching, a “pseudo enrollment” date defined as 2 days following a COVID-19 test result was assigned to the unenrolled comparator group. Patients were excluded if they were under age 18, lived more than 50 miles from Stanford Hospital or had a missing zip code, or if their positive test date was after September 2, 2020, to ensure complete follow-up of all patients; follow-up was defined as 28 days following the date of program enrollment (or pseudo-enrollment). Patients were also excluded if they were hospitalized in the 7 days preceding and inclusive of the enrollment (or pseudo-enrollment) date, a decision made to focus the evaluation on the intervention’s effectiveness at directing patients to the appropriate level of care early in their disease course.

Manual chart reviews were conducted by physician authors (with regular audits by author SV). These were used to identify the binary presence of comorbidities based on the COVID-19 risk stratification protocol (Additional file 1: Appendix A) in addition to emergency room and hospitalization encounters in the exclusion and observation periods that were not captured in the extracted dataset but were viewable within the electronic health record (Epic, Wisconsin, USA).

The matched cohort was finalized by matching each eligible enrolled patient to an unenrolled patient considering home distance from hospital (0–15, > 15–30, > 30–50 miles), health system affiliation (Stanford academic, Stanford non-academic made up of affiliated community practices, and unaffiliated), insurance type (private, Medicare, other) and race/ethnicity (Hispanic, White, Asian, other), age (closest match), and binary presence of relevant comorbidities (technical details in Additional file 1: Appendix B).

Quantitative data analysis

The outcomes of interest for the matched analysis were COVID-related unique patient and total ED encounters, unique patient and total hospital admissions (inclusive of an admission to observation status), and death in the 28 days following the date of program enrollment (or pseudo-enrollment). The ratio of a positive admission following an ED visit was reported for each group (defined as # inpatient admissions / (# inpatient admissions + # ED encounters)); these rates were statistically compared using a mixed effects logistic regression model with a random effect for patient. We also compared rates of COVID-related ED and hospital admissions by risk severity level for enrolled patients as an informal validation of the clinical risk assignment protocols. Analyses were conducted using R version 4.0.5 software and SAS version 9.4 software. Statistical significance was set at 0.05; p-values were adjusted for multiple comparisons using Tukey-Kramer correction where appropriate. Poisson regressions were used to determine statistical significance for counted events, including to validate the risk stratification tool where risk level was an explanatory variable and COVID-related ED and hospital admissions were outcomes.

We conducted a detailed chart review to extract clinical details of all COVID-19-related ED encounters and hospitalizations to provide a descriptive analysis only, as our evaluation was not powered to detect changed at this level. Data extracted included chief complaint, referring party (self, clinic, other), and suboptimal oxygenation (defined as a pulse oximeter reading of 95% or below at first clinical presentation) and are described as a percentage of a given event type. Given ever-shifting COVID-19 treatment guidelines at the time, for the purposes of analysis and discussion, we relied on a crude definition of “appropriate care” in which a hospital admission followed an ED encounter.

Qualitative stakeholder groups & data collection

We interviewed clinic stakeholders including enrolled patients, providers, and administrators. Interviews with unenrolled patients were not pursued given their experiences have been described elsewhere [31, 32], as well as our need to direct limited resources to better understand the clinic’s implications for the purposes of quality improvement. Enrolled patients were selected to include a diverse representation of gender, primary language spoken, and COVID-19 risk severity level from a list of those who completed clinic enrollment within the previous 3 weeks with no prior hospitalization for COVID-19. Patients were approached by telephone, given a verbal description of the evaluation, and gave their informed consent to participate in the interview. All providers who made up the core clinical team were contacted for interview, as were key administrators who were familiar with ambulatory clinical operations, acute care clinical operations, and finances related to clinic operations.

Semi-structured interview protocols were adapted for each stakeholder group based on its relevant perspective (Additional file 1: Appendix C). Interviews were conducted by either of two researchers (AL, SV), recorded and transcribed for analysis. Transcripts of interviews conducted in Spanish were first transcribed in Spanish and then translated to English by health system affiliates with professional translation training.

Qualitative analysis

We conducted thematic analysis of interviews using a combined inductive and deductive approach with separate codebooks developed for each stakeholder group. The deductive codes were derived from the topic guide content and key implementation outcomes [29]. All transcripts were imported into NVivo (released March 2020) for analysis. Two coders (SV, ASL) first independently coded one transcript then met to discuss and align coding practice. The process was repeated until the researchers agreed coding alignment was achieved for each stakeholder group. Coders continued to independently code transcripts and met regularly to review emerging themes and revise the codebook [33]. Coded data were then summarized into a thematic matrix with color-coding to identify positive and negative sentiments where rows represented individual participants and columns represented key themes; this visualization further supported the identification of intra and inter- group convergence and divergence of opinion [34].

Results

Quantitative analysis

Characteristics of enrolled and unenrolled patients

A total of 719 patients (89 Stanford academic, 20 Stanford non-academic, and 610 unaffiliated) enrolled in the clinic during the inclusion period. After applying exclusion and selection criteria for all populations, 137 eligible enrolled patients and 126 unenrolled matched patients were included in the analysis. There was no statistically significant difference between enrolled and unenrolled patients in terms of key demographic characteristics, suggesting successful matching of the groups on these variables (Table 1). There was also no statistically significant difference in enrolled and unenrolled patients who were hospitalized for non-COVID-19 related reasons, suggesting that these groups were reasonably balanced regarding comorbidities and their general hospital utilization (Additional file 1: Appendix D).

Table 1 Characteristics of Enrolled and Unenrolled Patients in the Specialized COVID-19 Clinic
Full size table

Covid-19 related ED encounters and hospitalizations

The enrolled and unenrolled groups had similar proportions of unique patients with a COVID-related ED-only encounter (7% enrolled vs 6% unenrolled; p = .76) as well as the total number of these encounters (13 enrolled and 9 unenrolled, p = .51) (Table 2). A larger proportion of enrolled patients had COVID-related hospitalizations, however, than unenrolled (8% vs 2%; p = .02); the enrolled patients also had a higher number of total hospitalizations (11 vs 3, p = .04). Chart review showed each admission event was preceded by an ED event (and direct admissions by primary care providers are not supported in this system), this reflects an increased admittance rate of 46% in the enrolled group (11 admittances of 24 possible events) versus 25% in the unenrolled group (3 admittances of 12 possible events). This could represent more appropriate use of the ED by enrolled patients, though this difference was not statistically significant (p = .32). Finally, there was no difference in length of stay during the observed hospitalizations (p = .67) or in mortality (p > .99).

Table 2 COVID-Related Events Within a 28-Day Observation Period by Enrollment Status in the Specialized COVID-19 Clinic
Full size table

Chart reviews revealed a total of 6 ED encounters for asymptomatic repeat COVID-19 testing, (4 of 13; 31% ED visits in the enrolled and 2 of 9; 22% in the unenrolled). Events in which patients self-referred to acute care and were not admitted were similar between enrolled and unenrolled patients (8 of 13; 62% vs 6 of 9; 67%, respectively). However, events in which patients self-referred to acute care and were ultimately admitted were slightly higher in the enrolled than in the unenrolled group (11 of 24; 46% vs 3 of 12; 33%, respectively). Finally, all admission events for unenrolled patients were associated with poor oxygen status (defined as a pulse oximeter reading of 95% or below) at first clinical presentation (3 of 3; 100%), whereas this proportion was slightly lower for enrolled patients (9 of 11; 82%).

Validation of COVID-19 risk severity clinic protocol

Most enrolled patients were determined to have low (39%) or medium (43%) COVID-19 Risk Severity based on the clinic protocols (Table 3). There was not a statistically significant difference between risk severity levels and COVID-19-related ED visits or admissions after adjusting for multiple comparisons. However, there was a statistically significant difference between high and low risk severities in rate of ED-only events (p = .05) and combined ED and hospital admission events (p < .01), thus suggesting the risk stratification tool (Additional file 1: Appendix A) holds some validity.

Table 3 Enrolled Patients COVID-related Events Within the 28-Day Observation Period by COVID-19 Risk Severity
Full size table

Interview findings

Characteristics of interviewed stakeholder groups

We conducted a total of 38 qualitative interviews June to October 2020 across the three stakeholder groups, including 21 patients, 9 (of 12 available) providers, and 8 (of 12 available) administrative stakeholders, 2 of whom had overlapping provider and administrative responsibilities and were therefore interviewed with both the provider and administrator protocols. Patient characteristics were predominantly female (67%), Hispanic/Latino (52%), Spanish speaking (52%) and were moderate risk based on clinical criteria (48%), though low and high risk patients were also represented (33, 19%, respectively). Reflecting the personnel serving in the CROWN clinic, providers were predominantly female (78%), physicians (56%) while administrators were also predominantly female (75%) and had diverse areas of operational focus (Table 4). Duration of interviews varied low to high (median) across stakeholder groups: 10–27 (19), 18–45 (25), 17–39 (24) minutes for patients, providers and administrators, respectively. The data below focuses on key themes related to care utilization that complement the quantitative analysis above, system considerations impacting the perceived benefits of the clinic, and clinic features impacting the acceptability of the clinic from patient and provider perspectives. We compare findings across methods to gain insight where applicable.

Table 4 Patient, Provider, and Administrative Stakeholder Characteristics
Full size table

Perceptions and utilization patterns suggest an increase in appropriate care escalation

Both providers and administrators believed the clinic played a crucial role in identifying patients who required care escalation but who may not have otherwise sought or been able to access this care. Providers felt this to be particularly important given the nature of the COVID-19 illness where low blood oxygen levels can be present without associated symptoms, so called “silent hypoxia”. Such cases were detected through symptom monitoring and pulse oximeter information, which was “… super helpful. We do a resting and an ambulatory pulse ox. And then if the O2 sat is worrisome, we either have the patient come to clinic or send them to the emergency room” (Provider 3).

Patients who had access to pulse oximetry overwhelmingly reported the devices were easy to operate, and they used them approximately 1–3 times a day. One patient explicitly described using the device to determine whether to seek care (though no patient interviewed reported the need for escalation of care). Patients expressed universally positive sentiment in having the pulse oximeter as a resource during their illness.

Another mechanism that may have increased appropriate care utilization was the interim step that an in-person evaluation in CROWN offered in place of the ED:

…I've said, ‘You need to go to the ER,’ and [patients] are like, ‘No, I'm not going to the ER.’ Then I say, ‘Well, if you refuse to go there, I can offer you the CROWN in person, but I might still tell you to go to the ER [emergency department] after I see you.’ They seem more willing to do that than to go to the ER.’ (Provider 2)

Some patients expressed concerns about possibly being contagious and appreciated the remote care and streamlined in-person care the clinic offered.

These perceptions were supported by the quantitative data described above in which enrolled patients had a significantly higher number of inpatient admissions as well as a higher admittance rate upon presenting to the ED. The lack of a significant difference in length of stay between enrolled and unenrolled patients does not support the notion that patients presented for care earlier because of the program. Clinic providers were a source of referral to ED evaluation in one third (8 of 24 referrals) of the encounters for enrolled patients, suggesting the clinic played a substantial role in escalation of care.

Perceptions of a reduction in inappropriate care escalation without corresponding quantitative evidence

Providers and administrators believed the clinic also played a strong role in redirecting patients away from inappropriate care, specifically in the ED. One administrator shared he would “panic” without this resource given it was the primary place to send patients with confirmed or suspected COVID-19 (Administrator 1). Another provider reported that keeping patients out of the ED had become an unexpected predominant role of the clinic:

We thought that we would be picking up hypoxia that people weren’t noticing and sending them to the ER. And instead, we’re almost doing the opposite where we have patients who may feel a little shortness of breath and then they get their [oxygen saturation] and it’s 99% [optimal] and that actually helps me keep people at home. (Provider 5)

Without the clinic, providers perceived that patients would be otherwise “completely lost” given restricted access for potentially contagious persons and the reported reluctance of PCPs and specialists to see these patients in person (Provider 2). Several patients corroborated these challenges in accessing primary care, stating their doctors were either slow to return their calls and/or were unable to physically see them in regular clinic: “…they [clinic] said no, that they could not attend to me unless it was an emergency” (Patient 10). CROWN was perceived by providers as offering “everything the patient needs in one place,” including imaging, labs, and other studies, thereby optimizing safety and efficiency (Provider 4).

Despite these beliefs and reported behaviors, the actual utilization data described above does not corroborate these beliefs, as there was no difference between enrolled and unenrolled ED-only utilization. Indeed, chart reviews suggested similar numbers between enrolled and unenrolled patients for an inappropriate ED presentation –asymptomatic repeat COVID-19 testing—despite provider efforts to educate patients during their enrollment.

Systems considerations impacting perceived value of the clinic

Several administrators and providers reported the clinic’s greatest impact was possibly its role in supporting access to health services for non-COVID patients. They reported that safety was increased throughout the system by directing potentially infected patients to a single site for non-acute services, thereby facilitating a gradual reopening of services following the initial lockdown:

Our ability to reopen all the sites…was dependent on the fact that we had a place that we could also appropriately case manage and track people with COVID …So the financial viability of the clinic itself can't be measured in the clinic financial performance, but the rest of organization’s capability to keep pace or open up quickly and stay open. (Administrator 7)

Administrators also reported challenges capturing reimbursement for the clinic due to structural challenges. Having been rapidly launched, the clinic lacked its own budget and instead operated under the umbrella of its neighboring urgent care clinic with “borrowed” resources (Admin 3). Many RPM activities, such as outreach by non-provider staff were not reimbursable, and when they were, the patient mix reportedly skewed towards the uninsured or underinsured, further limiting reimbursement.

Clinic features impacting acceptability of the specialized COVID-19 clinic

Patients and providers described how several clinic characteristics impacted their perceived acceptability of the clinic (Table 5). Patients and providers both expressed generally favorable perceptions related to improved access to care, benefits of patient education, benefits of concentrating COVID-19 expertise within a specialized clinic, support for mental health services, and use of pulse oximeters.

Table 5 Clinic Features Impacting Acceptability of the Specialized COVID-19 Clinic Across Patients and Providers
Full size table

Acceptability of other aspects was mixed. Patients overwhelmingly reported the frequency of outreach was appropriate, apart from one asymptomatic patient who felt outreach was too frequent. For providers, some debate existed as to the appropriate frequency of outreach, with a few providers questioning whether a proactive outreach approach was the optimal use of scarce resources, noting that many patients felt fine when they were contacted.

In addition, while most patients accepted virtual care via video or phone, noting community-wide shelter-in-place orders and their own need to isolate given their diagnosis, two participants expressed concerns that remote care could contribute to missed important clinical changes. Providers usually preferred video to phone so they could “eyeball” a patient to visually assess their clinical status (Provider 1). In addition, a few providers also regretted the system’s inability to provide home social services to patients in a time of significant need—an ED visit was sometimes the only solution when a patient could not adequately care for him- or herself.

Finally, acceptability was limited by the challenges associated with specialist care, particularly for patients with serious comorbid conditions (i.e., cancer, post-transplant). Patients wanted to know what their COVID-19 diagnosis meant for them in terms of their pre-existing condition; providers’ efforts to connect with these specialists on behalf of patients were sometimes fruitless. Particularly challenging for providers was convincing specialists that patients were no longer infectious after their 10-day quarantine and therefore qualified for in-person specialty care.

Discussion

The specialized COVID-19 clinic and RPM program was launched early in the COVID-19 pandemic to fulfill multiple stakeholder needs—primarily to promote access to non-emergency services for patients suffering from COVID-19 and secure patient and staff safety systemwide [30]. This convergent mixed methods analysis suggests stakeholder beliefs that the clinic reduced acute care utilization (ED and admission events) was not supported by the quantitative analysis. Rather, observed hospitalizations were actually higher in the enrolled group than the unenrolled group. Further, no difference was observed between groups for ED events that did not result in an admission. Despite these negative quantitative findings, qualitative data suggests healthcare access increased for patients with confirmed or suspected COVID-19 who were otherwise restricted from non-emergency care. Finally, the clinic’s unmeasured benefit supporting the safe reopening of health services systemwide was felt to be substantial.

These findings seem to contradict the reduction in acute care utilization previously seen following COVID-19 RPM efforts in which patients were enrolled following an ED encounter or hospitalization [11, 12]. Our analysis differed from these studies in that it focused on patients who were enrolled following an outpatient diagnosis. This difference, as well as variation in how RPM was implemented may contribute to our observation that hospitalization was higher in the enrolled patient group [22].

Such increase in the utilization of health services has been previously seen with increased healthcare access [35, 36]. Increased exposure to the health system through the COVID-19 clinic may have increased patient comfort with the health care system overall, thereby lowering patients’ perceived barrier to seeking a higher level of care. Though clinician perspectives presented here suggest increased utilization was often an appropriate escalation of care, we did find patients presenting to the ED for asymptomatic repeat COVID-19 testing in both the enrolled and unenrolled groups. This suggests increased opportunity for patient education regardless of enrollment status, and possibly at the time of their initial diagnosis.

At the time of the analysis, outpatient care for COVID-19 was also primarily supportive. Simply increasing the monitoring for clinical deterioration (rather than combining this effort with treatment) was perhaps unlikely in retrospect to change the clinical course of the disease. Instead, clinicians had more opportunity to identify worsening disease and therefore escalate patients to a higher level of care. Future work in this area should account for COVID-19 outpatient treatments that have become available since this analysis [37].

Other possible reasons for increased utilization in the enrolled group include increased patient psychological need for health services at a time of great vulnerability [38], ED physician bias to admit a patient sent in by another provider, and residual selection bias towards “sicker” patients in an enrolled population (i.e. patients with underlying disease not already accounted for in the analysis are more likely to opt into a monitoring program) [39].

The present work represents an example of a collaboration between researchers, clinicians and health system administrators to evaluate an ongoing initiative in order to inform future improvements and the direction of limited resources [40]. Evaluating the value of such a clinic from a mixed-methods, multi-stakeholder perspective is particularly important in this setting where typical value-oriented data (i.e., cost and outcomes) is missing or overshadowed by unmeasured benefits —for example, benefits from the gradual reopening of a health system following an initial lockdown where no comparator exists.

The present work is also notable in its examination of an important new use case for RPM: monitoring of acute infectious disease. All stakeholders largely believed that RPM added value in this clinical context, and patients and providers described how it shifted their care-seeking behavior and clinical decision-making, respectively. This novel use of RPM touches on growing trends to explore and expand “hospital-at-home” models in which patients who would otherwise meet inpatient criteria received care in the comfort of their own home, with the support of remote clinical experts [41,42,43]. Understanding the bridge between hospital-at-home and traditional RPM will be an important area for future research. Further, careful consideration should be given to which types of patients may benefit most from RPM given limited resources. For example, targeting clinic resources towards patients who are higher risk for adverse outcomes (given the link between increased risk and escalation of care demonstrated in Table 3), and/or with a lower level of health literacy may be more cost-effective than a uniform approach and is an area for future work.

Evaluation limitations include the retrospective case-control design in which patients had the option to opt into the program; we were also unable to verify through the data that each patient was equally offered enrollment following standard protocols. Our analysis therefore relies on successful matching across several variables. The absence of any statistically significant difference in these variables and acute care events unrelated to COVID-19 between these groups provides some reassurance against selection bias. Individuals without access to at least a phone were unreachable and therefore not enrolled, though prior work suggests this is a small population [44]. Manual chart review captured all ED encounters and hospitalizations that occurred within any health system in the area that used the predominant electronic health record system (Epic Systems), though uncaptured encounters outside this network were possible. We further note that diverse clinical circumstances involving a novel disease limited our ability to fully define “appropriate” care. Our assumption therefore focused on two extremes—an admission following an ED visit suggests that ED visit was likely appropriate, whereas an asymptomatic patient presenting to the ED for a repeat COVID-19 test was likely better served in an alternative setting. We recognize possible exceptions to these generalizations, including our inability to draw conclusions about cases between those extremes (i.e. value-added ED visits [21]). For these reasons, conclusions should be interpreted with caution.

Conclusion

Acute care utilization data and multi-stakeholder interviews suggest heightened outpatient observation through a specialized COVID-19 clinic and RPM program may have contributed to an increase in appropriate acute care utilization. A reduction in inappropriate care utilization was not seen, despite provider and administrator beliefs in this benefit. The clinic’s role securing safety systemwide, leading to a gradual reopening of health services systemwide was endorsed as a primary, if unmeasured, benefit. Additional evaluation is needed to understand the growing role of RPM in the novel support of acute infectious disease.

Availability of data and materials

The datasets generated and/or analyzed during the current evaluation are not publicly available due to institutional data use agreements but are available from the corresponding author on reasonable request.

Abbreviations

RPM:

Remote Patient Monitoring

ED:

Emergency Department

ER:

Emergency Room

SHC:

Stanford Health Care

CROWN:

Care and Respiratory Observation of patients With Novel coronavirus

References

  1. Mann DM, Chen J, Chunara R, Testa PA, Nov O. COVID-19 transforms health care through telemedicine: evidence from the field. J Am Med Inform Assoc. 2020. https://doi.org/10.1093/jamia/ocaa072.

  2. Hollander JE, Carr BG. Virtually perfect? Telemedicine for Covid-19. New England J. 2020. https://doi.org/10.1056/NEJMp2003539.

  3. Webster P. Virtual health care in the era of COVID-19. Lancet. 2020;395(10231):1180–1. https://doi.org/10.1016/S0140-6736(20)30818-7.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Accorsi TAD, Amicis KD, Brígido ARD, et al. Assessment of patients with acute respiratory symptoms during the COVID-19 pandemic by Telemedicine: clinical features and impact on referral. Einstein (Sao Paulo). 2020;18:eAO6106. https://doi.org/10.31744/einstein_journal/2020AO6106

  5. Morgan A, Blachandran M, Do D, et al. Remote monitoring of patients with Covid-19: design, implementation, and outcomes of the first 3,000 patients in COVID watch. NEJM Catalyst Published online July 21, 2020.

  6. Annis T, Pleasants S, Hultman G, et al. Rapid implementation of a COVID-19 remote patient monitoring program. J Am Med Inform Assoc. 2020;27(8):1326–30. https://doi.org/10.1093/jamia/ocaa097.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Aalam AA, Hood C, Donelan C, Rutenberg A, Kane EM, Sikka N. Remote patient monitoring for ED discharges in the COVID-19 pandemic. Emerg Med J. 2021;38(3):229–31. https://doi.org/10.1136/emermed-2020-210022.

    Article  PubMed  Google Scholar 

  8. Tabacof L, Kellner C, Breyman E, et al. Remote Patient Monitoring for Home Management of Coronavirus Disease 2019 in New York: A Cross-Sectional Observational Study. Telemedicine e-Health. 2020:tmj.2020.0339. https://doi.org/10.1089/tmj.2020.0339

  9. Medina M, Babiuch C, Card M, et al. Home monitoring for COVID-19. Cleve Clin J Med. 2020. https://doi.org/10.3949/ccjm.87a.ccc028.

  10. Hutchings OR, Dearing C, Jagers D, et al. Virtual health Care for Community Management of patients with COVID-19 in Australia: observational cohort study. J Med Internet Res. 2021;23(3):e21064. https://doi.org/10.2196/21064.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Gordon WJ, Henderson D, DeSharone A, et al. Remote patient monitoring program for hospital discharged COVID-19 patients. Appl Clin Inform. 2020;11(05):792–801. https://doi.org/10.1055/s-0040-1721039.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Misra-Hebert AD, Ji X, Jehi L, et al. COVID-19 home monitoring after diagnosis and health care utilization in an integrated health system. JAMA Health Forum. 2021;2(5):e210333. https://doi.org/10.1001/jamahealthforum.2021.0333.

    Article  PubMed Central  Google Scholar 

  13. Vegesna A, Tran M, Angelaccio M, Arcona S. Remote patient monitoring via non-invasive digital technologies: a systematic review. Telemed e-Health. 2017;23(1):3–17. https://doi.org/10.1089/tmj.2016.0051.

    Article  Google Scholar 

  14. Farias FAC de, Dagostini CM, Bicca Y de A, Falavigna VF, Falavigna A. remote patient monitoring: a systematic review. Telemed e-Health 2020;26(5):576–583. https://doi.org/10.1089/tmj.2019.0066.

  15. Ong MK, Romano PS, Edgington S, et al. Effectiveness of remote patient monitoring after discharge of hospitalized patients with heart failure: the better effectiveness after transition–heart failure (BEAT-HF) randomized clinical trial. JAMA Intern Med. 2016;176(3):310. https://doi.org/10.1001/jamainternmed.2015.7712.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Lee PA, Greenfield G, Pappas Y. The impact of telehealth remote patient monitoring on glycemic control in type 2 diabetes: a systematic review and meta-analysis of systematic reviews of randomised controlled trials. BMC Health Serv Res. 2018;18(1):495. https://doi.org/10.1186/s12913-018-3274-8.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Block VAJ, Pitsch E, Tahir P, Cree BAC, Allen DD, Gelfand JM. Remote physical activity monitoring in neurological disease: a systematic review. PLoS One. 2016;11(4):e0154335. https://doi.org/10.1371/journal.pone.0154335.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Kitsiou S, Paré G, Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. J Med Internet Res. 2015;17(3):e63. https://doi.org/10.2196/jmir.4174.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Noah B, Keller MS, Mosadeghi S, et al. Impact of remote patient monitoring on clinical outcomes: an updated meta-analysis of randomized controlled trials. NPJ Digit Med. 2018;1:20172. https://doi.org/10.1038/s41746-017-0002-4.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Peretz D, Arnaert A, Ponzoni NN. Determining the cost of implementing and operating a remote patient monitoring programme for the elderly with chronic conditions: a systematic review of economic evaluations. J Telemed Telecare. 2018;24(1):13–21. https://doi.org/10.1177/1357633X16669239.

    Article  PubMed  Google Scholar 

  21. Pekmezaris R, Tortez L, Williams M, et al. Home Telemonitoring in heart failure: a systematic review and Meta-analysis. Health Aff. 2018;37(12):1983–9. https://doi.org/10.1377/hlthaff.2018.05087.

    Article  Google Scholar 

  22. Taylor ML, Thomas EE, Snoswell CL, Smith AC, Caffery LJ. Does remote patient monitoring reduce acute care use? A systematic review. BMJ Open. 2021;11(3):e040232. https://doi.org/10.1136/bmjopen-2020-040232.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Breteler MJM, Numan L, Ruurda JP, et al. Wireless remote home monitoring of vital signs in patients discharged early after Esophagectomy: observational feasibility study. JMIR Perioper Med. 2020;3(2):e21705. https://doi.org/10.2196/21705.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Casale PN, Vyavahare M, Coyne S, et al. The promise of remote patient monitoring: lessons learned during the COVID-19 surge in new York City. Am J Med Qual. 2021;36(3):139–44. https://doi.org/10.1097/01.JMQ.0000741968.61211.2b.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Rajbhandary PL, Nallathambi G. Feasibility of Continuous Monitoring of Core Body Temperature Using Chest-worn Patch Sensor. In. 42nd annual international conference of the IEEE engineering in Medicine & Biology Society (EMBC). IEEE. 2020;2020:4652–5. https://doi.org/10.1109/EMBC44109.2020.9175579.

    Article  Google Scholar 

  26. Schinköthe T, Gabri MR, Mitterer M, et al. A web- and app-based connected care solution for COVID-19 in- and outpatient care: qualitative study and application development. JMIR Public Health Surveill. 2020;6(2):e19033. https://doi.org/10.2196/19033.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Pritchett JC, Borah BJ, Desai AP, et al. Association of a Remote Patient Monitoring (RPM) Program With Reduced Hospitalizations in Cancer Patients With COVID-19. JCO Oncology Practice. 2021:OP.21.00307. https://doi.org/10.1200/OP.21.00307

  28. Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs-principles and practices. Health Serv Res. 2013;48(6 Pt 2):2134–56. https://doi.org/10.1111/1475-6773.12117.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health Ment Health Serv Res. 2011;38(2):65–76. https://doi.org/10.1007/s10488-010-0319-7.

    Article  Google Scholar 

  30. Artandi M, Barman L, Srinivasan M, et al. A specialized acute COVID-19 outpatient clinic at an academic medical center. Am J Med Qual. 2021. https://doi.org/10.1097/JMQ.0000000000000006.

  31. Zhang H, Xie F, Yang B, Zhao F, Wang C, Chen X. Psychological experience of COVID-19 patients: A systematic review and qualitative meta-synthesis. Am J Infect Control. 2022:S0196–6553(22)00058-X. https://doi.org/10.1016/j.ajic.2022.01.023

  32. Aliyu S, Travers JL, Norful AA, Clarke M, Schroeder K. The lived experience of being diagnosed with COVID-19 among black patients: a qualitative study. J Patient Exp. 2021;8:2374373521996963. https://doi.org/10.1177/2374373521996963.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Miles MB, Huberman AM, Saldana J. Qualitative data analysis. 4th ed. Los Angeles: Sage; 2019.

    Google Scholar 

  34. Averill JB. Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qual Health Res. 2002;12(6):855–66. https://doi.org/10.1177/104973230201200611.

    Article  PubMed  Google Scholar 

  35. Wherry LR, Miller S. Early coverage, access, utilization, and health effects associated with the affordable care act Medicaid expansions: a quasi-experimental study. Ann Intern Med. 2016;164(12):795–803. https://doi.org/10.7326/M15-2234.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Kondo A, Shigeoka H. Effects of universal health insurance on health care utilization, and supply-side responses: evidence from Japan. J Public Econ. 2013;99:1–23. https://doi.org/10.1016/j.jpubeco.2012.12.004.

    Article  Google Scholar 

  37. Coronavirus Disease 2019 (COVID-19) Treatment Guidelines. National Institutes of Health. Published April 1, 2022. Accessed 7 Apr 2022. www.covid19treatmentguidelines.nih.gov

  38. Becerra MB. Factors associated with increased healthcare utilization among adults with asthma. J Asthma. 2017;54(4):376–82. https://doi.org/10.1080/02770903.2016.1218017.

    Article  PubMed  Google Scholar 

  39. Rafiq M, Keel G, Mazzocato P, et al. Extreme consumers of health care: patterns of care utilization in patients with multiple chronic conditions admitted to a novel integrated clinic. J Multidiscip Healthc. 2019;12:1075–83. https://doi.org/10.2147/JMDH.S214770.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Enticott J, Johnson A, Teede H. Learning health systems using data to drive healthcare improvement and impact: a systematic review. BMC Health Serv Res. 2021;21(1):200. https://doi.org/10.1186/s12913-021-06215-8.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Levine DM, Ouchi K, Blanchfield B, et al. Hospital-level Care at Home for acutely ill adults: a randomized controlled trial. Ann Intern Med. 2020;172(2):77–85. https://doi.org/10.7326/M19-0600.

    Article  PubMed  Google Scholar 

  42. Arsenault-Lapierre G, Henein M, Gaid D, Le Berre M, Gore G, Vedel I. Hospital-at-home interventions vs in-hospital stay for patients with chronic disease who present to the emergency department: a systematic review and Meta-analysis. JAMA Netw Open. 2021;4(6):e2111568. https://doi.org/10.1001/jamanetworkopen.2021.11568.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Leong MQ, Lim CW, Lai YF. Comparison of hospital-at-home models: a systematic review of reviews. BMJ Open. 2021;11(1):e043285. https://doi.org/10.1136/bmjopen-2020-043285.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Raven MC, Kaplan LM, Rosenberg M, Tieu L, Guzman D, Kushel M. Mobile phone, computer, and internet use among older homeless adults: results from the HOPE HOME cohort study. JMIR Mhealth Uhealth. 2018;6(12):e10049. https://doi.org/10.2196/10049.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgments

The authors would like to acknowledge Dr. Steven Asch, Dr. Kari Nadeau, and Dr. Megan Mahoney for their support of this project.

Funding

This project received direct support from the Sean N. Parker Allergy Center for Allergy and Asthma Research at Stanford University, the CROWN Foundation, and the Sunshine Foundation and indirectly by the Stanford-Intermountain Fellowship in Primary Care, Population Health and Delivery Science (author SV), Stanford Healthcare and the Stanford School of Medicine.

Author information

Authors and Affiliations

  1. Department of Medicine, Stanford University School of Medicine, 1265 Welch Rd, Stanford, CA, 94305, USA

    Stacie Vilendrer, Anna Lestoquoy, Maja Artandi, Linda Barman, Kendell Cannon, Donn W. Garvert, Douglas Halket, Laura M. Holdsworth, Sara Singer, Laura Vaughan & Marcy Winget

Authors
  1. Stacie Vilendrer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anna Lestoquoy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maja Artandi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Linda Barman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kendell Cannon
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Donn W. Garvert
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Douglas Halket
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Laura M. Holdsworth
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Sara Singer
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Laura Vaughan
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Marcy Winget
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Authors S.V., M.W., L.V. conceived of the presented idea. Regarding qualitative data, authors L.H., A.L., S.S., and S.V. contributed to the design of the methods and analysis while authors A.L. and S.V. contributed to data collection. Regarding quantitative data, authors M.W., D.G., L.V., S.V. contributed to the design of the methods and analysis, while authors M.A., L.B., K.C., D.G., D.H., L.V., and S.V. contributed to data collection. All authors contributed to the interpretation of results and contributed to the final manuscript, with the initial draft completed by author S.V. The author(s) read and approved the final manuscript.

Authors’ information

Authors M.W. and S.V. are the Director and Medical Director, respectively, of the Evaluation Sciences Unit at Stanford Medicine, a multidisciplinary group of clinician and non-clinician doctorate and masters-trained faculty and staff with expertise in implementation and evaluation sciences, health services research, epidemiology, and biostatistics.

Corresponding author

Correspondence to Stacie Vilendrer.

Ethics declarations

Ethics approval and consent to participate

The Stanford Institutional Review Board approved the present retrospective evaluation and determined this evaluation to be quality improvement (Protocol #56054). Informed consent was obtained from all interview participants and from legal guardians of deceased patients for the present retrospective evaluation. All methods were performed in accordance with relevant guidelines and regulations.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vilendrer, S., Lestoquoy, A., Artandi, M. et al. A 360 degree mixed-methods evaluation of a specialized COVID-19 outpatient clinic and remote patient monitoring program. BMC Prim. Care 23, 151 (2022). https://doi.org/10.1186/s12875-022-01734-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12875-022-01734-7

Keywords

BMC Primary Care

ISSN: 2731-4553

Contact us