Nudging General Practitioners to explore suicidal thoughts among depressed patients
BMC Primary Care volume 24, Article number: 88 (2023)
While frank discussion of suicidal thoughts in patients with depression is important for the prevention of suicide, suicide exploration of General Practitioners (GPs) is suboptimal. This study aimed to assess whether an intervention that prompts pop-up screens nudges GPs to more frequently explore suicidal thoughts over the course of two years.
From January 2017 to December 2018, the intervention was incorporated in the information system of the Dutch general practice sentinel network. New registration of an episode of depression triggered a pop-up screen referring to a questionnaire about GPs’ behaviour with regard to exploring suicidal thoughts. In two years, 625 questionnaires were completed by GPs and analysed using multilevel logistic regression analyses.
Compared to the first year, GPs were 50% more likely to explore suicidal thoughts among patients in the second year (OR 1.48; 95%CI 1.01–2.16). When adjusting for patients’ gender and age we found that the effect of the pop-up screens disappeared (OR 1.33; 95% CI 0.90–1.97). Suicide exploration occurred less frequently in women than in men (OR 0.64; 95% CI 0.43–0.98) and in older compared to younger patients (OR 0.97; 95% CI 0.96–0.98 per year older). In addition, 26% of variation in suicide exploration was because of differences in general practice. There was no evidence that general practices developed differently over time.
Although low cost and easy to administer, the pop-up system was not effective in nudging GPs to explore suicidality more frequently. We encourage studies to test whether implementing these nudges as part of a multifaceted approach will lead to a stronger effect. Moreover, we recommend researchers to include more variables, such as work experience or previous mental health training, to better understand the effects of the intervention on GPs’ behaviour.
Worldwide, over 700.000 people die every year as a result of suicide and approximately 20 times as many attempt suicide [1, 2]. In addition, an estimated 8% of people experience suicidal thoughts at some point in their live . Suicidal thoughts are an important risk factor for suicidal acts; over a quarter of people who experiences these thoughts attempts suicide later in life . Early recognition and intervention of patients at risk for suicide is therefore crucial.
In countries with a primary care network, such as the Netherlands, General practitioners (GPs) are the core of the health care system. They function as gatekeepers in referring patients to specialized care and often have enduring relationships with patients . Therefore, GPs have a large exposure to patients who are potentially at risk for suicide. In fact, people who died by suicide were more often in contact with a GP than with mental health services prior to death . A Dutch study reported that in the month before their suicide or attempted suicide, about half of the patients had consulted a GP . Supporting primary care is therefore seen as one of the most effective elements of suicide prevention strategies [8,9,10].
In primary care, suicide prevention practices are mostly delivered by physicians, and their behaviour serves as proxy for quality of care. Improving the quality of care often requires physicians to change behaviour in accordance with evidence-based practice or clinical guidelines . Various clinical guidelines recommend assessing the presence of suicidal thoughts among patients with an elevated risk, such as patients diagnosed with depression [12, 13]. Regardless, in a previous study we found that assessment of suicidal thoughts occurs in about two third of depression-related consultations . Other studies have also shown that suicide exploration rates in primary care are suboptimal [15,16,17].
Many strategies have been developed to influence professional behaviour, although most consist of providing education or training [18, 19]. Suicide prevention interventions in primary care also mostly rely on the provision of education and training . Although widely recommended, most studies were only able to report short-term effects on confidence, knowledge or skills, and fail to report long-term effects on behaviour change [21, 22]. Changing behaviour is complex and often requires more than influencing knowledge or skills . An important overview study of systematic reviews on changing provider behaviour also emphasized that education provides small and short-term effects. More passive methods, such as the delivery of information and materials, are only effective for creating awareness. Issuing reminders was the most promising individual intervention to substantially change medical practice, especially for the provision of preventive care . Besides, developing a reminder system is a relatively inexpensive intervention that is easy to implement . Issuing reminders may thus be a potential strategy to change provider behaviour and increase suicide exploration rates in general practice. To our knowledge, this has not been tested before.
This study aims to assess whether a pop-up reminder system nudges GPs to more frequently explore suicidal thoughts over the course of two years. We implemented a pop-up intervention in GPs’ Electronic Health Record (EHR) to alert GPs to explore suicide risk in patients consulting for depression. Because of the great variability in suicide exploration of GPs we found in a previous study , we also tested if general practices, as hypothesized, show a different development over time.
Design and setting
This pre-post study was performed among the network of Dutch Sentinel general practices of the Nivel Primary Care database . The sentinel network consists of approximately 40 general practices, which provide in-depth health care information on illnesses and procedures that cannot be obtained from electronic medical records. In 2018, the network of sentinel general practices had almost 130.000 registered patients and was representative in terms of patients’ age, gender, geographical distribution, and population density .
We created an automated pop-up screen that was implemented in GPs’ EHR. The pop-up screen was activated after recording a new episode of depression during a consultation. This pop-up screen referred to a questionnaire designed to gather data about GPs’ suicide exploration behaviour, described previously . Within this infrastructure, it was not possible to create a control group because the pop-up functioned as both intervention and measurement instrument. Effect of the pop-up screen was therefore only assessed in relation to consultation year.
Instrument and measures
The pop-up screen itself included some control questions about the consultation to make sure it concerned (1) a regular patient of the GP, (2) a patient with depression, (3) a new episode of depression, and (4) a face-to-face consultation. If these were answered affirmatively, the follow-up questionnaire was activated. GPs could access the questionnaire straight away or complete it at a later moment. The full questionnaire included 19 items, but not all questions were applicable during all consultations. On average, it took 66 seconds to complete the questionnaire. Our dichotomous main outcome measure ‘suicide exploration’ was assessed with the question: “have you asked the patient if he/she experiences suicidal thoughts?”.
Data was collected from January 2017 until December 2018. General practices that did not provide data over the full study period were excluded, yielding a total of 35 general practices. The present study is based on consultations with patients who were diagnosed with a new episode of depression.
We analysed our data using multilevel logistic regression techniques. The first model only includes year of consultation and was adjusted for clustered data of patients within general practices (two level structure). Since we found in a previous study that suicide exploration occurred more frequently in male and younger patients , we created a new model where we adjusted for these variables. The Intraclass Correlation Coefficient (ICC) is a measure to calculate the between group variance as proportion of the total variance and is used to estimate variation between general practices . We added a random slope for consultation year to the model and used the likelihood ratio test to assess whether the general practices showed different development over time. We computed and plotted the random effects to specify the variation between general practices.
Table 1 presents the descriptive statistics per year and in total. In total, we included 625 completed questionnaires about consultations with patients consulting for a new episode of depression. The majority (60%) were women, mean age of the patients was 49.9 years old (sd 19.0). Overall, in 69% of the consultations, suicidal feelings were explored and in 45% of those, patients reported suicidal feelings.
Table 2 shows the results of the multilevel analyses with a random intercept on general practice level. The first model shows the influence of contact year on suicide exploration while adjusted for nested data of consultations within general practices. This model shows that the number of consultations during which suicide was explored was significantly higher in 2018 compared to 2017 (OR 1.48; 95%CI 1.01–2.16). The ICC that was calculated for this model is 0.24, meaning that 24% of the variation in suicide exploration is explained by general practice.
In the second model, which is adjusted for patients’ gender and age, the effect of consultation year on suicide exploration of GPs attenuated (OR 1.33; 95%CI 0.90–1.97). Female gender and older age are both negatively correlated with suicide exploration (respectively OR 0.64; 95% CI 0.43–0.98 and OR 0.97; 95% CI 0.96–0.98 per year older). The ICC for this model is 0.26, indicating that 26% of the variation in suicide exploration among patients is explained by general practices. We also created a model with a random slope for consultation year to assess whether suicide exploration of general practices developed differently over time. According to the log likelihood ratio test, this was not the case (X2 = 1.212, P = 0.545). This model is therefore not presented here.
In Fig. 1 the random effects (intercepts) of general practices are plotted to show how suicide exploration differs per general practice. The lowest value represents a general practice where suicidality was explored in 1/12 depressed patients and the highest value represents a general practice where this was explored in 17/17 patients.
This study shows that patient-specific pop-up reminders were not effective in nudging GPs to more frequently explore suicidal thoughts over a period of two years. GPs were 50% more likely to explore suicidal thoughts in the second year compared to the first, however, this effect disappeared after adjusting for patients’ gender and age. Though there was considerable practice variation in suicide exploration, general practices showed no different development over time.
Issuing reminders effectively changed professional behaviour in previous studies across a range of settings [23, 24, 28, 29]. Shea at al. reported that issuing reminders was effective for improving preventive services with 77% overall. However, not all preventive services were improved; screening for cervical cancer, for instance, was not. The authors speculate this is related to contextual factors, specifically the time it takes to perform a pelvic examination and patients’ resistance for this procedure . The reason why we did not find evidence for the effectiveness of the reminder system in the present study, may be because of the suicide prevention context, which is perceived difficult and complex by many primary care professionals [31,32,33]. In order for interventions to effectively change professionals’ behaviour, it is important to adjust them to the context and consider barriers that are in place. For suicide prevention, important barriers are lack of knowledge, skills and/or confidence, lack of time, and limited access to mental health care [31,32,33,34,35,36,37].
Most interventions to change suicide prevention practices of professionals include providing education or training [19, 20]. This addresses the individual barriers lack of knowledge, skills and/or confidence. However, in order to effectuate true behaviour change, the environmental barriers should be addressed too. These are much harder to influence because they often require changes on system level. Effective interventions in primary care that address these include the institution of specialized nurses or other health care professionals, organizational changes to increase collaboration within primary care professionals, or collaborative or shared care practices . Not all these interventions have been specifically evaluated for suicide prevention. The mental health support staff was introduced in primary care about a decade ago. This positively influenced suicide prevention practices and was highly valued by GPs . A collaborative care model for suicide prevention was recently implemented and tested. Although we are awaiting the long-term results, short term results are promising and stakeholders especially valued the chain of care element to support collaboration .
The Behaviour Change Wheel (BCW) is a contemporary model for behaviour change that can be used for understanding and influencing behaviour in context. In the core of the wheel is the COM-B system, containing the three constructs of behaviour (B): Capability, Opportunity and Motivation. Surrounding the core are nine intervention functions, such as education, training, enablement, and environmental restructuring. These can be used to address deficits in one or more of these behaviour constructs. Education and training influence professionals’ capability and reflective motivation. Providing pop-ups in the EHR system to remind clinicians to engage in a certain behaviour is a type of environmental restructuring. Prompting these questions restructures the physical context, and influences the opportunity and automatic motivation conditions of clinicians’ behaviour . This increases the chance that newly adopted behaviour is structurally applied. Combining these with other interventions to overcome the environmental barriers for suicide prevention in primary care modifies behaviour more sustainably. This is in line with recommendations from other studies, arguing that multifaceted interventions targeting various barriers and influencing multiple behaviour constructs simultaneously are more likely to effectuate behaviour change and improve patient-level outcomes [18, 19, 23].
Strengths & limitations
The naturalistic design of this study was a major strength. We developed a feature that was inexpensive and easy to incorporate in the GPs’ natural workflow. Especially for GPs this is important, because they are known to have restricted time, so developing interventions that add to their high workload will only increase their burden further. The fact that we collected over 600 completed questionnaires in two years shows we were able to adapt to their workflow and create engagement among these GPs.
However, the design also caused some important limitations. First, it was not possible to compare the effects of the intervention group with a control group. Therefore, we cannot rule out whether any of the results are influenced by other extraneous variables. Second, only a limited number of variables were available for this study. Variables we expected to be of influence, such as work experience or previous mental health training , are missing. This limited our possibilities to adjust for confounding or effect modification. Finally, implementing pop-up reminders in the EHR system has become more prevalent in clinical practice. Even to the point that it may result in ‘alert fatigue’, causing health care professionals to ignore alerts due to the overload of prompts and reminders . Unfortunately, this data was not available so we could not take it into consideration.
We recommend researchers to test whether implementing these nudges as part of a multifaceted approach will lead to a stronger effect on GPs’ suicide exploration. Further, we recommend adopting a design that allows for comparison with a control group and to include more GP- en general practice variables to develop better understanding of the effect of the reminder system. Previous studies among the sentinel practices have indicated some clinical factors that are associated with suicide attempts and suicides, such as high consultation frequency and other psychological complaints or disorders . Furthermore, suicidal thoughts should also be assessed during later consultations concerning depression, and, preferably, during consultations concerning depressive mood or other psychological complaints. Future studies are therefore recommended to broaden their scope and include more illnesses and complaints. Finally, these insights and insights about sociodemographic variables should be used to improve the accuracy of the reminders and determine whether this affects the effectiveness of the pop-up system.
Developing a pop-up intervention and implementing it in the GPs’ information system is a low-cost and easy to administer intervention. Unfortunately, we were not able to report a robust effect of the intervention on GPs’ suicide exploration behaviour. We encourage studies to test whether implementing these nudges as part of a multifaceted approach to improve suicide prevention practices of GPs is more effective. In addition, we recommend researchers to include more GP- and general practice variables to better understand the effect of the intervention on GPs’ behaviour.
The datasets used during the current study are available from the corresponding author upon reasonable request.
Behavioural Change Wheel
Electronic Health Record
Intraclass correlation coefficient
World Health Organization. LIVE LIFE. An implementation guide for suicide prevention in countries. 2021 Jun.
World Health Organization. Preventing suicide: A global imperative. Geneva, Switzerland; 2014.
Nock MK, Borges G, Bromet EJ, Alonso J, Angermeyer M, Beautrais A, et al. Cross-national prevalence and risk factors for suicidal ideation, plans and attempts. Br J Psychiatry. 2008;192(2):98–105
ten Have M, van Dorsselaer S, Tuithof M, de Graaf R. Nieuwe gegevens over suïcidaliteit in de bevolking: Resultaten van de ’Netherlands Mental Health Survey and Incidence Study-2 (NEMESIS-2). Utrecht; 2011.
Kroneman M, Boerma W, Van Den Berg M, Groenewegen P, De Jong J, Van Ginneken E. The Netherlands: Health system review. Health Syst Transit. 2016;18(2):1–239.
Stene-Larsen K, Reneflot A. Contact with primary and mental health care prior to suicide: a systematic review of the literature from 2000 to 2017. Scand J Public Health. 2019;47:9–17.
de Beurs DP, Hooiveld M, Kerkhof AJFM, Korevaar JC, Donker GA. Trends in suicidal behaviour in Dutch general practice 1983–2013: a retrospective observational study.BMJ Open. 2016 May10;6(5):e010868.
Krysinska K, Batterham PJ, Tye M, Shand F, Calear AL, Cockayne N, et al. Best strategies for reducing the suicide rate in Australia. Aust N Z J Psychiatry. 2016;24(4):50.
Mann JJ, Apter A, Bertolote J, Beautrais A, Currier D, Haas A, et al. Suicide prevention strategies: a systematic review. JAMA. 2005;294:2064–74.
van der Feltz-Cornelis CM, Sarchiapone M, Postuvan V, Volker D, Roskar S, Grum AT et al. Best practice elements of multilevel suicide prevention strategies: A review of systematic reviews. Vol. 32, Crisis. 2011. p. 319–33.
Shin HD, Cassidy C, Weeks LE, Campbell LA, Rothfus MA, Curran J. Interventions to change clinicians’ behavior in relation to suicide prevention care in the emergency department: a scoping review protocol. JBI Evid Synth. 2021 Aug;19(8):2014–23.
van Hemert AM, Kerkhof AJFM, de Keijser J, Verwey B, van Boven C, Hummelen JW, et al. Multidisciplinaire richtlijn diagnostiek en behandeling van suïcidaal gedrag. Utrecht: De Tijdstroom; 2012. pp. 1–454.
National Institute for Health and Care Excellence. Depression in Adults: Treatment and Management. NICE Guideline [Internet]. 2022 [cited 2022 Oct 18]. Available from: www.nice.org.uk/guidance/ng222
Elzinga E, Gilissen R, Donker GA, Beekman ATF, de Beurs DP. Discussing suicidality with depressed patients: An observational study in Dutch sentinel general practices.BMJ Open. 2019;9(4).
Vannoy SD, Robins LS. Suicide-related discussions with depressed primary care patients in the USA: Gender and quality gaps. A mixed methods analysis. BMJ Open. 2011 Jan 1;1(2):e000198.
Verger P, Brabis PA, Kovess V, Lovell A, Sebbah R, Villani P, et al. Determinants of early identification of suicidal ideation in patients treated with antidepressants or anxiolytics in general practice: A multilevel analysis. J Affect Disord. 2007;99(1–3):253–7.
Feldman MD, Franks P, Duberstein PR, Vannoy S, Epstein R, Kravitz RL. Let’s not talk about it: Suicide inquiry in primary care. Ann Fam Med. 2007;5(5):412–8.
Chauhan BF, Jeyaraman M, Mann AS, Lys J, Skidmore B, Sibley KM, et al. Behavior change interventions and policies influencing primary healthcare professionals’ practice-an overview of reviews. Implementation Sci. 2017;12(1).
Shin HD, Cassidy C, Weeks LE, Campbell LA, Drake EK, Wong H, et al. Interventions to change clinicians’ behavior related to suicide prevention care in the emergency department: a scoping review. JBI Evid Synth. 2022;20(3):788–846.
Milner A, Witt K, Pirkis J, Hetrick S, Robinson J, Currier D, et al. The effectiveness of suicide prevention delivered by GPs: A systematic review and meta-analysis. J Affect Disord. 2017;210:294–302
Morriss R, Gask L, Battersby L, Francheschini A, Robson M. Teaching front-line health and voluntary workers to assess and manage suicidal patients. J Affect Disord. 1999;52(1–3):77–83.
Coppens E, van Audenhove C, Gusmão R, Purebl G, Székely A, Maxwell M et al. Effectiveness of General Practitioner training to improve suicide awareness and knowledge and skills towards depression.J Affect Disord. 2017;227:17–23.
Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior: an overview of systematic reviews of interventions. Med Care. 2001;39(8):2–45.
Cheung A, Weir M, Mayhew A, Kozloff N, Brown K, Grimshaw J. Overview of systematic reviews of the effectiveness of reminders in improving healthcare professional behavior. Syst Rev. 2012;1(1):1–8.
Nivel. Nivel Zorgregistraties Eerste Lijn. 2021.
Donker G. Nivel Zorgregistraties Eerste Lijn - Peilstations 2018. Utrecht; 2019.
Twisk JWR. Applied multilevel analysis: a practical guide. Applied Multilevel Analysis: a practical guide. Cambridge University Press; 2006. pp. 1–182.
Balas EA. The clinical value of computerized information services. A review of 98 randomized clinical trials. Arch Fam Med. 1996 May 1;5(5):271–8.
Grimshaw JM, Eccles MP, Walker AE, Thomas RE. Changing physicians’ behavior: what works and thoughts on getting more things to work. J Contin Educ Health Prof. 2002;22(4):237–43.
Shea S, DuMouchel W, Bahamonde L. A Meta-analysis of 16 randomized controlled trials to Evaluate Computer-based clinical reminder Systems for Preventive Care in the ambulatory setting. J Am Med Inform Assoc. 1996;3(1):399–409.
Elzinga E, de Kruif AJTCM, de Beurs DP, Beekman ATF, Franx G, Gilissen R. Engaging primary care professionals in suicide prevention: A qualitative study. Kleiman EM, editor. PLoS One. 2020;15(11):e0242540.
Saini P, Chantler K, Kapur N. General practitioners’ perspectives on primary care consultations for suicidal patients. Health Soc Care Community. 2016;24(3):260–9.
Leavey G, Mallon S, Rondon-Sulbaran J, Galway K, Rosato M, Hughes L. The failure of suicide prevention in primary care: Family and GP perspectives - a qualitative study. BMC Psychiatry. 2017;17(1).
Yarnall KSH, Pollak KI, Østbye T, Krause KM, Michener JL. Primary Care: Is There Enough Time for Prevention?Am J Public Health. 2003;93(4).
Bucher S, Maury A, Rosso J, de Chanaud N, Bloy G, Pendola-Luchel I, et al. Time and feasibility of prevention in primary care. Fam Pract. 2017;34(1):49–56.
Saini P, Chantler K, Kapur N. GPs’ views and perspectives on patient non-adherence to treatment in primary care prior to suicide. J Mental Health. 2017;27(2):112–9.
Saini P, While D, Chantler K, Windfuhr K, Kapur N. Assessment and management of suicide risk in primary care. Crisis. 2014;35(6):415–25.
Hofstra E. It takes a community to prevent suicide: The development, roll-out, and scientific evaluation of a regional systems intervention for suicide prevention (SUPREMOCOL). s.l.: Ipskamp Printing, 2021. 209 p.
Michie S, van Stralen MM, West R. The behaviour change wheel: A new method for characterising and designing behaviour change interventions. Implementation Science. 2011;6(1):1–12.
Bocquier A, Pambrun E, Dumesnil H, Villani P, Verdoux H, Verger P. Physicians’ characteristics associated with exploring suicide risk among patients with depression: A French panel survey of general practitioners.PLoS One. 2013;8(12).
Backman R, Bayliss S, Moore D, Litchfield I. Clinical reminder alert fatigue in healthcare: a systematic literature review protocol using qualitative evidence.Syst Rev. 2017;6(1).
van Mens K, Elzinga E, Nielen M, Lokkerbol J, Poortvliet R, Donker G et al. Applying machine learning on health record data from general practitioners to predict suicidality. Internet Interv. 2020;21:100337.
Wet WMO, [Internet]. Rijksoverheid 1998. Available from: https://wetten.overheid.nl/BWBR0009408/2022-01-31
Amsterdam UMC. Research Support: Ethical review [Internet]. [cited 2022 Dec 8]. Available from: https://www.amsterdamumc.org/en/research-support/ethical-review.htm
General Data Protection Regulation (GDPR). Subject-matter and objectives [Internet]. 24. Available from: https://gdpr-text.com/nl/read/article-24/
de Leeuw-Stravers E. Informatie voor (deelnemende) zorgverleners - Nivel Zorgregistraties Eerste Lijn [Internet]. 2022 [cited 2022 Dec 16]. Available from: https://www.nivel.nl/nl/nivel-zorgregistraties-eerste-lijn/informatie-voor-deelnemende-zorgverleners
We thank the sentinel GPs from Nivel Primary Care database and the data managers from the Nivel for contributing to this study.
This study was funded by the Ministry of Health of the Netherlands: grant number 326961, 2017.
Ethics approval and consent to participate
In the Netherlands, the Medical Research Involving Human Subjects Act (Wet Medisch wetenschappelijk onderzoek met mensen, WMO ) came into force since 1998. All medical scientific research in which human participants are subjected to (medical) procedures or are required to follow rules of behaviour, falls under the scope of this Act. All other research with humans (e.g. research with medical data or interviews) is not subject to the WMO, and does not require approval by an accredited independent committee of experts . According to this legislation, neither obtaining informed consent nor approval by a medical ethics committee is obligatory for this kind of observational studies [43,44,45]. Because this study does not subject participants to (medical) procedures or rules of behaviour, this study was never assessed by a medical ethical committee. Patients were never directly assessed by the researchers; patients’ characteristics were obtained by surveying GPs. General practices that participate in Nivel Primary Care Database signed a contract by which they provided written informed consent for participating in research. In addition, they are contractually obliged to inform their patients about their participation in Nivel Primary Care Database, and about the option to opt-out if patients do not wish to include their data in the database . The study design and procedures were discussed and approved in the Sentinel Practices and Surveillance Advisory Board Meeting of 11 October 2016, NZR064-111016 (meeting report). This study was conducted in accordance with relevant guidelines and the WMA Declaration of Helsinki.
Consent for publication
The authors declare they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Elzinga, E., de Beurs, D., Beekman, A. et al. Nudging General Practitioners to explore suicidal thoughts among depressed patients. BMC Prim. Care 24, 88 (2023). https://doi.org/10.1186/s12875-023-02043-3
- Suicide prevention
- Suicide exploration
- General practitioners
- General practice
- Mental health
- Behaviour change
- Clinical guidelines.