- Open Access
Validation of a modified questionnaire to assess Ukrainian Family Physicians’ readiness to implement the evidence-based screening recommendations into their clinical practice, using a mixed method study
BMC Primary Care volume 23, Article number: 225 (2022)
Ukrainian Family Doctors' knowledge and readiness for the implementation of new guidelines recommendations into practice has to be evaluated and taken into consideration, meanwhile they often use the old protocols of annual checkups. This study aimed to perform a linguistic and cultural validation of Ukrainian adopted questionnaire designed on the German prototype “Readiness of general practitioners to recommend and implement evidence-based screening recommendations questionnaire”.
This was a mixed method study. The English version of the original German prototype questionnaire was translated into Ukrainian with forward–backward method. Additionally Ukrainian version was modified by including an extra questions about evidence based screening of cardiovascular risks, infectious diseases, type 2 diabetes mellitus, depression, and some old-fashioned annual checkups which are still commonly used in routine family doctors’ practice. During the face validation process, 10 practicing general practitioners assessed all the items in the questionnaire as adequately structured, grammatically correct, and understandable. During qualitative part of content validation process 11 experts found and corrected some grammatical errors, indicated that items of the questionnaire were comprehensible and related to Ukrainian culture. During the quantitative part of content validation process experts accessed 5 of 6 items as essential, relevant, and clear. Internal consistency of the items, assessed by using Cronbach’s alpha method was acceptable. To access how stable can be results provided by the questionnaire, test–retest reliability was performed, where 19 general practitioners filled in the same questionnaire twice over a period of two weeks.
In our study CVR > 0.79 and CVI > 0.59 were acceptable. The internal consistency was evaluated by using Cronbach’s alpha method and had to be above 0.7. According to the test–retest reliability results of weighted kappa and Cohen's kappa coefficients, test–retest agreement of the questionnaire was moderate for 29%, substantial for 38%, and almost perfect for 5% of the items (p ≤ 0.05). Kappa coefficients were not computed for 10% of items as there was no variability in the assessments.
The Ukrainian version of the questionnaire can be used for the assessment Ukrainian Family Doctor’s readiness to implement the evidence-based screening recommendations into their clinical practice.
In a systematic analysis for the Global Burden of Disease Study 2019 it is highlighted that the global burden of chronic non-communicable and communicable diseases is increasing . Cardiovascular and oncological diseases keep leading positions in the list of the most common causes of death in Europe [2, 3]. Recent years many European countries increased their focus on preventive measures and a growing trend towards more screening for non-communicable diseases is there . Four separate categories of prevention have been defined by the World Organization of National Colleges and Academies of Family Medicine (WONCA) and the International Dictionary for General/Family Practice in 2003 (See Table 1) [5, 6].
Among European countries different screening recommendations and guidelines concerning breast cancer, cervical cancer and colorectal cancer screening exist. But not all countries follow the basic criteria for screening .
In central and western Europe there are different governing bodies that aid, support and provide screening recommendations to the citizens. However, the situation is different in Ukraine. Previous post-Soviet non-evidence-based screening protocol was canceled by the Ukrainian Ministry of Health Care legislation in 2018 . But the new one has not been offered yet . The government is still searching for means of implementing and introducing various screening recommendations . Clinicians' knowledge and readiness for the implementation of new guidelines recommendations into practice also have to be evaluated and taken into consideration, meanwhile they often use the old protocols of annual checkups in their routine practice . Besides just providing recommendations for Family Doctors (FDs) to use, screening needs to be state-based and covered by insurance as in other European countries.
Nowadays the Ukrainian family doctors' knowledge about evidence based screening and their readiness to implement these modern screening recommendations into clinical practice is unclear.
In this aim, we needed and elaborated a questionnaire to assess Ukrainian General practitioners’ (GPs) readiness to recommend and provide modern screening strategies based on evidence instead of the old ways of post-Soviet screening. This article will describe the elaboration and validation procedure of the questionnaire about readiness to recommend and implement such screening in Ukrainian FD’ practice (See Supplementary info file).
This study aimed to perform a linguistic and cultural validation of Ukrainian adopted questionnaire designed on the German prototype “Readiness of GPs to recommend and implement evidence-based screening recommendations questionnaire”.
Materials and methods
This mixed method study was conducted by the research team of Family Medicine and Outpatient Care Department of Medical Faculty 2 of Uzhhorod National University, Uzhhorod, Ukraine.
The questionnaire that we used for our study was a modified version of the questionnaire designed by the German scientists in their research for cancer-based screening . Prior German questionnaire was developed and validated in Uzhhorod, Ukraine. The original version of it included 5 questions. The permission to use the original version of the questionnaire for Ukrainian research objectives was granted by the German authors. Except oncological screening we were interested in a screening of cardiovascular diseases, type 2 diabetes mellitus, some common infectious diseases (hepatitis B and C, HIV, and others), i.e. we included extra questions using the same design and formulation as in the German prototype questionnaire. As far as the state-based screening program has not yet been implemented in Ukraine, some GPs still follow post-Soviet protocol and recommend adults to follow some routine non-evidence-based investigations during annual check-ups (for example, CBC, urine analysis, lungs X-ray) . To achieve our respondents' knowledge and readiness to follow the new evidence based screening strategies comparing with the old ones, commonly used in family medicine practice as routine annual checkups, we also included the list of such investigation into the questionnaire as well.
Two bilingual experts cooperated with the researcher to translate the questionnaire, using a forward–backward method. Firstly, two native Ukrainian experts (who had an advanced level in English) translated the original English version of the questionnaire into Ukrainian. Then, the two native English-speaking experts (who had an advanced level in Ukrainian) translated it back into English. After that, both versions were discussed and compared in an expert panel meeting and presented the final version of the Ukrainian questionnaire [11,12,13].
The adopted questionnaire was given to 10 practicing GPs with more than 5-years of practical experience. GPs had to assess if items in the questionnaire had been adequately structured if the grammar was correct if the questions were understandable. For the evaluation of the importance of questions, they were provided with a 5-point Likert scale for each of the items to range from 1 (not important at all) to 5 (highly important). Once all the results were collected, the impact score was calculated using the formula below and scores > 1.5 were considered acceptable.
Both qualitative and quantitative phases were assessed by experts who were familiar with both the topic and the purpose of the questionnaire. We recruited 11 experts who were practicing as GPs and had experience in teaching family medicine. The experts had to assess if there were any grammatical errors, if the information was comprehensible, if each item was related to Ukrainian cultural issues and if there were any variants to improve the suggested instrument. Once all the modifications have been made, the questionnaire was given back to the experts to assess each item of the questionnaire for content validity ratio (CVR). They had to review the items with respect to being essential, useful, or unnecessary. For that purpose, experts were provided with a 4-point Likert scale ranging from 1 to 4 to assess each question for clarity and relativity. Once this was done content validity indexes (CVI) were calculated for each question.
The CVR was calculated by the ratio of the total number of experts divided by the number of experts saying item essential/total number of experts on the panel.
The CVI of each item, both for relevance and for clarity, was calculated by the ratio of the number of responses “3” or “4” in relation to the total number of responses to the item .
In our study CRV > 0.79 and CVI > 0.59 were acceptable.
For the assessment of reliability, both the internal consistency and the test–retest reliability were evaluated. The sample size for internal consistency testing included 57 responses and was evaluated by using Cronbach’s alpha method. To prove it the internal consistency had to be above 0.7 .
To assess test–retest reliability the questionnaire was administered twice among 19 doctors over a period of two weeks.
Test–retest reliability was analyzed by weighted kappa (κw) for ordinal variables and by Cohen's kappa (κ) for nominal ones. Kappa compares an observed and expected accuracy with the evaluation of not only a single question, but also questions among themselves.
The suggestions of Muñoz and Bangdiwala were used for interpretation of the kappa statistic: < 0 = poor agreement, 0 to 0.20 = fair agreement, 0.20 to 0.45 = moderate agreement, 0.45 to 0.75 = substantial agreement, 0.75 to 1.00 = almost perfect agreement [17, 18].
Inclusion and exclusion criteria for data collection see in Table 2.
After comparison and discussion of both versions of the questionnaire at an expert panel meeting some grammatical issues were cleared and the final version of the Ukrainian questionnaire was accepted.
The face validity was carried out by the participants who mentioned that the contents of the questionnaire were straightforward, comprehensible, and was appropriate for their intended use. Impact scores for all questions were greater than 1.5.
In our study CVI > 0.79 and CVR > 0.59 were acceptable. See Table 3.
As Table 1 shows, question 5 didn’t have an acceptable CVR and SVI indexes that is why this question was removed from our questionnaire.
The internal consistency was evaluated by using Cronbach’s alpha method. To be acceptable, the internal consistency had to be above 0.7. The questionnaire’s internal consistency was 0.85.
Test–retest reliability was analyzed by weighted kappa (κw) for ordinal variables (Questions 1,4,5) and by Cohen's kappa (κ) for nominal ones(Questions 2,3).
According to the results of weighted kappa and Cohen's kappa coefficients, test–retest agreement of the questionnaire was moderate for 29%, substantial for 38%, and almost perfect for 5% of the items (p ≤ 0.05). Kappa coefficients were not computed for 10% of items as there was no variability in the assessments. In 15% of items, test–retest agreement was not significant (p > 0.05).
The distribution of weighted kappa (Questions 1,4,5) and Cohen's kappa (Questions 2,3) coefficients for the single items are shown in Tables 4, 5, 6, 7 and 8.
This study aimed to translate the original German prototype of “Readiness of GPs to recommend and implement evidence-based screening recommendations questionnaire”, and to test it concerning validity and reliability of the Ukrainian version was modified by including questions about evidence based screening of cardiovascular risks, infectious diseases, type 2 diabetes mellitus, depression, and some old-fashioned annual checkups which are still commonly used in routine family doctors’ practice (i.e. CBC, urine analysis, lungs X-ray).
First of all we started from forward–backward translation of the original English version of the questionnaire into Ukrainian language. Face validation process confirmed that the questions of the questionnaire were adequately structured, grammatically correct, and understandable. During the quantitative part of content validation process CVRs and CVIs indexes were calculated for all the questions. As a result only question number 5 from the questionnaire was not essential, relevant, and clear enough (CVR and CVIs were not acceptable). The GPs are not yet directly concerned by this question. That is why it was removed from the questionnaire.
There is no a state supported evidence-based screening program in Ukraine yet and some of GPs still are used to use post-Soviet routine investigations as a part of routine annual checkups (i.e. CBC, urine test) . That is why we added those investigations to the general list of investigations (See Additional file 1: Appendix).
Internal consistency of the items with the sample size of 57, assessed by using Cronbach’s alpha method was acceptable.
For the assessment of stability of the questionnaire, test–retest reliability was performed, where 19 GPs filled in the same questionnaire twice over a period of two weeks. According to the results of statistical analysis (Cohen's kappa and weighted kappa coefficients), test–retest reliability of 84,2% of items was acceptable (moderate for 29,4%, substantial for 39%, almost perfect for 5,3% and not computed for 10,5% of items because of no differences in respondents’' answers for the first and for the second time).
Up to the authors, the rest (15.8%) of items were not significant due to GPs doubts concerning what the right screening is.
The Ukrainian version of the questionnaire is reliable and adapted to the sociodemographic primary care context. The taken out questions did not directly concern the GPs in Ukraine because they don’t perform such procedures in their practice.
Why is this questionnaire useful in the Ukrainian primary care setting? In absence of national evidence-based screening guidelines supported by the state, GPs remain the key medical professionals in promoting and advising evidence-based screening strategies to the community.
We are aware that the current war situation in Ukraine will not foster screening implementation into clinical practice of Ukrainian Family doctors.
Before the new national screening protocol is developed it’s crucial to form the correct way of understanding by primary health professionals the right screening strategy of common non-communicable diseases.
Nowadays, Ukraine bears the second largest HIV epidemic in Eastern Europe and Central Asia. In 2019, an estimated number of 250.777people living has HIV (PLHIV) in Ukraine (Spectrum) and 2.977 AIDS-related deaths (14% less than in 2018) and 16.405 newly diagnosed HIV cases (4% more than in 2018) were reported . That’s why HIV has to be included in the list of mandatory to be screened diseases in primary medical care Ukraine.
According to the WHO 2017 data, Ukraine belongs to the countries with the average spread of Hepatitis C: approximately 3% of the population, i.e. 1.170.000 people, were infected. However, the results of selective monitoring of risk groups showed that the level of infection by Hepatitis C in some of them exceeds the average indicators and reaches 40–60% . That’s why HCV has to be included in the list of mandatory to be screened diseases in Ukraine.
Cervical cancer is a frequent disease in Ukraine. Among all women cancers, it ranks 5th for incidence and 6th for mortality. While first year mortality shows some decline, efforts on earlier detection are mandatory . That’s why cervical cancer has to be included in the list of mandatory to be screened diseases in primary medical care Ukraine.
The cardiovascular disease (CVD) mortality level in Ukraine is 772.1 per 100 000 for men and 440.9 per 100 000 for women. This accounts for 68,0% of the total mortality in the country [22, 23]. That’s why CVD screening has to be included in the list of mandatory to be screened diseases in primary medical care Ukraine.
Thus, this new and reliable questionnaire has a direct practical implication to determine GP’s readiness to get involved in evidence-based screening implementation and to meet the screening need in their country.
Strengths and limitations
This is the first concrete step to get involved GPs in Ukraine into screening routines. Our sample size of test–retest reliability calculation after the second-time questioning included only 19 doctors which is small. Initially we had a bigger number of participants (n = 60) but due to the new COVID-19 pandemic wave at that time which caused family doctors' workload the response rate for the second-time questioning was much lower (n = 19).
The original German prototype of “Readiness of GPs to recommend and implement evidence-based screening recommendations questionnaire”, has been translated to Ukrainian and tested concerning validity and reliability. The Ukrainian version was modified by including questions about screening of cardiovascular, infectious diseases, type 2 diabetes mellitus, depression, and prior local routine screening investigations (CBC, urine analysis, lungs X-ray). It can now be used as a reliable tool to determine GP’s readiness to get involved in evidence-based screening implementation and to meet the screening need in their country.
Availability of data and materials
Only summarized data sets generated or analyzed during the current study are included in this article.
Complete blood count
Content validity index
Content validity ratio
Hepatitis C virus
Human immunodeficiency virus
Oral glucose tolerance test
World Organization of National Colleges and Academies of Family Medicine
GBD 2019 Diseases and Injuries Collaborators T, Lim SS, Abbafati C, Abbas KM, Abbasi M, Abbasifard M, et al. Global burden of 369 diseases and injuries in 204 countries and territories, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet (London, England). 2020;396(10258):1204–22. Available from: http://www.ncbi.nlm.nih.gov/pubmed/33069326. [Cited 2021 Oct 29].
Townsend N, Kazakiewicz D, Lucy Wright F, Timmis A, Huculeci R, Torbica A, et al. Epidemiology of cardiovascular disease in Europe. Nat Rev Cardiol. 2021. Available from: https://doi.org/10.1038/s41569-021-00607-3
Ferlay J, Colombet M, Soerjomataram I, Dyba T, Randi G, Bettio M, et al. Cancer incidence and mortality patterns in Europe: Estimates for 40 countries and 25 major cancers in 2018. Eur J Cancer. 2018;103:356–87.
Screening programmes: a short guide. Available from: https://apps.who.int/iris/bitstream/handle/10665/330829/9789289054782-eng.pdf
Jamoulle M, Roland M, Bae J-M, Heleno B, Visentin G, Gusso G, et al. Ethical, pedagogical, socio-political and anthropological implications of quaternary prevention. Rev Bras Med Família e Comunidade. 2018;10(13):1–14.
Martins C, Godycki-Cwirko M, Heleno B, Brodersen J. Quaternary prevention: reviewing the concept. Eur J Gen Pract. 2018;24(1):106–11. Available from: https://pubmed.ncbi.nlm.nih.gov/29384397.
Policy Brief Screening in Europe. Available from: https://www.euro.who.int/__data/assets/pdf_file/0007/108961/E88698.pdf. [Cited 2021 Nov 11].
Order of the Ministry of Health of Ukraine dated March 19, 2018 № 504. “On approval of the Procedure for providing primary care". Available from: https://moz.gov.ua/article/ministry-mandates/nakaz-moz-ukraini-vid-19032018--504-pro-zatverdzhennja-porjadku-nadannja-pervinnoi-medichnoi-dopomogi
Kolesnyk P, Frese T, Vinker S, Shushman I, Zharkova A, Messaadi N, et al. Steps towards implementing evidence-based screening in family medicine in Ukraine: SWOT-analysis of an approach of multidimensional empowerment. BMC Fam Pract. 2021;22(1):1–11.
Engler J, Dahlhaus A, Güthlin C. The readiness of German GPs to recommend and conduct cancer screening is associated with patient-physician gender concordance. Results of a survey. Eur J Gen Pract. 2017;23(1):11–9. 2016/11/14. Available from: https://pubmed.ncbi.nlm.nih.gov/27841043.
Tsang S, Royse CF, Terkawi AS. Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi J Anaesth. 2017;11(Suppl 1):S80-9. Available from: https://pubmed.ncbi.nlm.nih.gov/28616007.
Wild D, Grove A, Martin M, Eremenco S, McElroy S, Verjee-Lorenz A, et al. Principles of Good Practice for the Translation and Cultural Adaptation Process for Patient-Reported Outcomes (PRO) Measures: report of the ISPOR Task Force for Translation and Cultural Adaptation. Value Heal J Int Soc Pharmacoeconomics Outcomes Res. 2005;8(2):94–104.
Nagasaki K, Shikino K, Nishimura Y, Kuriyama A, Nonaka S, Izumiya M, et al. Translation, Cultural Adaptation, and Validation of the Mini-Z 2.0 Survey among Japanese Physicians and Residents. Intern Med. 2021;60(15):2405–11. 2021/02/22. Available from: https://pubmed.ncbi.nlm.nih.gov/33612686
Lacasse Y, Godbout C, Sériès F. Health-related quality of life in obstructive sleep apnoea. Eur Respir J. 2002;19(3):499–503.
Silva DFO, Souza AL de S, Pimentel JB, Souza TO, Araújo EPDS, Sena-Evangelista KCM, et al. Development and content validity of an instrument for assessing the motivation for weight loss in adolescents with overweight and obesity. PLoS One. 2020;15(11):e0242680–e0242680. Available from: https://pubmed.ncbi.nlm.nih.gov/33237963.
Jones GL, Benes K, Clark TL, Denham R, Holder MG, Haynes TJ, et al. The Polycystic Ovary Syndrome Health-Related Quality of Life Questionnaire (PCOSQ): a validation. Hum Reprod. 2004;19(2):371–7.
Munoz SR, Bangdiwala SI. Interpretation of Kappa and B statistics measures of agreement. J Appl Stat. 1997;24(1):105–12. https://doi.org/10.1080/02664769723918.
Pistoia F, Shiel A, Ornello R, Saporito G, Gentili L, Carolei A, et al. Translation and Transcultural Adaptation of the Wessex Head Injury Matrix, Italian Version: A Preliminary Report. Brain Sci. 2021;11(6):810. Available from: https://pubmed.ncbi.nlm.nih.gov/34207277.
Global AIDS monitoring 2019:Ukraine. Available from: https://www.unaids.org/sites/default/files/country/documents/UKR_2020_countryreport.pdf. [Cited 2021 Nov 20].
WHO.Assesement of the viral hepatitis response in Ukraine,2017. Available from: https://www.euro.who.int/__data/assets/pdf_file/0007/372697/ukr-hepatitis-report-eng.PDF. [Cited 2021 Nov 12].
HPV information centre.Human Papillomavirus and Related Diseases Report. Ukraine. Available from: https://hpvcentre.net/statistics/reports/UKR.pdf. [Cited 2021 Nov 15].
European Society of Cardiology.EAPC Country of the month - Ukraine. Available from: https://www.escardio.org/Sub-specialty-communities/European-Association-of-Preventive-Cardiology-(EAPC)/Advocacy/Prevention-in-your-country/country-of-the-month-ukraine. [Cited 2021 Dec 11].
WHO.Too many Ukrainians die early from preventable causes – action is needed to curb noncommunicable diseases. Available from: https://www.euro.who.int/en/health-topics/noncommunicable-diseases/pages/news/news/2019/06/too-many-ukrainians-die-early-from-preventable-causes-action-is-needed-to-curb-noncommunicable-diseases. [Cited 2021 Nov 12].
This publication was financially supported by the publication funds of the Martin-Luther-University Halle-Wittenberg." Additionally the authors wish to thank all the study participants.
Open Access funding enabled and organized by Projekt DEAL. The study was not funded.
Ethics approval and consent to participate
Ethical approval has been obtained from the ethical committee board of Uzhhorod National University №6/5. All methods were performed in accordance with the relevant guidelines and regulations (Declaration of Helsinki). Informed consent was obtained from all subjects.
Consent for publication
The authors have no conflict of financial and non-financial interest to declare.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
ile 1. Readiness of general practitioners to recommend andimplement evidence-based screening recommendations questionnaire.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Shushman, I., Kolesnyk, A., Kolesnyk, P. et al. Validation of a modified questionnaire to assess Ukrainian Family Physicians’ readiness to implement the evidence-based screening recommendations into their clinical practice, using a mixed method study. BMC Prim. Care 23, 225 (2022). https://doi.org/10.1186/s12875-022-01818-4
- Family medicine
- Evidence-based screening