Assessing the scalability of health system interventions in Africa: protocol for a Delphi study

Abstract

Background

There is widespread enthusiasm for scaling interventions to strengthen health systems. However, little is known about the scalability of such interventions in Africa. In this study, we seek to assess the scalability of interventions for improving the functionality of health systems in Africa, as a key to large-scale implementation strategy of interventions with potential for impact.

Methods

The study will deploy a multi-pronged approach, grounded in an integrated knowledge translation (iKT) approach. First, a multidisciplinary steering committee will be established, involving key female and male stakeholders in all stages of our study from its inception and as equal members of the research team for overseeing the project. Second, as part of the RAND/University of California, Los Angeles (UCLA) Appropriateness Method, evidence from a published systematic review will be used to develop the African Scalability Assessment Framework (AFROSAF), a series of multiple attributes for assessing the ability to scale a health system intervention in Africa. Third, the content of the AFROSAF will be validated using Delphi survey (within a deliberative dialogue) following the Lavis’ framework for knowledge transfer and a conceptual framework developed by Boyko et al. a multi-stakeholder consensus exercise with experts from Africa will be convened. The Likert scaled scalability attributes developed will be rated and descriptive statistics and hierarchical cluster analysis will be used to synthesize the data. Finally, document analyses will be conducted to rate to which extent each intervention has data that meet criteria responding to the essential components of scalability using the AFROSAF. We will conduct an analysis to score and rank each intervention for scalability.

Discussion

This project proposes an approach aiming to catalyse the scale of interventions for effective functionality of health systems in Africa. The process will yield a scalability assessment tool for Africa and inventory scalable interventions. The findings will help African countries and policymakers understand the parameters to use and assess health system interventions for scaling.

Background

Countries in Africa are pursuing sustainable development goals and universal health coverage to end the suffering of their population through various interventions for health services and systems [1]. The increasing burden and evolving epidemiological changes in Africa, however, underscore the importance of strengthening health systems to deliver effective and efficient care. Various interventions have been developed and successfully piloted to strengthen health systems in low- and middle-income countries [2, 3]. However, there is a delivery gap between evidence-based interventions and those that actually reach the people who could benefit from them. International organizations, governments, and the scientific community are constantly developing and testing interventions, but fail to implement, sustain or scale those shown to be successful. Although substantial efforts have gone into creating a compendium of interventions for health systems and health services, little to no attention has been placed on ensuring that the identified effective interventions reach every corner of the community. Thus, there is a need for systematic efforts to help identify evidence-based interventions that could be successfully scaled to reach more beneficiaries in Africa [4, 5].

Within literature, different terminologies have been used to refer to the concept of scalability. The differences between “scaling up”, “scaling out”, “scaling deep”, “scaling” and “spread” are nuanced [1,2,3]. In this study, we use those terms to refer to scaling. By scaling, we mean systematic efforts to increase the impact, adoption and reach of effective interventions to improve health systems on a lasting basis [1, 3, 4]. Scaling process follows a number of steps after the development of an intervention [4,5,6]: (i) scalability assessment of the intervention, (ii) development of a scaling strategy, (iii) implementation and evaluation of this scaling strategy and (iv) addressing the long-term sustained use of the successfully scaled intervention [7]. Assessed in the first step, the “scalability” of an intervention is its potential for scaling [1, 4, 6]. By scalability, we mean the ability of an intervention to change in size (e.g. from district level to national or vice versa), be transferred (e.g. from a country in the world to an African country), or be sustained (e.g. local improvement within a health system), while retaining effectiveness, efficiency and equity [1, 8].

Recently, in depth knowledge syntheses [2, 3] and an analysis [1] were undertaken to consolidate effective interventions for the functionality of the health systems in the region. The WHO Regional Office for Africa inventoried a set of such interventions that contributed to improving the four capacities of health systems in Africa [2, 3], consisting of access to essential service, quality of care, demand for essential service, and resilience of health systems [2]. The study noted that 38.9% of the interventions contributed mainly to “access” and “quality of care”, 0.07% contributed mainly to demand while 0.06% interventions contributed mainly to resilience [2]. Some of the key interventions, which is eliciting growing awareness in Africa, addressed on how to leverage the potential benefit of mobile technology for service delivery, data availability and quality, surveillance, disaster preparedness and response among others [2]. The wired mothers experience in Zanzibar [6], the Ebola Surveillance and Contact Tracing in Guinea [7], and a mobile-based technical support on clinical management to doctors caring for patients with human immunodeficiency virus (HIV) in antiretroviral therapy (ART) centres in India [8], were a few from many with potential opportunities for scalability at the various levels of translation. Interventions focusing on mobilization of community health workers have also proven to function as leverage for health system strengthening in Africa and yield effective outcomes with minimal but adequate resources, thus becoming a fertile ground for scaling at subsequent translation stages [2]. Finally, innovative financing strategies also need to be considered for scaling while safeguarding financial protection and access to essential services. Inadequate and fragmented healthcare financing mechanisms in Africa is one of the most identified bottlenecks for health system strengthening [9]. Fostering synergies to achieve universal health coverage could be achieved by reducing fragmentation of health system financing marked by inequities in access to health services, low health workforce and limited capacity to implement guidelines [10]. The health sector-wide approach (SWAp) in Bangladesh offered an example of a successful adaptation of such an approach in a complex administrative structure [11]. These were examples among several interventions which need to be assessed for their scalability across African health systems.

However, little is known about the scalability of those interventions. To be scalable, a health system intervention should meet certain minimum criteria, such as effectiveness and cost effectiveness. Pitfalls, problems and difficulties with scaling even proven interventions suggest that it might be beneficial to identify phases as well as components of scaling of those interventions [12, 13]. Unfortunately, there are few theoretical, conceptual or practical frameworks to guide such assessments [4, 5]. These tend to have a global focus, specified income group (e.g. low-middle income countries) or program specific (e.g. HIV, maternal and child health, cardiovascular diseases, mental health) [14] lacking comprehensiveness to address system-wide approach and varying context of countries. Available scalability assessment tools lack methodological quality, and none have focused on the functionality of health systems, especially in the African context which is fit-for-purpose in attaining UHC [12]. Some innovations are scaled before undergoing a pilot trial or small-scale introduction [15, 16], while interventions often need considerable adaptation to enable implementation at scale. Scaling is a process that can reduce or remove the effects of interventions [17, 18]. Therefore, assessing the scalability of interventions comprehensively is critical to determine strategic investments and direction, as well as priority setting in a multi-faceted manner [12].

Objectives

The aim of this scoping review is to: (1) develop a framework for assessing the scalability of health system interventions in Africa, (2) validate the content of the tool with key stakeholders and (3) assess the scalability of interventions identified by the WHO Regional Office for Africa and their preparedness for scaling interventions aligned with functionality of health systems in the WHO African Region.

Methods

Overarching methodological approach (2 months)

A multi-pronged study will be conducted, grounded in an integrated knowledge translation (iKT) approach [19]. A multidisciplinary steering committee will be established, composed of key female and male stakeholders throughout all stages of the study, as equal members of the research team. Committee members will represent key stakeholder groups across African countries, including decision makers (H.C.K. and T.Y.), clinicians (H.C.K., A.B. and J.N.), and research investigators (H.C.K., A.B.C., S.N.K., H.K.K., S.Y.S., J.N. and T.N.M.). No patient or member of the public will be involved in this work. Team members will meet weekly to discuss the progress of work during the current week. They will use a virtual teamwork space using Google Drive and communicate using Microsoft Teams. Multidisciplinary consultations with key stakeholders will be conducted to discuss theoretical, conceptual or practical insights. Formal ethical approval from an ethics committee is not required because our research will assess research interventions and will not collect data from human participants.

Phase 1: development of the scalability assessment tool (2 months)

• First, as part of the RAND/University of California Los Angeles (RAND/ULCA) Appropriateness Method [20], the expert of scalability assessment (ABC) will use evidence from a published systematic review [12] to draft version zero (V0.0) of the framework, named the African Scalability Assessment Framework (AFROSAF – V0.0). English mother-tongue experts in health systems (S.N.K., A.B., J.N., T.Y. and H.K.K.) will review content and improvements will be discussed with all core team members through two 1-h virtual meetings. The AFROSAF will be a series of multiple attributes or questions for assessing the readiness to scale. This framework will be based on the Framework of Actions [1], a framework for health system strengthening towards universal health coverage in the WHO African Region. As such, the development of the framework will incorporate factors to consider when implementing in the WHO African Region, including socio-economic peculiarities, barriers and facilitators that may exist. The AFROSAF will be drafted following 15 interpretability criteria proposed for detecting scalability (Appendix 1) [12]. For example, each attribute will refer to a single idea, it shall not contain negated constructs or negative answers, will not ask a combination of two or more questions, and will not contain the words “and”, “or” or “because”. As there is no standard for the number of points on rating scales, we will use a seven-point Likert scale for rating each attribute: 1 = “to an extremely small extent”, 2 = “to a very small extent”, 3 = “to a small extent”, 4 = “to a moderate extent”, 5 = “to a large extent”, 6 = “to a very large extent” and 7 = “to an extremely large extent”. Indeed, a previous review suggests that seven-point scales are probably optimal in many instances [21]. Reliability beyond seven points is quite minimal for single attributes and the gains in validity become correspondingly smaller when scales grow longer [21]. Rating scales with 7, 9, or 10 responses is generally preferred [22]. Moreover, there will be two other possible responses: “not applicable” (NA) and “no data”. Space will also be provided for additional comments on each attribute, including rationale for choosing “not applicable”. Moreover, zero will not be used as a lowest point in any answer scale because it can be misinterpreted as “no answer”.

• Second, core team members (A.B.C., H.K.K., S.N.K., A.B., J.N., T.Y. and T.N.M.) will rate a random sample of about 50 health service interventions for their scalability. For each intervention, a review of published data will be undertaken to answer the specific attributes of the AFROSAF. Relevant information from these documents will be used to rate the corresponding attribute of the AFROSAF. Each intervention will be rated by one team member. Then, each team member will review and suggest improvements. In addition, a factor analysis will be used to guide these changes. All disagreements and results from the factor analysis will be discussed with all core team members through a 2-h meeting to reach consensus. Those improvements will be integrated to finalize a draft version of the AFROSAF (V1.0).

Phase 2: content validity of the AFROSAF (2 months)

• Design: aligned to the concept of integrated knowledge translation, a systematic multi-stakeholder consensus exercise (e-Delphi survey) will be undertaken with experts from academics, practitioners, advocates or policy makers functioning within countries in the WHO African region. It is anticipated that the two rounds will allow an acceptable degree of agreement on research priorities, but if not, a discussion meeting with committee members will be undertaken to finalize the AFROSAF.

• Participants: on the basis of sampling trends and recommendations from previous Delphi studies, there will be an invite to 100 participants [23] to help ensure that a minimum of 10 key stakeholders complete the two rounds [24]. They will be included on the basis of (1) their age (at least 18 years old); (2) their ability to participate, read and understand English; and (3) their knowledge or interest in the field.

• Data collection: to enable efficient and timely data collection with international participants, we will use an online platform. The questionnaire will be based on the AFROSAF. We will use a 5-point Likert score ranging from 1 = “strongly irrelevant” to 5 = “strongly relevant”.

• Data analysis: we will categorize participants’ responses as low (1–2), moderate (3) and high (4–5). For each criterion, consensus will be defined as 80% agreement for the priority score of 4–5. Results will be collated using Excel to calculate the mean score and percentage of agreement.

Phase 3: scalability assessment of health system interventions (2 months)

• Design: we will conduct a combination of document analysis and an online survey to rate to which extent each intervention meets the scalability criteria of the AFROSAF.

• Document analysis: for each intervention, we will undertake a review of existing data or literature to answer specific criteria or questions raised in the AFROSAF. Relevant information from these documents will be highlighted to populate the corresponding sections of the AFROSAF and then summarized across the sources.

• Online survey: an online questionnaire will be disseminated to participants to rate the scalability of each intervention. The questionnaire will be based on the readiness assessment questions provided in the AFROSAF. It will be circulated to the same participants who took part in the e-Delphi survey. However, as suggested [14], it may be possible to conduct this part of the assessment in person as a group if there are no aforementioned restrictions of COVID-19 or other epidemic at the time of the study. For each component of the AFROSAF, a condensed summary of the results from the document analysis will be provided and participants will be asked to rate scalability given the evidence for that domain. As an alternative, additional open-ended feedback sections will be included in the survey for participants to give more detailed feedback alongside single-choice questions.

• Data analysis: exploratory data analysis will be carried out; where there is a rating of interventions by more than one person, reliability analysis using cronbach’s alpha will be computed. In addition, factor analysis will be carried out to identify the relative factor contribution by each of the attributes. We will analyse the transformed Likert scaled data using descriptive statistics to look at the distribution of the attributes and explore potential associations or correlations between the different attributes. We will further carry out hierarchical cluster analysis, as proposed in a previous scalability assessment study [25]. This will allow us to group interventions into the most homogeneous clusters possible on the basis of the number of criteria met. The objective will be to rank interventions in order of their scalability. The unit of analysis will be the intervention. Finally, for each intervention, we will calculate an overall score of scalability with the associated 95% confidence intervals.

• Process documentation: for each system intervention, the AfroSAF scalability score, together with the evidence used will be generated using AfroSAF. This will allow for researchers and policymakers to contextualize the score appropriately and build further evidence to improve on its reliability. It is expected that the ranking scores of interventions will be “living”, as these will constantly be updated as countries and researchers explore and expand on the evidence base. Thus, the process will be documented to show evolution of scalability of interventions across time and countries.

Discussion

This project proposes an approach aiming to catalyse the scale of interventions for effective functionality of health systems in Africa. The core deliverables will be a validated AFROSAF and compendium of interventions with scalability scores. AFROSAF will be based on a set of attributes of scalability, that cater for niche peculiarities in implementation settings (e.g. country, district). Detailed information will be generated on scalability scores or rating for each health system intervention inventoried previously [9]. On the basis of our categorization of reviewed interventions into national and sub-national levels, we believe the findings from the scalability assessment will help African countries, policy makers and other decision makers across the health system better understand the parameter to use and qualify health services interventions for scaling, taking into account contextual variations. AFROSAF is expected to act as normative technical guide that countries can use at national and sub-national levels during the various stages of intervention scaleup. Furthermore, the compendium of interventions with scalability scores across the African region will provide rich succinct data on which interventions have been traditionally successful for scale up. This will be disaggregated by public health function (prevention, promotion, curative, rehabilitative and palliative care) as well as the age cohorts across the life course. Furthermore, by reflecting and documenting the processes of assessing scalability system stakeholders will be better informed about other underlying health system investment areas that would require improvement to accommodate and sustain scaled up interventions. The results will disseminated through publications in peer-reviewed journals, and through a technical brief. The brief will be shared using free public repositories, such as Open Science Framework and ResearchGate.