Can digitally enabling community health and nutrition workers improve services delivery to pregnant women and mothers of infants? Quasi-experimental evidence from a national-scale nutrition programme in India

ICDS Program and the ICDS-CAS Intervention

The ICDS programme, launched in 1975, is a national flagship nutrition programme to support the health, nutrition and developmental needs of children below the age of six, and pregnant and lactating women, through a network of early childhood development and feeding centres called the Anganwadi Centres (AWCs) at the village-level.27 Each AWC is served by a CHNW called Anganwadi worker (AWW) who is a full-time government paid female worker—but her official position is that of a contractual staff and not permanent government staff—from the community. The ICDS programme serves an estimated 158 million beneficiaries through India’s 1.4 million CHNWs. Under the ICDS, CHNWs provide five core services: (1) supplementary food including hot-cooked meals and take-home rations (THR) (2) home visits to provide health and nutrition education to pregnant and lactating women on pregnancy care and infant and young child feeding practices; (3) growth monitoring for children; (4) preschool education for children 3–6 years of age; and (5) conducting a monthly event called the Village and Health and Nutrition Day for immunisation and other health-related services in coordination with the National Health Mission (NHM) frontline workers.

In 2012, the ICDS Systems Strengthening and Nutrition Improvement Programme (ISSNIP) was launched across 162 districts having high undernutrition burden. Under ISSNIP, a mobile app-based intervention was piloted for ICDS and NHM frontline workers to improve coordination and collective service delivery to pregnant women. In 2016, a modified mobile-app based intervention called CAS was rolled out for cadre of CHNWs under the ICDS by the Ministry of Woman and Child Development. At the time of this evaluation, ICDS-CAS intervention covered over 600,000 CHNWs from 347 districts across 28 states and more were likely to be covered.28 29

ICDS-CAS sought to digitally enable CHNWs with a data capture module that digitised and replaced ten of the eleven paper registers maintained by the CHNWs to register and longitudinally track provision of services to different type of beneficiaries. ICDS-CAS also was designed to facilitate CHNW’s workflow management, remind her of upcoming home visits and services due to beneficiaries, provide checklists and a library of instructional videos as a job-aid during counselling, track growth status and immunisations for children, and report data for the programme’s monitoring. Figure 1 depicts the originally intended purpose and data flow for ICDS-CAS that was not only a CHNW level intervention but was also meant to support real time monitoring and decision making at all levels from the CHNWs’ immediate supervisors to the state-level ICDS director. ICDS-CAS intended to include three additional features : (1) a module for CHNWs to report supply chain constraints and other logistics issues; (2) a separate app for CHNW supervisors to monitor CHNWs remotely, assess quality of service delivery, and use as a job-aid to train CHNWs; and (3) web-based real-time dashboards for officials at the block, district, state and national levels to identify bottlenecks, prioritise local issues, and take data-driven decisions. But, the aforementioned features could not be implemented during our evaluation. Therefore, we evaluated the effect of digitally enabling CHNWs on the services they could deliver but not the effect of real-time monitoring or decision making features of the ICDS-CAS system. In 2020, Government of India discontinued CAS and replaced it with another system called the Poshan Tracker. It was beyond the scope of this impact evaluation to study why other components could not be implemented or the reasons why CAS was replaced with the Poshan Tracker.

Figure 2 maps different ICDS services to different modules on the CHNW app to track and identify beneficiaries due for the service, provide the service and update the records for longitudinal tracking. All CHNWs were provided training over multiple rounds on using the CHNW app. Helpdesks at block and district levels were available for technical support.

Based on the WHO classification of digital health intervention,30 this intervention would broadly fit the following classifications: (2.1) client identification and registration; (2.2) client health records; (2.3) healthcare provider decision support; (2.7) health worker activity planning and scheduling; (1.1) targeted client communication (during home visits) and (4.1) data collection, management and use.

Theory of change

Figure 3 presents the hypothesised theory of change we evaluated. The CAS app for CHNWs was expected to improve CHNW service delivery related to home visits and counselling as per ICDS guidelines specific to the life stages (trimester of pregnancy, postpartum period and age of child), growth monitoring consisting of weighing of the child and discussion of growth with the mothers, and counselling and provision of THR to pregnant women and mothers of children <12 months. We further hypothesised that improvement in these ICDS services can improve infant and young child feeding practices.

Study design

We used a village-matched, quasi-experimental design with repeated cross-sectional preintervention and postintervention measurements to assess the impact of the digital health intervention. The impact parameter of interest is intention-to-treat effect conditional on matching and sample restriction. All primary and secondary outcomes were measured based on interviews with pregnant women and mothers, and the community-level covariates were based on CHNWs’ interviews. Study methods, details of matching and rationale have been published previously.31 Reporting in this article follows the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) guidelines for quasi-experimental evaluation studies.32

Patient and public involvement

Patients or the public were not directly involved in the design, or conduct, or publication, or dissemination plans of our research. However, the research design was presented to government departments, funders, implementing partners of the intervention including non-governmental organizations (NGOs) to build their support for the study. Periodic updates on study progress were given to these stakeholders, and results were presented to them as well.

Study setting and timelines

The staggered rollout of the digital health intervention started in October 2016 across several districts and states under the larger umbrella of a system strengthening project for nutrition (https://projects.worldbank.org/en/projects-operations/project-detail/P121731). The evaluation was conducted in two purposively selected states, Madhya Pradesh (MP) and Bihar, in discussion with the Ministry of Woman and Child Development and the funding agency. The states differed in prevailing health and nutrition issues as well as the capacity of the state health and nutrition systems. Bihar has a higher burden of undernutrition than MP—49.3% of children <5 years in Bihar and 43.6% in MP were stunted and 63% of mothers in MP and 49% in Bihar reported that their children <6 years received some service from the AWCs.33 The under-5 mortality is high in both states at 65 and 58 per 1000 live births in MP and Bihar, respectively. Only 34% of infants in Bihar and MP were fed within the first hour of birth, and exclusive breast feeding at 6 months was 53% and 58%, respectively. Even more alarmingly, only 8% in Bihar and 7% in MP of children 6–23 months received an adequate diet.

Data were collected at two time points, before the launch of the full intervention (May–August 2017), and at follow-up (2019: January–February in MP and July–August in Bihar). The intervention and survey timelines are shown in figure 4. Prior to the baseline survey in May–August 2017, the CHNWs in the intervention districts in MP and Bihar had received smartphones, installed the app and received first of the four rounds of training on the app use. However, the full set of four trainings were concluded and the app usage started by July (MP) and December (Bihar), 2017 after the baseline data collection. The endline survey in Bihar was postponed by almost 6 months to allow ICDS services to normalise after a month-long CHNW strike in February–March 2019.

Statistical methods

We estimated intention-to-treat impacts of the intervention as pre-specified in the study protocol, separately for MP and Bihar, using two models.

where,    is the outcome of interest for beneficiary i in village j at endline (t=1);    is an indicator variable f denoting the treatment status of the digital health intervention;    is the village-level average of outcome Y at baseline (t=0);    and    are fixed effects for k pairs of matched villages or d pairs of matched districts; P_j is the propensity score estimated in the baseline; Z is a set of 20 covariates identified based on balance test results to control for observed imbalances and to improve precision; ε is the model error term; and    is the estimated risk difference or the effect of the intervention on outcome Y. Note, the intervention was assigned at the district level so that all villages in a district are in either intervention or comparison group. The pairs of intervention-comparison villages were matched using nearest neighbour propensity score matching method as described in the protocol.31

Preintervention balance was assessed using model 1 except the term    for a range of CHNW, household and beneficiary characteristics.

SEs in model (1) were clustered at the village level given village-pair fixed effects were specified, and standard errors in model (2) were clustered at the village-pair level to account for village pairing. Given the evidence of some imbalance between the intervention groups, results from model (2) are mainly discussed in this paper.

All analyses were done in STATA V.15, documented in a DO file, verified for code consistency (by AP and LG), and replicated by two different analysts (SN and SRP).

Sampling

Sample design and sampling procedures are published in the protocol31 and updated in figure 5. The sample was powered to detect a difference of 5–9pp from the counterfactual levels between 10% and 50% with intracluster correlation coefficient between 0.15 and 0.30 assuming 1200 respondents from 200 villages in each arm in each state.

The achieved sample consisted of 210 pairs of villages in MP and 216 pairs in Bihar matched using 1:1 nearest neighbour propensity score matching with Census 2011 village level characteristics from three pairs of CAS and non-CAS ISSNIP districts which were purposively selected in MP and Bihar. Baseline surveys were completed in 417 (out of 420) villages in MP and 428 (out of 432) villages in Bihar. At the endline, 4 out of 845 villages were lost to follow-up and this minor loss is assumed as random. The endline sample in MP was 684 CHNWs from 415 villages, 1048 pregnant women in last trimester and 3218 mothers of children <12 months. The endline sample in Bihar was 633 CHNWs from 426 villages, 1350 pregnant women in last trimester, and 3417 mothers of children <12 months. This study excluded women not registered with the CHNWs and mothers of children 12–59 months who were covered by the ICDS-CAS intervention, because the primary outcomes would be most relevant and sensitive for pregnant women and mothers with children <12 months given the exposure period of 16–18 months.

Outcomes and their measurement

The primary outcome indicators were constructed using aided recall by beneficiaries of home visits and counselling received from CHNWs during 3 months prior to the survey as prespecified in the study protocol. ICDS programme guidelines specify the number of home visits and counselling messages a beneficiary should receive at different life stages (ranging from the second trimester of pregnancy up to the child completing 24 months of age). Accordingly, the outcome indicator for adequate number of home visits was set to 1 if the respondent reported receiving the minimum number of home visits in the last 3 months, and 0 otherwise. The outcome indicator for life-stage appropriate counselling was set to 1 if the respondent recalls receiving at least half the counselling messages appropriate for their life stage as per ICDS guidelines in the past 3 months, and 0 otherwise.

Secondary outcomes included exposure to other nutrition programme services, such as food supplements and growth monitoring, as well as infant and young child feeding practices. Online supplemental table S1 describes in detail the construction of primary and secondary outcome indicators and the covariates used in the analyses, except those used only to test for preintervention balance.

Role of the funding source

The funder provided input to the overall evaluation design along with other partners. The funder was not involved in data collection, data analysis, data interpretation or writing of the manuscript. Funders reviewed the draft manuscript, but all final decisions rested with the authors. The corresponding author had full access to all the data and had final responsibility for the decision to submit for publication.

Sample characteristics and balance

Table 1 summarises sample characteristics at endline in the intervention and comparison groups in MP and Bihar. Compared with Bihar, MP had more pucca constructed AWCs (84% vs 54%), more centres with drinking water (65% vs 47%) and toilets (42% vs 18%), a larger proportion of CHNWs fully trained (68% vs 46%) and fewer CHNWs reporting problems in supply of supplementary nutrition or THR (11% vs 45%). A higher proportion of households in MP belonged to scheduled castes, had built a toilet and owned agricultural land.

As reported in online supplemental table S2, groups were well balanced at the baseline on all primary outcomes and most beneficiary characteristics, but there were some differences in CHNW characteristics, service delivery and secondary outcomes related to growth monitoring and complementary feeding. Even at the endline, there were some important differences between the intervention and comparison groups in CHNW and beneficiary characteristics and practices which could not have been influenced by the intervention (online supplemental table S3).

Given evidence of imbalance in some of the preintervention covariates and differences at the endline in two group, results from model 2 are mainly discussed in this paper. We also flag and discuss results where the magnitude of the effect estimated by models 1 and 2 are ‘qualitatively’ different; often they agree with each other.

Exposure to the intervention

As prespecified in the protocol,31 we estimated the intention-to-treat impact parameter which assumes that all CHNWs and beneficiaries were exposed to the intervention as per the original assignment. Nearly 98% of the CHNWs in the intervention districts in both states received the training, 100% in MP and 95% in Bihar reported owning a smart phone, and 99% in MP and 88% in Bihar carried the smart phone with app during home visits as reported in table 1. Close to 70% of the CHNWs reported ‘always’ using the app content during home visits.

Primary outcomes

At endline, mothers in the intervention group were more likely to receive the appropriate number of home visits in both states (MP: comparison mean=42%, effect=8 pp (95% CI 4pp to 13pp; p<0.001), Bihar: comparison mean=24%, effect=8 pp (95% CI 4pp to 12pp; p<0.001)) (table 2). Mothers were also more likely to recall at least half of the counselling messages that they were meant to receive (MP: comparison mean=28%, effect=12 pp (95% CI 7pp to 16pp; p<0.001). Bihar: comparison mean=9%, effect=8 pp (95% CI 5pp to 11pp; p<0.001)). The magnitude of the effect size is practically the same between models 1 and 2 suggesting lack of confounding bias, and thus, attributable impacts.

The proportion of pregnant women receiving adequate number of home visits was not different between the groups at the endline (MP: comparison mean=55%, effect=3 pp (95% CI −4pp to 11pp; p=0.353), Bihar: comparison mean=26%, effect=3 pp (95% CI −3pp to 9pp; p=0.327)). However, the recall of at least half of the life-stage appropriate messages by pregnant women in the intervention group was higher in MP (MP: comparison mean=43%, effect=9 pp (95% CI 2pp to 16pp; p=0.009)).

Secondary outcomes

In Bihar, the proportion of mothers reporting weighing of their child or reporting that the CHNW discussed child’s weight/growth more than doubled in the intervention group, with little difference in the magnitude of the effect between models 1 and 2 (table 3). In MP, the CAS intervention did not result in any difference between groups at endline. There was no effect of CAS on receipt of THR at least once a month by mothers. However, in both states 8%–10% more mothers in the intervention group reported receiving counselling on consumption of THR.

In both states, a higher proportion of mothers in the intervention areas recalled being counselled by the CHNW about exclusive breast feeding, timely initiation of complementary feeding, diet diversity, and adequate frequency of meals for children >6 months of age as was intended under the theory of change for CAS (table 4). However, no impacts were observed on infant and young child feeding practices.

Comparison with other digitally enabled CHNW evaluations

The magnitude of effects on appropriately timed home visits are somewhat comparable with results from randomised controlled trials of similar digital interventions for community health workers at pilot-scale. Compared with our finding of 8pp effect on homevisits to mothers, a trial in Bihar had found an impact of 12pp20 and another trial in Gujarat had found an impact of 10.2pp on visits during the first week after the birth of the child.22 However, our study did not find any impact on visits to pregnant women unlike the above pilot trials in Bihar and Gujarat. It is possible that the behaviour change tools and approaches used in CAS application were not as effective as the tools and approaches used in the pilot programmes. For example, Mobile Kunji intervention that focused on human-centric audio-video and interactive voice response messaging in eight districts of Bihar was strongly associated with a range of behavioural and nutrition outcomes.21

Across all four of the infant and young child feeding counselling areas—from early initiation of breast feeding to complementary feeding—we found that more mothers in the intervention group recalled receiving messages on recommended practices. This is promising and suggests that digital technologies can enable provision of more age-appropriate content of counselling. However, our study did not find impacts on infant and child feeding practices, unlike the pilot-scale trials.20 22 It is likely that these results highlight the challenges in replicating success of small, well-controlled, and intensive pilots in large-scale real-life programmes. In this respect, our findings are similar to an observational study of a national-scale SMS-based data reporting and health monitoring system in Rwanda which found no impact of the at-scale intervention on antenatal care visits, institutional deliveries, postnatal care visits, and malnutrition screening.19 It is also possible that despite the delivery of appropriate counselling content, other programme components that support nutrition behaviour change may have been limited. For example, other large-scale behaviour change programmes invested heavily in shaping other determinants such as community norms or engaged influential members such as fathers and grandmothers.34 It is also possible that other non-knowledge constraints to behaviour change—such as maternal time, workload, or household resources—were not fully addressed.

Limitations

Our study is not without limitations in identifying unbiased estimates of the programme effects. Due to the nature of the rollout, we could not design a randomised controlled trial that would have been an ideal design to prove causality. We relied on an observational design with the matching procedure that prioritises reducing selection bias or confounding but at the cost of representativeness of the sample. Therefore, the results are conditional on the matched sample which excludes villages not on common support of propensity score.31

Our village-level matching and use of a repeated cross-sectional design can theoretically ensure control of time invariant village-level confounders but not beneficiary-level or time-variant village-level confounders. However, the strong agreement between the results with and without controlling for household, individual and village-level covariates suggests that the bias due to imbalanced confounders was minimal.

We also mainly rely on recall by survey participants in constructing our indicators and not on any objective measurements or data. We did not have access to backend data from the CAS app or ICDS system which could have provided us with more objective measurements. However, even if we had these data, per the intervention’s design, there would have been a differential bias in measurements—the monthly progress data in the intervention group would have been more complete, timely, and less error prone because of CAS app, while the comparison group would have used the paper-based registers to compile such data which is a completely different system and cannot be compared with CAS. Although the recall biased indicators can have measurement errors, the responses of beneficiaries in the intervention and control groups should not be differentially biased due to John-Henry or Hawthorne biases because, arguably, the only difference beneficiaries could perceive was seeing a mobile phone in the hands of a CHNW. On other hand, access to the system-generated data on CHNW engagement with the app could have allowed us to perform heterogeneity analysis by CHNW performance and engagement with the app and to study if that lead to improved outcomes for beneficiaries.