Data-Driven Monitoring in Community Based Management of Children With Severely Acute Malnutrition (SAM) Using Psychometric Techniques: An Operational Framework

Background The success of the Community Based Management of Severe Malnutrition (CSAM) programme largely depends on the knowledge and skills of Front-Line Workers (FLWs). A robust supportive supervision system in CSAM should be tailored to individualistic learning needs by distinguishing the FLWs as per their ability and simultaneously identifying the task domains to be emphasized more in supervisory visits. This paper details the ability assessment strategy developed and employed in the selected geographical locations in Madhya Pradesh (Central India) among the 197 Anganwadi workers (FLWs involved in CSAM implementation). Methodology A 25 items tool was developed based on an analytical construct for ability estimation through Rasch Analysis (RA). RA models the probability of right/wrong answers as a function of a person (participants) and item (questions) parameters and calculates the item difficulty in relation to personability on the same unidimensional linear scale. Suitable visualization like item characteristic curve (ICC), person item map (PIM) and quadratic allocation were plotted in RA. The data fitting to the Rasch model (Rasch diagnostic) was tested by numeric (Anderson LR and Wald test) and graphical methods. Results The item easiness parameter (β) value related to Diarrhoeal assessment was lowest (-2.32, -2.91 to -1.73) and related to peer assessment meaningful action (2.009, 1.669- 2.348)) was highest (most difficult). Anderson LR test (LR=31.32, df=24, p=0.079) showed the absence of global outliers. Quadrant analysis using the permutations of ability score and adjusted burden of malnutrition further mapped 41/197 (20.8%) FLWs to low ability -high burden quadrant and 44/197(25%) as low ability low burden quadrant. Conclusion Rasch assessment may address the innate challenges to maintain homogeneity, discrimination capacity and linearity in a raw score-based measurement construct. The monitoring strategy developed on this thus may offer a judicious, pragmatic and thematic approach to supportive supervision in the CSAM program.


Introduction
Severely Acute Malnutrition (SAM) is a condition characterized by extremely low weight for height, muscle wasting and nutritional oedema. It is closely associated with high mortality and morbidity rates among children under five [1]. A child with SAM or MAM (Moderate Acute Malnutrition) has more likelihood of mortality than a well-nourished child if not intervened promptly [2]. Until the first decade of this century, all such children were primarily approached through facility-based care through a specifically designed intervention centre known as Nutritional Rehabilitation Center -NRCs. This facility-based care consisted of medical management of complications (such as correction of hypoglycaemia, hypothermia, shock, electrolyte imbalance), feeding interventions (starter and rehabilitative diet, food chart) and educational interventions (involving mother in food preparation, care of a child after discharge). This is also known as facility-based management of SAM children (FSAM). However, this strategy faced several challenges such as high operational and opportunity costs (for both systems and parents of SAM children), low coverage; longer stays leading to overcrowded NRCs, and cross-contamination [3][4][5]. Moreover, multiple studies have stated that only 10-20% of the SAM children developed complications requiring hospitalized care. In contrast, the rest could be treated at the community level with a package of services (medicines, nutritional supplements, nutrient-dense foods and weekly tracking) [3,6]. To address this gap, Community Based Management of SAM children (CSAM) came into existence [7][8][9]. It is also known as Community based Management of Acute Malnutrition (CMAM) globally.
CSAM offers an opportunity for longitudinal tracking of the child by an intersectoral peripheral designated team at their doorsteps to monitor their nutritional status continuously. This also includes the initial and follow up assessment for medical complications and subsequent referral to the health facility. Several processes beginning from initial screening, classification, management and subsequent follow-ups of the beneficiaries in CSAM are mostly performed by the peripheral Front-Line Worker (FLW) team constituted chiefly by Anganwadi Worker (AWW) and Auxiliary Nurse Midwife (ANM). Thus, the program's effectiveness in principle depends on the robust ability of these FLWs to understand, apply and make decisions as per programmatic guidelines.
Thus, this program's monitoring and supervision strategies must be sensitive enough to capture the extent of the ability of FLWs about the conceptual understating and translating deliverables in CSAM strategy, without considering this aspect negates the principles on which they were built. The ability estimation is an abstract and latent construct in the Psychometric paradigm. Generally, the measurement of this construct is attempted by some manifested variables (henceforth referred to as items), which are presumed to measure the unique fragment of that construct. The significant sum of these items is perceived as ability. This seemingly easy concept of ability measurement is a little tricky on the operational plane, especially in social sciences /psychometrics. The raw scores derived by this summation may ignore the equidistance and linearity of items on difficulty level [10]. Thus these items may not discriminate the high from low achievers in the true sense. Neither offers an insight to the evaluator whether he measures a homogeneous construct accurately only [11][12][13]. There are multiple studies on the assessment of skills and capacities of the frontline workers; a few regarding programmes about the management of malnutrition and few others on generic public health programmes [14][15][16]. However, most of them estimate the ability by raw score without any transformation that neither considers the item difficulty and associated discrimination capacity nor addresses the equidistance distribution of items and participants on a linear scale. Translating it from a programmatic stance, the differential capacities of FLWs on different domains of tasks may not be explicitly addressed through them.
Rasch Analysis (RA) offers an alternate strategy to estimate the ability is offered by Rasch Analysis (RA), which considers the homogeneity, discrimination capacity, and linearity of items [17]. This is a mathematical modelling technique routed in logarithmic transformation. It attempts to achieve a conjoint (person-item measurement on the same linear scale) additivity (equidistance linearity). In contrast, the homogeneity of the tool is simultaneously attempted to maximize by reducing items (poor discriminator, misclassifies etc.) [18]. It presumes that the probability of a correct response on an item is the product of calibrated item difficulty and calibrated person ability measured on a logarithmic scale. A further description is of this scale is given in the supplementary file.
We developed a data-driven monitoring strategy within the context above that considered malnutrition magnitude concerning ability (measured by RA) estimation. The objectives for this strategy development were to identify the AWCs requiring frequent supportive supervisory visits and to identify the programmatic components to be addressed during the visit.
This article was previously posted to the medRxiv preprint server in June 2021.

Materials And Methods
This study was designed by the CSAM unit of the Regional Center of Excellence for Nutritional Rehabilitation and Resource Training (RCoENRRT), Madhya Pradesh. The RCoE was established under the aegis of AIIMS Bhopal, an apex teaching tertiary care hospital in Central India. The RCoE has adopted a subdistrict development block (Babai) from the district of Hoshangabad for technical facilitation, execution and supportive supervision of the CSAM program. The methodology section is further subdivided into three different subsections for ease of understanding.

A1 Tool development-
The process of tool development started in September 2019. The first step in tool development was identifying the theoretical construct from literature review and in-house discussions among investigators. This process led to the identification of three domains, namely performance, adaptability and stability, that could be further contextualized for CSAM management [19,20]. The initial desk review was followed by four field visits and engagement with the field experts to refine identifying the subdomains specific to the context. The identified subdomain within the analytical framework is detailed in Table 1.  After identifying the subdomains, the research team developed the items and transected the items as a vital, essential and desirable matrix for each subdomain. We had developed 36 items initially based on the subdomains. The items were put to discussion with the research and field experts to decide on the validity of the content, structure of the contents, and the length of the questionnaire. The items were proposed to be closed-ended with multiple choices. This review of the initial pool of items reduced the number of questions to 30 once the redundant and repetitive questions were deleted from the list. A few of the questions were changed as per the expert opinion considering its contextual applicability. These questions were further translated into the local language with the help of a field expert proficient in the vernacular language. The translated items were put for pre-test with seven research participants to check for issues in the language, tone, structure, and design of a questionnaire. Incorporating the suggestions from the research participants and the identified need, changes were made to the questions, and the number of items was reduced to 27. A round of pilot tests was undertaken to assess the consistency and reception of questions. Relevant changes were made to the pool of items, and the item numbers were finalized as 25. Another round of pilot tests was undertaken with the items before finalizing the tool to assess clarity and ease of understanding.
Information on the burden of malnutrition for each Anganwadi centre was retrieved from a monthly report submitted at the block office. The number of children with Severe Acute Malnutrition (SAM-weight for height Z-score <-3) and Moderate Acute Malnutrition (MAM-weight for height Z-score <-2) was noted.

A2 -Tool institution
The pre-tested and piloted tool was ready to use by October 2019. It was a community based cross-sectional psychometric health system research. As planned and communicated earlier, the Anganwadi workers (AWW) of Babai block were called for a block-level review meeting on the CSAM programme to the Block Panchayat through the Child Development Project Officer (CDPO) on 11 October 2019. It was decided to conduct the Rasch assessment prior to the review meeting. Out of 217 AWWs in the Babai block, 197 AWWs were present for the exercise. The pre-printed questionnaires were handed out to the AWWs. None of the questions was explained or read out to the research participants even when in doubt, since an individual level of understanding of the question was the mandate for the assessment. It was also communicated with the participants not to have any discussions on the questions. All possible measures, including a time frame of 30 minutes for completing the questionnaire, were set to avoid potential bias in the execution process.

A3 Analysis and model creation
The data was entered in MS Excel in wide data format and was checked for duplication, missing values, outliers and redundancies. The adjusted total burden of malnutrition was calculated for each Anganwadi Centre (AWC-Geo-administrative unit for AWW) by assigning half of the weight (0.50) to a MAM child compared to a SAM child. The analysis comprised 25 items and 197 respondants i.e. Anganwadi Workers (AWWs). The analysis was performed in base-R and r-packages available in open domains. For each item, difficulty parameters with confidence intervals were obtained. Item Characteristic Curves (ICCs) were drawn for each domain item which showed the plot between the probability of correct response for an item on the y-axis and the underlying ability for the domain item on the x-axis. The purpose was to have a visual idea of the discrimination capacity and difficulty of the items clubbed under the domain construct.
A person-item map was plotted through the eRM package. This map displayed the range and position of the item (lower panel) and person parameters (upper panel) on the same axis of latent dimension.
The Rasch model was initially created using responses of 197 participants for 25 items. The data fitting to the Rasch model (Rasch diagnostic) was tested by numeric (Anderson LR and Wald test) and graphical methods. The reason behind choosing the two tests was to look at the model both from a global (all items simultaneously) and an individualistic item wise perspective. The Anderson LR statistic checked for the item bias (differential item functioning globally), while the Wald test looked for the individual item invariance by dividing the sample into two groups. The visual diagnostic was performed by using the Goodness-of-Fit plot.
Here, underlying assumptions of homogeneity (with confidence bend) was visualised by comparing the beta scores of two groups split by median score as a cut-off. Rasch model was scrutinised to check for deterministic patterns by infit MSQ value. All the diagnostics were run on eRM R-package.
A scatter diagram was drawn with the ggplot2 package. Each co-ordinate represented the adjusted burden of Malnutrition in AWC at the x-axis and Rasch value scored by corresponding AWW at the y-axis. The malnutrition burden in a centre was thought off in terms of the total weighted number of SAM and MAM children enrolled in the centre at the time of tool administration. A MAM child was given half of the weight compared to a SAM child for calculating the burden, which ensured the relatively high contribution of SAM into the total burden. This scatter plot was further divided into four quadrants by placing a horizontal and vertical line intersecting at the y and x-axis. These intersecting lines represented the median values of Rasch score (horizontal line) and adjusted burden of malnutrition (vertical line). This exercise led to the formation of four quadrants-Q1 (right lower)-low ability high burden, Q2(left lower)-low ability, low burden, Q3(left upper)-high ability, high burden and Q4(right upper)-high ability, low burden. Figure 1 illustrates the process from construct identification to model creation in a sequential manner.

Ethics issues
The Institutional Human Ethics Committee of AIIMS Bhopal reviewed and approved the study protocol. Committee also granted a waiver of written informed consent. All participants were provided with an information sheet that had details about objectives and procedures. It was assured that performance in the assessment did not have any unfavourable effect on their regular working. There are no competing interests. It is a funded project by UNICEF, Madhya Pradesh.

Results
Out of 217 AWWs in the Babai block, 197 completed the assessment. The average age of participants was 39.8 years, and the average duration of experience as AWW was 11.4 years. The item related to diarrhoeal assessment was found to be as easiest (-2.32, -2.91 to -1.73). Item related to peer assessment consequential action (2.009, 1.669-2.348) was most difficult on item difficulty parameters. The Item Characteristic Curve (ICC) is shown in Figure 2, in which items were listed as per their domain membership with mean item difficulty scores. The items displayed in Figure 2 relate to 'stability' domains that were perceived easy by AWWs and had less discriminatory ability between low and high ability AWWs. On the other hand, items related to adaptability and performance had a higher discriminatory capacity and were perceived as difficult by AWWs.

FIGURE 2: Item characteristic curve showing the monotonic homogeneity of ability on x-axis and probability to resolve at y-axis across the 3 domains of inquiry
Person item maps in Figure 3 show that items were distributed along with the whole range of latent dimensions, thus sensitive enough to capture the whole of the latent construct. However, this sensitivity was lower at very high ability (to the right upper end). Item no 4,21 and 23 mapped the maximum number of participants. The purpose of the Rasch analysis was to assess the difficulty level apart from the discriminatory capacity of items that ICC and the person item map showed. The difficulty level was assessed at item and person levels with the same latent dimension. Next, Table 2 shows the Item difficulty point estimates with their 95% confidence level. The items having beta estimate values with a negative sign connote the relatively easy items compared to items having beta estimates with a positive value 2021   Similarly, person parameters were also identified for 197 AWWs on the same latent dimension as of item parameter.
The descriptive analysis was followed by model diagnostic. Anderson LR test (LR=31.32, df=24, p=0.079) was applied to check the presence of global outliers, which showed the absence of such outliers. Similarly, local outliers were checked through Wald statistics. Twenty-one out of 25 items had a p-value more than the threshold, while four items (item-12,19,24 and 25) had p values less than the threshold (<0.05). However, these items were retained in the final model as they were in alignment with the construct.
The Rasch analysis and model diagnostic was followed by allocation of all the AWWs into one of the four quadrants created as under-low ability -high burden (P1-41 AWWs), low ability-low burden(P2-44 AWWs), high ability-high burden (P3-37AWWSs) and high ability -low burden(P4-75 AWWSs). This allocation of AWW as per the ability in CSAM domains and adjusted burden of the malnutrition (differential weightage of MAM and SAM child ) was shown through a visualization ( Figure 4). All the AWWs were categorized into one of the four quadrants. About 38% of AWWs had high ability and low burden (low priority for monitoring with an inherent assumption of correct reporting as a function of ability). However, about 1 in 5 AWWs had a low ability and reported a high burden of SAM (P1), and about 1in 4 AWWs had a low ability and low reported burden of SAM (P2). These P1 and P2 segments were thought of as a priority segment for supportive supervision. This conceptual priority assignment was operationalized in 2 ways-enhanced relative allocation of monitoring visits to P1 and P2 segments and identification of difficult items from poor performing domains (by looking at item easiness parameters ) with individualistic /group based capacity building.

FIGURE 4: Quadratic allocation of AWWs as per ability (determined through Rasch) and adjusted burden of malnutrition.
The yellow dashed line intersects the x and y-axis at their median value. The SAM number was colour coded, while the differential size of the dots showed MAM numbers.

Discussion
There has been a global shift in the way in which the process of supervision was perceived, from authoritative mechanistic supervision ("do as I say") to supportive facilitatory ("do as we agree upon") supervision [21,22]. To embark on this paradigm, the supervisor should have a fair orientation of the strength and scope of improvement of the supervisee in his individualistic capacity before and during interaction in an objective manner. In practice, it is often difficult to achieve due to previous cumulative predispositions and the absence of tangible performance measures which have the factual and accurate construct of performance discrimination [23,24]. Rasch modelling may attempt to overcome both concerns as it assigns objectivity to the whole evaluation framework and has an innate discriminant capacity for ability estimations [18]. In addition, any ability estimation has the meaning only if it can be translated into the extent of task accomplishment. With the argument, any monitoring strategy that simultaneously contemplates ability and an intended task may offer more vital cues for actions than one that focuses only on ability or intended task [23][24][25].
The aptness of any measurement depends on the notion of a single construct (without contamination) equidistance and linearity in measurement in general. This is relatively easy to achieve in physical measurements by concatenation. In the world of social science/psychometrics, this measurement allocates the numeral to events as per some pre-defined rubrics. The accumulation of these numbers is perceived as magnitude for that event [26]. This simpler and linear calculation of the magnitude of an event will be problematic in social sciences since the observations do not fall equidistant from each other, and the positioning of a score has deeper meanings than its numerical value. A linear assessment method might offer a flawed interpretation of the results and understanding the events in place [18].
Several supervisory insights may be derived through this strategy-At first, in the high burden and low ability quadrant, we will know whether this burden is an actual burden or an anthropometric classification error on the fieldworker side? This question becomes more relevant in the presence of high discordance in MAM and SAM numbers. Although the discordance between SAM and MAM is reported by other population-based surveys and plausible explanations for the same is offered, the transition from MAM to SAM should be perceived from the CAS (Complex Adaptive System) perspective where system outputs are the non-linear sums of underlying processes, and this transition should be seen on a continuous pathophysiological spectrum from normal to SAM child and not as disconnected dichotomized entity. In the Quadrant plot, the absolute SAM burden corresponds to the marker's colour, thus adding to another dimension in the plot.
Second, the low burden and low ability quadrant also conveys the same notion as mentioned above; whether are we missing some eligible SAM children here? The peripheral staff serves as 'gate-keeper of CSAM management ecosystem through identification of SAM/MAM children. Thus, the delayed entry or missing cases may adversely affect the program performance, leading to preventable child mortality. Third, the high ability quadrants also have a high and low burden of the problem. These quadrants may serve as benchmarks for estimating the true burden but with some caveats. The cut-offs are derived from the median scores; thus, these quadrants are the outcome of rank order arrangement in two dimensions.
Although this argument assigns more credibility to allocate the 'grey-zone markers into respective quadrants, yet farthest points at the quadrant plots should be interpreted with some caution. The standard error for the Rasch score is maximum at the extreme ends and minimum at the middle, which is the opposite of raw scores. Ability estimation accordingly may have more reliability at the middle compared to extreme ends [27]. This inherent limitation of Rasch transformation may not substantially affect the person-item map; the item density is more at the middle and none at the end. Finally, the difficulty scores having agreements between predicted difficulties by investigators and actual difficulties are the definite domains in which capacity building is required. The RCoE has organized the refresher training for AWWs targeting specifically these domains both as groups and on a one-to-one basis during monitoring visits. Apart from the innate weakness of lesser discrimination at both ends of the ability, the results should be interpreted with a caveat that the process of transformation of ability into Rasch score preserves order from a unidimensional perspective only while real-world ability might be multidimensional ( socio-environmental aspects, for instance ) in nature which is not accounted for while estimating the scores. However, all the responders belonged to the same socio-geographical strata and had comparable formal qualifications. Also, some participants may find it difficult to recall the correct numeric options in the presence of distractors.

Implications for policy and practice
The investigators have crafted the following intervention framework ( Figure 5) based on insights received from this data-driven monitoring mechanism. We were able to identify the domain requiring more attention and develop a more focused monitoring strategy catering to the differential needs of the frontline workers. Based on this experience, we further propose exploring the psychometric analysis tools (such as Rasch scores) to identify specific activities to be focused on for an individual as per his or her ability. It also emphasizes consideration of the burden of given health problems in deciding monitoring and supervision plan. Thus, a data-driven framework proposed in this study will help prioritize monitoring and supervision of community-based management of acute malnutrition outreach sites. This strategy will ensure mentoring in identified individual-specific domains. The same framework can be suitably modified and adapted for monitoring and supervision of other public health programs.

Conclusions
In summary, this strategy offers some distinct advantages compared to raw scored based ability estimation as discussed above. However, the effectiveness of the intervention framework will be achieved following the realistic application of focused monitoring and the improvement of the target indicators. 5. If a new child has come to Anganwadi Center during CSAM session, how will you confirm the age of the child.

Appendices
a. By measuring the height of the child b. By asking the mother 15. If a SAM child was staying at her mother's home, which is your duty station and have shifted her residence to her father's job location, How will you respond?
a. Report it to the supervisor b. Report it to the NRC c. Report it to the concerned Anganwadi at the child's current residence. 16. What is incorrect about conducting the appetite test a. The mother has to wash her hands before feeding the child b. The child needs to be forced to complete the therapeutic food c. It has to be made sure that the child was not fed for the last 2 hours.