This edition of the SOHS, like previous editions, has been created through a synthesis of findings from separate research components using distinct methods.
To better facilitate this synthesis and make the process more transparent, ALNAP developed a study matrix with indicators for each of the DAC criteria. This ensured a common conceptual framework and set of questions to be applied within each research component. The study matrix is available in annex 3.
For the first time, all of the interviews (key informant interviews and case studies) were recorded and transcribed verbatim. To ensure consistency between the various elements of the research, and to allow for comparison of all the data from different elements on the same topic, these interview transcripts and the evaluation synthesis were then coded in MaxQDA using a common coding framework derived from the study matrix.
In line with recommendations from The State of the Humanitarian System Methods Group, this edition makes a specific attempt to increase the amount of information collected in humanitarian operations, and in particular to increase the amount of information collected from aid recipients. This edition includes:
346 interviews with individuals in five country case studies and 17 other countries (compared to 201 interviews in four countries in 2015).
Responses from 5,000 aid recipients in five countries (compared to 1,189 aid recipients in four countries in 2015).
The element of the report that assesses the performance of the system over the period 2015–17 is based on a synthesis of the following components, each of which uses distinct methods. In addition, the SOHS was supported by Ground Truth Solutions, who kindly shared recent data from their new Humanitarian Voice Index, and the Peer2Peer support team (formerly STAIT), who allowed the SOHS to access P2P reports for the period 2015–17.
Summary information for each research component of the system performance assessment is provided below. For more information, please refer to Chapter 3: Components, method and approach.
A synthesis of relevant evaluation findings from the period January 2015 to December 2017 forms one component of the evidence base for the SOHS 2018. Around 170 evaluations were considered, based primarily on a search of the ALNAP evaluation database. Of these, 121 of the most relevant were included in the synthesis process. These were then scored for quality and depth of relevant evidence, and the synthesis process was organised in such a way as to prioritise findings from evaluations with the highest evidence scores.
For more information, please refer to Chapter 3: Components, method and approach
Full case studies were conducted in five countries (Bangladesh, Kenya, Lebanon, Mali and Yemen). In addition, team members interviewed individuals and focus groups in a number of other countries (Afghanistan, Cameroon, Central African Republic (CAR), Chad, Colombia, Côte d’Ivoire, DRC, Greece, Haiti, Nepal, Nigeria). In total, 346 people were interviewed through 171 bilateral or focus group meetings. Case study countries were chosen to provide a sample with geographical diversity (across regions) and contextual diversity (across the three main contexts considered in the report).
Of the total, 38% of interviewees were from Africa, 18% from the Middle East and North Africa, 23% from Asia, 20% from the Americas and 1% from Europe. As noted above, it is not entirely accurate to say that any country represents only one crisis context. However, we can say, broadly, that 44% of interviewees were from disasters linked to natural phenomena (Colombia, Haiti, Kenya and Nepal), 33% from conflict-affected areas (Afghanistan, Cameroon, CAR, Chad, Côte d’Ivoire, DRC, Mali, Myanmar, Nigeria and Yemen) and 23% from refugee-hosting contexts (Bangladesh, Greece and Lebanon).
The interviewees were selected to be representative of the diversity of organisations taking part in humanitarian action and its coordination. However, as the key informant section of the research was oriented towards HQ staff and international organisations, the case study interviews aimed to give a higher profile to national organisations and to aid recipients. As a result, local actors (national and local NGOs, national and local authorities) represent the larger group, with 36% of interviewees, aid recipients 34% and international actors (international organisations and INGOs) 30%.
Interviewee selection was partially purposive (inasmuch as interviewers attempted to interview a certain number of people from each category) and partially by convenience (interviewers were working on a short timeframe, and within categories tended to interview people who were available and qualified to speak on the situation). Aid recipients interviewed were selected on a convenience basis. Although the study team tried to achieve an appropriate level of diversity in terms of gender and age criteria this was not always possible and, in some cases, very few or no women were reached. Overall, 42% of interviewees were women and 58% men.
For more information, please refer to Chapter 3: Components, method and approach
The key informant interviews were designed to be as representative as possible. The team aimed to cover all of the major types of actor within the sector: UN agencies, the Red Cross and Crescent Movement (RCRC), international NGOs, national NGOs, donors, development banks and other multilaterals, think-tanks, academia, the media, affected governments and commentators. The team also sought out respondents at different levels of the system and of the organisations and bodies outlined above – from senior leaders to those working at functional, operational or operational coordination levels in humanitarian programmes. The team also used a snowball approach, asking interviewees to recommend people who had differing views or who represented a particular aspect of a discussion, or who had specific technical or geographic expertise. In all, 153 people were interviewed.
For more information, please refer to Chapter 3: Components, method and approach
For this State of the Humanitarian System report, ALNAP again commissioned GeoPoll to carry out telephone surveys in DRC, the Horn of Africa (Kenya and Ethiopia), Iraq and Afghanistan. The SOHS 2012 was one of the first major surveys of aid recipients in humanitarian action, reaching 1,104 aid recipients in DRC. This iteration surveyed 5,000 aid recipients across the five countries. These countries were chosen to represent humanitarian responses in a variety of geographical areas and contexts. The selection was partially influenced by the choice of case study countries, and aimed to include more conflict contexts to make up for their under-representation in the case studies.
For more information, please refer to Chapter 3: Components, method and approach
An online survey aimed at humanitarian staff working in country programmes, which received 1,170 responses.
The practitioner and government surveys for this iteration of the SOHS were updated to ensure that the questions asked covered all of the areas in the study matrix, but without sacrificing the comparability of the survey over time. The surveys were translated into French, Spanish and Arabic and uploaded to SurveyMonkey for dissemination. The ALNAP team prepared a dissemination plan mapping local NGO networks and national disaster management agencies (NDMAs), liaised with the SOHS Strategic Advisory Group to spread the word and supported ALNAP Members that are operational with messaging and channels so that the surveys could reach staff on the ground. Adverts were placed on ReliefWeb and Dawns Digest, and the survey was also promoted with social media campaigns. The surveys were open for six months (from August 2017 to January 2018) and were completed by 1,170 practitioners and 38 government representatives from a wide geographical spread. The ALNAP team cleaned and prepared all the answers collected through pivot charts in spreadsheets, to allow for cross-cutting and analysis of the data received. Datasets from past SOHS surveys were incorporated to allow for comparison of responses over time.
For more information, please refer to Chapter 3: Components, method and approach
The literature review was used mainly to provide information on specific areas not captured fully by other means in the Study Matrix. These related primarily to the functions of the humanitarian system beyond the provision of humanitarian assistance, including protection and resilience. The rationale for considering these elements in a separate literature review is that, while these activities should be addressed in humanitarian evaluations, there is some evidence that they do not receive consistent attention in humanitarian activities, and so will not be adequately covered in programme evaluations.
For more information, please refer to Chapter 3: Components, method and approach
