Qualitative evidence synthesis for universal curriculum approaches

1.1.4.1. Included studies

In total 47,322 references were identified through systematic searches. Of these, 558 references were considered relevant, based on title and abstract, to the protocols for universal approach interventions and were ordered for full-text review. Nine articles were ordered but not received, with the remaining 549 reviewed at full-text. Out of the 549 studies reviewed at full-text, 148 references were included and 401 were excluded. Of the 148 included references for universal interventions, 10 studies were included in this qualitative review only. An additional 3 studies were included in this qualitative review and the quantitative effectiveness review (see Evidence Review B).

1.1.4.2. Excluded studies

See Appendix F for full list of excluded studies.

1.1.6.1. Summary of themes and sub-themes

Iterative aggregation of codes generated the following key themes and sub-themes (table 3)

Table 3

Summary of themes and sub-themes.

1.1.6.2. Summary of qualitative findings

The qualitative evidence for review questions 3.2 and 3.3 are presented in table 4.

Table 4

Summary of qualitative findings.

See Appendix E for full GRADE-CERQual tables.

Are the results/findings from individual syntheses supportive or contradictory?

The results from the quantitative data and qualitative data are somewhat contradictory. There was a general perception among children and teachers that the interventions were beneficial for those receiving them and improved their social and emotional wellbeing. However, there were relatively few interventions that showed significant improvements across the outcomes measured in these reviews. This was specifically highlighted by a RCT and process evaluation of Zippy’s Friends (Sloan 2018), which reported that teachers believed that interventions were beneficial for children, but this was not necessarily reflected in impact evaluation of the intervention.

Does the qualitative evidence explain why the intervention is/is not effective?

The qualitative evidence highlighted time constraints and difficulties with timetabling as a barrier to successful implementation of interventions. A considerable number of interventions lasted several weeks, which may have contributed to a lack of significant effect demonstrated by multiple interventions across the measured outcomes. Additionally, distractions in the school setting were highlighted in the qualitative evidence as a barrier to practicing mindfulness. As interventions tended to be delivered in classrooms this may have contributed to the lack of significant effect shown for mindfulness interventions for all outcomes except anxiety / depression and academic outcomes.

Does the qualitative evidence explain differences in the direction and size of effect across the included quantitative studies?

The qualitative data did not provide clear explanations for the size of effect variations in the quantitative data. However, it did highlight areas that could contribute to making universal curriculum interventions successful or unsuccessful. For example, appropriate training for staff and a strong peer community among children were identified as elements for intervention success. Lack of allocated time for interventions and timetabling issues were factors likely to make interventions unsuccessful.

Which aspects of the quantitative evidence were/were not explored in the qualitative studies and which aspects of the qualitative evidence were/were not tested in the quantitative studies?

The limited overlap between the quantitative and qualitative findings for this review does not make any meaningful integration useful. This is predominantly because the qualitative evidence is very much focussed on process related understanding of very specific programmes. Regarding the Perceptions of Impact theme, the committee highlighted perceptions of children, teachers and parents are not objective measures for intervention efficacy. Therefore, there is limited benefit to integrating these qualitative and quantitative data, as was planned. The themes are very useful in understanding why the universal curriculum interventions worked (or did not work), but other than at a very superficial level they unable (without substantial speculation) to explain the pupil level outcomes.

1.1.8.1. The outcomes that matter most

Qualitative outcomes were perspectives, values, beliefs, experiences and attitudes relating to the acceptability and barriers and facilitators of:

• teachers and practitioners delivering interventions
• children and young people receiving interventions.
• parents/carers of children and young people receiving the interventions

The committee agreed that it was very important to contextualise some of the subthemes formed from the qualitative evidence base. For example, perceptions of the programme were self-reported and should not be used as measures of programme effectiveness. Additionally, some of the findings from the ‘training, coaching and feedback’ sub-theme were specifically linked to the PATHS intervention and had limited generalisability. Furthermore, teachers liking the prescriptive nature of interventions (identified in the ‘programme structure’ sub-theme) only tended to apply to those who were new to delivering the intervention. Issues surrounding over-prescriptiveness have been identified in the wider literature. Finally, the committee identified ’programme implementation’ as an important sub-theme, which was explored extensively in process evaluations conducted by the Education Endowment Fund. These process evaluations were captured in grey literature searches. Overall, the committee agreed that the themes identified in this review mostly resonated with their experience of these programmes and the way that teachers and children and young people experience them.

1.1.8.2. The quality of the evidence

The committee acknowledged there was a limited evidence base of qualitative data. The evidence base comprised of thirteen UK-based studies, which captured nine universal interventions. The quality of evidence varied considerably, with GRADE-CERQual ratings ranging from very low to high. Methodological limitations and concerns with evidence adequacy were the main reasons for downgrading quality. The committee noted that grey literature inclusions from the Education Endowment Foundation website were important additions to the evidence base. As a result of these factors, the committee felt that the qualitative evidence should be used to contextualise and aid the interpretation of the quantitative evidence, but should not be the sole basis for any recommendations. Therefore no recommendations were made directly from this review.

1.1.8.3. Benefits and harms

The committee agreed that many of the findings presented in the analysis mirrored findings from their own experience. Regarding the ‘perceptions of the programme’, it was recognised that when considering implementation of an intervention, conditions for success would be better if the views of children are taken into account. Additionally, equipping teachers to understand benefits of interventions and involving and engaging parents and carers throughout the process also promotes conditions for success. Important contextual considerations should be taken into account when interpreting these findings. Primarily, findings were self-reported, post-hoc perceptions of programme impact and should not be considered an objective measure of programme effectiveness. The committee were not minded to make recommendations, as the evidence was relatively intervention specific.

1.1.8.4. Cost effectiveness and resource use

Cost-effectiveness and resource use is captured in the committee discussion of Evidence Review B.

1.1.8.5. Other factors the committee took into account

Other factors the committee took into account are captured in the committee discussion of Evidence Review B.

Database name: Medline

Database: Ovid MEDLINE(R) <1946 to September 21, 2020>

Search Strategy:

------------------------------------------------------------

1. ((Social or emotional or social-emotional or socio or socio-emotional or pro-social or prosocial) and (wellbeing or well-being or wellness or learn* or competenc* or skills)).ti,ab. (77780)
2. ((SEL or SEAL or SEBS or EWB or EMHWB) and (school* or class* or curricul* or intervention* or program*)).ti,ab. (1620)
3. (“social learner*” or “social learning”).ti,ab. (2467)
4. (resilien* or coping).ti,ab. (68742)
5. Adaptation, Psychological/ or Resilience, Psychological/ (99149)
6. (self-control or “emotional regulation” or self-aware* or self-efficacy or self-regulat* or selfconfiden* or self-management or self-esteem or self-concept or “emotional intelligence” or mindful*).ti,ab. (83731)
7. Emotional Intelligence/ (2154)
8. exp Self Concept/ (110417)
9. Emotional Adjustment/ or Social Adjustment/ (23951)
10. ((social or interpersonal or communication or relationship*) adj2 (skill* or competence* or attribute*)).ti,ab. (19797)
11. (friendship* or friends).ti,ab. (26352)
12. ((social or peer or peers) adj2 (group* or network*)).ti,ab. (25987)
13. empathy.ti,ab. (9854)
14. (“social awareness” or socialisation or socialization or “social interaction*” or “social inclusion”).ti,ab. (23520)
15. Social Skills/ or Social Behavior/ or Social Values/ (73492)
16. (“personal development” or “youth development”).ti,ab. (2191)
17. (“decision making” or “problem solv*” or problem-solv*).ti,ab. (124516)
18. Decision Making/ (95890)
19. Problem Solving/ (24899)
20. (bully* or bullies or anti-bully* or “anti bully*” or antibully* or cyber-bully* or “cyber bully*” or cyberbully* or victimis* or victimiz* or stigma or anti-stigma or “anti stigma” or antistigma or prejudice*).ti,ab. (34249)
21. (delinquen* or anti-social or “anti social” or antisocial or “conduct disorder*” or “risky behavio*” or “problem behavio*” or (behavio* adj problem*)).ti,ab. (36063)
22. (((substance or drug* or alcohol) adj3 (“use” or abuse or misuse)) and (prevent* or reduc*)).ti,ab. (50561)
23. ((exclu* or expulsion or expel* or absent* or truant* or truancy or conflict or violent or violence or disengage*) and school*).ti,ab. (13080)
24. bullying/ or cyberbullying/ or problem behavior/ (6476)
25. ((school* or academic) adj2 (achieve* or attain* or engage* or progress* or motivat* or connectedness or belonging)).ti,ab. (8041)
26. Mental Health/ (39108)
27. (mental adj2 (health or wellbeing or well-being or “well being” or wellness)).ti,ab. (120613)
28. ((psychological or “psycho social” or psycho-social or psychosocial) adj2 (wellbeing or “well being” or well-being)).ti,ab. (10497)
29. (anxiety or anxious or depression or depressed or depressive or stress).ti,ab. (1040473)
30. or/1-29 (1763666)
31. (“Aban Aya” or “Academic and Behavioural Competency Program*” or “Active Citizens in Schools” or ACIS or “Adolescent Decision Making Program*” or “ALERT plus” or “Alcohol Education Package” or “Alcohol Education Program*” or “Alcohol Screening and Brief Intervention” or “All Stars” or “Al’s Pals” or “Alternatives to Trouble” or “Amazing Alternatives” or “Anti-bullying Program*” or “Attention Academy” or “Aussie Optimism” or BARR or “BBBS Ireland” or “Be the Best You can Be” or “Beat Bullying” or Beatbullying or “Befriending Intervention” or BeyondBlue or “Big Brothers Big Sisters” or “Bounce Back” or “Boys and Girls Club” or “Breathing Awareness Meditation” or “Building Assets Reducing Risks” or “Building Resiliency and Vocational Excellence” or “Bully Proofing” or Bullyproofing or “Bullying Eliminated from Schools Together”).ti,ab. (31676)
32. (CAPSLE or CASEL or “Caring School Community” or CharacterPlus or “Child Development Initiative” or “Circle Time” or “Classroom Centred Intervention” or “Classroom Centred Program*” or “Class-wide Function-based Intervention” or “Climate Schools” or Climb-UP or CMCD or “Coalition for Youth Quality of Life” or “Comer School Development Program*” or “Communities that Care” or “Community of Caring” or “Competence Support Program*” or “Competent Kids Caring Communities” or “Conscious Coping” or “Consistency Management and Cooperative Discipline” or “Coping Koala” or “Coping Power” or “Counsellor Peers” or “Creating a Peaceful School Learning Environment” or Cues-ed or CSRP or “Cultivating Awareness and Resilience in Education”).ti,ab. (501)
33. (“Early Risers” or “EiE-L” or “Empathic Discipline” or “Empower Youth” or “Engage in Education” or “Expect Respect” or “Expeditionary Learning” or “Facing History and Ourselves” or “Families and Schools Together” or “Family Check-up” or “Family School Partnership” or “Family SEAL” or “Fast Track” or “FearNot*” or “First Steps to Success” or “Formalised Peer Mentoring” or “Foundations of Learning” or “Fourth R-Skills” or “Fourth Step” or “Friendly Schools” or “FRIENDS program*” or FSP or “Gang Resistance Education and Training” or Gatehouse or GBG or “Get Wise” or “Girls First” or “Going for Goals” or “Going Places” or “Good Behaviour Game” or “Grades Attendance and Behaviour” or “Guided Self-change” or HASSP or “Head Start” or “healthy active peaceful playgrounds” or “Healthy for Life” or “Healthy Futures” or “Healthy Lifestyles” or “Healthy Minds in Teenagers” or “Healthy Relationships Training Program*” or “Healthy Schools and Drugs” or “Here’s Looking at You” or HighScope or “Home and School Support Program*” or “How to Thrive” or “I Can Problem Solve” or ICPS or “ICAN Kids” or “Improving Social Awareness” or “Incredible Years” or “Inner Explorer” or InnerKids or “Inspiring Futures” or “Interpersonal Cognitive Problem Solving Skills” or “In:tuition” or “ISA-SPS” or Jigsaw).ti,ab. (13869)
34. (“Keepin* It REAL” or “Kia Kaha” or KiVa or “klar bleiben” or “Knightly Virtues” or “Know Your Body” or “Learning for Life” or “Learning to BREATHE” or “Lessons for Living” or “Lessons in Character” or “Life Skills Program*” or “Life Skills Training” or Lift or “Linking the Interests of Families and Teachers” or “Lions Quest” or “Living with a Purpose” or “Love in a Big World” or LST or “Master Mind” or “Match Model” or “Michigan Model for Health” or “Middle School Success” or “Midwest* Prevention Project” or “Millennium Volunteers” or “Million Dollar Machine” or “Mind Up” or MindUP or MindfulKids or “Mindfulness in Schools” or MISP or “Mood Gym” or “My Character” or “My Teaching Partner” or “New Beginnings” or Narconon or OBPP or Olweus or “Open Circle” or “Op Volle Kracht” or “Over to You”).ti,ab. (11171)
35. (Paths or PATHstoPAX or “Paws B” or “Peace Builders” or “Peace Works” or “Peacemaking Skills for Little Kids” or “Peer Mentoring” or “Peer Acceleration Social Network” or “Penn Resiliency Program*” or “Personality Risk Factors” or PESSOA or Playworks or Ploughshares or “Positive Action” or “Positive Alternative Learning Support” or “Positive Adolescent Life Skills” or “Positive Youth Development Program*” or “Preparation through Responsive Education” or “Primary SEAL” or “Prime for Life” or “Proactive Classroom” or Pro-ACT or “Problem Solving Program*” or Progetto or “Project A.T.T.E.N.D.” or “project ALERT” or “project CHARLIE” or “Project Northland” or “Project Pride” or “project SMART” or “Project Based Learning” or “Project STAR” or “Promoting Alternative Thinking Strategies” or “Puppets for Peace” or “Pyramid Project” or “Raising Healthy Children” or RCCP or ReachOut or “Reaching Adolescents for Prevention” or “Reading Apprenticeship” or “Reading, Writing, Repect and Resolution” or “Recognizing, Understanding, Labeling, Expressing and Regulating Emotions” or “Reconnecting Youth” or REDI or “Resilience Program*” or “Resilient Families” or “Resolving Conflict Creatively” or “Respect Program*” or “Responsive Classroom” or “Risk Training Skills” or “Rochester Resilience Program*” or “Resourceful Adolescent Program*” or “Roots of Empathy” or Rtime or Ruler).ti,ab. (19411)
36. (“Safe and Civil Schools” or “Safe Dates” or “SafERteens” or “Say Yes First” or SBIRT or “Schoolbased Resilience Intervention” or “School Health and Alcohol Harm Reduction Project” or “Schoolwide Positive Behavioural Interventions and Support” or “Second Step” or SS-SSTP or “Secondary SEAL” or “Seattle Social Development Project” or “SEED Scotland” or “Self-determination Program*” or “Self-management and Resistance Training” or “Service Learning” or “SFP10-14” or SHAHRP or “Siblings are Special” or SIBS or “Skills for Adolescence” or “Skills for Change” or “Skills for Success” or SingUp or “Social Competence Training” or “Social Decision Making” or “Social Norms” or “Social Problem Solving Skills” or “Social Skills Group Intervention*” or “Social Skills Training” or “South Carolina Program*” or “Smart Moves” or “S.S.GRIN” or SST or “Steg fur Steg” or STAMPP or “STARS for Families” or “Start Taking Alcohol Risks Seriously” or “Staying Calm” or “Step II” or “Steps towards Alcohol Misuse Prevention” or “Talk about Alcohol” or “Step-by-Step” or “Steps to Respect” or “Stop Breathe Be” or “Strengthening Families Program*” or “Strengths Gym” or “Stress Inoculation Training” or “Stress Management Intervention” or “Student Success Skills” or “Student Success through Prevention” or “Student Threat Assessment” or “Success for Kids” or SWPBIS or SWPBS or “Teach Team” or “Teen Outreach Program*” or “Teen Talk” or “Theatre in Education” or “The GOOD life” or “The Incredible Years” or “Think Feel Do” or “Think Well, Do Well” or “Too Good for Violence” or “Tools for Getting Along” or “Tools of the Mind” or “Towards no drug abuse” or “Transition Mentoring” or “Tribes Learning Communities” or “UK Resilience Program*” or “Unique Minds” or ViSC or “Wise Mind” or Woodrock or YogaKid* or “Yo Puedo” or “You Can Do It!” or “Youth Development Project” or “Youth Matters” or “Zippy’s Friends” or “21st Century Community Learning” or “4Rs”).ti,ab. (32480)
37. (PSHE or “personal social health” or PSE or “personal and social education” or SMSC or “spiritual moral social and cultural”).ti,ab. (2268)
38. (“positive behavio* intervention*” or “positive behavio* support” or PBIS).ti,ab. (187)
39. (“school-wide positive behavio* support*” or SWPBS).ti,ab. (4)
40. “relationships and sex education”.ti,ab. (4)
41. or/31-40 (110898)
42. 30 and 41 (14752)
43. (mindful* or meditat* or yoga).ti,ab. (12908)
44. Mindfulness/ or Meditation/ or Yoga/ (7952)
45. “life skills”.ti,ab. (903)
46. “motivational interview*”.ti,ab. (3378)
47. Motivational Interviewing/ (1888)
48. ((brief or opportunist* or concise or short or direct) adj3 (counsel* or advice* or advise* or advisor* or therap* or support* or guide* or guidance* or intervention*)).ti,ab. (31871)
49. ((behaviour* or behavior* or cognitive) adj3 (technique* or therap* or chang* or modify or modifies or modifying or support* or intervention* or session* or program* or workshop*)).ti,ab. (119612)
50. counseling/ or directive counseling/ or child guidance/ or psychology, adolescent/ (52429)
51. Behavior Therapy/ or Cognitive Behavioral Therapy/ (52716)
52. (skills adj1 (train* or teach* or educat* or develop*)).ti,ab. (9521)
53. ((peer or pastoral or teacher*) adj2 (educat* or support* or group* or led)).ti,ab. (11763)
54. (prevent* and (intervention* or program*)).ti,ab. (210438)
55. “intervention program*”.ti,ab. (13835)
56. “social and emotional learning program*”.ti,ab. (28)
57. “play therap*”.ti,ab. (377)
58. (“mental health” adj3 (intervention* or program*)).ti,ab. (5492)
59. ((Wellbeing or “well being” or well-being) adj3 (intervention* or therap*)).ti,ab. (1042)
60. ((HIIT or fitness or “physical activity”) adj2 (intervention or program*)).ti,ab. (4796)
61. ((questionnaire* or survey* or self-report* or “self report*” or assessment*) adj3 (school* or class or classroom* or pupil* or student* or teach*)).ti,ab. (24811)
62. or/43-61 (484103)
63. (classroom* or “whole class*” or whole-class*).ti,ab. (14258)
64. ((multi*-component or multicomponent or “multi* component” or universal or brief or “group based” or group-based or groupbased or “group work*” or group-work* or groupwork* or “small group*” or small-group* or targeted) and (intervention* or program* or project* or pilot* or initiative* or approach* or activit* or lesson* or curricul*)).ti,ab. (208759)
65. (“whole school*” or whole-school* or wholeschool* or “school wide” or school-wide or schoolwide or “school based” or school-based or schoolbased).ti,ab. (11813)
66. (school* adj3 (ethos or culture or life or environment or governance or policy or policies or leadership or SLT)).ti,ab. (6050)
67. (school* and (intervention* or program*)).ti,ab. (67253)
68. or/63-67 (287833)
69. 62 and 68 (62272)
70. 30 and 69 (26390)
71. (school* or pupil* or student* or teach* or curricul* or lesson* or learner* or learning or syllabus).ti,ab. (795925)
72. (((city or technical) and (academy or academies or college*)) or sixth-form* or “sixth form*” or “6th form*” or “lower six*” or “upper six*” or “post 16” or post-16 or “further education”).ti,ab. (4912)
73. (“secure children* home*” or “young offender* institution*” or “secure training cent*” or “secure school*”).ti,ab. (52)
74. (“year one” or “year 1” or “year two” or “year 2” or “year three” or “year 3” or “year four” or “year 4” or “year five” or “year 5” or “year six” or “year 6” or “year seven” or “year 7” or “year eight” or “year 8” or “year nine” or “year 9” or “year ten” or “year 10” or “year eleven” or “year 11” or “year twelve” or “year 12” or “year thirteen” or “year 13” or “key stage one” or “key stage 1” or “key stage two” or “key stage 2” or “key stage three” or “key stage 3” or “key stage four” or “key stage 4” or “key stage five” or “key stage 5” or KS1 or KS2 or KS3 of KS4 or KS5 or “grade one” or “grade 1” or “grade two” or “grade 2” or “grade three” or “grade 3” or “grade four” or “grade 4” or “grade five” or “grade 5” or “grade six” or “grade 6” or “grade seven” or “grade 7” or “grade eight” or “grade 8” or “grade nine” or “grade 9” or “grade ten” or “grade 10” or “grade eleven” or “grade 11” or “grade twelve” or “grade 12” or “first grade” or “1st grade*” or “second grade*” or “2nd grade*” or “third grade*” or “3rd grade*” or “fourth grade*” or “4th grade*” or “fifth grade*” or “5th grade*” or “sixth grade*” or “6th grade*” or “seventh grade*” or “7th grade*” or “eighth grade*” or “8th grade*” or “ninth grade*” or “9th grade*” or “tenth grade*” or “10th grade*” or “eleventh grade*” or “11th grade*” or “twelfth grade*” or “12th grade*”).ti,ab. (105865)
75. curriculum/ or schools/ or teaching/ or school health services/ or school nursing/ or school teachers/ (169056)
76. or/71-75 (932670)
77. (medical or medicine or dental or dentist* or doctor* or physician* or nursing or “teaching hospital*” or undergraduate* or graduate* or postgraduate* or preschool* or pre-school* or nursery or “higher education” or university or universities).ti,ab. (2254098)
78. 76 not 77 (599715)
79. exp Child/ or exp Child Behavior/ or Child Health/ or Child Welfare/ or Child Development/ (1934624)
80. Adolescent Behavior/ or Adolescent/ or Adolescent Health/ or Adolescent Development/ (2036366)
81. (child* or adolescen* or kid or kids or youth* or youngster* or minor or minors or underage* or under-age* or “under age*” or “young person*” or “young people” or pre-adolescen* or preadolescen* or pre-teen* or preteen* or teen or teens or teenager* or juvenile* or boy or boys or boyhood or girl or girls or girlhood or schoolchild* or student* or pupil* or “school age*” or schoolage* or schoolage*).ti,ab. (1963224)
82. or/79-81 (3744899)
83. 78 and 82 (289372)
84. 42 or 70 (39392)
85. 83 and 84 (12583)
86. limit 85 to english language (12029)
87. limit 86 to (letter or historical article or comment or editorial or news or case reports) (183)
88. 88 86 not 87 (11846)
89. limit 88 to yr=“2007 -Current” (8286)
90. limit 89 to ed=20190924-20200901 (980)