Swedish schools are legally obliged to have anti-bullying policies. Many commercial programs are available. During the study of the practical experiences of working with eight (named) programs, the research team was forced to modify the main question: whether schools that use specific anti-bullying programs were more effective than schools without such programs. All nominally designated treatment schools, as well as non-treatment schools, were found to be contaminated. This discovery overthrew the planned quasi-experimental design approach. Instead of attempting to evaluate anti-bullying programs as complete entities, an evolved evaluation strategy was developed to evaluate an array of program components, use of which was assessed on the basis of detailed qualitative data. Hierarchical cluster analysis was used to designate intervention types and schools with similar approaches. The purpose of using cluster analysis in this context was to construct empirically based classifications of types of interventions and approaches to bullying.
IMPLICATIONS
A mixed methods approach, based on questionnaires and interviews, was used in the evaluation. This approach provides a number of significant insights into evaluation practice. A major difficulty in program evaluations is associated with monitoring program fidelity. Rather than taking program fidelity for granted, as well as ‘‘pure’’ control conditions (which is often a prerequisite in experimental controlled trials or in randomized controlled trials), the mixed methods approach provide data about whether, in this case the schools, actually use the constituent intervention elements (program components) in real-world conditions, i.e. more realistic and reliable information about what schools actually do compared to evaluation approaches where use of program elements in a prescribed manner is taken for granted. Through the gathering of extensive qualitative data our evaluation revealed a profound program contamination. The solution to this evaluation challenge was to focus on program elements, or components. Since component use is highly context dependent, the array of components was identified on the basis of the context of everyday practice in all participating schools. One implication of this is that any comprehensive program evaluation ought to consider contextual circumstances in order to gain detailed data about actual component use. Furthermore, results from this evaluation emphasize the importance of gathering contextual data because an effective component in one circumstance may prove ineffective or ever counter effective in outwardly similar circumstances.