oru.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
The Problem of Program Contamination in Evaluation: An Example From an Evaluation of Anti-bullying Programs in Sweden
Örebro University, School of Law, Psychology and Social Work.ORCID iD: 0000-0001-6465-0545
Örebro University, School of Law, Psychology and Social Work.ORCID iD: 0000-0002-3134-5461
2013 (English)In: Evaluation Practice in the Early 21st Century. Washington DC, USA, October 14-19, 2013, 2013Conference paper, Oral presentation with published abstract (Refereed)
Abstract [en]

Swedish schools are legally obliged to have anti-bullying policies. Many commercial programs are available. During the study of the practical experiences of working with eight (named) programs, the research team was forced to modify the main question: whether schools that use specific anti-bullying programs were more effective than schools without such programs. All nominally designated treatment schools, as well as non-treatment schools, were found to be contaminated. This discovery overthrew the planned quasi-experimental design approach. Instead of attempting to evaluate anti-bullying programs as complete entities, an evolved evaluation strategy was developed to evaluate an array of program components, use of which was assessed on the basis of detailed qualitative data. Hierarchical cluster analysis was used to designate intervention types and schools with similar approaches. The purpose of using cluster analysis in this context was to construct empirically based classifications of types of interventions and approaches to bullying.

 

IMPLICATIONS

A mixed methods approach, based on questionnaires and interviews, was used in the evaluation. This approach provides a number of significant insights into evaluation practice. A major difficulty in program evaluations is associated with monitoring program fidelity. Rather than taking program fidelity for granted, as well as ‘‘pure’’ control conditions (which is often a prerequisite in experimental controlled trials or in randomized controlled trials), the mixed methods approach provide data about whether, in this case the schools, actually use the constituent intervention elements (program components) in real-world conditions, i.e. more realistic and reliable information about what schools actually do compared to evaluation approaches where use of program elements in a prescribed manner is taken for granted. Through the gathering of extensive qualitative data our evaluation revealed a profound program contamination. The solution to this evaluation challenge was to focus on program elements, or components. Since component use is highly context dependent, the array of components was identified on the basis of the context of everyday practice in all participating schools. One implication of this is that any comprehensive program evaluation ought to consider contextual circumstances in order to gain detailed data about actual component use. Furthermore, results from this evaluation emphasize the importance of gathering contextual data because an effective component in one circumstance may prove ineffective or ever counter effective in outwardly similar circumstances.

Place, publisher, year, edition, pages
2013.
National Category
Social Work
Research subject
Social Work
Identifiers
URN: urn:nbn:se:oru:diva-32795OAI: oai:DiVA.org:oru-32795DiVA, id: diva2:679556
Conference
Evaluation Practice in the Early 21st Century. Washington DC, USA, October 14-19, 2013
Available from: 2013-12-16 Created: 2013-12-16 Last updated: 2017-10-17Bibliographically approved

Open Access in DiVA

No full text in DiVA

Authority records BETA

Flygare, ErikJohansson, Björn

Search in DiVA

By author/editor
Flygare, ErikJohansson, Björn
By organisation
School of Law, Psychology and Social WorkSchool of Law, Psychology and Social Work
Social Work

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 479 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf