Evaluation of DARE to make a Choice

The Schools Survey

We carried out a web-based survey of PEOs and teachers to see how far they felt that Choice met the principles of best practice for drug education set out by MYD. They were questioned about 14 of the 16 best practice principles. They were also asked about the degree to which they complied with the Teaching Guide. Respondents were given the opportunity to make written comments about the areas of questioning, and many took advantage of this.

Of 113 teachers asked to take part, 64 responded - a response rate of 57%. Of the 48 PEOs who initially volunteered to take part, 35 responded (73%). The teachers and PEOs came from 46 schools across New Zealand. There was a good spread across schools of different decile levels, and PEOs were likely to have offered views on Choice from their wider experience of teaching in several schools. At the same time, it was hard to know how representative the teachers and PEOs who responded to the survey were. It may be that that they had rather more commitment to Choice than others involved in its delivery.

Many questions used a five-point scale which spanned strong disagreement (or a similar strong negative sentiment), and strong agreement (or a similar very favourable sentiment). We combined the two top (agree) points to give a summary measure of the level of endorsement for how various aspects of each principle were being delivered. There were a number of questions each around the content, process, and context areas of best practice. In drawing these together we used a weighted average of the responses of teachers and PEOs. This was so that the teachers’ responses did not dominate.

The teaching load

The teachers and the PEOs, perhaps not surprisingly, had rather different views on how the Choice teaching load was spilt between them. Four in ten teachers said the load was shared equally, while another four in ten conceded the PEO carried all or most of the load. Nearly two-thirds of the PEOs, though, said they carried all or most of the load, and less than a quarter said it was shared equally.

The content principles

The main findings from the questions on the best practice principles as regards the content of Choice were:

• Choice was seen by 80% (on the weighted average) as successful in meeting its objectives to prevent and reduce drug-related harm through indirect means (principle 2). Rather more (85%) felt it had clear realistic objectives by 85% (principle 3).

• It was also felt to be reasonably successful in meeting the needs of young people (71%) - though the endorsement from teachers was greater than from PEOs (principle 4).

• On the less positive side, less than half felt that Choice met needs of different cultural groups well (principle 5). Again PEOs were more critical than teachers.

• Only four in ten felt that Choice involved parents in classroom sessions (though there was not wholehearted support for this anyway), and even fewer respondents felt parents were involved in planning (principle 6). However, the activities were written so that classroom discussions would spill over into homework and conversations in the children’s homes. There were much the same figures for the involvement of community groups in the classroom and in planning (principle 7). The programme designers noted that there is extensive community involvement in Choice, although much of it takes place outside the classroom.

The process principles

The questions on the elements of best practice principles as regards the process of Choice elicited generally favourable results.

• About three-quarters felt that the way Choice was being delivered encouraged an interactive teaching style (principle 8).

• The same proportion thought it promoted good social skills (principle 9).

• Another three-quarters felt Choice offered information that was both accurate and appropriate to the age group (principle 10), although the PEOs were rather more negative on this score than the teachers.

• The main shortcoming was in analysing mass media messages (principle 11). Only four in ten felt that Choice was successful in this, with the PEOs particularly doubtful.

The context principles

The main findings of the questions on the elements of best practice principles as regards the context of Choice were:

• There was strong agreement that procedures were well followed for discussing drug issues in a confidential and safe manner in the classroom (principle 12).

• In most schools, Choice is well supported by a comprehensive school-wide approach drugs (principle 13).

• While teachers and PEOs said they provided each other mutual support and advice, there was much less agreement that they had good opportunities for ongoing training in drug education in general, or Choice specifically (principle 15). The programme designers indicated that PEOs are required to complete initial training on drug education at the University of Auckland as well as regular in-service training.

• The use of evaluation forms was fairly low, although there was more activity as regards carrying out other evaluations of Choice (principle 16).

Suggestions for improving Choice

Survey respondents were energetic in offering views about areas where Choice was failing, and what was needed for the future. The two most dominant concerns were interrelated. One concerned making Choice more up-to-date - by Improving the content of the programme, the materials provided for it, and bringing programme coverage more into line with contemporary issues. The other set of comments related to the need for a better IT infrastructure for Choice to take advantage of media advances likely to engage students better.

There were also a number of comments on shortening the programme. One element was that, with current levels of resourcing, delivery of Choice could be too rushed, and that the coverage of schools was less than it might be. The other element was that certain parts of the programme were rather ‘long-winded’.

Support for Choice

The comments revealed that while there was fairly strong support for a rethink of Choice (and a radical one in the view of a few respondents), there was nonetheless a good deal of endorsement of what Choice was trying to achieve, and how it was going about it.

PEO-school collaboration

The comments that were offered by PEOs and teachers in particular, as regards their collaboration, were in large part very positive, although inevitably some in each group had criticisms of some in the other. A sentiment that emerged quite strongly from the teachers was that the involvement of PEOs was beneficial. For one, it greatly lessened the burden on the teacher. For another, it was seen as a way of bringing the police into schools in a supportive role and in a manner likely to enhance police-student relationships.

The Case Study Schools

We conducted case studies in two schools, each with a different PEO, to observe the implementation of Choice. The work aimed to highlight actual practice in relation to elements of content, process, and context outlined in the 16 principles.

Decisions on the number and location of the schools were taken in consultation with NZ Police. Both schools were located within the greater Wellington area. Two PEOs who had some experience in delivering Choice were selected by the Manager of YES. The case study schools were nominated by the PEOs, who usually taught at a number of schools.

School A was a decile 8 co-educational intermediate state school. It caters for students in Years 7 and 8. The PEO had around 14 years’ experience teaching Choice.

School B was a decile 7 co-educational Catholic integrated school. It is a full primary school that caters for around 250 students from Years 1 to 8. The PEO was in her sixth year of teaching Choice.

The case studies involved:

• observation of a Choice lesson, to monitor its delivery;

• a semi-structured interview with the PEO and teacher delivering Choice, to determine whether they were adhering to elements of best practice and to ask their views on programme delivery;

• a brief questionnaire completed by the Principal, to assess whether Choice was supported by a comprehensive school-wide approach, was associated with family-based training, and was co-ordinated with other community initiatives; and

• a group interview with students doing the programme, to determine whether the content was relevant to their needs. In School A, this involved 20 Years 7-8 students chosen by the PEO. The interview in School B involved six Years 7-8 students chosen by the teacher.

The case studies drew attention to ways in which actual practice did or did not adhere to MYD best practice principles. Where there were deviations from best practice, the results indicated that some matters were programmatic, some may have been associated with individual differences between those delivering the programme, and some were matters for schools, rather than those who design or deliver Choice.

The case studies provided a generally but not unanimously favourable view of the extent to which the delivery of Choice reflected MYD best practice principles. While it is clearly difficult to generalise from the case study results, there was a degree of convergence between them and the survey findings, which suggested that the two schools may not be atypical.

Key findings relating to the content elements

A few points stand out from the case studies as regards best practice principles relating to the content of Choice.

• The aims of the programme appeared to be realistic and the content relevant to students. Principles relating to these elements were reflected in the delivery of Choice in both schools.

• Teachers and PEOs believed that the programme’s general content was relevant to today’s students, although they acknowledged the need to update teaching materials. One PEO regarded the programme materials as a guideline only. He believed that PEOs should adapt strategies that have been shown to work, even if they are not part of the Teaching Guide.

• Choice was not delivered with any particular attention to different cultural views and realities. This reflected the PEOs’ and teachers’ view that many drug-related issues cut across cultural boundaries.

• There was room for strengthening links between Choice and family-based training in drug education. This may be a job for schools, rather than solely for PEOs or NZ Police / NZ DARE Foundation.

Key findings relating to the process elements

The delivery of Choice in the case study schools measured up well against the four principles relating to process.

• The classroom observations showcased the interactive nature of the lessons and the numerous opportunities for young people to develop social skills.

• In both schools, students’ responses to the programme were overwhelmingly positive. The interviews with students suggested that Choice is popular in part because the information provided was relevant and useful to students in this age group. It is reasonable to assume that students’ endorsement of the informative aspects of the programme reflected its success in achieving its goals.

• The case studies raised a question about whether programme deliverers specifically plan lessons that include critical analysis of mass media messages.

Key findings relating to the context elements

There were two main areas in which actual practice deviated from best practice principles relating to the context of Choice.

• The first area related to the long-term delivery of drug education (principle 14), which might be more a planning issue for schools. MYD recommends that young people should have access to drug education during the entire school career. Choice is not designed to be delivered across all school years, although other DARE programmes are available for younger students (Years 5-6) and senior secondary students. Whether this principle is met will depend on what other drug education schools offer. Choice would build on any other drug education offered through the Health and Physical Education Curriculum.

• Secondly, there seemed to be a deficit in respect of regular, structured self-review and evaluation of whether the programme is meeting its learning objectives. The evaluation forms included in the Choice Teaching Guide provide ample opportunities for this, but those delivering the programme did not use them.

There is a question around training and ongoing support for programme deliverers. For the teachers and PEOs in both schools there seemed to be little in the way of opportunities for professional development in delivering drug education. As the PEOs delivered most, if not all, of the lessons, it is important, from an MYD perspective, that they demonstrate competencies in quality teaching. However, as previously noted, the programme designers informed us that all PEOs complete a module on drug education as part of their training. Furthermore, they have in-service training several times a year, which often includes workshops on DARE programmes and the nature of drug crime.

Conclusions

This evaluation considered how well Choice aligns with best practice principles for drug education developed by MYD. Its premise was that if best practice principles were being met, this would provide good indirect information of Choice’s likely effectiveness. The evaluation did not look at whether Choice changed students’ knowledge, attitudes, self-esteem, or drug use.

The findings from each component of the methodology are in Box A.

• In respect of the qualitative results (i.e. document analysis and case studies) a tick indicates that we found evidence in a given source that Choice met the best practice principle. In the schools survey a tick indicates endorsement by at least three-quarters of respondents.

• For the qualitative results, a cross indicates that we did not find evidence in a particular source that Choice met the best practice principle. In the schools survey, it means less than 50% endorsement.

• For the qualitative results, a question mark means that there were some questions as to how Choice measured up against this criterion. In the schools survey, it indicates between half and three quarters endorsement.

There were two main areas where there was evidence from the sources of a deficiency in Choice delivery. One of these concerned its lack of responsiveness to different cultural views and realities. The other was the rather inadequate way in which Choice was geared to critically analysing mass media messages. After this, there was some question mark over how well Choice is associated with family-based training, which might enhance the impact of drug education messages on children. There was another question mark over the rigour with which teachers and PEOs engage in thorough and regular, ongoing review and evaluation. Finally, there was an issue as to whether Choice meets the best principle of long term delivery. Choice is targeted at Years 5-8 and a new DARE programme will cover senior secondary students. These programmes would reinforce the messages of any other drug education offered to students, but cannot fully meet principle 14, which is more a planning issue for schools.

Box A Findings from each component of the methodology

This study did not address the merit or otherwise of using PEOs in delivering Choice, since it centred on evaluating the current mode of delivery, in which PEOs are integral. However, both the case studies and the schools survey showed that Choice is popular with schools, partly because it upgrades their capacities and resources for delivering drug education. From an MYD perspective, this underscores the importance of ensuring that PEOs develop competencies in quality teaching and have ongoing training in drug education. At the same time, due recognition should be given to the fact that teachers should be contributing to programme delivery in a substantial way and that they are highly trained and skilled in teaching.

The case studies indicated that the presence of PEOs had a positive impact on student behaviour and they were highly regarded by staff and students. Teachers who responded to the school survey also welcomed their collaboration with PEOs and felt their presence in the classroom was helpful in enhancing police-student relationships. To some extent, the perceived success of Choice in meeting its learning objectives may reflect non-programmatic elements, such as the PEOs’ credibility, personality and rapport with students.

The ongoing popularity of Choice, coupled with the new DARE programme for secondary schools, suggests that PEOs may face increased demands on their time if they continue to be responsible for delivering most of or the entire Choice curriculum. The question arises as to whether it is preferable to spread existing resources more thinly, by increasing the number of schools / programmes allocated to each PEO, or whether to co-opt more PEOs.

1 Introduction

In June 2006 New Zealand Police contracted the Crime and Justice Research Centre (CJRC) to conduct an evaluation of the DARE to make a Choice (Choice) drug education programme. The evaluation was based on the ‘best practice’ approach outlined in the CJRC scoping study commissioned by NZ Police in 2005 to consider the most efficient evaluation approach. This report presents the results of the subsequent evaluation.

Choice was developed by the New Zealand DARE Foundation and New Zealand Police. The programme was named DARE to make a Choice (Choice) to differentiate it from the American DARE programme. While Choice was inspired by the American DARE, the programmes have little in common, as Choice was developed to address the local cultural context and to tie in with the New Zealand Curriculum Framework (Hallmark, 2004; Sanders, 1995). Although the DARE to make a Choice programme is commonly referred to as DARE within schools, we have chosen to call it Choice so as to differentiate the NZ drug education programme from the DARE programme in the United States, and the other DARE programmes which run in New Zealand.

Choice was originally launched in 1991. Different aspects were evaluated over the next six years (see below). It was rewritten in 1998 to bring it in line with best practice principles identified in a report commissioned by the New Zealand DARE Foundation (Pickens, 1998). The current study is the first evaluation of the revised version of Choice.

Choice is a school-based programme delivered by teachers and police education officers (PEOs). The curriculum comprises two separate programmes - one for children in Years 5-6; the other for those in Years 7-8. The current study looks at the programme delivered to the older group (aged around 12 to 13).

Programmes for both Years 5-6 and Years 7-8 have a common aim and a sequence of clusters which are laid out in the Teaching Guides for the two age groups (developed by the NZ DARE Foundation and New Zealand Police). They are in Box 1.

Choice is a widely used programme. It was taught to 34,160 Years 5-6 and Years 7-8 school students in New Zealand during the 2003/2004 police corporate year.

Choice is taught by teachers and PEOs. The curriculum has a flexible structure that allows teachers to plan programmes and choose activities within each cluster which meet the levels at

which students are working. The Teaching Guide recommends that, for behaviour change to occur, children require a minimum of 15 sessions of approximately one and a half hours duration (i.e., about 22 to 23 hours). Both the Years 5-6 and Years 7-8 programmes have activities designed to meet these guidelines. It also states that PEOs should be present for no less than 10 and no more than 15 hours of actual teaching over the course. Ideally children will undertake Choice twice during Years 5-8, without repetition of any clusters. This requires careful record-keeping by teachers and PEOs.