Authors: Sylvia Dixon and Sarah Crichton
Youth Service: Not in Employment, Education or Training (or YS: NEET) is a government programme designed to encourage and assist disadvantaged 16-17 year olds to stay in education or training and improve their qualification attainment. Community organisations are contracted to undertake needs assessments and provide mentoring and support for these youth. This paper evaluates the impact of the programme on the educational retention, qualification achievement, benefit receipt, inactivity and employment rates of participating youth in the 18-24 months after they enrol in YS: NEET. Administrative data from the Integrated Data Infrastructure (IDI) is used to measure individuals' outcomes. The impacts of the programme are estimated by comparing the outcomes of participants with those of a matched comparison group of similar youth who did not participate.
We find that YS: NEET raises the educational retention of participants in the first year, by up to 9 percentage points at peak. The proportion who complete a level 2 qualification is slightly raised, by around 2 percentage points. The programme appears to raise rather than lower participants' subsequent benefit receipt rates, and there is no improvement in their likelihood of being employed.
We would like to thank Marc de Boer for his help with the Youth Service data and Michelle Bly, Marc de Boer, Gulnara Huseynli, Dean Hyslop, and Judd Ormsby for their helpful comments on earlier drafts of this paper.
The views, opinions, findings, and conclusions or recommendations expressed in this Working Paper are strictly those of the author(s). They do not necessarily reflect the views of the New Zealand Treasury, Statistics New Zealand, or the New Zealand Government. The New Zealand Treasury, Statistics New Zealand, Ministry of Justice and the New Zealand Government take no responsibility for any errors or omissions in, or for the correctness of, the information contained in this Working Paper. The paper is presented not as policy but with a view to inform and stimulate wider debate.
The results in this report are not official statistics - they have been created for research purposes from the Integrated Data Infrastructure (IDI) managed by Statistics New Zealand. Ongoing work within Statistics New Zealand to develop the IDI means it will not be possible to exactly reproduce the data presented here.
Access to the anonymised data used in this study was provided by Statistics New Zealand in accordance with security and confidentiality provisions of the Statistics Act 1975. Only people authorised by the Statistics Act 1975 are allowed to see data about a particular person, household, business or organisation. The results in this report have been confidentialised to protect these groups from identification.
Careful consideration has been given to the privacy, security and confidentiality issues associated with using administrative and survey data in the IDI. Further detail can be found in the privacy impact assessment for the Integrated Data Infrastructure available from Statistics New Zealand.
The results are based in part on tax data supplied by Inland Revenue to Statistics New Zealand under the Tax Administration Act 1994. These tax data must be used only for statistical purposes, and no individual information may be published or disclosed in any other form or provided to Inland Revenue for administrative or regulatory purposes.
Any person who has had access to the unit-record data has certified that they have been shown, have read and have understood section 81 of the Tax Administration Act 1994, which relates to secrecy. Any discussion of data limitations or weaknesses is in the context of using the IDI for statistical purposes and is not related to the data's ability to support Inland Revenue's core operational requirements.
The Youth Service (YS) is a government programme for 16-18 year old youth who are considered to be at risk of poor outcomes, including long-term benefit dependency. It aims to help these young people achieve a qualification at level 2 or higher and develop life skills to reduce their risk of moving on to a working-age benefit. The Ministry of Social Development (MSD) contracts community-based social service providers to work with these young people and support them to enter and remain in education, training or work-based learning. The YS programme commenced in August 2012.
This paper evaluates the YS: NEET sub-programme. The 'Not in Employment, Education or Training' (or NEET) strand of YS is targeted at young people aged 16-17 who are not currently receiving any income support from the government, but are considered to be at risk of moving onto a benefit at a later date. Participation in YS: NEET is voluntary.
The evaluation examines two main questions: how well has the programme been targeted towards youth who are at high risk of poor outcomes, and what impact does participation have on their outcomes? Longitudinal administrative data from IDI are used to measure a range of different outcomes including educational participation, educational achievement, time spent on benefits, time spent NEET, time in employment, and time in custody.
To provide a longer-term picture of New Zealand's experience with youth transition programmes, we also evaluate the impacts of an earlier programme, the Youth Transition Service (YTS). The YTS operated from 2006 until 2012, when it was replaced by the Youth Service.
Targeting of YS: NEET
We begin by taking all youth who started YS: NEET in the 2012-14 period and asking whether the majority of these new participants were at high risk of experiencing poor outcomes at age 18. To evaluate this, we calculate a series of risk scores for everyone who was aged 16-17 in 2012-14, representing their predicted likelihood of experiencing poor outcomes at age 18, and then compare the risk scores of the YS: NEET participants with those of other youth in the wider age group.
We find that high-risk youth were more likely to be recruited to YS: NEET than low-risk youth. However, a significant proportion of the participants were at relatively low risk of poor outcomes at age 18 (based on their observed characteristics, modelled risk scores, and subsequent outcomes). Only about half of all 2012-14 participants were drawn from the highest-risk 20% of the 16-17 year resident population.
About one third of all new recruits to YS: NEET were still enrolled in school when recruited, and among these high-school-student participants, targeting was particularly weak. Only about 30% were drawn from the highest-risk 20% of the youth population.
Impacts of YS: NEET
Our impact evaluation focuses on the youth who started YS: NEET in either 2012 or 2013 and stayed enrolled in the programme for at least three months. The time frame for assessing the impacts of YS: NEET is the first 18-24 months after enrolment. The impacts of participation are estimated by comparing the outcomes of the participants with those of a matched comparison group, made up of youth who were very similar on a wide range of measured characteristics but did not participate in YS: NEET.
Higher educational participation
We find that YS: NEET had a positive impact on education and training enrolment rates, although the effects were modest in size and not sustained much beyond one year. The proportion who were enrolled in education or training was 9 percentage points higher at 6 months after starting YS: NEET, 4 percentage points higher 12 months after, and not significantly different 18 months after.
The increase in educational participation during the first year after starting YS: NEET was largely due to higher enrolment in tertiary programmes. We did not find any significant change in the likelihood of staying in school.
Slight increase in qualification attainment
The impact on qualification achievement rates was positive but very small. YS: NEET raised both the level 1 and level 2 qualification achievement rates by about 2.0 percentage points on average.
Slightly more time spent on benefits
Participation in YS: NEET was associated with higher rather than lower benefit receipt in the following two years, with the impact on benefit rates increasing though time. This was true for all sub-groups of participants. Thirty-one percent of YS: NEET participants were on a benefit two years after they started YS: NEET, compared with 27% of the matched comparison group.
Negative impact on employment rates
YS: NEET reduced participants' employment rates in the first year (reflecting their higher rate of retention in education) and did not have any significant impact in the second year.
The limited impact of YS: NEET on qualification attainment is the most likely reason for the programme's lack of labour market benefits. The programme's impacts on level 1 and level 2 qualification attainment rates were small (2 percentage points on average), and there was no impact on level 3 attainment. A previous study of the employment impacts of tertiary study at levels 1-3 for low-qualified school leavers (Tumen, Crichton and Dixon, 2015) found that only students who were successful in completing a qualification were more likely to be employed afterwards.
Educational achievement impacts were slightly larger for higher-risk youth
While the average effects of YS: NEET on qualification attainment were very small, there was some variation between sub-groups of participants.
The qualification attainment impacts were slightly larger for youth who were not continuously enrolled in formal education at the time they started YS: NEET, who made up about half of all participants. On average, they had a level 2 qualification achievement rate by the end of the following year that was 6 percentage points higher than that of their matched comparisons.
Very high-risk youth - defined here as those in the highest decile of the predicted risk of poor outcomes at age 18 - also experienced larger increases in educational attainment. Participation in YS: NEET was associated with a 5 percentage point improvement in the level 2 qualification attainment rate of this subgroup, which made up around 29% of all participants in 2012-14.
However, these larger impacts on qualification attainment were not translated into larger or more beneficial impacts on either employment rates or benefit receipt rates.
Limitations of the YS: NEET evaluation
The two-year follow-up period used in this evaluation is relatively short. However, the time pattern of the impacts on education and training rates shows that these impacts were declining rather than increasing over time, so it seems unlikely we would find larger positive impacts if the follow-up period was longer.
The most important limitation of this study is that the method used can't ensure that the study population and the comparison group (which provides the benchmark for estimating impacts) are perfectly matched on all characteristics that may affect outcomes. We acknowledge there is a risk that the impact estimates given in this paper could be either too high or too low. We see them as the best available estimates in the absence of a more rigorous evaluation method, which would need to randomly allocate eligible youth to either participation in YS: NEET or a control group.
Impacts of the Youth Transition Service
The Youth Transition Service (YTS) was in operation from 2006 until mid 2012, when it was replaced by the Youth Service. We estimated its impacts on the youth who participated during 2008-2011. A key objective of this supplementary impact analysis was to identify whether the benefits of participation in a youth transition programme tend to be larger in the medium term (up to five years) than in the 1-2 years immediately after participation.
We find that the average impacts of YTS were insignificant or slightly negative. Participants were 1-3 percentage points less likely to be studying in the follow-up period than matched non-participants, about 2 percentage points more likely to be on a benefit, 1-2 percentage points more likely to be NEET, and 1 percentage point less likely to be employed. There was no significant impact on qualification achievement. In addition, there was little change in the estimated impacts from one year after YTS enrolment to five years after: using a longer follow-up window did not materially alter the size or pattern of results.
YS: NEET raised participants' rates of enrolment in formal education by up to 9 percentage points at peak. This positive impact on the likelihood of studying was sustained for around one year. The flow-on to qualification achievement was quite modest, however: the level 2 attainment rate was increased by around 2 percentage points. The programme did not raise participants' employment rates, and their benefit receipt rates were slightly raised rather than lowered.
Prior international research on the impacts of youth mentoring programmes has found that they either have no impacts or small positive effects on academic achievement. The results for YS: NEET are in line with this literature.
The Youth Service (YS) is a government programme for 16-18 year old youth who are considered to be at risk of poor outcomes such as long-term benefit dependency. It aims to help these young people achieve a qualification at level 2 or higher and develop life skills in order to reduce their risk of moving on to a working-age benefit after their 18th birthday (Ministry of Social Development, 2015). Under the YS, the Ministry of Social Development (MSD) contracts community-based social service providers to work with these young people and support them to enter and remain in education, training or work-based learning.
YS has three strands: Youth Payment, Young Parent Payment, and Not in Employment, Education or Training (or NEET). Youth aged between 16 and 18 years who are receiving a benefit are required to participate in the Youth Payment or Young Parent Payment strands. Those who are not receiving any income support from the government, but are considered to be at risk of moving onto a benefit at a later date, can participate in the YS: NEET strand on a voluntary basis.
This paper evaluates the YS: NEET strand of the programme. The other two strands are evaluated in an associated Treasury working paper.
We examine the following two questions: how well has the programme been targeted towards youth who are at high risk of poor outcomes, and what impact does participation have on outcomes in practice? Longitudinal administrative data from Statistics New Zealand's Integrated Data Infrastructure (IDI) are used to measure a range of different outcomes, including educational participation, educational achievement, time spent on benefits, time spent NEET, time in employment, and time in custody.
Because the YS is a relatively new programme, the impact evaluation focuses on people who enrolled in YS: NEET during the first 18 months of operation (August 2012 to December 2013), and their outcomes during the 18-24 months after their enrolment. To provide a longer-term picture of New Zealand's experience with youth transition programmes, we also evaluate the impacts of an earlier programme, the Youth Transition Service, which operated from 2006 until 2012 when it was replaced by YS. We analyse the outcomes of YTS participants over a five-year follow-up period.
The paper is structured as follows. Section 2 summarises relevant literature, describes the YS: NEET and YTS programmes in more detail, and describes the previous evaluations of these programmes. Section 3 outlines the data and methods used in this evaluation. Section 4 examines the targeting of the YS: NEET programme. Section 5 provides descriptive information on the characteristics of participants, the duration of time they were enrolled in YS, the education or training they undertook, and their educational outcomes. Section 6 presents our estimates of the programme's impact on educational participation, educational achievement, benefit take-up rates, NEET rates and other relevant outcomes. Section 7 presents our estimates of the impacts of the YTS, the predecessor programme. We draw conclusions in Section 8.
2 Literature review and description of the programme
2.1 Literature review
The core element of both YTS and YS: NEET is the provision of customised support and guidance. Many other countries also have youth programmes that provide customised support and guidance to 'at risk' youth.
DuBois et al (2011) undertook a comprehensive and systematic review of studies of the effects of youth mentoring programmes published up to 2010, covering both experimental and non-experimental evaluations. The authors conclude that the evidence is promising, showing that mentoring programmes can have positive impacts on attitudes, interpersonal relationships, various indicators of psychological well-being, conduct problems, academic performance at school, and physical health. Compared with other interventions, the effect sizes are modest (equivalent to a difference of 9 percentile points from the scores of non-mentored youth on the same measures). A few studies have found evidence that the positive impacts of mentoring persist during the following two years, but the number of such studies is low. The authors conclude that more evidence is needed on the long-term effects of mentoring on key outcomes of policy interest such as educational attainment, employment, and juvenile offending.
A more recent review by Rodriguez-Planas (2014) gives priority to evidence from studies that used rigorous evaluation methods, such as random assignment of youth to either the mentoring programme or a control group. Rodriguez-Planas concludes that the evidence is mixed: some studies have found positive effects on youth behaviour and academic success while others have not. In part, this diversity of findings is likely to reflect the diversity of the programmes. Mentoring programmes tend to be better at improving young people's non-cognitive and social skills than their academic performance. The positive effects of mentoring programmes that do lead to improvements can dissipate quickly in the year after the mentoring ends (p.4).
In addition, Rodriguez-Planas finds that mentoring programmes can sometimes have negative as well as positive effects. In the case of the Quantum Opportunity Program for high-school aged youth in the US, for example, male participants were more likely to commit crimes and be arrested in their 20s than students who had not been in the program (p.5). One possible explanation is that grouping together high-risk youth can expose them to negative peer influences, leading to increased rates of substance abuse or criminal offending.
Since youth mentoring and transition support programmes vary widely in their objectives, target populations and design, one would expect variation in their effectiveness. Compared with programmes that have operated elsewhere, YS: NEET has fairly well-defined and academically-oriented objectives - raising participation in formal education or training and raising qualification achievement - and it uses outcome-based funding to incentivise the programme providers to achieve these objectives. These features would be likely to promote effectiveness. On the other hand, YS: NEET does not necessarily provide a high level or intensity of support to its participants.
2.2 Youth Service: NEET
This section describes the YS: NEET programme, drawing on MSD documents (MSD, 2012 and MSD, 2014). YS: NEET is a voluntary programme that is available to 16 and 17 year olds (and some 15 and 18 year olds) who are either not in education, employment or training (ie,NEET), or are considered to be at risk of becoming NEET after they leave the education system. The service aims to engage and support these young people to return to education or training and raise their qualifications, with a particular focus on achieving NCEA Level 2 or equivalent qualifications. Participants in YS: NEET do not receive financial assistance through the programme. Young people who are already on a benefit or who successfully apply for one are channelled into the Youth Payment or Young Parent Payment strands of YS.
MSD documents (MSD, 2012) state that the expected outcomes of participation in YS: NEET are that young people:
- reduce their risk of long-term benefit dependency
- achieve at least NCEA Level 2 or an equivalent qualification
- sustain participation in education, training or work-based training for at least three months
- develop important life skills, such as budgeting, to improve their social outcomes
- have an achievable plan for employment, further education, or training upon exiting the service
- not be on benefit or serving a custodial sentence three months after the end of the school year or end of the training course following their 18th birthday.
MSD contracts community-based service providers to work with young people to support them into education, training or work-based learning.
When a new participant is enrolled in YS: NEET the provider is expected to:
- discuss with the youth the goals they want to achieve in employment, education or work-based training and create an Action Plan, which details the agreed activities and goals and how the provider will work with them to help them achieve the goals
- refer them to appropriate education, training or work-based learning
- refer them to other interventions as identified in the needs assessment
- meet with the youth on a regular and ongoing basis
- monitor their progress to ensure they are on track to achieve their goals.
YS providers are funded using a mix of fee-for-service and outcome-based payments. For each YS: NEET participant, providers receive an enrolment fee, a quarterly administration fee, a quarterly milestone fee if the youth is still engaged in education, a one-off achievement fee for each further level of qualification obtained, and an exit fee if the participant leaves at 18 years and is not on benefit 3 months after leaving.
To help providers identify high-risk youth, MSD calculates a model-based risk score for all 15-17 year old school leavers, using administrative data held by MSD and the Ministry of Education, and a model of the likelihood of moving to a benefit at age 18. MSD passes information about these school leavers and their risk scores to the contracted YS providers in each region. The providers can then contact the young people and invite them to participate in YS. They can also enrol eligible youth they have contacted themselves, for example by approaching schools directly, or been referred through other channels.
Children who have not left school yet or who left school before 2012 are not assigned a model-based risk score by MSD. In this situation, the provider completes a paper questionnaire as an alternative method of assessing their risk and assigning a risk category. The provider may also apply to have a young person's existing risk category changed if they believe it is inappropriate.
The payments that YS providers subsequently receive for their client's retention in education or training, qualification achievement and off-benefit status are linked to the client's risk score, with higher-risk youth attracting higher payments. In the 2015-16 financial year, for example, the quarterly payment for a low risk youth who was engaged in education was $381, while the quarterly payment for a very high risk youth who was engaged in education was $663.
2.3 Youth Transition Service
The Youth Transition Service (YTS) operated from 2004 to 2012 and was broadly similar in objectives to the YS: NEET programme. The goal of the YTS was to ensure that all 15-19 year olds were in work, education, training, or other activities that would contribute to their long-term economic independence and well-being (MSD, 2008, p1). Specifically, community organisations around the country were contracted to:
- Follow-up school leavers and engage with young people who were at risk of prolonged disengagement from work, education or training
- provide these at-risk youth with customised support and guidance to facilitate their re-engagement into appropriate work, education or training
- identify and support the development of appropriate labour market, education and training opportunities for young people
- provide a forum for the strategic planning and co-ordination of services for young people.
According to MSD (2008), a successful outcome of YTS was 'when a young person maintains ongoing involvement in education, training or employment and no longer requires the support of YTS to remain in that activity'.
According to the operational manual, youth were to be allocated to 'follow-up' if they had a plan or destination after secondary school, and had no significant issues putting them at risk of long-term inactivity. Follow-up required the provider to make contact with the youth, and to ensure they transitioned into work, education, training or some other meaningful activity as intended.
Youth were to be allocated to 'customised support' if they did not have a plan or destination following secondary school, or had significant issues or barriers placing them at risk of long-term inactivity.The YTS provider would provide intensive support, including mentoring, coaching, counselling and interventions to help transition the youth into work, education, training or some other meaningful activity. Customised support could continue after the transition to ensure the youth remained in the activity, by providing in-work or in-training support.
The main differences between YTS and YS: NEET are that:
- YTS provided services to a wider age group, with the majority of participants being 15-19 years old at the time of enrolment (rather than 16-17 years in the case of YS: NEET)
- YTS engaged with its clients for a shorter period of time on average. Sixty-seven percent of the 2008-2011 intakes were enrolled for less than 12 months, compared with 47% of YS: NEET participants.
- YTS providers were not given assistance by MSD to identify and locate 'at risk' youth. They were expected to identify the youth who needed assistance by developing relationships with schools and community organisations (who could then make referrals), and to determine a youth's level of need through their own assessments.
- YTS providers were funded on a fee-for-service basis, while YS funding uses a mixture of fee-for-service payments and payments for specific outcomes.
- The target outcomes were not as clearly defined under YTS as they are under YS.
These are quite significant differences. However, the vast majority of providers that held YTS contracts in 2011 continued to provide services under YS in 2012, after YTS was replaced by the Youth Service. This continuity suggests that the nature of the services provided may have evolved rather than changed abruptly.
-  After excluding people who were enrolled for less than 30 days.
2.4 Previous evaluations of the YTS and YS: NEET
A process and outcome evaluation of the Youth Transition Service was published by MSD in 2008 (MSD, 2008). The evaluation analysed the proportions of youth who had a positive outcome at the time of exiting from YTS, such as entry to employment, enrolment in tertiary education, or a return to school, using outcome data recorded by the providers. Due to a lack of access to suitable data, the study did not attempt to measure the impact of participation in YTS (compared with the counterfactual of not participating).
MSD published an evaluation of the Youth Service, covering the first 18 months of its operation, in June 2014 (MSD, 2014). This study also did not attempt to assess the impact of the NEET strand of YS. It simply reports data on participant outcomes. For example, it reports that half of the YS: NEET participants increased their number of NCEA credits within 12 months of starting Youth Service; 15 per cent met the requirements of NCEA Level 2 within their first 12 months in the Service; and 17 per cent had already met the requirements of NCEA Level 2 prior to starting the Service.
3.1 Data sources
The study uses data from Statistics New Zealand's Integrated Data Infrastructure (IDI), which combines administrative data from the tax system with data collected by other government agencies and covers all persons in New Zealand.
At the time the research was carried out, the main IDI data sources used in this study provided longitudinal information on individuals':
- employment and earnings over the period from 1999 to mid 2015
- benefit payments over the period from 1993 to mid 2015
- interactions with Child, Youth and Family over the period from 1993 to mid 2015
- school enrolments over the period from 2006 to mid 2015
- tertiary education enrolments over the period from 2003 to the end of 2014
- NQF-registered qualifications completed from 2006 to the end of 2014
- custodial and community sentences served with the Department of Corrections
- places of residence within New Zealand; and
- movements in and out of New Zealand from 1997 to mid 2015.
The information on individuals' places of residence within New Zealand is derived from several administrative sources, including the National Health Index, Primary Health Organisation enrolments, and address data held by Inland Revenue, MSD and the Ministry of Education.
Before finalising the paper, additional data were obtained on school and tertiary enrolments in 2015 and school and tertiary qualifications completed in 2015. These data were used to update and extend the main impact estimates which are set out in Table 6 and Figure 5. We decided not to update any other results (such as the impact estimates for sub-groups) because the work was already at an advanced stage and the new data did not show any material changes in programme impacts.
-  Addresses are encrypted in IDI to preserve confidentiality.
3.2 Methods used in the impact evaluation
3.2.1 Study population selection criteria
The study population for the YS: NEET impact evaluation comprises people who first enrolled in the NEET strand of Youth Service in either 2012 or 2013 and met various other criteria. The selection criteria, and the impact they had on the numbers of people in the sample used for the impact evaluation, are set out in Table 1.
|First enrolled in YS: NEET between 1 August 2012 and
31 December 2013, without previously enrolling
in the YP or YPP strands
|Identity was linked to the IDI spine||12,327||89.0|
|Also has IRD and Ministry of Education data linkages in IDI||12,162||87.8|
|Aged 15-18 years when enrolled in YS: NEET||12,129||87.6|
|Survived until 20th birthday||12,102||87.4|
|Had at least one New Zealand school enrolment record during
the period when they should have been enrolled in Years 9-11
|Not overseas for more than 6 months in total during the 5.5-year
study period, comprising 4 years before the YS: NEET
enrolment date and 18 months following it
|Did not attend a school that offers international qualifications
|Last school enrolment record has a valid end date**||10,827||78.2|
|Not on a benefit in the month of enrolment in YS: NEET||10,500||75.8|
|Stayed enrolled in YS: NEET for at least 90 days in the 12 months
after the first enrolment
|Matched with at least one comparison group individual||9,081||65.6|
*This restriction was only applied if the youth was aged 15 years or older when attending the school.
**This restriction was only applied if more than five years had elapsed since the enrolment start date, indicating that the school had failed to supply the end date. The numbers in this table are randomly rounded.
A linkage to the IDI spine is required to provide the data needed for this study. Connections to IRD and Ministry of Education identity numbers are also required to ensure we have access to individuals' income, benefit and education and training data.
We exclude a small number of people whose age when they started YS: NEET, as recorded in IDI, was less than 15 or greater than 18 (in conflict with their recorded age in the YS enrolment dataset). We also exclude a small number of people who died before their 20th birthday, who may have incomplete outcome data for the follow-up period.
We exclude people who did not enrol in a New Zealand school at least once during the period when they should have been attending Year 9 to Year 11, and those who were overseas for more than 6 months in total during the main study period (comprising the 4 years leading up to the YS: NEET enrolment date and the 18 months following it).
We also exclude individuals who, when aged 15 or older, attended a school that offers qualifications that are not part of the National Qualifications Framework (such as the International Baccalaureate), because there is no information in IDI about those non-NQF qualifications. We exclude a small number of people whose last school enrolment record had a missing end date after five years (preventing us from identifying when they left school). We also exclude youth who were on a benefit in the reference month, because these youth should have been enrolled in the Youth Payment or Young Parent Payment strands of YS (even though the YS enrolment data do not show that this was the case).
Finally, we exclude youth who were enrolled in the YS for less than 90 days during the first year of enrolment, because they may not have received enough assistance to experience a measurable change in outcomes. (Note, we do not require that they were enrolled for 90 days consecutively, although the vast majority of our study population members were.)
The figures in the right-hand columns of table 1 show the impact of these restrictions.
Of the approximately 13,848 individuals who first enrolled in YS: NEET during 2012 or 2013, around 12,160 or 88% could be linked by Statistics NZ to the IDI spine and the Education and Inland Revenue domains in IDI. The most likely reason for linking failure at this point is that the individual's names or birthdate were inaccurately or incompletely recorded.
A further 12% were dropped from the potential study population as a result of all the other study selection criteria, leaving 10,500 persons. Of these, we then retain the 88 percent who stayed enrolled in YS for at least 90 days, giving 9,285 potential study group members. Of this latter group, 98% could be matched to at least one suitable matching non-participant and were therefore retained in the impact analysis sample (and used to produce the estimates in this paper).
The final sample for the impact analysis represents 66% of the individuals who first enrolled in YS: NEET in 2012 or 2013. Although this means the sample does not include about one-third of all youth who (appear to have) enrolled in YS: NEET at some point in 2012 or 2013, the restrictions are designed to ensure that we evaluate the impacts of the programme as fairly as possible. Removing individuals with incomplete data or enrolment periods of less than 90 days should increase the likelihood of detecting programme impacts, if there are any.
No data are available on the characteristics of the YS: NEET participants who were not linked to IDI. However, we compared the characteristics of the participants who were linked to IDI but excluded from the final study sample (approximately 2,877) with of those of the participants who were retained (approximately 9,285). We found that the youth who were excluded were much more likely to be 18 years at the time of their enrolment. It is possible that these 18 year olds were more likely to leave the programme (or have their enrolment terminated by their provider) in less than three months. In other respects, the youth who were excluded from the final sample appear to have been slightly less disadvantaged, in terms of the indicators of disadvantage used in this paper, than those who were retained. For example, 55% of the youth who were excluded had no NCEA qualifications, compared with 62% of the final study population. Our decision to exclude youth who were overseas for long periods, enrolled at schools offering non-NCEA qualifications, or YS: NEET participants for less than 90 days, may have caused a slight bias in the final sample towards more highly disadvantaged youth.
Note that our selection criteria mean most recent migrants to New Zealand will not be included. This is appropriate because their administrative data in IDI will be too limited to provide comparable measures of their characteristics and lifetime experiences.
Note also that the study population for the YS: NEET evaluation includes young people who first enrolled in YS: NEET but subsequently transferred to the Youth Payment or Young Parent Payment strands. This would normally be a result of them applying for and receiving a benefit. Seventeen percentof the study population had transferred by the middle of 2015, with most moving to YP. Individuals who first enrolled in the YP strand of the Youth Service and later transferred to YS: NEET are not included in this study population.
3.2.2 Outcome measures
The objectives of YS: NEET are to raise participation in formal education or training, raise qualification attainment, and reduce the likelihood that a youth will move onto a benefit when they become eligible for income support at 18 years of age. YS providers receive quarterly payments for participants who are enrolled in formal education or training, one-off payments for their achievement of a qualification, and a one-off payment for participants who are not on benefit three months after leaving the YS. Therefore enrolment rates, qualification achievement rates and benefit receipt rates are the core outcomes against which the effectiveness of the programme should be assessed.
We also consider a range of other outcomes measures that may be influenced by YS participation, including whether the youth was NEET at various points in time during the follow-up period, whether they were employed, whether they received a student allowance, whether they were serving any custodial or community sentences, and their earnings growth (if employed). We also construct some additional measures of level 2 qualification attainment, benefit take-up and NEET during the three months following the young person's 18th birthday.
The outcomes of programme participants are assessed over the 18-24 months following the first YS enrolment date. We have a complete set of data covering the first 18 months after YS: NEET enrolment for most participants. Because 2015 tertiary enrolment data were not available until near the end of the project, we did not have data on tertiary enrolments for some portion of months 13-18 for a minority of participants (those who enrolled in the second half of 2013). We constructed our outcome measures using the individuals for whom the enrolment data were available.
Due to the manner in which tax data are collected in New Zealand, the employment and earnings measures in IDI are available on a calendar month basis only. There are no measures of weekly earnings, hourly earnings, or hours of work in IDI. In this study, a person is classified as 'employed' in a given calendar month if they received any wage and salary earnings in that month (that were reported through the tax system). For consistency, we use calendar months to construct all of our measures of post-YS enrolment activity and incomes, even though some of them (such as whether or not a benefit was received) are recorded in IDI on a daily basis. For example, the 'employment rate' measures the proportion of people in a particular group who received wage and salary earnings (above a minimum threshold of $10) at any time during a particular calendar month. Similarly, a person is classified as 'in receipt of a benefit' if they received any income from one of the main income support benefits during the calendar month, and a 'benefit receipt rate' is the proportion of people in a particular group who received benefit income in that month.
3.2.3 Method of estimating the program impact using matched comparisons
The impact of participation in YS: NEET is estimated by selecting a comparison group of youth who were as similar as possible to the individuals in the study population but did not participate. The outcomes of the comparison group individuals in the follow-up period provide the 'counterfactual' against which the actual outcomes of the study population members are compared. We use a combination of exact case matching and propensity score matching to select the most appropriate comparison group members for each individual in the study population.
Tables A.1 and A.2 in the Appendix show that YS: NEET participants and the potential comparison group of all 16-17 year old non-participants who met the criteria in Table 1 above were very different in their measured characteristics. In tables A.1 and A.2 we compare the characteristics of YS: NEET participants in the month they were recruited to the programme with those of all the potential comparisons who were aged 16-17 in the month of December 2013. (This time point is an arbitrarily chosen one.)
The YS: NEET participants were much more likely to live in high-deprivation neighbourhoods, to have parents or caregivers who had served custodial or community sentences, and to have been supported by a benefit for a high proportion of their childhood. They were more likely to have had Child, Youth and Family (CYF) care and protection notifications, findings or placements during their childhood. They were more likely to have attended low decile schools, to have had recorded episodes of truancy, suspensions or stand-downs from school, to have left school while aged 15 or 16. They were less likely to have achieved an NCEA qualification. There are material differences on every characteristic shown.
The matching process is designed to create a comparison group that is as similar as possible to the YS: NEET participant sample. The method used to construct the comparison group had three parts.
In the first stage, a pool of potential comparison group members was created by selecting all youth who met the criteria listed in Table 1, with the exception that they did not enrol in YS: NEET before or during our study period (including the follow-up period) and did not enrol in any Youth Service programme before the month of selection. Note that we did not exclude individuals who moved onto YS: YP or YS: YPP at a later stage, during the follow-up period, because those programmes are compulsory for youth receiving benefits, and we did not want to exclude youth who went onto a benefit from either the study group or the comparison group.
The characteristics, prior activities and childhood histories of these youth can be measured in each calendar month from August 2012 through to December 2013. For each person in the potential comparison group, we generated 17 monthly records corresponding to each month in this time period, and randomly assigned a reference date within the month. The characteristics, prior activities and childhood histories of each individual were then measured as at the reference date. The purpose of creating this large pool of potential control group records was to ensure we could match each person in the study population with a group of other youth whose characteristics were as well matched as possible in the reference month - the month when the study population member first enrolled in YS.
In the second stage, the study population and potential control records were divided into three sub-samples, according to whether the child was still at school, enrolled at a tertiary institution, or not in either group, at the reference date. The three sub-samples were defined as follows:
- 'at school' - enrolled at a secondary school when they enrolled in YS and stayed in school for at least one further month (meaning they had school enrolment records for three consecutive months centred on the YS enrolment month)
- 'in tertiary' - not at school but enrolled in a tertiary education programme when they first enrolled in YS, and stayed enrolled for at least one further month (meaning they had tertiary enrolment records for three consecutive months centred on the YS enrolment month)
- 'not enrolled' - those who were either not enrolled in school or tertiary education at all, or not enrolled for the minimum 3 month period.
The logistic regressions were then estimated separately for each sub-sample. This meant that explanatory variables that were not relevant to the circumstances of children who were still at school (for example) could be omitted from the model for the 'at school' sub-sample but included in the other two models. It also allowed the parameters to differ between the three sub-samples.
To run the logistic regressions, we took a random sample of 10,000 of the potential comparison group records, so that the treatment group individuals would make up a larger proportion of the total.
The explanatory variables included in the models include:
- The reference month (the month of first YS: NEET enrolment in the case of the study population, and a randomly assigned reference date in the case of the potential comparisons)
- An indicator for having previously participated in the Youth Transition Service (a similar programme that preceded YS and was superseded by it)
- Age at the reference month
- Ethnic groups (using indicators for each non-European ethnic group)
- Birth cohort
- New Zealand Deprivation Index score for the neighbourhood that was lived in at the reference date
- Region of residence at the reference date
- A measure of the proportion of time the child had been supported by a parent's benefit during their life time
- Several variables capturing the youth's lifetime care and protection history with CYF (the total number of notifications, an indicator of whether there had been any substantiated findings, and an indicator of having had a CYF care and protection placement)
- A measure of the child's total number of CYF youth justice referrals
- An indicator of whether the youth's mother or female caregiver had no qualifications (available if the mother or caregiver has received a benefit at some time in the past)
- Indicators of whether the child's parents or caregivers had ever served a custodial or community sentence (available if the parent or caregiver has received a benefit at some time in the past)
- The proportion of the individual's childhood that was spent out of NZ, up to the reference date
- Characteristics of the school that the youth currently or most recently attended, including its decile and ownership type
- The level of the highest qualification held at the reference date
- The numbers of NCEA credits that had been completed at levels 1, 2 and 3
- An indicator of whether the child had ever been granted special education funding
- Several measures of 'disengagement' from school, including the total numbers of stand-downs and suspensions from all schools attended and whether there were any truancy records
- The number of schools attended since 2006
- Time elapsed since the date of leaving school (where relevant)
- Whether any vocational or general skill tertiary programmes had been undertaken before the reference date
- Whether any tertiary qualifications had been completed before the reference date
- Measures of the number of months the individual was enrolled at school, enrolled in tertiary education, employed, or NEET during the past 48 months
- Measures of the number of months the individual was in receipt of a benefit, in their own name or as a partner, during the past 18 months.
A full list of the explanatory variables included in the regressions is given in Table A.2 in the Appendix.
Predicted probabilities of participating in YS: NEET were then calculated for all members of the treatment group and potential comparison group (not just the sub-sample of potential comparisons that was used in the logistic regressions), using the propensity scores from each regression model. These predicted probabilities are referred to as 'propensity scores'.
The third stage of the method was to match each individual in the study population with a group of comparison individuals (within each sub-sample). Matches were only made between records with the same reference month, gender, age, highest qualification level, and region of residence. We exact match cases by reference month and region to provide some degree of control for the effects of variations in the business cycle and local labour market effects.
Within those exact matching constraints, each study population individual was matched to up to 10 comparison group individuals with the closest values of their propensity score, within a radius of plus or minus 0.01 or 0.03 propensity score points (the narrower band was used when there were more available matches). Fewer than 10 matches were selected if less than 10 people met these criteria. Matching with replacement was used, meaning that each comparison group individual could be matched to more than one study population member.
Each matched comparison individual was assigned a weight based on the number of matches made (eg, 0.10 if the person was one of 10 matches for a particular study sample member). These weights are applied in the subsequent analysis of impacts, to ensure that the distribution of comparison group characteristics mirrors that of the study population.
We dropped individuals in the study population who could not be matched with one or more comparisons. The match rates for the three sub-samples were >99% for the 'at school' and 'not enrolled' subsamples and 94% for the 'in tertiary' subsample.
The matching method was designed to balance the average characteristics of the study population and matched comparison groups. After matching, there were no remaining statistically significant differences in variable means between the study and comparison groups, for any of the model variables.Although we did not exact match on every variable, the method ensured that the matched samples were very similar in terms of their demographic and regional profiles and prior employment and income support histories.
Once the matched comparison groups were constructed, the impacts of YS: NEET participation were estimated as the difference between the mean outcome of the study population and the mean outcome of the matched comparison group. Standard errors and confidence intervals for each impact estimate were estimated using bootstrapping methods.
We used a single set of matched comparisons for each member of the study population to estimate the impacts across all outcome variables. We did not re-run or alter the matching process for different outcomes.
In our final impact estimates, we also excluded individuals who were out of New Zealand in the month of interest. For example, the estimates for outcomes at 24 months after starting YS: NEET do not include people who were out of New Zealand for more than 25 days during that particular month.
3.3 Models used to estimate young people's risk of having poorer outcomes at age 18
One of the objectives of this evaluation is to assess whether the 16 and 17 year olds who are at highest risk of future inactivity or benefit receipt have been participating in YS: NEET. To do so, we construct three different measures of 'poor' outcomes at age 18, model the likelihood of a young person experiencing these poor outcomes, and then score all teenagers who were aged 16-17 during the period of YS operation (our 'target' population) according to their estimated likelihood of experiencing each outcome. We can then compare the predicted risk scores of the YS: NEET participants with those of all 16-17 year olds to see whether participants were drawn from the segments of the youth population that had the highest risk scores.
Outlining the approach in more detail, the measures of poor outcomes at age 18 are:
- Being NEET for at least 6 calendar months of the year when aged 18
- Not having obtained a level 2 qualification or higher by the end of the calendar year in which the teenager turned 18
- Receiving a benefit in at least 6 calendar months of the year when aged 18.
We used IDI data for all youth who were born in 1993-95 to construct an estimation sample. The 1993-95 birth cohorts were used because they are old enough to have complete data for age 18 in IDI, enabling us to construct the above measures for everyone, but are as close as possible in birth year to our target youth population. We constructed multiple records for each individual in the estimation sample, capturing their characteristics and lifetime history at each quarter-year of age from 16 through to 17.75. The explanatory variables used in the risk models were largely the same as those used in our models of the likelihood of YS: NEET participation, and are described above in Section 3.2.3.
The records in the estimation sample were divided into three sub-samples, according to whether the youth was still at school, enrolled at a tertiary institution, or not in either group, at the reference date. Logistic regressions were then estimated separately for each dependent variable and each sub-sample. Although not all of the explanatory variables were statistically significant, all were retained in the regressions to provide the best possible predictions.
The coefficients obtained from the nine regressions (one for each combination of the three sub-samples and three outcomes measures) were then used, along with data on the characteristics and lifetime histories of the target sample members, to calculate three risk scores for each target sample member, representing their predicted relative risk of experiencing each outcome at 18.
Note that our risk assessment models provide risk ratings for only around 80% of youth in the total 16-17 year old resident population during the period of interest. We only model the risk of the youth who met the IDI data linkage, residence in New Zealand and other selection criteria listed in Table 1, so that we have a full set of information on the variables required for the risk prediction models.
|Risk model||At school||Sub-sample
In tertiary education
|Long-term NEET when aged 18||0.784||0.918||0.884|
|Did not achieve a level 2 qualification by end of year when turned 18||0.899||0.962||0.966|
|Long-term benefit receipt when aged 18||0.869||0.938||0.905|
Table 2 shows the area under the Receiver Operating Characteristic (or ROC curve) for each of the risk prediction models. The ROC curve is derived by plotting the model's sensitivity versus (1-specificity). The area under the ROC curve gives a summary measure of model performance across the range of predicted risk values. It ranges from 0.5 for a model with no predictive power to 1 for a model with perfect predictive power.
The ROC statistics range from 0.784 to 0.966, indicating moderately good through to good predictive accuracy. For all three risk measures, the predictive accuracy is higher for youth who had left school than for those who were still at school. This is not surprising, as the post-school activities of 16-17 year old school leavers are likely to be correlated with their post-school activities at 18, but we do not have this information for youth who are still at school. The predictive accuracy of the models of level 2 non-achievement is also higher than that of the models of NEET and benefit receipt at 18, reflecting the use of highly predictive variables such as the number of NCEA credits already obtained. However, we use the results of the NEET risk model in our targeting analysis because this outcome measure is most closely aligned with the targeting intent of the YS: NEET programme.
To further illustrate the predictive power of the risk models, we plot the actual NEET rates of youth who were 16-17 years in August 2012 over the following three years, grouping them by their predicted risk score in August 2012 from the NEET risk model (see Figure 1 panel A). Those with risk scores in the lowest 6 deciles are classified as 'low risk'. Those with risk scores in the 7th and 8th deciles are classified as 'medium risk'. Those in the highest two deciles as classified as 'high' and 'very high' risk, respectively.
Figure 1 shows that the actual monthly NEET rates of the youth who were in the highest deciles of predicted NEET risk were much higher than those of youth who were classified as being at lower risk. There are large and consistent differences in actual NEET rates across the four predicted risk groups. The same is true when we graph the actual monthly benefit receipt rates of youth who were 16-17 years in August 2012, grouped by their predicted risk of receiving a benefit in at least 6 calendar months of the year when they were aged 18 (Figure 1 panel B). Those in the higher-risk groups had substantially higher actual benefit receipt rates.
- Figure 1 - Actual NEET and benefit receipt rates of youth in different predicted risk groups
4 The targeting of YS: NEET
In this section of the paper we consider whether the YS: NEET programme has been well targeted, in the sense that the majority of participants were youth who were at a high risk of becoming NEET or moving on to a benefit at age 18. We also provide estimates of cumulative total programme participation rates among 16-17 year olds by the end of our impact evaluation study period, December 2013, to give an indication of the 'reach' of the programme by that time.
4.2 Programme participation rates by level of risk
As discussed above, participation in YS: NEET is voluntary. To help providers identify and contact high-risk youth, MSD calculates a model-based risk score for all 15-17 year old school leavers, using administrative data obtained from MSD and the Ministry of Education and a model of the likelihood of moving to a benefit at age 18. Information about these school leavers and their risk scores is passed to the contracted YS providers in each region. The providers can then contact the young people and invite them to participate in YS. They can also enrol other eligible youth they have contacted or been referred through other channels. The payments that YS providers subsequently receive for each young person in their programme are linked to the youth's risk score, with higher-risk participants attracting higher payments.
To assess the extent to which the most disadvantaged or 'at risk' youth participated in YS: NEET, we developed our own measures of relative risk, based on the predicted likelihood of experiencing each of three measures of poor outcomes at 18: being NEET for at least 6 months while aged 18 years; not having obtained a level 2 qualification or higher by 18 years; and receiving a benefit for at least 6 months while aged 18. These prediction models are described above in Section 3.3.
We were able to draw on a wider range of administrative data to model individuals' likelihood of experiencing poor outcomes at age 18 than is available to MSD, which means our predicted risk scores are likely to be more accurate. They also have broader coverage, because we calculate them for the majority of 16-17 year olds and not just school leavers.
Using our prediction models, all teenagers who were aged 16 or 17 between August 2012 and November 2014 were scored, in each calendar month, according to their predicted likelihood of experiencing each of the three 'poor' outcomes when they were aged 18. For each calendar month and each outcome measure, all 16-17 year olds were then classified into one of 10 risk deciles according to their relative risk score at that time (ranked from lowest to highest). We then use the decile groups to compare the risk profiles of 16-17 year old youth who were recruited into YS: NEET with those of all 16-17 year olds.
- Figure 2 - Rates of recruitment into YS: NEET by predicted risk decile
Figure 2 illustrates this by plotting the proportion of all 16-17 year olds in each risk decile who enrolled in YS: NEET for the first time in 2012, 2013 and 2014. The programme began in August 2012 and was only in operation for five months (building up participant numbers in that time), which explains the lower rate of recruitment in 2012 compared with the other two years and the lower position of the 2012 line on the graph.
The plot lines show that in its first 29 months of operation, YS: NEET recruited a few youth from all risk deciles, but a higher proportion of those in the highest risk deciles.
- Figure 3 - Proportion of new YS: NEET participants in each risk group
Turning to the new participants themselves, we show the proportion of new YS: NEET participants in each month from August 2012 to November 2014 who were in the different deciles of risk in Figure 3. Those with risk scores in the lowest 6 deciles are classified as 'low risk'. Those with risk scores in the 7th and 8th deciles are classified here as 'medium risk'. Those in the highest two deciles as classified as 'high' and 'very high' risk, respectively.
We can see that the proportion of new YS: NEET participants who were in the highest two risk deciles was initially 70-80%, but it declined gradually over 2012 and 2013 to around 50% and then plateaued during 2014. Both the medium and low risk groups increased their shares.
-  YS: NEET participants who were aged 15 or 18 at the time they started the programme were excluded from this analysis simply to reduce the scale of the work required. They represent a very small percentage of all participants.
4.3 Risk profile of participants by their enrolment status at the start
New YS: NEET recruits who were still in school when they started YS tended to have lower predicted risk scores than those who had already left school. This is illustrated in Figure 4. On the left side, we show the proportions who were in the low, medium, high and very high risk bands, focusing on the youth in our impact evaluation sample (who started YS: NEET in 2012 or 2013). On the right side, we show the same proportions for all youth who had started YS: NEET by November 2014.
- Figure 4 - Risk group in month of recruitment into YS: NEET
We find that:
- Only 30% of the 'at school' participants in the impact evaluation sample were classified as very high or high risk, compared with 75% of those in the 'in tertiary' and 72% of those in the 'not enrolled' groups.
- Turning to all YS: NEET participants from the start of the programme until November 2014, only 29% of participants in the 'at school' group were classified as very high or high risk, compared with 69% of those in the 'not enrolled' group.
Similar results were obtained using our alternative measures of poor outcomes at age 18, namely the predicted risk of not having completed a Level 2 qualification, and receiving a benefit for 6 months or more.
- Figure 5 - Enrolment status of new YS: NEET participants over time
We also find evidence that the proportion of new recruits to YS: NEET who were still enrolled at school was rising over time. Figure 5 shows the enrolment status of new YS: NEET participants over the period from August 2012 until the end of the September 2014 quarter, by quarter. It indicates that after the middle of 2013, the majority of those starting YS: NEET were already enrolled at a school or at a tertiary institution.
The decline in the average risk rating of new participants was partly due to the increase in the proportion who were still at school. However, risk scores also appear to have declined over time within groups. For example the proportion of 'not enrolled' youth who were in the 'high' or 'very high' risk groups fell from about 80% in the fourth quarter of 2012 to about 66% in the fourth quarter of 2013. Similarly, the proportion of 'at school' youth who were in the 'high' or 'very high' risk groups fell from about 32% in the fourth quarter of 2012 to about 24% in the fourth quarter of 2013.
One possible explanation for the declining 'risk rating' of new participants is that it became harder over time for providers to identify and successfully recruit high-risk youth who had not already been contacted. Another possibility is that the payment structure for providers made enrolling lower-risk youth more financially worthwhile than enrolling high-risk youth, and providers discovered this during the first year of the programme's operation.
4.4 Cumulative rates of participation by the end of 2013
The impact evaluation study design, in which we select non-participants who are as similar as possible to YS: NEET participants but did not enrol, requires a substantial pool of young people in each region who could have participated in YS: NEET but did not do so. If most young people with 'high risk' profiles enrolled in YS, we would not be able to find a sufficient number of eligible non-participants.
We demonstrate that this was not the case by estimating total rates of participation by region and risk group, as at December 2013, the end of our study selection period. These estimates are shown in Table 3.
At the regional level, the proportion of 16-17 year olds who had already enrolled in YS: NEET by 31 December 2013 ranged from about 2% in Otago to about 15% in Gisborne. If we focus solely on the 16-17 year olds who were in the two highest risk deciles, the proportion who had enrolled by the end of December 2013 is estimated to range from around 13% in Otago to about 35% in Gisborne.
These estimated participation rates should be regarded as indicative only, because there are limitations to the coverage and quality of the available data which will have reduced their accuracy. Nevertheless, the numbers in Table 3 suggest that there were a substantial number of youth in all regions who had not yet participated by the end of 2013, even in the highest risk deciles.
|All 16-17 year olds||16-17 year olds
in highest two risk deciles
|Bay of Plenty||6.3||17.7|
Notes: These are estimates based on the data currently available in IDI, and are subject to some significant data coverage errors.
- Figure 6 - Actual monthly NEET rates of non-participating youth by their predicted risk of becoming long-term NEET
Finally, Figure 6 plots the actual monthly NEET rates of the 16-17 year old youth in August 2012 who did not go on to participate in the Youth Service during the next two years (by which time, the full sample would have completely aged out of the programme's target age range). Based on the available data in IDI, it appears that the higher-risk non-participants were frequently inactive (ie,NEET), and so in principle would have been available to participate in YS. Moreover, because our measure of NEET is a conservative one which excludes individuals with any employment during the month, no matter how limited the hours, the true proportion who were essentially inactive in each month and could have been available to participate in the Youth Service is probably higher than shown. What we don't know, of course, is whether they were given the opportunity to participate in YS: NEET and declined, or whether they did not know of the existence of the programme.
4.5 Summary of findings on targeting
The results of this section show that the likelihood of becoming enrolled in YS: NEET in 2012-14 was positively correlated with a youth's relative risk of experiencing poor outcomes at age 18, with high risk youth being far more likely to participate. However, a substantial minority of participants were not drawn from the higher risk groups, and the proportion who were higher-risk decreased over time. The proportion of new recruits who were in the highest two risk deciles was initially 70-80%, but it declined gradually over 2012 and 2013 to around 50% and then plateaued during 2014. Both medium-risk and low-risk youth increased their participation shares.
The reduction in the strength of targeting was associated with an increase in the enrolment of youth who were still enrolled at school, who were less likely to be in the highest deciles of risk.
Our cumulative total participation rate estimates suggest that in all regions of the country, two-thirds or more of the high-risk youth who could have participated in YS by the end of 2013 had either not been contacted by that time or had refused to participate.
The fact that there are a substantial number of medium to high-risk non-participants means there was a sizeable pool of non-participants in every region from whom individuals with matching characteristics could be selected, to form a matched comparison group. Unfortunately, we are not able to distinguish those who refused to participate from those who were simply not contacted.
5 Participant characteristics and activities while enrolled
We summarise the characteristics and schooling history of YS: NEET participants in this section of the paper. We show that the matched comparison groups are well matched with the study population members in terms of these measured characteristics and experiences.
We also provide information on the nature of the education or training that was undertaken by YS: NEET participants while they were engaged in the programme or afterwards, and the qualifications they completed.
5.2 Characteristics of YS: NEET participants at the time of their enrolment
Tables A.3-A.5 in the Appendix summarise the characteristics and childhood experiences of our final matched sample of YS: NEET participants. There are four columns of data for the study population, giving results for those who were enrolled at school, enrolled in tertiary education, and not enrolled at the time of their recruitment to YS: NEET, and the total. Comparable results for the matched comparison group are given in the right-hand columns of the tables.
The vast majority of YS: NEET participants were aged 16 or 17 at enrolment, but 5% were aged 15 and 1.6% were aged 18. The gender split was roughly even, although participants who enrolled in YS: NEET while still at school were more likely to be female. Counting all ethnic group affiliations that have been recorded in IDI, 42% identified as European, 45% as Maori, 18% as one of the Pacific ethnicities, and 3% as Asian. The low proportion of Asians may be partly due to our study design.
The residence data indicates that participants tended to come from poorer socio-economic backgrounds. About 46% were living in a neighbourhood with a New Zealand Deprivation Index score of either 9 or 10 (the most deprived deciles). We estimate that 24% had a mother or female caregiver who did not have any formal qualifications - although data quality limitations mean this measure is not considered to be very accurate. Nearly 20% had a parent or caregiver who had served a custodial sentence at some earlier time, and 41% had a parent or caregiver who had served a community sentence. These measures of parental corrections history are also subject to data limitations and should be treated as indicative only.
Twenty-eight percent of participants had spent three-quarters or more of their childhood (from birth to the birthday before their YS: NEET enrolment) living with a parent or caregiver on a benefit. Forty-seven percent had spent at least half of their childhood with a parent/caregiver who was receiving a benefit. Fifty percent had been the subject of at least one CYF care and protection notification during their childhood. Twenty-four percent had been the subject of at least one substantiated CYF care and protection finding, and 7% had had a CYF care placement.
Eleven percent of participants had had at least one referral to CYF youth justice during their childhood. Twenty-eight percent had used mental health or addiction services in the secondary health care sector at some stage during their childhood.
A comparison of the data for the three enrolment sub-groups indicates that the 'at school' subgroup was the least disadvantaged of the three in terms of these measures of socio-economic status and childhood hardship, while the 'not enrolled' subgroup was the most disadvantaged.
Summary measures of school characteristics and educational achievement are set out in Table A.4, focusing on the current or most recently attended school. Most YS: NEET participants were attending or had attended low decile schools, but a minority were at, or had been at, high decile schools. Thirteen percent had attended five or more different schools since 2006, which is considerably higher than the expected 2-3 and represents a high level of movement between schools. Two percent had received special education funding at some time during their schooling. Special education funding is provided to assist children with disabilities.
Other results in Table A.4 show that about 28% had a truancy record, 37% had had at least one stand-down from a school and 15% had had at least one suspension from a school.
Of the participants who had left school before they enrolled in YS: NEET (the 'in tertiary' and 'not enrolled' subgroups), about a quarter had left school at 15 years or younger (before the official minimum school leaving age), and half at 16 years.
Due to data limitations, we know the calendar years in which qualifications were obtained but not the exact timing within the year. Sixty-three percent held no formal qualifications in the year before their YS enrolment. The rest had NCEA level 1 (23%), NCEA level 2 (12%), or a tertiary qualification (2%). Nearly all of the tertiary qualifications were National Certificates at level 1 or level 2.
The youth who began YS: NEET while they were still at school were more likely to have obtained NCEA level 2 already (17%) than those who had left school before enrolling in YS: NEET (about 10%). Others would have been working towards NCEA level 2 during the year that they enrolled in YS. The youth who enrolled in YS: NEET while they were still at school also tended to have lower school mobility and fewer past school infringements than the other two sub-groups.
Table A.5 presents data on other activities that were undertaken before participation in YS. The majority of the 'in tertiary' and 'not enrolled' participants had left school within the past 12 months. Twenty-six percent of all new YS: NEET participants had been enrolled in a tertiary education programme in the past 18 months (including tertiary courses that can be undertaken at the same time as attending school). Fifteen percent of youth in the 'in tertiary' group and 27% of those in the 'not enrolled' group had been 'NEET' for more than 6 months of the past 18.
Finally, comparison of the data on the two sides of tables A.3 to A.5 shows that the YS: NEET participants and matched non-participants are very similar on all of the indicators shown.
- We do not include recent migrants to New Zealand because the administrative data in IDI does not fully capture their characteristics and past activities.
- The education of the mother is only known if the mother has received a benefit at some time in the past, and the educational attainment of beneficiaries has not always been accurately recorded in the benefit records.
5.3 Retention in YS: NEET, studying rates and educational achievement patterns
Statistics on participants' enrolment and achievement patterns are given in Table 4. About two-thirds of the study population remained in the YS scheme for more than one year. Another 26 percent stayed in the scheme for between 6 months and one year. These figures include the enrolment time of individuals who switched from YS: NEET to the YP or YPP options. Recall that the youth who stayed enrolled in YS for less than 90 days are not included in our study population.
|Time enrolled in YS
in the first 18 months
|Education or training
participating in YS: NEET
|Some school attendance||100||10.8||19.4||46.3|
|Some tertiary attendance||36.2||100||54.7||56.3|
|Tertiary programmes that
started before YS enrolment
|Tertiary programmes that
started after YS enrolment
|Any school or tertiary attendance||100||100||64.8||83.6|
|No school or tertiary attendance||0||0||35.2||16.4|
|Any industry training||2.2||2.8||3.8||3|
|Any education or training||100||100||66.6||84.4|
Notes: All sample size numbers are randomly rounded.
Eleven percent of participants in the 'in tertiary' group and 19% of those in the 'not enrolled' group had a school enrolment during their YS participation period, indicating that they returned to school at least briefly, or perhaps enrolled at the Correspondence School.
Fifty-six percent of all YS: NEET participants were enrolled in a tertiary programme at some point during their YS enrolment. This include enrolments that had been started before the student was recruited into YS. Forty-five percent started a new tertiary programme during their YS enrolment.
Rates of participation in industry training were very low. About 3% of all participants and 4% of the 'not enrolled' subgroup were enrolled in industry training at some stage during their YS spell.
The purpose of YS: NEET is to raise participation in formal education or work-based training, but 35% of youth in the 'not enrolled' subgroup (and 16% of all participants) did not undertake any formal education or formal work-based training while they were enrolled. These youth tended to disengage from YS sooner than other participants, but nevertheless 68% remained enrolled in the YS for more than 6 months and 29% for more than one year.
|Study load, first year||Less than 0.5 EFTS||26.2||29.5||25.6||26.6|
|Type of provider||University||15.4||1.1||2||5.4|
|Private training establishment||41.9||67.6||60.1||56.7|
|Funding source||Student component||43.5||31.6||29.3||33.7|
|Field||Natural and Physical Sciences||2.2||s||s||0.8|
|Engineering and Related Technologies||9.6||5.8||7.3||7.6|
|Architecture and Building||6.9||6.9||6.8||6.8|
|Agriculture, Environmental and Related Studies||6.1||5.8||8.2||7.1|
|Management and Commerce||10.2||10.2||7.3||8.6|
|Society and Culture||12.1||10.2||8.4||9.8|
|Food, Hospitality and Personal Services||9.4||16.7||12.8||12.7|
|Employment or life skills (mixed field)||26.7||31.6||35.6||32.5|
Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.
Table 5 presents data on the tertiary programmes that were started by participants after enrolling in YS: NEET. (Recall that 44% started a new tertiary programme or programmes.) If more than one programme was started on the same day, we sum the study load and select the highest qualification that was enrolled for.
The majority (83%) of these tertiary programmes were aimed towards qualifications at levels 1, 2 or 3, with level 2 being most common. Three-quarters required less than one year of full-time study in the first year. Two-thirds were taught by a private training establishment (PTE) or a wananga, while the rest were mainly courses at polytechnics.
Nearly two-thirds of these courses were funded through the Youth Guarantee, which provides fee-free places to eligible students. About one third (33%) were 'mixed field' programmes, that is, courses on employment or life skills. The remaining two-thirds were courses teaching occupationally-focused skills in fields such as food, hospitality and personal services, management and commerce, and education.
|Industry training qualification||None||99.1||99.1||98.7||98.8|
|Field of highest tertiary qualification||None obtained||82||34.9||69||67.4|
|Natural and Physical Sciences||s||s||s||s|
|Engineering and Related Technologies||1.3||4.4||2.8||2.5|
|Architecture and Building||1.2||3.5||2||1.9|
|Agriculture, Environmental and Related Studies||1.7||4.2||3.1||2.7|
|Management and Commerce||2.2||5.5||3.1||3.2|
|Society and Culture||2||7.9||3.1||3.6|
|Food, Hospitality and Personal Services||2.1||11.2||4.3||4.8|
|Employment or life skills (mixed field)||4.7||18.5||8.7||9|
Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.
Data on the qualification attainment of the participants is given in Table 6. Because the qualification attainment data only record the year in which a qualification was completed and not the month of the year, we can't attribute qualifications exactly to the period of YS: NEET enrolment. Instead we simply report all qualifications that were completed in the year of first YS enrolment or the following year. The studying required to obtain the qualification may have been partly or fully carried out either before or after the YS: NEET enrolment period.
Starting with the highest school qualification that was obtained in the year of starting YS: NEET or the following year, 12% completed NCEA level 1 in this period, 32% NCEA level 2, and 16% NCEA level 3. Thirty percent completed a level 2 or higher tertiary qualification. Only 1 percent completed an industry training qualification at any level. Putting all types of qualification together, 57% completed a level 2 or higher qualification, 12% completed a level 1 qualification and 31% did not complete any new qualifications.
Looking at the fields of the tertiary qualifications, nine percent of the YS: NEET participants completed a qualification in life skills or general employment skills and 24% completed an occupationally-oriented qualification. Nearly all of these qualifications were National Certificates.
Participants in YS: NEET had many markers of potential disadvantage or risk, such as living in a low decile neighbourhood, having had past contacts with CYF, and showing signs of disengagement from school. There was some diversity in the profiles of different groups of participants, however, particularly when we compare the youth who started YS: NEET while they were still at school with those who started after they had left school.
The majority had either no qualifications or just NCEA level 1 in the year before their year of enrolment in YS, but 12% had completed NCEA level 2 already. Others would have been working towards NCEA level 2 during the year they were recruited into YS: NEET. It is unclear why the youth who had obtained NCEA level 2 already were recruited into the programme.
The majority (84%) of the youth in our study population undertook some form of education or training during their YS enrolment period. Forty-six percent were at school during some of the period they were enrolled in YS: NEET and 56% were enrolled in a tertiary programme. This includes tertiary enrolments that had started before the student was recruited into YS.
The majority of the tertiary programmes attended were at levels 1, 2 or 3, with level 2 being most common. The majority of these programmes required less than one year of full-time study to complete. About one-third were general employment or life skills courses.
Fifty-seven percent of YS: NEET participants completed a level 2 or higher qualification, and 12% a level 1 qualification, in either the year when they started YS: NEET or the following year. The other 31% did not complete any new qualifications in this period. As noted, the studying that was required to obtain these qualifications may have been partly or fully carried out before or after the YS: NEET enrolment period.
6 Impact estimates
In this section we present estimates of the impact of participation in YS: NEET. The overall results for all participants in the main study population are given in Section 6.1. In section 6.3, we consider the question of whether YS: NEET impacts differ for youth who were still at school when they started YS, enrolled in tertiary study, or not studying at all. In section 6.4, we look at whether the impacts vary by the youth's risk rating at the time they started YS: NEET. In section 6.5 we consider the variation in impacts by gender, ethnic group, and highest qualification. In section 6.6, we provide impact estimates for a broader study population that includes the youth who were enrolled in YS: NEET for less than three months, and look at variations in impact size by duration of enrolment in YS. Section 6.7 provides a summary of the findings.
6.2 Main impact estimates
Our main estimates of the impacts of YS: NEET participation are summarised in Table 7 and illustrated in Figure 7. Previewing these results, they show a moderate increase in the proportion who were studying over the first year of participation, which was not sustained after 18 months, and a very small increase in the proportion achieving level 1 or level 2 qualifications. On the other hand, participants were slightly more likely to receive a benefit during the follow-up period than matched non-participants, contrary to the programme objectives, and this effect was increasing over time as an increasing proportion of those in the study population reached their 18th birthday and become eligible for a benefit.
Results at the top of Table 7 show that the proportion of the YS: NEET participants who were enrolled in formal education (either school or post-school) was 9 percentage points higher 6 months after starting YS, and 4 percentage points higher 12 months after, than the comparable proportion of matched non-participants. About 66% of participants were enrolled in school or tertiary education after 6 months and 53% of participants were enrolled after 12 months. By eighteen months after enrolment in YS, the impact on educational participation was no longer significant.
As shown in Figure 7, the higher rate of formal study was essentially due to a higher rate of tertiary enrolment. The school enrolment profiles of the YS: NEET participants and their matched comparisons were similar.
Note that our measures of tertiary participation are based on tertiary enrolment records without the benefit of any data on attendance. If a student withdraws from a programme within the first few of weeks their enrolment record will be cancelled, but if they withdraw at a later stage no change is made to the administrative records and they will be counted in our estimates as 'studying' until the end of the enrolment period. Therefore, tertiary enrolment rates will tend to be overstated.
This matters for our study results if there was a significant difference between YS: NEET participants and non-participants in the likelihood of dropping out early. If the participants were more likely to drop out before the end of their programme, our estimates of the impact of YS: NEET on studying rates are likely to be overstated. On the other hand, if YS: NEET participants were less likely to drop out than the youth in the matched comparison group, our estimates of the impact of the impact on the programme on studying rates are likely to be understated. We have no evidence either way.
|Outcome||Time after starting
|N. partic.||Partic. mean
|Enrolled in formal education||6 months||9,063||66.2||57.6||8.6||0.67||*|
by one calendar year after
starting YS: NEET
|NCEA Level 1+||9,084||59.3||58.1||1.2||0.5||*|
|NCEA Level 2+||9,084||47.2||45.9||1.3||0.51||*|
|NCEA Level 3+||9,084||15.8||16.9||-1.2||0.41|
by two calendar years after
starting YS: NEET
|Receiving a benefit||6 months||9,063||8.7||6.7||2||0.42||*|
|Other outcomes targeted
by the programme
|Level 2 qualification by year turned 18||7,944||59||57.7||1.2||0.65|
|Benefit receipt in the 3 months after the 18th birthday||7,905||27.3||21.9||5.4||0.66||*|
|Custody in the 3 months after the 18th birthday||7,908||1.3||1.1||0.2||0.18|
|In employment||6 months||9,063||25||28.3||-3.2||0.57||*|
Notes: All sample sizes are randomly rounded. Estimates that are statistically significant at the 95% confidence level are marked with an asterisk. The sample sizes for the activity measures decline with time because we do not include individuals who were overseas for most of the month. We also exclude youth who were 18 or close to 18 when they started YS: NEET from the measures of 'other outcomes targeted by the programme'.
- Figure 7 - Outcomes of all YS: NEET participants and their matched comparisons
We next show measures of qualification achievement, giving the percentages of youth who had completed an NCEA level 1, level 2 or level 3 qualification, or any type of level 1, level 2 or level 3 qualification, in the first and second calendar years after participants started the programme.
NCEA achievement rates at levels 1 and 2 were slightly higher for the participants than their matched comparisons, although only by around 1 percentage point.
When we count tertiary qualifications as well as NCEA, the impact of YS: NEET on attainment is slightly greater. We see increases of 2.0 and 1.6 percentage points at level 1 and level 2 respectively, in the first calendar year after starting YS, and increases of 2.3 and 2.0 percentage points respectively, in the following calendar year. By the end of the second year, 74% had achieved a level 2 qualification compared with 72% of the matched non-participants.
There is no evidence of a positive impact on level 3 qualifications. In fact, the YS: NEET participants' attainment rates were about 2 percentage points lower than those of the matched non-participants.
We find that benefit receipt rates were raised by participation in YS: NEET. The proportion who were on a benefit was 2.0 percentage points higher 6 months after starting YS, 3.3 percentage points higher 12 months after, 3.9 percentage points higher 18 months after, and 3.3 percentage points higher 24 months after, than the comparable proportion of matched non-participants. By two years after starting YS: NEET, 28 percent were on a benefit compared with 25 percent of the comparison group.
We next consider three specific measures of outcomes at age 18:
- the proportion who had achieved a level 2 qualification by the end of the year when they turned 18
- the proportion who received a benefit in the 3 months after their 18th birthday, and
- the proportion who were in custody (prison or remand) during the 3 months after their 18th birthday.
We find that participation in YS: NEET was associated with a 1.2 percentage point increase in the proportion that had attained a level 2 qualification and a 5.4 percentage point increase in the proportion who were on a benefit in the first three months after their 18th birthday. Participation was not associated with a significant difference in the proportion who were in custody, however.
Turning to other measures of activity during the follow-up period, we find that the NEETrates of the participants were initially reduced by their higher rate of educational enrolment in the first year. Six months after starting YS, the proportion of participants who were NEET was 4.3 percentage points lower than the comparable proportion of youth in the comparison group. The impact was 1.8 percentage points after 12 months and not significant after 18 or 24 months.
Participants' employment rates were also reduced slightly by YS: NEET in the first year, by 3.2 percentage points after 6 months and 2.3 percentage points after 12 months. There was no significant effect after 18 or 24 months.
We provide impact estimates for a small number of supplementary outcome measures in Table 8. These cover:
- the proportion who served any custodial or community sentence during the 18 months after starting YS: NEET
- the proportion who were enrolled in industry training 6, 12, 18 and 24 months after starting YS: NEET
- the proportion who received a student allowance at 6, 12, 18 and 24 months after starting YS: NEET, and
- the change in mean monthly earnings, conditional on being in employment in both the six months immediately before YS enrolment and the 13-18 months afterwards.
|Outcome||Time after starting
|N. partic||Partic. mean (%)||Comp. mean (%)||Impact (%)||Std error (x100)||Sign.|
|Corrections sentences||Custody during first 18 months||9,081||1.8||1.7||0.1||0.2|
|Community sentence during first 18 months||9,081||5.8||4.9||0.9||0.3||*|
|Receiving a student allowance||6 months||9,063||2.1||2.2||0.0||0.2|
|Industry training||6 months||9,063||1.4||1.8||-0.5||0.2||*|
|Earnings growth: change in mean monthly earnings||13-18 months ($)||1,836||754||766||-12||29|
Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are marked with an asterisk.
There was no impact on the (very small) proportion who served a custodial sentence during the follow-up period. On the other hand, the proportion who served a community sentence was almost 1 percentage point higher for the participants than the matched non-participants.
Industry training enrolment rates were around 0.5% percentage points lower for YS participants than their matched comparisons, suggesting that by channelling some youth into formal education, YS: providers may have slightly lowered the likelihood of them entering industry training. However, the proportion of both participants and comparisons who were enrolled in industry training at any given time during the follow-up period was very low, at 1-1.5 percentage points.
Participation in YS had no significant impact on student allowance take up rates, or on the change in mean monthly earnings from before to after participation.
- Because the quality of the industry training enrolment data is relatively poor, with end dates frequently not being supplied, and only 3 percent of people in the study population did any industry training, we do not construct impact measures capturing participation in any form of education or work-based training but instead focus on formal education.
6.3 Impacts by enrolment status at the time of starting YS: NEET
Estimates of impacts by enrolment status at the time of recruitment to YS: NEET are shown in Table 9. The outcomes of the two largest sub-groups, the 'at school' and 'not enrolled' sub-groups, are also illustrated in Figure 8 and Figure 9.
These results suggest that participation in YS was of little benefit to the one-third of participants who enrolled when they were still at school. Over the next 18 months, these young people's rates of enrolment in formal education were slightly lower than those of their matched comparisons (by 2-3 percentage points at 6 and 12 months and 6 percentage points at 18 months). Participants were less likely to have had attained a level 1 or level 2 qualification by the end of the year following the year of commencing YS, by around 4 percentage points. Benefit receipt rates during the follow-up period were 1-2 percentage points higher.
|Outcome||Time after starting YS: NEET||Enrolled at school||Enrolled in tertiary education||Not enrolled|
|N Partic.||Partic mean
|Std E (x100)||Sign.||N Partic.||Partic mean
|Std E (x100)||Sign.||N Partic.||Partic mean
|Std E (x100)||Sign.|
|Enrolled in formal education||6 months||3,216||84.2||-2.9||0.7||*||1,632||70.9||7.9||1.6||*||4,215||50.6||17.7||1||*|
|Qualification achievement by calendar year after starting||Level 1+||3,225||83.9||-4.5||0.7||*||1,632||77.7||1.9||1.6||4,221||55.5||7||0.8||*|
|Receiving a benefit||6 months||3,216||3.9||1.3||0.4||*||1,632||9||1.6||1.1||4,212||12.3||2.7||0.7||*|
|Other outcomes targeted by the programme||Level 2 qualification by year turned 18||1836||79.8||-4.2||1||*||1,029||65.6||0.3||2.3||2,691||42.1||6.7||1.2||*|
|Benefit receipt in the 3 months after the 18th birthday||2,496||13.9||2.9||0.8||*||1,290||32.2||6.2||2.1||*||3,183||36.3||7.8||1.1||*|
|Custody in the 3 months after the 18th birthday||2,496||0.3||0||0.2||1,290||0.9||-0.2||0.4||3,183||1.8||0||0.3|
|In employment||6 months||3,216||24.6||-0.1||0.9||1,632||26.6||-2.4||1.8||4,215||24.8||-5.9||0.9||*|
Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are marked with an asterisk.
- Figure 8 - Outcomes of youth who were enrolled at school when they started YS
- Figure 9 - Outcomes of youth who were not enrolled in education when they started YS: NEET
The results for the group who were enrolled in tertiary programmes at the time they were recruited into YS (18% of the total) show small positive effects on subsequent educational enrolment rates, qualification achievement rates, and the proportion who were NEET in the follow-up period, but these estimates are mostly insignificant. Benefit receipt rates were slightly higher.
The results for the group who were not enrolled in education at the time they started YS (49%) show the largest positive impacts. These are set out in the right-hand columns of Table 9 and illustrated in Figure 9. We estimate that the proportion who were enrolled in formal education was nearly 18 percentage points higher 6 months after starting YS, 10 percentage points higher 12 months after starting, and 5 percentage points higher 18 months after starting. Thus, participation in YS had a fairly large impact on educational enrolment rates initially, with this impact diminishing over time.
The level 1 and 2 qualification achievement rates of this group were 6-7 percentage points higher than those of the matched comparison group in the year after the year of starting YS. The level 3 qualification attainment rate was not affected.
Despite the beneficial impacts of YS: NEET on educational enrolment, the benefit receipt rate of this group was a few percentage points higher than those of the matched non-participants throughout the follow-up period, with the difference rising over time to 5 percentage points 18 months after YS enrolment. NEET rates were lower than those of the matched comparisons at 6 and 12 months after YS enrolment, but no different at 18 months. The employment rates of the YS: NEET participants were also lower in the follow-up period, by 6, 4 and 2.5 percentage points at 6, 12, and 18 months respectively.
Note that the impact estimates presented in this paper could potentially be biased upward or downward by inexact matching on unobserved characteristics. One possible explanation for the different patterns of results by prior enrolment status is that the intakes from schools to YS: NEET were more negatively selected in terms of their unmeasured characteristics - such as their underlying attitudes to further study - than the intakes from the community.
Participation in YS: NEET is voluntary. Teenagers who hold negative attitudes about further study but are still enrolled at school might be more likely to agree to participate in YS than those with equally negative attitudes who have left school already. This is because those who are still at school do not have to make any significant changes to join YS, while those who have left school are expected to re-engage in education or training after enrolling in the programme - a much more significant change in behaviour.
If YS providers actively search out and enrol groups of youth who are still at school but somewhat disengaged from learning, and our matched comparisons are youth who had similar measured characteristics but were not quite as disengaged, then the impact estimates for the 'at school' group could be biased downward (ie, lower than they really should be). The true impacts of YS participation on enrolment retention and achievement could be neutral or slightly positive rather than negative.
Teenagers who have left school already are in a different situation. Since the YS promotes re-engagement in formal education or training, those with negative attitudes to education or little interest in further study may be less likely to agree to participate than youth with more positive attitudes. If so, those who enrol in YS: NEET could be positively selected and have more favourable unmeasured characteristics - in relation to the likelihood of studying and achieving further qualifications at least - than the non-participants that we have selected for comparison purposes. If this is the case, it would mean that our impact estimates for the 'not enrolled' groups would be biased upwards (ie, higher than they really should be).
- Drawing on results not presented here, these impacts on qualifications attainment were sustained in the following calendar year but not improved.
6.4 Impacts by the predicted risk of poorer outcomes at age 18
Our measures of the predicted risk of experiencing relatively poor outcomes at age 18 can be used to explore the question of whether 'higher risk' youth benefit more from participation in YS than 'lower risk' youth.
We present impact estimates for youth in four different risk groups in Table 10. The risk groups are constructed using our modelled predictions of the relative risk of being long-term NEET at 18. Recall that four risk groups were constructed after allocating all 16 and 17 year olds to decile groups according to their predicted long-term NEET risk score. The 'low risk' group comprises individuals in the lowest 6 deciles; the 'medium risk' group comprises individuals in deciles 7 and 8; the 'high risk' group comprises individuals in decile 9 and the 'very high' risk group those in decile 10. Nineteen percent of the YS: NEET participants in our study population were in the low risk group, 22% in the medium risk group, 25% in the high risk group and 34% in the very high risk group.
To obtain the impact estimates reported in Table 10, we exactly matched participants and comparisons within these four risk groups as well as by the other variables included in the matching procedure. The results in Table 10 are restricted to 16 and 17 year olds - the small number of 15 and 18 year olds in the study population and comparison groups were dropped.
The results show that YS: NEET participants in the 'low risk' group gained little benefit from participation. A small positive impact on educational participation was short-lived. Most of the estimated impacts on enrolment rates and educational achievement were small and non-significant, with the exception of benefit receipt rates, which were slightly raised.
The results for participants in the 'medium risk' group are substantively the same as for those in the low risk group.
Teens in the 'high risk' group had higher educational enrolment rates after starting YS, by 10 percentage points at 6 months and 5 percentage points at 12 months. But their educational achievement rates were not significantly higher, and members of this group has significantly higher benefit receipt rates in the follow-up period than their matched comparisons (by 5 percentage points at 12 months after starting YS).
Only participants in the 'very high risk' group (representing the top decile of risk) appear to have benefitted in the manner intended by the programme. Their enrolment rates were approximately 14, 9 and 5 percentage points higher at 6, 12 and 18 months after starting YS. Level 1 and level 2 achievement rates were approximately 6 and 5 percentage points higher than those of the matched non-participants in the year after YS enrolment. However, the proportion who were on a benefit was also around 3 percentage points higher than the comparable proportion of matched non-participants. Employment rates were also lower during the two-year follow-up period.
We ran similar impact estimates using the 'benefit receipt at 18' risk measure and these were substantively similar.
|Outcome||Time after starting YS: NEET||Low risk||Medium risk||High risk||Very high risk|
|Std E (x100)||Sign.||Partic mean
|Std E (x100)||Sign.||Partic mean
|Std E (x100)||Sign.||Partic mean
|Std E (x100)||Sign.|
|Enrolled in formal education||6 months||72.3||3.1||1.3||*||74.3||4.1||1.5||*||69.6||9.9||1.6||*||54.5||14.3||1.7||*|
|Qualification achievement by calendar year after starting YS||Level 1+||93||-0.3||0.6||85.6||-0.1||0.9||69.2||1.3||1.3||46.9||5.7||1.4||*|
|Receiving a benefit||6 months||3.3||1.3||0.5||*||5.6||2.8||0.7||*||9.1||2.1||0.9||*||13.3||1.2||0.9|
|Other outcomes targeted by the programme||Level 2 qualification by end year turned 18||83.2||-0.9||1.3||75.9||-0.9||1.6||57||-1||1.9||36.2||4.7||1.7||*|
|Benefit receipt in the 3 months after 18th birthday||7.6||2.3||1||*||16.2||5.6||1.2||*||32.7||7.6||1.7||*||42||6.4||1.6||*|
|In employment||6 months||55.2||0.4||1.7||27.5||-5.1||1.6||*||18.3||-3.6||1.6||*||11.3||-2.9||0.9||*|
Notes: 'Low risk' means the lowest 6 deciles in the predicted risk of being long-term NEET at age 18. 'Medium risk' means being in the 7th or 8th deciles, 'high risk' means being in the 9th decile and 'very high risk' means being in the top decile. The (rounded) sample sizes are as follows: 1,515 low risk, 1,776 medium risk, 2,001 high risk, 2,691 very high risk. Estimates that are statistically significant at the 95% confidence level are marked with an asterisk.
6.5 Impacts by gender, ethnic group, and highest qualification
A breakdown of the impacts by gender is given in Table A.6. The impacts estimated for males are slightly larger than those for females. For example, the increases in level 1 and level 2 achievement rates associated with YS: NEET participation are nearly 3 percentage points for males, but smaller and not statistically significant for females. Participation in YS: NEET was associated with larger increases in studying and reductions in NEET rates for males than for females, but also larger increases in benefit receipt rates.
The fact that the impacts were larger for males may be partly due to a higher proportion of male participants having left school at the time they started YS. As shown above, we did not find positive impacts on average for youth who were recruited when they were still at school.
A breakdown of impacts by ethnic group is set out in Table A.7. To obtain these results, we used a priority ranking scheme to assign each individual to a single ethnic group, and re-selected their comparisons using exact matching by ethnic group as well as gender, age, highest qualification, region and reference month. We show results for the European, Maori and Pacific groups only.
Youth in all three ethnic groups were more likely to be enrolled in formal education in the 12 months after starting YS. The impacts on studying rates were larger for Europeans and Maori than for Pacific youth. On the other hand, the impacts on educational achievement were larger for Pacific youth. We do not find statistically significant increases in the proportion of European or Maori youth who held a level 2 qualification by the end of the year after starting YS. For Pacific youth, in contrast, we find a 4 percentage point increase in this measure of achievement.
Disaggregating the impacts by highest qualification in the year before the year of enrolment in YS (Table A.8), we find somewhat larger increases in both studying rates and qualification achievement rates among unqualified youth than those who held NCEA level 1 or level 2 already. We estimate that YS: NEET was responsible for a 2 percentage point increase in the level 2 achievement rate of youth who previously had no qualifications and a 1 percentage point increase in the level 2 achievement rate of youth who already had NCEA level 1.
6.6 Impacts by duration of enrolment in Youth Service
So far, we have focused on a study population restricted to youth who stayed in YS: NEET for at least 90 days in the first 12 months. We now relax that constraint and estimate average impacts for participants of all durations. We also consider whether the impacts are different for different periods of enrolment.
Table A.9 gives an alternative set of main results based on an expanded study sample, including those who left after fewer than 90 days. These results are similar to those reported previously, but weaker. For example, the estimated impact on the level 1 qualification achievement rate is 1.5 percentage points rather than 2.0, and the impact on the level 2 achievement rate is 1.0 percentage points rather than 1.6.
Table A.10 shows how the pattern of impacts varies by the length of time the youth stayed enrolled: 0-3 months, 3-6 months, 6-12 months, and 12 months or more. Some of the people in the last group were in the programme for more than two years.
The results show negative or insignificant programme impacts for those who were enrolled for less than 3 months, slightly negative but mostly insignificant impacts for youth who were enrolled for 3-6 months, mixed results for those who were enrolled for 6-12 months, and larger positive results for those who stayed in the programme for more than 12 months. For the latter group, rates of enrolment in education were estimated to be 14, 9 and 5 percentage points higher at 6, 12 and 18 months after starting YS: NEET. The proportion with a level 2 qualification is estimated to be 4 percentage points higher by the end of the year after the year of YS enrolment.
These results suggest that the programme was more likely to benefit youth who stayed in it for at least a year. Caution is needed in drawing this conclusion, however. The duration of the YS: NEET enrolment period could be an endogenous variable that is influenced by whether or not the young person chooses to continue studying. This could be the case if providers terminate the YS enrolments of the teenagers who drop out of school or leave their tertiary programmes, for example. If this is the case, it would be wrong to conclude that a longer YS enrolment period leads to better outcomes.
The duration of the YS: NEET enrolment period is also a very blunt measure of the amount of support that was delivered through the programme. If data on the number of contacts and the type of services provided to each participant were made available in IDI in future, we could explore the question of whether those who receive more assistance experience greater benefits in a more meaningful way.
- Youth who leave the local area, for example, will probably be dis-enrolled.
6.7 Summary of findings on the impacts of YS: NEET
YS: NEET is intended to help youth who are at high risk of becoming inactive and/or moving on to a benefit when aged 18. Our analysis of the targeting of the programme showed that the proportion of new participants who were in the highest two deciles of risk (relative to all other 16-17 year olds) was initially around 70-80%, but it declined gradually over 2012 and 2013 to only around 50%, and then plateaued at this level during 2014. Thus, the targeting of the programme weakened over time. The available data in IDI show that a high proportion of the high-risk non-participants were NEET during the study period, suggesting that in principle, they could have participated in the programme. Unfortunately, we don't have any information on whether they were offered a place and refused to participate or were not contacted.
Targeting was particularly weak among the one-third of participants who were recruited to the YS while still at school. More than 70% of youth in this sub-group were not in the two highest deciles of risk.
Turning to the impacts of YS: NEET, we find evidence of a positive impact on participants' educational participation rates over the first 12 months, a very small increase in the attainment of level 1 and level 2 qualifications, and a small increase (rather than decrease) in the proportion that moved onto a benefit during the two-year follow-up period.
Specifically, by comparing the outcomes of participants with those of a matched comparison group, we estimate that:
- the proportion who were enrolled in formal education was 9 percentage points higher 6 months after starting YS: NEET and 4 percentage points higher 12 months after. This was mostly due to a higher rate of enrolment in tertiary programmes at levels 1-3.
- the proportion who had completed a level 2 qualification was 1.6 percentage points higher one year after enrolment in YS: NEET and 2.0 percentage points higher two years after.
- benefit receipt rates were raised. The proportion who were on a benefit was 2.0 percentage points higher 6 months after starting YS, 3.3 percentage points higher 12 months after, and 3.9 percentage points higher 18 months after, than the comparable proportion of matched non-participants. By 24 months after starting YS, 28 percent were on a benefit.
- the NEET and employment rates of the YS: NEET participants were reduced during the first year after enrolment in the programme (reflecting the effects of higher participation in education), but these effects did not last. In the second year after starting the programme, the NEET and employment rates of participants were not significantly different from those of the matched non-participants.
- there was no significant change in the proportion who were in custody during the follow-up period. The proportion who served a community sentence was slightly higher among participants than among the matched non-participants.
While the average effects of YS: NEET on qualification attainment were small, there were some material differences in these impacts between sub-groups of participants. On average, those who enrolled in YS: NEET while they were still at school or already enrolled in tertiary education did not benefit from the programme. In contrast, those who were disengaged from formal education at the time of starting YS: NEET had a level 2 qualification achievement rate that was 6 percentage points higher than that of the matched non-participants by the end of the year after starting YS.
Youth with a relatively high risk of experiencing poor outcomes at age 18 - and particularly those in the highest decile of risk - also show larger positive educational impacts following participation than other youth. We estimate a 5 percentage point improvement in the level 2 qualification attainment rate of youth in the highest risk decile, but no significant impact for youth in all other risk groups.
Our results also suggest that youth with no qualifications when they enrolled in YS: NEET benefitted more from their programme participation, in terms of increases in subsequent qualification attainment, than youth who held NCEA level 1 or higher qualifications at the time they started.
We find evidence of higher benefit receipt rates in the follow-up period for every sub-group of participants - regardless of whether or not YS was found to have positive impacts on their rates of studying or their educational achievement. We think this could be a genuine programme effect, which is likely to be a consequence of the needs assessment work carried out by YS providers (leading to better identification of income needs and eligibility) or perhaps increased contact with other youth who are on benefits (encouraging the YS: NEET participants to apply when they turn 18).
Because this is an observational study and not a randomised trial, we acknowledge that the estimates in this paper could be biased by differences between the participants and matched non-participants in unobserved characteristics, such as mental health, disabilities, substance addictions, personality traits, or the motivation to learn.
There are signs that unobserved characteristics could be influencing some of our results. For example, our results for the youth who started YS: NEET while they were still at school show that participation had small negative effects on tertiary enrolment and qualification achievement rates. Our results for youth who stayed enrolled in YS: NEET for less than 6 months also imply that participation reduced rather than improved educational enrolment rates and qualification achievement. Since it is unlikely that participation in YS had these negative effects, it is more likely that these groups of participants differed from their matched comparison in terms of some unobserved characteristics that we were not able to control for in our matching strategy, such as the motivation to continue studying. For these youth, the impact estimates we report may be biased downwards, due to 'negative selection' on unobserved characteristics (eg, the participants had less interest in further education than the non-participants we matched them to).
For other groups of participants, positive selection on unobserved characteristics could be causing upward bias in the results. In particular, young people who were recruited from the community and were not already engaged in formal education are likely to have been more motivated to undertake further study than other similar youth - or they would not have agreed to participate. If they were more motivated, they would have been more likely to return to education and gain a qualification than other youth with similar measured characteristics, even if the YS: NEET programme did not exist. For these groups, the true impact of YS: NEET on studying rates and qualification attainment may be smaller than our estimates suggest.
We are not able to say whether the overall results are more likely to be biased downward or biased upward by unobserved characteristics. Only a well-designed and well-implemented experimental study could give unbiased estimates of the true programme impacts.
7 Impacts of the Youth Transition Service
The Youth Transition Service (YTS) operated from 2004 to 2012 and was broadly similar in its objectives to YS: NEET. The goal of YTS was to ensure that all 15-19 year olds were in work, education, training, or other activities that would contribute to their long-term economic independence and well-being (MSD, 2008, p1). Community organisations around the country were contracted to contact school leavers, engage with young people who were at risk of prolonged disengagement from work, education or training, and provide these at-risk youth with customised support and guidance to facilitate their re-engagement in appropriate work, education or training.
The YTS programme had two main streams: 'follow-up', intended for youth who had a plan or destination after secondary school and no significant issues putting them at risk of long-term inactivity, and 'customised support', intended for youth who did not have a plan or destination following secondary school, or had significant issues or barriers placing them at risk of long-term inactivity. The 'customised support' strand of the programme was more similar to YS: NEET than the 'follow-up' strand.
We investigated the impacts of YTS using IDI data on young people who enrolled in the programme between 1 January 2008 and 31 December 2011. Their educational and labour market outcomes were tracked for the following 3-5 years (depending on their start date and the period of data available in IDI).
Our study sample comprises all YTS participants who were successfully matched to other data in IDI and met certain other criteria, such as being aged 15-18 when they started, attending a school that offered only NCEA qualifications, being in New Zealand for most of the study period, and staying enrolled in YTS for at least 30 days. The selection criteria were almost identical to those listed in Table 1 for the YS: NEET study sample. The final YTS study sample comprises approximately 54% of everyone who enrolled in YTS between 1 January 2008 and 31 December 2011. This fraction is relatively low because one-quarter of all YTS records could not be linked to IDI, due to the poorer quality of the name and birthdate records obtained from YTS providers.
The evaluation uses the same methods as were used for the evaluation of YS: NEET. We select a comparison group of youth who were as similar as possible to the individuals in the YTS study sample, but who did not participate. The outcomes of the comparison group members in the follow-up period provide the 'counterfactual' against which the outcomes of the study population members are compared. Propensity score matching methods were used to select the most appropriate comparison group matches for each individual in the study population. The propensity score models were essentially the same as those outlined in Section 3.2.3 above. The main difference is that we used two years of historical data on individuals' prior education and employment rather than four years, because the period of data available in IDI was more limited.
7.2 Profile of YTS participants and their educational participation
Nine percent of the youth in our study sample registered with YTS but received no further assistance. Forty-six percent received 'follow-up' services and 45% received 'customised support'.
Summary statistics on the characteristics of the YTS participants are set out in Table A.11 in the appendix. In terms of their socio-economic circumstances, they appear to have been similarly disadvantaged as participants in YS: NEET. For example, the proportion that attended decile 1-2 schools was similar, as was the proportion living in neighbourhoods classified to categories 9 or 10 in the NZ Deprivation Index (representing the most disadvantaged neighbourhoods).
One significant difference is that a higher proportion of the YTS group were still at school when recruited (nearly two-thirds), and a higher proportion remained at school afterwards. About 50% of YTS participants were enrolled at school two months after starting YTS, compared with 35% of the YS: NEET sample.
A second major difference is that 23% were aged 18, compared with less than 2% of the YS: NEET sample. A third difference is that they were more likely to hold a school qualification before they enrolled: 50 percent had no school qualifications compared with 63% of the youth who were recruited to YS: NEET. In view of the evidence presented above on variations in the impacts of YS: NEET, these differences could be expected to weaken the impact of YTS compared with that of YS: NEET.
Summary statistics on the activities of YTS participants while on the programme are set out in Table A.12 in the appendix. Sixty-two percent attended school while they were on YTS and 39% took some tertiary courses. Of those who enrolled in tertiary courses, more than half enrolled in level 4 or higher programmes, compared with just 17% of the YS: NEET participants who enrolled at tertiary level. The fact that YTS participants were more likely to be taking intermediate-level rather than basic tertiary programmes would lead us to expect better outcomes for YTS participants, on average, than we observed for YS: NEET participants, but not necessarily larger programme impacts.
7.3 YTS impact estimates
Impact estimates for all YTS participants, regardless of what level of assistance they received, are given in Table 11 and illustrated in Figure 10 (showing monthly activities) and Figure 11 (showing qualifications achieved).
The average impacts of YTS were insignificant or slightly negative. Participants were 1-3 percentage points less likely to be studying in the follow-up period than their matched non-participants, about 2 percentage points more likely to be on a benefit, 1-2 percentage points more likely to be NEET, and up to 1 percentage point less likely to be employed. We find no significant impact on qualification achievement. In addition, there is no real improvement in the estimated impacts from one year after YTS enrolment to five years after: using a longer follow-up window does not lead to better results.
Impact estimates for the main sub-streams of YTS are set out in Table A.13 in the appendix and illustrated in Figures 12-14 below. These results show that the young people who were allocated to the 'follow-up' stream had somewhat better outcomes than the non-participants we matched them to. They were around 3 percentage points more likely to complete a level 2 qualification, 1-2 percentage points less likely to be on a benefit or NEET, and 1-2 percentage points more likely to be employed, during the five-year follow-up period.
|Outcome||Time after starting YTS||N. partic||Partic. mean (%)||Comp. mean (%)||Impact (%)||Std error (x100)||Sign.|
|Enrolled in formal education||6 months||19,812||59||61||-2||0.38||*|
|Level 2 qualification achievement||First year||19,881||61.1||60.8||0.4||0.27|
|Receiving a benefit||6 months||19,812||12.4||10.4||2||0.25||*|
|In employment||6 months||19,812||31.3||32.2||-0.9||0.38||*|
Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are marked with an asterisk.
- Figure 10 - Outcomes of all YTS participants and their matched comparisons
- Figure 11 - Qualification attainment of all YTS participants and their matched comparisons
- Figure 12 - Outcomes of participants in the 'follow-up' option of YTS
The opposite is the case for the 'customised support' group, who were 3-6 percentage points less likely to be enrolled in formal education during the following five years, 6-7 percentage points more likely to be in receipt of a benefit, 5-8 percentage points more likely to be NEET and about 4 percentage points less likely to be employed, than their matched comparisons. The 'customised support' group were also 2-3 percentage points less likely to attain a level 2 qualification.
Given the fairly low level of support that was provided to youth in the 'follow-up' steam, it is questionable whether the small positive impacts found in our estimates should be attributed to the programme. An alternative explanation for the overall pattern of results - positive for 'follow-up' but negative for 'customised support' - is that youth were allocated to one of the two streams on the basis of both characteristics that can be measured in IDI and characteristics that we lack adequate information about, such as their mental health, use of alcohol or drugs, whether they had children, or whether they had recently experienced a crisis or set-back in their lives. The 'follow-up' group could be positively selected on these unmeasured characteristics, when compared to both the 'customised support' groups and their matched comparison group. The 'customised support' group could be negatively selected. Heterogeneity on unmeasured characteristics, and our inability to select matched comparisons with exactly the same unmeasured characteristics, could alone explain the pattern of small positive and small negative impacts, independently of any effects of the YTS programme.
In summary, the impacts of YTS participation appear to have been small or insignificant. Although our impact estimates could be a little too low or a little too high because of the effects of unmeasured variables that we weren't able to take into account, we conclude that participation in YTS probably did not have substantial impacts on participants' outcomes. Large impacts are likely to be detectable using this method even if the participant and non-participant groups are imperfectly matched.
- Figure 13 - Outcomes of participants in the 'customised support' option of YTS
- Figure 14 - Qualification attainment of YTS participants by sub-programme
This paper has evaluated the impact of YS: NEET on the educational retention, qualification achievement, benefit receipt, NEET rates and employment rates of participating youth in the 18-24 months after they enrolled in YS: NEET. Administrative data from the Integrated Data Infrastructure (IDI) was used to measure participants' outcomes. The impacts of the programme were estimated by comparing the outcomes of participants with those of a matched comparison group of similar youth who did not participate.
The programme is intended to help youth who are at high risk of becoming NEET and/or moving on to a benefit when aged 18. However, we estimate that in practice, only about half the participants in 2012-14 were high-risk youth - defined here as those in the top 20% of the risk spectrum. This evaluation can't determine the reasons for this, but the finding suggests the programme's design or contracts with providers could be reviewed to see if targeting can be strengthened.
Our impact estimates show that participation in YS: NEET led to increases in young people's participation in study or training, by up to 9 percentage points at the peak. For the average participant, however, it failed to materially raise qualification achievement. On average the programme raised level 1 and level 2 achievement rates by about 2.0 percentage points, both one year and two years after the year of enrolment in YS: NEET.
One possible reason for the limited translation of higher educational participation into higher qualification attainment is that the participation impacts reported in this paper are over-estimated - because they are based on enrolment data alone and don't take into account students' actual attendance patterns, which may be lower. Another possibility is that the support provided through YS: NEET promotes retention in education, but is not well tailored to increase academic achievement.
Very high risk youth - those in the highest decile of the predicted risk of poor outcomes at age 18 - appear to have benefitted from participation in YS: NEET to a greater degree than other youth, showing increases in their level 2 qualification attainment rates of around 5 percentage points. This finding suggests that the programme might have had slightly larger impacts per participant if it had been more tightly targeted on very high risk youth. However, we need to be somewhat cautious in drawing this conclusion. There is a possibility that our estimates of the programme impacts for high-risk youth are too high, if those who participate in YS: NEET are more motivated to study and gain further qualifications than other high-risk youth. If this is the case, then changing the programme to increase its focus on high-risk youth - but recruiting a greater number of the less motivated - might not raise qualification attainment to the same extent.
We did not find any evidence of positive effects on young people's likelihood of being NEET (not in employment, education or training) beyond the period of programme participation. Nor did we find evidence of positive impacts on employment rates. This is consistent with the findings of a previous study on the impacts of tertiary study for young people who leave school with either no qualifications or very low qualifications (Tumen, Crichton and Dixon, 2015). The earlier study found that level 1 to level 3 tertiary programmes only increased employment rates if the student was successful in completing a qualification. Given that YS: NEET only raised level 1 and level 2 attainment rates by 2 percentage points and had no effect on level 3 qualification attainment, we would not expect much improvement in employment rates to follow.
In contrast, benefit receipt rates were a few percentage points higher in the follow-up period, for every sub-group of YS: NEET participant. We think this could be a genuine programme effect that is caused either by the needs assessment work carried out by YS providers (leading to better identification of participants' income needs and eligibility), or by increased contact with other youth who are on benefits (encouraging YS participants to apply when they turn 18).
Two limitations of this evaluation are the relatively short follow-up period, and the use of propensity score matching rather than experimental methods to construct the 'control' group. At this time the analysis was done, we only had information on outcomes for the 18-24 months after enrolment in YS: NEET. Although this follow-up period is fairly short, the outcome and impact measures do not show signs of positive impacts becoming larger through time. Our evaluation of the Youth Transitions Service - the predecessor programme that was replaced by the Youth Service - used a five-year follow-up period, but also did not find any sign of positive impacts becoming larger with the passage of time. Therefore, it seems unlikely that the results reported here for youth who started YS: NEET during 2012 or 2013 will improve when more time has passed.
Evaluations based on propensity score matching methods have been found to show biases when compared with evaluations based on well-implemented randomised controlled trials (McKenzie et al, 2006). This is due to the fact that non-experimental methods can't ensure that the 'study' and 'control' groups in an impact evaluation are perfectly matched on all factors that influence outcomes. We acknowledge there is a risk that the impact estimates given in this paper could be too high or too low, due to the method used. We see them as the best available estimates in the absence of a more rigorous evaluation approach.
-  As noted above, IDI doesn't include data on course attendance.
Caliendo, Marco and Sabine Kopeinig (2005) 'Some practical guidance for the implementation of propensity score matching.' IZA DP No. 1588.
Dixon, Sylvia and Sarah Crichton (2011) 'The labour market returns to further education for working adults'. Department of Labour research paper.
DuBois, David L., Nelson Portillo, Jean E. Rhodes, Nadia Silverthorn, and Jeremy C. Valentine (2011) 'How effective are mentoring programmes for youth? A systematic review of the evidence'. Psychological Science in the Public Interest, 12(2) 57-91.
McKenzie, David, John Gibson and Steven Stillman (2006) 'How important is selection? Experimental versus non-experimental measures of the income gain from migration.' Motu Working Paper 2006-02.
Ministry of Social Development (2008) 'Youth Transition Service Evaluation: Final Report'. Centre for Social Research and Evaluation, MSD, unpublished report.
Ministry of Social Development (2012) Youth Service Operational Manual. Ministry of Social Development, unpublished report.
Ministry of Social Development (2014). Youth Service Evaluation Report. Ministry of Social Development. Wellington.
Ministry of Social Development (2015) Youth Service outcomes update March 2015: Technical report. Ministry of Social Development, unpublished report.
Rodriguez-Planas, Nuria (2014) 'Do youth mentoring programs change the perspectives and improve the life opportunities of at-risk youth?' IZA World of Labor 2014: 62, wol.iza.org
Tumen, Sarah, Crichton, Sarah and Dixon, Sylvia (2015) 'The impact of tertiary study on the labour market outcomes of low-qualified school leavers'. Treasury Working Paper 2015-07.
- Table A.1 - Demographic characteristics and childhood and schooling history of the study population and potential comparison group, before matching
Table A.1 continued - Demographic characteristics and childhood and schooling history of the study population and potential comparison group, before matching
Table A.2 - Variables used in the propensity score regressions
Table A.2 continued - Variables used in the propensity score regressions
Table A.3 - Personal characteristics and childhood history of the matched samples
Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.
Table A.3 continued - Personal characteristics and childhood history of the matched samples
Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.
Table A.4 - Schooling history and achievement of the matched samples
Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.
Table A.5 - Other activities of the matched samples before enrolling in YS: NEET
Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.
Table A.6 - Impacts by gender
Table A.7 - Impacts by ethnic group
Table A.8 - Impacts by highest qualification in the year before enrolling in YS: NEET
Table A.9 - Impacts of YS: NEET for an expanded sample including early leavers
Table A.10 - Impacts by duration of enrolment in YS: NEET
Table A.11 - Characteristics of YTS participants and their matched comparisons, by sub-programme
Table A.12 - Enrolment patterns of YTS participants, by sub-programme
Table A.13 - Impacts of YTS by sub-programme
- Table A.13 continued - Impacts of YTS by sub-programme