Back to top anchor
Working paper

Evaluation of the impact of the Youth Service: Youth Payment and Young Parent Payment (WP 16/07)

 

Abstract

The Youth Service is a programme administered by the Ministry of Social Development, designed to encourage and assist disadvantaged youth to stay in education and achieve qualifications. There are three main strands of the programme. The Youth Service (YS) is provided to recipients of the Youth Payment (YP) and Young Parent Payment (YPP) benefits, while the YS:NEET service is aimed at disadvantaged young people at risk of becoming detached from employment, education and training. This report focusses on the YP and YPP strands. Community organisations are contracted to provide mentoring and support for youth participating in the service. This is complemented by other changes to youth benefits intended to encourage continued study, including; obligations to participate in the service and in formal study, financial incentives, sanctions for failing to meet obligations, and access to childcare payments.

This paper evaluates the impact of the programme on the educational retention, qualification achievement, benefit receipt, and employment rates of participating youth in the 24 to 30 months after they come onto benefit. Administrative data from the Integrated Data Infrastructure (IDI) is used to measure outcomes. The impacts of the programme are estimated by comparing the outcomes of participants with those of an historical comparison group of youth beneficiaries. This comparison is adjusted to control for other changes over time that could have affected outcomes, using two matched comparison groups of similar young people who were not receiving a youth benefit. Results from the study should be treated with some caution, as they are reliant on a number of assumptions which could not be empirically tested. While we believe these results represent the best that can be done to get at the true impact of the programme, the results may not be robust if the assumptions do not hold.

We find that YS raises enrolment in formal education for young beneficiaries, relative to the previous youth beneficiary case management approach, with the effect being largely sustained over a two-year period for young parents, and a shorter period for other youth beneficiaries. The proportion of YP recipients who complete a level 1 or 2 qualification is raised slightly through participation in the programme, while the impact for young parents occurs slightly later, is larger, and occurs at levels 1, 2 and 3. Participation in the programme appears to raise subsequent benefit receipt rates in the short term, but there is some evidence that it encourages a move off benefit and into work in the medium term (24 to 30 months after starting benefit), especially for YPP participants.

Acknowledgements

We would like to thank Dean Hyslop for his suggestions on the methods used in this paper; Marc de Boer for his advice on the benefit data; and Marc de Boer, Michelle Bly, Gulnara Huseynli, Dean Hyslop, and Judd Ormsby for their helpful comments on earlier drafts of this paper. The opinions expressed in the report, and any remaining errors remain the responsibility of the authors.

Disclaimer

The views, opinions, findings, and conclusions or recommendations expressed in this Working Paper are strictly those of the author(s). They do not necessarily reflect the views of the New Zealand Treasury, Statistics New Zealand, or the New Zealand Government. The New Zealand Treasury, Statistics New Zealand, Ministry of Justice and the New Zealand Government take no responsibility for any errors or omissions in, or for the correctness of, the information contained in this Working Paper. The paper is presented not as policy but with a view to inform and stimulate wider debate.

The results in this report are not official statistics - they have been created for research purposes from the Integrated Data Infrastructure (IDI) managed by Statistics New Zealand. Ongoing work within Statistics New Zealand to develop the IDI means it will not be possible to exactly reproduce the data presented here.

Access to the anonymised data used in this study was provided by Statistics New Zealand in accordance with security and confidentiality provisions of the Statistics Act 1975. Only people authorised by the Statistics Act 1975 are allowed to see data about a particular person, household, business or organisation. The results in this report have been confidentialised to protect these groups from identification.

Careful consideration has been given to the privacy, security and confidentiality issues associated with using administrative and survey data in the IDI. Further detail can be found in the privacy impact assessment for the Integrated Data Infrastructure available from Statistics New Zealand.[1]

The results are based in part on tax data supplied by Inland Revenue to Statistics New Zealand under the Tax Administration Act 1994. These tax data must be used only for statistical purposes, and no individual information may be published or disclosed in any other form or provided to Inland Revenue for administrative or regulatory purposes.

Any person who has had access to the unit-record data has certified that they have been shown, have read and have understood section 81 of the Tax Administration Act 1994, which relates to secrecy. Any discussion of data limitations or weaknesses is in the context of using the IDI for statistical purposes and is not related to the data's ability to support Inland Revenue's core operational requirements.

Executive Summary

Background and approach

The Youth Service (YS) is a programme for youth (aged 16-18 years) who are considered to be at risk of poor outcomes such as long-term benefit dependency. It aims to help these young people to achieve qualifications reduce their risk of moving on to a working-age benefit after their 18th birthday (Ministry of Social Development 2015). Under the YS, the Ministry of Social Development (MSD) contracts community-based social service providers to work with these young people and support them to enter and remain in education, training or work-based learning.

YS has three main strands: Youth Payment, Young Parent Payment, and Not in Employment, Education or Training (or NEET). Young parents aged 16 to 18 who are receiving a Young Parent Payment benefit, or youth aged 16 or 17 who are receiving a Youth Payment benefit are required to participate in the Youth Service. Those who are not receiving any income support from the government, but are considered to be at risk of moving onto benefit can participate in the YS: NEET strand on a voluntary basis.

This paper evaluates the Youth Payment and Young Parent Payment (YP/YPP) strands of the programme. The focus of our research is to determine what impact participation in YS has on benefit, employment, and educational outcomes, relative to the previous approach to youth beneficiary case management. Longitudinal administrative data from Statistics New Zealand's Integrated Data Infrastructure (IDI) are used to measure a range of different outcomes, including educational participation, educational achievement, time spent on benefits, time in employment, and time serving a corrections sentence.

YP and YPP incorporate a number of features that did not exist under earlier benefit regimes. Participants are expected to be in, or available for, full-time education, and they are expected to participate in budgeting and parenting courses. They receive incentive payments for meeting these obligations, and sanctions if they fail to meet them. The abatement regime was designed to encourage participation in education, creating financial disincentives to move off benefit and into low-paid fulltime work whilst eligible for a youth benefit.

MSD published results from an initial impact evaluation for the YP and YPP components of the Youth Service in June 2014, and these were updated in 2015. No significant impacts were found at that stage on either benefit receipt or qualification attainment for YPP participants, but YP participants were estimated to be more likely to achieve NCEA Level 1 or 2 qualification over a two-year follow-up period. They were also found to be more likely to be on benefit over this period, consistent with an increased focus on participation in education rather than employment.

The participant group was matched to an historical comparison group of youth beneficiaries who had not participated in YS on the basis of a number of characteristics derived from MSD administrative data and other linked administrative data held by MSD. Matching was done using propensity scores, on a 1-to-1 basis.

The MSD impact evaluation is subject to a number of limitations that the current evaluation seeks to address. In particular, the reliance on a comparison across time can be seen as problematic. New Zealand's labour market was still moving out of economic recession in the period leading up to the implementation of Youth Service in July 2012, and youth unemployment was particularly high over the 2009 to 2012 period. Similarly, the introduction of government targets has seen a focus on raising qualification achievement for young New Zealanders.

The programme was implemented in such a way as to make robust impact evaluation challenging. The service was universally accessible to eligible youth as soon as it was implemented, and as such, there was not an obvious contemporaneous comparison group against which to contrast outcomes.

The approach we take to estimating the impact of the Youth Service on youth beneficiaries uses a mix of historical comparisons as well as matched contemporaneous comparisons. As with the MSD study the main basis for our estimation is a comparison between outcomes for YP and YPP participants and outcomes for historical comparison groups of youth beneficiaries. This captures both the effect of the Youth Service, and the effect of different temporal conditions experienced by the two populations. It is therefore not likely to be a good estimator for the effect of the Youth Service alone.

In order to control for changes in temporal conditions that might affect the outcomes of participant and historical comparison groups, we look to the broader youth population, contrasting changes in outcomes for populations of young people not on youth benefits over the same time period. If the temporal effect on outcomes for YP and YPP participants can be assumed to be the same as the temporal effect on outcomes for the broader youth population over the same period, this could provide a good estimator of the impact of the Youth Service.

This assumption is unlikely to hold, given the very different characteristics of the youth beneficiary population from the broader youth population. As such we select propensity score matched comparison groups for both our participant and historical comparison populations. We do this matching using a wide range of background characteristics and activity measures from the linked administrative data. From these comparison groups we derive temporal adjustments for each outcome measure, and these are applied to our historical comparison-based impact estimates in order to adjust for changes over time.

We dropped individuals from the study population where they did not meet a range of criteria affecting measurement of their background or outcomes (for example linked data from key sources being available, having spent sufficient time in schooling in New Zealand, and being in the country for most of the study period). In combination this resulted in 12 to 16 percent of the participant and historical comparison groups being excluded from the study. Where the remaining participants could not be matched to one or more people from the broader youth population, they were also excluded. Around 96 percent of YP participants and their pre-YS equivalents, and 90% of young parents were successfully matched.

Results

Youth Service is strongly focused on educational participation, and as such we would hope to see positive impacts on enrolment and achievement of qualifications. In the first 12 to 18 months there's strong evidence of a positive impact on enrolment in formal education, with estimated impacts of 11 percentage points after 6 months for both YP and YPP. YP impacts declined from that point however to 3 percentage points after 18 months, and were no longer significant after 24 months. YPP participation effects were more long-lasting with significant impacts at 24 and 30 months (6 to 7 ppts).

Impacts on qualification achievement were somewhat smaller than the participation effects, possibly due to participants enrolling but not continuing with a course, not completing the course, or simply not achieving enough credits to gain a formal qualification. YP participants were estimated to be 3 to 4 percentage points more likely to gain a level 1 or 2 qualification in the two calendar years following the calendar year of first YS participation. YPP participants were estimated to be 4 to 6 percentage points more likely to gain a level 1 to 3 qualification, but only in the second calendar year following first YS participation. The sustained positive enrolment effects could be indicative of further qualification impacts in future years.

Participation and achievement effects were through both schools and tertiary providers for YPP, while YP effects were primarily through tertiary providers. Effects on tertiary attainment for YP participants were larger than the overall qualification effects, indicating that many of these qualifications were at the same level as school qualifications they already held.

In the first 12 months there is some evidence that YS results in beneficiaries on YP being more likely to stay on benefit. This is consistent with a focus on education rather than employment. We estimate that YP beneficiaries are 8 percentage points more likely to still be on benefit 6 months after first coming onto benefit than if they hadn't participated in Youth Service, and 5 percentage points more likely after 12 months. Over the longer term both YP and YPP participants are estimated to be more likely to move off benefit and into employment as a result of Youth Service, although this is only significant for YPP (an estimated 6 percentage point effect after 30 months).

Conclusion

Results from this study should be treated with some caution. The service was introduced to all young beneficiaries at the same time, and so an obvious comparison was not available to assess the impact of the service. The service was also implemented over a period of considerable change. While we believe these results represent the best that can be done to get at the true impact of the programme, the results may not be robust if our assumptions do not hold. In particular we assume that the historical population of youth beneficiaries provides a valid counterfactual for YP and YPP participants, setting aside temporal effects. We further assume that changes in outcomes for matched populations of non-beneficiary youth provide valid estimates of the temporal effect for YP and YPP participants, allowing us to account for these effects.

The programme is intended to help youth beneficiaries, and has a focus on participation in education, rather than a shift off benefit and into work. These study participation effects were particularly sustained for recipients of YPP, who were more likely to be enrolled in education even 30 months after first participating in YPP. We also estimate that YS raised qualification achievement, albeit to a lesser degree. The impact of YS on YPP achievement was more delayed than for YP, but larger. We also found some evidence that YS resulted in more youth beneficiaries moving off benefit and into work, although these effects were smaller and not significant for YP.

While we estimate some positive effects for youth beneficiaries arising out of the introduction of the Youth Service, the introduction of the service consisted of a number of simultaneous changes. The changes were introduced simultaneously and we did not have reliable information about which participants were affected by which aspects of the service. As such it was not possible to assess which changes contributed most to the positive outcomes observed.

1 Introduction

The Youth Service (YS) is an MSD-administered programme for youth (aged 16-18 years) who are considered to be at risk of poor outcomes such as long-term benefit dependency. It aims to help these young people achieve a qualification at level 2 or higher and develop life skills in order to reduce their risk of being on a working-age benefit after their 18th birthday (MSD, 2015). Under the YS, MSD contracts community-based social service providers to work with these young people and support them to enter and remain in education, training or work-based learning.

YS has three main strands: Youth Payment, Young Parent Payment, and Not in Employment, Education or Training (or NEET). Young parents aged 16 to 18 who are receiving a Young Parent Payment benefit, or youth aged 16 or 17 who are receiving a Youth Payment benefit are required to participate in the Youth Service. Those who are not receiving any income support from the government, but are considered to be at risk of moving onto a benefit at a later date, can participate in the YS: NEET strand on a voluntary basis.

This paper evaluates the Youth Payment and Young Parent Payment (YP/YPP) strands of the programme. The YS: NEET strand is evaluated in an associated Treasury working paper.

The focus of our research is to answer the following question: what impact does participation have on outcomes for young people on participating in the Youth Service relative to the previous policy settings, and approach to youth beneficiary case management? Longitudinal administrative data from Statistics New Zealand's Integrated Data Infrastructure (IDI) are used to measure a range of different outcomes, including educational participation, educational achievement, time spent on benefits, time in employment, and time serving a corrections sentence.

Because the Youth Service is a relatively new programme, the evaluation focuses on people who came onto a YP or YPP benefit during the first 18 months or so of operation (August 2012 to December 2013), and their outcomes during the 24-30 months after they first participate. Future evaluations will be able to focus on a larger participant group, followed over a longer outcome window.

The paper is structured as follows. Section 2 summarises relevant literature, describes the YS: YP/YPP programme in more detail, and describes previous evaluations of the programme. Section 3 outlines the data and methods used in this evaluation. Section 4 provides descriptive information on the characteristics of participants, the education or training they undertook, and their educational outcomes. Section 5 presents our main estimates of the programme’s impact on educational participation, educational achievement, benefit rates, and other relevant outcomes. We draw conclusions in Section 6.

2 Literature review and description of the programme

2.1  Literature review

As with the YS: NEET programme, the core element of YS: YP/YPP is the provision of customised support and guidance, although in the case of YP and YPP there are also obligations associated with receiving the benefit payments and sanctions that may be imposed if these obligations are not met.

As well as engaging with the service, participants are required to be enrolled in study and to undertake budgeting training, and much of their money is managed for them by the service provider. There are also additional payments that provide an incentive to undertake certain activities, such as attending budgeting courses, ongoing participation in education or training, and meeting ‘parenting requirements’ (for young parents). These requirements are outlined in greater detail in section 2.2.

Many other countries have youth programmes that provide customised support and guidance to ‘at risk' youth. The accompanying report on the NEET stream of the Youth Service (Dixon et al. 2016) outlines the international evidence around youth mentoring programmes such as the Youth Service. The results of these studies are somewhat mixed, possibly due to the diversity of the programmes evaluated. As with the YS: NEET strand of the programme, the YS: YP/YPP strands differ from many such programmes in that they have fairly well-defined educational objectives, using outcome-based funding to incentivise the programme providers to help clients achieve these objectives.

Financial incentives have also been used internationally to encourage ongoing participation of young people in education and training. The Emergency Maintenance Allowance (EMA) was piloted in the United Kingdom from 1999, and rolled out nationally in 2004[1]. At least three evaluations of the impact of the programme on participation in education were undertaken (see Middleton et al. 2005, Chowdry et al. 2008, and Dearden et al. 2009). Although impact estimates varied, all three studies found significant positive effects of the programme on participation, with estimated effects generally in the order of approximately three to six percentage points.

The compulsory money management aspect of the Youth Service was examined by Fletcher et al (2013). The authors concluded that the approach was almost unique in the world, and identified a number of concerns. Some concerns with the money management aspect of the programme were also identified by the MSD evaluation of the programme, although the evaluation notes that these have been addressed subsequent to the evaluation.

Notes

  • [1]The EMA has since been discontinued in England, but is still available in Scotland, Wales, and Northern Ireland.

2.2 Youth Service: Youth payment and young parent payment

The Youth benefit component of the Youth Service consists of two main benefits available to young people; Youth Payment (YP), which is available to 16 and 17 year olds, and Young Parent Payment (YPP), which is available to 16 to 18 year old parents[2]. The new benefits were introduced in July 2012 and replaced existing benefits available to young people. The benefits primarily replaced; the Emergency Maintenance Allowance (EMA)[3], a sub-category of the Emergency Benefit available to 16 and 17 year old sole parents (amongst others); the Domestic Purposes Benefit (Sole Parent), available to sole parents aged 18 to 64 years; and the Independent Youth Benefit, available to young people aged 16 to 17 years who were unable to live at home due to exceptional circumstances, or who were in a relationship.

The new Youth Service (YS) benefits, and equivalent pre-YS benefits are outlined in Table 1 below.

Table 1 - YS and pre-YS equivalent benefit types
Youth service benefit Equivalent pre-YS benefits Eligible population
Youth Payment Independent Youth Benefit Aged 16-17 years, without dependent child/ren and either unable to live with parents or has a partner/spouse
Young Parent Payment Emergency Maintenance Allowance Aged 16-17 years, with dependent child/ren and no partner/spouse
  Domestic Purposes Benefit - Sole Parent Aged 18 years, with dependent child/ren and no partner/spouse
  Sickness Benefit Aged 16-18 years, with a partner/spouse and dependent child/ren, and unable to work due to ill health/injury/disability
  Unemployment Benefit Aged 16-18 years, with a partner/spouse and dependent child/ren

Detailed information on Youth Payment and Young Parent Payment is provided on the ‘Map' section of the Work and Income website and the Youth Service website[4]. These include information about eligibility criteria, entitlements, responsibilities, sanctions, and current and historical payment rates.

Expected outcomes for young people in the YP and YPP streams of Youth Service are “NCEA Level 2 or higher qualifications” and “improved social outcomes for the young people, and their children (for young parents)” (MSD 2014 II). MSD contracts community-based service providers to work with young people to support them into education, training or work-based learning. YP and YPP incorporate a number of other new features that did not exist under earlier benefit regimes as detailed below:

  • Educational obligations
  • Financial incentives and sanctions
  • Money management
  • Childcare payments
  • New abatement regime.

Educational obligations

Participants are expected to be in, or available for, full-time education, training or work-based learning leading towards at least an NCEA level 2 or equivalent qualification. Participants are also expected to participate in budgeting and parenting courses.

Young parents are exempt from these obligations when their child is less than six months old, or is between six and 12 months old where there is no suitable place available for them in a teen parent unit.

Financial incentives and sanctions

Participants receive incentive payments for meeting certain obligations, and sanctions if they do not meet these. Young people receive an additional $10 per week if they complete six months of education or training, or if they complete a budgeting course. Young parents also receive a further $10 per week if they complete a parenting course[5].

Sanctions may also be applied if a participant doesn't meet the obligations expected of them. The first or second time obligations are not met, the participant's in-hand allowance and any incentive payments may be stopped. If the participant doesn't meet them within four weeks, the whole benefit payment may be stopped. The third time a participant fails to meet their obligations, the whole benefit payment may be stopped immediately.

Money management

Part of the participant's payment is re-directed to cover costs such as accommodation and utilities, another $50 is provided as an in-hand payment which they can use at their discretion, and the remainder is loaded onto a payment card which can only be used in specific shops to pay for food and groceries.

Childcare payments

A payment is available to cover childcare costs for young parents with children aged under 5 years.

New abatement regime

Whilst payment rates were unchanged with the introduction of the Youth Service benefits, a new abatement regime was put in place. When they earn above certain thresholds, beneficiaries lose a specified percentage of their benefit income, affecting incentives to engage in different levels of employment. The Youth Service was designed to allow young people to work up to around 15 hours at the adult minimum wage without their benefit being affected[6], but any employment above this level of earnings would be dis-incentivised.

Whilst participants are able to earn more than previously without their benefit rate being reduced, the benefit thereafter reduces quickly with each additional dollar of income (the participant loses a dollar of benefit for every dollar of additional income earned at this point). Above a specified threshold the benefit cuts out completely, strongly dis-incentivising any earnings above this threshold (this was around $257 per week before tax for a single participant in 2012).

Setting aside other forms of support (such as accommodation supplement and working for families), and using the payment and abatement rates when Youth Service was introduced in July 2012, single participants in YP and YPP had a slightly stronger incentive than earlier youth beneficiaries to earn between approximately $80 per week and $207 per week (before tax). They are then strongly dis-incentivised from earning more than $257 per week, only receiving more money once they earn almost $400 per week before tax for YP and almost $550 per week for YPP.

This created a far stronger financial disincentive than was previously the case for youth beneficiaries to move off benefit and into fulltime work whilst eligible for a youth benefit, unless that work were paid at significantly above the minimum wage. The cabinet paper outlining the Youth Service policy changes notes that “the focus of the Youth Package is on supporting young people to learn not work”[7].

Notes

2.3 Previous evaluations of the Youth Service

MSD published an evaluation of the Youth Service, covering the first 18 months of its operation, in June 2014 (MSD, 2014). The evaluation involved:

  1. an examination of the outcomes achieved by Youth Service participants, including their participation in education and training, budgeting and parenting activities, and achievement of NCEA qualifications
  2. an evaluation of the impact of the Youth Service on the time young beneficiaries spend on benefit
  3. a process evaluation of the Youth Service implementation, six months after it started assessing whether it operated as intended, what worked well, and what could have been improved.

The process evaluation was undertaken in 2013, based on interviews conducted in February 2013 and was updated in 2014. A number of things were considered to be working well, including the wrap-around service model, the low youth-coach ratio, the community-focussed approach, the experience and commitment of coaches, and the way financial assistance met young people's needs.

Areas for improvement were also identified, and the report noted that a number of changes had been implemented to address these. These included issues related to the training and support given to Youth Service providers, providing better information to providers about the young people they are working with, and improving the range of retailers and transport providers who accept the payment card.

The impact evaluation was based on matching cohorts of Youth Payment and Young Parent Payment recipients to earlier cohorts of youth beneficiaries who were not exposed to the Youth Service intervention. The participant group was matched to the comparison group on the basis of a number of characteristics derived from MSD administrative data, and other linked administrative data held by MSD. Matching was done using propensity scores, on a 1-to-1 basis.

Key findings of the impact evaluation include that:

  • YP and YPP participants were more likely to gain NCEA credits in the first 12 months of enrolment in Youth Service.
  • YP and YPP participants were more likely to achieve an NCEA Level 2 qualification within their first 12 months in Youth Service. After this time 14 per cent of YP participants and 7 per cent of YPP participants met the requirements of NCEA Level 2 - nine percentage points and two percentage points higher than similar young beneficiaries before Youth Service was established respectively.
  • Early evidence suggests that Youth Payment participants spend less time on benefit under Youth Service, and fewer Youth Payment participants were transitioning to a working-age benefit.
  • It was considered to be too early to assess the impact of Youth Service for Young Parent Payment. The report noted that significant impacts were not anticipated until four to five years after teen parents start the Youth Service. This was based on timeframes from earlier evaluations of the Training Incentive Allowance for sole parents, and the likelihood that childcare responsibilities may restrict young parents from moving into employment until their children reach school age.

The initial MSD impact evaluation appears to have been repeated in 2015. MSD (2015) refers to an updated report from March 2015 and summarises some of the key findings from this report:

  • In terms of qualification achievement, the paper reported an estimated 9 percentage point (ppt) impact on NCEA Level 1 achievement and an 11 ppt increase in NCEA Level 2 achievement over a two-year follow-up period. NCEA Level 3 achievement impacts were not statistically significant, nor were any impacts for YPP participants.
  • There was a statistically significant increase in benefit receipt for YP participants in the first two years of participation, but no significant change for YPP participants. An increase in benefit receipt was considered to be consistent with most programmes with an education and training focus in the initial post-participation period.

The MSD impact evaluations conducted in 2014 and 2015 were the most robust approaches that could be undertaken at the time, given the data available. Nevertheless, they are subject to a number of limitations that the current evaluation seeks to address:

  1. The reliance on a comparison of cohorts observed at different points in time.
  2. A somewhat limited set of measures to match on.
  3. A relatively limited set of outcomes measures.

The reliance on a comparison across time is particularly problematic, and could be seen as limiting the ability to draw conclusive assessments from the evaluation. Such an approach did not account for the different economic conditions the different cohorts were exposed to, nor does it account for the impact of other policy changes which may have happened between the two periods.

A particular concern in this regard is that New Zealand's labour market was still moving out of economic recession in the period leading up to the implementation of Youth Service in July 2012, and youth unemployment was particularly high over the 2009 to 2012 period, before falling somewhat subsequently. Similarly, the introduction of Better Public Services (BPS) targets has seen a focus on raising qualification achievement for young New Zealanders. BPS results area 5[8] has set targets for increased level 2 qualification attainment by 18 year olds, with year-on-year improvements being achieved since 2011. In 2011, 74 percent of 18 year olds achieved NCEA level 2, while this had increased to 84 percent by 2014.

These are difficult issues to get around, largely because the programme was implemented in such a way as to make robust impact evaluation challenging. The service was universally accessible to eligible youth as soon as it was implemented. Piloting, a staggered roll-out, eligibility cut-offs, or differentiating between aspects of the service offered in different regions, could have been considered as ways to make the effect of the service more easily identifiable.

The availability of linked data in the IDI has enabled us to develop an approach that addresses many of these issues. The IDI provides a broad set of background pre-participation characteristics and post-participation outcome measures for both participants and non-participants, as well as similar information for historical groups of beneficiaries and non-beneficiaries before the implementation of the service.

It is of particular importance that this evaluation is able to distinguish between the effects of the introduction of the Youth Service on outcomes from changes due to other things occurring over the same time period. Our approach seeks to do this, but is subject to a number of assumptions and caveats, and results should be treated with some caution.

3 Methods

3.1  Data sources

The study uses data from Statistics New Zealand's Integrated Data Infrastructure (IDI), which combines administrative data from the tax system with data collected by other government agencies and covers all persons in New Zealand[9].

Within the IDI, the main data sources used in this study provide longitudinal information on individuals':

  • employment and earnings over the period from 1999 to the end of 2015
  • benefit payments over the period from 1993 to the end of 2015[10]
  • interactions with Child, Youth and Family over the period from 1993 to the end of 2015
  • school enrolments over the period from 2006 to the end of 2015
  • tertiary education enrolments over the period from 2003 to the end of 2015
  • NQF-registered qualifications completed from 2006 to the end of 2015
  • custodial and community sentences served with the Department of Corrections
  • places of residence within New Zealand; and
  • movements in and out of New Zealand from 1997 to the end of 2015.

The information on individuals' places of residence within New Zealand is derived from several administrative sources, including the National Health Index, Primary Health Organisation enrolments, and address data held by Inland Revenue, MSD and the Ministry of Education.[11]

Notes

3.2 Impact estimation approach

3.2.1  Estimation approach

The approach we take to estimating the impact of the Youth Service on youth beneficiaries uses a mix of historical comparisons as well as matched contemporaneous comparisons, and uses a combination of propensity score matching and difference-in-differences, as outlined below. Propensity score matching approaches are summarised in Caliendo and Kopening (2005), while examples of studies combining these two approaches are outlined in Gertler at al. (2016).

Figure 1 illustrates the way our participant population is defined for the purposes of this study. The Youth Service was implemented in July 2012, with new and existing youth beneficiaries (Youth Payment and Young Parent Payment) being provided the service, and being subject to its requirements.

Figure 1 - Study and comparison populations

 

Figure 1 - Study and comparison populations.

For the purpose of this study we exclude young people who were already on benefit, and therefore were only exposed to the Youth Service intervention for part of their time on benefit.

Our study population is represented by area A of Figure 1, young people entering benefit from August 2012 through to the end of 2013. We are then able to observe outcomes for this group through to the end of 2015, giving us a 24 month outcome window for all participants, and a 30 month outcome window for a majority of participants.

We also observe an historical comparison population B of youth beneficiaries who were not exposed to the Youth Service, and were instead exposed to the previous youth beneficiary regime. We restrict this population to young people who were first on a youth benefit (primarily on IYB, DPB, or EMA as discussed above) in 2009 and 2010. We are able to observe at least 18 months of outcomes for these young people before the Youth Service was implemented. As such, few of the historical comparison group participated in the Youth Service during the outcome window, and those who did were only exposed to the service for a relatively short period of time, at least a year and a half after first coming onto a youth benefit.

As discussed above, the problem with making a simple comparison between the outcomes of groups A and B is the different ‘temporal conditions' these groups were exposed to; both varying economic conditions, and different policies and Government interventions, such as those implemented to support the Better Public Services targets.

In order to control for changes in these temporal conditions that might affect the outcomes of our participant (A) and historical comparison (B) groups, we look to the broader youth population who are not on benefit - groups C and D respectively in our two time periods of interest.

  • Group A is exposed to both the Youth Service and Post-YS temporal conditions.
  • Group B is exposed to neither the Youth Service, nor to Post-YS temporal conditions, but is otherwise similar to Group A.
  • Group C is not exposed to the Youth Service[12], but is exposed to Post-YS temporal conditions. Group C is otherwise very different from groups A and B.
  • Group D is not exposed to the Youth Service, nor to Post-YS temporal conditions. While Group D is likely to be broadly similar to Group C, it is very different from groups A and B.

Generally speaking, if we are interested in an outcome y, for group X at a particular point in time we can define our estimate of that outcome as ȳX. We can then define the estimated impact of the Youth Service on group A,

 

A as being equal to ȳA - ȳB, i.e. the difference between group A's outcomes at that time and group B's outcomes in the earlier period. This captures both the effect of the Youth Service and the effect of different temporal conditions experienced by populations A and B, and is therefore not likely to be a good estimator for the effect of the Youth Service alone. If the characteristics of the group B are different from those of group A, it is likely that this estimator will also pick up these compositional differences.

 

The effect of this temporal change on the wider no benefit youth population can be defined in the same way for the non-benefit population, as δC = ȳC - ȳD. This could be seen as capturing the effect of different temporal conditions over the same period on the non-benefit youth population, but excludes the effect of YS, since groups C and D were not exposed to the service.

Going a step further we could define the impact of the Youth Service as being:

 

 

 

This is similar to the standard difference-in-difference estimator, but instead of contrasting outcomes pre and post-intervention for two groups, we are contrasting outcomes post-intervention for two groups with outcomes for two earlier pre-intervention cohorts.

If the temporal effect on outcomes for group A between time t-1 and time t can be assumed to be the same as the temporal effect on outcomes for group C over the same period (essentially the ‘parallel trends' assumption in a difference-in-difference context), then this could provide a good estimator of the impact of the Youth Service, controlling for changing temporal conditions.

In practice however, this assumption is difficult to justify. Economic conditions and changes in government policy and services are likely to impact on youth beneficiaries in a different way than they impact on youth non-beneficiaries. As such, it's important that we make some attempt to make groups C and D as similar to groups A and B respectively as is possible.

To do this, we select propensity matched comparison groups C* and D* (as indicated in Figure 1) as sub-sets of groups C and D, but which match as closely as possible the background characteristics of the members of groups A and B. Thus we define the impact of the Youth Service on group A as being:

 

 

 

With this specification, a key assumption is that the temporal effect from time t-1 to time t is likely to affect outcomes for group C* in the same way as it affects outcomes for group A.

A second important assumption is that there is no selection bias between youth beneficiaries before and after the introduction of Youth Service (i.e. groups A and B). In other words, differences in observed outcomes can be assumed to be due to the change in the policy settings and case management of youth beneficiaries following the introduction of YS, and are not due to a change in the type of people who come onto youth benefits.

We choose not to match our groups A and B, and C* and D* respectively across time for two key reasons. The first is that many of the measures we have available to us for matching are not measured consistently in different time periods, or are expected to have different implications in the different time periods. For example, people who had left school at age 16 in the earlier period are likely to be less generally disadvantaged on average than those who left school at age 16 in the post-YS period. This is because rates of retention in education increased considerably over this period due to broader policy and operational changes. There is a risk that matching on observed characteristics could result in the groups becoming more, rather than less, different in terms of their underlying characteristics.

The second, less important, reason we do not match across time as well as contemporaneously is because we already have relatively small participant and historical comparison populations, and to do so would further limit the number of participants included in the study, affecting our ability to generalise the results.

This approach differs from a standard difference-in-difference framework in a number of ways. Firstly the comparison group is formed by matching. Second, the members of the groups change over time, although this may be the case where difference-in-differences are used for geographically identified groups for example. Relatedly, the difference in difference is not defined with respect to contemporaneous outcomes but to outcomes occurring at certain intervals of time since participants entered the programme. Finally the comparison group is matched on benchmark historical observations of the outcomes of interest.

A number of studies have combined propensity score methods (weighting or matching) with difference-in-differences approaches (for example Werner et al. 2009, Song et al. 2012, and Stuart et al. 2014). These studies differed in the way propensity scores were used, in the groups being matched, and in the way the difference-in-differences estimator was constructed. In the first study matching was done across time separately for treatment and control groups, in the second study matching was done only within a specific time period between treated and untreated groups, and in the final study four-way matching was undertaken both contemporaneously and over time.

Notes

  • [12]Note that some of this group are actually exposed to aspects of the Youth Service through the YS: NEET stream of the service. These young people are excluded from our analysis.

3.2.2 Study and comparison population selection criteria

As discussed above, key to the method adopted is the assumption that a comparison group formed from an historical cohort of youth coming onto benefit in an earlier period (2008 and 2009) present a robust counterfactual for Youth Service participants in 2012 and 2013. It is therefore important that the historical comparison group be as similar as possible to current participants.

The primary selection criteria for our two participant groups are based on data derived from Ministry of Social Development administrative data on benefit receipt (the SWIFTT database), and are outlined in Table 2 below[13].

Table 2 - Participant population primary selection criteria
Population group Criteria
YP participant

Aged 16-17 years AND

Recorded in SWIFTT as receiving Youth Payment AND

No previous record of receipt of a youth benefit[14] AND

No move to YPP within 6 months of YP being granted AND

Benefit granted between 1 July 2012 and 31 December 2013

YPP participant

Aged 16-18 years AND

Recorded in SWIFTT as receiving Young Parent Payment AND

No previous record of receipt of a youth benefit or Moved from a YP benefit within 6 months of that benefit being granted AND

Benefit granted between 1 July 2012 and 31 December 2013

The primary selection criteria for our historical comparison groups are also based on SWIFTT data, with selection made more complicated by the different eligibility criteria applied to youth and other benefits prior to Youth Service implementation. The criteria were designed to make the groups as alike as possible, as outlined in Table 3.

Table 3 - Historical comparison population primary selection criteria
Population group Criteria
YP historical comparison

Aged 16-17 years AND

Recorded in SWIFTT as receiving the Independent Youth Benefit or of receiving another benefit and having a partner aged 16-17 AND

No previous record of receipt of a youth benefit AND

No move to a YPP historical comparison benefit within 6 months of the benefit being granted AND

Benefit granted between 1 January 2008 and 31 December 2009

YPP historical comparison

Aged 16-18 years AND

Recorded in SWIFTT as receiving the Domestic Purposes Benefit: Sole Parent or the Emergency Maintenance Allowance or another benefit where they have both a partner also aged 16-18 and at least one dependent child AND

No previous record of a youth benefit or Moved from a YP historical comparison benefit within 6 months of that benefit being granted AND

Benefit granted between 1 January 2008 and 31 December 2009

The study and historical comparison populations were further refined according to the availability of matched data, and the characteristics revealed by this matched data. The criteria are similar to those outlined in some detail in the accompanying report on the NEET stream of the Youth Service (Dixon et al. 2016). In summary they are:

  • Linkages to the IDI spine, IRD identity numbers, and Ministry of Education identity numbers are required to provide data needed for this study, including income, benefit and education information. People who died before their 20th birthday were also excluded.
  • People aged less than 15 or greater than 18 were excluded (17 for Youth Payment), as this would indicate they are ineligible for Youth Service benefits. A number of potential issues could underlie this. The birthdate could be incorrect, they may be labelled as being on the wrong benefit, or data from different sources may have been matched incorrectly.
  • People who did not enrol in a New Zealand school in the years they would have been in Year 9 to Year 11 were excluded, as were those who attended a school that offered qualifications that are not part of the National Qualifications Framework, or whose school enrolment record had a missing end date after five years. Those who were overseas for more than 6 months in total during the period comprising the 2 years leading up to the YS benefit start date and the 2 years following it were also excluded.

The selection criteria mean that recent migrants to New Zealand will not be included. The data held in IDI will be too limited to provide comparable measures of the characteristics and experiences of these young people.

The impact of applying these criteria to the participant and historical populations is illustrated in Table 4 Below. Around 5 percent of each group were excluded through not matching the spine, IRD, or Ministry of Education data. This restriction particularly affected the number of young parents in the earlier period with around 10 percent being excluded at this point. 18 year old young parents in this earlier period were particularly likely to not match to the Education data. Nevertheless the final sample for our analysis represented around 85 percent of each of the groups of beneficiaries we initially selected, and this was reasonably consistent across benefits and time periods.

Table 4 - Participant and historical comparison population
selection criteria and their impact on the sample size
Selection criteria   Participant A Historical comparison B
    YP YPP YP YPP
Initial participant and historical comparison populations N 2,496 1,698 5,226 3,939
Identity linked to the IDI spine, linked to IRD and Ministry of Education and survived until 20th birthday N 2,376 1,626 4,914 3,558
  % 95.2 95.8 94.0 90.3
Some NZ school enrolment in years 9-11, did not study in a non-NQF school, last school enrolment with a valid end date, and in NZ for most of the 4 year study period N 2,145 1,494 4,371 3,321
  % 85.9 88.0 83.6 84.3

Notes: The numbers in this table are randomly rounded.

As discussed earlier, one of the key assumptions in this report is that historical comparison groups of beneficiaries provide a good counterfactual for participants. We can assess this using measured characteristics, however they may also differ on unmeasured characteristics. Differences in selection into the programme could translate to differences in unmeasured characteristics which may relate to our outcomes of interest, biasing our results.

The numbers in Table 4 give us some information about the rate at which young people enter the programme. Differences could suggest that selection may have changed over time (i.e. that different people now come onto youth benefits, possibly due to different circumstances). If we divide the number of participants by the number of months covered by each period (17 months in the case of participants, and 24 months in the case of the historical comparison), we see that the number of participants coming onto youth benefits has dropped considerably over time. Before Youth Service, 382 new youth beneficiaries were coming onto benefit each month, whereas after the introduction of YS there were 247 new recipients of YP or YPP each month[15]. These differences could signal unobservable differences that could bias our results.

Notes

  • [13]An alternative source of Youth Service participation data is administrative data from the system that supports the Youth Service (known as ART). There are inconsistencies between this data and the SWIFTT benefit data however. For many participants recorded in ART there is no evidence of them receiving a Youth Service related benefit. For the most part these young people are only recorded as participating in the Youth Service for a very short period of time, and it may be that they are only provisionally registered as participating, pending confirmation of being granted a benefit. In any case, we believe SWIFTT is likely to be a more reliable indicator of participation.
  • [14]As defined for YP/YPP participants and historical comparison groups.
  • [15]There were 147 new YP participants each month and 100 new YPP participants each month following the introduction of YS. This compares with 218 and 164 new beneficiaries respectively in the historical comparison period.

3.2.3 Outcome measures

The primary objectives of YS: YP & YPP are to raise participation in formal education or training and raise qualification attainment. YS providers receive quarterly payments for participants who are enrolled in formal education or training (and childcare in the case of YPP), one-off payments for their achievement of a qualification, and a one-off payment for participants who leave benefit and stay off benefit for at least three months. In accordance with these measures, engagement in education, qualification achievement and benefit receipt are the main outcomes against which we assess the effectiveness of the programme.

We also consider a range of other outcomes measures that may be influenced by YS participation, including whether the youth was employed at various points in time during the follow-up period; whether they were not in employment, education or training (NEET); whether they received a student allowance; and whether they were serving any custodial or community sentences. People are able to earn a certain amount while still on benefit, and so we also look at those who are earning and not on benefit as an indicator of a successful transition into work independently of benefit.

The outcomes of programme participants are assessed over the 24-30 months following the first YS enrolment date. We have a complete set of data covering the first 24 months after YS benefit start for most participants, whilst we are able to observe up to 30 months of outcomes for most participants who started on benefit before July 2013. Most data was available up until December 2015, however benefit data was only available up until November. We construct our outcome measures using the individuals for whom data are available at that point in time.

Due to the manner in which tax data are collected in New Zealand, the employment and earnings measures in IDI are available on a calendar month basis only. There are no measures of weekly earnings, hourly earnings, or hours of work in IDI. In this study, a person is classified as ‘employed' in a given calendar month if they received any wage and salary earnings in that month (that were reported through the tax system). For consistency, we use calendar months to construct all of our measures of post-YS enrolment activity and incomes, even though some of them (such as whether or not a benefit was received) are recorded in IDI on a daily basis. For example, the ‘employment rate' measures the proportion of people in a particular group who received wage and salary earnings at any time during a particular calendar month. Similarly, a person is classified as ‘in receipt of a benefit' if they received any income from one of the main income support benefits during the calendar month, and a ‘benefit receipt rate' is the proportion of people in a particular group who received benefit income in that month.

3.2.4 Method of estimating the program impact using historical and matched comparisons

As discussed above the impact of participation in YS: YP/YPP is estimated by comparing outcomes for youth beneficiaries who participate in the Youth Service with an earlier cohort of youth beneficiaries who did not participate in the Youth Service. The difference in outcomes between these groups are used as a counterfactual for what might have occurred to participants if they had not participated in the Youth Service (ie, the programme effect), and if temporal conditions had not changed (ie, the temporal effect).

Comparison groups of youth who were as similar as possible to the individuals in the participant and historical comparison populations but did not come onto youth benefit were also constructed and their outcomes compared. Changes in outcomes between these groups provide an estimate of the effect of temporal conditions on a similar cohort of young people over a similar period, and we use this as an estimate of the counterfactual temporal effect on youth beneficiaries if Youth Service were not implemented.

We use a combination of exact case matching and propensity score matching to select the most appropriate matched comparison group members for each individual in the study population. Firstly, a pool of potential comparison group members was created by selecting all youth who met the criteria listed in Table 1, with the exception that they did not come onto a youth benefit before or during our study period (including the outcome period). The characteristics, prior activities and childhood histories of these youth can be measured in each calendar month from July 2012 through to December 2013.

For each person in the potential comparison group, we generated 17 monthly records corresponding to each month in this time period, and randomly assigned a reference date within the month. The characteristics, prior activities and childhood histories of each individual were then measured as at the reference date. The purpose of creating this large pool of potential control group records was to ensure we could match each person in the study population with a group of other youth whose characteristics were as well matched as possible in the reference month - the month when the study population member first enrolled in YS.

Next logistic regressions were estimated for YP and YPP separately, for both participants and their equivalent historical comparisons. To run the logistic regressions, we took a random sample of 10,000 of the potential comparison group people, so that the treatment group individuals would make up a larger proportion of the total.

The explanatory variables included in the models include:

  • The reference month (the month of first benefit start in the case of the benefit populations, and a randomly assigned reference date in the case of the potential matched comparisons)
  • An indicator for having previously participated in the Youth Transition Service (a similar programme that preceded YS and was superseded by it) or the NEET stream of the Youth Service
  • Demographic characteristics, such as birth year, age at the reference month in quarter years, sex, and ethnic group (using indicators for each non-European ethnic group)
  • Region of residence and New Zealand Deprivation Index score for the neighbourhood that was lived in at the reference date
  • A measure of the proportion of time the child had been supported by a parent's benefit during their life time
  • Several variables capturing the youth's lifetime CYF care and protection history and youth justice history
  • An indicator of whether the youth's mother or female caregiver had no qualifications (available if the mother or caregiver has received a benefit at some time in the past)
  • Indicators of whether the child's parents or caregivers had ever served a custodial or community sentence (where the parent or caregiver has received a benefit at some time in the past)
  • The proportion of the individual's childhood that was spent out of NZ, up to the reference date
  • The number of schools attended since 2006, characteristics of the school that the youth currently or most recently attended, including its decile and ownership type, and an indicator of whether the child had ever been granted special education funding
  • The level of the highest qualification held at the reference date, and the number of NCEA credits that had been completed at levels 1, 2 and 3
  • Several measures of ‘disengagement' from school, including the total numbers of stand-downs and suspensions from all schools attended and whether there were any truancy records
  • Whether any tertiary qualifications had been completed before the reference date, and the type of programme undertaken (vocational or general skills)
  • Measures of the number of months the individual was enrolled at school, enrolled in tertiary education, employed, on benefit, or NEET during the past 24 months.

A full list of the explanatory variables included in the regressions is given in Table A.1 in the Appendix. All explanatory variables were constructed and entered in the model as categorical, and as such treated as a set of binary predictors.

Most variables included were the same for each model, however matching on the number of months at school, in tertiary education, employed, on benefit, or NEET during the past 24 months differed somewhat between YP and YPP (and their respective historical comparisons), with young parents being matched on a smaller set of activity variables. Specifically:

  • Young parents were not matched on benefit receipt in the 12 months leading up to starting on a young parent benefit. We were seeking to match young parents to similar young people who did not have children, and therefore did not come onto a young parent benefit. Whilst more than half of young parents came onto benefit for the first time around the time their child was born, many were already on benefit as a result of the pregnancy - most commonly coming onto sickness benefit or Youth Payment in the months leading up to receiving YPP (or its historical equivalent).
  • Young parents were not matched on school or tertiary enrolment in the 6 months leading up to starting on a young parent benefit. Many young parents moved out of school or tertiary study in the months leading up to receiving a young parent benefit. In many cases this will be related to the impending birth of their child, and as such we did not want to include it as a matching characteristic.

Predicted probabilities of participating in YS: YP/YPP were then calculated for all members of the participant (or historical comparison) group and potential matched comparison group (not just the sub-sample of potential comparisons that was used to calculate the probabilities in the logistic regressions), using the propensity scores from each regression model. These predicted probabilities are referred to as propensity scores.

The third stage of the method was to match each individual in the study population with a group of comparison individuals. Matches were only made between records with the same reference month, in order to control for the effects of variations in the business cycle and labour market.

Apart from reference month, exact matching (or ‘blocking') was also undertaken on a number of other criteria that differed according to the model, as indicated in Table 5. Variables selected for exact matching were those that were considered most likely to be crucial to post-participation outcomes (in particular age, baseline measures of pre-benefit educational achievement and participation, and benefit activity). In the case of YPP we also blocked on sex, given the dominance of women among young parent beneficiaries.

Table 5 - Characteristics used in exact matching
Description Categories YP YPP
    Participants
A
Historical cohort
B
Participants
A
Historical cohort
B
Reference month (start of benefit) Calendar month Y Y Y Y
Age in quarter years 15-18 Y Y Y Y
Highest qualification obtained before benefit start None, NCEA Lvl 1, NCEA Lvl 2, NCEA Lvl 3, Tertiary Y Y Y Y
Female Yes, No     Y Y
Number of months on benefit 0-6 months before reference month 0, 1-3, 4-6 Y Y    
Number of months on benefit 12-24 months before reference month 0, 1-12     Y  
  0, 1-3, 4-6, 7-9, 10-12       Y
Number of months enrolled in school 6-12 months before reference month 0, 1-3, 4-6 Y Y    
Number of months enrolled in school 12-24 months before reference month 0, 1-12     Y Y
Number of months enrolled in tertiary 6-12 months before reference month 0, 1-3, 4-6 Y Y    
Number of months enrolled in tertiary 12-24 months before reference month 0, 1-12     Y Y

As with the matching variables more generally, exact matching was done slightly differently for YP and YPP (and their historical equivalents). YPP included a requirement to match on gender, as YPP is predominantly received by young women. Exact matching on participation in benefit, school and tertiary also focussed on an earlier period for young parents, reflecting the pre-benefit impacts of pregnancy on participation rates.

Within those exact matching constraints, each study population individual was matched to up to 20 comparison group individuals with the closest values of their propensity score, within a radius of plus or minus 0.03 propensity score points. Fewer than 20 matches were selected if less than 20 people met these criteria. Matching with replacement was used, meaning that each comparison group individual could be matched to more than one study population member.

Each matched comparison individual was assigned a weight based on the number of matches made (eg, 0.10 if the person was one of 10 matches for a particular study sample member). These weights are applied in the subsequent analysis of impacts, to ensure that the distribution of comparison group characteristics mirrors that of the study population.

We dropped individuals in the study population who could not be matched with one or more comparisons. Table 6 shows the match rate for each participant historical comparison group. In general matches were easier to obtain for YP participants, and their pre-YS equivalents, with 96% being successfully matched, compared to 90% of young parents in both time periods.

Table 6 - Population size and match rate after propensity score matching
Selection criteria Population after
exclusions applied
Successfully
matched records
%
matched
Participant A      
YP 2,145 2,061 96.1
YPP 1,494 1,350 90.4
Historical comparison B      
YP 4,371 4,209 96.3
YPP 3,321 3,003 90.4

Notes: All sample size numbers are randomly rounded.

The matching method was designed to balance the average characteristics of the study population and matched comparison groups. A comparison of the participant, historical comparison, and their respective matched comparison populations is provided in sections 4.2 and 4.3.

As a further test of balance we re-estimated the propensity score model using only matched individuals from the potential comparison group to predict selection into the programme, as suggested in Caliendo and Kopening (2005). Comparing the pseudo-R squared before and after matching we would appear most of the variation explained by the covariates to have disappeared, resulting in much smaller values that are very close to zero (all pseudo-R squared statistics are 0.005 or less after matching). Chi-squared tests of joint significance also find insufficient evidence to reject the null hypothesis that the covariates are equal to zero after matching.

There were few remaining statistically significant differences in variable means between the study and comparison groups for any of the model variables.Although we did not match exactly on every variable, the method ensured that the matched samples were very similar in terms of their demographic and regional profiles and prior employment and income support histories.Overall we conclude that the models are well-balanced on observed co-variates after matching.

Once the matched comparison groups were constructed, the impacts of YP and YPP participation were estimated as the difference between the mean outcome of the study population and the mean outcome of the historical comparison population, adjusted by the difference between the mean outcomes of their respective matched comparison populations, as outlined in section 3.2.1. Standard errors and confidence intervals for each impact estimate were estimated using bootstrapping methods.

4 Participant characteristics and activities while enrolled

4.1  Introduction

We summarise the characteristics and schooling history of YS: YP & YPP participants and comparison populations in this section of the paper. We show that the historical and matched comparison groups are well matched with participants in terms of these measured characteristics and experiences.

We also provide information on the nature of the education or training that was undertaken by YP and YPP participants while they were engaged in the programme or afterwards, and the qualifications they completed.

4.2 Characteristics at the time of benefit start - Youth Payment

Demographic and background characteristics

Table 7 provides summary statistics on the characteristics and childhood experiences of YP participants and the historical and matched comparison populations, using the sample of participants that is used in our impact evaluation. The third and sixth columns of data present the same information for the population of potential matches, including multiple monthly records for young people who were aged 16 or 17 in multiple months during the period of interest, and met the other criteria for inclusion (matches to the spine and other datasets, spending sufficient time in NZ etc). This gives an idea of the characteristics of the broader population of young people). Care needs to be taken when comparing pre-YS results with post-YS results. The underlying data covers different time periods for different data collections, and in some cases available data is much more limited for the earlier pre-YS birth cohorts.

The Youth Payment age distribution was similar to the pre-YS historical comparison. Almost a half of both populations were aged between 16 3/4 and 17 1/2 when starting on benefit, while around double as many were aged between 16 and 16 1/4 than were aged between 17 3/4 and 18. The gender split was roughly even in both periods, with slightly more female than male participants. The population of potential matches over the same period had slightly more males than females. Matched comparison groups had a similar gender split to participants, and the same age profiles (this latter was forced through exact matching as discussed earlier).

Almost half of participants identified as Māori in both periods, whilst only one in five potential matches identified as Māori. Participants were less likely to identify as being of Pacific ethnicities, Asian, European, or Other ethnicity. The low proportion of Asians may be partly due to our study design, since recent migrants to New Zealand are not included in the study.

The residence data indicates that participants tended to come from poorer socio-economic backgrounds. More than two fifths of participants were living in a neighbourhood classified in the two most deprived deciles. Less than a quarter of potential matches lived in similar neighbourhoods.

There was quite a marked shift in the region of residence between the pre-YS and post-YS periods, with youth beneficiaries far less likely to live in Auckland in the more recent period (32% compared to 18%), despite the underlying distribution of young people being essentially unchanged. On the other hand, youth beneficiaries were more likely to live in Northland, Hawkes Bay, Taranaki and Manawatu-Wanganui after the introduction of Youth Service.

Over two-fifths of Youth Payment recipients and half of the historical comparison group of youth beneficiaries had a mother or female caregiver who was recorded as not having any formal qualifications, although this data was only available where the mother received a benefit in the past and their qualifications were recorded by Work and Income. Only a fifth of the wider population of youth had a mother with no recorded qualifications.

Almost a third of YP recipients had a parent or caregiver who had served a custodial sentence, and over half had a parent or caregiver who had served a community sentence. This was almost unchanged since the pre-YS comparison period, with rates around three to four times higher than for the wider population of potentially matched young people. These measures of parental corrections history are also subject to data limitations and should be treated as indicative only.

Around a third of participants had spent three-quarters or more of their childhood (from birth to the birthday before receiving benefit) being supported by a parent or caregiver on a benefit (compared to around 10 percent of the wider population of young people). Three quarters had been the subject of at least one CYF care and protection notification during their childhood, two fifths had been the subject of at least one substantiated CYF care and protection finding, and almost a fifth had had a CYF care placement. These measures were slightly smaller for the pre-YS cohort of young people, reflecting broader trends in CYF contact rates for young people.

Almost a quarter of Youth Payment participants had at least one referral to CYF youth justice, and almost half had used mental health or addiction services in the secondary health care sector at some stage before receiving a youth benefit. In the earlier period, only a fifth of youth beneficiaries were recorded as having accessed mental health services, but this largely reflects the lack of historical mental health service data collection rather than a shift in prevalence or a change in mental health needs of the benefit population.

Educational participation and other activities

Summary measures of school characteristics and educational achievement are set out in Table A.2, focusing on the current or most recently attended school. Youth Payment participants were more likely than the general population to have attended low decile schools, were more likely to have attended correspondence school, and to have attended a state school. Almost a half of post-YS youth beneficiaries had attended four or more schools since 2006, compared to only around 10 percent of the broader youth population. A quarter of pre-YS youth beneficiaries had attended three or more schools, compared to 6 percent of the broader youth population at the time.

Other results in Table A.2 show that about 40 percent had a truancy record, almost a half had at least one stand-down from school and over a fifth had had at least one suspension. These figures were almost identical before and after YS was implemented, and were all 3 to 4 times the rates of the broader youth population. Youth payment recipients were slightly more likely to still be at school (37 percent) than the historical comparison group of youth beneficiaries (31 percent).

Due to data limitations, we know the calendar years in which qualifications were obtained but not the exact timing within the year. Seventy percent of Youth Payment recipients had no formal qualifications in the year before starting benefit, compared to 76 percent of pre-YS youth beneficiaries. A little over a third of the broader population of potential youth matches had no qualifications.

Table A.3 summarises the other activities that were undertaken before receiving a youth benefit. In the more recent period, following the implementation of Youth Service, youth beneficiaries were more likely to have spent time enrolled at school or in tertiary education in the previous 18 months than before YS was implemented, but were less likely to have been employed.

Comparison with the YP historical comparison

Comparisons between the YP participant population and the historical comparison population are important as differences in characteristics could illustrate whether the historical population provides a good counterfactual for the YP population. The use of matched comparisons to adjust for temporal effects will also be impacted on by these differences, as the historical matched comparison is selected to have similar characteristics to the historical comparison group.

Table 7, Table A.2 and Table A.3 illustrate that the YP and historical youth benefit populations are broadly very similar in terms of their background and characteristics. The biggest differences between the populations relate to measurement differences in different years, especially in the number of schools attended since 2006 and the use of mental health services.

Other differences are likely to reflect broader temporal effects that have affected young people over the period, particularly those most at risk of poor outcomes. These differences are evident in the broader population of potential matches, and can be seen as reflecting more general population trends. Examples are the increase in educational participation and attainment, and drop in employment rates for the more recent cohort. We might reasonably expect the temporal adjustment we make using matched comparisons in the two periods to account for the influence of these differences on our impact estimates, overcoming the deficiencies of using a single historical comparison.

Finally, there are a few differences that are less easy to explain through either measurement effects or broader population shifts. The most obvious of these is the drop in the Auckland youth beneficiary population after YS was implemented. This was despite the Auckland youth population actually growing over the period, from 28 percent of potential historical matches to 29 percent of YP potential matches. Conversely, an increase in youth beneficiaries resident in the Bay of Plenty was not reflective of an increase in the youth population living in that region.

The other area of change that was not reflected in broader temporal change was the gender split of the youth beneficiary population. It's possible that this reflects a shift in the benefit classification of young male parents who previously might have received the Independent Youth Benefit, and are now classified as being a young parent (and receiving YPP) instead of a youth beneficiary (and receiving YP). We consider this shift (of 4 percentage points) as not being large enough to be likely to have had a significant influence on the results.

As we would expect, the matching approach largely results in matched comparisons that are broadly similar in their observable characteristics to the participant and historical comparison populations. This will not necessarily be the case however, as, outside of the exact matched characteristics, we are matching on the ‘propensity' to be a participant rather than on the actual characteristics. Some differences are worth noting:

  • Both pre-YS and post-YS the matched non-beneficiaries were less likely to be European and more likely to be of Maori or Pacific ethnicity than the youth beneficiaries they were matched to.
  • In both periods matched comparisons had slightly less contact with CYF care and protection or youth justice, to have attended more than 3 schools or to have used mental health services, but were more likely to have been supported by benefit for an extended period.
  • Matched comparison populations were slightly more likely to live in a deprived (decile 9 or 10) neighbourhood in both periods. They were also more likely to have been employed, and to have earned more than $1,000 per month while employed.

Matched comparison youth were more likely to have left school in the previous 3 months than youth beneficiaries. This is perhaps not surprising given that the ‘exceptional circumstances' that have led to taking up a youth benefit could have also impacted on school participation. As discussed earlier, exact matching was done on school participation in the 6-12 month period before benefit, but not in the more recent pre-benefit period.

Table 7 - Youth Payment participant and comparison group characteristics
  Post-YS Pre-YS
  Participants
A
Matches
C*
Potential matches
C
Historical cohort
B
Historical matches
D*
Potential matches
D
N 2,058 2,058 1.6 mil 4,218 4,215 2.3 mil
Year started in YS            
    2009       55.6 55.7 50.2
    2010       44.3 44.3 49.8
    2012 21.1 21.1 29.5      
    2013 78.9 78.9 70.5      
Age at start of YS participation            
    16 to 16.25 13.4 13.3 12.3 15.1 15.1 12.8
    16.25 to 16.5 10.3 10.2 12.3 11.2 11.2 12.7
    16.5 to 16.75 9.8 9.9 12.4 11.2 11.2 12.6
    16.75 to 17 14.9 14.9 12.6 12.8 12.9 12.5
    17 to 17.25 17.3 17.3 12.7 16.5 16.5 12.5
    17.25 to 17.5 14.3 14.1 12.6 13.2 13.2 12.4
    17.5 to 17.75 12.7 12.8 12.5 12.1 12.1 12.3
    17.75 to 18 7.4 7.4 12.6 7.8 7.8 12.3
Gender            
    Male 44.0 44.8 51.0 47.4 48.5 51.4
    Female 56.1 55.2 49.0 52.5 51.5 48.6
Ethnicity (including multiple responses)            
    European 53.2 47.8 65.1 53.0 47.3 67.4
    Māori 49.3 51.6 22.3 47.9 51.2 21.3
    Pacific 6.3 7.9 9.9 7.3 8.3 9.5
    Asian 1.6 1.5 7.7 1.9 1.6 7.3
    Other 1.3 1.5 2.3 1.3 1.3 2.0
    Region of residence            
    Northland 7.4 8.0 3.9 5.0 7.1 3.7
    Auckland 18.2 20.1 29.1 31.9 30.5 28.3
    Waikato 12.2 11.7 10.1 10.1 11.2 10.0
    Bay of Plenty 7.7 8.2 7.2 3.4 4.6 7.0
    Gisborne 1.9 2.0 1.3 2.3 2.7 1.3
    Hawkes Bay 6.0 7.4 4.3 4.8 4.3 4.1
    Taranaki 4.2 4.8 2.8 2.9 3.0 3.0
    Manawatu-Wanganui 10.9 9.5 5.9 6.6 7.3 5.8
    Wellington 7.9 7.3 10.7 8.0 8.0 10.9
    West Coast/ Tasman/ Nelson/ Marlborough 6.0 5.1 4.0 5.7 5.1 4.0
    Canterbury 9.8 9.9 13.0 10.0 9.7 12.8
    Otago 4.7 3.5 4.4 5.4 3.9 4.6
    Southland 2.6 2.8 2.4 3.6 2.7 2.4
    Not available s s 0.8 s s 2.3

Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.

Table 7 cont. - Youth Payment participant and comparison group characteristics
  Post-YS Pre-YS
  Participants
A
Matches
C*
Potential matches
C
Historical cohort
B
Historical matches
D*
Potential matches
D
Deprivation index            
    1-2 5.8 5.1 19.9 6.5 5.3 18.3
    3-4 10.2 9.2 18.8 10.2 9.6 18.3
    5-6 16.6 14.6 18.5 15.6 16.1 18.4
    7-8 25.7 26.5 18.8 24.7 23.4 19.1
    9-10 41.5 44.2 23.2 42.7 45.5 23.5
    Not available s s 0.8 0.3 s 2.3
Time overseas in childhood            
     92.4 92.7 84.9 93.5 93.5 86.5
    10- 4.8 4.4 8.7 3.1 3.3 5.1
    50%+ 2.9 2.9 6.5 3.3 3.2 8.4
Mother unqualified 42.1 44.3 20.2 53.1 54.3 22.0
Parent custodial sentence 31.0 31.9 7.1 27.5 28.0 5.8
Parent community sentence 56.7 57.3 17.2 53.2 54.2 15.8
Childhood supported by benefit            
    None 8.2 6.1 45.6 8.2 6.2 45.9
    1-9% 6.9 6.9 13.1 6.8 6.1 13.1
    10-24% 9.3 10.6 9.4 8.3 7.9 9.0
    25-49% 21.3 19.4 11.7 19.6 19.4 11.4
    50-74% 24.2 23.6 9.4 24.5 24.6 9.5
    75+% 30.3 33.7 10.8 32.6 35.7 11.1
CYF care & protection notifications            
    None 25.1 25.9 78.5 31.6 32.7 82.7
    1-2 22.2 24.1 12.0 24.5 27.7 10.7
    3-9 37.5 36.4 7.8 35.6 33.7 5.8
    10+ 15.3 13.6 1.7 8.3 6.0 0.7
CYF care & protection finding 43.4 42.1 8.6 38.8 37.1 7.2
CYF care & protection placement 18.8 15.9 1.9 17.9 13.2 1.6
CYF youth justice referrals            
    None 78.7 80.2 97.0 76.2 79.1 96.5
    1-2 10.5 10.6 1.9 13.4 12.2 2.4
    3-9 9.2 7.6 1.0 9.2 7.5 1.0
    10+ 1.6 1.7 0.2 1.1 1.1 0.1
Mental health, drug or alcohol services 46.1 42.6 11.7 19.0 16.6 3.9

Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.

4.3 Characteristics at the time of benefit start - Young Parent Payment

Benefit characteristics and history

Almost all young parents on benefit aged 16 to 18 are granted a Young Parent Payment after the introduction of Youth Service, whereas before YS a number of different benefits could be granted. The circumstances of people granted YPP also varied somewhat from those of the earlier cohort of young parent beneficiaries. The characteristics associated with the benefits granted to young parents are shown in Table 8.

Table 8 - Young Parent Payment participant and historical
comparison group benefit history and characteristics at grant
  Participants
A
(%)
Historical cohort
B
(%)
Number of children on benefit at benefit grant    
    None 21.8 2.1
    1 77.3 95.7
    2+ 1.1 2.2
Age of youngest child at benefit grant    
    No child recorded on benefit 21.6 2.1
    Less than 1 month 31.2 36.2
    1 to 3 months 17.8 25.4
    3 months to 1 year 21.8 25.2
    1 to 2 years 5.8 8.2
    2+ years 1.3 2.9
Partner on benefit 19.2 12.3
Benefit type    
    Young Parent Payment 100.2 s
    Emergency Maintenance s 40.2
    Sole Parent s 47.5
    Job Seeker s 9.1
    Sickness s 2.0
    Other s 1.3
Previous benefit receipt    
    Never on benefit 56.3 43.8
    Sickness Benefit or equiv 24.9 43.8
    Unemployment Benefit or equiv 6.5 7.6
    Other 12.5 4.8

Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.

Demographic and background characteristics

As we would expect, for the historical cohort, almost all young parents (96 percent) have one child on benefit at the time they're granted benefit, while around 2 percent have two children, and another 2 percent have no children, perhaps because they are granted benefit shortly before the birth of the child, or possibly because the child is recorded against the benefit at a later date.

More unusually, a fifth of YPP participants had no children recorded against their benefit. This is because, unlike other benefits, where a YP or YPP recipient has a partner, the benefit is granted to both partners separately, and the child is only recorded against the benefit of one of them. In addition there is no way of linking the partners together, so for many young parents, we have no information about their children.

Consistent with this just under a fifth of YPP recipients had a partner, up from 12 percent in the historical cohort. Most of the historical cohort of beneficiaries received a sole parent benefit (40 percent), or the Emergency Maintenance Allowance (47 percent). This latter benefit is an emergency benefit that was available to young sole parents who did not meet the eligibility criteria for the usual sole parent benefit, normally due to being aged under 18.

As well as being granted a wider range of benefits, young parents pre-YS were more likely to have a history of benefit receipt before coming onto a young parent benefit (56 percent compared to 44 percent post-YS). The most common benefit type for both groups was a sickness-related benefit, often granted due to pregnancy-related health issues.

The characteristics and childhood experiences of YPP participants and the historical and matched comparison populations are summarised in Table 9, along with the potential match population. As with Youth Payment this includes multiple monthly records for young people during the period of interest, although in the case of YPP 18 year olds are also included. This gives an idea of the characteristics of the broader population of young people at that point in time. Note however, that the results will be influenced by the fact that the majority of young parent beneficiaries are female, whilst the potential matches are evenly split between young men and women.

The Young Parent Payment age distribution was similar to the pre-YS historical comparison distribution, with almost half of participant being aged 18. Nevertheless there were slightly more 16 year olds on YPP than in the earlier period (18 percent compared to 24 percent) and slightly fewer 18 year olds (45 percent compared to 52 percent). The gender split was female-dominated in both periods, although there were more male participants after the introduction of Youth Service. This could be the result of more male young parents being identified as parents by Youth Service providers than was previously the case with the Work and Income administered youth benefits.

Over half of participants identified as Māori in both periods. Participants were also more likely than the broader youth population to identify as being of Pacific ethnicity (unlike for YP). The ethnic profile did not change markedly following the introduction of Youth Service, although there was a drop from 40 percent to 35 percent of participants identifying as European. Again, the low proportion of Asians may be partly due to the exclusion of recent migrants to New Zealand from the study.

The residence data indicates that participants tended to come from poorer socio-economic backgrounds. Over half of participants were living in a neighbourhood classified in the two most deprived deciles, compared to fewer than a quarter of the broader population of young people. Young parent beneficiaries were more likely to live in Northland, Bay of Plenty, Gisborne, Hawkes Bay and Manawatu-Wanganui than the broader youth population.

Around a half of young parent beneficiaries had a mother or female caregiver who was recorded as not having any formal qualifications in both time periods, although this data is of questionable quality, as discussed earlier. Only a fifth of the wider population of youth had a mother with no recorded qualifications. Around a quarter of YPP recipients had a parent or caregiver who had served a custodial sentence, and over half had a parent or caregiver who had served a community sentence. This was slightly higher than in the pre-YS comparison period. These measures of parental corrections history are also subject to data limitations and should be treated as indicative only.

As with YP around a third of young parent beneficiaries had spent three-quarters or more of their childhood being supported by a parent or caregiver on a benefit. Almost two thirds had been the subject of at least one CYF care and protection notification during their childhood, a third had been the subject of at least one substantiated CYF care and protection finding, 8 percent had had a CYF care placement, and 14 percent Almost a quarter of Youth Payment participants had had at least one referral to CYF youth justice. These measures were slightly smaller for the pre-YS cohort of young people, reflecting broader trends in CYF contact rates for young people, and were lower than the rates observed for YP participants.

Almost a third of YPP participants had used mental health or addiction services before receiving benefit. In the earlier period, only 7 percent of young parent beneficiaries were recorded as having accessed mental health services, but this is expected to reflect measurement changes rather than a change in prevalence or access to mental health services.

Educational participation and other activities

The school characteristics and educational achievement of young parent beneficiaries and matched comparisons are set out in Table A.4. YPP participants were more likely than the general population to have attended low decile schools and a third of post-YS young parent beneficiaries had attended four or more schools since 2006, compared to only around 10 percent of the broader youth population. Fourteen percent of pre-YS young parent beneficiaries had attended three or more schools since 2006, compared to 5 percent of the broader youth population at the time. As with YP, YPP participants were more likely to have attended correspondence school as their most recent school, and to have attended a state school.

Table A.4 also shows that about 40 percent of YPP recipients had a truancy record, around two fifths had at least one stand-down from school and 17 percent had had at least one suspension. These figures were slightly higher than before YS was introduced. As with YP, YPP recipients were more likely to still be at school (24 percent) than the historical comparison group of youth beneficiaries (17 percent).

Two thirds of Young Parent Payment recipients had no formal qualifications in the year before starting benefit. Unlike with YP, where qualification rates were higher in the post-YS period, YPP participants had very similar qualification attainment patterns to the earlier cohort of young parent beneficiaries.

Table A.5 summarises the other activities that were undertaken prior to a young parent benefit being granted. After the introduction of Youth Service, young parent beneficiaries were more likely to have spent time enrolled in school and tertiary education in the previous 18 months than before YS was implemented, but were less likely to have been employed. They were also less likely to have a history of benefit receipt.

Comparison with the YP historical comparison

As with YP, comparisons between the YPP participant population and the historical comparison population of young parent beneficiaries provide an important assessment of the appropriateness of using the historical group as a counterfactual (once temporal effects are set aside).

Table 9, Table A.4 and Table A.5 illustrate that the YPP and historical young parent populations are broadly similar in terms of their background and characteristics. As noted earlier, there was a shift in young male beneficiaries from a youth beneficiary classification to a young parent benefit classification after the introduction of Youth Service. As with YP, the biggest differences between the populations relate to measurement differences in different years, for example in the number of schools attended and the use of mental health services.

Other differences are likely to reflect broader temporal effects that have affected young people over the period, particularly those most at risk of poor outcomes. These differences are evident in the broader population of potential matches, and can be seen as reflecting more general population trends. Examples are the increase in educational participation (although not in attainment), the drop in employment rates, and the increased contact with CYF in the more recent period. As with YP, we might reasonably expect our approach using matched comparisons to adjust for temporal effects to account for the influence of these differences on our impact estimates.

There are a few differences that are less easy to explain through either measurement effects or broader population shifts. These include the increase in the number of male young parents as discussed previously, and matching a decline in the share of other male youth beneficiaries, and an increase in the share of 16 year olds in the young parent population and consequent decrease in the number of 18 year olds. There was also a shift away from young parent beneficiaries being located in Auckland (and consequent increase in the Bay of Plenty in particular), however this was much more muted than for YP.

As with YP the matching approach largely results in matched comparisons that are broadly similar to the participant and historical comparison populations:

  • Unlike for YP, in both the periods before and after YS was introduced the matched comparison groups were more likely to identify as European and less likely to identify as Maori ethnicity than the young parent beneficiaries they were matched to.
  • In the more recent period young parent beneficiaries were less likely than their matched comparison group to have a parent with a corrections sentence, more likely to have a mother with no qualification, were slightly less likely to live in a deprived neighbourhood, and were more likely to have had a CYF care and protection finding during their childhood. In the earlier period the converse was true in each case.
  • Matched comparison groups in both periods were slightly less likely to have been referred to youth justice, or to have a record of truancy.

Given the tendency for young parent beneficiaries to move onto a benefit in the months leading up to the birth of the child, and consequent receipt of a young parent benefit, young parents were not matched on benefit receipt in the 12 months leading up to the young parent benefit. As such it is not surprising that the matched comparison are much less likely to receive a benefit over this period.

Table 9 - Young Parent Payment participant and comparison group characteristics
  Post-YS Pre-YS
  Participants
A
Matches
C*
Potential matches
C
Historical cohort
B
Historical matches
D*
Potential matches
D
N 1,347 1,347 2.4 mil 3,024 3,024 3.4 mil
Year started in YS            
    2009       53.2 53.3 50.2
    2010       46.7 46.8 49.8
    2012 30.7 30.7 29.3      
    2013 69.5 69.5 70.7      
Age at start of YS participation            
    16 to 16.5 11.8 11.6 16.4 9.2 9.2 17.1
    16.5 to 17 12.7 12.9 16.6 8.6 8.6 16.9
    17 to 17.5 13.1 13.4 16.8 13.8 13.6 16.7
    17.5 to 18 16.9 16.7 16.8 16.0 16.1 16.6
    18 to 18.5 25.4 25.2 16.8 29.5 29.4 16.5
    18.5 to 19 20.0 20.0 16.5 23.0 23.1 16.1
Gender            
    Male 14.0 14.0 51.2 8.1 8.1 51.3
    Female 86.2 86.0 48.8 91.9 91.8 48.7
Ethnicity (including multiple responses)            
    European 35.2 38.1 65.4 40.4 41.3 67.7
    Māori 58.8 57.9 22.1 55.8 55.1 21.0
    Pacific 13.1 13.4 9.7 14.2 14.2 9.4
    Asian 1.3 1.1 7.7 1.0 1.0 7.4
    Other 1.1 1.3 2.3 1.0 0.7 2.1
Region of residence            
    Northland 5.8 6.5 3.8 6.1 6.8 3.6
    Auckland 25.2 26.3 29.3 29.0 28.6 28.5
    Waikato 11.4 11.6 10.1 13.0 13.1 9.9
    Bay of Plenty 11.8 11.1 7.0 9.8 9.8 6.8
    Gisborne 2.9 2.4 1.3 2.6 2.4 1.2
    Hawkes Bay 6.7 8.0 4.1 5.8 4.8 4.0
    Taranaki 3.1 2.7 2.8 3.0 2.5 3.0
    Manawatu-Wanganui 8.5 6.7 5.8 7.0 7.1 5.8
    Wellington 8.5 8.9 11.0 9.1 9.1 11.1
    West Coast/ Tasman/ Nelson/ Marlborough 2.4 2.4 3.9 2.6 2.6 3.9
    Canterbury 8.0 8.2 13.0 7.5 8.8 13.0
    Otago 3.1 3.1 4.9 1.7 1.6 4.8
    Southland 2.7 2.2 2.3 2.7 2.4 2.4
    Not available s s 0.7 s s 2.0

Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.

Table 9 cont. - Young Parent Payment participant and comparison group characteristics
  Post-YS Pre-YS
  Participants
A
Matches
C*
Potential matches
C
Historical cohort
B
Historical matches
D*
Potential matches
D
Deprivation index            
    1-2 3.6 3.1 19.3 4.5 4.3 18.2
    3-4 8.0 8.0 18.4 8.0 9.2 18.2
    5-6 13.1 13.1 18.3 13.7 13.9 18.4
    7-8 24.1 23.6 19.4 23.3 22.6 19.6
    9-10 51.0 51.9 23.7 50.2 49.6 23.5
    Not available s s 0.8 0.3 s 2.1
Time overseas in childhood            
     94.4 94.0 84.8 94.5 94.8 86.8
    10- 3.8 3.3 8.5 2.9 2.8 4.8
    50%+ 2.0 2.4 6.6 2.6 2.3 8.4
Mother unqualified 47.2 46.5 20.1 52.8 53.1 21.5
Parent custodial sentence 25.8 26.9 7.1 19.3 18.8 5.7
Parent community sentence 50.8 52.8 17.4 44.9 44.6 15.7
Childhood supported by benefit            
    None 8.5 7.6 45.4 11.9 10.6 46.5
    1-9% 8.0 8.2 13.2 9.7 9.5 13.1
    10-24% 7.6 8.5 9.5 8.9 9.1 9.0
    25-49% 20.3 19.2 11.8 19.2 19.7 11.3
    50-74% 22.0 22.5 9.6 20.9 20.9 9.4
    75+% 33.4 34.1 10.5 29.3 30.0 10.7
CYF care & protection notifications            
    None 36.7 37.2 78.8 49.2 50.1 83.0
    1-2 25.2 25.2 11.9 24.9 25.3 10.6
    3-9 29.0 28.3 7.7 22.5 21.2 5.7
    10+ 9.1 9.6 1.6 3.3 3.4 0.7
CYF care & protection finding 31.8 33.6 8.6 25.2 24.5 7.2
CYF care & protection placement 7.8 9.1 2.0 5.7 6.1 1.6
CYF youth justice referrals            
    None 86.0 87.3 96.5 88.8 89.9 96.1
    1-2 8.7 7.8 2.2 7.6 7.0 2.6
    3-9 4.9 4.5 1.2 3.3 2.9 1.1
    10+ s 0.9 0.2 s s 0.1
Mental health, drug or alcohol services 30.1 30.5 12.2 7.3 6.8 3.9

Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.

4.4 Studying rates and educational achievement patterns

Information about participants' education enrolment in the two years after coming onto a youth benefit is outlined in Sixty-eight percent of YP participants and 44 percent of YPP participants were enrolled in a tertiary programme at some point over this time, a rate that was also considerably higher than for youth beneficiaries in the pre-YS period. This include enrolments that had been started before the student was recruited into YS. Although some were already enrolled in a tertiary programme, most also started a new tertiary programme during the following two years.

Table 10. Fifty-two percent of YPP participants and 45 percent of YP participants were enrolled at school at some stage in this two year period. Both of these rates are much higher than for the historical comparison (particularly for YPP).

The majority of young parents both before and after YS who were still at school were enrolled either at Correspondence School or a school with a Teen Parent Unit (TPU)[16]. While attendance at Correspondence School increased by 8 percentage points (from 10 percent to 18 percent) and attendance at a school with a TPU increased by 7 percentage points (from 12 percent to 19 percent), this was in line with the more general increase in school attendance for this group.

A recent evaluation of TPUs (Vaithianathan et al. 2016) found that access to a TPU increased the post-birth school enrolment of teen mothers by 11 to 15 percentage points, the chances of attaining an NCEA Level 1 qualification by 22 percentage points, and found weaker evidence of an impact on Level 2 attainment. Youth Service and TPUs are likely to be complementary and mutually reinforcing services. To the degree to which YS increases the enrolment in TPUs our impact estimates are likely to include the effects of this increased TPU attendance on participant outcomes.

Sixty-eight percent of YP participants and 44 percent of YPP participants were enrolled in a tertiary programme at some point over this time, a rate that was also considerably higher than for youth beneficiaries in the pre-YS period. This include enrolments that had been started before the student was recruited into YS. Although some were already enrolled in a tertiary programme, most also started a new tertiary programme during the following two years.

Table 10 - YP/YPP education or training undertaken
  YP YPP
  Participants
A
 (%)
Historical cohort B
(%)
Participants
A
(%)
Historical cohort B
(%)
N 2,061 4,206 1,350 3,003
Some school attendance 51.8 42.1 44.7 26.4
    Correspondence School 15.3 9.7 17.6 9.6
    School with a Teen Parent Unit 5.1 3.6 18.9 12.2
    Other 31.4 28.7 8.2 4.6
Some tertiary enrolment 67.7 51.5 44.4 32.9
    Started before benefit start 24.6 10.3 9.1 6.8
    Started after benefit start 59.2 46.9 40.4 29.6
Some school or tertiary attendance 90.5 71.9 70.7 49.9
Some industry training 3.2 1.9 1.6 0.9
Some education or training 91.1 72.3 71.1 50.2

Notes: All counts are randomly rounded. Teen Parent Unit indicates a young person was enrolled in a school that had a Teen Parent Unit attached to the school.

Rates of participation in industry training were very low. About 3 percent of YP participants and few than 2 percent of YPP participants were enrolled in industry training at some stage during the two years after coming onto benefit.

Overall around 90 percent of YP participants, and 70 percent of YPP participants were enrolled in some form of education or training in the two years after coming onto benefit. As noted earlier, participation in study is compulsory for most youth beneficiaries, however exemptions may be applied where a young parent has a child under the age of a year.

Note

  • [16]We're unable to establish whether a young person attended the Teen Parent Unit, but assume most YPP recipients at a TPU school attend the unit, while most YP recipients attend the main school.

Tertiary study characteristics

Table 11presents data on the tertiary programmes that were started by participants after enrolling in YP/YPP – 59 percent and 40 percent of all YP and YPP participants respectively. If more than one programme was started on the same day, we sum the study load and select the highest qualification that was enrolled for.

Table 11 - Characteristics of the first tertiary programme after starting benefit
  YP YPP
  Participants A
(%)
Historical cohort B
(%)
Participants
A
(%)
Historical cohort B
(%)
N - With a tertiary enrolment only 1,221 1,971 546 888
Level        
    Level 1 24.1 9.0 22.0 6.4
    Level 2 36.9 30.3 28.0 20.9
    Level 3 25.3 35.9 30.2 36.8
    Level 4+ 14.0 25.3 19.8 37.2
Study load, first year        
    Less than 0.5 EFTS 42.8 30.9 41.8 42.6
    0.5- 44.0 36.4 45.1 35.8
    1.0+ EFTS 13.0 33.0 13.2 23.3
Type of provider        
    University 3.7 5.9 2.2 5.7
    Polytechnic 31.7 44.1 33.0 45.3
    Wananga 7.4 8.1 15.4 13.5
    Private training establishment 57.5 42.0 49.5 36.8
Funding source        
    Student component 31.4 68.0 50.5 91.9
    Youth Guarantee 60.4 9.1 39.6 1.7
    Other funding 8.1 23.1 9.9 7.8
Field        
    Natural and Physical Sciences 1.2 s s s
    Information Technology 5.4 7.0 3.8 9.5
    Engineering and Related Technologies 7.4 6.7 3.3 2.0
    Architecture and Building 4.7 3.8 2.2 1.4
    Agriculture, Environmental and Related Studies 7.1 9.4 4.9 2.7
    Health 2.7 2.4 2.7 4.7
    Education 1.5 1.4 2.2 4.1
    Management and Commerce 8.6 11.9 13.7 23.6
    Society and Culture 7.9 10.4 14.3 15.5
    Creative Arts 3.7 4.9 2.7 3.4
    Food, Hospitality and Personal Services 13.0 12.6 15.4 15.9
    Employment or life skills (mixed field) 37.1 28.8 34.6 17.9

Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.

Whilst more young beneficiaries participated in tertiary education after the introduction of YS, Table 11 shows that lower level courses were more prevalent for both YP and YPP, with Level 1 courses showing a particularly large change. Both YP and YPP participants were also much more likely than pre-YS youth beneficiaries to take courses that were less than a full-time equivalent year of study in the first year.

Around half of courses were undertaken at a private training establishment (PTE), with a third at a polytechnics. Fifteen percent of YPP participants and 7 percent of YP recipients studied at a Wānanga. In general there was a shift from polytech-based study before YS to PTE-based study after YS.

Sixty percent of courses taken by YP recipients were funded through the Youth Guarantee, which provides fees-free places to eligible students, but only 40 percent of YPP study was funded through this source. The policy was implemented in 2010, and as such few of the historical comparison groups had study funded through the policy. Eligibility for fees-free places under Youth Guarantee was initially available to 16 and 17 year olds, and extended to 18 year olds in 2014. This could explain the lower use of the policy by YPP participants relative to YP participants.

An evaluation of the Youth Guarantee (Earle 2016) produced slightly unclear and contradictory findings. The evaluation indicated that the policy had been successful in retaining young people in education in the first year, but not subsequently, and had increased attainment of Level 2 qualifications. There was some evidence of a positive impact on fulltime employment, but also an increase in benefit receipt and a decrease in educational participation.

It's possible that our analysis could pick up the effects of Youth Guarantee to some degree and ascribe them to the Youth Service. Looking at our matched comparison groups for YP and YPP respectively, we saw that many also accessed funding through Youth Guarantee, although rates were somewhat lower than for YP and YPP participants[17]. It could be that Youth Service increased access to Youth Guarantee, and that these policies may have complemented each other in helping young people achieve better outcomes, however the Youth Guarantee is unlikely to have influenced our findings to any great degree.

About a third were ‘mixed field' programmes, that is, courses on employment or life skills. These increased substantially following the introduction of YS. The remaining two-thirds were courses teaching occupationally-focused skills in fields such as food, hospitality and personal services, management and commerce, and education.

Note

  • [17]We found that 44 percent of the YP matched comparison group and 30 percent of the YPP matched comparison group engaged in tertiary study accessed Youth Guarantee funding. This compared to 60 percent and 40 percent for YP and YPP participants respectively.

Qualification attainment

Data on the qualification attainment of participants is given in Table 12. Because the qualification attainment data only records the year in which a qualification was completed and not the month of the year, we can’t attribute qualifications exactly to the period of YP/YPP participation. Instead we simply report all qualifications that were completed in the year of first YS enrolment or the following two years. The studying required to obtain the qualification may have been partly or fully carried out either before or after the YS enrolment period.

Table 12 - Highest qualification in the two calendar years after starting benefit
  YP YPP
  Participants
A
(%)
Historical cohort B
(%)
Participants
A
(%)
Historical cohort B
(%)
N 2,061 4,206 1,350 3,003
School qualifications        
    None 46.4 61.6 54.9 65.4
    NCEA 1 13.5 15.8 13.3 14.9
    NCEA 2 29.8 14.3 22.2 14.3
    NCEA 3 10.0 8.5 9.6 6.1
Tertiary qualifications        
    None 64.0 78.7 72.7 79.9
    Level 1 4.8 0.6 4.0 1.2
    Level 2 14.8 4.3 7.8 3.1
    Level 3 11.5 9.6 10.0 8.7
    Level 4 or higher 4.8 7.1 5.3 7.8
Industry training qualifications        
    None 98.7 99.1 98.9 99.6
    Level 1 0.6 s s s
    Level 2 0.6 0.9 s 0.8
    Level 3+ s s s s
All qualifications        
    None 35.8 55.9 46.5 59.1
    Level 1 13.5 15.0 14.4 14.3
    Level 2 29.1 14.4 19.3 13.8
    Level 3 17.5 11.0 13.8 8.3
    Level 4 or higher 3.9 4.0 4.7 5.1
Field of highest tertiary qualification        
    No tertiary qualification obtained 65.1 79.9 77.1 83.8
    Natural and Physical Sciences s s s s
    Information Technology 2.3 1.5 1.3 1.4
    Engineering and Related Technologies 2.0 1.4 s s
    Architecture and Building 1.9 0.9 s s
    Agriculture, Environmental and Related Studies 2.9 2.6 1.1 0.6
    Health 1.0 0.6 s 0.8
    Education s 0.4 0.9 0.7
    Management and Commerce 3.6 3.4 3.3 4.7
    Society and Culture 3.1 2.8 3.1 2.1
    Creative Arts 1.3 1.2 s 0.7
    Food, Hospitality and Personal Services 6.7 4.1 4.7 3.5
    Employment or life skills (mixed field) 9.5 1.6 5.6 1.9

Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.

Starting with the highest schoolqualification that was obtained in the year of starting on benefit or the following two years, around 13 percent of YP/YPP participants completed NCEA level 1 in this period, slightly fewer than pre-YS. Level 2 achievement rates were considerably higher than for the pre-YS benefit groups however, 30 percent for YP and 22 percent for YPP.

Around a third of YP and more than a quarter of YPP participants completed a tertiary qualification, with most YP completing a level 2 or 3 qualification, and YPP participants most commonly completing a level 3 qualification. Putting all types of qualification together, around half of YP participants and 38 percent of YPP participants completed a level 2 or higher qualification. These were both less than 30 percent for the pre-YS beneficiary populations.

Looking at the fields of the tertiary qualifications, ten percent of YP participants completed a qualification in life skills or general employment skills and 25% completed an occupationally-oriented qualification. Nearly all of these qualifications were National Certificates. Rates for YPP participants were 6 percent for life or employment skills and 14 percent for occupational qualifications.

5 Impact estimates

5.1  Introduction

In this section we present estimates of the impact of participation in the Youth Service for young people receiving either YP or YPP. The overall results for YP and YPP participants are given in Sections 5.2 and 5.3 respectively. In section 5.4, we look at alternative impact estimates for YPP, where matching on past activity uses the period write up to when they come onto YPP. In section 5.5, we look at whether the impacts vary for YPP participants who do not have a partner (ie, sole parents), or according to whether they transfer onto YPP from another benefit.

5.2 Main impact estimates - Youth Payment

Our main estimates of the impacts of YS participation for beneficiaries on YP are summarised in Table 13 and illustrated in Figure 2. Two impact estimates are presented. The first is an unadjusted impact estimate, which does not attempt to account for temporal effects. This is simply a comparison of outcomes for YP participant with outcomes for the historical comparison group of youth beneficiaries at the same number of months after coming onto a youth benefit. The second impact estimate applies an adjustment to our initial impact estimate to account for temporal changes between the pre-YS and post-YS periods. This adjustment is equal to the difference between outcomes for the YP and historical cohort matched comparisons. In most cases we believe the adjusted impact estimate to be the most robust measure of the impact of the Youth Service programme for beneficiaries on YP. The graphs in figure 2 show more detailed monthly activities prior to and after coming onto youth benefit in the pre-YS and post-YS periods, as well as activities for the matched comparison groups.

5.2.1  Impact on benefit receipt

In the first 12 months there is some evidence that YS results in beneficiaries being more likely to stay on benefit. This is potentially consistent with the YS focus on education rather than employment, and is consistent with the estimates reported in MSD (2015). We estimate that YP beneficiaries are 8 percentage points more likely to still be on benefit 6 months after first coming onto benefit than if they hadn't participated in Youth Service, and 5 percentage points more likely after 12 months. After 24 to 30 months, we estimate YP recipients to be more likely to have moved off benefit as a result of YS, with a 2 to 3 percentage point estimated reduction in benefit rates, but these were not statistically significant.

5.2.2 Impact on educational participation and achievement

A strong emphasis of Youth Service was on educational participation, and as such we would hope to see positive impacts on enrolment and achievement of qualifications. Impacts on enrolment in formal education and achievement of Level 1 to 3 qualifications are summarised in Table 13, while more detailed estimates by type of institution and type of qualification are given in the Appendix in Table A.6 and A.7.

In the first 12 to 18 months there's strong evidence of a positive impact on enrolment in formal education, with an estimated impact of 11.5 percentage points after 6 months, declining to 3 percentage points after 18 months. There is no evidence of a statistically significant impact after 24 or 30 months however.

Impacts on qualification achievement are somewhat muted compared to enrolment, with an estimated 3 percentage point impact on Level 1 qualification achievement in the two calendars years following the year participants first came onto a youth benefit. Impacts were slightly higher (3.5 ppt in year 1 and 3.7 ppt in year 2) for level 2 qualifications, but we did not estimate a significant impact on qualification attainment at level 3.

These estimates are smaller than those found in the earlier MSD study (MSD 2015) although as discussed earlier, this study did not control for temporal effects. Interestingly, our un-adjusted estimates were much larger than those found in the MSD study, although this study included YS participants who were already on a youth benefit when YS was introduced. It may be that YS had a smaller impact on educational participation and achievement for this group relative to the new youth beneficiaries that are the focus of our study. The MSD study also used propensity score matching to ensure YS participants and earlier youth beneficiaries were as similar as possible, and this may have influenced some of the differences in the results between the two studies.

Note that our measures of tertiary participation are based on tertiary enrolment records without the benefit of any data on attendance. If a student withdraws from a programme within the first few of weeks their enrolment record will be cancelled, but if they withdraw at a later stage no change is made to the administrative records and they are will be counted in our estimates as ‘studying' until the end of the enrolment period. Therefore, tertiary enrolment rates will tend to be overstated.

This matters for our study results if there was a significant difference between YS participants and non-participants in the likelihood of dropping out early. If the participants were more likely to drop out before the end of their programme, our estimates of the impact of YS on studying rates are likely to be overstated. On the other hand, if YS participants were less likely to drop out than the youth in the matched comparison group, our estimates of the impact of the impact on the programme on studying rates are likely to be understated. We have no evidence either way.

The lower impacts on qualification achievement relative to enrolment could indicate that some YP participants may not have actively engaged in education, despite their enrolment, that their participation was not sustained for long enough to achieve a qualification, or that they achieved insufficient credits to meet the standards for the qualification.

Unadjusted impacts are far larger than our final adjusted impacts, highlighting the general increases in educational participation and achievement over recent years. For example, whilst YP participants were 26 percentage points more likely to be enrolled in formal education 6 months after coming onto benefit than a comparable earlier cohort of youth beneficiaries, we ascribe less than half of this effect to Youth Service. Similarly, Youth Payment recipients have a 20 percentage point higher attainment of a Level 2 qualification in the calendar year after coming onto benefit, but we only associate a 3.5 percentage point impact to YS.

Table A.6 shows that YP participants were mostly supported to study while on benefit as we would expect. Impacts were mainly on tertiary (9 ppt after 6 months, and 7 ppt after 12 months) rather than school (3 ppt at 6 and 12 months) level. In line with this, qualification achievement mainly related to tertiary rather than school qualifications, although there was a positive impact on NCEA qualifications at Level 2 (Table A.7). The impacts on tertiary attainment (4 to 6 ppt) were higher than the overall attainment impacts, indicating that many young people were supported to attain tertiary qualifications at the same level at which they already held school qualifications. There was also a small significant positive impact on tertiary qualification attainment at level 3, counteracted by a small non-significant negative impact on NCEA level 3 attainment. It could be that YS resulted in an increase in tertiary enrolment that came, to some degree, at the expense of school enrolment.

5.2.3 Impact on employment

Two measures were used to look at employment outcomes for YS participants. The first simply measures whether the person had any salary and wage earnings in the month in question. This will capture low levels of part-time employment that may be undertaken while a person is in school or tertiary study, and/or while on benefit. A benefit recipient is able to earn a certain amount before their benefit is reduced, while they do not receive any benefit if they earn over a specified ‘abatement threshold'.

Section 2.2 describes the changes made to abatement rates and thresholds for youth beneficiaries following the introduction of Youth Service. These encouraged YP participants to work part-time (up to around 15-20 hours per week at the minimum wage), but strongly dis-incentivised working any more than this, including in low-paid full-time work. These effects change as participants aged out of YS, potentially moving to adult benefits, and a different abatement regime.

The picture is further complicated in the period before coming onto YP by the general reduction in teenage employment over the years before and after YS was introduced. This is illustrated in Figure 2, with a 10 percentage point gap in the employment rates of the YP and historical youth beneficiary cohorts evident in the months leading up to benefit. It is also reflected in Table A.3, with far more young people in the earlier period having a history of employment, including in the broader population of potential matches.

Employment in this period, when a young person is 15 or 16 and possibly still at school, is quite different from employment in our outcome period, when they may be aged 18 or over, and are likely to be seeking fulltime work. Nevertheless, we match on employment history as a way of controlling for differences that could lead to different future employment outcomes. As a result, comparisons over time on this measure are somewhat fraught, and our impact estimates should be treated with some caution.

As is illustrated in in Figure 2, moving onto a youth benefit causes employment rates for YP and the historical comparison to converge, but the same effect does not happen for their respective matched comparisons, with convergence occurring later, at around 12 months post-YS start, as many reach age 18. The result of this is that we do not expect our estimation method to provide sensible estimates of impact over the period immediately following participation in YS.

Nevertheless, the convergence of employment rates for the two matched comparison groups at 12 months after participation start gives us some confidence regarding the robustness of estimates from that point on. At 18 months, we estimate that YS has had a 5 percentage point impact on employment rates for YP recipients, while this declines slightly to 4 percentage points at 24 months. The impact at 30 months is smaller again, and no longer significant, although we only observe outcomes for less than two thirds of participants at this point.

As noted earlier, people are able to earn a certain amount without their benefit being affected. We construct an alternative outcome measure of employment while off benefit, to signal a move to employment enables a young person to be free of benefit support. In most cases this will signal a move to fulltime employment. Note that this looks the same as the standard employment measure in the pre-benefit period, as few youth beneficiaries had a history of benefit receipt. In the outcome period rates are lower than previously however, and start at 0 percent at time 0, the point when all youth beneficiaries move onto a youth benefit.

The impact of YS on ‘employment and off benefit' rates is smaller than the employment rate (at 2 to 2.8 ppt between 18 and 30 months), and is not statistically significant. While YS seems to have a positive impact on employment for youth beneficiaries, there is less evidence that it results in levels of employment sufficient for youth beneficiaries to become free of benefit, at least in the two years after coming onto a youth benefit.

5.2.4 Other impacts

We observe a number of other potential outcomes for YP participants, as summarised in Table A.8. The NEET measure identifies whether the young person is not in employment, education, or training, and reflects a combination of these effects, as illustrated in Table 13. Not surprisingly, given the positive impact on enrolment in formal education, YS is estimated to also have a strong reduction in NEET rates in the first 18 to 24 months after coming onto YP. There are small and generally non-significant estimated effects of YS on sentencing rates for YP participants, a small (less than 2 ppt) but significant negative impact on student allowance rates in the first 12 months after coming onto YP, possibly signalling a small shift from student allowances to YP for support while studying, and a small but significant reduction in the number of participants moving overseas 24 to 30 months after first coming onto YP.

Table 13 - Main impact estimates Youth Payment participants
Outcome Time since YS benefit start Participants Participants Matched comparison Historical cohort Historical matched comparison Unadjusted impact Adjusted impact Sig Std error
      ȳA ȳC* ȳB ȳD* ȳAB AB)-(ȳCD)    
    (N) (%) (%) (%) (%) (ppt) (ppt)   (ppt)
Receiving a benefit                    
  6 months 2,058 81.8 6.6 74.5 7.3 7.3 8.0 ** 1.22
  12 months 2,058 64.1 17.1 59.6 17.8 4.5 5.2 ** 1.47
  18 months 2,058 54.1 26.7 54.9 27.0 -0.8 -0.4   1.65
  24 months 1,959 48.2 32.3 52.9 34.7 -4.6 -2.3   1.62
  30 months 1,068 45.2 34.3 50.5 36.7 -5.3 -2.9   2.00
In employment                    
  6 months 2,058 20.1 29.0 18.4 31.6 1.7 4.3 ** 1.33
  12 months 2,058 27.1 33.8 24.6 34.3 2.6 3.0 * 1.37
  18 months 2,058 33.7 37.0 28.5 37.0 5.1 5.1 ** 1.51
  24 months 2,058 35.7 39.2 31.3 39.1 4.4 4.3 ** 1.55
  30 months 1,203 38.4 42.4 34.6 41.4 3.8 2.8   1.96
In employment and off benefit                    
  6 months 2,058 6.6 28.6 8.6 30.7 -2.1 0.1   1.19
  12 months 2,058 15.0 31.5 16.8 32.1 -1.8 -1.2   1.32
  18 months 2,058 23.2 33.7 21.0 33.5 2.2 2.0   1.46
  24 months 1,959 27.1 35.1 24.0 34.4 3.1 2.5   1.48
  30 months 1,068 29.8 36.8 26.5 36.3 3.3 2.8   1.92
Enrolled in formal education                    
  6 months 2,058 68.2 57.4 42.6 43.3 25.7 11.5 ** 1.48
  12 months 2,058 55.1 48.0 35.0 36.6 20.1 8.7 ** 1.57
  18 months 2,058 38.3 36.3 29.4 30.7 8.9 3.4 * 1.41
  24 months 2,058 30.2 30.2 24.0 25.1 6.2 1.1   1.42
  30 months 1,203 24.4 25.7 20.8 21.6 3.7 -0.4   1.57
Level 1 qualification or higher                    
  Year started 2,058 51.7 52.5 36.6 39.8 15.2 2.5 * 1.24
  Year started+1 2,058 64.6 63.4 46.1 48.0 18.5 3.0 * 1.39
  Year started+2 2,058 69.4 68.1 51.5 53.5 17.9 3.3 * 1.36
Level 2 qualification or higher                    
  Year started 2,058 32.8 35.6 20.0 23.8 12.8 1.0   1.18
  Year started+1 2,058 50.7 51.9 31.0 35.6 19.8 3.5 * 1.45
  Year started+2 2,058 57.0 58.2 38.0 42.9 19.0 3.7 * 1.46
Level 3 qualification or higher                    
  Year started 2,058 9.5 9.9 6.2 7.5 3.3 0.9   0.86
  Year started+1 2,058 21.7 23.5 15.4 17.7 6.3 0.6   1.24
  Year started+2 2,058 28.0 31.0 22.1 25.8 5.9 0.6   1.43

Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are indicated by an asterisk (*) and at the 99% confidence level by two asterisks (**).

Figure 2 - Outcomes over time for Youth Payment participants and their matched comparisons

 

Figure 2 - Outcomes over time for Youth Payment participants and their matched comparisons.
Figure 2 cont. - Outcomes over time for Youth Payment participants and their matched comparisons

 

Figure 2 cont. - Outcomes over time for Youth Payment participants and their matched comparisons.
Figure 2 cont. - Outcomes over time for Youth Payment participants and their matched comparisons

 

Figure 2 cont. - Outcomes over time for Youth Payment participants and their matched comparisons.

5.3 Main impact estimates - Young Parent Payment

The main estimates of the impacts of YS participation for beneficiaries on YPP are summarised in Table 14 and illustrated in Figure 3. As with YP two impact estimates are presented, an unadjusted impact estimate which does not account for temporal effects, and an adjusted impact estimate aiming to account for temporal changes between the pre-YS and post-YS periods.

The graphs in Figure 3 show monthly activities prior to and after coming onto a young parent benefit in the pre-YS and post-YS periods, as well as activities for the matched comparison groups. As discussed earlier, we would expect these patterns to change for young parent beneficiaries prior to coming onto a young parent benefit, and at an earlier time than for youth beneficiaries (YP and the YP historical comparison) due to possible pre-birth impacts of pregnancy. These effects are evident in the activity graphs, with benefit rates increasing, school enrolment rates dropping slightly, and tertiary enrolment rates flattening off around 9 months prior to YS participation. Employment rates seem to be impacted even earlier, starting to reduce around 12 months prior to YS participation. These pre-benefit impacts are not reflected in the matched comparison activity graphs, as we made a conscious decision to not match on these activities over the time period where pregnancy effects might be occurring. We believe these provide a better counterfactual for outcomes in the absence of pregnancy, and consequential receipt of a young parent benefit.

5.3.1 Impact on benefit receipt

Unlike for YP, we find little evidence that YS results in YPP beneficiaries staying for longer periods on benefit in the first months after coming onto benefit. While our adjusted impact of 4.6 percentage points 6 months after coming onto YPP might lead us to believe that YS has increased benefit rates for YPP participants at this point in time, the result is driven by differences in the matched comparison groups that carry over from the pre-young parent benefit period, and should be treated with some caution.

Over the longer term, we believe the results are likely to be more robust, and estimate that YS results in a small and non-significant reduction in benefit receipt among YPP recipients after 18 to 24 months. After 30 months the estimated reduction in benefit receipt, albeit for a reduced sample, is 6 percentage points and is statistically significant.

The earlier MSD study (MSD 2015) did not identify a significant impact on benefit receipt for YPP participants, either positive or negative. It's possible that benefit impacts were more muted for youth beneficiaries who were already on benefit when YS was introduced. These participants have been excluded from our study.

5.3.2 Impact on educational participation and achievement

As for YP participants, a strong emphasis of Youth Service is on young parents continuing to participate in education and training. As such we would again hope to see positive impacts on enrolment and achievement of qualifications. Impacts on enrolment and achievement are summarised in Table 14, while more detailed estimates are given in the Appendix in Table A.9 and A.10.

As for YP, in the first 12 to 18 months there's strong evidence of a positive impact on enrolment in formal education, however unlike YP this effect is largely sustained for a much longer period. Estimated impacts peak at 12 percentage points after 12 months, before declining to 6 percentage points at 30 months. As with the benefit impacts discussed above the earlier impact evaluation did not find significant qualification impacts for YPP.

Again, impacts on qualification achievement are somewhat more muted than the enrolment effects, nevertheless we find estimated positive significant impacts at levels 1, 2 and 3 in the second calendar year following first coming onto YPP (5.6, 5.4, and 4.4 percentage points respectively). As for YP, unadjusted impacts are far larger than our final adjusted impacts, highlighting the general increases in educational participation and achievement over recent years, and reinforcing the importance of using an approach that tries to account for this.

Table A.9 shows that the impact on YPP participant enrolment in formal education was split evenly between school and tertiary institutions in the first year or so after coming onto benefit, unlike for YP, where the impact was concentrated at the tertiary level. In the longer term, impacts on school enrolment tailed off, consistent with an ageing out of the school system, but tertiary enrolment showed an ongoing positive impact (of around 6 percentage points after 30 months). These results are reflected in impacts that are spread across the types of qualification (NCEA, non-NCEA, and tertiary).

Note that we estimate negative impacts on qualification attainment at almost all levels and types in the year of first YS participation. These are likely to be unreliable for some of the same reasons as we discussed around employment impacts for YP and benefit impacts for YPP. The matched comparison groups may be over-controlling for temporal effects in this period shortly after beginning YS, and these results should be treated with caution.

5.3.3 Impact on employment

As for YP, we use two measures of employment outcomes for YPP participants; any employment for a wage or salary in the month, and employment for a wage or salary in a month where someone did not receive a benefit. This analysis is subject to the same considerations and cautions as discussed for YP in section 5.2.3.

The impact of YS on employment rates is statistically significant and estimated to be 5.6 percentage points at 24 months, and 5.3 ppt at 30 months. Unlike for YP, this translates through to a positive and statistically significant employment off-benefit rate impact of 3.8 ppt at 24 months and 4.7 ppt at 30 months. These rates match the estimated off-benefit impacts discussed above closely, indicating that most of the impact of a shift off benefit seems to be related to a move into employment.

5.3.4 Other impacts

Other potential outcomes for YPP participants are summarised in the appendix in Table A.11. As we would expect from the positive education participation impacts, YS is estimated to reduce NEET rates for young parents considerably. There is no evidence of any broader impacts of YS for young parents, either on sentencing rates, student allowance receipt, or a move overseas. These rates were all very low already for young parents, and so the lack of an effect is not surprising.

Table 14 - Main impact estimates Young Parent Payment participants
Outcome Time since YS benefit start Participants Participants Matched comparison Historical cohort Historical matched comparison Unadjusted impact Adjusted impact Sig Std error
      ȳA ȳC* ȳB ȳD* ȳAB AB)-(ȳCD)    
    (N) (%) (%) (%) (%) (ppt) (ppt)   (ppt)
Receiving a benefit                    
  6 months 1,350 90.9 23.3 91.3 28.2 -0.4 4.5 ** 1.72
  12 months 1,350 85.1 28.4 88.0 34.2 -2.9 2.9   1.72
  18 months 1,350 80.2 35.6 86.0 38.9 -5.8 -2.5   1.78
  24 months 1,290 77.0 39.1 83.7 42.5 -6.8 -3.4   1.81
  30 months 834 73.0 41.7 80.9 43.8 -7.8 -5.7 * 2.23
In employment                    
  6 months 1,350 11.1 30.0 9.4 34.1 1.7 5.8 ** 1.64
  12 months 1,350 13.8 32.4 12.1 35.9 1.7 5.1 ** 1.68
  18 months 1,350 16.0 35.3 14.3 36.9 1.7 3.3   1.81
  24 months 1,350 19.8 36.7 15.7 38.2 4.1 5.6 ** 1.88
  30 months 897 23.4 39.1 18.5 39.5 5.0 5.3 * 2.14
In employment and off benefit                    
  6 months 1,350 3.3 26.7 2.8 30.0 0.6 3.8 ** 1.39
  12 months 1,350 5.1 29.1 4.9 31.5 0.3 2.7   1.48
  18 months 1,350 7.8 31.1 5.5 31.9 2.3 3.2 * 1.50
  24 months 1,290 10.0 31.6 7.3 32.7 2.7 3.8 * 1.56
  30 months 834 13.3 33.5 8.9 33.7 4.4 4.7 * 1.91
Enrolled in formal education                    
  6 months 1,350 43.8 41.3 24.3 33.0 19.5 11.2 ** 2.09
  12 months 1,350 43.6 35.8 24.0 28.1 19.5 11.8 ** 2.11
  18 months 1,350 34.0 27.8 21.6 24.3 12.4 8.9 ** 1.84
  24 months 1,350 29.6 23.6 20.4 21.1 9.1 6.7 ** 1.86
  30 months 897 26.1 19.7 19.0 18.8 7.0 6.1 ** 2.12
Level 1 qualification or higher                    
  Year started 1,350 41.6 48.7 36.9 43.3 4.7 -0.8   1.51
  Year started+1 1,350 52.9 57.6 41.9 50.1 11.0 3.6 * 1.66
  Year started+2 1,350 59.6 62.2 46.8 55.3 12.7 5.8 ** 1.76
Level 2 qualification or higher                    
  Year started 1,350 27.6 35.3 22.0 29.1 5.5 -0.7   1.40
  Year started+1 1,350 38.7 46.9 27.9 38.1 10.8 2.0   1.74
  Year started+2 1,350 47.1 52.2 34.3 44.7 12.8 5.3 ** 1.84
Level 3 qualification or higher                    
  Year started 1,350 10.7 16.7 8.2 13.9 2.4 -0.3   1.10
  Year started+1 1,350 19.1 25.8 13.5 22.6 5.6 2.5   1.38
  Year started+2 1,350 26.7 32.2 19.7 29.6 6.9 4.3 ** 1.56

Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are indicated by an asterisk (*) and at the 99% confidence level by two asterisks (**).

Figure 3 - Outcomes over time for Young Parent Payment participants and their matched comparisons

 

Figure 3 - Outcomes over time for Young Parent Payment participants and their matched comparisons.
Figure 3 cont. - Outcomes over time for Young Parent Payment participants and their matched comparisons

 

Figure 3 cont. - Outcomes over time for Young Parent Payment participants and their matched comparisons.
Figure 3 cont. - Outcomes over time for Young Parent Payment participants and their matched comparisons

 

Figure 3 cont. - Outcomes over time for Young Parent Payment participants and their matched comparisons.

5.4 Alternative impact estimates - Young Parent Payment

As discussed earlier, we decided to limit the degree to which young parent beneficiaries were matched on activities prior to coming onto benefit, particularly in the 12 months leading up to benefit. This was in the expectation that choices about activities would be impacted on by the upcoming birth of a child, and that some activities might be impacted on by the pregnancy itself in the months leading up to the birth. In addition to this, some young parent beneficiaries only come onto benefit some weeks and months after the child is born, meaning employment and educational participation could be impacted on at an earlier date than the receipt of a young parent benefit might signal.

We consider that it is likely to be more robust to match young parents to non-beneficiaries who have similar background characteristics and were following a similar track prior to the young parents' pregnancy, than to match to those who follow the same track following the pregnancy. Nevertheless, we wanted to explore the impact of this choice by matching young parents using educational participation right up to the benefit start date and benefit receipt up to 6 months pre-young parent benefit.

These alternative impact estimates are presented in Table 15. Generally, the impacts estimated using this alternative approach are larger than those estimated using our preferred approach, but are similar in magnitude and direction. Differences are greatest for qualification achievement, with estimated impacts around 2 percentage points larger at each level.

Table 15 - Impacts by for Young Parent Payment, alternative matching on activities up to participation
Outcome Time since YS benefit start Participants Participants Matched comparison Historical cohort Historical matched comparison Unadjusted impact Adjusted impact Sig Std error
      ȳA ȳC* ȳB ȳD* ȳAB AB)-(ȳCD)    
    (N) (%) (%) (%) (%) (ppt) (ppt)   (ppt)
Receiving a benefit                    
  6 months 1,329 91.0 22.1 91.6 29.7 -0.6 7.0 ** 1.65
  12 months 1,329 85.1 28.4 88.3 35.2 -3.2 3.5 * 1.69
  18 months 1,329 80.4 34.5 86.4 39.6 -6.1 -1.0   1.77
  24 months 1,272 77.1 39.6 83.9 43.1 -6.8 -3.3   1.79
  30 months 825 72.7 41.1 81.2 44.6 -8.5 -5.0 * 2.19
In employment                    
  6 months 1,329 11.3 31.2 9.6 34.6 1.7 5.1 ** 1.61
  12 months 1,329 14.0 34.1 12.0 36.2 2.0 4.2 * 1.63
  18 months 1,329 16.0 36.1 14.4 37.1 1.6 2.6   1.78
  24 months 1,329 19.6 37.7 15.9 38.1 3.7 4.1 * 1.68
  30 months 891 23.2 37.7 18.4 39.1 4.9 6.2 ** 1.98
In employment and off benefit                    
  6 months 1,329 3.2 28.2 2.7 30.6 0.5 2.9 * 1.44
  12 months 1,329 5.4 30.2 4.7 31.4 0.7 1.9   1.35
  18 months 1,329 7.7 31.6 5.4 31.8 2.3 2.5   1.47
  24 months 1,272 10.1 32.1 7.1 32.3 3.1 3.3 * 1.48
  30 months 825 13.8 32.4 8.8 33.3 5.0 5.9 ** 1.83
Enrolled in formal education                    
  6 months 1,329 42.4 36.6 23.7 28.5 18.8 10.7 ** 1.89
  12 months 1,329 43.1 32.1 23.3 25.9 19.8 13.7 ** 1.89
  18 months 1,329 33.6 25.5 21.3 22.6 12.3 9.4 ** 1.86
  24 months 1,329 29.1 21.9 19.7 19.9 9.4 7.4 ** 1.84
  30 months 891 26.3 19.9 18.4 17.4 7.9 5.4 ** 2.01
Level 1 qualification or higher                    
  Year started 1,329 42.7 46.0 36.6 40.9 6.0 0.8   1.16
  Year started+1 1,329 53.7 55.3 41.3 47.5 12.5 4.7 ** 1.49
  Year started+2 1,329 60.5 60.0 46.1 52.6 14.4 6.9 ** 1.56
Level 2 qualification or higher                    
  Year started 1,329 29.1 32.7 21.8 26.5 7.3 1.1   1.22
  Year started+1 1,329 40.4 43.8 27.4 35.4 13.0 4.6 ** 1.57
  Year started+2 1,329 48.5 49.4 33.6 41.6 14.9 7.1 ** 1.69
Level 3 qualification or higher                    
  Year started 1,329 10.8 14.2 8.0 11.8 2.9 0.4   1.06
  Year started+1 1,329 19.6 22.6 13.2 19.9 6.5 3.8 ** 1.43
  Year started+2 1,329 27.5 28.7 19.1 26.4 8.5 6.2 ** 1.72

Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are indicated by an asterisk (*) and at the 99% confidence level by two asterisks (**).

5.5 Impact estimates - sole parents on YPP

Sole parents are a particular group of interest in terms of the potential impact of the Youth Service, as the challenges they face may be different than for other young parents, who have the support of a partner.

The majority of Young Parent Payment recipients (81 percent) do not have a partner on benefit, and are assumed to be parenting alone. Consistent with a shift towards more young men being classified as young parents, the percentage of young parents who are sole parents has decreased considerably following the introduction of YS (from 88% in the historical comparison population). In both the pre-YS and post-YS periods, most young sole parents were female (98% and 95% respectively).

This section looks at the impact of Youth Service for sole parents on YPP, and impact estimates for this group are presented in Table 16. Generally speaking impact estimates are similar in direction and magnitude to our overall YPP estimates, perhaps not surprisingly, given that sole parents dominate YPP. Almost all impacts are slightly smaller for sole parents, however key impacts on educational enrolment, and qualification attainment are still statistically significant. There is some evidence of an increased shift off benefit and into employment, but for the most part this is not statistically significant[18].

Table 16 - Impacts for Young Parent Payment, sole parents only
Outcome Time since YS benefit start Participants Participants Matched comparison Historical cohort Historical matched comparison Unadjusted impact Adjusted impact Sig Std error
      ȳA ȳC* ȳB ȳD* ȳAB AB)-(ȳCD)    
    (N) (%) (%) (%) (%) (ppt) (ppt)   (ppt)
Receiving a benefit                    
  6 months 1,092 92.3 25.3 94.6 29.0 -2.3 1.4   1.75
  12 months 1,092 88.2 29.9 91.4 35.3 -3.2 2.1   1.80
  18 months 1,092 85.2 37.1 89.7 40.3 -4.5 -1.4   1.82
  24 months 1,044 81.9 40.8 87.3 43.9 -5.4 -2.3   1.96
  30 months 702 78.6 43.6 84.8 45.4 -6.2 -4.4   2.48
In employment                    
  6 months 1,092 9.9 29.4 7.6 33.4 2.3 6.3 ** 1.75
  12 months 1,092 12.6 31.9 9.7 35.1 2.9 6.1 ** 1.80
  18 months 1,092 14.3 34.3 11.9 36.1 2.4 4.2 * 1.92
  24 months 1,092 17.0 35.4 13.2 37.1 3.8 5.5 * 2.12
  30 months 750 20.4 38.0 16.0 38.3 4.4 4.8 * 2.27
In employment and off benefit                    
  6 months 1,092 2.2 26.1 1.1 29.3 1.1 4.3 ** 1.48
  12 months 1,092 3.6 28.3 2.8 30.5 0.7 3.0   1.55
  18 months 1,092 5.5 29.9 3.2 31.0 2.3 3.4 * 1.58
  24 months 1,044 7.5 30.2 4.9 31.6 2.6 4.0 * 1.72
  30 months 702 9.4 32.5 6.3 32.5 3.1 3.1   1.95
Enrolled in formal education                    
  6 months 1,092 42.6 41.2 24.1 33.4 18.5 10.6 ** 2.31
  12 months 1,092 42.9 36.3 24.2 28.5 18.6 10.9 ** 2.41
  18 months 1,092 34.3 28.0 22.1 24.5 12.3 8.8 ** 2.05
  24 months 1,092 30.2 23.6 20.9 21.5 9.3 7.2 ** 2.01
  30 months 750 26.8 19.2 19.5 19.0 7.3 7.1 ** 2.28
Level 1 qualification or higher                    
  Year started 1,092 41.5 48.9 36.2 42.9 5.3 -0.7   1.73
  Year started+1 1,092 52.7 58.0 41.2 49.9 11.6 3.5   1.86
  Year started+2 1,092 59.3 62.4 46.0 55.2 13.3 6.1 ** 1.98
Level 2 qualification or higher                    
  Year started 1,092 28.0 35.4 21.6 28.8 6.4 -0.2   1.55
  Year started+1 1,092 38.2 47.0 27.5 37.9 10.7 1.6   1.93
  Year started+2 1,092 46.4 52.5 33.8 44.6 12.6 4.7 * 2.08
Level 3 qualification or higher                    
  Year started 1,092 10.7 17.6 7.9 14.3 2.8 -0.5   1.24
  Year started+1 1,092 18.7 26.4 12.8 22.9 5.9 2.4   1.50
  Year started+2 1,092 26.1 32.7 19.3 29.9 6.8 3.9 * 1.75

Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are indicated by an asterisk (*) and at the 99% confidence level by two asterisks (**).

Note

  • [18]Estimates of the employment and off benefit effect are positive and statistically significant in the short-term after coming onto benefit, however we don’t consider these estimates to be robust, for reasons discussed earlier (see section 5.2.3).

5.6 Impact estimates - Young Parent Payment by previous benefit receipt

As indicated in Section 4.3, YPP participants were less likely than earlier young parent beneficiaries to have a history of benefit receipt when they moved onto YPP (44 percent, compared to 56 percent). This difference in the history of the two beneficiary groups could undermine the use of the historical comparison as a counterfactual for YPP. To assess the impact this could have, particularly on our key educational attainment estimates, we split the YPP population according to whether they had a benefit history or not, and calculated impact estimates separately. These results are presented in the appendix, in Tables A.12 and A.13. Graphs of selected outcomes over time for YPP participants and comparisons according to whether they have a history of benefit receipt are contrasted in Figure A.1.

Detailed comparisons of the characteristics of the two sub-populations are not presented, however some differences are worth noting. In particular, the group with a benefit history was much more likely to be older (around a half being aged 18 ¼ or over, compared to a fifth of those with no benefit history) both before and after YS was introduced. Those with a benefit history were also more likely to be female, but they otherwise had very similar composition to those without a benefit history across a broad range of background characteristics. In line with their age and time on benefit, they were also less likely to have spent time in school in recent months.

The results for the two YPP sub-populations are similar in many respects, although the statistical power of the analysis is reduced due to the smaller sample sizes. While the direction of the estimated impacts are generally consistent for YPP participants regardless of their history of benefit receipt, the magnitude and significance of the results does vary. In particular, the group with no benefit history is estimated to experience a higher likelihood of moving off benefit and into work after 30 months (8 percentage points) and slightly larger qualification attainment impacts in the second calendar year after starting to participate (5 to 6 ppt). These results are positive but non-significant for the group with a benefit history.

Regardless of previous benefit receipt, YPP participants were significantly more likely to be enrolled in formal education. These impacts were larger in the short-term (especially after 12 months) for young parents with no benefit history, but slightly larger after 24 months for those with benefit history. The adjustment for temporal effects was unimportant for the group with previous benefit receipt as there was little change over time for the pre-YS and post-YS matched comparison groups.

In general these results can be seen as supporting the general approach taken. The overall YPP impact estimates lie between the two sets of estimates for the two sub-populations when split according to whether they had a benefit history. The program seems to have a larger effect for participants without a history of benefit receipt. This group are generally younger and have more recent contact with the school system.

6 Conclusion

In this paper we evaluated the impact of the Youth Service on the educational participation, qualification achievement, benefit receipt, and employment rates of youth beneficiaries in the 24-30 months after they first received a youth benefit and were enrolled in YS. The impact was assessed relative to the previous case management regime, and benefit policy settings, and as such captures a number of changes made with the introduction of Youth Service.

The two YS benefits were evaluated separately; Youth Payment, available to 16 and 17 year olds who have no parental support due to exceptional circumstances; and Young Parent Payment, available to 16,17, and 18 year old parents. Administrative data from the Integrated Data Infrastructure (IDI) was used to measure participants’ outcomes. The impacts of the programme were estimated by comparing the outcomes of participants with those of an historical comparison group of similar youth beneficiaries, and this was adjusted for temporal effects through the use of non-beneficiary matched comparison groups.

Results should be treated with some caution. The service was introduced to all young beneficiaries at the same time, and so an obvious comparison was not available to assess the impact of the service. The service was also implemented over a period of considerable change, both in terms of economic conditions, and in terms of a broader focus on educational attainment for youth, particularly those most at risk of poor outcomes. Taken together, these circumstances present challenges in conducting an impact evaluation. While we've attempted to isolate the impact of the Youth Service from broader changes, we have had to make reasonable assumptions that may not hold. Nevertheless, we believe these results represent the best that can be done to get at the true impact of the programme.

The programme is intended to help youth beneficiaries, and has a focus on participation in education, rather than a shift off benefit and into work. Our impact estimates show that participation in YS led to increases in young people's participation in study or training, by around 12 percentage points at the peak. These study participation effects were particularly sustained for recipients of YPP, who were more likely to be enrolled in education even 30 months after first participating in YPP.

We also estimate that YS raised qualification achievement, albeit to a lesser degree. On average the programme raised the qualification of YP recipients at levels 1 and 2 by 3 to 4 percentage points in both of the two calendar years after first receiving YP. The impact of YS on YPP achievement was more delayed, but larger, with effects of 4 to 6 percentage points in the second calendar year after first receiving YPP.

YPP participants had an increased likelihood of studying at both school and tertiary levels, and achieved both school and tertiary qualifications, while most of the impact of YP was on tertiary enrolment and qualifications. YP participants were 5 to 6 percentage points more likely to gain tertiary qualifications at levels 1 and 2, indicating that these were often additional to school qualifications they may have already held at these levels.

We also found some evidence that YS resulted in more youth beneficiaries moving off benefit and into work, although these effects were smaller and not significant for YP (2.5 percentage points after 24 months, compared to a significant 4 percentage point impact for YPP).

Note that this evaluation has focused solely on outcomes that can be measured using administrative data, and on the first two and a half years after enrolment. The programme still seems to be having a positive impact on YPP participation in education at this point, and longer term outcomes on educational attainment, benefit receipt, and employment are likely. It is also possible that YS has been more effective in raising other dimensions of well-being, such as mental health, personal skills or social skills.

While we estimate some positive effects for youth beneficiaries arising out of the introduction of the Youth Service, the introduction of the service consisted of a number of simultaneous changes. These changes included a change in the way benefit rates were affected by earnings, changes to incentives and obligations, changes to sanctions for failing to comply with these obligations, changes in the way participants received their benefit payments, and a change in the model of case management. It is not possible to assess which of these changes contributed to the positive outcomes observed. Some changes may have had less positive, or even negative, effects.

References

Caliendo, M. and Kopenig, S. (2005) Some Practical Guidance for the Implementation of Propensity Score Matching. IZA Discussion Paper No. 1588.

Chowdry, H., Dearden, L. and Emmerson, C. (2008) Education maintenance allowance evaluation with administrative data: the impact of the EMA pilots on participation and attainment in post-compulsory education. Learning and Skills Council (LSC), London, UK.

Dearden, L., Emmerson, C., Frayne, C. and Meghir, C., ‘Conditional Cash Transfers and School Dropout Rates', The Journal of Human Resources, 2009, 44:4, pp. 827-857.

Dixon, S., Crichton, S. and McLeod, K. (2016) ‘Evaluation of the Impact of the Youth Service: NEET programme'. Treasury work paper, 2016 16/08.

Earle, D. (2016) Monitoring the Youth Guarantee policy 2014. Ministry of Education. Wellington.

Fletcher, M., Hanna, K. and Andersen, D. (2013). ‘The introduction of compulsory income management into New Zealand's social security system: The case of the Youth Service package', 15th Labour, Employment and Work Conference, Victoria University of Wellington, Wellington.

Gertler, P., Martinez, S, Premand, P., Rawlings, L., Vermeersch, C. (2016) Impact Evaluation in Practice, Second Edition. World Bank, Washington, DC.

Middleton, S., Perren, K., Maguire, S., and Rennison, J. (2005) Evaluation of Education Maintenance Allowance Pilots: Young People Aged 16 to 19 Years Final Report of the Quantitative Evaluation, Department for Education and Skills Research Report number 678, 2005.

Ministry of Social Development (2012) Youth Service Operational Manual. Ministry of Social Development, unpublished report.

Ministry of Social Development (2014) Youth Service Evaluation Report. Ministry of Social Development. Wellington.

Ministry of Social Development (2014 II) Youth Service Process Evaluation Report. Ministry of Social Development, unpublished report.

Ministry of Social Development (2015) Youth Service outcomes update March 2015: Technical report. Ministry of Social Development, unpublished report.

Song, Z., Safran, D., Landon, B., Landrum, M., He, Y., Mechanic, R., Day, M., Chernew, M. (2012) ‘The ‘Alternative Quality Contract,' Based on a Global Budget, Lowered Medical Spending and Improved Quality', Health Affairs 31(8), 1885-1894.

Stuart, E., Huskamp, H., Duckworth, D., Simmons, J., Song, Z., Chernew, M., Barry, C. (2014) ‘Using propensity scores in difference-in-differences models to estimate the effects of a policy change', Health Serv. Outcomes Res. Method. 14, 166-182.

Vaithianathan, R., Maloney, T., Wilson, M., Staneva, A. and Jiang, N. (2016) ‘Impact of school-based support on educational outcomes of teen-mothers: evidence from New Zealand's “teen parent units”' (forthcoming).

Werner, R., Konetzka, R., Stuart, E., Norton, E., Polsky, D. and Park, J. (2009) ‘The impact of public reporting on quality of postacute care', Health Service Research 44(4), 1169-1187.

Appendix

Table A.1 - Variables used in the propensity score regressions
Description Categories YP YPP

Personal/family characteristics and childhood experiences

     
Reference month (start of benefit) Calendar month Y Y
Previously participated in Youth Transition Service or YS: NEET Yes, No Y Y
Birth year 1991-1997 Y Y
Age in quarter years 15-18 Y Y
Female Yes, No Y Y
Asian ethnicity Yes, No Y Y
Maori ethnicity Yes, No Y Y
Pacific ethnicity Yes, No Y Y
Other (non-European) ethnicity Yes, No Y Y
Region of residence Regional council area (West Coast, Tasman, Nelson, Marlborough combined) Y Y
NZ Deprivation index of the meshblock of residence 1-10 Y Y
Proportion of childhood spent overseas 0-9%, 10-49%, 50%+ Y Y
Parent/caregiver served a community sentence Yes, No Y Y
Parent/caregiver served a custodial sentence Yes, No Y Y
Mother/caregiver unqualified Yes, No Y Y
Proportion of childhood supported by benefit 0%, 1-9%, 10-24%, 25-49%, 50-74%, 75%+ Y Y
CYF care and protection finding in childhood Yes, No Y Y
CYF care and protection placement in childhood Yes, No Y Y
Number of CYF care and protection notifications in childhood 0, 1-2, 3-9, 10+ Y Y
CYF youth justice placement Yes, No Y Y
Number of CYF youth justice referrals 0, 1-2, 3-9, 10+ Y Y
Used secondary mental health, drug or alcohol services Yes, No Y Y

Schooling history prior to Youth Service

     
Decile of last school attended 1-10 Y Y
Number of schools attended since 2006 0-2, 3, 4+ Y Y
Ever had special education funding Yes, No Y Y
Any record of truancy Yes, No Y Y
Authority of school last attended State, Other Y Y
Type of school last attended Correspondence, Other Y Y
Number of stand-downs from school 0, 1-2, 3-9, 10+ Y Y
Number of suspensions from school 0, 1-2, 3-9, 10+ Y Y
Highest qualification obtained before benefit start None, NCEA Level 1, NCEA Level 2, NCEA Level 3, tertiary qualification (e.g. national certificate) Y Y
Highest qualification a non-NCEA school qualification Yes, No Y Y
Highest qualification a tertiary qualification Yes, No Y Y
Table A.1 cont. - Variables used in the propensity score regressions
Description Categories YP YPP
Types of tertiary programme previously enrolled in Life or employment skills only, occupational skills only, Both types of programme Y Y
Number of NCEA level 1 credits held 0-4, 5-49, 50-59, 60-79, 80+ Y Y
Number of NCEA level 2 credits held 0-4, 5-49, 50-59, 60-79, 80+ Y Y
Number of NCEA level 3 credits held 0-4, 5-49, 50-59, 60-79, 80+ Y Y

Other involvement in education, benefit and employment prior to Youth Service

     
Number of months on benefit 0-6 months before reference month 0, 1-3, 4-6 Y  
Number of months on benefit 6-12 months before reference month 0, 1-3, 4-6 Y  
Number of months on benefit 12-24 months before reference month 0, 1-3, 4-6, 7-9, 10-12 Y Y
Number of months employed 0-6 months before reference month 0, 1-3, 4-6 Y Y
Number of months employed 6-12 months before reference month 0, 1-3, 4-6 Y Y
Number of months employed 12-24 months before reference month 0, 1-3, 4-6, 7-9, 10-12 Y Y
Number of months NEET 6-12 months before reference month 0, 1-3, 4-6 Y Y
Number of months NEET 12-24 months before reference month 0, 1-3, 4-6, 7-9, 10-12 Y Y
Number of months enrolled in school 0-6 months before reference month 0, 1-3, 4-6 Y  
Number of months enrolled in school 6-12 months before reference month 0, 1-3, 4-6 Y Y
Number of months enrolled in school 12-24 months before reference month 0, 1-3, 4-6, 7-9, 10-12 Y Y
Number of months enrolled in tertiary 0-6 months before reference month 0, 1-3, 4-6 Y  
Number of months enrolled in tertiary 6-12 months before reference month 0, 1-3, 4-6 Y Y
Number of months enrolled in tertiary 12-24 months before reference month 0, 1-3, 4-6, 7-9, 10-12 Y Y

Table A.2 - School and educational characteristics for Youth Payment

  Post-YS Pre-YS
  Participants Matches Potential matches Historical cohort Historical matches Potential matches
Decile of last school attended            
    1-2 19.7 22.6 11.9 22.1 24.3 12.2
    3-4 20.7 19.8 15.5 20.9 22.1 16.3
    5-6 24.2 23.2 24.9 21.7 22.6 25.4
    7-8 16.5 15.7 24.0 16.5 15.1 23.5
    9-10 5.8 5.0 18.7 7.4 6.5 17.7
    NA 13.1 13.8 5.0 11.4 9.4 4.9
Last school was correspondence school 8.9 10.3 2.5 8.6 7.5 2.2
Last school was state school 92.6 94.6 85.2 93.4 94.3 85.4
No. of schools attended since 2006            
    0-1 2.8 2.9 7.3 41.8 45.7 66.0
    2 21.6 21.9 48.6 32.7 29.4 27.4
    3 28.4 29.9 31.2 16.2 17.4 5.2
    4+ 47.4 45.5 12.8 9.2 7.5 1.3
Had special education funding 4.2 4.1 1.5 1.8 1.5 1.3
Had truancy record 40.4 39.2 11.4 41.3 39.6 9.9
Number of stand-downs from school            
    None 53.1 54.1 86.0 54.3 54.0 86.4
    1-9 25.4 24.9 9.4 25.1 25.9 9.6
    10+ 21.7 21.0 4.6 20.5 20.1 4.1
No. of suspensions from school            
    None 77.3 79.6 95.3 78.7 78.8 95.6
    1-9 10.8 9.3 2.4 8.4 8.4 2.1
    10+ 12.1 11.2 2.3 12.9 12.9 2.3
Age when left school            
    Still at school 36.6 42.9 77.9 30.7 35.9 74.0
    15 or less 20.8 17.8 4.2 28.3 27.5 6.1
    16 34.7 33.5 11.4 35.7 31.7 13.3
    17 7.9 6.0 6.5 5.2 4.8 6.6
Highest qualification held, year before participation            
    None 69.7 69.7 34.5 75.9 75.9 36.7
    NCEA Level 1 18.8 19.0 37.9 15.6 15.6 36.8
    NCEA Level 2 8.3 8.3 24.2 6.1 6.1 23.2
    NCEA Level 3 s s 1.4 s s 1.5
    Tertiary qualification 2.9 2.9 1.9 2.3 2.3 1.8
No. of Level 1 NCEA credits year before YS            
    None / Missing 32.2 32.1 18.3 36.3 35.6 17.3
    Less than 40 26.2 26.5 9.0 25.9 26.1 9.7
    40-59 10.2 10.2 6.3 10.6 10.2 6.8
    60-79 10.1 9.8 9.2 9.7 11.4 9.9
    80+ 21.3 21.3 57.3 17.5 16.6 56.2
No. of Level 2 NCEA credits year before YS            
    None / Missing 72.3 71.1 54.5 72.8 71.0 47.9
    Less than 40 18.4 19.4 18.7 19.6 21.4 25.3
    40-59 3.4 3.8 4.5 3.0 3.1 4.8
    60-79 3.1 2.9 5.7 2.3 2.6 5.8
    80+ 3.2 2.9 16.6 2.2 1.9 16.2
No. of Level 3 NCEA credits year before YS            
    None / Missing 95.0 94.9 91.3 95.7 95.4 89.5
    Less than 40 4.8 4.8 7.0 4.1 4.4 8.6
    40+ s s 1.7 s s 1.9

Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.

Table A.3 - Other activities before enrolling in Youth Payment

  Post-YS Pre-YS
  Participants Matches Potential matches Historical cohort Historical matches Potential matches
Time since last school enrolment            
     17.1 9.9 6.9 14.9 8.5 7.3
    4-6 months 10.3 10.2 3.6 9.8 10.9 4.2
    7-12 months 16.5 17.2 5.6 17.1 16.4 6.2
    1-2 years 16.9 18.1 5.3 21.9 22.6 6.9
    2+ years 2.5 1.7 0.7 5.7 5.6 1.5
    Not applicable 36.6 42.9 77.9 30.7 36.0 74.0
School in previous 18 months            
    None 8.3 7.6 2.7 13.9 13.4 4.0
    1-6 months 14.7 15.0 3.9 16.9 17.7 4.9
    7-12 months 20.4 20.6 6.3 20.8 19.6 6.9
    13-18 months 56.7 56.7 87.2 48.4 49.2 84.2
Tertiary in previous 18 months            
    None 68.2 67.5 88.6 79.6 79.4 89.5
    1-6 months 20.6 20.7 6.4 11.7 12.4 6.6
    7-18 months 9.8 10.5 4.2 7.1 6.8 3.2
    13-18 months 1.5 1.5 0.8 1.6 1.5 0.7
Employment in prev. 18 months            
    None 62.5 60.6 56.4 46.8 47.4 45.5
    1-6 months 22.9 22.6 20.9 29.1 25.8 21.6
    7-18 months 8.9 9.3 10.6 13.2 13.3 12.9
    13-18 months 5.7 7.4 12.2 10.9 13.6 20.0
NEET in previous 18 months            
    None 42.6 46.2 84.2 34.6 40.4 81.2
    1-6 months 32.7 31.3 10.4 35.8 31.3 12.3
    7-18 months 16.2 15.0 3.3 18.2 17.4 3.9
    13-18 months 8.6 7.4 2.1 11.4 11.0 2.6
Benefit receipt in prev. 18 months            
    None 99.4 99.3 98.4 98.6 98.6 97.9
    1-6 months 0.7 s 0.5 0.9 0.8 0.7
    7-18 months s s 0.5 0.4 0.4 0.7
    13-18 months s s 0.5 s s 0.7
Average monthly earnings in previous 6 months            
    None 71.6 68.5 62.3 61.5 59.8 54.0
    Less than $500 11.2 11.1 16.1 13.0 10.6 17.0
    $500-$1000 9.3 8.9 12.4 11.0 11.4 16.0
    $1000-$2500 7.1 9.8 7.7 12.3 15.0 11.1
    More than $2500 0.9 2.0 1.5 2.1 3.3 1.9
Tertiary before participation            
    Occupational skills 15.9 15.5 8.2 12.2 12.0 8.7
    Life or employment skills 11.4 11.7 2.4 8.0 8.3 2.2
    Both types of  programme 7.1 6.7 1.4 1.7 1.7 0.7

Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons.

Table A.4 - School and educational characteristics for Young Parent Payment

  Post-YS Pre-YS
  Participants Matches Potential matches Historical cohort Historical matches Potential matches
Decile of last school attended            
    1-2 29.2 28.3 11.9 27.2 27.6 12.1
    3-4 20.7 21.6 15.5 23.6 23.1 16.4
    5-6 20.5 21.6 24.9 19.7 21.6 25.4
    7-8 10.5 10.5 23.7 11.2 10.2 23.4
    9-10 3.8 3.3 18.4 3.8 3.4 17.5
    NA 15.1 14.9 5.7 14.5 13.9 5.3
    Last school was correspondence school 14.0 13.4 3.2 12.1 11.3 2.4
    Last school was state school 96.2 96.2 85.5 93.6 93.9 85.3
No. of schools attended since 2006            
    0-1 5.1 5.3 10.9 57.9 60.5 72.7
    2 31.2 32.1 52.2 28.2 25.7 22.4
    3 28.5 27.4 26.1 9.3 9.8 4.0
    4+ 35.4 35.6 10.8 4.6 4.0 1.0
    Had special education funding 1.6 1.6 1.5 0.4 0.4 1.2
    Had truancy record 44.1 41.4 11.5 37.0 34.9 9.1
No. of stand-downs from school            
    None 59.2 61.2 85.4 67.1 67.5 86.2
    1-9 25.2 25.2 9.9 22.4 22.1 9.7
    10+ 15.6 13.6 4.7 10.6 10.4 4.0
No. of suspensions from school            
    None 84.0 84.6 95.1 87.6 88.0 95.5
    1-9 8.0 7.6 2.5 6.1 6.0 2.2
    10+ 8.5 7.8 2.4 6.4 6.2 2.3
Age when left school            
    Still at school 24.3 26.7 58.5 16.8 19.7 54.7
    15 or less 21.2 20.3 4.2 27.8 27.1 6.6
    16 34.5 34.5 12.6 35.3 34.6 14.8
    17 16.3 14.7 17.2 16.7 15.0 16.3
Highest qualification held, year before YS            
    None 67.3 67.5 28.2 67.6 67.6 31.4
    NCEA Level 1 13.1 13.4 28.4 14.5 14.5 28.8
    NCEA Level 2 11.8 11.6 28.6 10.2 10.2 26.2
    NCEA Level 3 2.4 2.2 11.2 1.0 1.0 10.0
    Tertiary qualification 5.6 5.6 3.6 6.7 6.6 3.6
Level 1 NCEA credits year before YS            
    None / Missing 27.4 28.3 14.0 29.7 29.5 14.2
    Less than 40 32.5 31.0 9.0 29.9 27.9 9.8
    40-59 10.5 11.6 6.5 11.5 12.3 6.9
    60-79 11.6 10.9 9.8 10.8 12.6 10.2
    80+ 18.3 18.7 60.6 18.2 17.7 58.9
Level 2 NCEA credits year before YS            
    None / Missing 63.3 61.9 41.4 62.6 60.6 38.2
    Less than 40 22.3 23.8 17.8 24.7 25.6 23.0
    40-59 6.0 6.2 6.5 4.9 6.2 6.4
    60-79 5.1 5.3 8.9 4.2 4.6 8.2
    80+ 3.8 2.7 25.4 3.7 3.1 24.1
Level 3 NCEA credits year before YS            
    None / Missing 88.6 88.4 77.3 89.4 89.5 76.6
    Less than 40 8.9 8.9 10.4 8.9 8.7 11.7
    40+ 2.7 2.7 12.3 1.7 1.8 11.7

Notes: All sample size numbers are randomly rounded.

Table A.5 - Other activities before enrolling in Young Parent Payment

  Post-YS Pre-YS
  Participants Matches Potential matches Historical cohort Historical matches Potential matches
Time since last school enrolment            
     4.0 3.6 4.6 3.0 3.0 4.4
    2-3 months 4.9 4.5 4.3 4.5 3.8 5.1
    4-6 months 7.8 6.0 6.0 6.5 4.7 7.1
    7-12 months 16.0 15.6 12.9 14.9 12.9 11.1
    1-2 years 26.1 25.6 9.6 28.2 29.2 11.5
    2-3 years 14.7 16.3 3.6 22.0 22.7 5.2
    3+ years 2.7 2.0 0.5 4.3 4.2 0.9
    Not applicable 24.3 26.7 58.5 16.7 19.7 54.7
School in previous 18 months            
    None 30.1 31.0 8.1 39.8 41.3 10.8
    1-6 months 18.3 16.7 6.2 19.5 18.6 7.5
    7-12 months 20.0 18.9 13.6 17.9 15.5 11.8
    13-18 months 31.8 33.6 72.1 22.8 24.7 69.9
Tertiary in prev. 18 months            
    None 72.6 66.1 77.4 75.7 69.0 78.4
    1-6 months 13.1 17.4 11.8 10.8 16.1 12.6
    7-12 months 12.2 13.1 9.1 10.1 11.3 7.3
    13-18 months 2.0 3.1 1.8 3.4 3.6 1.7
Employment in prev. 18 months            
    None 64.8 56.6 47.5 44.8 37.8 37.7
    1-6 months 18.7 21.6 21.5 27.4 26.1 21.3
    7-12 months 9.6 10.5 12.9 16.0 16.3 14.5
    13-18 months 7.1 11.4 18.1 11.8 19.9 26.4
NEET in previous 18 months            
    None 18.3 28.1 74.6 14.3 27.8 72.7
    1-6 months 34.5 32.3 17.3 35.0 30.9 18.4
    7-12 months 23.8 20.3 4.8 27.0 21.3 5.2
    13-18 months 23.6 19.2 3.3 23.7 19.9 3.8
Benefit receipt in previous 18 months            
    None 57.7 82.4 94.4 44.9 77.4 93.2
    1-6 months 33.6 11.1 3.2 43.4 13.4 3.9
    7-12 months 8.0 5.8 1.3 9.1 6.4 1.5
    13-18 months 0.9 0.7 1.1 2.6 2.8 1.4
Average monthly earnings in the 6 months before YS participation            
    None 78.8 66.1 54.4 71.8 54.1 47.1
    Less than $500 7.1 9.1 15.4 8.1 9.9 15.6
    $500-$1000 6.0 8.5 13.7 7.8 10.7 16.5
    $1000-$2500 6.9 13.4 12.8 10.6 20.9 16.4
    More than $2500 1.1 3.1 3.8 1.8 4.4 4.4
Nature of any tertiary enrolments before YS enrolment            
    Occupational skills 16.7 21.6 19.3 19.2 24.5 19.5
    Life or employment skills 7.6 9.6 2.8 5.8 6.5 2.8
    Both types of programme 5.3 5.6 2.0 2.8 3.4 1.7

Notes: All sample size numbers are randomly rounded.

Table A.6 - Youth Payment impact estimates by type of enrolment

Outcome Time since YS benefit start Participants Participants Matched comparison Historical cohort Historical matched comparison Unadjusted impact Adjusted impact Sig Std error
      YA YC* YB YD YA-YB (YA-YB)-(YC*-YD*)    
    (N) (%) (%) (%) (%) (ppt) (ppt)   (ppt)
Enrolled in formal education                    
  6 months 2,058 68.2 57.4 42.6 43.3 25.7 11.5 ** 1.48
  12 months 2,058 55.1 48.0 35.0 36.6 20.1 8.7 ** 1.57
  18 months 2,058 38.3 36.3 29.4 30.7 8.9 3.4 * 1.41
  24 months 2,058 30.2 30.2 24.0 25.1 6.2 1.1   1.42
  30 months 1,203 24.4 25.7 20.8 21.6 3.7 -0.4   1.57
Enrolled in formal education with no benefit support                    
  6 months 2,058 9.3 54.8 9.7 41.6 -0.4 -13.6 ** 1.49
  12 months 2,058 15.6 42.0 14.4 33.0 1.1 -7.8 ** 1.37
  18 months 2,058 16.9 29.4 15.0 25.8 1.9 -1.7   1.25
  24 months 1,959 17.5 23.3 14.2 19.3 3.2 -0.8   1.20
  30 months 1,065 17.2 20.3 12.7 16.2 4.4 0.3   1.42
Enrolled in school                    
  6 months 2,058 31.6 33.4 24.0 29.0 7.6 3.2 * 1.45
  12 months 2,058 21.4 23.6 15.0 20.2 6.4 3.0 * 1.28
  18 months 2,058 11.4 12.7 10.5 13.1 0.8 1.3   1.00
  24 months 2,058 6.6 6.9 6.3 6.6 0.3 0.1   0.79
  30 months 1,203 3.7 3.0 4.3 3.3 -0.6 -0.2   0.71
Enrolled in tertiary education                    
  6 months 2,058 40.8 27.3 20.1 15.3 20.7 8.7 ** 1.46
  12 months 2,058 37.2 26.5 21.1 17.2 16.0 6.7 ** 1.48
  18 months 2,058 28.1 24.8 19.8 18.3 8.3 1.9   1.33
  24 months 2,058 24.1 23.9 18.5 18.9 5.5 0.5   1.32
  30 months 1,203 21.2 22.7 17.0 18.6 4.2 0.1   1.54
Enrolled in tertiary education and not at school                    
  6 months 2,058 36.6 23.9 18.7 14.4 17.9 8.3 ** 1.37
  12 months 2,058 33.8 24.3 20.0 16.4 13.8 5.8 ** 1.45
  18 months 2,058 27.1 23.6 18.9 17.7 8.3 2.3   1.31
  24 months 2,058 23.5 23.3 17.7 18.4 5.7 0.9   1.30
  30 months 1,203 20.7 22.4 16.4 18.4 4.3 0.2   1.53

Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are indicated by an asterisk (*) and at the 99% confidence level by two asterisks (**).

Table A.7 - Youth Payment impact estimates by type of qualification gained

Outcome Time since YS benefit start Participants Participants Matched comparison Historical cohort Historical matched comparison Unadjusted impact Adjusted impact Sig Std error
      YA YC* YB YD YA-YB (YA-YB)-(YC*-YD*)    
    (N) (%) (%) (%) (%) (ppt) (ppt)   (ppt)
Level 1 NCEA qualification or higher                    
  Year started 2,058 43.1 44.6 30.6 33.2 12.5 1.2   1.22
  Year started+1 2,058 52.3 52.0 34.9 36.9 17.5 2.4   1.34
  Year started+2 2,058 54.7 54.2 38.1 39.8 16.6 2.1   1.44
Level 2 NCEA qualification or higher                    
  Year started 2,058 25.1 27.8 13.5 16.7 11.6 0.5   1.04
  Year started+1 2,058 38.2 40.4 18.0 23.4 20.2 3.2 * 1.26
  Year started+2 2,058 41.7 43.6 22.5 27.8 19.2 3.4 * 1.41
Level 3 NCEA qualification or higher                    
  Year started 2,058 4.2 5.0 2.3 2.6 1.9 -0.4   0.53
  Year started+1 2,058 9.6 12.7 5.9 8.0 3.7 -1.0   0.77
  Year started+2 2,058 11.1 15.5 8.5 11.7 2.6 -1.1   0.91
Level 1 non-NCEA qualification or higher                    
  Year started 2,058 19.8 20.8 5.4 5.2 14.4 -1.2   1.19
  Year started+1 2,058 31.8 31.0 12.1 12.3 19.7 0.9   1.40
  Year started+2 2,058 35.6 35.6 17.1 17.9 18.5 0.8   1.45
Level 2 non-NCEA qualification or higher                    
  Year started 2,058 10.1 11.1 3.6 3.9 6.5 -0.7   0.91
  Year started+1 2,058 20.8 21.0 8.8 9.1 12.1 0.2   1.28
  Year started+2 2,058 24.6 25.7 13.1 14.0 11.5 -0.1   1.37
Level 3 non-NCEA qualification or higher                    
  Year started 2,058 2.6 2.3 1.1 1.2 1.5 0.4   0.49
  Year started+1 2,058 6.3 5.7 3.6 3.1 2.6 0.1   0.75
  Year started+2 2,058 8.3 8.2 6.0 5.7 2.3 -0.2   0.89
Level 1 Tertiary qualification or higher                    
  Year started 2,058 17.5 15.7 5.9 6.7 11.6 2.5 * 1.12
  Year started+1 2,058 34.3 28.3 14.6 14.2 19.7 5.6 ** 1.42
  Year started+2 2,058 42.1 36.9 21.4 22.1 20.7 6.0 ** 1.51
Level 2 Tertiary qualification or higher                    
  Year started 2,058 14.1 13.1 5.7 6.5 8.4 1.8   1.05
  Year started+1 2,058 29.7 25.2 14.2 13.9 15.6 4.2 ** 1.33
  Year started+2 2,058 37.6 33.4 20.8 21.6 16.8 5.0 ** 1.43
Level 3 Tertiary qualification or higher                    
  Year started 2,058 6.4 6.1 3.8 4.8 2.6 1.3   0.79
  Year started+1 2,058 15.5 13.8 10.7 11.0 4.8 1.9   1.16
  Year started+2 2,058 21.9 20.1 16.7 17.6 5.2 2.7 * 1.31

Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are indicated by an asterisk (*) and at the 99% confidence level by two asterisks (**).

Table A.8 - Youth Payment impact estimates on secondary outcomes

Outcome Time since YS benefit start Participants Participants Matched comparison Historical cohort Historical matched comparison Unadjusted impact Adjusted impact Sig Std error
      YA YC* YB YD YA-YB (YA-YB)-(YC*-YD*)    
    (N) (%) (%) (%) (%) (ppt) (ppt)   (ppt)
NEET - Not in employment, education or training                    
  6 months 2,058 23.0 27.3 45.3 36.6 -22.2 -12.9 ** 1.39
  12 months 2,058 29.0 30.8 46.6 39.3 -17.6 -9.1 ** 1.50
  18 months 2,058 37.2 36.2 48.4 41.2 -11.2 -6.2 ** 1.65
  24 months 2,058 41.4 38.0 50.1 43.2 -8.7 -3.6 * 1.67
  30 months 1,203 43.4 39.4 48.5 43.5 -5.1 -1.0   1.91
Received a community sentence                    
  6 months 2,058 4.7 2.8 6.0 4.1 -1.3 0.0   0.69
  12 months 2,058 5.5 4.2 9.4 6.2 -3.9 -1.9 * 0.76
  18 months 2,058 7.4 5.4 11.1 8.0 -3.7 -1.1   0.84
  24 months 2,058 7.7 5.8 11.7 8.2 -3.9 -1.6   0.83
  Any time to 24 mths 2,058 15.9 11.4 22.3 16.2 -6.4 -1.5   1.13
Received a custodial sentence                    
  6 months 2,058 0.9 0.9 1.1 1.3 -0.3 0.1   0.33
  12 months 2,058 1.9 1.3 1.7 1.7 0.2 0.6   0.43
  18 months 2,058 2.0 1.9 2.4 2.1 -0.4 -0.1   0.51
  24 months 2,058 2.3 2.2 2.8 2.3 -0.5 -0.4   0.55
  Any time to 24 mths 2,058 6.6 4.8 7.5 5.8 -0.9 0.1   0.88
Receiving a student allowance                    
  6 months 2,058 0.9 1.3 2.6 1.2 -1.7 -1.8 ** 0.39
  12 months 2,058 4.2 4.1 5.1 3.5 -0.9 -1.5 * 0.61
  18 months 2,058 7.7 6.7 7.0 5.8 0.7 -0.2   0.84
  24 months 2,058 9.8 8.3 7.9 7.8 1.9 1.3   0.91
  30 months 1,203 8.7 8.7 7.4 8.1 1.3 0.7   1.03
Located overseas                    
  6 months 2,058 s 0.9 0.6 0.8 s s   s
  12 months 2,058 s 1.0 0.6 0.8 s s   s
  18 months 2,058 0.9 1.2 0.6 0.9 0.2 -0.1   0.28
  24 months 2,058 0.9 1.6 1.1 0.9 -0.2 -0.9 ** 0.31
  30 months 1,203 1.7 1.5 2.4 1.1 -0.7 -1.1 * 0.50

Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons. Estimates that are statistically significant at the 95% confidence level are indicated by an asterisk (*) and at the 99% confidence level by two asterisks (**).

Table A.9 - Young Parent Payment impact estimates by type of enrolment

Outcome Time since YS benefit start Participants Participants Matched comparison Historical cohort Historical matched comparison Unadjusted impact Adjusted impact Sig Std error
      YA YC* YB YD YA-YB (YA-YB)-(YC*-YD*)    
    (N) (%) (%) (%) (%) (ppt) (ppt)   (ppt)
Enrolled in formal education                    
  6 months 1,350 43.8 41.3 24.3 33.0 19.5 11.2 ** 2.09
  12 months 1,350 43.6 35.8 24.0 28.1 19.5 11.8 ** 2.11
  18 months 1,350 34.0 27.8 21.6 24.3 12.4 8.9 ** 1.84
  24 months 1,350 29.6 23.6 20.4 21.1 9.1 6.7 ** 1.86
  30 months 897 26.1 19.7 19.0 18.8 7.0 6.1 ** 2.12
Enrolled in formal education with no benefit support                    
  6 months 1,350 4.0 35.6 2.4 27.4 1.6 -6.6 ** 1.55
  12 months 1,350 6.0 29.3 2.8 21.8 3.2 -4.3 ** 1.46
  18 months 1,350 4.9 21.3 3.0 18.3 1.9 -1.2   1.26
  24 months 1,290 5.6 17.4 2.8 15.0 2.8 0.3   1.31
  30 months 834 6.1 14.0 3.4 13.3 2.7 2.0   1.43
Enrolled in school                    
  6 months 1,350 26.2 18.7 16.2 14.1 10.1 5.5 ** 1.60
  12 months 1,350 21.6 12.4 13.3 9.5 8.3 5.3 ** 1.39
  18 months 1,350 12.9 6.7 8.3 6.0 4.6 3.8 ** 1.17
  24 months 1,350 8.9 4.2 5.6 3.4 3.3 2.5 * 1.01
  30 months 897 4.3 1.7 4.1 1.6 0.3 0.2   0.81
Enrolled in tertiary education                    
  6 months 1,350 19.6 24.7 9.1 19.7 10.4 5.5 ** 1.72
  12 months 1,350 24.0 24.9 11.5 19.1 12.5 6.8 ** 1.78
  18 months 1,350 22.7 21.8 14.0 18.7 8.7 5.6 ** 1.67
  24 months 1,350 22.0 19.8 15.5 18.1 6.5 4.8 ** 1.70
  30 months 897 22.4 18.4 15.5 17.4 6.9 5.9 ** 2.03
Enrolled in tertiary education and not at school                    
  6 months 1,350 17.3 22.9 8.1 18.8 9.2 5.2 ** 1.60
  12 months 1,350 22.0 23.6 10.7 18.5 11.3 6.2 ** 1.75
  18 months 1,350 20.9 21.1 13.3 18.4 7.6 4.8 ** 1.65
  24 months 1,350 20.4 19.3 15.1 17.8 5.4 3.8 * 1.71
  30 months 897 21.7 18.1 15.1 17.2 6.7 5.8 ** 2.03

Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are indicated by an asterisk (*) and at the 99% confidence level by two asterisks (**).

Table A.10 - Young Parent Payment impact estimates by type of qualification gained

Outcome Time since YS benefit start Participants Participants Matched comparison Historical cohort Historical matched comparison Unadjusted impact Adjusted impact Sig Std error
      YA YC* YB YD YA-YB (YA-YB)-(YC*-YD*)    
    (N) (%) (%) (%) (%) (ppt) (ppt)   (ppt)
Level 1 NCEA qualification or higher                    
  Year started 1,350 35.6 40.9 30.3 33.0 5.3 -2.6   1.36
  Year started+1 1,350 42.9 46.0 32.8 35.7 10.1 -0.2   1.55
  Year started+2 1,350 46.7 48.2 34.3 37.8 12.3 1.9   1.62
Level 2 NCEA qualification or higher                    
  Year started 1,350 21.6 26.9 14.4 17.2 7.2 -2.6 * 1.30
  Year started+1 1,350 28.9 34.9 17.6 21.5 11.3 -2.0   1.49
  Year started+2 1,350 34.0 37.6 19.9 24.4 14.1 0.9   1.60
Level 3 NCEA qualification or higher                    
  Year started 1,350 5.3 7.3 2.0 3.5 3.3 -0.5   0.70
  Year started+1 1,350 8.7 10.9 4.3 7.1 4.4 0.7   0.96
  Year started+2 1,350 11.1 12.9 6.0 9.9 5.2 2.2   1.15
Level 1 non-NCEA qualification or higher                    
  Year started 1,350 14.4 18.7 6.4 9.5 8.0 -1.1   1.37
  Year started+1 1,350 21.8 26.4 10.5 15.1 11.3 -0.1   1.51
  Year started+2 1,350 26.2 30.2 14.8 19.9 11.4 1.2   1.67
Level 2 non-NCEA qualification or higher                    
  Year started 1,350 9.3 12.4 5.0 8.0 4.4 0.0   1.12
  Year started+1 1,350 14.9 19.6 8.0 12.7 6.9 0.0   1.37
  Year started+2 1,350 19.6 23.1 11.8 17.2 7.8 1.8   1.56
Level 3 non-NCEA qualification or higher                    
  Year started 1,350 2.4 4.9 2.1 4.0 0.4 -0.6   0.79
  Year started+1 1,350 5.6 8.2 3.8 6.9 1.8 0.5   0.97
  Year started+2 1,350 8.7 11.1 6.3 10.0 2.3 1.2   1.19
Level 1 Tertiary qualification or higher                    
  Year started 1,350 12.9 18.9 8.7 13.1 4.2 -1.6   1.28
  Year started+1 1,350 24.7 30.2 13.5 21.2 11.2 2.2   1.73
  Year started+2 1,350 33.3 37.1 20.5 28.5 12.8 4.2 * 1.84
Level 2 Tertiary qualification or higher                    
  Year started 1,350 10.9 16.9 8.1 12.7 2.8 -1.4   1.17
  Year started+1 1,350 20.7 27.6 12.7 20.6 8.0 1.0   1.64
  Year started+2 1,350 29.3 34.2 19.3 27.7 10.0 3.4 * 1.75
Level 3 Tertiary qualification or higher                    
  Year started 1,350 7.6 12.0 6.4 10.5 1.1 -0.4   1.04
  Year started+1 1,350 14.0 19.6 10.5 17.8 3.5 1.7   1.34
  Year started+2 1,350 20.9 25.1 16.3 23.9 4.6 3.4 * 1.49

Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are indicated by an asterisk (*) and at the 99% confidence level by two asterisks (**).

Table A.11 - Young Parent Payment impact estimates on secondary outcomes

Outcome Time since YS benefit start Participants Participants Matched comparison Historical cohort Historical matched comparison Unadjusted impact Adjusted impact Sig Std error
      YA YC* YB YD YA-YB (YA-YB)-(YC*-YD*)    
    (N) (%) (%) (%) (%) (ppt) (ppt)   (ppt)
NEET - Not in employment, education or training                    
  6 months 1,350 49.1 36.9 67.7 41.2 -18.5 -14.3 ** 2.20
  12 months 1,350 46.9 38.9 65.4 43.3 -18.5 -14.1 ** 2.17
  18 months 1,350 53.1 43.6 66.2 45.3 -13.1 -11.3 ** 2.28
  24 months 1,350 54.0 44.9 65.7 46.3 -11.7 -10.2 ** 2.17
  30 months 897 54.5 46.2 64.6 46.8 -10.1 -9.4 ** 2.59
Received a community sentence                    
  6 months 1,350 1.8 2.4 2.8 3.5 -1.0 0.0   0.68
  12 months 1,350 2.7 3.1 3.7 4.1 -1.0 0.0   0.82
  18 months 1,350 2.9 3.8 3.9 4.6 -1.0 -0.2   0.78
  24 months 1,350 3.6 3.6 4.0 4.3 -0.4 0.3   0.83
  30 months 1,350 6.9 8.0 9.1 10.3 -2.2 0.1   1.19
Received a custodial sentence                    
  6 months 1,350 s s s 0.4 s s   s
  12 months 1,350 s s s 0.5 s s   s
  18 months 1,350 s s s 0.5 s s   s
  24 months 1,350 s s 0.5 0.5 s s   s
  30 months 1,350 s s 1.3 2.2 s s   s
Receiving a student allowance                    
  6 months 1,350 s 4.7 1.1 5.4 s s   s
  12 months 1,350 1.3 6.0 0.9 5.9 0.4 0.3   0.67
  18 months 1,350 1.8 6.9 1.2 6.3 0.6 0.0   0.74
  24 months 1,350 2.2 7.6 1.1 6.6 1.1 0.2   0.80
  30 months 897 2.3 6.4 1.2 6.3 1.2 1.0   0.93
Located overseas                    
  6 months 1,350 s 0.9 0.6 0.9 s s   s
  12 months 1,350 s 1.1 1.0 1.2 s s   s
  18 months 1,350 s 1.3 0.9 1.1 s s   s
  24 months 1,350 0.9 1.8 0.8 1.2 0.1 -0.5   0.42
  30 months 897 s 1.7 1.8 1.5 s s   s

Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons. Estimates that are statistically significant at the 95% confidence level are indicated by an asterisk (*) and at the 99% confidence level by two asterisks (**).

Figure A.1 - Outcomes over time for Young Parent Payment participants and their matched comparisons by benefit history

 

Figure A.1 - Outcomes over time for Young Parent Payment participants and their matched comparisons by benefit history.
Figure A.1 cont. - Outcomes over time for Young Parent Payment participants and their matched comparisons by benefit history

 

Figure A.1 cont. - Outcomes over time for Young Parent Payment participants and their matched comparisons by benefit history.

Table A.12 - Young Parent Payment impact estimates, participants with no previous benefit receipt

Outcome Time since YS benefit start Participants Participants Matched comparison Historical cohort Historical matched comparison Unadjusted impact Adjusted impact Sig Std error
      YA YC* YB YD YA-YB (YA-YB)-(YC*-YD*)    
    (N) (%) (%) (%) (%) (ppt) (ppt)   (ppt)
Receiving a benefit                    
  6 months 759 90.9 17.8 88.6 21.4 2.3 5.8 ** 2.09
  12 months 759 83.8 24.1 84.3 28.2 -0.5 3.5   2.36
  18 months 759 77.5 32.4 82.3 33.0 -4.8 -4.3   2.65
  24 months 726 75.2 36.8 80.5 37.3 -5.2 -4.8   2.63
  30 months 453 70.2 40.4 77.0 39.1 -6.8 -8.2 ** 2.83
In employment                    
  6 months 759 11.1 28.9 14.5 35.7 -3.5 3.3   2.05
  12 months 759 14.2 32.4 16.4 37.7 -2.1 3.2   2.23
  18 months 759 15.8 36.0 19.1 39.5 -3.3 0.3   2.35
  24 months 759 19.0 38.3 19.3 40.9 -0.3 2.2   2.39
  30 months 504 24.4 38.7 23.0 43.2 1.5 5.9 * 2.81
In employment and off benefit                    
  6 months 759 3.6 26.5 4.5 32.3 -1.0 4.8 ** 1.78
  12 months 759 5.5 29.6 7.5 34.1 -2.0 2.5   1.88
  18 months 759 8.3 31.6 8.0 35.0 0.3 3.7   2.08
  24 months 726 9.9 33.1 9.5 35.9 0.4 3.2   2.08
  30 months 453 15.9 33.8 11.6 37.7 4.3 8.3 ** 2.65
Enrolled in formal education                    
  6 months 759 51.8 47.8 28.6 36.1 23.1 11.5 ** 2.77
  12 months 759 52.2 39.5 27.3 30.5 24.9 15.8 ** 2.70
  18 months 759 40.3 31.2 25.0 26.6 15.3 10.7 ** 2.68
  24 months 759 32.4 26.1 23.4 22.3 9.0 5.2 * 2.45
  30 months 504 29.2 21.4 20.9 19.5 8.3 6.4 * 2.72
Level 1 qualification or higher                    
  Year started 759 40.3 47.8 39.3 45.7 1.0 -1.1   2.17
  Year started+1 759 54.5 58.1 44.8 53.0 9.8 4.6 * 2.34
  Year started+2 759 61.7 63.2 50.5 57.7 11.2 5.7 * 2.51
Level 2 qualification or higher                    
  Year started 759 26.5 33.6 22.5 29.8 4.0 0.2   1.99
  Year started+1 759 39.9 47.4 30.2 40.2 9.7 2.5   2.47
  Year started+2 759 49.8 53.4 37.0 47.0 12.8 6.4 * 2.62
Level 3 qualification or higher                    
  Year started 759 8.3 14.2 7.7 13.0 0.6 -0.7   1.51
  Year started+1 759 19.0 24.1 14.1 22.7 4.9 3.5   1.98
  Year started+2 759 27.7 32.4 21.1 30.5 6.5 4.6   2.42

Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are indicated by an asterisk (*) and at the 99% confidence level by two asterisks (**).

Table A.13 - Young Parent Payment impact estimates, participants with previous benefit receipt

Outcome Time since YS benefit start Participants Participants Matched comparison Historical cohort Historical matched comparison Unadjusted impact Adjusted impact Sig Std error
      YA YC* YB YD YA-YB (YA-YB)-(YC*-YD*)    
    (N) (%) (%) (%) (%) (ppt) (ppt)   (ppt)
Receiving a benefit                    
  6 months 591 90.9 29.9 93.1 33.3 -2.3 1.1   2.49
  12 months 591 86.8 34.0 90.8 38.7 -4.0 0.7   2.51
  18 months 591 83.8 39.1 88.9 43.5 -5.2 -0.8   2.50
  24 months 564 78.7 42.0 86.1 46.7 -7.4 -2.7   2.66
  30 months 378 76.2 43.7 83.8 47.5 -7.6 -3.7   3.64
In employment                    
  6 months 591 11.2 31.5 5.6 32.9 5.5 7.0 ** 2.44
  12 months 591 13.7 32.5 9.0 34.5 4.7 6.7 ** 2.44
  18 months 591 16.2 34.5 10.6 34.9 5.7 6.0 * 2.69
  24 months 591 20.3 35.0 12.7 36.1 7.6 8.7 ** 2.74
  30 months 396 22.7 38.6 14.8 36.6 7.9 5.9   3.38
In employment and off benefit                    
  6 months 591 2.5 26.9 1.4 28.2 1.1 2.4   2.07
  12 months 591 4.6 28.9 2.6 29.6 1.9 2.6   2.12
  18 months 591 7.1 29.9 3.5 29.6 3.6 3.2   2.34
  24 months 564 10.1 29.3 5.8 30.1 4.3 5.1 * 2.25
  30 months 381 11.0 33.1 6.9 30.6 4.2 1.7   2.87
Enrolled in formal education                    
  6 months 591 33.0 33.0 21.0 30.6 12.0 9.7 ** 3.05
  12 months 591 32.0 31.5 21.7 26.2 10.3 5.1   3.00
  18 months 591 25.9 22.8 19.0 22.5 6.9 6.6 ** 2.55
  24 months 591 25.9 20.3 18.3 20.2 7.6 7.5 ** 2.65
  30 months 396 22.7 18.2 17.8 18.1 4.9 4.9   3.18
Level 1 qualification or higher                    
  Year started 591 43.1 49.7 35.0 41.4 8.1 -0.3   2.07
  Year started+1 591 50.8 56.9 39.4 47.9 11.3 2.4   2.50
  Year started+2 591 55.8 60.9 44.0 53.3 11.8 4.3   2.61
Level 2 qualification or higher                    
  Year started 591 28.9 36.5 21.7 28.5 7.3 -0.7   1.98
  Year started+1 591 37.6 46.2 26.2 36.6 11.3 1.8   2.44
  Year started+2 591 43.7 50.8 32.2 43.0 11.4 3.6   2.71
Level 3 qualification or higher                    
  Year started 591 13.7 19.8 8.8 14.8 4.9 -0.1   1.74
  Year started+1 591 19.3 27.9 13.0 22.4 6.3 0.7   2.09
  Year started+2 591 25.4 32.5 18.7 28.9 6.7 3.1   2.42

Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are indicated by an asterisk (*) and at the 99% confidence level by two asterisks (**).