In 2014, the Social Innovation Fund agreed to invest $3 million in the United Way of Greenville County over three years to support OnTrack Greenville. The goal of OnTrack Greenville is to increase success in four high poverty middle schools where students have historically been at risk of not graduating high school. At the same time, the Social Innovation Fund is interested in funding innovations like OnTrack Greenville to, as they say, “find out what works and make it work for more people.”
The key to “finding out what works” is evaluation.
OnTrack Greenville funders met on August 7, 2017 for an update session focused on evaluation.
Dr. Tracy Waters with the Riley Institute at Furman University reported that evaluation supports OnTrack Greenville partners in celebrating success, building evidence, scaling to serve more students, and improving the programs and the overall initiative. There are a few layers of evaluation in the initiative (see Figure A).
- Implementation Partners are being evaluated on the implementation of their program, including fidelity to the model – are they doing what they said they’d do? – and to understand factors such as how the work varies across school sites and how the partners are learning and adjusting
- Implementation Partners are also evaluated for impact: are they impacting participants in the way they intended and are they improving attendance, behavior, and course performance?
- The overall impact of OnTrack Greenville is being evaluated – is OnTrack Greenville as a whole having an impact on attendance, behavior, and core course performance? Is it impacting other things important to educational outcomes, such as academic self-confidence and perseverance and relationships with teachers and adults?
- The process of collaboration is also being evaluated – are OnTrack Greenville partners living their shared collective impact values?
Dr. Waters presented the first findings from the implementation evaluations and clarified that the evaluations are not intended to be punitive or to be used as report cards, but rather; as tools for the partners to use to aid in adjusting their performance and ensure they are continually learning.
Implementation partners shared ways that the implementation evaluations have increased their capacity and their learning. Below are several ways the partners reported they have been impacted in their work.
Inspire action. Communities In Schools Project Director Kim Mahaffey shared that in August of 2015 when the OnTrack initiative began, it felt like trial by fire. While CIS had been at work in Greenville County Schools since 1991, the additional partners and intentional alignment were at first challenging. In addition, their own staff had rapidly expanded from one CIS staff member per school to one per grade level at each school (in fact, CIS serves 2,300 schools nationwide, though the staffing saturation levels of CIS Greenville is uncommon elsewhere).
With new people engaged – both internal to CIS and through the OnTrack Greenville partnership – it meant referrals were coming to CIS from all directions, including teachers, administrators, parents, and the students themselves. This was a good thing; they want students who needed services to receive them, however, the process needed to be efficient and effective. The implementation evaluation helped CIS understand how to clarify roles and responsibilities and improve communication. It resulted in a new referral form that ensured CIS captured all relevant information about the student and their needs while also allowing the referring party to efficiently refer students in need to CIS.
As a result, referrals are now purposeful and action-oriented. The referral process has made CIS’s work more effective and has been added to CIS’s national database of best practices.
Create awareness of services and roles. Greenville Health System’s School Based Health Centers are a tremendous asset in each school, but students cannot use them unless their parents or guardians complete a consent form. GHS has worked with their legal counsel to make the form as brief and straightforward as possible, but it still includes some degree of legal language that can be off-putting to parents, and some parents may question signing a form for a service they can’t anticipate their students will need.
GHS and OnTrack partners unintentionally discovered a way to increase signed consent forms: sports physicals. Before students can participate in school sports, they must obtain a sports physical performed by a physician, which can run around $40 and require a parent’s time off from work. The School Based Health Centers provide them free of charge at school…as long as the consent form is signed. Many families seized on this opportunity, which allowed GHS to create awareness of what else the School Based Health Center could do for their children, and it boosted other families’ completion of consent forms (even those who didn’t plan to participate in sports). Megan Shropshire, School Health Manager with the Bradshaw Institute for Community Child Health and Advocacy at GHS says, “We’ve seen an increase of referrals from the EWRS teams to the SBHC of students dealing with behavioral, emotional, and social health issues which has allowed us to get to the bottom of what is causing our students to struggle in and out of the classroom.”
As a result, the School Based Health Centers saw 160 more visits in the ’16-17 academic year than the prior year, and 94% of students returned to class after a visit to the center, which helps OnTrack meet the goal of increasing attendance and course performance.
Build relationships and communication. Jennifer Driscoll, Data and Quality Specialist with Greenville County Schools, remembered that when OnTrack Greenville began two years ago there were many meetings inside and outside the partnership to get the initiative launched. Now she reflects that the successes that all of the partners have seen are a result of relationships and trust, and those could be built only through the intentional investment of time.
She shared an example of the transition the partners have experienced. “At the beginning, we were simply learning about the services that each other provides. Now, we’ve moved from wondering what Communities in Schools does (for example), to hearing directly from our students how CIS has helped them, to wanting to actively engage CIS in our work. And we are enriched by Greenville Health System’s assessment of why our students are off track – and they are reasons we would not uncover in the classroom.”
Ms. Driscoll stated that with the early establishment of the partnership value of “students at the center,” now there is a prevailing focus on “how can I work with you to help you help your students?” It with the time invested in building relationships and communication that the partners have worked through challenges to see early impact and success. Further, this collaborative success and the visibility of OnTrack has created a sense of accomplishment among the partners, which has even further strengthened trust.
Clarifying purpose. Dr. Ansel Sanders, President & CEO of Public Education Partners (PEP), noted that PEP’s role in OnTrack is to serve at the instructional level, coaching and supporting teachers as they work with students.
However, he acknowledged that schools have Instructional Coaches, Title I ELA and Math Support Specialists, and principals –who are the instructional leaders of schools – so, PEP began OnTrack Greenville with efforts to integrate into an existing system of instructional and academic supports.
He said, “I believe that the implementation evaluations are a secret sauce of OnTrack Greenville. Having objective information about what is working well and what is not working and why, allows us to have productive conversations to adjust the work in real time to ultimately have a better impact on students.”
Using the impact evaluation, PEP and Greenville County Schools revised their coaching support schedule for the 2017-2018 school year, determining for PEP to primarily focus on instructional coaching at two of the OnTrack schools and Greenville County Schools to offer more coaching capacity at a different one. That way, the coaching support could be more consistently and efficiently be deployed to better serve each the target schools and “keep students at the center,” OnTrack’s core value.
Evaluation has already informed OnTrack Greenville partners in how to best serve students and families on their journey to high school graduation. All partners are anticipating the first impact evaluation results, which will be released by the Riley Institute in early 2018. However, a few exciting preliminary results were shared from the 2015-2016 pilot baseline year to the second year (2016-2017), which is considered the first full implementation year:
Attendance:
- Overall, negligible change in chronic absenteeism
- 18% decrease in chronic absenteeism at one of the OnTrack Greenville Schools
Behavior:
- 22% decrease in total number of referrals
- 6% decrease in number of students receiving referrals
- 20% decrease in out of school suspension referrals
- 5% decrease in percent of students with out of school suspensions
- 18% decrease in total number of days in out of school suspensions
These are positive statistics for such a complex initiative, which is in some ways, still in the toddler stage. Significant change usually takes five years. Therefore, partners are very excited about the impact that will occur in the future.
Evaluation is what helps us know what interventions work and helps us offer them to more people. But it’s only possible when funders pay for evaluation and when practitioners – whether educators, health care providers, human service workers, artists or others – are given enough time to try, build trust, adapt, and try again as those involved with OnTrack have been able to do. GPP is proud to have been a part of OnTrack Greenville, and we look forward to continuing the learning.
