Everyone in education is asking the same question:
Are we on the right track?
Like most fundamental questions, it’s elegantly simple and devilishly hard to answer. You want to know your plans will lead to increased success in serving your students, but do you know what you mean by “success”?
When I ask school leaders “what is success to you?” I typically hear two kinds of responses:
And, and, and…
College-ready graduates. Low drop-out rates. Positive school climate. Well-rounded students. High levels of community engagement.
These are all laudable, worthy goals. However, an essential truism of leading any team is that everyone needs to know and agree on both the destination and path to reach it. Vision matters.
Deciding that success is “Mushville Middle School Graduates Independent Learners” doesn’t mean that having a welcoming school isn’t important. It simply means that you know the primary objective of your educational institution and all other choices will flow from that objective.
Honing down on a crisp, clear definition of success is very hard work. There are multiple questions you will need to explore. But if you have too many definitions of success, your team will not be aligned, your energy will dissipate, and you will have little control over your outcomes.
Check out Innovating with an Exhausted Staff to explore how to use metric-building to rally your team
The State Metric
Another approach we’ve heard is to adopt the state accountability metric. You know you’ll be called to account for your performance against this metric, so using it is a logical choice. But before adopting the metric that’s handed to you, remember that those hard choices were made. They just weren’t made by you.
Every definition of “successful school” inherently benefits particular outcomes and disadvantages others. There is no way around this. It’s not because of nefarious intentions to benefit particular students or school structures or neighborhood contexts. It’s simply because choices have consequences. To understand what we mean, consider one of the most talked-about components of state accountability scores: the growth metric.
There are multiple growth metrics in use across the country. Each growth metric is a legitimate response to of a set of philosophical questions about the goal of the public education system. The table below outlines 4 of the most common growth metrics in use today.
|GROWTH MODEL||ESSENTIAL QUESTION|
|Growth to Proficiency (GTP)||Are students on a trajectory to meet proficiency by a particular grade?|
|Value Tables (VT)||Are students successfully moving from one level to the next level?|
|Growth Percentiles (GP)||How well are students doing compared to other students who got the same scores in the past?|
|Value-Added Metrics (VAM)||Are students growing as much as we expected they would, based on past performance?|
The psychometricians who build these models have to many multiple difficult questions, including:
- How to score students who enter the year proficient
- How to score students who make significant but insufficient growth
- Where to place cut scores
- How to aggregate multiple scores
- How to account for the impact of factors that impact scores, but are not directly attributable to school performance (e.g. English proficiency, migratory status, poverty)
The result of these decisions can be seen in the scores schools receive. Take the three schools highlighted in the graphs below. Orange School, for example, scores very well on the GTP metric, with 81% of students meeting their benchmark. They also perform very well on the Value Table, with a Scaled +2 score. However, their performance is essentially average (51st percentile) under the SGP. In short, school performance can look very different depending on how you define “success.”
All 3 of these metrics are statistically defensible, but as a school leader, you need more than “statistically defensible” to rally your stakeholders behind your metric of success. Which brings us to another essential truth of leadership: you aren’t leading if you don’t have followers.
VALUE OF INDIVIDUALIZED METRICS
Building Metrics Aligns Stakeholders
When people don’t agree on the metrics, they are not going to line up behind the conclusions. If your staff doesn’t agree with the state score (and most don’t), it’s going to be hard to galvanize them to action.
When you bring your team together to build metrics that matter to them, they will own the outcome.
Creating a shared metric of success was a driving goal behind Polarys. We needed to get beyond arguments about whether it was the unprepared students or the unfair tests that were giving us low scores.
By lowering a 25 point achievement gap to an 8 point gap, it helped staff create a more manageable and achievable goal.
You don’t need to build an entirely new statistical model, but you can use the metric building process to identify the few key performance indicators you can agree are essential to enacting that shared vision.
Building Metrics Focuses on Local Priorities
Finally, building your own metric can help you – the school leader – to focus on whatever is most important to you and your school. By nailing down what is really important, you’re also making an affirmative stand about what is NOT important.
The number one reason we end up overwhelmed by data is because we think we have to pay attention to all of it. You don’t. You only need to focus on the data that is going to help you answer your key questions.
In a business setting, this is called “Key Performance Indicators.” I once knew a chicken farmer who told me he knew how well he was doing by how high the truck was stacked when it left with the day’s sales. He didn’t need spreadsheets. He didn’t need sales reports. He just needed to see the truck as it rolled out every night.
That’s the kind of clarity you can achieve when you know what your Key Performance Indicators will be.
PUTTING YOUR KPI TO WORK: METRICS IN ACTION
Now let’s look at how having a clear metric can help you drive change to meet your goal.
Brook Street Middle School serves a high needs population. Over 40% of the students begin at the school two or more grade levels below math expectations.
To be successful, the school needs to graduate more 8th grade students ready to continue their math education and achieve proficiency on state exams in 10th grade. In order to meet that goal, the school needs to take entering 6th graders, 40% of whom are 2 grade levels below, and achieve an average of 1.5 years of progress per year in math.
After going through the Ends-Outcomes-Indicators protocol, the leadership determines that their metric will be “percent of students meeting their individualized growth plans”, a growth-to-proficiency metric that will allow them to track students on a personal level.
Once the KPI is set, the next step is to explore the RTI model with a focus on how well the system works in tailoring interventions to meet a variety of needs. The leadership wants answer three essential questions:
- Is the RTI model going to help us achieve our goal?
- What in the RTI model can be adjusted to improve outcomes?
- Is the RTI system adequately resourced to achieve our goals?
Brook Elementary School Growth Goals
Individual Student Growth Goals
Question 1: Is our RTI model going to get us there?
The first step is to articulate the question in a quantifiable way: how well our intervention students are doing in making gains in math proficiency each year, compared to our non-intervention students?
Analyzing the program from this perspective, it appears the model is on the right track, in that roughly as many students receiving intervention support are seeing score increases as those not receiving interventions. We can also see that too many kids are sliding backward. This suggests there’s a percent of the population, both intervention and non-intervention, that aren’t being served well by the programming.
Now we put our KPI to work by charting the percent of students making sufficient annual growth to reach proficiency by 10th grade.
What this shows is that the model is working, in that RTI students are seeing significant growth. But the school is still falling short of their goal. To achieve the growth-to-proficiency goals, the school will need to take a close look at the details of the program.
Question 2: What’s working for us?
Looking at the intervention options, we can see there are 9 different choices that teachers and interventionists employ to address the math gap. This list begs the question: can the school optimize to reduce the number of programs it supports and increase value per solution?
|Student-Interventions Assigned||Unique Uses|
|Repeated Practice w/Feedback||207|
Table 1: Total annual program implementations.
Note: Intervention programs listed are as an example only and should not be considered an accurate reflection of the typical implementation or outcomes of these programs.
In order to determine what programs provide the biggest returns, benchmark assessment data and RTI data are essential. This data allows the school to see which students got which interventions for which subtopics and whether they experienced growth in that subtopic or not. Of the 9 total interventions, 5 are clearly getting the biggest results.
By focusing on the KPI, individual student growth outcomes, the school can hone in on opportunity for improvement on their most important metric.
Question 3: Does our RTI system have enough resources?
Finally, the school needs to know whether the RTI system is adequately resourced to meet the ambitious goal it set.
Best practices call for 10 to 15% of students served in Tier 2. In this case, one grade has over 40% of students receiving Tier 2 support.
Further exploration shows that about 30 to 35% of student math time is spent with RTI staff.
The RTI staff is serving more students than recommended, but in this case the school may decide that’s necessary to do to reach growth goals. Next they need to determine if they’re adequately staffed for that strategy.
In each grade the percent of available staff time by interventionists is significantly lower than the percent of student time spent with interventionists. For example, 30% of 6th grade math hours are spent with the interventionists, who are only 25% of 6th grade math staff. This is a particularly important metric, because interventionist hours should be dedicated to intensive, personalized support.
To meet the individualized growth goals they’ve set, the school will need to decide how to balance resources with needs. For example, will they shift math teachers into interventionists roles and train them accordingly? Will they change direct instruction practices so more student math hours are spent with primary instructors? Will they offer primary instructors professional development on math differentiation?
Additional investigation into the direct instruction model could explore:
- Which teachers are seeing the most growth in particular subtopics?
- Which student subgroups are seeing the least growth in mathematics?
- Are there correlations between student subgroups and growth?
The important thing to remember here is the indicator: individual student growth. Every question you will ask is in service of understanding “how will this impact individual student growth on benchmark assessments?”
Through focusing on the Key Performance Indicator, the leadership has honed in on a few focus areas for improvement as well as the data they’ll focus on to monitor their success.
|RTI Program||% of students making sufficient annual progress to be proficient by 10th grade||State Exams|
|Individual Interventions||% of students making sufficient progress on benchmark assessments by subtopic performance||Benchmark Assessments|
|RTI Resource Allocation||% of student math time spent with interventionists||SIS, Staff Time Sheets|
Table 3: Sample Progress Monitoring Plan
The next step for this team will be to research and implement strategies that can address their challenges, but by knowing what they’re monitoring and why, they have significantly improved their chances of success.
So, Are You on the Right Track?
As we’ve seen, there is no single answer to the question “are we on the right track?” That’s because there’s no single answer to the question “this is what you should be aiming for.” Even in highly regulated public school system, every school is different and the goals you set should be different, too.
Taming the Data Monster doesn’t require a magic flute or a black belt in Excel. You simply need to treat data as a resource at your service.
When we ask people who’ve been through our data boot camp “what’s the most important thing you learned?” they almost universally tell us “it was to ask a good question and leave the data in the drawer until it was needed.”
Instead of asking “does the data say we’re doing a good job?” ask “How are we doing in accomplishing our goal?” Take the time to articulate exactly what that goal is and how you’re going to know when you’ve achieved it. Assess your performance against your goal from multiple angles. Make a plan to address any challenges you identify along the way. And importantly, lay out your monitoring plan, so you know when and how you’ll look to see how well you’re doing in meeting your goal.
You’re now in control of your data and your destination.
Want to learn more? Join us on July 17th at Data Empowerment Boot Camp to learn the practical protocols for transforming your vision into action.