Evaluators on Autopilot: Is Data Crunching Drowning Out Educator Insight? Or Beyond the Numbers: Is It Time to Rethink Data-Driven Decisions in Education?

 

The real power of data in education isn’t fully realized when it’s just about collecting scores and ticking off assessments. I spent a good portion of my time in my doctoral program focused on my emphasis—Assessment and Evaluation. Though often used interchangeably by educators, these terms have different, more than just nuanced, definitions and roles. I believe that since these two terms are often considered synonyms, the depth of knowledge and understanding of the term “evaluation” has basically been lost. The term “data-driven decisions” is ubiquitous in education and often a hallmark of a skilled educator. Data’s real worth, however, comes from the questions it makes us ask, not just the answers it provides. Research by RAND emphasizes this point: “having data alone does not guarantee effective decision-making” (Marsh, Pane, & Hamilton, 2006, paraphrased). This observation underscores the importance of advancing beyond mere data collection to a deeper, more strategic use of data to genuinely improve teaching and learning.

Data is only the initial part of the story; the adage “Data rich, information poor” is a clear way to demonstrate what I see as the current practice and the current problem in education. Educators must develop the skills, knowledge, and experience to move from data to evaluation—moving from the “what” to the “why” and into the realm of “how.”  

• The “what” helps us define our current reality. 
• The “why” defines our current practices that got us to where we now are. 
• The “how” helps us to plan for the next steps to help us achieve the next loop in our planning and goals. 

Being able to explain the “why” and “how” is the essence of evaluation and the key to continuous, systemic change and improvement. 

There’s significant opportunity to shift from accumulating data points to interpreting this data to foster better outcomes. Much like watching a pot won’t make it boil any faster, merely testing students more frequently without altering teaching approaches doesn’t improve their learning. It’s the heat, not the watching, that brings the pot to a boil! Through academic and professional experiences, I’ve seen too many instances where data is collected meticulously but utilized minimally. Educators often feel swamped by the sheer amount of data or are stuck in old ways of looking at it, which prevents them from using it creatively. By changing how we approach this wealth of information, we can use data as a launchpad for deeper understanding and more effective teaching strategies.

In the classroom, the practice of assessment often fills our lesson plans, serving as the foundation for understanding student performance and achievement. It is the information we use to help us determine next steps and to serve as a marker of proficiency. Yet, the true transformative impact in teaching extends well beyond mere collection and exploration of data; it lies in the art of evaluation - turning raw data into actionable, impactful strategies.

Assessment often marks the completion of a learning cycle, offering a snapshot of student achievements at a particular point. While it highlights what students have learned, it seldom provides insights into how teaching methods or curriculum can be refined to improve further learning outcomes. Taking the next step towards evaluation, however, extends this process by using the data to craft a strategic roadmap, aimed at make the necessary classroom changes to increase student access to proficiency. 

Consider a typical scenario: after a unit on fractions, a teacher discovers that a significant number of students are struggling. Traditional assessment would end at quantifying this struggle through assessment data. Evaluation, by contrast, digs deeper and provides further insight. It seeks to uncover why students found fractions challenging and what the next steps could be to increasing more positive outcomes.

Another problem with being solely data-minded is that there is a bigger story being told beyond the numbers. Evaluation helps us extend beyond just the tangible data. It also involves the intangible elements that contribute to the atmosphere and spirit of a learning environment. Smith and Smith (1966) highlighted this with the idea that appraisal inevitably depends to some extent on the beholders themselves. Therefore, evaluation must consider both measurable outcomes and the subtler signs of understanding, engagement, and the overall classroom climate. It also must include teacher perspectives, including the philosophies and understandings of the context of the data. 

Expanding our view, let’s consider another scenario involving a science class learning about ecosystems. The end-of-unit test results show mediocre comprehension. Here, evaluation steps in to question the efficacy of the teaching tools and styles employed—were the sessions too theoretical, lacking in hands-on experiments that connect students with real-world ecological interactions? An evaluative approach might suggest more fieldwork or simulation-based learning to enhance conceptual grasp. It allows the teacher to paint a clearer picture of what needs to happen next. 

For teachers, embracing evaluation over mere assessment can lead to significant changes in the classroom dynamics. It challenges educators to not only recognize areas of student difficulties but also to innovate and adapt their teaching strategies to meet these challenges effectively. It places the data discussion into the realm of variables within a teacher’s control and away from the oft-used complaint about poor results centered around, “my students just didn’t try hard enough.” It’s also important to note that "Dialogue in evaluation places the responsibility for evaluation squarely on the educators and other participants in the setting" (Jeffs and Smith, 2005, paraphrased). This dialogue is essential for turning assessments into comprehensive educational evaluations that benefit all stakeholders in the learning process.

Consider These Strategies

To harness the full potential of evaluation, educators might consider several strategies:

• Use assessment data as a starting point, not the finish line: Transform assessment data into a foundation for deeper investigation and targeted action. Consider the broader implications of students' performances and look for patterns that suggest areas for pedagogical improvement.
• Engage in regular reflection and adjustment of teaching strategies based on data: Incorporate a routine of reflecting on what teaching methods are working and which are not, using the data as a guide. Adjustments could range from tweaking lesson plans to overhauling assessment methods to better capture student learning.
• Encourage a culture where data is seen as a tool for growth, not just a measure of performance: Foster an educational environment where data is viewed as a constructive tool for enhancing teaching efficacy and student learning, not merely as a metric of achievement.
• Transform PLCs into Professional Improvement Communities (PICs) where evaluation drives collaboration and planning: Shift the focus from data as the conversation-driver to evaluation as the catalyst for improvement. In these communities, educators would use evaluation not just to assess what has been learned but to actively shape how teaching and learning occur. This could involve collaborative evaluations of teaching methods, student feedback sessions, and strategic planning that all aim to continuously refine educational practices.

Commission for Educators

- This month, take a moment to reflect on the story your data is telling. Choose one class or subject area and dig deeper into the evaluations—not just the scores. Create a small action plan based on your findings, focusing on one change that could enhance understanding or engagement. Share your plan and its impacts at your next team meeting.

- Partner with a fellow educator to observe and evaluate each other's assessment methods. Discuss the outcomes together and brainstorm ways to shift from merely collecting data to making impactful educational decisions based on that data. Implement one strategy from this collaboration over the next term, and observe the difference it makes in your students' learning experiences.

By prioritizing evaluation alongside assessment, we strengthen our abilities to make informed decisions that lead to better teaching and learning experiences. We shouldn’t settle for data collection alone. We need to build a system that leverages evaluation to support meaningful, lasting improvements.

References

Jeffs, T., & Smith, M. K. (2005). Informal education: Conversation, democracy and learning. Educational Heretics Press.

Marsh, J. A., Pane, J. F., & Hamilton, L. S. (2006). Making sense of data-driven decision making in education: Evidence from recent RAND research. RAND Corporation.

Smith, H. W., & Smith, M. J. (1966). The dynamics of educational evaluation. Academic Press.