How Valid and Reliable Are Social Emotional Learning Surveys?

Over 1,000 schools use Panorama Education's research-backed social emotional learning survey to measure and improve their students’ social-emotional skills and competencies. But how valid and reliable are SEL assessments for students? 

Using data from the 2015-2016 school year, we’ve published a preliminary report on the reliability and validity of Panorama’s social-emotional learning measures. After outlining how the measures are designed, the report presents correlations between SEL measures with other student data and shares new evidence of the reliability and validity of the measures.

Panorama offers 22 measures of social-emotional learning—including grit, growth mindset, and self-management—that draw on and relate to the broad domains of social relationships, motivation, and self-regulation. Psychology and education literature, outlined in the preliminary report, has shown that social relationships, motivation, and self-regulation consistently produce outsized impacts on student outcomes. The 22 measures are customizable, so educators can decide which subset of measures are most important to their students and teachers.

Learn how districts are measuring students' SEL skills in our free guide for school and district leaders.

Each measure consists of several questions. Each item is worded and structured in accordance with modern best practices in the science of survey design. However, this step alone does not insure high quality measures. We regularly test our items and how well they function together as a holistic scale to measure the underlying aspect of social-emotional learning that they are supposed to measure.

Testing Panorama's Social Emotional Learning Survey for Reliability and Validity

To provide initial evidence of the reliability and validity of Panorama's social emotional learning survey measures, we analyzed data from three school districts on seven measures. Each measure contained between five and 10 questions. These schools were measuring student SEL for formative purposes only — they were not evaluating students or teachers based on the results. 

Reliability of the measures speaks to how well the individual items, or questions, measure the same underlying concept. Our measures are reliable, with an average Cronbach alpha coefficient of .78 and minimum of .68.

Validity assesses how well the measures get at the underlying concept they are supposed to address. We looked at validity two ways. First, we found that the measures correlate with each other as expected. Similar measures (e.g. sense of belonging and teacher-student relationships) show a higher correlation than less related constructs (e.g. sense of belonging and grit). The table shows the SEL intercorrelations, with the shading and numbers indicating the size of the (Spearman) correlations.

Correlations of social-emotional learning measures

Next, we looked to other student data, including student GPA, absences, and behavioral referrals, to be sure they correlate with SEL measures as we expected. Indeed, we found that correlations work as we would predict. For example, there is a relatively strong positive correlation (r = .40) between the self-efficacy SEL measure and GPA.

What's Next for Social Emotional Learning Measures

Our report finds that Panorama’s social emotional learning survey — administered for formative purposes — exhibits the psychometric properties of good instruments: reliability and validity. We invite you to read the preliminary report to learn more about the design of the measures and to read the full explanation of the tests of reliability and validity. As we continue to collect, analyze, and interpret SEL data, we will update and expand the report accordingly.