Amidst the flurry of op-ed pieces about BC's annual standardized testing for students in Grades four and seven, I felt my nerves fray at this quote from BC's Minister of Education:
The FSA helps parents and educators determine the students and schools in need of extra support. The results complement the assessment and reporting by teachers and help school districts direct resources to where they are needed most. Additionally, it helps the Ministry of Education identify trends to improve teaching and learning for the coming school year.
...
I believe that one of strengths of the FSA is that by sampling every child we are able to get a comprehensive picture of how students are doing.
Using random samples for the FSA would make it very difficult to get the level of detail we obtain from all our students including aboriginal students, English-language learners and students with special needs. It would not provide the individual, school and district level information critical for both student intervention and long-term planning.
Hmmm. Let's see. The provincial framework for school accountability is the Achievement Contract. Each District develops a contract which outlines how the District will address learning challenges it identifies. Then, in January, each Superintendent files a report on how well the District is meeting its goals. Given the comments above, one would assume that FSA results are a primary source for objectively identifying issues at the school and district level. Is this true?
Have a look at the recent report from Delta. http://web.deltasd.bc.ca/sites/files/Superintendent%27s%20Annual%20Report%20on%20Student%20Achievement-2012.pdf The FSA test does not appear once in the report. This District chose instead to use report card marks and six year completion statistics. Perhaps that is because, in Delta's case, the FSA results are remarkably uninteresting. Here is a graph from the results published on the Ministry web site:
Each of the FSA graphs look remarkably similar. Take a look yourself to see all six FSA results: http://www.bced.gov.bc.ca/reports/pdfs/fsa/037.pdf. Yet here is a quote from the local news report based on the Achievement report:
Students in the Delta School District are performing better than they were five years ago, according to the district’s annual student achievement contract. (http://www.southdeltaleader.com/news/187337201.html)
One can only assume that in this case, the FSA results did not "help inform" the district about achievement rates.
Another interesting example is Greater Victoria. Here is a link to the Achievement report (attached to a recent Board meeting agenda): http://www.sd61.bc.ca/super/pdf/agendas/12_13/2013_01_21.pdf
In this report, some FSA data is used, although interestingly, it is pretty selective.
The report begins with the title: What is improving? And lists several areas:
"Our data indicates an upward trend in student achievement. However, despite the 19.9% improvement in completion rate for aboriginal students and the 7% improvement in completion rates for all students since 2008, we continue to be acutely focused on, and determined to improve, our Dogwood completion rate for each and every student."
There is no mention of the areas lacking improvement.
Here are the grade 4 FSA results graphs produced by the Ministry:
I'm not sure I would use the words "upward trend" to describe this data. Before making any claims, I would want to know if whatever upward movement in numbers there is was statistically significant.
Instead of using these charts, the report includes the following text:
The FSA helps parents and educators determine the students and schools in need of extra support. The results complement the assessment and reporting by teachers and help school districts direct resources to where they are needed most. Additionally, it helps the Ministry of Education identify trends to improve teaching and learning for the coming school year.
...
I believe that one of strengths of the FSA is that by sampling every child we are able to get a comprehensive picture of how students are doing.
Using random samples for the FSA would make it very difficult to get the level of detail we obtain from all our students including aboriginal students, English-language learners and students with special needs. It would not provide the individual, school and district level information critical for both student intervention and long-term planning.
Hmmm. Let's see. The provincial framework for school accountability is the Achievement Contract. Each District develops a contract which outlines how the District will address learning challenges it identifies. Then, in January, each Superintendent files a report on how well the District is meeting its goals. Given the comments above, one would assume that FSA results are a primary source for objectively identifying issues at the school and district level. Is this true?
Have a look at the recent report from Delta. http://web.deltasd.bc.ca/sites/files/Superintendent%27s%20Annual%20Report%20on%20Student%20Achievement-2012.pdf The FSA test does not appear once in the report. This District chose instead to use report card marks and six year completion statistics. Perhaps that is because, in Delta's case, the FSA results are remarkably uninteresting. Here is a graph from the results published on the Ministry web site:
Each of the FSA graphs look remarkably similar. Take a look yourself to see all six FSA results: http://www.bced.gov.bc.ca/reports/pdfs/fsa/037.pdf. Yet here is a quote from the local news report based on the Achievement report:
Students in the Delta School District are performing better than they were five years ago, according to the district’s annual student achievement contract. (http://www.southdeltaleader.com/news/187337201.html)
One can only assume that in this case, the FSA results did not "help inform" the district about achievement rates.
Another interesting example is Greater Victoria. Here is a link to the Achievement report (attached to a recent Board meeting agenda): http://www.sd61.bc.ca/super/pdf/agendas/12_13/2013_01_21.pdf
In this report, some FSA data is used, although interestingly, it is pretty selective.
The report begins with the title: What is improving? And lists several areas:
- Dogwood completion rate for all students
- Dogwood completion rate for aboriginal students
- FSA results for grade 7 aboriginal students in reading, writing and numeracy
- FSA results for grade 4 aboriginal students in numeracy
- FSA results for grade 4 students in writing and numeracy
- FSA results for grade 7 students in reading, writing and numeracy
"Our data indicates an upward trend in student achievement. However, despite the 19.9% improvement in completion rate for aboriginal students and the 7% improvement in completion rates for all students since 2008, we continue to be acutely focused on, and determined to improve, our Dogwood completion rate for each and every student."
There is no mention of the areas lacking improvement.
Here are the grade 4 FSA results graphs produced by the Ministry:
Instead of using these charts, the report includes the following text:
The first thing I notice when I looked at this list, is that for each bullet, a different date range is used. Or, put another way, certain data is excluded from the analysis. Given that the Ministry provides five year data for comparison, why would some years, for some grades/subjects, be left out? What was the criteria for omitting them?
Another example is the reporting of graduation rates in this chart:
Here is another chart, from the previously published Achievement Contract (http://www.bced.gov.bc.ca/schools/sdinfo/acc_contracts/2013/61.pdf):
Looking at these two charts, I wonder why the year 2008 was selected as the start year for the data chart demonstrating "improvement" for all students?
Greater Victoria also was in the news after this report was brought to a Board Meeting. Kudos to Lindsay Kines, who reported that the graduation rates increased from 76%, the previous high, in 04/05, to 78%, the current number. This is a very different way of describing the data than the chart showing the changes beginning from the low point in 2008. (http://www.timescolonist.com/news/local/graduation-rate-reaches-new-high-in-greater-victoria-school-district-1.51362)
I have to give the Ministry some credit. At least their charts and graphs all use the same date ranges, and includes the same information for every District. They are not picking and choosing which data to publish. But the scheme for "contracting" measurable improvements and then justifying performance with numbers appears to me to be a shell game. I wonder what "long term planning" the Minister imagines is happening? What objective information can anyone get from either the selective reporting from the Districts or the plethora of un-analysed data from the Ministry? What "comprehensive picture" does the Minister see here?
Another example is the reporting of graduation rates in this chart:
Here is another chart, from the previously published Achievement Contract (http://www.bced.gov.bc.ca/schools/sdinfo/acc_contracts/2013/61.pdf):
Looking at these two charts, I wonder why the year 2008 was selected as the start year for the data chart demonstrating "improvement" for all students?
Greater Victoria also was in the news after this report was brought to a Board Meeting. Kudos to Lindsay Kines, who reported that the graduation rates increased from 76%, the previous high, in 04/05, to 78%, the current number. This is a very different way of describing the data than the chart showing the changes beginning from the low point in 2008. (http://www.timescolonist.com/news/local/graduation-rate-reaches-new-high-in-greater-victoria-school-district-1.51362)
I have to give the Ministry some credit. At least their charts and graphs all use the same date ranges, and includes the same information for every District. They are not picking and choosing which data to publish. But the scheme for "contracting" measurable improvements and then justifying performance with numbers appears to me to be a shell game. I wonder what "long term planning" the Minister imagines is happening? What objective information can anyone get from either the selective reporting from the Districts or the plethora of un-analysed data from the Ministry? What "comprehensive picture" does the Minister see here?
Do you know what year we started including Grade 10 marks into the Graduation Plan? It seems to me, that rates started "improving" once the bar was lowered.
ReplyDelete