Data representation in Achievement Reports

Amidst the flurry of op-ed pieces about BC's annual standardized testing for students in Grades four and seven, I felt my nerves fray at this quote from BC's Minister of Education:

The FSA helps parents and educators determine the students and schools in need of extra support. The results complement the assessment and reporting by teachers and help school districts direct resources to where they are needed most. Additionally, it helps the Ministry of Education identify trends to improve teaching and learning for the coming school year.
...
I believe that one of strengths of the FSA is that by sampling every child we are able to get a comprehensive picture of how students are doing.

Using random samples for the FSA would make it very difficult to get the level of detail we obtain from all our students including aboriginal students, English-language learners and students with special needs. It would not provide the individual, school and district level information critical for both student intervention and long-term planning.

Read more: http://www.vancouversun.com/news/helps+ensure+resources+directed+where+they+needed/7815610/story.html#ixzz2IuZ6wlu3

Hmmm. Let's see. The provincial framework for school accountability is the Achievement Contract. Each District develops a contract which outlines how the District will address learning challenges it identifies. Then, in January, each Superintendent files a report on how well the District is meeting its goals. Given the comments above, one would assume that FSA results are a primary source for objectively identifying issues at the school and district level. Is this true?

Have a look at the recent report from Delta. http://web.deltasd.bc.ca/sites/files/Superintendent%27s%20Annual%20Report%20on%20Student%20Achievement-2012.pdf The FSA test does not appear once in the report. This District chose instead to use report card marks and six year completion statistics. Perhaps that is because, in Delta's case, the FSA results are remarkably uninteresting. Here is a graph from the results published on the Ministry web site:



Each of the FSA graphs look remarkably similar. Take a look yourself to see all six FSA results: http://www.bced.gov.bc.ca/reports/pdfs/fsa/037.pdf. Yet here is a quote from the local news report based on the Achievement report:

Students in the Delta School District are performing better than they were five years ago, according to the district’s annual student achievement contract. (http://www.southdeltaleader.com/news/187337201.html)

One can only assume that in this case, the FSA results did not "help inform" the district about achievement rates.

Another interesting example is Greater Victoria. Here is a link to the Achievement report (attached to a recent Board meeting agenda): http://www.sd61.bc.ca/super/pdf/agendas/12_13/2013_01_21.pdf

In this report, some FSA data is used, although interestingly, it is pretty selective.

The report begins with the title: What is improving? And lists several areas:

• Dogwood completion rate for all students
• Dogwood completion rate for aboriginal students
• FSA results for grade 7 aboriginal students in reading, writing and numeracy
• FSA results for grade 4 aboriginal students in numeracy
• FSA results for grade 4 students in writing and numeracy
• FSA results for grade 7 students in reading, writing and numeracy

Given this list, one might expect to find similar list items in the section entitled: Challenging areas - which trends in student achievement are of concert to you? In particular, I would expect to see FSA results for grade 4 aboriginal students in writing and numeracy, and FSA results for grade 4 students in reading. But these don't appear. Instead, this section reads:

"Our data indicates an upward trend in student achievement. However, despite the 19.9% improvement in completion rate for aboriginal students and the 7% improvement in completion rates for all students since 2008, we continue to be acutely focused on, and determined to improve, our Dogwood completion rate for each and every student."

There is no mention of the areas lacking improvement.

Here are the grade 4 FSA results graphs produced by the Ministry:

I'm not sure I would use the words "upward trend" to describe this data. Before making any claims, I would want to know if whatever upward movement in numbers there is was statistically significant.

Instead of using these charts, the report includes the following text:

The first thing I notice when I looked at this list, is that for each bullet, a different date range is used. Or, put another way, certain data is excluded from the analysis. Given that the Ministry provides five year data for comparison, why would some years, for some grades/subjects, be left out? What was the criteria for omitting them?

Another example is the reporting of graduation rates in this chart:

Here is another chart, from the previously published Achievement Contract (http://www.bced.gov.bc.ca/schools/sdinfo/acc_contracts/2013/61.pdf):

Looking at these two charts, I wonder why the year 2008 was selected as the start year for the data chart demonstrating "improvement" for all students?

Greater Victoria also was in the news after this report was brought to a Board Meeting. Kudos to Lindsay Kines, who reported that the graduation rates increased from 76%, the previous high, in 04/05, to 78%, the current number. This is a very different way of describing the data than the chart showing the changes beginning from the low point in 2008. (http://www.timescolonist.com/news/local/graduation-rate-reaches-new-high-in-greater-victoria-school-district-1.51362)

I have to give the Ministry some credit. At least their charts and graphs all use the same date ranges, and includes the same information for every District. They are not picking and choosing which data to publish. But the scheme for "contracting" measurable improvements and then justifying performance with numbers appears to me to be a shell game. I wonder what "long term planning" the Minister imagines is happening? What objective information can anyone get from either the selective reporting from the Districts or the plethora of un-analysed data from the Ministry? What "comprehensive picture" does the Minister see here?

Educators and Enbridge

There was some "controversy" earlier this year when the media (main stream) picked up a story about the BCTF producing teaching materials about the Enbridge pipeline. I use quotes because I don't really understand where the conflict in the story came from. I was interviewed on the day of the large rally in Victoria organized by Defend Our Coast and asked if I thought it was apppropriate for the BCTF to produce materials with a point of view. Given that corporations and interest groups routinely produce teaching materials that are biased, I see no problem so long as the teacher uses a variety of sources, and this is precisely what the BCTF intended and even recommended. So where was the controversy?

Like many British Columbians, many teachers and educators are very concerned and aware of the issues related to the pipeline, tankers, the tar sands and climate change.  I noticed that at the Victoria Enbridge hearings, two recently retired teachers from the Victoria school district and one Victoria School Trustee (Diane McNally) gave presentations expressing their opposition to the project. Former teacher Janet Simpson expressed her exasperation with a hearing process so limited in producing any thorough or meaningful exploration of ideas:

"...what I have is 10 minutes, only ten minutes to try to express the magnitude of what we are going to lose.

10 minutes to attempt in vain to stop us from careening down a path that will result in 10 decades of environmental devastation. 

10 minutes to urge you to look beyond the short-sighted false economy that the tar sands and its related projects are creating.

10 minutes to ask why we have learned nothing about value-added resource extraction from our habit of selling raw logs.

10 minutes to point out what an incredible insult this is to every First Nation.

10 minutes to stress the value inherent in a fossil fuel and the folly in squandering it all in such a destructive way.

10 minutes to question why we should all pay such a huge price for Alberta’s diminishing prospects for sales of its polluting tar sand oil in the U.S.

10 minutes to remind you of the two major earthquakes within a recent month span and the effect of another one and its accompanying tsunami on supertankers moving down Kitimat inlet.  How prepared can we possibly be for the chaos that would result if simultaneous disasters occurred?  How much attention would one pipeline break gushing millions of gallons of oil into a remote, pristine river system get?"

Beyond viewing these issues as political/environmental concerns, teachers also take a pedagogical interest in the natural environment and teaching and learning. I was very interested to read this fall about a new Kindergarden program in Sooke based almost wholly in the outdoors, and the similar program in the Gulf Islands. World-wide, there is an educational movement stressing the importance of children just simply spending time in nature and organizing curricula around one's immediate environment.  Research indicates that there are a myriad of potential benefits, including behavioural, academic and simply an appreciation for our natural surroundings and our collective responsibility as stewards of the earth. (See, for example, http://www.psychologytoday.com/blog/people-in-nature/200809/how-nature-can-transform-education.)

The Enbridge hearings are a very teachable moment - for students, for schools, and for all of society.