[Andrew Field]

I'm rather bewildered by this week's news. For the past four years the new compulsory ICT test for all KS3 students has been pushed, promoted and mentioned at any and every moment. I was pleased to get my school into the pilot for its second year and we have run the test for the past three years. (I think!)

To cut a huge story short, the software was built by RM and promised great things. Until last year it crashed unreliably. Last year we ran the pilot tests in our state of the art ICT suite with 50 students at a time. Still we had problems - we had to use an entire morning for each set of students. The two tests each lasted 50 minutes, but at the end of the first one the transmission of data took about 20 minutes. Thus after that we've purchased a new server to run the final round of pilot tests on this year.

I really like the concept of the tests, but hate the implementation. However, I have persevered with the tests - and the technicians at school must have spent well over a hundred hours getting the tests setup. This was enormous hassle, but we kept pushing things on believing that the compulsory tests in 2008 were coming whatever happened, so we had to get organised.

The key issue was that teacher assessment for ICT was no longer going to be used and that instead the test results would be forwarded automatically and become the fourth SAT exam. I have had grave concerns about the validity of the tests hearing terrible reports of schools with 80%+ Level 5+ being reported that their students are all at Level 4. However, we were told issues with the tests were being ironed out and that was the point of pilots. As a school we had just had immense investment in ICT facilities and still we had to purchase more hardware to get the tests running. I was staggered and wondering what kind of spending would be required across the country to make the test happen.

This week, on the way into school I heard on the radio that the QCA are recommending that the test is not compulsory at all.

What?

I was utterly amazed by this. Having looked further, the QCA minutes from their meeting in early November 2006 do indeed say exactly this:

http://www.qca.org.uk/downloads/qca_item_a...vember_2006.pdf [page 5]

Quote

Key Stage 3 ICT test

10. Managing Director of the NAA, David Gee, reported that, since assuming control of the
programme in April 2006, NAA has implemented new governance and control
mechanisms, and has made good progress towards addressing the concerns raised by
OGC.

10.1 David Gee recommended to the Board that the focus of the KS3 ICT test should
change. The DfES has new Ministers and senior officials, who are currently questioning
whether this test should be rolled out as a statutory test in 2008. David Gee
recommended that the test should be used as an integral and standardised tool for
informing teacher assessment, where it would contribute as a component of the overall
mark, rather than making it a full statutory test with results included in national performance tables.

David Gee recommended to the Board that this approach be the
formal advice of the QCA to the DfES.

10.2 The Board discussed the recommendation and agreed that it was not necessary to
burden schools with an additional statutory test, considering that ICT was something
that should be embedded into other subjects. The Board felt that the work that had
been undertaken should not be lost and agreed that the test should be used to inform
teacher assessment.

10.3 The Board endorsed the recommendation that a formal advice letter should be sent to
the Secretary of State, recommending that the KS3 ICT test not be made statutory.

In some ways this appears to be a fantastic turn around having listened to teachers. If we were to have e-Assessment tools that help us reach conculsions about students' ability that would be marvellous. Yet what on earth has been going on here? What, indeed is going to happen? Millions of pounds has been spent on the development of the test and I'm sure millions of pounds has been spent in schools upgrading and developing equipment in preparation for the test.

There are some incredibly serious issues to consider here. Don't get me wrong, I'm delighted that there appears to be a change in policy, but I await with interest what will be announced next week at the BETT show. This has impliciations for everyone who teaches ICT and much wider implications for the way ICT is taught.

In the QCA minutes it also mentions how ICT should be embedded in other subjects. It should - we tried this several years ago, but discovered that when the students aren't taught specific skills it falls to the individual subject teachers to teach the skills and thus ICT takes over their lesson, making it ICT first and their subject second. What should happen is that schools teach ICT capability within ICT lessons and then students apply their skills in their subjects.

More about this in an excellent thread on the TES forums:
http://www.tes.co.uk/section/staffroom/thr...p;messagePage=1
Plus a BBC news report:
http://news.bbc.co.u...ion/6232207.stm
Grumpy Old Teacher blog:
http://ict-grump.blogspot.com/2007/01/chan...head.html#links

... but I will be very surprised if there isn't a huge furore about all this. Wasted money, poor planning, experts working in opposite directions towards different goals? I suppose the only thing we could say here is that finally ICT in schools is becoming more and more like the real business world.

More news is expected at BETT.

[Andrew Field]

Also - via posters on the TES forum - the National Assessment Agency have posted a response on their website:
http://www.naa.org.uk/naaks3/327.asp

Quote

05 January 2007

Following discussion at the Qualifications and Curriculum Authority (QCA) Board meeting in November 2006, QCA sent advice to the Secretary of State about the development of the KS3 ICT test. This advice stressed the importance of this test as a tool to inform teaching and learning. Any decision about changes to the development programme will be announced by the DfES in due course.

The KS3 ICT test pilots have been very successful and the majority of schools are now participating. The NAA will continue to communicate to schools about developments through the regular KS3 ICT Countdown newsletter. The next issue will be mailed to schools in mid January.

The pilots have been extremely successful in developing and evaluating a highly innovative test and have contributed greatly to improving our understanding of e-assessment methods and the challenges schools face in implementing e-assessment. Currently over 90% of schools are getting ready to take part in the 2007 full national pilot and the NAA encourages all schools to carry on with their preparations for the 2007 pilot.

This is, quite frankly, a disgusting statement. It makes no mention of any issues that schools have been having. I know from my own experience that my school actually had things very easy. I've been told that we did the most tests of any school in the county. So if we had enormous trouble and run the most successful pilot (in terms of number of tests delivered) what about all those who must have struggled even more.

The statement also seems to be quite carefully worded to justify the NAA's position. I really do wonder what is really going on here. Is this all just to soften the blow when it is announced that the test will be optional, or - more likely - that the test will be compulsory but the results will also take account of teacher assessments too? I just don't know.

Obviously the teachers who are more annoyed by the whole thing are those most likely to post on the TES forums, but there has been such a groundswell of anger and annoyance about this test I simply cannot see how the NAA can release such as positive statement like this. It is bewildering. The above statement just isn't true at all.

[Andrew Field]

Unbelievable.  dfes_letter_10_jan_07.pdf (19.19K)
Number of downloads: 13

Quote

ICT TEST AT KEY STAGE 3

Jim Knight will announce today at the BETT Show 2007 a new focus for the ICT test, moving from an end of key stage summative assessment to an assessment that will provide formative information to support individual progression.

Following advice from the QCA, Ministers have decided not to put the test on a statutory footing in 2008. Piloting will continue in 2007 and as the test is developed further there will be a new focus on assessment materials that can be used more flexibly throughout the key stage. These materials will provide formative feedback that teachers and pupils can use to allow each pupil to progress and inform teaching.

ICT is one of the necessary life skills for the 21st century and we want every child to have the capability to live, learn and work successfully in today’s IT mediated world. That is why it is the subject chosen to pioneer this new approach to testing for progression.
The majority of schools will have taken part in the summative test pilots and this experience will be extremely valuable in familiarising pupils and teachers with onscreen assessment.

I would encourage every school to take part in the summer pilot to support the further development of the test. The National Assessment Agency will continue to keep schools updated on the 2007 pilot through the KS3 ICT Countdown newsletter. The next issue will be issued in mid January.

I would like to thank you for all the hard work schools have put in which has made the pilots so valuable.

Related BBC news story: http://news.bbc.co.u...ion/6247939.stm

[Andrew Field]

Depending on the way the new version of the test turns out, this could actually become a fantastic tool. If we can use the tool for end of level assessments, providing mini-assessment exercises to help students prove that they have understood a topic and can produce work at a particular level it will then be fantastic.

Yet surely this should have been the vision since the beginning? I have no idea about the way the test has actually been put together, but I've always had concerns when waiting for the test data to be sent back to the server. It always seemed to take absolutely ages. I simply couldn't work out why so much data had to be transferred.

If it is possible to adapt the test so it can be used in this new way, it will be a great additional tool for us all to use. Yet I do seriously wonder how this will be possible. Having looked at the test it doesn't appear to have been built so it is modular in nature with parts that can slot in and out as required.

Interesting times!

[Andrew Field]

Very good article on the Register about the tests and the wider implications:
http://www.theregist...t_school_tests/

[Andrew Field]

Also see the Chalkface blog: http://www.chalkface...TS_RIP_213.html

I felt really sorry for the representatives on the QCA stand at the BETT show - they had clearly been answering questions about the QCA ICT test the entire time. Those responsible - the chief developers and directors of RM should have been there to face the music. I did leave the stand feeling slightly less negative about the whole thing, but was amazed that they still think 90% of schools will run the tests this year.

I feel that if a school can run them without any problems, they probably will. However, the first time they experience network slowdown, interface issues, crashes or admin problems they will just not bother. It was promised the test this year will be much better, so I think we will still give it a go. The jury is most certainly out though - in fact I think they've gone on a long haul flight.

[Andrew Field]

The other main issues here are the implications for ICT and computer-based assessment. With the statutory ICT test, schools were bound to focus on ICT as they knew it was the fourth core subject. Now, no matter how fantastically switched on a school is to ICT they will naturally not see ICT in the same 'light' as English, Maths and Science.

According to the National Strategy we have English, Maths, Science, ICT and the Foundation subjects. The issue now is to ensure that ICT is still seen as important, that investment in ICT resources has a widespread impact on the whole school community and - most importantly - how ICT is given its key place at the heart of the curriculum. The recent announcements about the ICT test have caused some serious heart surgery.

It seriously weakens the case for computer-based assessment. I suppose what is now vital is for the issue to be examined in depth - why did the tests not work well? The concept was excellent - a computer-based test that would provide an indication at what level each student was working at. However, the implementation was flawed. The non-standard interface, the immense technical demands and the bloated nature of the software made it an enormous millstone around schools' necks.

I am pleased that the test has been made non-statutory. However, I am very concerned that the real reasons for its failure will not be recognised. It was poor implementation rather than a poor concept. Why, for example, were ICT subject advisors not involved from the beginning? These poor souls have been the ones who have been beating the drum about the test, encouraging schools to get involved. Now they are the ones who have to go into schools and pick up the pieces.

What a mess this whole thing is

[Andrew Field]

Update today:
http://news.bbc.co.u...ion/6734233.stm

I've copied the whole article below too - best point is the last comment.

Quote

A computer exam intended to be taken by all England's 14-year-olds would have involved "a substantial risk", an evaluation by the exams watchdog said.

The Qualifications and Curriculum Authority blames itself and the test's developer, but also cites "tensions" in the government's original remit.
Its report questions the reliability of the results pupils were obtaining.Ministers have already accepted the QCA's advice not to make the Key Stage 3 test compulsory from 2008. Most pupils found the pilot version difficult and half could not complete it in the time available, the QCA said.

The computer-based test involved a series of tasks such as e-mailing, spreadsheet work and data handling. It was innovative and did not use commercial software such as Microsoft's Office suite - which pupils would have preferred. The QCA's internal evaluation, by electronic assessment researcher Andrew Boyle, defends the decision to use non-standard software. The prime contractor was Research Machines (RM), which provides ICT software and services to a host of UK educational institutions. But concerns included the reliability of results when pupils were tested then re-tested, and test results were out of kilter with their teachers' assessments of their ICT abilities. In 6% of cases their results were two whole test levels below their teachers' assessments.

"At the time of writing it is not possible to make a completely watertight argument that the KS3 ICT test provides reliable measurement," Mr Boyle wrote. It would have been "a risky test in a risk-averse environment". He stresses that responsibility for the "state" of the test model lies within the project - with the development agency, the QCA and its National Assessment Agency, and others who approved work or provided advice. But he says "the obligation to develop an innovative e-assessment implicitly required any developer ... to go outside the comfort zone offered by traditional techniques".

There was a "tension" between the two demands of the Department for Education and Skills. It had demanded that the test be robust enough to be used in "high stakes" national testing but also an innovative demonstration of ICT-based assessment. Part of the evaluation involved getting the views of those involved, including 434 pupils in 16 of the schools that conducted pilot tests.

The report says 42% did not understand the instructions, 52% did not have enough time to complete the test and 69% found it difficult or very difficult.
Pupils said they liked using a computer for the test, e-mailing and learning and experimenting with the software. But many, asked what they had liked about it, replied "nothing" or "finishing it".

[Rob W]

Hi Folks

I just think you're forgetting one big issue here... RM sucks - always have done and quite likely, always will. They produce second rate products at massively inflated prices and the ones that suffer are the schools that buy in to their crap! Now it seems even the Government have turned their back on them. But this could have been settled at the start all they had to do was ask any IT Professional that's been around in education for more than 6 years and the majority would have said no way to RM. Suckered again!

[Andy Hulse]

Surely the biggest outcome is that this totally flawed method of 'assessing' yet more students has been recognised for what it actually was - Rubbish