After a long baking time, the survey results on the use of eAssessment in the post 16 education sector, supported by Jisc and the awarding bodies, were released last week. The survey was made available to all post 16 organisations delivering vocational qualifications over a five month period last year and 175 responses were collected.
The survey finds that most organisations are using e-assessment on a day-to-day basis but for many its impact is still limited. e-testing is more embedded and mature than e-portfolios and tracking systems. While many organisations have a strategy or plan for e-assessment, senior managers don’t necessarily see it as a priority. There still seem to be significant barriers and challenges to increased use of eAssessment, with the survey respondents citing a lack of strategy or policy; organisational culture; staff and learner capability; lack of appropriate accommodation and technology resourcing, reliability or support.
Some members of the eCom team give their personal views on this latest report:
Learning Technology Consultant
The report highlights two issues which I frequently observed and reported on at Jisc: the lack of senior management strategic direction/ guidance; and staff confidence. There are two sides to staff confidence: digital literacy, and the confidence that the system will do the job when it is needed. Comments recorded in the report cite aged systems: for example, problematic wifi provision and inappropriate browsers, such as IE8. We should think of having four utilities – gas, water, electric and wifi - standard expectations for most people. Robust, dependable systems encourage people to use them, and the cautious to learn to use them. After all, why learn to use an unreliable system? Why not just rely on what you know always works!
Managers tend to see the cost of a new system as prohibitive, and that’s why there is little support from this level. However, what is often missed is the cost of running the in-situ system. If a proper cost/ benefit comparison of existing paper assessment versus eAssessment is performed, it may well turn out it doesn’t cost that much more, or even costs less in the end. When someone says it’s too expensive, have they considered transport (moving assessors or learners), postage (scripts to exam boards), storage, administration, reporting of results, results compilation, printing (I had some assessments that hit over 100 pages!), marking time and availability? Not to mention Ofsted’s requirement for more robust data management. I know one place that did this, and at the time, found an eportfolio cost around £30 per learner, but paper-based cost over £70 and that was taking paper and printing into consideration, without any other benefits.
VP Business Development
Given the wide potential survey cohort (e.g. 400 FE colleges and 60 UTCs) and lengthy response period, the respondent response is disappointingly small. The survey/research process is questionable and with so many stakeholders involved, a professional researcher should have designed the questionnaire and acted as an overseer. The irony of an e-Assessment survey not utilising professional item writers is confounding.
There is a worrying lack of context with other, more in-depth studies. Following the QCA’s 2004 “Blueprint for eAssessment”, two concurrent studies were performed by Acritas Research for Prometric (with FAB and QCA as stakeholders). They interviewed two-thirds of all regulated awarding bodies in 2004, and 81% in 2005. Freshminds conducted an equivalent, later study for Pearson VUE, involving 86 organisations from an overall list of 450 regulated and non-regulated organisations. They claimed that four of the largest global summative post-16 e-Assessments were UK originated back in 2005! More recently, large data sets were submitted to the Competition and Mergers Authority enquiry into e-Assessment published in January 2016. Could these not have been made available to further inform the picture?
The absence of any questions on accessibility and how e-Assessment is helping learners with accessibility issues is surprising. JISC themselves have pointed out that 13% of all learners have this issue.
The e-Assessment survey has suffered from trying to please too many stakeholders with a poor design, resulting in a weak response and extremely limited insights. We must be careful not to judge the sector or make any policy or strategic decisions based on this limited survey. I encourage the stakeholders to properly scope, fund and engage a full-blooded research project that does the e-Assessment sector justice.
Chief Executive Officer
In 2005 many Awarding Bodies were piloting or deploying high-stakes eAssessment, and it was seen as the way forward. The BECTA report from 2007 predicted the growth of eAssessment. Here we are in 2016 and from the recent Jisc report it seems not a lot has happened on high-stakes assessment in mainstream education in that 10-year period. While in commercial organisations and professional institutes there has been a big push on high stake exams to increase speed and reduce costs of exams, many awarding bodies only offer eAssessment for a small number of awards and accept electronic evidence as proof of success.
So what is delaying the up-take? A general lack of funding in education seems to be the main obstacle. The 2016 report blames a lack of technology in classrooms, no funding for staff development and lack of modular courses and question banks for subjects. Respondents indicated that they were waiting on being provided with a funded solution, rather than taking on the provision for themselves.
Although 70% of the survey respondents say they have used eAssessment, these apply to online learning using formative assessment and ePortfolio, so it is difficult to know from the report about any increase in high-stakes eAssessment testing. It would have been helpful if the report had broken down the percentage uptake into High-Stakes, Formative and ePortfolio adoption. This would have given a clearer steer on where more emphasis is required in the sector.
eCom challenge organisations who are “not yet convinced that e-assessment is going to be as widespread as suggested”. Education Technology is booming in the commercial sector with training aligned to how learners play, work and spend their leisure time. The number of providers of e-Assessment expands on a weekly basis.
It is not enough to recommend more investment as a simple panacea. Staff need to make the case for investment based on lowering costs, enhancing accessibility, increasing efficiency and giving learners better education outcomes.
Lack of funding for hardware could be overcome by a trust-based BYOD policy and supported by existing experienced technical staff. Poor wi-fi can be addressed by insisting on devices and software that work off-line or with minimal bandwidth/ concurrency. For risk-averse management, staff should share the good practice that’s going on at institutions across the world. In the competition for learners, institutions who stick with what they know will cease to exist.
Some awarding bodies are keen to change to a digital world– many have already changed through the noughties, when analogue television was being decommissioned. The apprenticeship levy is going to shake up a lot of outmoded practice and give employers a stern voice to outdated delivery, while embracing those who blend experience with contemporary methods, such as eAssessment.
If you need to make a case for eAssessment in your organisation or have technical or cultural barriers to overcome, our Learning Technology Consultants are happy to discuss what is possible. Call us on 01383 630032 or email us on firstname.lastname@example.org to arrange an exploratory discussion.
Discover how the Institute of Chartered Accountants Scotland have successfully implemented eAssessment.
Read about the benefits the Chartered Bankers Institute have achieved with online assessment.