The 3rd version of the Department of Health’s ETD Standards were published in August this year and clearly state the best practice for evaluation and assessment across all NHS training departments. Furthermore the publication of the government white paper, Equity and Excellence: Liberating the NHS (Department of Health 2010), has resulted in a greater emphasis on improving patient care and value for money.
ePC work with a number of different NHS trusts and foundation trusts to provide solutions for their evaluation needs. From clear and simple form design, allowing trainers to create their own tools to get the answers they need to meet their objectives, through to complete implementation of a system that will capture the data automatically and send information to where it needs to go. The aim of this document is to highlight the changes in attitude toward training evaluation in the NHS and other government departments and subsequently help us to understand their future needs.
Workforce Development for NHS Wales says this:
‘Education and Training should be evaluated as robustly as any activity which uses up precious resources and it is important that a form of measurement is introduced so that not only is the quality of the learning experience measured in some way, but also the impact of the learning experience on the learner…. …with this, more robust, model in mind it would be better to think of evaluation as a form of measurement of effectiveness…
We look at the five levels of maturity laid out by the Department of Health in their ETD standards and how all of these can be met.
Improving Quality Feedback
‘Many people think that evaluation of Education and Training is about checking that “things were okay” during the training. Where evaluation happens it is often characterised by “happy sheets” which usually check with the learners that the conditions were “okay” and how highly the learners rate the facilitator and the training as a whole. In fact, this is only part of what evaluation is about.’ Workforce Development, NHS Wales
When ePC spoke to over 1200 government run departments and organisations in March 2011 about the difficulties they have with their evaluation processes, a point that came up repeatedly with staff was that they spent all their time inputting the data from their evaluation forms, but never did anything with the information. ETD Standard 6 – Evaluation and Assessment states that ‘Results of delegate’s reaction to the training are analysed and reported internally’ to reach stage 2 of 5 in the maturity ladder, this gives us some indication of just how below par some departments are performing currently.
Storage is another factor in the process that has changed enormously in recent years. Paper storage is becoming inconvenient and prone to lost forms and of course lengthy retrieval processes.
In a digital world where so few activities involve writing with a pen and paper, handwriting is becoming messier and harder to read in a lot of cases, especially at the end of a tiring training session when attendees just want to go home. Data inputters can spend inordinate amounts of time deciphering the scrawl on their forms in order to make sense of what is written.
The NHS is a typical organisation where those inputting this data have a number of other responsibilities which have to be prioritised, making the manual keypad work a slow and arduous task when it has to be done.
NHS trainers also find that they are handing out one generic feedback form to every learner regardless of the course they are taking, meaning the data that they are collecting is vague and often difficult to put into any meaningful report, with staff and budget cuts across the board, there just is not time to write multiple feedback forms and keep on top of the extra administration that this would entail. Unfortunately this means that those training departments unable to put together reports for stakeholders are not meeting the required criteria to meet level3 in the maturity ladder for the ETD Standard for Evaluation and Assessment.
Level 5, the top level of maturity states that ‘the training service uses innovative tools and techniques to assess and evaluate all training solutions, taking into account care setting, audience and environment’
A final thought on the importance of saving time when collating the data from the evaluation forms comes from focussing on the end point. Why do we use an evaluation cycle? Feedback is collected so that departments can review their processes and make changes where they are needed. If collecting and looking at the information takes so long that the same mistakes are continuing to be made despite key staff having information that would prevent this, it compounds the ineffectiveness of the whole process.
One of the biggest priorities for most organisations is that of reducing costs, as business expenses rise, budgets continue to be cut. Every resource must pull it’s own weight and the analysis of return on investment has become a necessary tool for deciding how those budgets are apportioned.
‘The evaluation and assessment of training has delivered a Return on Investment.’
The real challenge here is finding a solution to these issues without the need for additional staff and capping the resources being used on expanding office spaces and storage facilities. Unfortunately in a bid to save costs on staffing, departments often find that senior members of the team are completing high numbers of hours of administration and data inputting, despite being paid for higher levels of work, making their productivity output poor.
Anthony Kiss, the Learning Technology Officer for Oxford and Buckinghamshire Mental Health NHS Foundation Trust moved his department to an automated solution and say this of the transfer:
“Automation of the data capture process enables the L&D Department to concentrate on the analysis of consolidated feedback, impacting on the future delivery of learning interventions”
In his department, Kiss can now show his return on investment through these interventions and from clear analysis of good quality data.
Conclusion & Potential
If learning departments are not acting upon the feedback that is handed to them then how can we monitor the progress of what efficacies are being made and if we are at all…progressing during this difficult time. To not make use of the feedback could mean we are not heading in any particular direction and questions the point of the overall training deliverance. Learning and development is fundamental but only if we have the time to hear what is being said and if we can act upon that information accordingly.
ePartner Consulting Ltd have more experience in implementing TeleForm and LiquidOffice based data capture and Business Process Management solutions throughout the NHS than any other UK company. We have implemented all kinds of systems from simple questionnaire processing through to mission critical CRF capture. We have faced unique challenges from high volume requirements of over 120,000 pages per day through to complicated installation requirements.
# # #
ePC specialise in implementing and consulting on data capture and Business Process Management solutions. We specialise in electronic and paper based systems for automated capture from paper, fax, eMail and web with on-going process workflow.