Measuring outcomes in Vocational Education – Are we doing enough?

With the deadline for Total VET activity reporting fast approaching us my mind has turned to how we evaluate outcomes in Vocational Education in Australia, or at least the data that we collect nationally which seeks to provide us with information on the Vocational Education Industry in this country.  Now we all know and love the Learner Questionnaire and the Employer Questionnaire, but really what are we capturing here and is it really giving us any real useful information.  Lets be honest, when we look at the Questionnaire’s, even the employer one, they really nothing more simple level one satisfaction (smile) sheets and anyone who has been in the L&D industry for even a few weeks knows how resounding useless smile sheets are if you want to convince anyone about anything that has to do with training.  Now I know that these are only one source of data, but if I had used that sort of data to convince an executive team as to why they should invest in a learning program I would have been laughed out of the boardroom.

If we look at the well know Kirkpatrick model of evaluation with its four levels

  1. Reaction.
  2. Learning.
  3. Behavior.
  4. Results.

realistically here all we are doing is evaluating reaction.  We are not really looking at anything that comes close to the other 3 steps, at least not on any consistent regular basis.  Now of course, someone could suggest that levels 2-4 and even level 5 (ROI) if we want to include that is something that should be done in the organisations that are sending their employees to be trained and that is a fair point and I would suggest that most organisations with L&D units are already doing this.  However, what about all of the people who are undertaking training not at the behest of their employer, they may be unemployed, casual, wanting to change careers, we seem not to be collecting and analysing any higher level data on these people.  We also seem not to be asking those employers who are collecting robust data on the learning of their employees for their data either.  Now admittedly in Queensland, and this may be the case in other states as well there is some collection being done of employment outcomes for students undertaking funded programs, but really if we are trying make a better system don’t we need some real data to allow us to make decisions about what is working and what isn’t.  (if we are collecting this data and someone knows about it and how we are doing it please let me know)  I come back again to the point that if I was to try and justify an organisational L&D budget on the kinds of data that is being collected on training, I would be told to go away and come up with some real figures.

So what should we be capturing, If we look at the Kirkpatrick model (I will talk about some other options later)  we should at least be capturing whether or not the student actually learnt something, where the learning objectives met, and how much change has there been in their skills, knowledge or attitude.  Now of course to do this properly, we need to collect data before and after training, and it could be suggested that by virtue of the person going from not having a qualification to having a qualification (becoming competent) that there is an increase in knowledge, but is this actually the case, has their actually be a change in the persons skills, knowledge or attitude.  With the data we are collecting at the moment we just don’t know we are making assumptions, but without good data those assumptions could very well be wrong.

What about whether or not we are seeing any changes in behaviour, are the learners putting what they ‘learnt’ into action in the workplace, are their behaviours changing as a result of the training that was delivered to them.  This is not something that can be answered with a smile sheet at the end of training, or an employer questionnaire with general questions about perceived improvement.  This is data that needs to be collected once the learner has been in the workplace for a while, once they have had time to adjust and integrate their new learnings into their work environment.

Then of course there is the question of what the result was, did this training improve anything organisationally, on a state level, on a national level.  Was it good for the organisation, the country, the learners, the bottom line, whatever.  We need to figure out what areas we a seeking to see improvements in and then track and see whether or not we are seeing those improvements.  Again this is a step up in complexity in terms of the types of data needed and how we might collect it.

Finally of course there is the question of Return on Investment.  It lets us begin to answer the question, has the billions of dollars the government has handed out in VET-FEE HELP and other programs and the money spent by business and organisations on top of that has been worth it, have we got the return on that money that we expected or did it just swirl away into a black hole where it didn’t improve anything which we would consider enough to measure.

Is all of this easy to do? No.  Am I suggesting that the reporting requirements on RTO’s change so that they are required to capture a whole range of additional data over a timeframe long after the student has finished their courses?  No.  Am I suggesting that perhaps we need to do a yearly survey, akin to the kind done by ATD (what was ASTD) on the state of learning, and include employers, organisations, training providers and learners and try to ask them some serious questions?  Maybe.  Am I suggesting we need to do more with this so called big data thing that has been talked about so much?  Maybe.  I don’t have a definitive answer, I just think we need to be doing more if we really want to capture the true outcomes from our Vocational Education system.


My First 90 days #first90 – A repost from linkedin

I wrote this piece a couple of days ago on Linkedin and thought I would share it hear as well.



It is really tempting I think whenever you are starting a new role to want to show yourself in the best light, to be the best person you can be, to try and really fit and become part of the company culture. The problem is that sometimes what we end up doing is giving people a false impression of what we are like, of how to approach us and of how we work.

My single biggest piece of advice is that from day one follow just one simple rule

Be Yourself

Why? You are either going to fit or you are not, and the company hired you so they must have at least thought there was a good chance you were going to fit and in the long run if it is not going to work out for either you or the organisation would not it be better if everyone realised it as early on as possible when everyone can cut there losses with the minimum of damage.

There is also another much stronger reason for adopting this simple rule as well. You just might find that your job becomes far more than that, it becomes some where where get to enjoy what you do, form strong relationships and most of all actually have one.


So who are these Private RTO’s really

I have heard a lot of talk recently about private RTO’s, the need to restrict the number of them, the funding available, stop funding free enterprise with public money and the like, so I thought that at least for a moment I might explore who these people and organisations are who seem to be being demonised a little bit in this whole discussion of TAFE, public education and the VET sector in general.  It seems quite easy I think to lump all non-public (read TAFE) suppliers of VET education into the private provider category, however it is not as simple as that by any stretch of the imagination.  There are at least 3 major distinctions which can be made with the ‘private’ RTO sector:

  1. For Profit commercial
  2. Enterprise
  3. Not for profit

and even within these broad categories there are going to be a huge variety.

If we take for example For Profit Commercial providers, the vast majority of these providers are small to medium size businesses, who are not making substantial profits, work in niche areas in one maybe two sectors at the most.  There are very few private providers who are making millions out of government funding or out of training in general.  Most of them started their businesses not because they wanted to make money or get rich, but because they saw a need, they saw sometimes personally that people were dissatisfied with the training they were getting, the quality of students, the outcomes and thought they could do better.  And a lot of them do, a lot of these smaller ‘for profit’ providers provide services which are at least as good as and a lot of time better than those offered by the large corporate or government providers.  Why? That is easy, they actually care about the work they are doing, the businesses and industries they work with and the people.

So what about enterprise RTO’s  I have talked long and often about Enterprise RTO’s , their place and purpose in the VET sector and why an organisation would choose to become an ERTO.  For most of these organisations becoming and maintaining an RTO came from two reasons, firstly, it is a relatively natural extension from the standard operations of an L&D unit to provide accredited training to the organisations staff and want to be able to provide it in-house so that the content, delivery, outcomes, and costs can be better controlled and managed.  Secondly is dissatisfaction, dissatisfaction with the quality of people being recruited who already have qualifications and dissatisfaction with the quality of outcomes for existing staff doing accredited training.  It therefore becomes a relatively easy decision to move to a position where those qualifications which are vital to the business are delivered  in-house, as part of the normal regime of Learning and Development activities.  Again however for most of these ERTO’s the number of qualifications are small and in sector that relate very strongly to the core business of the organisation.  Now some of these ERTO’s operate entirely in-house, they train only their own staff and no one else, most of them utilise government funding where it is available but also provide a wide range of training that is funded, others  choose to take their expertise to the market place and provided training services to a wider (though usually only within their sector) group of stakeholders.  Again, why?  Because they know what is needed to make a competent worker, they know what a good outcome is and how it is best delivered and they know this because they are actually embedded in the industry.

Then we have the not for profits, most of which are either enterprise rto’s of some description or deeply embedded in another not for profit organisation.  These providers don’t seek to make huge profits, they seek to ensure that both those who currently work and those who want to work in their sectors (usually community services) have the best opportunities to be able to do that.  They work with the disabled, the disenfranchised, the disadvantaged and they do it well.  They do it well because that is their core business, they know and understand the world in which they work, they are embedded in the industry, they live it day in and day out and want to pass that knowledge and experience onto other.  Now not only do a lot of these providers like all of the others provide services and outcomes that are at least equal if not better than those provided by the large and public providers, they also tend to do it cheaper, making it more affordable and accessible to those who are both most vulnerable and most in need of these programs, but in general they do it with far less fees for students.  Take for example a Certificate III in Disability which attracts funding of around $2500 in QLD.  Now a student even with a concession card wanting to do this qualification at a TAFE in Queensland would have to come up with around $1000 in student fees.  How much would they pay at a not for profit provider between $20 and $100. Why? because these providers know that most of the people who are looking to do this course can’t afford $1000 and because they are connected to their communities and understand the need for people to be able to access affordable training in order to be able to change their lives.

So for me I sometimes tend to get upset when people use the term private provider like a stick, to beat the non-public side of the industry.  They drag out the horror stories of the small number of providers who do the wrong thing and suggest that is the case for everyone.  They make the assumption that TAFE is good and non-TAFE is bad.  They make the assumption that only a public provider can provide services for people with challenges and disabilities, or at a price that people can afford.  They make the assumption that public providers know the industry better than the organisations and people who actually work in the industries.  They talk about cutting access to funding and only allowing TAFE to provide Government funded courses as if this would have no effect on the lives of the thousands of people and organisations who use these non-public providers, let alone the effect it would have on the lives and livelihoods of the people who own, manage and work in these providers.

As I have said on so many occasions we need a public provider, but we also need the non-public providers as well and to suggest that TAFE can do everything that the non-public providers do, or to lump  all non-public providers into the ‘Private’ handbasket.  Is a few that misunderstands who these providers are and the services they bring to their communities.

Anyway that’s my opinion

How do you lose $70 million and still have a job? Work for TAFE in Victoria it seems.

I posted the article on Linkedin yesterday about the foreshadowed losses in the Victoria TAFE system and I have to admit I have been thinking about it a little more overnight. I mean where else but in the public sector could you amass such substantial losses and then get an offer from the government to give you more money.

I struggle to see how that can come about. Did they spent money that they didn’t have in the budget for things like a new SMS, did they grossly underestimate the amount of students. Unfortunately all of these things point to really bad management practices. I have managed very large budgets and even losses of $1 million plus are heavily questioned. Surely they know their business well enough that they can plan forward or are they simply working on the assumption that it doesn’t really matter in the long run because the government will just bail them out and pay the shortfall. Not a way to run any kind of business be it public or private in my opinion.

Now that being said we need to have a public VET education system, but we need to have one that doesn’t constantly need to be bailed out by the government, throwing more money at a system that is clearly broken isn’t going to fix it.

TAFE has a place, but it is struggling to find that place. There are arguments that suggest that it provides support and training in places were non-public providers don’t because of economic reasons, but even that is changing there are plenty of providers particularly NFP’s who are willing to and already do the kinds of work and in the kinds of places that traditionally only TAFE did.

If a business (and even though it is a public entity TAFE it is still a business) can’t support itself with the population base it has around it or its infrastructure costs are too high or whatever the reason is, then something needs to be done. Unless there are strong social and economic reasons for keeping it then closing or merging needs to be seriously considered, as does staffing levels and executive salaries and packages.

Why have separate CEO’s and boards and executive teams at each TAFE why not centralise at least a bit, cut down on management overheads and put the money where it needs to go, the teaching of participants.

Anyway thats just my opinion

My favourite L&D Books

I thought as a start to the year I would begin building a list of my favourite Learning and Development books, these are all books that I have read and would recommend highly.  If you have any other suggestions then feel free to let me know.

The business of learning – David Vance

The success case method – Robert O Brinkerhoff

High Impact Learning – Robert O Brinkerhoff

Ten Steps to a Learning Organisation – Kline and Saunders

Building the Learning Organisation – Marguardt

The Fifth Discipline – Senge

The Training Measurement Book – Bersin

The Six Disciplines of Breakthrough Learning – Wick, Pollock and Jefferson

How to measure training results – Phillips and Stone


So as I said if you have any other books that you like let me know.  I intend to build this list over the year.


Why I work here.

It is that time of year where everyone thinks about the year ahead and what they want to do and achieve, but sometimes amongst all of these thoughts and plans it is easy to forget the why behind the things that we do.  I was reminded of that today by a post by a friend on LinkedIn.  We talk about compliance and standards, about how to improve the things that we do, about best practice, trends and new technologies.  We talk about training needs and delivery processes, how to fund and manage learning.  We talk about policy and theory and academic positions and theories, informal and formal learning, elearning, mlearning and all of the things we would like to do or try it we had the time and the resources.   While this is all fine it is very easy to lose sight of the simple facts about the sector that we all work in, it is about the participants and more importantly every single day this sector changes people’s lives. Not just the lives of individuals but of their families, those around them and their communities.

We need to remember the person who failed at school but who has learnt new skills through a well structured adult learning program

We need to remember the staff who through the things that we provide are able to life their careers and their lives to heights they never thought possible

We need to remember the clients and stakeholders who get better quality of service and outcomes and walk away happy rather than disgruntled and take that happiness into other parts of their lives

And most importantly we need to remember that working in this sector more than many others gives us such an opportunity to have a real and lasting effect on the lives of others.

And I for one and grateful to have such a wonderful opportunity.

%d bloggers like this: