Chasing Butterflies – Evaluating the organizational impact of informal learning
May 23, 2014 4 Comments
We all learn things all of the time. It is part of being human; we pick up a snippet of information during a conversation that we remember later, which makes whatever we are doing easier or quicker. We watch a video on the internet to show us how to do something we haven’t done before or can’t quite remember how to do. We read, we interact with people and things and we learn informally every single day.
Informal Learning and ideas like 70:20:10 have been on the minds of Learning and Development Departments and organisations for a while now. Everyone knows that staff learn while they are at work, and that while a proportion of that learning comes to them through formal means, another proportion comes to them informally, through talking and interacting with others, through reading and watching videos, posting on forums, attending MOOC’s and this learning can happen both inside and outside of working hours.
What does all of this informal learning mean for organisations though, what kinds of impacts does it have on the overall performance of the organisation or the individuals which make it up? Does it increase productivity and efficiency, does it increase the competency of staff, and does it improve the bottom line? Anecdotal evidence suggests that it does, and if you believe the pundits and evangelists for the value of informal learning, it is far more valuable to an organisation than all of the formal learning that happens and it is the thing that organisations should be investing in and assisting their staff access in order to maximise their ability to operate in their chosen markets. Is this really the case however, and is it the case across the board for organisations, does informal have a real and significant business impact, does it make a business and its staff better at what it does, and does it provide individuals and the organisation with identifiable increases in competency, efficiency and overall effectiveness?
This post will not actually argue the case either way for informal learning, it will not pit informal against formal learning and it won’t make an argument for an investment strategy based around informal learning. What this book will seek to do however is;
- Look at what organisations need to think about when they think about the value of informal learning,
- Look at the information an organisation needs if it seriously wants to determine the value of informal learning, and
- Look at how to capture that information and what methodologies to use to make sense of it.
This post is also only concerned with learning that has some organisational impact, where the skills and knowledge gained by the learner, is translatable in some way to their current or future role within the business. If a persons what to go off in their own time and as a colleague of mine likes to call it, study underwater basket weaving, then that is fantastic, but it is only of interest to an organisation if some formal link to an improvement in day-to-day work can be seen.
Formal Learning is any course or program designed using industry-recognised disciplines and methodologies that have a formal structure, and specific, well-defined learning objectives, which may lead to formal outcomes or qualifications for the participant and which is delivered through a form of Student-Teacher relationship. That is formal learning is what we are all used to as learning, it is the learning we did at school, at university and at the courses and programs that we attend at work. It is structured learning, which has learning outcomes defined not by the participant, but in the training itself.
So then what is informal learning? One could say that it is simply everything else, every other way in which we manage to learn new skills and be fairly accurate. Informal learning is where there is no set learning outcome, at least not one that is set by anyone other than the participant. It is unstructured, not designed using robust principles of instructional design and often not even seen as learning by the participant. It is as I have said before the day to day conversations, the videos, the books, the jobs that we do, they all for part of our informal learning.
If we look at the definition of informal learning as opposed to that of formal learning, the problem in terms of evaluating its impact can be clearly seen. There are no outcomes that can be looked at to see if the participant has successfully achieved, it is not delivered in traditional methods and formats, there is by its very nature no assessment as part of it and in a lot of cases we may not even be sure when or where the learning itself actually occurred. So it is no wonder we struggle with not just the concept but with the practicality of how to evaluate the informal learning that staff do during the course of their days and what if any personal or organisational impact that learning may have.
It seems to me that there are two types of informal learning which are important to organisations, learning that simply increases, builds or improves a skill and learning provides some ‘formal’ recognition pathway for the learner. It is important to note that it is some kind of formal recognition that is important here, recognition, where there is some ‘qualification’ style outcome as a result of the learning, where the participant is assessed in some way prior to being deemed as competent in the skill or knowledge. I am not talking about badges or other such methods to capture the results of informal learning, be they peer-reviewed or not, because from an organisational point of view they in these methods are (at least currently) virtually meaningless.
Why meaningless; I have often recounted a story of being asked as part of an investigation into an incident involving a member of staff, “How did you know this person was competent?” Now if my answer had been, well he has a badge for it, I think I may have gotten a much different reaction to saying, as I did, “Well, they have completed all of the assessment tasks, including a third-party observation, necessary for us to be satisfied that they were competent under the rules of evidence set out in the legislation pertaining to the operations of Registered Training Organisations, in Australia.”
This is not to say that the person with the badge was not competent, or the one with the assessments didn’t make a mistake or had forgotten what they had learnt. It is to say however that it seems more likely that we can be confident that the second person was actually competent than we can be of the first.
Is this to say then that if informal learning does not lead to some kind of qualification that it is of no use, or much less use than what we would normally see as formal learning? No, not all informal learning though is going to lead a staff member to a qualification, some of it is not related to or captured by the range of qualifications available, some simply adds to the skill set they already have, making them better at their role, but not providing them with a new skill. This suggests that any process of evaluation that we may seek to apply to informal learning must be capable of dealing with both that learning that will lead to a formal outcome and that which will not.
To this end there seems to be a number of things that we need to know in order to be able to begin to evaluate informal learning, and I am indebted to Saul Carliner for some of his thinking around this. In order to be able to evaluate effectively the impact of informal learning both at an individual participant and an organisational level we will need the following information;
- A baseline – what is the staff members current skill level,
- What they have learned,
- How they have learned it,
- New skill level,
- Is there a competency attachable to the learning,
- Effect of learning on organisational metrics – reduction in customer complaints, less injuries etc., and maybe
- Return on Investment?
Interestingly this list looks very much like what we need know about any learning process in an organisation. This of course then leads to the question, if this is what we need to know to evaluate the effectiveness of organisational learning, why are we treating the outcomes of informal learning differently to how we would treat formal learning.
Now it may not be relevant, we may not or able to or we may not want to, capture the standard ‘smile’ sheet satisfaction style data that we collect from formal training and yes, the natural of the learning, pull not push, driven by the individual, just in time etc., all make the nature of the process of learning different. However, when we look at it from an organisational point of view are we not looking for the same thing as with formal learning? We are looking for an increase in the skill level of the staff member, such that increase in skill will have an effect on the relevant workforce metrics that relate to their role in the organisation. If we aren’t looking to improve the skills of our staff and the organisation as a whole, what are we investing in informal learning systems in the workplace, and why has it become so important.
So how do we achieve this, the process outlines what we need to have in order to really make this idea work;
- Skills Outline for each role type within the organisation,
- Assessment of Staff member against skills outline – there are a range of options here, but I think there has to be at least self-assessment + manager assessment at the very least,
- System for capturing staff informal learning activities,
- Regular (6-12 Month) updates of Staff skill assessments,
- Data capture of changes in skills levels across the organisation,
- Method of mapping skills changes to competencies. and
- Methodology of converting skills changes to organisational metrics and ROI.
If we start with the idea of a baseline we might be able to sort out some structure and processes around this idea. So, where might a baseline come from?
- Position Description,
- Performance and Professional development plans,
- Formalised Assessment, or
- Job skills analysis plus a rating system.
But how can we do it without it being onerous on everyone involved. If we use position descriptions as our starting point, we have the problem of there not being enough detail or they are not skills based or we don’t assess the person against them in a really formal way that gives us any real data to work with in the first place. They could be coupled with self-assessment, and direct manager assessment to give a fuller picture of the skill set and levels of an individual staff member. PPD plans can be seen in the same light, in order to make them more useful in terms of presenting us with a baseline we need to capture more granular detail about the role and the staff member’s skills relating to that role.
To really make this process as robust as possible there is a definite need to skill groups determined for each role type within an organisation. These skill groups are entirely separate to any particular position description and are tied to role types and levels rather than to specific staff or positions. Now to what level of granularity an organisation is going to need to go to is going to have to be determined by each organisation. My thinking however is that for most organisations there would be general role types to which skill groups could be attached. The skills contained in these skill groups would also have some similarity through the hierarchy of the organisation, everyone in the organisation needs to be able to communicate, but the level of skill expected may well be different.
Once we have the skill groups established, we then have to come up with a way of assessing the current level of competent staff have in relation to the skill group which applies to their role.
Again a number of ways of doing this have been suggested;
- Peer assessment
- Manager assessment
- Formal evaluation and testing
The problem associated with most of these suggestions is that they either don’t really provide us with strong evidence of skill levels or competency or they are or are likely to become onerous and time consuming. Self-assessment is probably not going to provide us with the robust kind of evaluation that we require, particularly if we are looking at this assessment process as providing us with a baseline from with to build evidence towards a qualification from.
Peer assessment and management assessment may also fall into the same trap and in addition they may also depending on the number of assessments and the number of people managed, become onerous and time consuming. While formal assessment processes certainly could meet the criteria in terms of robustness, there is again, depending on the kind of assessment and how it is delivered the real problem of it becoming expensive and time consuming for everyone concerned.
This is of course less of a problem with new hires into the organisation than it is with those staff currently employed. With new hires a system which captures their current skill levels with respect to their specific skill group can be quite easily developed and maintained, providing baseline data, at least on that group relatively easily. The reason for this is that the process for capturing the data can be made part of the recruitment and induction process. Current staff are going to be the group for which capturing this data will be most difficult, as any process is going to additional to their current workloads. One way would be make the Performance and Professional Development review process more robust and include a form of assessment against the skill groups as part of it, in addition to staff and manager perceived assessments of skill levels.
All of this data begins to present us with other problems as well however. How and where do we capture this data so that it is accessible and usable both in terms of individual staff progress and advancement in the relevant skill areas, but also on an organisational level where roll ups of this data could and would be invaluable in determining trends and needs across the entire organisation? How this will be achieved will of course a decision that individual organisation make according to their current systems and needs. It is vital however that this and the other information that is going to be generated through this process is captured, because it provides a wealth of information not just for the organisation, but for the individuals themselves. It allows the organisation to see the value and the benefit of its learning investments, be they formal or informal and allows individuals to track their own progress towards a variety of goals.
Having the baseline data available gives organisations a place to start when looking at the evaluation of learning, be it formal or informal, however one of the problems with capturing and evaluating the effects and impact of informal learning, is quite simply identifying when and how it happened. Unlike traditional formal learning, there are no classes, no events, no calendar of activities to point to, to show when a staff member undertook training on a particular subject. There are no learning outcomes, no standard content to point to establish what is a participant may have actually learnt. This has and does present a significant problem for evaluating the impact and effects of informal learning, if we don’t know what, when or how someone learnt something then how can we determine where that knowledge came from? Unfortunately from the perspective of this work there is little to talk about here except to say that organisations will need to find a way to capture this data, which is meaningful to them and suits their needs and individual situations. One method would be to establish a database of ‘learning activities’ which staff could update as needed or which could capture data from a range of sources using technology solutions around a staff members interactions with informal learning opportunities. There is no single answer here and no single best way of achieving this. There will even be differences of opinion about the level of granularity that is sufficient to show that a participant has in fact had an informal learning experience. Again, as with the baseline data what is important here is that data of some description that is meaningful to the organisation and individual is captured.
We have captured the baseline skills data across individuals in the organisation and we have also captured when and how they are learning, what is needed now is a method or a process of regularly assessing individual’s fluctuations in the skills within their skill group. Again it is going to be up to an individual organisation how robust they make these assessments, at the lowest end would sit, I think, a 6-12 month self-assessment by staff of where there skills now sit. At the highest end would be some form of regular, controlled, formalised testing process which provided solid evidence in relation to changes in key skills. If we were thinking about say desktop applications such as Word, staff members could at set time intervals be required to undergo a standardised external test of their skills and knowledge, which should give solid, meaningful data on changes to skill levels. The other point of note here is that there needs to be a correlation between the kind of assessment done at this stage and initial baseline assessment. While some variety of methods of assessment will work and provide meaningful data, this cannot be said of all methods. For example if the initial assessment had be a rigorous formal evaluation, carried out through validated assessment tools, administered professionally, but then when it came to the point of reassessment a simple self-assessment tool was utilised, it may be the case that the results of the data are not as useful as they could be. My thinking here would be simply that the same type of assessment, even the same assessment should be used to detect changes in an individual’s skills and knowledge; otherwise we are simply not comparing the same things.
Once this data has been collected we are at some point going to have to decide what it means either on an individual level, an organisational level or both. It is here, I think, that the real challenge may lie.
If we look at the following example; an organisation has been capturing data on its staff skill levels across, they have also been providing staff with access to and encouragement to utilise and learn through informal methods. After 12 months they see an average rise in skill levels across the organisation of 5% and over the same period a rise in sales of 3%. Can we make a connection between the two, is there any correlation between the increases.
At least at this point I think you would be hard pressed to make much out of it. It may even be difficult to make the correlation between the informal learning and the rise in skill levels, why? Well simply because unless all they have been doing is engaging in informal learning and there has been no formal activities, or coaching and mentoring, supervision etc., then we are going to need to come up with a process of separating out the gains make from formal learning and the gains made from informal learning. At this point we might be tempted to fall back on something like 70:20:10 and say something like, well 70% of learning is informal so 70% of the skills increase is due to informal learning, namely a 3.5% increase. I that I would and I think that a lot of other people would find an argument like that to be somewhat less than satisfying. It could be the case that it was the 10% formal learning component that produced the entire 5% rise in skill and informal learning did not contribute to the skills increase at all.
Even if we can make a strong case that informal learning contributed to the skills increase, we then still have to make the case that the skills increase contributed to the sales increase and unless we can show a link between the learning, the skills and the increase in sales that is going to be difficult. Here I think it is the level of granularity that we apply to the data we are collecting. If we can show that members of the sales department accessed a range of informal learning resources all related to closing sales, and that their skill level at closing sales went up (by whatever means of assessment we are using) and then the overall sales figures for that sales department increased, then I think we might have a strong enough case to suggest that the informal learning the staff did, had a correlative effect on the sales figures. For me however, this is still quite a big question mark. It is hard enough to do real, robust return on investment calculations with traditional formal learning; with the complexities surrounding informal learning it is much more difficult. Does this mean it is impossible? No, it simply means that organisations need to really figure out what it is that they are looking for in terms of success and what kinds of metrics and measurements will provide them with evidence of that success. If an increase in sales is an indicator of success and sales increase there has been a success, the problem comes with trying to determine what it was that caused this success.
One way which might allow us to hone in more on what it was the catalyst for the changes and therefore the success would be apply a process such as Brinkerhoff’s ‘success case method’ for evaluating training. If we formally ask those staff who met the criteria for success what they did, or what they think was the reason they were successful and we formally ask those who were not as successful what they did and what they think was the reason they were not as successful, we can then compare the sets of answers. This sort of process may give us some solid insights into whether there were specific things which were the root cause of the success. The successful staff may tell us that they think the formal training was responsible for about 50% of the success, because it gave them the skills they needed, but the other 50% came from the fact that their manager was supportive of them utilising the new skills they had learnt. On the other hand staff might tell us that one member of the team found a really good YouTube video that had a lot really helpful ideas and skills in it and shared it with the entire group and the successful staff were the ones that watched and applied the information in the video. This sort of information would give us data from which we could begin to look at what were the significant drivers of the successes.
The other part of the puzzle around the evaluation of informal learning for me is the link to formal qualifications and measurements of competency. Now for non-Australian readers and readers not familiar with the Australian competency based vocational education training system some of this might seem a bit foreign to start, but essentially it is all part of the same puzzle. One thing that I think it is necessary to be clear on here is that I am talking about formally recognised types of qualifications, there is a lot of talk about things like badges (peer-reviewed or otherwise) and other types of ‘endorsements’ of people skills. These are not what I am talking about here, and from a talent management, recruitment and learning perspective I see little value in these badges and related concepts. What I am talking about here are formally recognised kinds of qualification or certification where there is a robust, standardised and formalised assessment attached to the awarding of the qualification itself.
One of the values I see coming from informal learning is the ability to use the information collected about the kind and type of learning undertaken and results this learning has caused, is that it provides evidence which could be used to show that staff members met the requirements for certain formal qualifications.
Take for example the following;
A staff member, who is in a retail role and has a desire to move into store management, begins to take some online courses/MOOC, both in their own time and with the support of their manager, at times when they are able to during their working hours. The manager also allows them to undertake some stretch tasks around stock management and ordering and financial management. The staff member also actively becomes involved in a number of online and face to face discussion groups with other retail managers and staff around increasing store sales, better stock management and staffing and HR issues in the retail environment. As a result of these activities the staff members own personal sales increase, they receive numerous positive compliments from staff, the manager is comfortable letting them handle some of the stock ordering and closing off of the store financials at end of day. The manager and the staff member then decide that as a result of this the staff member should look at undertaking a formal qualification around retail or frontline management and decide that they will undertake the Certificate IV in Frontline Management (An Australian Vocational Education Qualification). If good records of the learning activities and achievements have been kept by both the manager and the staff member, they may find that significant parts, if not all depending on the length of time and the amount of experience the staff member has, of the qualifications requirements may have already have been met and that the staff member may be able to undergo a process, in Australia referred to as recognition of prior learning, to show evidence of their competency and be awarded the qualification by the appropriate authority.
This for me is one of the great values that lies untapped as part of the informal learning process, that is transitioning from just learning that may have quite a significant outcome in terms of both the business and the individual in terms of quality of work, productivity, effectiveness etc. and adding to that the additional value of giving the staff member access to formal qualification outcomes. This will be of particular advantage where the staff member in question does not have formal qualifications either in general or in the particular area they are working in. This is a definite value added outcome for the concept of formal learning and one that I think really needs to be explored further.
Informal learning is an incredibly valuable tool for both organisations and individuals, it provides individuals with the learning that they need, when they need it, in environments which may be very conducive to their learning. It provides organisations with an avenue to reinforce and build on the skills and knowledge that staff obtain through formal training activities and to provide them with additional skills through alternative methods of learning. We need however to understand the value, both in terms of what that value is and what that value means for organisations. Utilising anecdotal rules of thumb about the value of informal learning and what it provides are not useful tools, particularly when considering investment strategies around Learning and Development in organisations. We need to treat to treat informal learning like any other part of the learning business and make sure that we know what our success criteria are, what the real value is and the return on investment that we are getting for informal learning. We also need to embrace the concept that informal learning can be a pathway for staff to formal qualification outcomes and that this can only enhance the value that is gained both individually and by the organisation from its investment in informal learning.