Can we measure learning?

Somewhere in recent conversations someone came up with the line "if we can't measure it we can't manage it". I have heard this many times before and I'm not sure of its origin - if I was asked in a pub quiz I would suggest Jack Welsh of GE but I could be wrong.
Anyhow, the axiom is part of everyday management speak and is often cited as a core principle used in change management and strategic planning.
We've had a good example of this recently where financial systems and governments appear unable to 'measure' the extent of the bad bank loans (aka toxic debt) and, so the argument goes, we need to get these bad loans out of the system not because they are 'bad' as such but because they are unmeasurable.
The Irish Government plan is to establish a new agency - the National Asset Management Agency (NAMA) - this will take all the bad loans out of the banking system in order to free up the regular banks to continue to do business in the normal way.
I can see how all this will be used in case studies to further reinforce the axiom of 'can't measure can't manage'. The problematisation may even be reduced to "an unmeasurable got into the system and we had to clear it out".
Wouldn't life be really simple if it wasn't for all these unmeasurables getting in the way! They seem to crop up everywhere in finance, in politics, in sport and in nature.
Let's look critically at the relationship between measurement and management and see if the axiom holds true.
What do we mean when we say to measure something? Our usual response is that we put a value, preferably a numerical value, on something.
What do we mean when we say we manage something? We usually mean that we can exercise some control over a system or process and we can use this control to direct the process toward a particular goal.
Here are some everyday examples of measurement and management:

I can manage to keep my my driving below the speed limit because the speedometer provides me with a measure of my speed.

I manage my finances by keeping a regular check on my bank account balances.

The election agent manages a political campaign by measuring the public mood through opinion pools.

Notice how in the first example the system for measuring speed and the system for controlling speed are independent. In fact the interface between these two systems is me when I drive. I react to what I read on the speedometer and thereby adjust my speed. My car also has a cruise control function - so what's happening when this is engaged? I input the desired speed and take myself out of the loop - the speedometer 'talks' to the accelerator and the desired speed is maintained. This is a very good example of measurement-based management.
Or is it? What exactly is being managed? Is the system driving the car? Could I commute everyday using such a speed management system? The answer is of course not I need to manage brakes, gears, indicators, road conditions, other traffic, pedestrians etc. and respond to many, many more complex inputs than the reading on the speedometer.
Remaining alert while driving is perhaps just as important as driving at the appropriate speed. And yes I would be breaking the law if I was driving under the influence of drink or drugs as these are known to affect alertness and for these we also have measures such as blood alcohol levels but note that these are not measures of 'alertness' just indications of factors likely to influence alertness.
So is alertness measurable? This is a more difficult and interesting question. Let's see - we could start by looking at the extremes - I could say that when I am asleep I have a low measure of alertness (and should not be driving!) and when I am wide awake and concentrating on my driving I have a high measure of alertness. But is there any point in developing a scale, say from 1 to 10 on alertness - we could then introduce a new law - driving below the alert limit!
Ah! you might say this is nonsense - alertness is about potential to respond - we cannot really say anything about alertness except in retrospect. People doze off in the middle of the day even when driving a car and sleeping people will quickly escape from a burning building if they have prepared for this in advance. Alertness is not just a immediate state - it is a complex of influences involving past experience, planning and a sensitivity to immediate stimuli.
So to summarise so far - to manage the process of driving a car we have some measurable conditions such as speed that we can monitor and some, let's say, far less measurable but very important conditions such as alertness that we also need to monitor.
So now you say - aha! - you've just used the word 'monitor' in both cases so in a way you are measuring alertness.
Yes I agree but there is a fundamental difference between the two forms of measurement - in the case of speed the system to measure and the system to respond are separate but in the case of alertness the system to measure is part of your level of alertness.
The simple act of asking yourself how alert you are will increase your level of alertness.
So alertness is important for management (of many things apart from driving a car) cannot really be measured.
What about the other examples I give above? Yes my bank account balance is an important measure to help me manage my finances but it is not sufficient. In business, quoted companies are required to report full audited accounts and to make these available to investors and yet despite these measures, many banks and businesses have had to reevaluate their balance sheets by many billions of Euro.

So here's the first take away -
In today's society we place too much emphasis on what can be measured and not enough emphasis on what is important.

And here is the second take away -
There difference between the use of the terms measurement and management in relation to discrete processes such as the speedometer and the accelerator and connected processes such as when we wish to monitor our own alertness while driving.

I'll leave it to you to make the connection between the second take away and the systems of financial regulation for banks!



And so to my question - can we measure learning?

Gregory Bateson (Steps to an ecology of mind University of Chicago press 2000 edition) deals with a similar question by means of a metalogue - a conversation about some problematic subject. He uses a father daughter conversation to explore the question of How Much Do You Know?.

Here is a brief extract:

Daughter: Daddy how much do you know?
Father: Me? Hmm - I have about a pound of knowledge.
D: Don't be silly. Is it a pound sterling or a pound weight? I mean really how much do you know?
F: Well, my brain weighs about two pounds and I suppose I use a quarter of it - or use it at a quarter efficiency. So let's say half a pound.
D: But do you know more than Johnny's daddy? Do you know more than I do?
Father: Hmm - I once knew a little boy in England who asked his father, "Do father's always know more than sons?" and the father said, "Yes". The next question was, "Daddy who invented the steam engine?" and the father said, "James Watt". And the son came back with " - but why didn't James Watt's father invent it?"

And so the conversation continues as Bateson skillfully challenges our everyday assumptions about knowledge, quantity and measurement.

There are certainly aspects of learning that we can measure - we can design tests and assessments to demonstrate knowledge and competence in certain circumstances. However, as with my example of 'alertness' in relation to driving a car it is not possible to measure everything that is important.
We often make the following mistakes when we try measurement of learning:

We measure what we can measure easily (e.g. facts and information) and not necessarily what is important (e.g. problem-solving or coping skills).

We neglect to recognise that there are aspects of learning that are unmeasurable but important.

We measure out of context - an exam setting rather than a usage setting.

We neglect to recognise that assessment itself is a learning rather than a measurement process.

So to return to the management axiom of:

"if managers can't measure it they can't manage it".

I suggest that we will need to replace it with:

"if a manager can't question the measurement then we should question the manager"