Measuring targets

The justification given by the Environment Agency for the bonuses taken by their managers is that they had achieved 42 of their 45 performance-related targets. There has been much debate about whether the bonuses were appropriate in the circumstances, but besides the important points of principle, has anyone checked to see what those targets consisted of?

For the benefit of our readers, I have now waded through them. They are listed with comments below, but in summary, it was 42 out of 59 (not 45), most of the 42 were insignificant, bureaucratic or suspect (often some combination of the three), some of the other 17 were more significant than the 42, and most of the 59 were inappropriate tools for measuring their performance regardless of success. Highlights included:

• Success claimed in influencing planning decisions in relation to development in the floodplain, in offering an appropriate flood warning service to properties in the flood plain, in getting more people to take their advice on flooding, and in delivering their flood risk management programme. If this is success, you've got to wonder what their definition of failure would be.
• They redefined their failure to achieve their targets for the number of houses protected from flooding and the condition of the flood risk management systems as "partial achievement" (in much the same way that teachers wanted to refer to failure as "deferred success"). This allowed them to leave these factors out of the consideration of their overall success in meeting their targets - their logic seems to be: "it's neither success nor failure, so we just won't count it at all". In the case of the flood risk management systems, this "almost success" consisted of a little over half the systems being in their target condition.
• One of the targets which they admitted they had not achieved was for emissions of priority pollutants to be going down. Just a minor part of the EA's role, but obviously not as important as their successes in increasing the proportion of ethnic-minority employees in their workforce to a full 2.7%, getting positive media coverage, putting "workforce plans" in place, and not having too many accidents.
• They have, apparently, made themselves more efficient, dealt with permit- and planning-applications on a timely basis, reduced the administrative burden that they place on business, and generally delivered such a great "service" that most of their "customers" are happy with them. Honestly. Why are you laughing....

++++++++++++

The Tories, and their environment spokesman, Peter Ainsworth, in particular, have joined the attack on the bonuses recently awarded to Environment Agency managers. Iain Dale has posted on the subject, rejoicing that Mr Ainsworth has finally decided to say something publicly about the floods, and getting some publicity from it, though he appears neither to have led the criticisms in this regard, nor to have responded to questions on it as effectively as Chris Huhne, the LibDem spokesman on the environment.

Some commenters on Iain Dale's site pointed out that it ought not to be Tory policy to renege on government contracts. I suggested that the bonuses were nevertheless a potentially useful political weapon, if Mr Ainsworth were inclined (and not inhibited by Cameroon centralism) to point it at the Government rather than his own foot:

There is a broader point that could be drawn out of this, if the Tories had the wit. It is a fair point that governments should honour employment contracts, but who designed these contracts? Bonuses should reward skill or effort in achieving goals which might not otherwise have been achieved and which make a real difference to the organization if achieved. Who included bonuses for meeting meaningless, misleading, subjective and sometimes suspect targets? How well-designed are targets that suggest that the EA are performing well, given their numerous, highly-visible failings? (About as well-designed as all the Government's other targets.) If it comes to that, is it possible to design rational performance-related incentives for civil servants, or are bonuses inappropriate to these jobs? Answer: the latter.

This episode is a beautiful illustration of this Government's incompetent blurring of the line between public and private, incomprehension of the difference between the two, and complete lack of management skills and commercial nous. If you could combine all their practical experience into one person, you'd get the equivalent of a junior management trainee in a provincial bank. The criticism should be not that EA managers took a bonus to which they were entitled, but that some fool in government offered it to them in the first place.

Baroness Young is an appalling apologist for a dysfunctional organization, but the Tories ought to consider not only whether she is the appropriate target in this case, but also whether it serves their purposes better to attack her or the Government. But of course, it's very difficult to criticize the use of management techniques imported inappropriately from the world of management consultancy, when that's all you really know about management, and you're planning to do more of the same yourself, because you claim that the problem with government is that it's badly run, rather than impossible to run effectively on account of its size.

Gordon is vulnerable to attack as an incompetent managerialist, but for that attack to be credible, the opposition has to position itself as anti-managerialist. That means committing to more that just "sharing the proceeds of growth" as a means to shrink the size of government. It means stating overtly that government is too fat, sticking its nose into too many things, trying to calculate things it can't and shouldn't calculate, and running things it shouldn't run. It means disclaiming the use of inappropriate tools from the commercial world in the properly bureaucratic management of those things that it should be doing. It requires a public commitment to making cuts and changes to the way government is run. This was an opportunity for the Tories to show that they, unlike the Government, understand the difference between the public and private realms, and they fluffed it.

What is extraordinary in this context is the EA spokeswoman's claim, in defence of their bonuses, that "agency directors' bonuses were less than half the public sector average of 22.5 per cent". And that's supposed to make us feel better?

Is it really fair to call them "meaningless, misleading, subjective and sometimes suspect targets"? Well, firstly, the EA's spokeswoman justified the bonuses on the basis that "the agency hit 42 of 45 targets across a wide range of areas, not only flooding". She has a strange way of counting. According to the annual report, from which this is all drawn, the EA had 59 targets, not 45. They did indeed judge themselves to have achieved 42 of them, but 7 others were "on course or substantially achieved" (which, from the other perspective, means "not achieved"), 3 were "not achieved or only partially achieved" (i.e. "not even close"), four had received "new baselines" (i.e. if at first you don't succeed, change your definition of success), and three simply didn't record performance one way or the other. The spokeswoman's 45 was clearly achieved by only counting targets that were either achieved or not close to being achieved, and ignoring the others. Strange sort of accounting, but par for the course for government self-assessment.

Still, 42 out of 59 sounds quite impressive, until one looks at the nature of the targets that were achieved and missed:

Achieved:-

1. "We reduce the number of the most serious illegal waste dumping incidents." How does one measure this? The EA measure it as simply the number of incidents reported. So how do we distinguish between a real reduction and the classification of incidents by agency staff as slightly less than the "most serious" kind? What are their incentives here? To take a tough line or to turn a blind eye?

2. "More of our navigation assets are in safe working order". How is that measured? Apparently, by the number of "assets" that the EA themselves judge are in "good working order". It is not enormously reassuring that "Activity throughout the year has been aimed at addressing priority health and safety issues, so that by 2011 less than 5 per cent of our navigation assets will be in critical condition." What does that mean? That by 2011 only one lock or weir in 20 will be a deathtrap? It's not a ratio and schedule that would be tolerated in business, particularly by the enforcement agencies themselves.

3. "We deliver our navigation programme". This one's easy. It means they've spent (or rather "invested") the money they've been given for these purposes. Well, woopety-doo!

4. "Reduce peoples’ impact on the environment through World Environment Day". Measured by the numbers of pledges obtained, where people promise to do things that notionally reduce their impact on the environment. So it doesn't measure real reductions, just promised reductions. This is the sort of decisive action, and clear incentive, that is going to make all the difference.

5. "We improve the condition of Sites of Special Scientific Interest (SSSIs) on Environment Agency owned land". The target is for 85% of SSSIs to be in "favourable condition". So these are sites that are so important that the government has ring-fenced them and applied tough planning restrictions to protect them, and yet the EA are satisfied that more than one in seven is not in a "favourable condition". I guess whether these sites are important or not depends on whether you are a business somewhere nearby, or the government agency tasked with looking after them.

6. "Increase in length of watercourse restored or significantly enhanced for wildlife". Again self-assessed, they "improved 144 kilometres of river habitat through our Flood Risk Management, biodiversity and fisheries work". Looks like the flood-risk they were focusing on might have been to the wrong animals.

7. "Population of key fish and wetland species are going up." Their success in this regard didn't actually mean that populations were going up. Their success was measured by the fact that they had established a baseline against which to measure future improvements. Given that the quality/accuracy of the baseline was irrelevant to the success of even this limited task, this was always going to be a hard one to fail.

8. "More rivers are progressing towards sustainable salmon stocks." Not actually measured in terms of the levels of salmon stocks, but in terms of the number of rivers judged by themselves to be "at risk", relying largely on persuading anglers to throw most of their salmon back.

9. "Actions are completed to improve the status of principal salmon rivers." One cannot do better than quote their own self-assessment - "We completed 862 actions from our Salmon Action Plans: more than our target of 831." The mind boggles at the notion of a Salmon Action Plan that includes 862 actions. Was that one for each salmon? Good thing they didn't include perch as well.

10. "The quality of bathing water is getting better". Once again, self-assessed. And they've given themselves a glowing report: "99.4% of bathing waters in England and Wales met the EC directive's 'imperative standard'". There is a fly in the ointment, though: "However, compliance with the stricter 'guideline standards' fell from 85.2 per cent in 2005 to 81.1 per cent in 2006." So we don't have many waters left that will kill you, but the number of beaches where you wouldn't really like to go for a swim is increasing. That's what I call an improvement.

11. "Water companies deliver planned improvements (the National Environment Programme) agreed through the periodic review of water companies’ prices (PR04)". This isn't even prima facie an achievement of the EA. And in that aspect which is their responsibility to enforce - investigations - "The performance of water companies for investigations is disappointing."

12. "Delivery of key milestones in the Water Framework Directive programme". The production of reams of paper to marry with reams of paper from Europe. The EA are undoubtedly skilled in this sort of task.

13. "Increase the number of Restoring Sustainable Abstraction (RSA) schemes completed." Consists of investigating opportunities to take away from people rights to which they thought they were entitled. The EA have pursued this joyful task with gusto ("completed 133 investigations against a target of 76").

14. "More rural land is covered by environmental agreements that protect soil and water". The EA are successfully imposing more bureaucracy on farmers, who otherwise would simply be twiddling their thumbs wondering what to do with their time, or setting out actively to wreck the land from which they earn their living.

15. "More contaminated land is brought back into use". A full 1,884 ha of contaminated land have been brought back into use (0.01% of the land-mass of England and Wales). It is not recorded how much would have been brought back into use without the intervention of the EA, nor what the EA's role was in achieving this regeneration, nor what standards were applied. This is another target in which the EA is prosecutor, defence, judge and jury.

16. "More companies we regulate under Pollution Prevention and Control (PPC) and Waste Management Licensing (WML) will have Environmental Management Systems (EMS)". Have you seen a typical ISO14001 manual? What a bunch of bureaucratic nonsense. More paper for the sake of it. The EA are undeniably expert at foisting this sort of thing on businesses, by the simple expedient of charging you more if you can't present the desired tomes, until the cost of non-compliance exceeds the cost of compliance.

17. "We reduce the number of businesses with higher risk OPRA scores". By the simple expedient of judging as low-risk those businesses who put together sufficient volume of paper to satisfy the EA's lust for bureaucracy, and as high-risk those who put their efforts instead into management of their businesses.

18. "There are fewer serious and significant (Category 1 and 2) pollution incidents". Recorded. Does this mean that there are fewer incidents, or that the EA are recording fewer incidents, or grading the incidents less severely in order to meet their targets? We will never know, because the EA are once again judge and jury.

19. "Our compliance activity is focused where needed". Is it possible to have a more subjective target? What would be the appropriate metrics? For what it's worth, their sole metric is "to spend a minimum of 25 per cent of our total compliance assessment activity on programmed major audits." And they know that this is the appropriate scope and scale of the necessary activity because...

20. "We reduce our own environmental footprint." Fair enough. These are genuine savings, although as they are expressed per Full-Time Equivalent (FTE) employee, their benefit is diminished by the fact that the number of FTE employees has increased by 2.6%. And the claim that they have reduced their travel emissions by "driving less miles than we drove in 2001-02 even though we have had a large increase in new duties since that time" gives the lie to other claims that reduced enforcement levels were because people were behaving better, rather than the EA checking less.

More comical is the claim that "Our investment strategy for our pensions fund requires fund managers to take account of material environmental risks and opportunities (for example, climate change) in their investment process to manage our £1.5 billion fund." Does this have any relation to the fact that their pension-fund position has worsened by £270 million (over 15% of total liabilities) during the year? It's green, but is it wise?

21. "Reduction in Global warming Potential (GWP) from processes we regulate to help meet the UK’s Kyoto commitment to reduce levels by 12.5 per cent below 1990 levels between 2008 and 2012". Ah, the great lie. The Government is failing to reduce CO2 levels, following the initial impact of the "dash for gas". So they turn to the EA to tell them that emissions of the other greenhouse gases (principally methane and nitrous oxide) are falling sufficiently to still be on target to meet the overall reduction target. Except no one's measuring these emissions. In fact, they are (near enough) impossible to measure. So the EA and their sub-contracted consultants invent methods to estimate emissions, based on assumptions about changing behaviour. So, for instance, the assumption is that generation of electricity from landfill gas (which has increased dramatically since the witching date of 1990) has led to a dramatic reduction in emissions of methane to atmosphere from this source. No doubt, the encouragement of landfill-gas electricity generation has led to an improvement in collection efficiency, but the inconvenient fact that the Government like to ignore is that a large number of landfills had installed collection systems and flarestacks before 1990 - I know, because the company I ran for the past seven years sold many of them. The increase in utilisation and the improvement in flaring and collection efficiencies offered worthwhile but marginal improvements - the big gains were from the initial introduction of flaring, much of which preceded 1990. The Government assumes exactly the opposite. Similar convenient assumptions can be found in the assumptions about emissions of methane from coal mines and of agricultural emissions of methane and nitrous oxide. These are demonstrably not empirically verifiable figures. It is remarkable how conveniently the assumed emissions match the levels that the Government needs, in order to be able to claim to be on course to meet its Kyoto targets.

22. "Our policies and processes are adapted to take account of climate change". Including "Key areas such as Flood Risk Management" - I am not kidding, that is what they lead with. Are you absolutely sure that's a success, then?

23. "We successfully influence planning decisions in relation to development in the floodplain". Just what they've been saying since the floods, isn't it? Ah....

24. "More properties in the floodplain (extreme flood outline) are offered an appropriate flood warning service". Breathtaking.

25. "More people respond to flooding based on our advice". Great advice.

26. "More new Biodiversity Action Plan (BAP) habitats are created". A lot of lakes and swamps, for instance.

27. "We deliver our flood risk management programme." What can one say?

What they mean is that they successfully spent the money they were given. That may not be the most appropriate measure of success in this regard, though it is, of course, the average bureaucrat's favourite measure.

28. "We deliver permits more quickly". Anyone involved with the landfill PPC permitting process won't know whether to laugh or cry at this claim. Barely a single landfill permit was handled within the time allotted, and many ran over two or three times longer than allowed (though always with the operator's "permission", the alternative to which was a lengthy judicial-review process). Where the EA took so long that even they became embarrassed by the delay, they were known to issue permits before the client had had a chance to review and comment, as the process provides for. If there is any truth in this claim, it is not on account of improved efficiency, but because they have finally worked through the backlog of permits, and because anyone thinking of a new activity requiring a permit will be allowing very much longer lead-times than is officially required, stretching out the incoming applications.

29. "We reduce the administrative burden that we place on business." ROFLMAO. It appears that the way this has been calculated is to make an assumption of the worst-case costs that the EA were inclined to inflict on businesses, and then counting any efforts to be reasonable as "reducing the burden". I wonder if these "efficiencies" also count towards the red-tape reductions that Gordon claims he is making?

30. "Our policies and procedures meet quality criteria". 89% of policies passed quality criteria. It is the definition of quality systems that 100% should pass. A strange definition of success.

31. "We are trusted with new duties". Achieved 10 milestones in the new duties they have been assigned. What does that mean? Surely trust should be judged by external assessors, not by internal criteria.

32. "We are more efficient". They claim to have delivered in excess of £27 million of savings. Expenditure was up by £26 million, despite a reduction in capital expenditure, so these savings must be notional, relative to how much less efficient they would otherwise have been. Wouldn't it be great if we could all define efficiency as being better than the worst we might have been?

Any improvement in their financial position was on account of increases in charges to "customers", rather than reductions in costs. These increases amounted to £33 million, but there is no truth in the suggestion that the EA's burden on industry is ever-increasing. Mind you, any marginal improvement in the balance of income and expenditure is massively outweighed by the £270m deterioration in the pension position, let alone the £400m increase in the value of the liabilities (which will lead to further deterioration if equity-values stall or fall, which seems increasingly likely). Not that there is a problem with public-sector pension-provision, or anything.

33. "We successfully influence key audiences around priority issues". How does one measure the EA's influence on issues? Can we measure the volume of nonsense being talked? There is tragi-comic value to one of their own metrics: "There was also a 96 per cent increase in the numbers of people signing up for our new 'Flood Warnings Direct' service in 2006." Return customers may be more limited.

34. "Our customers say we are providing a good service." Don't you just love this notion of the subjects of regulation as the "customers" of the regulators, and the act of regulating being a "service"? Perhaps we "customers" should shop around for other providers of this "service". Maybe we should discuss with our "suppliers" which of their "services" we wish to take, and what their rates should be.

35. "We effectively deliver priority communications programmes and campaigns". Another vital service. Again, one cannot beat the irony of the EA's own words on their performance in this regard: "The success of our priority communications programmes and campaigns resulted partly from ensuring that most of the key products were well planned and delivered to time, cost and quality. Milestones included implementing a new campaign strategy for World Environment Day and completing market research to measure the effectiveness of our communications in waste and agriculture and the effectiveness of our Flood Awareness campaign." Do you think the bureaucrats who produce this stuff are giggling to themselves as they write it?

36. "We respond to planning enquiries on time". Target: 78% on-time. Delivery: 88% on-time. What we ought to demand: 100% on-time, and with a balanced and reasonable assessment. Individuals and businesses don't get this leeway. Why should the EA?

37. "We receive positive media coverage and are seen at the forefront of environmental debate." See any reason to question this claim of success? Perhaps they can claim that they have successfully raised the profile of their role in managing flood defences.

38. "We provide a safe place to work". They are paper-pushers. How difficult can it be to keep injuries to a minimum? 73 (the number of injuries) is a lot of paper cuts.

39. "Our workforce is more diverse". I'm sure your "customers" will be delighted to hear it. Equal-opportunities meddling is so much more tolerable. Not that it's very equal. BME employees now make up 2.7% of the workforce. The increase in their numbers was less than the EA's aspirational target, which doesn't look too ambitious. But why let failure to achieve targets stand in the way of claiming to have achieved the target.

40. "We achieve the right balance of resources". Meaning only 8% of their workforce quit during the year. That may be partly to do with the fact that many of the good ones have already gone to work for the environmental consultancies (whose businesses are booming thanks to the ever-increasing burden of environmental regulation), and many of the rest are being retained by the giving them extended leave, as anyone who has tried to maintain correspondence with the same EA contact for more than a few months will be aware.

41. "We have workforce plans in place." I've got a "kicking EA butt" plan in place. Can I have a bonus?

42. "Our expenditure is in accordance with our plans". They were £5m over budget, which itself was up £22m on last year. But this is government - who's counting the pennies?

 

Not achieved:-

1. "Emissions of priority pollutants are going down". Just a minor part of the EA's responsibilities, on which they are failing. Still, their workforce is more diverse, so let's not let details like this spoil the party.

2. "Reduction in waste disposal from industries we regulate under PPC". Result: 15% over. "Increase in waste recovery or reuse from industries we regulate under PPC". Result: 17% under. There is a serious question about whether they should be aiming for these targets in the first place, but they demonstrate yet again that, where the EA is judged on its ability to make a physical difference, rather than simply pushing paper, it underperforms badly.

3. "We secure the funding necessary to deliver our objectives." Bizarre. One of the targets on which their political masters will judge them is a factor over which the politicians, not the EA, have control. Imagine the reverse. If they had been given more money by government, would they have been judged to have performed even better. If this is a measure of good performance, why don't the politicians give them the money?

 

Partially achieved:-

1. "We have plans in place to improve local quality of life indicators". More measurement by delivery of plans rather than action. And they couldn't even succeed at that.

2. "Site air quality improvement conditions are on track". The improvement conditions are in place but not on track. Planning but not delivery again.

3. "The quality of rivers is getting better". It has to be said, they didn't miss their targets by much. Let them off this one.

4. "Increase the number of Catchment Abstraction Management Strategies (CAMS) published". More strategies, more paper. Another target that wasn't missed by much. But producing paper is their forte, so that's no excuse.

5. "More houses are protected from flooding". Should have protected 31,000 houses, only protected 29,000 houses. Pity about the rest, though. The targets look pretty arbitrary in the light of recent events.

6. "More flood risk management systems are at or above the target condition". Management systems again - should be the EA's bread and butter. But as we are now all too well aware, only "57 per cent of flood risk systems met their target condition including third party assets". They only had to get to 63% to hit their target, and they couldn't even manage that. They were patting themselves on the back for hitting their targets at the same time as admitting that nearly half the flood risk systems weren't ready.

7. "Leadership Group attendance at Leadership Programme". A target you can pass simply by turning up. They should make it a GCSE. And yet they didn't quite manage to hit their targets on this testing measure.

 

New baselines:-

1. "We take remedial action to improve SSSIs in unfavourable condition". There's nothing like prevarication to make a problem seem a little less serious.

2. "More regulated businesses comply with permit conditions." They appear to have achieved this target (or at least reduced their measurement of non-compliance). It's not clear why this is listed as having a new baseline.

3. "We identify and reduce illegal waste sites." Couldn't compare with before, because they had discovered there are more illegal waste sites out there than they thought. Which is a bit worrying, as this has been their responsibility for a while. Doesn't this count as failure?

4. "We are successful in taking action against those who damage the environment." Classical example of the difficulty in interpreting the data. Numbers of prosecutions fell, but was this because they had successfully reduced the number of serious incidents, or because they were being less diligent in pursuing culprits?

 

No score:-

1. "Water supply and demand is properly managed". Saved by our summer weather from yet more hosepipe bans. No wonder they didn't want to score themselves.

2. "Develop and improve the use of our management information systems to help drive better performance." Their summary of progress on this target would make every card a winner in buzzword bingo: "We have seen year-on-year improvement in corporate performance against Corporate Scorecard measures, which has been driven by regular management review. We have extended the ‘balanced scorecard’ as a performance management tool to more parts of our business and have trained staff in business planning and performance management. We have used zero-based planning techniques to review areas of our business and activity-based costing data has been used to inform our process re-engineering work programme."

3. "Key stakeholders agree we are good at working with them towards shared goals." Didn't complete the survey because of budget cuts. Everyone is a stakeholder in the environment, by definition. How big would the budget have had to be to get the views of all their key stakeholders. Or do they mean "the people we care about", rather than "the people affected"?

 

A well-run organization, with the right priorities, a clear sense of purpose and responsibility, and simple, comprehensible and justifiable objectives? Or a confused, lumbering behemoth? You decide.