Blog: Big or small - are numbers a tall order?
Blog by Helen Chambers
From my experience of the last three decades working in policy and service delivery, many people have a poor underpinning foundation of statistical analysis and struggle to meaningfully read and interpret data.
And for good reason. Most people in social policy, political and public service realms have had absolutely no training in how to go about this.
Tim Harford, the economist, Financial Times columnist and BBC broadcaster often asks in his programme More or Less “is this a big number?” when starting to get under the skin of any rather dubious number assertions.
In the UK, I don't think we ask ourselves often enough "is this a meaningful number?".
And often, even if we did, would people have the mental analytical tools to be able to give a competent answer?
An example lies within the statement, exaggerated for the purposes of illustration: “the success of this intervention has increased by 100%”. If the initial success rate was 0.0001% then increasing it to 0.0002% is close to meaningless; but often that second step of analysis isn’t taken and the initial declaration accepted.
You might find this example derisory. Perhaps you can find your way round a set of financial or economic outputs but how comfortable are you with a paired t-tests, chi square tests and p-values?
And what about your colleagues?
These are fundamental tools when it comes making sense of the world, especially in social policy. Development of policy, public services and their delivery sits across a very wide realm in Scotland: from small charities, the NHS, private companies and government at all levels.
The Scottish Government is able to retain the services of qualified and skilled analysts, but in many of the spaces that are crucial to changing the outcomes for communities, access to this type of support is absent.
So we rely on thousands of individuals being able to understand the material put in front of them in various oversight, governance and decision roles.
A high percentage of the time, people do not have significant comfort or understanding when they are reviewing the reams of data we are currently capable of producing and sharing with each other, in the name of evidence for decision making.
Developments in AI mean we are experiencing the dawn of some of the strongest analytical tools ever seen. But this brings risks, if we do not understand the data we are manipulating with AI or potential malevolent actors wish manipulate our perception and understanding of the world.
We are now at a point where this matters more than ever. AI could make things worse rather than better. In computing worlds there is the concept of GIGO, ”Garbage In, Garbage Out”.
Decision makers as well as the general public need to be able to sense check and interrogate numbers that are placed in front of us.
And beyond that, with so many organisations in a position of extreme resource squeezing, we have to know how to allocate assets whether financial, human or physical to have the greatest impact.
Right now I don't think we are in a position to do that at the level required. Perhaps it is time for a little more honesty, humility and access to Statistics 1.01, until more of us are confident in understanding and working with the data and evidence placed before us.
About Helen Chambers
Helen is a freelance consultant specialising in strategy, implementation, and influencing skills. She works with senior teams to optimise impact and resources for social good across the private, public and voluntary sectors. Her practice is underpinned by a belief in using strong critical thinking skills to deliver conscious intent.
Further reading and listening
Tim Harford’s More or less and Cautionary tales podcasts
Blog and discussion podcast from the Office for National Statistics, Communicating Uncertainty: how to better understand an estimate.
