During World War Two, the United Sates Air force (USAF) were tasked with the daytime bombing of enemy industrial and military locations. Raids were highly dangerous, with 6000 aircrew dying monthly, an average of 220 a day. By the end of the war 52,000 aircrew were reported dead or missing¹.
As the USAF sustained these heavy losses, a number of solutions were sought to improve the chances of survivorability amongst aircrew. The USAF decided to analyse hit points on bombers that returned from missions to determine where the planes should be better armoured.

Aircraft needed a positive ratio between speed and armour limited to the most vulnerable locations. A summary of the points of impact for returning aircraft is highlighted here with red dots.
The USAF argued for the armouring of wings, tails and central fuselage reflecting the most common points of impact. At this point, a naval mathematician called Abram Wald made a controversial suggestion. He argued that the data used to make this decision was flawed as it did not factor in any of the aircraft that didn’t return from their missions.

A ‘survivorship bias’ had selected only those planes that had returned, ignoring the data from the aircraft that were destroyed and were more likely to have been hit (and destroyed) elsewhere. Wald’s analysis suggested that the lost aircraft were likely to have been hit in the cockpit, engines and rear fuselage (causing the plane to split in half as shown below) rather than those illustrated in the USAF study. Wald concluded the hit points outlined by the initial study should be considered as areas that could sustain damage and still return home.

Wald’s model helped shape USAF strategy for the remainder of WW2, Korea and Vietnam and began to be used more broadly as a form of analysis in economics,science and the business world.

Survivorship bias illustrates the importance of data that is not available or accessible that might, in some meaningful way, impact the final outcome of an analysis. It overlooks factors and data sets that have failed to pass a selection process which can lead to a set of false conclusions.
Similar to ‘Black Swan’ events, which cannot be predicted or forecast, surviorship bias provides us with a framework to reappraise our approach to business. It allows us to address risk and outcomes through an understanding of ‘invisible or ignored’ data². Understanding the impact of this bias can help shape our approach to problem solving, managing our businesses and delivering projects and programmes effectively.

Survivorship bias and ‘invisible data’ can be observed across several areas in business and economics:

New Business Start Up: Entrepreneurs look at successful market leaders who have converted concepts into multi billion-dollar industries. Jeff Bezos at Amazon or Bill Gates at Microsoft are often cited as examples of how entrepreneurs have started small and built scaled successful companies.
However, many aspiring entrepreneurs fail to look at the range of data that shows how most start-ups fail. The ‘invisible data’ outlined in the chart below, often fails to influence the strategy for entrepreneurs who show a bias towards those who succeed, rather than the volume of those who fail, and factoring that into their model³.

Supply Chain Logistics Costs: Supply chains are complex and expensive structures that are often targeted for cost savings by businesses through initiatives such as network optimisation, inventory reduction and outsourcing of manufacturing to CMOs⁴.
In the Logistics space, many large brokers operate and control high profile distribution routes (landing slots and sea routes), preventing smaller, and at times more efficient companies, from competing with their alternative distribution options.
Many of these smaller companies are ‘selected out’ of consideration by supply chain owners who may otherwise generate significant cost reduction.

Hidden Unemployment: Unemployment statistics are a critical indicator of economic strength and performance. Following the 2008 recession, subsequent UK governments were focussed on reducing the level of unemployment across the UK. However, many economic commentators in 2017 have argued that the official government figure of 1.5m unemployed ignored the nearly 800k individuals of working age that were claimants of incapacity benefits⁵.
This data that was ‘selected out’ of the official unemployment data both paints an inaccurate picture of overall unemployment numbers. In addition, as outlined in the image here⁶ this also prevents a clearer understanding of where this unemployment is most prevalent – in this case in the poorest areas of the UK. This diminishes the ability of UK policymakers to drive the right outcomes and economic decisions.

Returning Customer Revenue: Many companies calculate their ARR based on statistical data that reflects revenue from their current customer base. This fails to consider those that never returned or rejected the company for a competitor.
This can drive the wrong business culture that assumes the strategy is fit for purpose and that competitors do not pose a long-term threat. Considering the behaviour of ‘non customers’ is a critical piece of invisible data that should be used to influence strategy and assess actual organisation performance.

As observed above, data invisibility and survivorship bias are highly prevalent in many areas of business and economics. There is frequent and widespread ignorance of bias principles and selective use of some data sets while ignoring others. This is also common in the field of programme and project delivery.

Much of the PPM training materials available often provide an academic approach to delivery with standard phases of mobilisation, delivery and closure. Many case studies assume each project begins at ‘ground zero’ and should be mobilised using a consistent model or methodology (agile/waterfall).

Many projects are scoped and planned as individual pieces of work, and frequently fail to consider broader historical context of organisational behaviour, parallel or unrelated programmes and quite critically what has come before. In a similar vein to the armouring of aircraft, PPM delivery focuses on the tangible and ‘as is’ position rather than recognising the project has a range of data sets that are invisible and yet influence its future success. It is here, that the survivorship bias model and invisible data might be applied to delivering better programmes and projects.

Survivorship bias and ‘invisible data’ principles argue that key data is often ‘selected out’ of decision-making processes which in turn provides a less accurate picture of realty and negatively impacts on planned outcomes.
In the PPM space many new programmes of work begin with a blank sheet of paper and are assumed to be completely new activities undertaken by the organisation. Deploying an ERP system for the first time, undertaking a new acquisition or optimising a warehouse network all begin with the assumption that this is a new piece of work. Conventional risks such as cost, governance and desire for change are mapped as key constraints or unknowns until delivery begins.
However, many programmes of work fail to factor in invisible or unconventional data which may exist in abundance because it has already been selected out of consideration. Some of this data might include;

• Number of similar projects undertaken elsewhere which have succeeded/failed;
• Project success rates internally;
• Leadership sponsorship rates and pace of decision making;
• Historical adherence to milestones within the delivery organisation;
• Forecast spend accuracy and use of contingency;
• Rapidity of risk mitigation;
• Ratio of project delivery times vs churn rate of staff;

Building on the idea that these sets of data which have been selected out of programme initiation and therefore expose the initiative to risk and failure, several key metrics might be considered to support a more data driven approach to an initiative outlined below;

This paper has attempted to outline the origin and principles of survivorship bias and its application across a range of economic and business principles. It has shown that the selection and subsequent exclusion of specific data sets out of statistical analysis can result in poor decision-making.
These principles can also be applied at a tactical PPM delivery level by seeking out data that is selected out for consideration and yet would have a significant impact on understanding the viability and potential success of new initiatives.
The key final consideration of this paper is that when undertaking PPM delivery, the programme or project manager should apply not only the conventional tools of planning, risk management and financial analysis but also lateral thinking and innovative practices. Seeking out thematic invisible data that may provide an insight into future behaviour and ultimate success should also be a key deliverable.
An initial set of proposed metrics have also been outlined in this paper and will be further explored in following white papers during 2019.

Seeing the invisible makes the visible all the clearer.

For further information and to find out how P2 Consulting can help your business today please contact me:

[email protected]
: +44 (0) 7813 900337

References:
¹In contrast to this death toll, according to the US Department of Defence 4,424 US servicemen died in Iraq between 2003-2016
²For a detailed breakdown of Wald statistical approach see Mangel and Samaniego ‘Abraham Wald’s Work on Aircraft Survivability’ – Journal of the American Statistical Association Volume 79 (June 1984)
³www.failory.com
⁴Contract Manufacturing Organisations
⁵https://www4.shu.ac.uk/research/cresr/sites/shu.ac.uk/files/real-level-unemployment-2017.pdf
⁶ibid
⁷Annual Recurring Revenue