Mastering the Art of Measurement – Drive Performance when Mere Reporting is not Enough

By Pat Shafer

A World of Opportunity

Imagine a world where metrics can identify issues before they occur, where forecasts are accurate, and stakeholders can perform effectively and grow in their jobs thanks to full process transparency. And imagine that these dashboards are automatically populated, forecasting models automatically updated, periodic reports automatically generated, and issues automatically identified and reported. We need not rely upon artificial intelligence, simply using the data and metadata within our systems (see figure below) can provide the insights to effectively manage and continuously improve regulatory performance.

Analytics support multiple capabilities

A Slow Start

Unfortunately, the biopharma and medical device industries have been lagging behind other industries in effectively using data and analytics to support business processes. A recent survey by the medical device innovation consortium (MDIC) highlights the current state of practice: Over 3/4 of companies surveyed still use manual steps to populate dashboards and reports. Just over 50% have live dashboards, and fewer than 10% are able to report metrics in real time. The vast majority of the respondents rely on lagging metrics to report performance, and a very few drive performance through leading indicators. Interestingly, just over 20% admitted to using metrics to punish low performers.

Our current work with life sciences companies tells us that most metrics programs are very compliance focused. Metrics highlighting past non conformances, CAPA aging and other QMS-related processes abound. Few have implemented a more mature, hierarchical approach.

Metrics that Work

Metrics must be fit-for-purpose. The C-Suite is often focused on key performance indicators (KPI’s) that report progress towards strategy over the past 30, 60, or 90 days. KPI’s do little to inform future direction. Senior management, those responsible for more tactical course setting and correction, rely on metrics reported in management reviews. Senior managers need to monitor trends and, as appropriate, take action or initiate more thorough investigations. Line managers and operational teams must be able to spot issues as they happen or, preferably, before they happen. Use of real-time data and predictive analytics is essential to identify opportunities for intervention in order to achieve performance goals.

An effective metrics hierarchy supports aligned, fit-for-purpose measurement

In all cases, strategy should drive what metrics are reported. As strategy changes, metric stewards must realign both the metrics and the target values. Metrics typically “roll up” from operational metrics to KPI’s, leveraging common data sources. In this way, the entire enterprise remains aligned.

Another key concept considers proactive versus reactive measures. One way to be proactive is to measure earlier in the value chain. Looking for quality or measuring progress early translates to less waste or greater opportunity to address delays. Going a step further, more mature companies identify leading indicators, metrics reporting forces that can predict or influence outcomes. Additionally, performance measurement supports the full regulatory life cycle. Every step, from submissions strategy through life cycle maintenance has related quality, progress or other critical metrics.

Measuring Intangibles

While many metrics have a clear, precise relationship to the data (e.g. using time stamps to calculate durations), other measures must rely on more or less loosely related data points. Many companies have recognized the significant influence of culture on performance. We consider culture as a leading or predictive measure. Yet most efforts to measure culture rely on periodic surveys that may or may not reflect broad participation. A more responsive approach is to identify surrogate measures that can reflect more timely, and in some cases, real-time data. Culture can be defined in many dimensions such as communication, empowerment, and sense of team. These in turn can be directly informed by structured data. For example, empowerment can be measured as a function of the percentage of employee-initiated changes or a monthly count of employees’ suggestions.

Use of “surrogate” metrics to measure culture

Implementation

There are many factors critical to the success of a regulatory performance measurement program. Analytics rely on accurate, high-quality data. Measurement programs must reflect and be applied consistently across the full ecosystem, affiliates and headquarters alike. They must be relevant to the work being performed. And perhaps most importantly, there must be a supportive culture. Leadership must position the metrics program as an opportunity for all stakeholders to benefit from the resulting transparency and insights. Metrics that trigger effective action or help identify a root cause are most valuable. When metrics are used to punish, there is a tendency to sabotage the program by corrupting data.

Best results are achieved when leadership and functional teams align with HR and Communications to introduce programs that support transparency, career growth and sustainability as well as performance. The first attempt will likely produce questionable results. Use initial readings to establish a baseline. The metrics themselves, as well as operational performance, will improve over time.

Pat Shafer
Managing Director
FTI Consulting
Pat.shafer@fticonsulting.com