http://www.energypulse.net/centers/article/article_display.cfm?a_id=1113

Nuclear plant owners and asset managers use more than a hundred performance indicators (PIs) to monitor and analyze plant and organizational performance. The most common approach is to use benchmarks and other pre-defined limits to identify ranges of acceptable or desired performance. Similarly the Nuclear Regulatory Commission has increased its emphasis on objective data to assess licensee performance, relying on the Reactor Oversight Process (ROP) to evaluate each licensee on a quarterly basis. A benefit missing from the current approach to PIs and other performance measures has been the lack of forward looking or leading indicators that provide advance warning of degrading or trending unsafe performance. Our continued research into nuclear plant performance dynamics has led to the formulation of a new metric, Plant Consistency Measure, or CM, that offers benefits in signaling future performance risks.

Integrated Performance Model

Consideration of leading indicators was one issue addressed in the noted Arthur Andersen study performed for the NRC in 1996. (1) As part of its assessment of the NRC’s senior management meeting process, Arthur Andersen recommended that NRC management move their assessments of licensees “upstream” in the performance chain. The study suggested an “integrated performance model” where “results”, as measured by performance indicators, is at the far downstream end of the chain. In order to identify precursors to results, it is necessary to move back up the chain, to operational effectiveness, management effectiveness and external conditions. (2) The study recommended that the NRC use assessments of licensee management effectiveness to identify adverse trends prior to the time where results and corresponding performance indicators deteriorated.

However, the NRC determined that development of leading indicators of performance should not use licensee management performance or competency as an input. (3) Nor has there been agreement on alternate measures that could provide a comparable leading indicator of performance risk. A significant issue has been the availability of a simple measure or proxy for management effectiveness – one that did not require sensitive and intrusive assessments.

Our more than 15 years of consulting engagements with nuclear operators, including intensive interventions in organizations with significant performance problems, validate the concept of an integrated performance model. In our view, performance is best described as a system where key elements continuously interact to produce results. A system representation captures important “self-generated” behaviors that can strongly influence performance but are not as visible as other contributors. For example, resource allocation decisions which impact equipment reliability, set in motion consequences such as changes in equipment outages and corrective maintenance that are removed in time and which in turn create new demands on the performance management system. Within the system, management effectiveness has a pervasive impact on performance and is a determinant of resource utilization and allocation, regulatory performance, organizational effectiveness including safety culture, and on a wide variety of policies and performance standards. These directly affect operational performance and the risk of significant events and/or regulatory intervention.

Recently we have continued to explore the system dynamics of high reliability nuclear power operations. While our probabilistic simulations of plant performance provide powerful insights into expected performance, we wanted to identify simpler, objective metrics of performance reliability and their ability to correlate with indicators such as management effectiveness and organizational effectiveness. Our work led us to a new metric that we have labeled, plant performance consistency measure or CM.

Performance as Characterized by Capacity Factor

CM is derived from the most common existing measure of plant operating results, plant capacity factor (CF). CF is the product of the many human and equipment contributors that combine uniquely to determine a plant’s generating output in a given period. Generally CF is expressed as an annual value or an average value over several years or a fuel cycle (to account for refueling shutdowns that do not occur every year).

The nature of plant performance and the many factors that combine to determine net generation gives rise to an approach where CF can be expressed as a probability function. This formulation accounts for the relative likelihood of each combination of contributing events to determine the frequency and range of specific values of CF outcomes. We believe this approach provides the most insight into expected performance.

The following chart illustrates CF distributions for two plants: a relatively high performing plant and the other a lower performing plant. The overall probability distributions illustrate the relative likelihood of specific CFs occurring. Most of the points lie between 70% and 100% CF with the remainder spread over a “tail” ranging to very low values of CF. The average or mean values of CF are approximately 80% (lower performing) and 90% (high performing). (4) In addition each CF distribution can be characterized by its standard deviation, or how tight performance is around the mean. It is clear from inspection of the distributions that the SD for the low performing plant is greater than the SD for the high performing plant. (5) Another way of expressing the efffect of SD is the use of statistical confidence limits. For a specified CF value on the distribution, the associated confidence limit indicates the percentage of time that actual CFs would be greater than or equal to the specified CF. For example the 95% limits are shown on the figure. For the lower performing plant the 95% limit is a CF of approximately 67%, meaning 95% of expected CF outcomes will be equal to or greater than 67%. The mean value is simply the 50% confidence limit and is most closely associated with the annual or average CF values commonly reported.

Consideration of CF as a distribution of possible outcomes leads to the observation that both the mean and the standard deviation of the distribution are important to provide the most complete picture of performance. The key to reliable, consistent operation is for SD to be as small as possible, and anchored to a mean value of CF that is consistent with good industry performance.

Performance Characterized by Consistency Measure (CM)

The above observations about the nature of CF distributions suggested to us that a performance measure that embodied both the mean and standard deviation could be useful in more fully characterizing nuclear plant performance. In addition our work has indicated that consistency of performance is a differentiating attribute of well-managed and effective organizations. We hoped that a measure that properly captured performance consistency might provide a simple and accurate leading indicator of performance results.

We define performance consistency as the ability to produce operational results that are predictable, at levels consistent with good industry performance, and do not vary significantly from year to year. (6) Plant performance consistency encompasses but is, in effect, more stringent than plant capacity factor.

To develop a formulation for measuring performance consistency we elected to use confidence limits since they are designed to encompass a specified percentage of all outcomes. We selected the 95% confidence limit as an appropriate benchmark as it is the most commonly used.

Because confidence limits cannot easily be derived from year to year CF data, we needed to identify a general relationship between statistical confidence and standard deviation. The specific relationship is a function of the type of distribution that is associated with the distribution of CF data. For example, if the data were distributed in accordance with a normal distribution (symmetric about the mean), the 95% confidence limit would be 1.65 standard deviations from the mean. But nuclear plant CF data is distributed asymmetrically, limited on the upside by 100% and subject to a long “tail” on the downside where there are small, but finite probabilities of very low or even zero CFs. Thus CF data may be best characterized by various statistical distributions where the relationship between confidence limit and standard deviation are unique to each distribution. Our examination of a variety of nuclear plant CF data and distributions indicated that for these data, the 95% confidence limit was associated with 1.7 to 1.8 standard deviations. On this basis we selected 1.75 standard deviations as an appropriate basis for normalizing CF data against a 95% statistical confidence limit.

With the above correlation we were able to arrive at a formulation for plant performance consistency measure or CM, as follows:

Thus, CM is essentially the 95% lower limit of the CF distribution. A CM of 70 would mean that 95% of the actual CFs would be expected to equal or exceed 70%. In the preceding figure the lower performing plant has a CM = 67 while the higher performing plant has a CM = 81. Note that these values are lower than the mean CFs but represent minimum CF values that could be expected to be achieved with high confidence in each case.

Using this formulation of CM provides a balance of average performance (CF) and performance variability (SD). CF performance tends to reflect actual current or near term performance, while SD indicates the potential for performance to vary above or below the average. The relative weighting of CF and SD is also important. Our formulation assigns 1.75 times the weight for SD than for CF, reflecting our belief that consistency is somewhat more important and intrinsic to performance risk.

We also wanted to be able to calculate CM relatively simply. Since statistical distributions of CF by plant are generally not available, we have decided to use three year averages as the basis. Thus the value of CF in the CM formula is the running three year average of annual capacity factor (combined for multiple unit plants). The value of SD is also determined based on three years of annual CF data. The resultant CM value for a given period carries with it the performance of the most recent three years, buffering abrupt changes in CM from year to year.

CM as a Leading Indicator of Performance

To this point we have developed the definition of CM based on an understanding of the statistical characteristics of plant CF and established a reasonable metric for capturing the contribution of both current performance and performance variability. Our objective was to define a metric that could provide insight as to the future performance risk. Our belief was that performance did follow a hierarchy of influence similar to the Integrated Performance Model introduced at the beginning of this paper. Thus, the effectiveness of management performance is considered a precursor to the effectiveness of organizational performance (and we would add plant state including material condition and equipment reliability), which in turn is a precursor to the actual results that are observed through traditional performance indicators. With this in mind we undertook a two-step process to validate the potential predictive value of CM.

First we looked at determining if there was a reasonable correlation between CM and management effectiveness. As previously noted, assessments of management effectiveness are not the province of the NRC and if performed at all, require an intrusive and sensitive examination of the management team and process. Over a period from the early 1990s to the present we have been involved in a number of such assessments, generally at the request of the executive management or Board of Directors of the enterprise. Often these assessments have been triggered by serious lapses in performance and/or regulatory interventions. Others have been pursued within a continuous improvement/operational excellence context or as part of merger/acquisition or asset management transactions. Taken as a whole they cover a representative range of management capabilities and associated performance results.

We have combined the assessments of management effectiveness (quantified on a calibrated scale) (7) with computations of CM for the corresponding plants over the applicable periods of time. The next figure provides the results of the correlation of CM (bottom, red line) and management effectiveness (ME). A similar correlation is shown for CF (top, blue line) for comparison purposes. The figure shows increasing ME consistently correlates to increasing CM and that the regression of data leads to an exponential relationship as the best fit. The R2 value for regression indicates that data fit is quite good. In the mid range of ME, CM increases almost rapidly, then at higher levels of ME, CM change gradually achieves asymptotic response. (8) We believe there is strong intuitive appeal to this correlation.

By comparison, the best regression of data for the CF plot results in an overall linear relationship but an R2 value that is much lower than for CM. Of most interest is the behavior of the correlations over the left half of the plot, where management effectiveness is in the lower range. The CM plot shows a dramatic drop off (sometimes referred to as a tipping point) in the range of CM = 60 to 70, where small reductions in ME lead to large changes in CM. On the other hand, the CF plot does not display similar characteristics limiting its ability to generate a reliable signal that management performance may have reached a critical stage.

Our second test looked at the CM performance of actual plants over time to ascertain if the tipping point nature of CM could be validated. The next figure shows CM trendlines for U.S. nuclear plants in the period from 1990 to 2004. This period is interesting because there was significant differentiation in the quality of performance of plants across the industry, including plants that were subject to significant regulatory actions including extended shutdowns.

The figure plots trendlines of the average CM for each of three groups of plants. The top (green) and middle (red) trendlines on the figure are for plants that did not experience significant, performance related shutdowns. The green trendline is limited to plants that maintained CM above 60 (approximately 70 plants) and the red trendline is for plants where CM was below 60 for a period of time (approximately 16 plants). The bottom (black) trendline is the average CM track for plants (approximately 16) that eventually experienced an extended shutdown or were permanently shutdown based on degraded performance. CM values for these plants are based on the years prior to shutdown and following restart. The vertical line separates the performance history of each plant into pre-shutdown and post-shutdown. CM values during the shutdown period are not plotted. (9)

Several observations can be made based on these data. First, CM values have been increasing consistent with overall performance improvement in the industry. This reflects improvements in both average CF and SD. Second, CM < 60 appears to be a threshold or tipping point below which the risk of significant performance deficiencies becomes quite high. For plants operating in the gray zone of the chart, 50% experienced an extended performance-related shutdown. Following shutdown and recovery, CM performance of these plants was raised to industry standards and associated lower performance risk. (10) The remaining 50% of plants were able to address their performance issues while avoiding shutdowns.

We examined one additional source of nuclear plant data to ascertain any linkage to CM. Over the last ten years approximately 25 nuclear plants have been subject to consolidation either through direct purchase, merger and acquisition activity of their owners, and/or 3rd party management contracts. (11) Of these transactions, plant performance issues including the transfer of nuclear operating performance risk, appear to have played a significant or determinative role for ten plants. For these ten plants we computed their 3 year average CF and CM prior to the consolidation activity (see table). The results indicate that low CM, in the 60 – 65 range, appears to be an accurate precursor of significant action by plant owners. In contrast, the CFs of most of the plants were greater than 80%, levels that would not normally signal unusual performance risk.

Concluding Thoughts

Development of objective performance indicators that provide enhanced understanding of nuclear plant operations continues to be a priority for the industry and regulators. The risk and the resources needed to address deficient performance increases as one moves downstream from first signals to visible results, making the value of early detection significant. The improving trend of current plants further increases the stakes while at the same time it may make such signals ever more difficult to distinguish.

Our work suggests that plant performance Consistency Measure shows promise in generating recognizable signals (i.e., CM < 60) when management may no longer be effective in rectifying adverse performance trends. Our conclusion is that performance below the threshold of CM = 60 creates a significant likelihood, perhaps on the order of 50%, of performance failure sufficient to impact asset value, and therefore sufficient to trigger mitigating action.

References:

(1) Recommendations to Improve the Senior Management Meeting Process, Arthur Andersen, December 30, 1996.
(2) Arthur Andersen used the term “economic stress” to refer to external conditions and focused on market forces coupled to deregulation of the electric utility industry.
(3 )STAFF REQUIREMENTS:SECY-98-059 - PROPOSED OPTIONS FOR ASSESSING THE PERFORMANCE AND COMPETENCY OF LICENSEE MANAGEMENT, June 29, 1998. The NRC opted to continue to infer licensee management performance based on inspections, assessments and event follow up activities
(4) Industry average CF for 2000-2004 was approximately 90.3%.
(5) Industry average SD for 2000-2004 was approximately 7.7%.
(6) Some variation is expected and, in fact, unavoidable. Refuel outages, generally occurring every 18 months, are the most significant contributors. For some plants there may be dispatch, environmental or other external factors that restrict operations but are not associated with performance reliability. With plant aging there has also been increased needs for extended shutdowns for major equipment replacements such as steam generators and reactor vessel heads. To the extent these are known in advance and executed per plan, they do not contribute to unreliability.
(7) Management effectiveness is based on our assessment of the management performance at various plants. The assessments use anchored scales, which are numerical scales with verbal descriptions at various numbers. Rating statements are associated with numerical values between 1 - 10, producing the desired quantitative output. Our quantification of ME values is confidential due to the sensitive nature of the engagements from which data were derived.
(8) As higher levels of performance are achieved there is a diminishing amount of additional improvement possible notwithstanding improving management. The limits are both physical (e.g., 100% limit for CF) and irreducible in that there are contributors to lost performance (e.g., random errors or failures) that are essentially beyond direct management control.
(9) CM values during the shutdown period are generally not particularly illuminating since it is really a shutdown that is being managed, not plant operations.
(10) Analysis of O&M spending for plants of varying CM demonstrate a key additional benefit of consistent performance – reduced operating costs.
(11) Most recently, Nebraska Public Power District entered into a management agreement with Entergy Nuclear for Cooper Nuclear Station.