NTSB Safety Recommendation P-14-003

NTSB safety recommendation P-14-003.

TO NISOURCE, INC. (FORMERLY COLUMBIA GAS TRANSMISSION CORPORATION): Modify your supervisory control and data acquisition system to (1) provide the controller with operating parameter trend data that can be used to evaluate the significance of a system change and (2) assign an alarm function to trends that are likely significant system malfunctions and therefore require immediate action by the controller.

Priority: CLASS II

Overall Status: Closed - Acceptable Action

Issued Date: 2014-03-05

Adopted Date: 2014-02-19

Overall Date Closed: 2017-12-15

Synopsis: On December 11, 2012, at 12:41 p.m. eastern standard time, a buried 20-inch-diameter interstate natural gas transmission pipeline, owned and operated by Columbia Gas Transmission Corporation, ruptured in a sparsely populated area, about 106 feet west of Interstate 77 near Route 21 and Derricks Creek Road, in Sissonville, West Virginia. About 20 feet of pipe was separated and ejected from the underground pipeline and landed more than 40 feet from its original location. The escaping high-pressure natural gas ignited immediately. An area of fire damage about 820 feet wide extended nearly 1,100 feet along the pipeline right-of-way. Three houses were destroyed by the fire, and several other houses were damaged. There were no fatalities or serious injuries. About 76 million standard cubic feet of natural gas was released and burned. Columbia Gas Transmission Corporation reported the cost of pipeline repair was $2.9 million, the cost of system upgrades to accommodate in-line inspection was $5.5 million, and the cost of gas loss was $285,000. Major safety issues identified in this investigation were external corrosion mitigation of the ruptured pipeline, supervisory control and data acquisition alert setpoint configuration, use of automatic shutoff valves and remote control valves to improve isolation of high-pressure pipelines, and exclusion of pipelines in the vicinity of highways from integrity management regulation. The National Transportation Safety Board makes safety recommendations to Columbia Gas Transmission Corporation and the Pipeline and Hazardous Materials Safety Administration.

Ntsbnumber: DCA13MP003

Report Number: PAR-14-01

Addressee Name: NiSource, Inc. (formerly Columbia Gas)

Addressee Status: Closed - Acceptable Action

Addressee Date Closed: 2017-12-15

Addressee Organization Type: P-Private Industry

Communication Date: 2017-08-14

Communication Type: Official Correspondence

Communication Contents: -From Charles (Randal) R. Broussard, TransCanada US Gas Operations, Senior Vice President, Operations: On behalf of TransCanada US Gas Operations, provided are the annual update to the National Transportation Safety Board (NTSB) with respect to the recommendations issued by the NTSB to Columbia Pipeline Group (CPG) in its report on the rupture of Line SM-80 in Sissonville, WV in 2012 (the Sissonville Report).1In addition to the annual update, we are requesting a determination from the NTSB that the two remaining open recommendations contained in the Sissonville Report (recommendations P-14-2 and P-14-3) should be classified Closed—Acceptable Action. Over the last three years, CPG has dedicated significant effort in carefully evaluating and implementing meaningful processes within the organization to address the recommendations contained within the Sissonville Report. Following the NTSB determination in 2016 that recommendation P-14-4 should be classified as Closed— Acceptable Action, CPG strongly considers that recommendations P-14-2 and P-14-3 should also be classified as Closed—Acceptable Action. This letter identifies the measures taken by CPG to address NTSB recommendations P-14-2 and P-14-3, which relate to the Control Room response following the incident that occurred on Line SM-80 in Sissonville, WV. This work was broken down into four main efforts: (1) removing the mental tally required by Controllers when confronted with multiple alerts, (2) providing easy access to additional information when alerts, alarms, or communication failures are received, (3) adding alarms to reinforce the severity of an event, and (4) defining expectations through documentation and training, around Controller actions and response times. In addition to providing detailed information on actions undertaken within the past year, this letter summarizes work that was initiated in prior years and reported through annual updates previously submitted to the NTSB. Additional details on work undertaken in prior years are contained in the annual updates submitted to the NTSB in 2015 and 2016. Thank you for the opportunity to provide this follow-up information. We look forward to a positive disposition on the remaining two open recommendations. CPG is continuing to drive improvement through internal initiatives, with an emphasis on its Plan-Do-Check-Act principles and a Root-Cause-Analysis program, as detailed below. CPG remains committed to advancing public safety and striving for zero incidents. CPG is requesting that the NTSB classify Recommendation P-14-3 as Closed—Acceptable Action based on the work described in this letter. In the Sissonville Report, NTSB identifies the purpose of Recommendation P-14-3 as for CPG to provide immediate access to additional information for the Controller using trends and alarms and to make certain that significant occurrences result in immediate Controller action. CPG has addressed this recommendation in two parts below. Part 1 describes the work performed to incorporate additional information into the SCADA display for alerts, alarms, and communication failures. This additional information enhances the ability of the Controller to recognize the significance of a recurring alert, thereby reducing the overall response time. Part 2 of this response addresses the alarm portion of the recommendation related to “significant malfunctions” and actions taken by CPG to make certain that critical events on the pipeline generate an alarm in the Control Room. Part 1: P-14-3(1) The ability to readily access and view key SCADA trends and additional information relevant to incoming alerts, may have assisted and even accelerated the recognition of the severity associated with the Sissonville incident. CPG has performed a comprehensive review of the SCADA system and how Controllers navigate through the screens. Based on this review, CPG has implemented modifications that provide Controllers immediate access to additional information such as: point description, time of latest refresh, trend data or historical events, status, safety related classification, as well as other relevant data for evaluating the significance of the system change. This modification has been fully implemented across CPG’s SCADA system. Starting in April 2013, CPG implemented a new method for acknowledging alerts which includes a pop-up dialog box for each alarm, alert, or communication failure (Figure 3). This required developing functionality that was not part of the existing SCADA system. Prior to implementing this new feature, the existing SCADA system allowed Controllers to acknowledge alarms, alerts, and communication failures from a single summary page that presented numerous pieces of information in tabular format. This tabular summary page was passive and offered Controllers a limited amount of information. Under the new method, when the controller clicks the row header, a pop-up dialog box appears (centered on the display). The pop-up dialog box provides additional information about the point and makes certain that Controllers are acknowledging the correct alert, alarm, or communication failure. Consistency was maintained on the summary page and no changes were made to the visualizations. Table 2 includes a listing of the features associated with the new pop-up dialog box. Figure 3 shows a screenshot of the pop-up dialog box as it appears in SCADA. Figure 4 shows the pop-up dialog box associated with an analog point (e.g. pressure point). When the alert or alarm is tied to an analog point the pop-up dialog box includes a 1-hour trend so a Controller can quickly determine whether the trend is ascending or descending. The creep alert value is also shown and can be adjusted from within dialog box. If the Controller wants an updated or refresh of the SCADA point from the Remote Terminal Unit (RTU), a “Poll Now” button is available. Figure 5 shows how the pop-up dialog box appears for a discrete point (e.g. valve state open or closed). For discrete alerts or alarms, where a trend is not possible, the dialog box includes the ten previous events so a Controller can quickly see the history of the changes in state. If the Controller wants an updated or refresh of the SCADA point from the RTU, a “Poll Now” button is available. Figure 6 shows how the pop-up dialog box appears for a communications failure. Similar to the display for discrete points, the communications failure dialog box includes the ten previous events so a Controller can quickly see the history associated with the communication failure. If the Controller wants an updated or refresh of the SCADA point from the RTU, to determine if the communication failure will clear, a “Poll Now” button is available. Part 2: P-14-3(2) Following an extensive review of the Control Room response the SM-80 rupture and the NTSB recommendation for additional SCADA measures, Rate of Change (ROC) alarms were introduced to highlight potential significant system changes that would require immediate action. CPG has now enabled ROC alarms across its entire transmission system including all active storage facilities. In addition, Controllers have been trained to recognize and respond to ROC alarms. In May 2014, CPG Gas Control chartered a Steering Committee to investigate the implementation of Rate of Change (ROC) alarms in the SCADA system. Core team members included the Director of Gas Control, all Gas Control Managers, and one Operations Support Analyst. The goal of the Steering Committee was to determine the feasibility of ROC alarms, beneficial areas of use, and limitations of ROC alarm setpoints. During the first phase of the study, a sample population of pressure readings were selected from various locations to represent a multitude of installations such as single line operation (e.g. consistent pressures), looped line operation (e.g. varying pressure profile), lines near major mar

Addressee Organization Type: P-Private Industry

Communication Date: 2014-06-05

Communication Type: Official Correspondence

Communication Contents: -From Shawn L. Patterson, President, Operations and Project Delivery: We have reviewed our process for acknowledging alerts within our SCADA system. In April 2013, we implemented a new method for acknowledging alerts which features a pop-up dialog box for each alarm, alert, or communication failure. Previously, controllers acknowledged alarms, alerts, and communication failures from a summary page – a method which provided controllers with limited information. Under the new method, when the controller clicks the row header, a pop-up dialog box opens. This provides more information and ensures that controllers are acknowledging the correct point. No visual changes were made to the summary page. The new pop-up dialog boxes have the following features: •?Point Name •?Point Description •?State: Normal vs. Alarm/Alert type (analog only) •?Remote: shows the RTU containing the point (analog and discrete only) •?Point type: telemetered vs. calculated (analog and discrete only) •?Last Krunch (Update): shows date and timestamp of last update •?Alarm/Alert Message: shows current alarm/alert message •?Safety Related Point Icon: indicates whether the point is considered safety related •?Event Button: shows events specific to the data point •?Priority Display Button: shows the home display associated with the data point •?Email Monitoring Center Button: sends an email with the alarm/alert directly to the Monitoring Center; a Monitoring Center log is automatically generated •?Acknowledge Button: used to acknowledge the alarm, alert, or communication failure •?Dismiss Button: closes the dialog box For Analog Alarms or Alerts, the pop-up dialog box includes a 1 hour trend so that a controller can quickly determine the direction the value is trending. For analog points, the creep alert value is also shown and can be adjusted from the dialog box. Shown below is an example of an analog alert dialog box. For Discrete Alarms or Alerts, the dialog box includes the 10 previous events so that a controller can quickly see the history of the changes in state. Shown below is an example of a multistate (discrete) alarm dialog box. For Communication Failures, the dialog box includes the 10 previous events so that a controller can quickly see the history of the communication failure. The Communication Failure pop-up dialog box also includes a Poll Now button, which allows the controller to force a poll from this display to determine if the communication failure will clear. Shown below is an example of a communication failure dialog box. We are currently investigating the feasibility of using rate-of-change alarms. This specific type of alarm would be used for identifying trends that are likely significant system malfunctions and therefore require immediate action by the controller. A rate-of-change alarm is used to detect changes in a measured value in units per second. The rate-of-change alarm monitors an input for a change in value over time. The alarm is set to trip when the input rate-of-change exceeds a defined criterion over a specified period of time. An abnormal variation of data, such as a sudden increase or decrease in operating pressure, would trigger the rate-of-change alarm. While data may not be high or low enough to trigger a basic alarm, unusually rapid fluctuations in value can indicate abnormal operating conditions. Many factors must be considered when determining rate-of-change alarm set points so that normal operating conditions that result in rapid fluctuations in pressure do not present false alarms to controllers. Some of these factors that occur often include: •?Starting/Stopping Compression •?Change in regulator set points •?Fluctuating market conditions or interconnects •?Pipeline diameter •?Isolated vs. parallel pipelines •?Distance between telemetered points In order to determine the feasibility of using rate-of-change alarms, we have selected 18 points across our system and are currently evaluating possible rate-of-change alarm occurrences using 2 years of historical data and various rates of change set points. The goal of this study is to determine the feasibility of establishing rate-of-change alarm set points that will notify controllers of an abnormal operating condition without causing additional false or nuisance alarms. Future activities being considered for the feasibility study include performing a trial using some of the points that have been evaluated so far, training controllers on responding to rate-of-change alarms, and selecting additional points for evaluation. We plan to complete the rate-of-change alarm feasibility study by December 2014.

Addressee Organization Type: P-Private Industry

Communication Date: 2015-01-15

Communication Type: Official Correspondence

Communication Contents: -From Shawn L. Patterson, President, Operations and Project Delivery: Columbia Gas has reviewed our process for acknowledging alerts within our SCADA system. In April 2013, Columbia Gas implemented a new method for acknowledging alerts which features a pop-up dialog box for each alarm, alert, or communication failure. Previously, controllers acknowledged alarms, alerts, and communication failures from a summary page – a method which provided controllers with limited information. Under the new method, when the controller clicks the row header, a pop-up dialog box opens. This provides more information and ensures that controllers are acknowledging the correct point. No visual changes were made to the summary page. The new pop-up dialog box has the following features: ??Point Name ??Point Description ??State: Normal vs. Alarm/Alert type (analog only) ??Remote: shows the RTU containing the point (analog and discrete only) ??Point type: telemetered vs. calculated (analog and discrete only) ??Last information update: shows date and timestamp of last update ??Alarm/Alert Message: shows current alarm/alert message ??Safety Related Point Icon: indicates whether the point is considered safety related ??Event Button: shows events specific to the data point ??Priority Display Button: shows the home display associated with the data point ??Email Monitoring Center Button: sends an email with the alarm/alert directly to the Monitoring Center; a Monitoring Center log is automatically generated ??Acknowledge Button: used to acknowledge the alarm, alert, or communication failure ??Dismiss Button: closes the dialog box For Analog Alarms or Alerts, the pop-up dialog box includes a 1 hour trend (see Figure 3) so that a controller can quickly determine the direction the value is trending. For analog points, the creep alert value is also shown and can be adjusted from the dialog box. Shown below is an example of an analog alert dialog box. Figure 3: Sample of an analog alarm/alert pop-up dialog box with 1-hour trend For Discrete Alarms or Alerts, the dialog box includes the 10 previous events so that a controller can quickly see the history of the changes in state (fault, out of service, available, running, etc.). Figure 4 below is an example of a multistate (discrete) alarm dialog box. Figure 4: Example of a multistate (discrete) alarm dialog box For Communication Failures, the dialog box includes the 10 previous events so that a controller can quickly see the history of the communication failure. The Communication Failure pop-up dialog box also includes a “Poll Now” button, which allows the controller to force a poll from this display to determine if the communication failure will clear. Shown below is an example of a communication failure dialog box. Figure 5: Example of a communication failure dialog box In May 2014, Columbia Gas’ Gas Control Group formed a team to investigate implementing Rate of Change (ROC) alarms in the SCADA system. Core team members included the Director of Gas Control, Gas Control Managers, and Operation Support Analysts. The goal of the ROC Team was to determine the feasibility of ROC alarms, beneficial areas of use, and limitations of ROC Alarm setpoints. For the first phase of the study a small set of pressures were used from various types of locations including single lines, looped lines, and near major markets. Data was scrutinized over a 24-month period in order to capture multiple seasons and variable flow scenarios. The study concluded that a 0.2 lbs per second ROC setting would be used as a starting point. The ROC Team acknowledged that 0.2 lbs per second may not work in every scenario and that adjustments may be necessary after implementation. The second phase of the study included taking a look at each Area of Responsibility (AOR) using a larger set of points. The goal for this phase was to confirm the 0.2 lbs per second setting. This study confirmed the 0.2 lbs per second setting. Data from the SM-80 incident was also reviewed to confirm that the 0.2 lbs per second setting would have resulted in a ROC alarm immediately after the incident occurred. Gas Controllers were trained on all aspects of ROC alarms. Training included how the alarms would be presented in SCADA along with actions to be taken for the alarms. The SCADA Information Technology Team has completed the necessary modifications to the alarm screen. Similar to other alarm setpoints, only the Gas Control managers will have the authority to set and modify ROC alarm setpoints. In addition, a “Poll Now” button has been provided on the alarm pop-up panel. The controller can manually force a single poll in the event of an ROC alarm to retrieve data before the next scheduled poll. This will reduce delays in investigating ROC alarms. A small group of pressures on one of our systems were selected to have ROC limits implemented in late September. A trial was completed by lowering the ROC setpoint in order to force an alarm. This confirmed that the ROC alarm was working as expected. The pilot program is still ongoing. At this time, the responsibility for implementing ROC alarms and documenting the process was passed to the newly formed team that is focusing on alarm and alert management. This team is mentioned above in the response to P-14-2.

Addressee Organization Type: P-Private Industry

Communication Date: 2016-04-13

Communication Type: Official Correspondence

Communication Contents: -From Shawn L. Patterson, Executive Vice President and Chief Operating Officer, Operations and Project Delivery: Item 1 response: In the update submitted to the NTSB on January 15, 2015, Columbia Gas described a new method for acknowledging alerts that features: a pop-up dialog box containing point description, time of latest update, trend data, historical events, current status, safety related classification and other data that aids the controllers in evaluating the significance of the system change. Controllers have been trained on this method for acknowledging alerts, and it has been fully implemented across all of the Columbia Gas company SCADA systems. Item 2 response: A team was formed to study the implementation of Rate of Change (ROC) alarms across the Columbia Gas system. The team determined the feasibility, benefits, limitations and initial settings for ROC alarms. The team identified an initial group of points across the Columbia Gas companies system and enabled ROC alarms for this group. After the points were monitored and the settings were deemed to be adequate, the team enabled ROC alarms on other groups of points based on geographical area. Controllers have been trained on the ROC alarms, possible causes of false ROC alarms and how to react to them. Columbia Gas currently has ROC alarms enabled across its footprint and incorporated into its comprehensive alarm/alert collection. The team is researching possible additional implementation sites.

Addressee Organization Type: P-Private Industry

Communication Date: 2017-12-15

Communication Type: Official Correspondence

Communication Contents: We note that you have incorporated additional information into your SCADA system display for alerts, alarms, and communication failures to enhance your pipeline controllers’ ability to recognize the significance of a recurring alert. We further note that, to make certain that significant occurrences result in immediate controller action, you have made revisions to ensure that critical events on the pipeline generate an alarm in the control room. These actions satisfy Safety Recommendation P-14-3 which is classified CLOSED--ACCEPTABLE ACTION.

Addressee Organization Type: P-Private Industry

Communication Date: 2015-03-17

Communication Type: Official Correspondence

Communication Contents: We note that you are working to implement a process for enhancing your supervisory control and data acquisition system to improve controller response time, are revising your Alarm Management Philosophy and Alarm Management Plan, and expect to implement the resulting new procedures this year. We further note that you will develop metrics that measure alert response time(s), will incorporate a review of these metrics as part of your alarm/alert review process, and, as part of the new process, will record the user and machine name along with the point or alarm data, as appropriate to the message. We also understand that you will train controllers on all new and updated procedures. Pending completion of these efforts, Safety Recommendation P-14-2 and -3 are classified OPEN—ACCEPTABLE RESPONSE.

Addressee Organization Type: P-Private Industry

Communication Date: 2014-07-29

Communication Type: Official Correspondence

Communication Contents: We understand that you reviewed the process for acknowledging alerts within your SCADA system and in April 2013, you added a new method for acknowledging alerts that features a pop-up dialog box for each alarm, alert, or communication failure; provides more information; and ensures that controllers acknowledge the correct point. We note that, in addition, you are investigating the feasibility of using a rate-of-change alarms plan and that you expect to be finished with this study in December 2014. Pending our receipt and review of the results of that study, Safety Recommendation P-14-3 is classified OPEN—ACCEPTABLE RESPONSE.

Addressee Organization Type: P-Private Industry

Communication Date: 2016-05-18

Communication Type: Official Correspondence

Communication Contents: We note that a team formed to study the implementation of rate-of-change (ROC) alarms across the Columbia Gas system determined the feasibility, benefits, limitations and initial settings for ROC alarms. After the points were monitored and the settings deemed to be adequate, the team enabled alarms on other groups of points based on geographical area. We understand that controllers have been trained on these alarms, possible causes of false alarms, and how to react to them. Currently, Columbia Gas has enabled ROC alarms across its footprint and has incorporated them into its comprehensive alarm/alert collection. Further, the team is currently researching possible additional implementation sites. Pending completion of these efforts, Safety Recommendation P 14-3 is classified OPEN—ACCEPTABLE RESPONSE.