NTSB Safety Recommendation P-05-002

NTSB safety recommendation P-05-002.

TO THE PIPELINE AND HAZARDOUS MATERIALS SAFETY ADMINISTRATION: Require pipeline companies to have a policy for the review/audit of alarms.

Priority: CLASS II

Overall Status: Closed - Acceptable Action

Issued Date: 2005-12-23

Adopted Date: 2005-11-29

Overall Date Closed: 2010-04-28

Synopsis: In the pipeline industry, Supervisory Control and Data Acquisition (SCADA) systems are used to collect data from pipeline sensors in real time and display these data to humans who monitor the data from remote sites and remotely operate pipeline control equipment. This study was designed to examine how pipeline companies use SCADA systems to monitor and record operating data and to evaluate the role of SCADA systems in leak detection. The number of hazardous liquid accidents investigated by the National Transportation Safety Board in which leaks went undetected after indications of a leak on the SCADA interface was the impetus for this study. The Safety Board developed a survey to obtain data about the liquid pipeline industry’s use of SCADA systems with input from industry. In addition to obtaining survey data, the Safety Board visited 12 pipeline companies that had operating SCADA systems. Based on information from previous accidents investigated by the Board, survey results, and site visit results, the Safety Board’s review of SCADA systems in the hazardous liquid pipeline industry uncovered five areas for potential improvement: display graphics, alarm management, controller training, controller fatigue, and leak detection systems.

Ntsbnumber: 80595

Report Number: SS-05-02

Addressee Name: PHMSA

Addressee Status: Closed - Acceptable Action

Addressee Date Closed: 2010-04-28

Addressee Acronym: PHMSA

Addressee Organization Type: G-Federal Government

Communication Date: 2008-12-23

Communication Type: NPRM Response

Communication Contents: The Safety Board notes that the NPRM would implement new requirements for control room management at 49 CFR Parts 192, 193, and 195 for natural gas pipelines, liquefied natural gas (LNG) facilities, and hazardous liquid pipelines, respectively. Under the proposed requirements, each operator of a natural gas pipeline, LNG facility, or hazardous liquid pipeline with a controller and control room would have to establish and follow written control room management procedures. The procedures also would have to be integrated into the operator's operations and maintenance manual, qualification program, and emergency plan, all of which are currently required under 49 CFR Parts 192, 193, and 195. The Safety Board further notes that in order to implement the proposed control room management procedures an operator would be required to do the following: define the roles and responsibilities of the controllers; provide controllers with accurate and timely system data; implement methods to prevent controller fatigue; ensure appropriate controller response to alarms and notifications when Supervisory Control and Data Acquisition (SCADA) systems are used; establish thorough and frequent communications among controllers, management, and field personnel when planning and implementing physical changes to pipelinelfacility equipment and configuration; review control room procedures following any event that must be reported under existing regulations; establish and implement a training program that includes an annual review to identify potential improvements; have a qualification program for controllers; and conduct an annual validation by a senior executive for the operator verifying that adequate control room management procedures have been implemented and are being followed. The Safety Board supports the overall direction of the NPRM and believes that the NPRM is comprehensive and focuses on the major elements of effective control room management. The Board also commends PHMSA for proposing to apply the standards to both natural gas and hazardous liquid pipelines and LNG facilities. Still, the Board has more detailed comments to provide concerning the specific safety recommendations discussed in the NPRM and the proposed requirements pertaining to mitigation of controller fatigue. The NPRM, if implemented, would require that each operator using a SCADA system ensure appropriate controller response to alarms and notifications. Operators would be required to review SCADA operations at least once a week and review SCADA configuration and alarm management operations at least once each calendar year but at intervals not to exceed 15 months. Notation 8070: The National Transportation Safety Board has reviewed the Pipeline and Hazardous Materials Safety Administration’s (PHMSA’s) notice of proposed rulemaking (NPRM), “Control Room Management/Human Factors,” that was published at 73 Federal Register 53076 on September 12, 2008. PHMSA is proposing to revise the Federal pipeline safety regulations, 49 Code of Federal Regulations (CFR) Parts 192, 193, and 195, to address human factors and other components of control room management. Overview The Safety Board notes that the NPRM would implement new requirements for control room management at 49 CFR Parts 192, 193, and 195 for natural gas pipelines, liquefied natural gas (LNG) facilities, and hazardous liquid pipelines, respectively. Under the proposed requirements, each operator of a natural gas pipeline, LNG facility, or hazardous liquid pipeline with a controller and control room would have to establish and follow written control room management procedures. The procedures also would have to be integrated into the operator’s operations and maintenance manual, qualification program, and emergency plan, all of which are currently required under 49 CFR Parts 192, 193, and 195. The Safety Board further notes that in order to implement the proposed control room management procedures an operator would be required to do the following: define the roles and responsibilities of the controllers; provide controllers with accurate and timely system data; implement methods to prevent controller fatigue; ensure appropriate controller response to alarms and notifications when Supervisory Control and Data Acquisition (SCADA) systems are used; establish thorough and frequent communications among controllers, management, and field personnel when planning and implementing physical changes to pipeline/facility equipment and configuration; review control room procedures following any event that must be reported under existing regulations; establish and implement a training program that includes an annual review to identify potential improvements; have a qualification program for controllers; and conduct an annual validation by a senior executive for the operator verifying that adequate control room management procedures have been implemented and are being followed. The Safety Board supports the overall direction of the NPRM and believes that the NPRM is comprehensive and focuses on the major elements of effective control room management. The Board also commends PHMSA for proposing to apply the standards to both natural gas and hazardous liquid pipelines and LNG facilities. Still, the Board has more detailed comments to provide concerning the specific safety recommendations discussed in the NPRM and the proposed requirements pertaining to mitigation of controller fatigue. Safety Board SCADA Safety Recommendations The Safety Board notes that the NPRM references the five safety recommendations, Safety Recommendations P-05-1 through -5, issued to PHMSA as a result of the Board’s 2005 safety study titled Supervisory Control and Data Acquisition (SCADA) in Liquid Pipelines. The NPRM specifically addresses Safety Recommendations P-05-1, -2, and -3, which, respectively, pertain to the use of graphics for SCADA systems, alarm management, and controller training. The three recommendations also were directly incorporated into the Pipeline Inspection, Protection, Enforcement, and Safety Act of 2006 (PIPES Act). These recommendations are listed below; a discussion of the related provisions in the NPRM follows. Require operators of hazardous liquid pipelines to follow the American Petroleum Institute’s Recommended Practice 1165 [API RP-1165] for the use of graphics on the SCADA screens. (P-05-1) The NPRM states that whenever a SCADA system is used, the operator must implement API RP-1165 in its entirety, unless the operator can adequately demonstrate that a provision of API RP-1165 is not applicable or is impracticable in the SCADA system used. Require pipeline companies to have a policy for the review/audit of alarms. (P-05-2) The NPRM, if implemented, would require that each operator using a SCADA system ensure appropriate controller response to alarms and notifications. Operators would be required to review SCADA operations at least once a week and review SCADA configuration and alarm management operations at least once each calendar year but at intervals not to exceed 15 months. Require controller training to include simulator or non-computerized simulations for controller recognition of abnormal operating conditions, in particular, leak events. (P-05-3) The NPRM proposed training provisions would require that training programs for controllers include the use of a simulator or noncomputerized (tabletop) method to enable controllers to recognize abnormal operating conditions, in particular leak and failure events. The Safety Board notes that under the NPRM, such simulations and tabletop exercises would include representative communications between controllers and individuals that operators expect to be involved during actual events. Further, controllers would also be required to participate in improving and developing tabletop or simulation training scenarios. The Safety Board believes that implement

Addressee Acronym: PHMSA

Addressee Organization Type: G-Federal Government

Communication Date: 2010-04-02

Communication Type: Official Correspondence

Communication Contents: The NTSB is currently reviewing the action taken by PHMSA on the above recommendations, identified in your letter, and expects to have a more substantive response back to you shortly. We apologize for the delay in responding regarding some of the recommendations; however, additional documentation was needed for these before their evaluation for closure could be completed. Action on Safety Recommendation R-89-53 (see enclosure) was completed on March 19, 2010.

Addressee Acronym: PHMSA

Addressee Organization Type: G-Federal Government

Communication Date: 2010-04-28

Communication Type: Official Correspondence

Communication Contents: The NTSB is pleased that the December 3, 2009, final rule addresses Alarm Management at 49 CFR Parts 192 and 195, requiring each operator using a SCADA system to ensure appropriate controller response to alarms and notifications and to review operations at designated intervals. Because the final rule satisfies the recommendation, P-05-2 is classified CLOSED -- ACCEPTABLE ACTION.

Addressee Acronym: PHMSA

Addressee Organization Type: G-Federal Government

Communication Date: 2009-09-22

Communication Type: Official Correspondence

Communication Contents: The NTSB is pleased that the September 2008 NPRM addresses alarm management for Parts 192, 193, and 195, requiring each operator using a SCADA system to ensure appropriate controller response to alarms and notifications. Operators are required to review SCADA operations at least once a week and review SCADA configuration and alarm management operations at least once per calendar year and at intervals not to exceed 15 months. The NTSB believes that if the applicable text in the proposed rule is adopted in its current form in the final rule, the recommendation will be satisfied. Accordingly, pending publication of the final rule, Safety Recommendation P-05-2 is classified OPEN -- ACCEPTABLE RESPONSE.

Addressee Acronym: PHMSA

Addressee Organization Type: G-Federal Government

Communication Date: 2008-03-18

Communication Type: Official Correspondence

Communication Contents: The Safety Board appreciates PHMSA’s comprehensive overview of action either underway or planned for implementation of these recommendations. The Board has reviewed the requirements of the PIPES Act, which, when implemented, will satisfy Safety Recommendations P-05-1 through -4. Section 19 of the PIPES Act requires implementation of Safety Recommendations P-05-1 through -3 by June 1, 2008, and implementation of Section 20 of the Act will satisfy Safety Recommendation P-05-4, as it requires PHMSA to amend the accident report form by December 31, 2007. Section 21 of the PIPES Act, as stated below, requires PHMSA to submit a report on leak detection systems with discussion about what can be done to foster development of more effective technologies. The Safety Board reminds PHMSA that to satisfy Safety Recommendation P-05-5, it needs to require operators to install computer-based leak detection systems. Not later than December 31, 2007, the Secretary of Transportation shall submit to Congress a report on leak detection systems utilized by operators of hazardous liquid pipelines. The report shall include a discussion of the inadequacies of current leak detection systems, including their ability to detect ruptures and small leaks that are ongoing or intermittent, and what can be done to foster development of better technologies as well as address existing technology inadequacies. The Safety Board is aware that the public comment period for PHMSA’s report was extended through January 18, 2008, and that PHMSA is currently compiling these comments before submitting the report to Congress. Because PHMSA is working to address the recommendations, Safety Recommendations P-05-1 through -5 are classified OPEN -- ACCEPTABLE RESPONSE, pending the completion of these efforts.

Addressee Acronym: PHMSA

Addressee Organization Type: G-Federal Government

Communication Date: 2006-08-08

Communication Type: Official Correspondence

Communication Contents: The Safety Board understands that PHMSA is monitoring the American Petroleum Institute's (API's) completion of the Recommended Practice and that API will consider the Board's recommendation P-05-1 while evaluating the draft Recommended Practice. Further, PHMSA's Controller Certification (CCERT) Project Team is examining the impact of SCADA graphics on controllers, the impact of alarms on controllers, controller training issues, and the need for data to evaluate controller fatigue. PHMSA announced in the May 1, 2006, Federal Register a public workshop to discuss opportunities to enhance the safety of pipeline control and findings from the CCERT Project. The Safety Board understood that the workshop would also assist PHMSA in completing the CCERT Project's report and support API's continued work on API RP 1165. The Safety Board would appreciate an update on the outcome of the workshop and how PHMSA now plans to address these recommendations. Pending a further response from PHMSA on these initiatives, Safety Recommendations P-05-1 through -4 are classified OPEN -- ACCEPTABLE RESPONSE.

Addressee Acronym: PHMSA

Addressee Organization Type: G-Federal Government

Communication Date: 2007-07-31

Communication Type: Official Correspondence

Communication Contents: Letter Mail Controlled 8/8/2007 2:24:08 PM MC# 2070395 - From Stacey L. Gerard, Assistant Administrator/ Chief Safety Officer: PHMSA has completed our CCERT project, discussed above in response to P-99-12, which covered various human factor control room issues. In January 2007, PHMSA submitted a report to Congress on the project that identified several areas for enhancing safety including improved graphics on SCADA screens, alarms, and training. The Pipeline Inspection, Protection, Enforcement, and Safety Act of 2006 (Act) requires PHMSA to issue regulations by June 1, 2008 that require operators to use the American Petroleum Institute’s Recommended Practice 1165, to review and audit alarm systems, and to develop training standards that include the recognition of abnormal operating conditions. The Act also requires PHMSA to submit a report to Congress on leak detection systems used by operators of hazardous liquid pipelines by December 3 1,2007. We are actively working on this report and intend to meet this deadline. PHMSA plans to include control room issues in a "Prevention Through People" regulatory effort that will incorporate the concepts of integrity management programs into risk-based regulations addressing human factors. PHMSA has begun work on this regulatory effort and expects to complete it this year. It will address both the Congressional direction and NTSB recommendations on use of graphics, review of alarms, controlling training, and fatigue. On May 23,2007 PHMSA held a public workshop that addressed best practices in addressing fatigue, man-machine interface, and qualifications and training and we are currently reviewing the workshop data. PHMSA has also begun work on a project to add data elements to accident reporting forms to capture information regarding the impact of fatigue on safety. PHMSA has ccnsistent!y pushed the hazardous liquid operators through integrity management to shore up the adequacy of their leak detection capabilities.

Addressee Acronym: PHMSA

Addressee Organization Type: G-Federal Government

Communication Date: 2010-02-01

Communication Type: Official Correspondence

Communication Contents: Letter Mail Controlled 2/3/2010 12:47:42 PM MC# 2100037 - From Cynthia L. Quarterman, Administrator: PHMSA has been in contact with the NTSB throughout the development of the Control Room Management (CRM) Rule. After several years of analytical work, PHMSA held public workshops on June 27,2006 and May 23,2007, as best practice forums to address fatigue, man-machine interface, and qualifications and training. The Notice of Proposed Rule-Making (NPRM) “Pipeline Safety: Control Room Management/Human Factors” was published on September 12,2008. After PHMSA considered 144 comments to the NPRM, the Technical Advisory Committees Meeting on December 1 1,2008, resulted in a nearly unanimous positive vote for CRM regulations. The Final Rule was published in the Federal Register (Vol. 74, No. 231, Page 63310) on December 03, 2009. Under the Final Rule, affected pipeline operators must define the roles and responsibilities of controllers and provide controllers with the necessary information, training, and processes to fulfill these responsibilities. Operators must also implement methods to prevent controller fatigue. The CRM final rule further requires operators to manage SCADA alarms, assure control room considerations are taken into account when changing pipeline equipment or configurations, and review reportable incidents or accidents to determine whether control room actions contributed to the event. In our January 2007 report to the Congress on the CCERT Project, PHMSA identified several areas for enhancing safety including management of alarm systems. The PIPES Act requires PHMSA to issue regulations requiring operators to review and audit alarm systems. The CRM final rule addresses both the congressional direction and the NTSB recommendation on use and review of alarm systems. The CRM final rule requires pipeline operators to develop written alarm management plans. These plans must include monthly reviews of data points that have been taken off scan or have had forced or manual values for extended periods. Operators will also need to verify correct alarm set-points, eliminate erroneous alarms, and review their alarm management plans at least annually. Operators will also be required to monitor the content and volume of activity being directed to their controllers (including alarms and actions directed to controllers from sources other than the SCADA system) at least annually. The American Petroleum Institute has established a committee to develop a recommended practice on Alarm Management. Recommended Practice 1167 is currently under development. The gas industry trade associations are also preparing guidance material to their industry sector. PHMSA will participate in and monitor the development of this guidance and best practice material, and consider such material in the development of operator inspection guidance.

Addressee Acronym: PHMSA

Addressee Organization Type: G-Federal Government

Communication Date: 2006-04-26

Communication Type: Official Correspondence

Communication Contents: Letter Mail Controlled 4/27/2006 9:59:08 AM MC# 2060218 4-26-06: - From Stacey L. Gerard, Assistant Administrator/ Chief Safety Officer: PFMSA is monitoring the American Petroleum Institute's (API) completion of the Recommended Practice and API has informed PHMSA of its intention to evaluate the draft Recommended Practice in light of the NTSB recommendations. PHMSA's Controller Certification (CCERT) Project Team is examining the impact of SCADA graphics on controllers, the impact of alarms on controllers, controller training issues, and the need for data to evaluate controller fatigue. This week PHMSA intends to announce in the Federal Register a public workshop in June to discuss opportunities to enhance the safety of pipeline control and findings from the CCERT Project. The workshop is a forum to discuss the adequacy of current regulations, some new concepts for improvement of control room management and operations, and for industry to provide feedback on any enhancements PHMSA offers for discussion on the topic. PHMSA would like to hear from the public and industry its experiences related to pipeline management processes, human fatigue issues, controller qualification training and other programs to assure the effective control of pipelines. Among the topics on the agenda will be the following: "Alarm and event management; "Shift scheduling and hand-off; "Current qualification programs; and "Need to assess knowledge, skills, and abilities such as eyesight and hearing. The discussion at this workshop will also aid PHMSA in completing the CCERT Project's report and support API's continued work on API RP 1 165. This discussion will enable us to provide a more informed response to these recommendations at a later date. We request the classification of this response to recommendations P-05-1 through P-05-4 as "Open-Acceptable Action." - Stacey L. Gerard, Acting Associate Administrator/Chief Safety Officer; (202) 366-4433.

Addressee Acronym: PHMSA

Addressee Organization Type: G-Federal Government

Communication Date: 2008-12-08

Communication Type: Official Correspondence

Communication Contents: Letter Mail Controlled 12/10/2008 3:05:08 PM MC# 2080723 - From Rick Kowalewski, Acting Assistant Administrator/ Chief Safety Officer: Safety recommendation P-05-02 asked that PHMSA require pipeline companies to have a policy for the reviewlaudit of alarms. In January 2007, PHMSA submitted a report to the Congress on the project that identified several areas for enhancing safety including improved graphics on SCADA screens, alarms, and training. On May 23, 2007, PHMSA held a public workshop that addressed best practices in addressing fatigue, man-machine interface, and qualifications and training. The PIPES Act requires PHMSA to issue regulations requiring operators to review and audit alarm systems. PHMSA issued an NPRM Pipeline Safety: Control Room Management/Human Factors on September 12,2008. The NPRM address both the congressional direction and the NTSB recommendations on use of graphics, review of alarms, controller training, and controller fatigue. PHMSA anticipates publishing a final rule in the summer of 2009.

Addressee Acronym: PHMSA

Addressee Organization Type: G-Federal Government

Communication Date: 2010-03-15

Communication Type: Official Correspondence

Communication Contents: Letter Mail Controlled 3/16/2010 1:16:29 PM MC# 2100096 - From Cynthia L. Quarterman, Administrator: I greatly appreciated meeting with you last month concerning the National Transportation Safety Board's (NTSB) recommendations to the Pipeline and Hazardous Materials Safety Administration (PHMSA). I look forward to meeting with you again, along with staff, on Tuesday, March 16th. Your recommendations, based on lessons learned from accident investigations, provide valuable safety information to our programs. We are committed toconsidering each of the recommendations and implementing those that are positive improvements in safety. I have been meeting with PHMSA's hazmat and pipeline safety programs to assess our actions on the NTSB recommendations. We are addressing these safety issues by taking actions to assure that the "unacceptable actions" are moved into the "open-acceptable" category and to achieve a "closed-acceptable" in a timely manner on as many recommendations as possible. I recognize that a number of "open-acceptable" recommendations are works in progress and maytake a year or more to complete. You indicated your interest in getting these issues resolved as well. As you requested, I asked our pipeline and hazmat staff to compile a copy of outstanding letters to the NTSB that request a change in the classification of a recommendation. I have attached a copy of those letters and am hopeful you and I can successfully resolve a number of these issues. In addition, I have askedour Chief Safety Officer, Cindy Douglass, and our Associate Administrators, Jeff Wiese for Pipeline Safety and Magdy El-Sibaie for Hazardous Materials Safety, to meet with your staff to help us better understand each recommendation and to clarify the actions the NTSB considers necessary for closure. Again, I look forward to meeting with you on these safety concerns and believe that, together, we will make a positive difference in the safe transportation of hazardous materials, including those transported by pipelines.