NTSB Safety Recommendation Letter P-98-021 through P-98-023

Page 1Official PDFNational Transportation Safety Board Washington, D.C. 20594 Safety Recommendation Date: In reply refer to: P-98-21 through -23 Ms. Mary Ellen Peters President Marathon Ashland Pipe Line LLC 539 Main Street Findlay, Ohio 45840 On May 23, 1996, a 68-mile-long, 20-inch-diameter steel pipeline owned by Marathon Pipe Line Company ruptured at a location near Gramercy, Louisiana. The rupture went undetected by the pipeline controller for about 1 hour. The ruptured pipeline ultimately released about 475,000 gallons of gasoline into a common pipeline right-of-way within a designated “wetland.” Gasoline also entered the Blind River, causing environmental damage and killing fish, wildlife, and vegetation in the area. The National Transportation Safety Board determined that the probable cause of the accident was damage done to the Marathon pipeline during excavations of a nearby pipeline operated by LaRoche Industries, Inc., which resulted from the failure of LaRoche either to take adequate measures to ensure that excavations performed under its supervision did not damage underground utilities or to notify Marathon that those excavations may have damaged the Marathon pipeline. Contributing to the severity of the accident was Marathon’s delay in recognizing the rupture, which delayed shutting down the pipeline and isolating the rupture. Shortly after the rupture, the supervisory control and data acquisition (SCADA) system in the Findlay, Ohio, control center displayed alarms consistent with a leak occurring between the Garyville and Zachary stations in Louisiana. The pipeline controller observed and acknowledged the series of alarms, but he did not associate the alarms with a possible pipeline leak until about an hour after the rupture had occurred. The Safety Board investigation determined that this delay was due to the fact that the pipeline operator did not immediately interpret the SCADA alarms as indicating a leak and instead associated the alarms with more common problems that could result in similar SCADA indications. When the leak occurred, the SCADA system correctly reported that certain pumps had shut down at the Garyville refinery. Within 2 minutes, the SCADA system reported a line balance alarm and displayed a message indicating that less product was exiting the pipeline at Zachary#

Page 22 than was being introduced at Garyville. The pipeline controller, however, because he believed that the alarms were due to activity at the Garyville refinery, did not read the entire displayed message on the SCADA monitor, thereby missing the opportunity to interpret the information as a leak in the pipeline. When the SCADA system repeated a display of the product volume information an hour later, the controller did note the abnormally low volume at Zachary and began taking actions to deal with the leak. The controller said that after receiving the initial alarms, he called Garyville and discussed the situation with the station operator there. The station operator confirmed the pump shutdowns and informed the controller that the refinery was loading product to a barge.1 Even though refinery personnel reported that the volume of product being delivered to the barge was insufficient to have caused the SCADA system to alarm, the pipeline controller and the station operator concluded that the loading of the barge had precipitated the alarms and the pump shutdowns. The pipeline controller’s confidence that the problem was related to refinery operations may have lessened the value he placed on subsequent alarms. In other words, his anticipation of a particular series of alarms may have reduced his vigilance in monitoring the automation and its parameters. Consequently, although the controller did observe the line balance alarm, he had already assessed the situation and therefore did not examine the numerical data closely enough to recognize that they signaled a leak. Another possible reason the pipeline controller did not adequately attend to the alarms was the high concentration of alarms presented to him during a relatively short period of time. The line balance data was displayed during a high-workload situation, 11 seconds after the previous alarm and just 4 seconds before the next alarm. During periods of high workload, operators tend to focus on new alarm information at the top of a display panel or monitor and to ignore or attribute less importance to older alarm information as it scrolls down the display. As new data messages quickly displaced the line balance alarm from the top of the SCADA screen, the pipeline controller likely gave more attention to the incoming data and placed ever-diminishing value on older reports. When the controller was unable to restart the pumping units, he became uncertain about the true source of the problem. About an hour after the initial alarm, the SCADA system reported a second line balance alarm. This alarm was received more than 7 minutes after the most recent alarm and more than 1 minute before the next alarm. Probably because of a combination of the lower workload represented by the increased time between alarms and the pipeline controller’s renewed desire to find another explanation for the abnormal situation, the controller correctly attended to the latest data, made a quick interpretation of their significance, and took immediate action to remedy the situation. 1 According to Marathon employees, the high-volume delivery of product from the refinery to river barges sometimes decreased the pressure in the pipeline beyond a predetermined set point, which resulted in the automatic shutdown of pumping units and consequent SCADA system alarms. The resulting SCADA alarm messages were similar to what would be expected in the event of a leak except for the values of the numerical data associated with the line balance alarm.#

Page 33 Because the controller erroneously attributed the first line balance alarm to the effects of operations at the Garyville refinery, he did not access data that would have more conclusively indicated a leak. Although critical information was available from the SCADA system, the data were not displayed in a manner that prompted the controller to scrutinize them. System safety depends on equipment design that considers the needs of the human operator. In high-workload situations, for instance, pipeline controllers can be at a great disadvantage if actions taken by the automated systems are not clearly displayed.2 A problem in many complex systems is the lack of information salience that may accompany automation.3 Cluttered displays reduce the perceptual salience of information, even if the data are available. In a complex environment with many activities occurring simultaneously, controllers may easily lose track of such information. The investigation determined that Marathon’s SCADA system did not have adequate safeguards or redundancies to assist the pipeline controllers with detecting vital information. In this accident, the 1 hour that elapsed between the first and second line balance data alarms limited the pipeline controller’s opportunity to detect deviations in normal operating conditions and thus to determine that a leak had occurred in the pipeline. The pipeline controller reported that the SCADA system, as configured at the time of the accident, would typically report well over 100 alarms during a 12-hour shift. The majority of these were low-priority or informational alarms serving mostly to report that the operator had just made a particular SCADA input. Marathon pipeline controllers have termed these low-priority reports “nuisance alarms.” Because of the frequency of such alarms, pipeline controllers may not have responded to them with appropriate vigilance. The Safety Board notes that since the accident, Marathon has taken measures to improve the likelihood that pipeline controllers will be alerted to leaks and will respond appropriately. For example, Marathon has substantially reduced the frequency of nuisance alarms to ease the workload of the pipeline controllers and increase the likelihood of their detecting important changes to the operations. The Safety Board commends Marathon for taking such action but believes that additional changes are necessary to ensure timely response to future pipeline emergencies. Even though the controller determined shortly after 11 p.m. that a leak had occurred and took action to shut the line down and isolate the leak, Marathon crews did not complete manual closure of valves on either side of the rupture until about 2:30 a.m. The Safety Board notes that some hazardous liquid pipeline operators have installed remotely or automatically operated valves (including check valves) in their pipelines. Some of these operators have designed their systems carefully and have taken steps to prevent inadvertent valve closures or to avoid excessive pipeline pressures should a valve close unexpectedly. Some of these measures have included selecting appropriate controls and fail-safe positioning during communication failures, selecting optimum 2 Van Cott, H., Wiener, E., Wickens, C., Blackman, H., and Sheridan, T., “Smart Automation Enhances Safety: A Motion for Debate,” Ergonomics in Design, Vol. 4, No. 4, 1996. 3 Endsley, M.R. “Automation and Situation Awareness,” Automation and Human Performance: Theory and Applications, Parasuraman, R., and Mouloua, M. (Eds.), 1996, Lawrence Erlbaum Associates, N.J.#

Page 44 valve closure times, shutting down pumping stations during indications of possible valve closure or certain communications failures, carrying out strict maintenance procedures, and building in surge and lightening protection. The National Transportation Safety Board therefore makes the following safety recommendations to Marathon Ashland Pipe Line LLC: Use recurrent pipeline controller training to (1) emphasize the importance of carefully and completely reading the text of and evaluating all alarm messages, and (2) increase controller proficiency in interpreting and responding to control system data that may indicate a system leak. (P-98-21) Evaluate the effectiveness of alternative display formats and frequencies of alarming critical information for your supervisory control and data acquisition system and modify the system as necessary to ensure that controllers are specifically prompted to consider the possibility of leaks during system deviations that are consistent with a loss of product from a pipeline. (P-98-22) Evaluate remote and automatic valve control technology to facilitate the rapid isolation of damaged or leaking pipelines, and incorporate the appropriate valve control technology in your pipeline system, especially in those segments located in urban or environmentally sensitive areas. (P-98-23) Also, the Safety Board issued Safety Recommendation P-98-24 to LaRoche Industries, Inc. The National Transportation Safety Board is an independent Federal agency with the statutory responsibility “to promote transportation safety by conducting independent accident investigations and by formulating safety improvement recommendations” (Public Law 93-633). The Safety Board is vitally interested in any action taken as a result of its safety recommendations. Therefore, it would appreciate a response from you regarding action taken or contemplated with respect to the recommendations in this letter. Please refer to Safety Recommendations P-98-21 through -23 in your reply. If you need additional information, you may call (202) 314-6469. Chairman HALL, Vice Chairman FRANCIS, and Members HAMMERSCHMIDT, GOGLIA, and BLACK concurred in these recommendations. By: Jim Hall Chairman#