NTSB Safety Recommendation Letter P-02-004 through P-02-005

Page 1Official PDFNational Transportation Safety Board Washington, D.C. 20594 Safety Recommendation Date: October 11, 2002 In reply refer to: P-02-4 and -5 Honorable Ellen G. Engleman Administrator Research and Special Programs Administration 400 Seventh Street, S.W. Washington, D.C. 20590 About 3:28 p.m. on June 10, 1999, a 16-inch-diameter steel pipeline owned by Olympic Pipe Line Company ruptured and released about 237,000 gallons of gasoline into a creek that flowed through Whatcom Falls Park in Bellingham, Washington. About 1 1/2 hours after the rupture, the gasoline ignited and burned approximately 1 1/2 miles along the creek. Two 10-year- old boys and an 18-year-old young man died as a result of the accident. Eight additional injuries were documented. A single-family residence and the city of Bellingham’s water treatment plant were severely damaged. As of January 2002, Olympic estimated that total property damages were at least $45 million.1 The National Transportation Safety Board determined that the probable cause of the June 10, 1999, rupture of the Olympic pipeline in Bellingham, Washington, was (1) damage done to the pipe by IMCO General Construction, Inc., during the 1994 Dakin-Yew water treatment plant modification project and Olympic Pipe Line Company’s inadequate inspection of IMCO’s work during the project; (2) Olympic Pipe Line Company’s inaccurate evaluation of in-line pipeline inspection results, which led to the company’s decision not to excavate and examine the damaged section of pipe; (3) Olympic Pipe Line Company’s failure to test, under approximate operating conditions, all safety devices associated with the Bayview products facility before activating the facility; (4) Olympic Pipe Line Company’s failure to investigate and correct the conditions leading to the repeated unintended closing of the Bayview inlet block valve; and (5) Olympic Pipe Line Company’s practice of performing database development work on the supervisory control and data acquisition (SCADA) system while the system was being used to operate the pipeline, which led to the system’s becoming non-responsive at a critical time during pipeline operations. In December 1998, Olympic completed construction of the new Bayview products terminal about 2 miles upstream of the existing Allen station. Because the accident pipeline entering the terminal could be operated at pressures considerably higher than the pressure limit for the terminal, three control devices were employed to protect station piping and components 1 For additional information, see National Transportation Safety Board, Pipeline Rupture and Subsequent Fire in Bellingham, Washington, June 10, 1999, Pipeline Accident Report NTSB/PAR-02/02. 7264A#

Page 22 from overpressure. First, a control valve, CV-1904, was installed on the inlet side of the station and set at 600 pounds per square inch, gauge, (psig) to throttle back the flow of product into the station. Second, a relief valve, RV-1919, was installed just downstream of the control valve. The relief valve was designed to open and transfer excess product to the transmix tank if the pressure downstream of CV-1904 exceeded the set pressure of the relief valve. Finally, a receiver manifold arrangement, consisting of three motor-operated and remotely controlled block valves (MV-1902, MV-1903, and MV-1907) controlled product flow upstream of control valve CV-1904. Either MV-1902 or MV-1903, depending upon the selected configuration, was set to close in approximately 60 seconds and completely block the flow of product into the Bayview terminal if a set pressure of 700 psig was reached inside the facility. RV-1919 was an 8-inch Brooks Model 760 pilot-operated control valve2 manufactured by Fisher-Rosemount. The valve is designed to remain closed until the pressure in the pipeline on the inlet side of the valve reaches a predetermined pilot set point. When this pressure is reached, the pilot opens, allowing the relief valve itself to open and permit product flow through the valve. The Model 1760 pilot is available in either a low-pressure (0 to 180 psig) or a high- pressure (150 to 650 psig) configuration. The two configurations have different pistons, valve covers, and O-rings. Because an employee of the valve vendor apparently misinterpreted the valve specifications, the vendor configured RV-1919 as a low-pressure relief valve with a set point of 100 psig. Even though all the valve documentation and the valve itself indicated that RV-1919 was configured as a low-pressure valve, this went unnoticed by Olympic. During the night from December 16 into December 17, 1998, Olympic personnel began filling the pipeline to bring the Bayview facility into operation. The employees noted that as the accident pipeline filled and the pressure increased above 100 psig, RV-1919 opened and diverted product to a breakout tank. The employees recalled that the engineering manager was on the site during this activity and that when he noticed that the relief valve was operating at a pressure lower than intended, he reviewed drawings and directed efforts to determine why this was happening. The employees were aware that the available pressure range adjustment on the relief valve was limited by the type of pilot spring. Without consulting the manufacturer’s literature on the valve, which was available, the employees decided that they could increase the set point by replacing the pilot spring. One of the mechanics had a spring in his truck that he gave to another of the mechanics who used it to replace the existing pilot spring in RV-1919. The set point was then increased, after which the employees were able to fill and pressurize the pipeline. The mechanic was not aware that, because the same spring was used for either the 70- to 180-psig or the 350- to 650-psig pressure ranges, depending on the valve configuration (high or low pressure), the spring that he placed in RV-1919 was identical to the one he removed. He said that after he increased the set point, he used a hydraulic pump to apply pressure to the pilot to determine the pressure at which the pilot operated.3 He said he tested the pilot several times and that it opened at the correct pressure each time. 2 Although “control valve” is the terminology used in the Brooks literature, RV-1919 functioned as a pressure relief valve in this installation. 3 To conduct the test, the mechanic isolates the pilot from the main relief valve and applies hydraulic pressure to the pilot through the sensing line. A gauge on the test unit registers (by showing a drop in pressure) the point at which the pilot operates. This is the same test the company used to perform annual valve tests required by Federal regulation.#

Page 33 It was determined during the evaluation of the relief valve after the accident that the pilot spring had been compressed to the point that the rising inlet pressure could not lift the piston, rendering operation of the pilot valve completely unreliable. Even though the mechanic who replaced the valve spring in RV-1919 and reset the pressure set point said he tested the pilot several times using the same test procedure the company used for annual valve tests, those tests did not reveal that the valve was improperly configured and thus would not consistently open at the intended set pressure. If this valve did not open and the pressure at the Bayview terminal increased above 700 psig, the inlet block valve upstream of the Bayview terminal would close and increase pressure across the damaged section of Olympic pipeline, which is what occurred on the day of the accident. Federal regulations at 49 CFR Part 195 require pipeline operators to test pressure limiting devices, relief valves, and other pressure control equipment once each calendar year at intervals not exceeding 15 months to determine that they are functioning properly, are in good mechanical condition, and are adequate from the standpoint of capacity and reliability of operation for the service in which they are used. These regulations do not identify specific testing procedures to be used to determine whether the relief valve is functioning properly. Although RV-1919 was a new valve and not yet subject to the requirement for periodic inspections, the annual inspections that Olympic performed on other relief valves within its system consisted of a visual inspection and a test to determine the set point of the pilot. The test used to check the set point was the same one used by the mechanic to test the operation of RV-1919. But, as noted above, the tests used by Olympic were inadequate to determine whether the pilot was configured properly or whether the relief valve was operating reliably. The Safety Board concluded that the Federal regulations establishing performance standards for the testing of relief valves and other safety devices installed on hazardous liquid pipelines provide insufficient guidance to ensure that test protocols and procedures will effectively indicate malfunctions of the relief valves and/or their pilot controls. On the day of the accident, the SCADA system that controllers used to operate the pipeline became unresponsive, making it difficult for controllers to analyze pipeline conditions and make timely responses to operational problems. The SCADA system became unresponsive at a critical time, as the controller was attempting to switch delivery points. Had the controller been able to operate the pipeline normally using the SCADA system, it is probable that the pressure backup that accompanied the change in delivery points would have been alleviated and the pipeline operated routinely for the balance of the fuel delivery. Even if the controller had been unable to prevent the pressure buildup and the subsequent closure of the inlet block valve at Bayview, had he had full SCADA control, he may have been able to slow down the pipeline sufficiently to reduce the severity of the pressure increase when the block valve did close. The Safety Board concluded that if the SCADA system computers had remained responsive to the commands of the Olympic controllers, the controller operating the accident pipeline probably would have been able to initiate actions that would have prevented the pressure increase that ruptured the pipeline. Investigators attempted to determine why the SCADA system, which was not reported to have experienced operational problems before the accident, became slow or unresponsive at a critical time during the pipeline operations. About the same time the accident controller was preparing to change delivery points on the 16-inch pipeline, the SCADA system administrator#

Page 44 was in the control center computer room entering two new records into the SCADA historical database. A few minutes after the new records were entered into the system, the SCADA computer began to generate error messages related to the historical database. The SCADA problems grew more pronounced over the next 20 minutes, during which, at one point, the system became completely unresponsive. This period of non-responsiveness coincided with the rupture of the pipeline. The SCADA problems encountered by the controllers occurred shortly after the system administrator inserted the new records into the system computer and were resolved after the control center supervisor deleted the new records. Also, the systems administrator said that as the new records were being deleted, he noticed a typographical error in the records that had not been there when the records were checked earlier. Because of this and the fact that the SCADA system had not previously exhibited a similar non- responsiveness, the Safety Board concluded that the degraded SCADA performance experienced by the pipeline controllers on the day of the accident likely resulted from the database development work that was done on the SCADA system. The system administrator was working on the “live” system. And even though the SCADA system was configured to permit alterations to be made to the historical database while the system was on line, the Safety Board does not consider this to be prudent practice. Computer systems, while they have proven their worth in all modes of transportation, are not infallible, nor are their operators and administrators. Newly developed computer routines do not always work correctly at first and must be revised. Sometimes, seemingly simple mistakes can result in catastrophic consequences, even on the most robust of operating systems. Olympic personnel used the operational system as a test bed to develop changes and upgrades to the database without first testing the changes on a separate off-line system. SCADA developmental work or database modifications should be performed on a developmental workstation that allows any revisions to be thoroughly tested off line. Only after such tests have verified that the system works as intended and the testing has been reviewed by personnel trained in analyzing the test methods and results, should the changes be entered into the SCADA real-time computer. The Safety Board concluded that, had the SCADA database revisions that were performed shortly before the accident been performed and thoroughly tested on an off-line system instead of the primary on-line SCADA system, errors resulting from those revisions may have been identified and repaired before they could affect the operation of the pipeline. The National Transportation Safety Board therefore makes the following safety recommendations to the Research and Special Programs Administration: Develop and issue guidance to pipeline operators on specific testing procedures that can (1) be used to approximate actual operations during the commissioning of a new pumping station or the installation of a new relief valve, and (2) be used to determine, during annual tests, whether a relief valve is functioning properly. (P-02-4) Issue an advisory bulletin to all pipeline operators who use supervisory control and data acquisition (SCADA) systems advising them to implement an off-line workstation that can be used to modify their SCADA system database or to#

Page 55 perform developmental and testing work independent of their on-line systems. Advise operators to use the off-line system before any modifications are implemented to ensure that those modifications are error-free and that they create no ancillary problems for controllers responsible for operating the pipeline. (P-02-5) Please refer to Safety Recommendations P-02-4 and -5 in your reply. If you need additional information, you may call (202) 314-6177. Acting Chairman CARMODY and Members HAMMERSCHMIDT, GOGLIA, and BLACK concurred in these recommendations. Original Signed By: Carol J. Carmody Acting Chairman#