NTSB Safety Recommendation P-98-022

NTSB safety recommendation P-98-022.

THE NTSB RECOMMENDS THAT THE MARATHON ASHLAND PIPE LINE LLC: EVALUATE THE EFFECTIVENESS OF ALTERNATIVE DISPLAY FORMATS & FREQUENCIES OF ALARMING CRITICAL INFO FOR YOUR SUPERVISORY CONTROL & DATA ACQUISITION SYSTEM & MODIFY THE SYSTEM AS NECESSARY TO ENSURE THAT CONTROLLERS ARE SPECIFICALLY PROMPTED TO CONSIDER THE POSSIBILITY OF LEAKS DURING SYSTEM DEVIATIONS THAT ARE CONSISTENT WITH A LOSS OF PRODUCT FROM A PIPELINE.

Priority: CLASS II

Overall Status: Closed - Acceptable Action

Issued Date: 1998-09-28

Adopted Date: 1998-09-21

Overall Date Closed: 1999-04-28

Synopsis: 0N 5/23/96, A 68-MILE-LINE, 20-INCH-DIAMETER STEEL PIPELINE OWNED BY MARATHON PIPE LINE COMPANY RUPTURED AT A LOCATION NEAR GRAMERCY, LOUISIANA. THE RUPTURE WENT UNDETECTED BY THE PIPELINE CONTROLLER FOR ABOUT 1 HOUR. THE RUPTURED PIPELINE ULTIMATELY RELEASED ABOUT 475,000 GALLONS OF GASOLINE INTO A COMMON PIPELINE RIGHT-OF-WAY WITHIN A DESIGNATED "WETLAND." GASOLINE ALSO ENTERED THE BLIND RIVER, CAUSING ENVIRONMENTAL DAMAGE & KILLING FISH, WILDLIFE, & VEGETATION IN THE AREA.

Probable Cause: Excavations of a nearby pipeline operated by LaRoche Industries, Inc., which resulted from the failure of LaRoche either to take adequate measures to ensure that excavations performed under its supervision did not damage underground utilities or to notify Marathon that those excavations may have damaged the Marathon pipeline

Ntsbnumber: DCA96MP004

Report Number: PAB-98-01

Addressee Name: Marathon Pipe Line LLC

Addressee Status: Closed - Acceptable Action

Addressee Date Closed: 1999-04-28

Addressee Acronym: MPL

Addressee Organization Type: P-Private Industry

Communication Date: 1998-12-30

Communication Type: Official Correspondence

Communication Contents: THE BOARD IS PLEASED THAT MARATHON HAS MADE CHANGES TO THE SCADA SYSTEM DISPLAYS AND ALARMS, INCLUDING DELETING OR MODIFYING ALARMS SO THAT THE OPERATOR CAN FOCUS ON THE IMPORTANT INFORMATION GENERATED BY THE SYSTEM. THE BOARD IS ALSO PLEASED TO LEARN THAT SUPERVISORS REVIEWED THE LESSONS LEARNED FROM THIS ACCIDENT WITH EACH SCADA OPERATOR. THE BOARD FURTHER UNDERSTANDS THAT MARATHON HAS INSTITUTED CONTINUOUS EVALUATION AND OPTIMIZATION OF BOTH THE SCADA SYSTEM AND THE TRAINING OF THE OPERATORS ON THE SYSTEM. THE BOARD WOULD APPRECIATE LEARNING IF MARATHON HAS (1) USED RECURRENT CONTROLLER TRAINING TO EMPHASIZE THE IMPORTANCE OF CAREFULLY AND COMPLETELY READING THE TEXT OF AND EVALUATING ALL ALARM MESSAGES, AND (2) MODIFIED ITS SCADA SYSTEM AS NECESSARY TO ENSURE THAT CONTROLLERS ARE SPECIFICALLY PROMPTED TO CONSIDER THE POSSIBILITY OF LEAKS DURING SYSTEM DEVIATIONS THAT ARE CONSISTENT WITH A LOSS OF PRODUCT FROM A PIPELINE. THE BOARD WOULD ALSO APPRECIATE LEARNING THE DETAILS OF MARATHON'S IMPLEMENTATION OF P-98-21 AND P-98-22. PENDING FURTHER RESPONSE FROM MARATHON ON THE STATUS OF THESE EFFORTS, P-98-21 AND -22 HAVE BEEN CLASSIFIED "OPEN--ACCEPTABLE RESPONSE."

Addressee Acronym: MPL

Addressee Organization Type: P-Private Industry

Communication Date: 1999-04-28

Communication Type: Official Correspondence

Communication Contents: THE SAFETY BOARD UNDERSTANDS THAT MARATHON HAS IMPLEMENTED A RECURRENT CONTROLLER TRAINING MECHANISM TO EMPHASIZE CAREFUL READING OF ALARM MESSAGES AND THAT IT HAS MADE SCADA SYSTEM MODIFICATIONS ENABLING ITS CONTROLLERS TO FOCUS ON CRITICAL INFORMATION. THE RESULT HAS BEEN THAT MARATHON HAS MADE SIGNIFICANT PROGRESS IN REDUCING FALSE COMPUTATIONAL PIPELINE MONITORING (CPM) ALARMS AND THAT IT INTENDS TO CONTINUE TUNING EACH OF ITS CPM SYSTEMS TO FURTHER REDUCE THE NUMBER OF FALSE ALARMS. BASED ON THESE EFFORTS, P-98-21 AND -22 HAVE BEEN CLASSIFIED "CLOSED--ACCEPTABLE ACTION."

Addressee Acronym: MPL

Addressee Organization Type: P-Private Industry

Communication Date: 1998-10-23

Communication Type: Official Correspondence

Communication Contents: Letter Mail Controlled 10/27/98 5:00:07 PM MC# 981296 THE FIRST TWO RECOMMENDATIONS RELATE TO THE SCADA SYSTEM, EITHER IN THE TRAINING OF THE SCADA OPERATORS OR IN THE PHYSICAL SETUP OF THE SYSTEM THROUGH ITS DISPLAYS AND ALARMS. (P-98-21 AND -22). I WILL ADDRESS THEM TOGETHER. AT THE TIME OF THE INCIDENT ON 5/23/96, MAPL LLC HAD JUST SWITCHED TO A NEW SCADA SYSTEM. THE PIPELINE SYSTEM THAT SUFFERED THE RUPTURE HAD ONLY BEEN "ONLINE" FOR LESS THAN THIRTY (30) DAYS. THE MAPL LLC COMPUTER STAFF WAS JUST STARTING THE PROCESS OF FINE TUNING THE SYSTEM. I DO NOT WANT TO LEAVE AN IMPRESSION THAT THE NEW SCADA SYSTEM WAS NOT WORKING CORRECTLY OR HAD BEEN BROUGHT ONLINE TOO EARLY. THE NEW SYSTEM WAS A VAST IMPROVEMENT OVER THE OLD SYSTEM. HOWEVER, LIKE ANY OTHER NEW SYSTEM, THE OPERATORS HAVE TO HAVE TIME TO ADJUST TO WORKING ON IT. IN ADDITION, FEATURES OF THE SYSTEM NEED TO BE REVIEWED AND ADJUSTED TO MEET THE REALITIES OF OPERATION. THE OPERATOR IN QUESTION HAD JUST STARTED WORKING ON THE SYSTEM AFTER A PERIOD OF INTENSIVE, EXTENSIVE TRAINING. HOWEVER, HE HAD NOT HAD A CHANCE TO FULLY "REPROGRAM" HIMSELF FROM HIS HABITS LEARNED ON THE OLDER SYSTEM, WITH WHICH HE HAD MUCH EXPERIENCE. ALSO, THE NUMBER OF ALARMS USED BY THE NEW SYSTEM AT THAT TIME PRESENTED VAST AMOUNTS OF INFORMATION, THAT WHILE TECHNICALLY ACCURATE, SERVED TO OBFUSCATE THE CRITICAL INFORMATION. FINALLY, THE OPERATOR MISREAD A NUMBER IN HIS HASTE TO REACT TO THE ALARMS - (HE DID NOT NOTICE THE "--" IN FRONT OF A NUMBER, WHICH CAN EASILY HAPPEN ON A SMALL SCREEN IF YOU ARE READING QUICKLY). SINCE THE TIME OF THE INCIDENT, MAPL LLC HAS MADE MANY CHANGES TO THE SCADA SYSTEM DISPLAYS AND ALARMS. FIRST, MANY TYPES OF ALARMS HAVE BEEN DELETED OR MODIFIED SO THAT THE OPERATOR CAN FOCUS ON THE TRULY IMPORTANT INFORMATION GENERATED BY THE SYSTEM IN A MEANINGFUL FASHION. THE MAPL LLC SCADA ROOM SUPERVISORS WENT OVER THE "LESSONS LEARNED" WITH EACH SCADA OPERATOR AFTER THE INCIDENT TO FOCUS ON WHAT WENT WRONG AND ON HOW TO PREVENT IT. LASTLY, NWL LLC MAINTAINS A CONTINUOUS EVALUATION OF THE SCADA SYSTEM AND THE TRAINING OF THE OPERATORS ON THE SYSTEM. MAPL LLC DOES NOT WAIT FOR SPECIFIC TIME PERIODS TO REVIEW THE SYSTEM AND THEN MAKE CHANGES. THE EVALUATION IS CONTINUOUS AND CHANGES ARE MADE AS QUICKLY AS POSSIBLE TO ENSURE THAT THE SYSTEM IS WORKING IN AS OPTIMAL A FASHION AS POSSIBLE.

Addressee Acronym: MPL

Addressee Organization Type: P-Private Industry

Communication Date: 1999-02-19

Communication Type: Official Correspondence

Communication Contents: Letter Mail Controlled 2/25/99 12:59:16 PM MC# 990173 YOUR FIRST QUESTION ADDRESSED MAPL LLC'S RECURRENT CONTROLLER TRAINING TO EMPHASIZE CAREFULLY READING ALARM MESSAGES. EVEN BEFORE THE AIRLINE HIGHWAY RELEASE, THE SCADA CONTROLLERS HAD BEEN TRAINED TO READ THE ALARM MESSAGES THOROUGHLY. A RECURRENT CONTROLLER TRAINING MECHANISM WAS RECENTLY IMPLEMENTED TO REINFORCE THE IMPORTANCE OF THIS ISSUE. A COPY OF THE MESSAGE BEING SENT TO THE CONTROLLERS IS ATTACHED. A RELATED QUESTION ADDRESSED MODIFICATIONS TO THE SCADA SYSTEM ITSELF TO PROMPT CONTROLLERS TO CONSIDER THE POSSIBILITY OF LEAKS DURING SYSTEM DEVIATIONS. MODIFICATIONS TO THE SCADA SYSTEM WERE MADE SHORTLY AFTER THE INCIDENT, SOME IN PART AS A RESULT OF THE AIRLINE HIGHWAY RELEASE, TO FOCUS THE CONTROLLERS ON THE CRITICAL INFORMATION. DOCUMENTATION OF THE MAJOR CHANGES MADE IS ATTACHED. SIGNIFICANT PROGRESS HAS BEEN MADE IN REDUCING "FALSE" CPM ALARMS AND ELIMINATING EXTRANEOUS ALARMS. IN LATE 1996, CPM ALARMS AVERAGED 150-200 PER DAY. TODAY, AFTER ADDING ADDITIONAL SYSTEMS RELATED TO THE MARATHON/ASHLAND MERGER, CPM ALARMS ARE AVERAGING 60 PER DAY FOR 53 DIFFERENT SYSTEMS. ONGOING EFFORTS CENTER AROUND TUNING EACH CPM SYSTEM TO FURTHER REDUCE THE NUMBER OF "FALSE" CPM ALARMS RECEIVED. THIS WILL ENHANCE THE IMPORTANCE OF THE CPM ALARMS.