CPF 32021017WL
CPF 32021017WL
32021017WL_Warning Letter_04092021_(20-173061)_text.pdf, page 1Official PDFWARNING LETTER VIA ELECTRONIC MAIL TO: stan.horton@bwpipelines.com, dick.keyser@bwpipelines.com April 9, 2021 Stanley C. Horton President, CEO Texas Gas Transmission, LLC 9 Greenway Plaza, Suite 2800 Houston, TX 77066 CPF 3-2021-017-WL Dear Mr. Horton: From June 15 through June 19, 2020, a representative of the Pipeline and Hazardous Materials Safety Administration (PHMSA) pursuant to Chapter 601 of 49 United States Code (U.S.C.) inspected your subsidiary, Texas Gas Transmission, LLC’s (Texas Gas), Control Room Management Program procedures and records in Owensboro, KY. Texas Gas Tranmsission is the primary for the CRM Safety Program Relationship which supports the following OPID’s: 31278 Texas Gas Pipeline Company, 39210 Boardwalk Storage Services , 39470 Louisiana Energy and Power Authority. The inspection was conducted remotely. As a result of the inspection, it is alleged that you have committed probably violations of the Pipeline Safety Regulations, Title 49, Code of Federal Regulations (CFR). The items inspected and the probable violations are:#
32021017WL_Warning Letter_04092021_(20-173061)_text.pdf, page 21. § 192.631 Control room management. (a) . . . . (j) Compliance and deviations. An operator must maintain for review during inspection: (1) Records that demonstrate compliance with the requirements of this section; and Texas Gas Point to Point (P2P) verification records were insufficient to demonstrate compliance with the regulation because they did not provide details to demonstrate thoroughness of the point-to-point verification. Section 192.631(c)(2) requires operators to “[c]onduct a point-to- point verification between SCADA displays and related field equipment when field equipment is added or moved and when other changes that affect pipeline safety are made to field equipment or SCADA displays.” Texas Gas presented for inspection electronic point to point records for Greenville Point to Point Verification (2012), Columbia Point To Point Verification (2012), GS024327-CB-NGPL-ControlWave-24328 GS024347-CB-Transco-ControlWave-24348 and TG009910/TG 9910 Lepa VLV 18 Stat. The record presented for GS024327-CB-NGPL-ControlWave-24328 (2017-2019) provided a cover sheet with technical and mapping information for the RTU. It also provided notes related to the site from May 19, 2017 to February 26, 2019. These notes relate to different work or projects related to the facility and associated equipment. The excel document contains 5 tabs: Info, Analogs, Status, Analog-Verify and Status-Verify. The information on the tabs list the signal description, Modbus register and SCADA Tag Name, but not with a consistent layout between the tabs. There is a column that is labeled “checked out with field” where only an X is placed in the cell. There is no indication of what the Control Room HMI displayed and the related field end device. There are also columns related to Limits for alarming which provided no documentation for the validation check, nor actual field outcome except for one entry for Chromatograph Stream 1’s GPM, which was not checked out. Similar results were found for record GS024347-CB-Transco-ControlWave-24348. The record related to TG009910 point to point for an “added remote valve 18”, is a completely different excel file form. The tab labeled General has a date at the top of the form as 7/21/2017 and it indicates the verification date was 4/17/19. A summary of the items reviewed provide the following: In the Analogue Verify and Status Verify there were columns to document the field value or status and the SCADA value or status, these were either blank or had an X. Additionally, the column with the tag description was colored green. When asked what, this meant, the answer was that they assumed it had been checked. There were columns for alarm limits that were blank and dates for the checkout were either not provided or provided in an undiscernible manner. The procedure was reviewed with the team to try to relate completing the form with the procedure and the responses were vague. The P2P procedure, when done in a thorough manner, should include information to verify a match between the field device and the HMI SCADA values or status, the individuals involved in the test, the limits established for the points and that they presented as alarms as designed 2#
32021017WL_Warning Letter_04092021_(20-173061)_text.pdf, page 3(correct value, message, priority, priority color, safety related, audible alert, etc.) and any comments related to that point. There should also be verification that the point responded consistently on each screen where it has been designed to present. The records of P2P were not thorough to provide such documentation. 2. § 192.631 Control room management. (a) . . . . (e) Alarm management. Each operator using a SCADA system must have a written alarm management plan to provide for effective controller response to alarms. An operator's plan must include provisions to: (1) … (4) Review the alarm management plan required by this paragraph at least once each calendar year, but at intervals not exceeding 15 months, to determine the effectiveness of the plan; Texas Gas reviews of the alarm management plan for 2017, 2018 and 2019 were insufficient to demonstrate adequate implementation of the operator’s process and to demonstrate compliance with the regulation in determining the effectiveness of the plan. Texas Gas presented for inspection form WI 06610 BWP Alarm Management Plan Review for 2017, 2018 and 2019. The document provided 4 statements of review, with no back up documentation as to what was reviewed. Each document was signed and dated which also included a brief comment. The review provided no discernable content nor criteria to determine any level of effectiveness of the plan. Simply stating the plan is effective is not adequate. Different documents were provided for 2018 and 2019 than 2017, but similar in nature. 3. § 192.631 Control room management. (a) . . . . (g) Operating experience. Each operator must assure that lessons learned from its operating experience are incorporated, as appropriate, into its control room management procedures by performing each of the following: (1) . . . . (2) Include lessons learned from the operator's experience in the training program required by this section. Texas Gas failed to document lessons learned training and review when they were delivered to controllers. Texas Gas did develop lessons learned, after a variety of events, and indicated they delivered them to the controllers for discussion and review. However, they were not able to provide records in any format to validate the lesson was delivered, reviewed and acknowledged by the controller. Lessons learned is required to be part of the training content and therefore, when delivered to a controller the lesson needs to be recorded. 3#
32021017WL_Warning Letter_04092021_(20-173061)_text.pdf, page 4Boardwalk provided a response and indicated that going forward, they will assign Lessons Learned to Controllers through its online Learning Management System (“LMS”) so that a training roster can be created to provide better documentation. The result of insufficient review and documentation stems from the procedure for the annual alarm review lacking substance in criteria, content, conclusions. There are no metrics for determination of effectiveness. Simply stating a plan, with no reference for that determination is not adequate nor acceptable. 4. § 192.631 Control room management. (a) . . . . (h) Training. Each operator must establish a controller training program and review the training program content to identify potential improvements at least once each calendar year, but at intervals not to exceed 15 months. An operator's program must provide for training each controller to carry out the roles and responsibilities defined by the operator. In addition, the training program must include the following elements: Texas Gas ’s annual review of the training program were insufficient to demonstrate adequate implementation of the operator’s process and to demonstrate compliance with the regulation in reviewing the training program content to identify potential improvements. Texas Gas presented for inspection form WI 06616 BWP Training Program Review for the years 2017, 2018 and 2019. The form for 2017 was different in style, but very similar in content. The 2017 form consisted of 3 statements dates, signature and 5 comments related to changes that have been made to the training content. The 2018 and 2019 documents provided responses of Yes or No to 5 questions with no comments in 2018 and a simple statement in 2019. There is no back up documentation to substantiate any of the responses or comments. An example of the lack of back up documentation for a question rendering a response of Yes or No is “Was the overall effectiveness of the Training Program reviewed?”; answer yes. “Were any changes necessary to improve Controller performance?”; answer no. The follow up question begs: what was reviewed, who reviewed it why was this relative to training, how is effectiveness measured, were controllers surveyed, was all content reviewed or just a sample. Without the details, there is no relevance to the review exercise. A good review process lays out a process that asks questions about content and performance, provides objective responses and findings that are relevant to those outcomes. Texas Gas failed to provide sufficient evidence of review to demonstrate adequate implementation of the operator’s process for annual training content review. 4#
32021017WL_Warning Letter_04092021_(20-173061)_text.pdf, page 55. § 192.631 Control room management. (a) . . . . inspection: (j) Compliance and deviations. An operator must maintain for review during (1) Records that demonstrate compliance with the requirements of this section; and Texas Gas failed to provide records for 2018 and 2019 that sufficiently demonstrated that they tested and verified their internal communication plan to provide adequate means for manual operation of the pipeline safely, at least once each calendar year, but at intervals not to exceed 15 months. Section §192.631(c)(3) requires operators to “test and verify an internal communication plan to provide adequate means for manual operation of the pipeline safely, at least once each calendar year, but at intervals not to exceed 15 months. Under §192.631(j)(1), Boardwalk is required to keep these records for review during inspection. Over the course of the inspection, three different sets of records were provided as evidence of compliance with the 2018 and 2019 internal communication plan tests and verification requirement. The first set of records provided were blank logs with dates of purported tests, but with no actual test information or signatures to verify that tests had been performed as required. The second set of records were emails that provided notification of the upcoming internal communication manual test to operator personnel for the dates provided in the first set of records. The third set of records were emails and operator log entries of actual SCADA outage events that were managed in the control room, but with no associated verbiage in the log that indicated personnel had been dispatched to the field to call in periodic readings to the controller. There was also no log recording the field data in this third set of records. Under Texas Gas ’s Control Room Management (CRM) plan, actual events can be considered a test in place of a drill or exercise. Section 4.5 of the CRM plan references another Texas Gas procedure titled “Risk of Failures and the Problem Resolution Items 3 and 4 of the Gas Control Business Continuation Plan (GCBCP), October 17, 2019” as the guiding document for the manual operation of the pipeline in the event of loss of SCADA or communications. Per this procedure, "Gas Control will record this information in the Emergency Ledger Sheets (Exhibit F), analyze it, and provide direction to Operations." (emphasis added). Section 4.5, “Internal Communication Plan”, of the CRM plan provides in relevant part as follows: "The test shall ensure the equipment is working properly as designed and that employees are familiar with how communications may be conducted. Functions that must be verified during testing include, but are not limited to, (1) communication between and among operational and maintenance personnel using voice, fax, messaging, radio, etc., and (2) communication of pipeline operational data such as dial-in polling of field equipment, manually reading gauges and field instrumentation, etc." Texas Gas equates the Internal Communication Plan to a "local control plan", which is referenced in Texas Gas procedure GCBCP, and provides in relevant part as follows: 5#
32021017WL_Warning Letter_04092021_(20-173061)_text.pdf, page 6"Should the local control plan be put in effect, Operations will monitor critical locations (Exhibit B) on the pipeline. Critical information identified in the Gas Control Local Control Plan will be accumulated and communicated by Operations to Gas Control via the most efficient mode of communication available at least every 2 hours. Should the satellite phone be the most efficient mode of communication available, Operations will gather and be prepared to communicate to Gas Control the specified data contained in the Emergency Ledger Sheets (Exhibit F)." The evidence of actual events as the test and verification required by §192.631(c)(3), provided in the third set of records submitted by Texas Gas , did not include the Exhibit F Emergency Ledger Sheets. When asked why there were not Emergency Ledger Sheets provided, Texas Gas stated that it did not execute the Local Control Plan during the actual events test, and Texas Gas procedures require use of the Exhibit F form only when the Local Control Plan is executed. The 2018 and 2019 actual event records provided for verification of a test of the internal communications manual operation plan test do not qualify as an acceptable test because Section 4.5, “Internal Communication Plan”, of the CRM plan establishes functions that must be verified during testing as: (1) communication between and among operational and maintenance personnel using voice, fax, messaging, radio, etc.; and (2) communication of pipeline operational data such as dial-in polling of field equipment, manually reading gauges and field instrumentation, etc. Because Texas Gas did not execute the Local Control Plan they did not fulfill the functional requirement defined in number 2. Therefore, no test can be counted through the actual events. Additionally, no other records could be provided for any tests in 2018 or 2019 that verified the functional requirements of a valid test. Under 49 U.S.C. § 60122 and 49 CFR § 190.223, you are subject to a civil penalty not to exceed $218,647 per violation per day the violation persists, up to a maximum of $2,186,465 for a related series of violations. For violation occurring on or after November 27, 2018 and before July 31, 2019, the maximum penalty may not exceed $213,268 per violation per day, with a maximum penalty not to exceed $2,132,679. For violation occurring on or after November 2, 2015 and before November 27, 2018, the maximum penalty may not exceed $209,002 per violation per day, with a maximum penalty not to exceed $2,090,022. For violations occurring prior to November 2, 2015, the maximum penalty may not exceed $200,000 per violation per day, with a maximum penalty not to exceed $2,000,000 for a related series of violations. We have reviewed the circumstances and supporting documents involved in this case, and have decided not to conduct additional enforcement action or penalty assessment proceedings at this time. We advise you to correct the item(s) identified in this letter. Failure to do so will result in Texas Gas Transmission being subject to additional enforcement action. No reply to this letter is required. If you choose to reply, in your correspondence please refer to CPF 3-2021-017-WL. Be advised that all material you submit in response to this enforcement action is subject to being made publicly available. If you believe that any portion of your 6#
32021017WL_Warning Letter_04092021_(20-173061)_text.pdf, page 7responsive material qualifies for confidential treatment under 5 U.S.C. 552(b), along with the complete original document you must provide a second copy of the document with the portions you believe qualify for confidential treatment redacted and an explanation of why you believe the redacted information qualifies for confidential treatment under 5 U.S.C. 552(b). Sincerely, Gregory A. Ochs Director, Central Region, Office of Pipeline Safety Pipeline and Hazardous Materials Safety Administration cc: Richard Keyser, Sr. VP Operations dick.keyser@bwpipelines.com 7#
This material provides agency context. It does not replace binding regulatory text, and its legal effect depends on the underlying authority and facts.