NUSTAR PIPELINE OPERATING PARTNERSHIP L.P. — Warning Letter

party submissionOfficial PDF42023021WL_Operator Response to Notice_06072023_(20-173936).pdf#

42023021WL_Warning Letter_04032023_(20-173936)_text.pdf, page 1Official PDFWARNING LETTER VIA ELECTRONIC MAIL April 3, 2023 Mr. Brad Barron President and Chief Executive Officer NuStar Pipeline Operating Partnership L.P. 19003 IH-10 West San Antonio, Texas 78257 CPF 4-2023-021-WL Dear Mr. Barron, From December 7 through December 11, 2020, and on January 13, 2021, a representative of the Pipeline and Hazardous Materials Safety Administration (PHMSA), Office of Pipeline Safety, pursuant to Chapter 601 of 49 United States Code (U.S.C.), virtually inspected NuStar Pipeline Operating Partnership L.P.’s (NuStar) control room in San Antonio, Texas. As a result of the inspection, it is alleged that NuStar has committed probable violations of the Pipeline Safety Regulations, Title 49, Code of Federal Regulations (CFR). The items inspected and the probable violations are: 1. § 195.446 Control room management. (a) . . . . (e) Alarm management. Each operator using a SCADA system must have a written alarm management plan to provide for effective controller response to alarms. An operator’s plan must include provisions to: (1) . . . . (2) Identify at least once each calendar month points affecting safety that have been taken off scan in the SCADA host, have had alarms inhibited, generated false alarms, or that have had forced or manual values for periods of time exceeding that required for associated maintenance or operating activities;#

42023021WL_Warning Letter_04032023_(20-173936)_text.pdf, page 2NuStar failed to identify at least once each calendar month points affecting safety that have had alarms inhibited for a period of time exceeding that which would be required for associated maintenance or operating activities in accordance with § 195.446(e)(2) and its procedures. NuStar’s October 2020 REV 6 Alarm Management Plan (AMP) procedures sections 2.3.3, 9.3, and 9.4 require the operator to record specific data on inhibited alarms and review these alarms at least once each calendar month. NuStar must identify the reason for inhibiting the alarm, ensure a tag is associated with a point/alarm, record the date and time it was inhibited, include a description for the point/alarm, document the name of the controller/operator responsible and review them at least once each calendar month. NuStar failed to include the required information for numerous alarms inhibited: • July 16, 2020 - July 31, 2020. NuStar identified 21 alarms inhibited and 14 analog points that had been inhibited without providing a reason for inhibiting them and without tagging each point. 38 alarms failed to include one or more of the following required details: no date/time recorded for when they inhibited, no alarm descriptions, and no operator/controller names. • November 1, 2019 - November 15, 2019. NuStar identified 34 alarms and 5 analog points that had been inhibited without providing a reason for the inhibiting and without tagging each point. 32 alarms failed to include one or more of the following required details: no date/time recorded for when they inhibited, no alarm descriptions, and no operator/controller names. • February 16, 2017 - February 28, 2017. NuStar identified 15 alarms and one analog point that had been inhibited without providing a reason for the inhibiting and without tagging each point. Nine alarms failed to include one or more of the following required details: no date/time recorded for when they inhibited, no alarm descriptions, and no operator/controller names. After the inspection NuStar conducted an investigation and identified the issue. Shift supervisors were removing inhibited points from the inhibited list without realizing this removed documentation of the inhibited point on the monthly report. NuStar created a new SCADA screen environment to prevent this and similar issues from recurring. NuStar reviewed supervisory training on inputting/documenting inhibited points in the new SCADA screen environment and its monthly reports to ensure the issue had been corrected. PHMSA finds that the corrective actions resolved the identified issues. Therefore, NuStar failed to identify at least once each calendar month points affecting safety that have had alarms inhibited for a period of time exceeding that which would be required for associated maintenance or operating activities in accordance with § 195.446(e)(2) and its procedures.#

42023021WL_Warning Letter_04032023_(20-173936)_text.pdf, page 32. § 195.446 Control room management. (a) . . . . (b) Roles and responsibilities. Each operator must define the roles and responsibilities of a controller during normal, abnormal, and emergency operating conditions. To provide for a controller’s prompt and appropriate response to operating conditions, an operator must define each of the following: (1) . . . . (4) A method of recording controller shift-changes and any hand- over of responsibility between controllers; and NuStar failed to define a method of recording controller shift-changes and any hand-over of responsibility between controllers in its control room management plan in accordance with § 195.446(b)(4) and its procedures. Specifically, NuStar’s controllers failed to include all required information in the shift change records, as required by NuStar’s procedures, General Systems Manual Section 3 (9/2020). NuStar’s Console 1 controller shift-change records failed to document the status of several facility locations including, the Alamose Pumping Station in the August 1, 2017 shift change records. After the inspection NuStar demonstrated new features to its electronic shift change documentation. NuStar added a Safety Issues section which requires documentation of any safety issues. PHMSA finds that the corrective actions resolved the identified issues. Therefore, NuStar failed to define a method of recording controller shift-changes and any hand- over of responsibility between controllers in its control room management plan in accordance with § 195.446(b)(4) and its procedures. 3. § 195.446 Control room management. (a) . . . . (j) Compliance and deviations. An operator must maintain for review during inspection: (1) Records that demonstrate compliance with the requirements of this section; and § 195.446 Control room management. (a) . . . . (d) Fatigue mitigation. Each operator must implement the following methods to reduce the risk associated with controller fatigue that could inhibit a controller’s ability to carry out the roles and responsibilities the operator has defined: (1) . . . . (2) Educate controllers and supervisors in fatigue mitigation strategies and how off-duty activities contribute to fatigue;#

42023021WL_Warning Letter_04032023_(20-173936)_text.pdf, page 4NuStar failed to provide records that demonstrate compliance with § 195.446 in accordance with § 195.446(j)(1). Specifically, NuStar failed to provide records for 2018 demonstrating it had educated controllers and supervisors in fatigue mitigation strategies and how off-duty activities contribute to fatigue in accordance with § 195.446(d)(2). 4. § 195.446 Control room management. (a) . . . . (j) Compliance and deviations. An operator must maintain for review during inspection: (1) Records that demonstrate compliance with the requirements of this section; and § 195.446 Control room management. (a) . . . . (d) Fatigue mitigation. Each operator must implement the following methods to reduce the risk associated with controller fatigue that could inhibit a controller’s ability to carry out the roles and responsibilities the operator has defined: (2) . . . . (4) Establish a maximum limit on controller hours-of-service, which may provide for an emergency deviation from the maximum limit if necessary for the safe operation of a pipeline facility. NuStar failed to maintain records that demonstrate compliance with § 195.446(d)(4) in accordance with § 195.446(j)(1). Specifically, NuStar failed to provide records demonstrating it followed its deviation procedures to reduce the risk of controller fatigue in accordance with § 195.446(d)(4) and its control room management (CRM) Plan. NuStar’s procedure for deviations from a controller’s hours of service (HOS) require the Shift Supervisor to perform multiple steps in the review and documentation for planned and emergency deviations and they must provide this information to NuStar’s Fatigue Risk Manager for approval. For HOS deviations, NuStar’s procedures require written approval, and the Fatigue Risk Manager must determine and substantiate how the increased risk of fatigue will be mitigated. NuStar’s CRM Plan, in Appendix B.3.3, lists acceptable hours of service (HOS) deviations, and refers to the review and approval process for each deviation. However, NuStar approved HOS deviations for other, non-emergency reasons, such as the unavailability of other controllers due to vacation and inadequate staffing levels. Between March 19, 2017, and November 16, 2020, NuStar failed to approve 16 HOS deviations using one of the listed acceptable deviations. These records cited non-listed, non-emergency reasons to justify the HOS deviation.#

42023021WL_Warning Letter_04032023_(20-173936)_text.pdf, page 5For those 16 HOS deviations, NuStar also failed to document why the deviation was necessary for safe operation of the facility, why the deviation was an emergency, and how the increased risk of fatigue would be mitigated. After the inspection, NuStar revised and clarified the documentation and approval requirements for HOS emergency deviations and mitigation in its CRM Plan. PHMSA finds that the corrective actions resolved the identified issues. Therefore, NuStar failed to provide records that demonstrate compliance with § 195.446 in accordance with § 195.446(j)(1). 5. § 195.446 Control room management. (a) . . . . (j) Compliance and deviations. An operator must maintain for review during inspection: (1) Records that demonstrate compliance with the requirements of this section; and § 195.446 Control room management. (a) . . . . (h) Training. Each operator must establish a controller training program and review the training program content to identify potential improvements at least once each calendar year, but at intervals not to exceed 15 months. An operator’s program must provide for training each controller to carry out the roles and responsibilities defined by the operator. In addition, the training program must include the following elements: (1) . . . . (6) Control room team training and exercises that include both controllers and other individuals, defined by the operator, who would reasonably be expected to operationally collaborate with controllers (control room personnel) during normal, abnormal or emergency situations. Operators must comply with the team training requirements under this paragraph no later than January 23, 2018. NuStar failed to maintain records that demonstrate compliance with § 195.446(h)(6) in accordance with § 195.446(j)(1). Specifically, NuStar failed to provide team training records for one of its controllers. Under 49 U.S.C. § 60122 and 49 CFR § 190.223, you are subject to a civil penalty not to exceed $225,134 per violation per day the violation persists, up to a maximum of $2,251,334 for a related series of violations. For violation occurring on or after January 11, 2021 and before May 3, 2021, the maximum penalty may not exceed $222,504 per violation per day the violation persists, up to a maximum of $2,225,034 for a related series of violations. For violation occurring on or after#

42023021WL_Warning Letter_04032023_(20-173936)_text.pdf, page 6July 31, 2019 and before January 11, 2021, the maximum penalty may not exceed $218,647 per violation per day the violation persists, up to a maximum of $2,186,465 for a related series of violations. For violation occurring on or after November 27, 2018 and before July 31, 2019, the maximum penalty may not exceed $213,268 per violation per day, with a maximum penalty not to exceed $2,132,679. For violation occurring on or after November 2, 2015 and before November 27, 2018, the maximum penalty may not exceed $209,002 per violation per day, with a maximum penalty not to exceed $2,090,022. We have reviewed the circumstances and supporting documents involved in this case, and have decided not to conduct additional enforcement action or penalty assessment proceedings at this time. We advise you to correct the items identified in this letter. Failure to do so may result in NuStar Logistics, L.P. being subject to additional enforcement action. No reply to this letter is required. If you choose to reply, in your correspondence please refer to CPF 4-2023-021-WL. Be advised that all material you submit in response to this enforcement action is subject to being made publicly available. If you believe that any portion of your responsive material qualifies for confidential treatment under 5 U.S.C. § 552(b), along with the complete original document you must provide a second copy of the document with the portions you believe qualify for confidential treatment redacted and an explanation of why you believe the redacted information qualifies for confidential treatment under 5 U.S.C. § 552(b). Sincerely, Bryan Lethcoe Director, Southwest Region Pipeline and Hazardous Materials Safety Administration#