# CALIBER SPRING CREEK LLC — Notice of Amendment **Citation:** CPF 32024017NOA **Type / status:** enforcement / historical **Agency:** Pipeline and Hazardous Materials Safety Administration **Effective:** Not stated **Published:** 2024-07-03 CLOSED notice of amendment citing 195.402(a), 195.408(b)(3), 195.446(a), 195.446(b)(1), 195.446(b)(2), 195.446(b)(3), 195.446(b)(4), 195.446(b)(5), 195.446(c)(1), 195.446(c)(2), 195.446(c)(3), 195.446(c)(4), 195.446(d), 195.446(d)(1), 195.446(d)(4), 195.446(e)(1), 195.446(e)(2), 195.446(e)(4), 195.446(e)(5), 195.446(e)(6), 195.446(f)(1), 195.446(f)(2), 195.446(g)(1), 195.446(g)(2), 195.446(h), 195.446(h)(1), 195.446(h)(3), 195.446(h)(4), 195.446(h)(5), 195.446(h)(6), 195.446(i), 195.446(j)(1), 195.452(a), 195.452(i)(3). ## Document text Notice of Amendment involving CALIBER SPRING CREEK LLC. PHMSA's enforcement data identifies the cited regulations as 195.402(a), 195.408(b)(3), 195.446(a), 195.446(b)(1), 195.446(b)(2), 195.446(b)(3), 195.446(b)(4), 195.446(b)(5), 195.446(c)(1), 195.446(c)(2), 195.446(c)(3), 195.446(c)(4), 195.446(d), 195.446(d)(1), 195.446(d)(4), 195.446(e)(1), 195.446(e)(2), 195.446(e)(4), 195.446(e)(5), 195.446(e)(6), 195.446(f)(1), 195.446(f)(2), 195.446(g)(1), 195.446(g)(2), 195.446(h), 195.446(h)(1), 195.446(h)(3), 195.446(h)(4), 195.446(h)(5), 195.446(h)(6), 195.446(i), 195.446(j)(1), 195.452(a), 195.452(i)(3). The case was opened on 2024-07-03 and is reported as closed as of 2026-05-21. Open the official case record for notices, responses, orders, and the latest status. Official case documents: 32024017NOA_Closure Letter_05212026_(22-236545).pdf: https://primis.phmsa.dot.gov/enforcement-documents/32024017NOA/32024017NOA_Closure%20Letter_05212026_(22-236545).pdf 32024017NOA_Closure Letter_05212026_(22-236545)_text.pdf: https://primis.phmsa.dot.gov/enforcement-documents/32024017NOA/32024017NOA_Closure%20Letter_05212026_(22-236545)_text.pdf 32024017NOA_Notice of Amendment_07032024_(22-236545).pdf: https://primis.phmsa.dot.gov/enforcement-documents/32024017NOA/32024017NOA_Notice%20of%20Amendment_07032024_(22-236545).pdf 32024017NOA_Notice of Amendment_07032024_(22-236545)_text.pdf: https://primis.phmsa.dot.gov/enforcement-documents/32024017NOA/32024017NOA_Notice%20of%20Amendment_07032024_(22-236545)_text.pdf 32024017NOA_Operator Response to Notice and Request Time Extension_07152024_(22-236545).pdf: https://primis.phmsa.dot.gov/enforcement-documents/32024017NOA/32024017NOA_Operator%20Response%20to%20Notice%20and%20Request%20Time%20Extension_07152024_(22-236545).pdf 32024017NOA_Closure Letter_05212026_(22-236545)_text.pdf U.S. Department of Transportation Pipeline and Hazardous Materials Safety Administration 1100 Main Street, Suite 800 Kansas City, MO 64105 (816) 329-3800 VIA ELECTRONIC MAIL TO: bkent@calibermidstream.com; dwerth@calibermidstream.com; cmaybee@calibermidstream.com May 21, 2026 Mr. Tad True, COO Caliber Spring Creek, LLC 455 North Poplar Street Casper, Wyoming 82602 RE: CPF 3-2024-017-NOA Dear Mr. True: On April 25 through April 29, May 2 through May 4, and May 9 through May 11, 2022, a representative of the Pipeline and Hazardous Materials Safety Administration (PHMSA), pursuant to Chapter 601 of 49 United States Code (U.S.C.), conducted an inspection of Caliber Spring Creek, LLC’s (“Caliber”) procedures for Control Room Management (CRM), SCADA, Leak Detection, and Operations and Maintenance (O&M) relevant to the control room located in Houston, Texas. On July 3, 2024, pursuant to 49 CFR § 190.206, PHMSA issued a Notice of Amendment which proposed amendment of Caliber’s procedures. Caliber submitted its amended procedures on October 18 and November 18, 2024 and provided supplements on November 21 and December 2, 2025. PHMSA has reviewed the amended procedures, and it appears that the inadequacies outlined in the Notice of Amendment have been corrected. This letter is to inform you that no further action is necessary, and this case is now closed. Thank you for your cooperation. Sincerely, Amy McKean Director, Central Region, Office of Pipeline Safety Pipeline and Hazardous Materials Safety Administration cc: Ken Dockweiler, Director Land and Government (ken.dockweiler@truecos.com) Kevin Pena, Pipeline Compliance Coordinator (kevin.pena@truecos.com) 32024017NOA_Notice of Amendment_07032024_(22-236545)_text.pdf NOTICE OF AMENDMENT VIA ELECTRONIC MAIL TO: bkent@calibermidstream.com; dwerth@calibermidstream.com; cmaybee@calibermidstream.com July 3, 2024 Bill Kent Chief Executive Officer Caliber Midstream Partners, LP Caliber Spring Creek, LLC 1805 Shea Center Drive, Suite 120 Highlands Ranch, CO 80129 CPF 3-2024-017-NOA Dear Mr. Kent: From April 25 through April 29, May 2 through May 4, and May 9 through May 11, 2022, a representative of the Pipeline and Hazardous Materials Safety Administration (PHMSA), pursuant to Chapter 601 of 49 United States Code (U.S.C.), inspected by video conference Caliber Spring Creek, LLC’s procedures for Control Room Management (CRM), SCADA, Leak Detection, and Operations and Maintenance (O&M) relevant to the control room located in Houston, Texas. Caliber Midstream Partners, LP (Caliber) operates multiple systems: Caliber North Dakota, Caliber Bear Den Interconnect, and Caliber Spring Creek Pipeline System (CSC). On October 29, 2019, CSC assets were monitored and controlled by a third party control room, NuGen Automation. On October 1, 2021, NuGen Automation became part of Everline Automation (Everline). At the time of the inspection, CSC assets were operated by a third-party control room, Everline. Everline was still using some of the NuGen Automation procedures. Everline’s CRM procedures apply to CSC. As a result of the inspection, PHMSA has identified the apparent inadequacies found within Caliber and CSC’s plans or procedures. The items inspected and the inadequacies are described below: 1. § 195.402 Procedural manual for operations, maintenance, and emergencies. (a) General. Each operator shall prepare and follow for each pipeline system a manual of written procedures for conducting normal operations and maintenance activities and handling abnormal operations and emergencies. This manual shall be reviewed at intervals not exceeding 15 months, but at least once each calendar year, and appropriate changes made as necessary to insure that the manual is effective. This manual shall be prepared before initial operations of a pipeline system commence, and appropriate parts shall be kept at locations where operations and maintenance activities are conducted. “Caliber Midstream Operations and Maintenance Manual Hazardous Liquids Pipelines” (revised September 23, 2020) procedure (O&M), section 2A, entitled “Normal Operations and Maintenance, Caliber Spring Creek Pipeline System,” dated February 20, 2020, was inadequate as it referenced incorrect control room management procedures and did not adequately describe the process used by Caliber and CSC to determine which facilities are control rooms, as required by 49 CFR §§ 195.402 and 195.446(a). Specifically, O&M section 2A.15, entitled “Communications and Control System,” stated, “The Caliber Spring Creek System’s Control Room Management program is operated by NuGen Automation. A full copy of NuGen CRM program has been made readily available. Please see Control Center-SOP-CRM. (NGA-CNTL- SOP-0001).” However, at the time of the inspection, the CRM procedures presented by Caliber as the relevant operating procedures for the third-party control room was “Everline CRM Pipeline Control Room Management” (CSC CRM), version 15, dated May 20, 2016. Caliber’s O&M procedure section 2A.15 specific to CSC requires amendment to address the correct CRM procedure currently used for operations and it must be properly integrated as required. Additionally, during the inspection, when Caliber was asked how a control room was determined, Caliber referenced section 1.3 of CSC’s CRM procedures. However, this section of the procedure was inadequate as it did not describe the process used by Caliber and CSC to identify control rooms; rather it described the process used by Everline to define a control room. Further, this section only applied to the CSC assets authorized for Everline to operate. Caliber’s procedures require amendment to clarify the process used by Caliber on any assets to identify a control room (including those used by CSC). During the inspection, PHMSA reviewed Caliber’s O&M procedure section 4.5.4, entitled “Suspected Leaks,” and this section referenced SOP 1001-2002 entitled “Crude Oil System Operating Procedures.” PHMSA requested this procedure (RI #11) and Caliber responded, “This SOP does not exist in the Caliber system.” Caliber’s O&M procedures require amendment to clarify what will be done for suspected leaks and reference the correct shutdown procedures. The incorrect procedure referenced requires removal. For these reasons, Caliber and CSC's O&M procedures were not adequate and require amendment, per § 195.402 and § 195.446(a). 2. § 195.408 Communications. (a) . . . . (b) The communication system required by paragraph (a) of this section must, as a minimum, include means for: (1) . . . . (3) Conducting two-way vocal communication between a control center and the scene of abnormal operations and emergencies; and Caliber’s O&M procedure section 4, entitled “Emergency Procedures,” did not adequately address communications between a control center and the scene of abnormal operations and emergencies as required by § 195.408(b)(3). Throughout sections 4.4 through 4.9 (entitled “Receiving, Identifying, Classifying Notices,” “Vapor Detected Inside or Near a Building,” “Natural Disasters,” “Extreme Weather and Natural Disasters,” “Fire and Explosion,” and “Extortion Threats”), the communication with the control room is only referenced once, in section 4.6.2, entitled “Wildfires.” Section 4.6.2 stated, “If a wildfire is observed by Caliber personnel in an open field, forest, or other area it should be immediately reported to the control room….” This is the only section that required communication with the control room; however, the control room would need to be made aware of any of the conditions described in sections 4.4 through 4.9. Even within section 4.6.2, the control room communication was not properly addressed. Section 4.6.2 included two conditions that do not require contacting the control room yet clearly could impact the control room. Thus communication should be required. First, the procedure did not require contacting the control room in a situation where a wildfire is near or approaching an above ground valve station, despite the fact that the above ground valve station can impact valve position or valve controls on the pipeline. Second, the procedure did not require contacting the control room in a situation when a fire is reported by a third party. This would also be significant to communicate to the control room. The procedure did not adequately address communication with the control room in multiple other sections. Sections 4.15, entitled “Alerting and Notification Procedure,” 4.2.1, entitled “Readiness,” 4.2.2, entitled “Availability of Resources,” 4.3.3, entitled “Containment,” 4.3.4, entitled “Protection of Public,” 4.10.2, entitled “Operations Manager or Designee,” and 4.14.1, entitled “Media Relations,” all referenced completing notification as described in Section 4A.14. However, nothing in section 4A.14 required communication with the control room. The Emergency Procedures referenced oil response or oil spill response procedures in sections 4.2.3, entitled “Organization Basics,” 5.4.5, entitled “Spill Control Training,” and 5.4.6, entitled “Emergency Prevention Training;” however none of these sections referenced communication with the control room. The Emergency Procedures also addressed oil spill response and Federal Response Plan (FRP) relevant to Part 194 (located in sections 4.2.1 “Readiness,” 4.5.3, “Duties & Requirements,” 4.3.5, “Reviews,” and 5.4.1 “Emergency Response Drill /Tabletop”) but again, nothing 3. referenced communication with the control room. Additionally, nothing in the CSC’s CRM procedures addressed aspects of FRP plans and Part 194 for CSC. For these reasons, the Emergency Procedures relevant to all Caliber pipeline systems (applicable also to CSC) require amendment to identify a process for communication with any control room, including a third-party control room, to comply with § 195.408(b)(3). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) Roles and responsibilities. Each operator must define the roles and responsibilities of a controller during normal, abnormal, and emergency operating conditions. To provide for a controller's prompt and appropriate response to operating conditions, an operator must define each of the following: (1) A controller's authority and responsibility to make decisions and take actions during normal operations; CSC’s CRM procedures did not adequately define a controller’s authority and responsibilities to make decisions and take actions during normal operations, as required by § 195.446(b)(1). This control room monitors and controls multiple operators’ assets in addition to CSC. At the time of the inspection, the control room had 10 consoles that were manned by individual controllers, and 2 supervisors located in the control room. Console 8 has CSC assets but this control room cross- trained controllers for console 7 and console 8. During the inspection, Caliber and the control room personnel communicated to PHMSA inspectors that all controllers can access any console if they have log-in privileges to the SCADA system, including those assets on consoles for which the controllers are not qualified. Any person with SCADA log-in access could execute commands on other assets. Further, more than one controller could access a given pipeline system at the same time (such as CSC). Nothing prevented command actions for the same asset from being executed by multiple controllers on different consoles. The procedures did not clarify which controller is to make decisions and take actions on the CSC system at any given time. CSC’s CRM procedure Exhibit B, entitled “Domain of Responsibility Table,” indicated that Watford PMP101 and PMP102 pumps are commanded by a controller; however, the indications on the display did not allow for commanding these pumps. PHMSA was verbally told during the inspection that these pumps were not commanded as indicated in Exhibit B. Additionally, CSC’s CRM procedures included inaccurate titles of control room personnel. For example, at the time of inspection, “Lead Controllers” were referred to as “Control Room Supervisors.” Amendment is required so that each employee knows their roles and responsibilities as stated in the procedures without confusion. Further, during the inspection it was confirmed with Caliber personnel that the control room did 4. 5. not receive 811 calls during or after normal work hours. CSC CRM procedures in section 3.4, entitled “Controller Responsibility,” Item 14 in the “Controller-Normal Operations” table indicated the control room does address 811 calls. CSC CRM procedure requires amendment specific to 811 calls for CSC assets. For these reasons, Caliber and CSC’s procedures were inadequate to define a controller’s authority and responsibilities during normal operations as required by § 195.446(b)(1). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) Roles and responsibilities. Each operator must define the roles and responsibilities of a controller during normal, abnormal, and emergency operating conditions. To provide for a controller's prompt and appropriate response to operating conditions, an operator must define each of the following: (1) . . . . (2) A controller's role when an abnormal operating condition is detected, even if the controller is not the first to detect the condition, including the controller's responsibility to take specific actions and to communicate with others; CSC’s CRM procedures were inadequate to define a controller’s roles and responsibilities during abnormal operations per § 195.445(b)(2). The control room does not receive 811 calls. The CRM procedure section 3.4, entitled “Controller Responsibility,” Item 9 in the “Controller - Abnormal Operations/Abnormal Operating Conditions” table requires amendment to clarify that 811 calls are not addressed for CSC assets by the control room during abnormal operations/abnormal operating conditions. CSC’s CRM procedures were inadequate to define a controller’s authority and responsibilities during abnormal operations as required by § 195.446(b)(2). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) Roles and responsibilities. Each operator must define the roles and responsibilities of a controller during normal, abnormal, and emergency operating conditions. To provide for a controller's prompt and appropriate response to operating conditions, an operator must define each of the following: (1) . . . . 6. (3) A controller's role during an emergency, even if the controller is not the first to detect the emergency, including the controller's responsibility to take specific actions and to communicate with others; Caliber’s O&M and CSC’s CRM procedures were inadequate to define a controller’s role during an emergency, even if the controller is not the first to detect the emergency, including the controller's responsibility to take specific actions and to communicate with others, per § 195.446(b)(3). Specifically, the CSC’s CRM procedures had multiple inconsistencies when defining a controller’s role and responsibility during an emergency. CRM procedure section 1.8 referenced a document entitled “Everline’s Emergency Evacuation Procedure,” but that document did not exist. Also, CRM procedure section 4.9.3, entitled “Unplanned Evacuations (e.g. bomb threat, fire threat),” in Item C referenced “mobile go kits” that are no longer used during emergencies, as confirmed by control room personnel during the PHMSA inspection. Additionally, CRM procedure, section 3.4, entitled “Controller Responsibility,” Item 6 “Controller – Emergency Operations” table did not incorporate or provide reference to the appropriate Internal Communication Plan. If an evacuation occurs of the control room, the Internal Communication Plan may be implemented until the back-up center is up and running with controllers present or until the specific situation that caused the evacuation is resolved. Thus, reference to the Internal Communication Plan is necessary. Further, CRM procedure section 4.9.1 did not identify whether 800 calls would be received for CSC. During the inspection, PHMSA asked how 800 calls were handled for Caliber, and CSC, and conflicting answers were provided by Caliber, CSC, and the control room. As a result, this needs to be clarified in procedures. This was not presented in the O&M procedures as to how this was integrated with the control room. Caliber’s O&M and CSC’s CRM procedures require amendment to adequately define a controller’s role during an emergency, even if the controller is not the first to detect the emergency, including the controller's responsibility to take specific actions and to communicate with others, per § 195.446(b)(3). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) Roles and responsibilities. Each operator must define the roles and responsibilities of a controller during normal, abnormal, and emergency operating conditions. To provide for a controller's prompt and appropriate response to operating conditions, an operator must define each of the following: (1) . . . . (4) A method of recording controller shift-changes and any hand-over of responsibility between controllers; and. . . . CSC’s CRM procedures were inadequate to define a method of recording shift-changes and any hand-over of responsibility between controllers, per § 195.446(b)(4) and § 195.446(c)(5), which requires the operator to implement section 5 of API RP 1168 (First Edition, September of 2008) to establish procedures for when a different controller assumes responsibility, including the content of information to be exchanged. During the inspection, CSC and control room personnel explained that the Shift Turnover process had moved from the “Shift Turnover Checklist,” Form 11-1 (this was referenced in CSC’s CRM procedures, sections 3, entitled “Responsibility,” and 5, entitled “Shift Turnover”) to an electronic form known as Elogger. However, the Elogger and the Shift Turnover Checklist were not the same, so the CRM procedures require revision in order to confirm what information is to be exchanged between controllers. Additionally, when changing the shift turnover form to the Elogger, the CSC CRM procedure section 5, entitled “Shift Turnover,” did not include step-by-step instructions for populating the form nor did it clarify that the “Yes/NA” data field will be blank at the start of each shift change, and comments must be added for each element when completing the shift change as explained by the operator during the inspection. CSC’s CRM procedures did not require controllers to log off the SCADA system when leaving the console. However, this was clearly communicated during the inspection as an expectation by control room representatives, and older shift turnover documentation from 2021 had this step identified as an instruction rather than a requirement. CSC’s CRM procedures in sections 3 and 5 did not describe this required action. Shift Turnover documentation from 2022 also did not include this instruction. During the inspection, PHMSA determined that manual tracking sheets were populated for operational changes made during the shift. CSC’s CRM procedures did not address this documentation. CSC’s CRM procedures require amendment to include any manual sheets that are populated as part of the shift turnover process and associated records. Caliber’s O&M and CSC’s CRM procedures did not include how volume movements and any imbalances will be monitored and communicated to controllers as part of the shift change process. PHMSA determined during the inspection that Caliber provided a summary of all meters in and out of the system to the control room daily to assist with imbalance tracking on the CSC system. Caliber’s O&M procedures and CSC’s CRM procedures did not define how a controller received the operations schedule (shipping or movement requirements) or imbalance information on a daily basis, however all of this information would need to be included as part of the Shift change documentation. This information is required for a controller to perform normal pipeline operations and to assist with abnormal operations detection. It was not clear if Caliber or CSC provides the information by email to a control room manager or supervisor for distribution to controllers, if controllers receive this information directly, or if the SCADA system automatically receives this information. CSC’s CRM procedures did not define whether any of the operations scheduling values require manual entry by a controller. CSC’s CRM procedure section 3 did not address an emergency where the controller is incapacitated and does not plan to leave the console. Similarly, CSC’s CRM procedure did not address if anyone can log off of the console for someone else (such as would be required should a controller not be able to log off the system). As such, the procedures need to be amended to include what actions will be required of the operator. 7. CSC’s CRM procedure section 3.3, “Impromptu Change in Responsibility,” did not clearly define the role and responsibilities of a controller when planning to leave the console for less than 30 minutes. The procedure should clarify that the controller must confirm before leaving the control room that the individual answering the phones has the controller’s correct contact information as part of the process for handover. Section 3.3 did not clarify if the individual assigned to answer the phone must have qualification to the console being vacated for 30 minutes. Moreover, section 3.3 did not address how the individual answering the phones will also be responsible for confirming that no other controller or person in the control room activates commands or makes keyboard actions while the qualified and scheduled controller has stepped away. Section 3.3 did not adequately define what would be required for a shift turnover if a controller has left the console for more than 30 minutes. An adequate procedure would include how the incoming controller would handle logging off for the previously scheduled controller. For these reasons, Caliber’s O&M and CSC’s CRM procedures were inadequate for shift turnover when a different controller assumes responsibility, for implementing section 5 of API RP 1168, in identifying the content of information to be exchanged, for clarifying actions to be taken by the operator or controller, and in defining a method of recording shift-changes and any hand-over of responsibility between controllers as required by § 195.446(b)(4) and § 195.446(c)(5). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) Roles and responsibilities. Each operator must define the roles and responsibilities of a controller during normal, abnormal, and emergency operating conditions. To provide for a controller's prompt and appropriate response to operating conditions, an operator must define each of the following: (1) . . . . (5) The roles, responsibilities and qualifications of others who have the authority to direct or supersede the specific technical actions of controllers. Caliber’s O&M and CSC’s CRM procedures were not adequate to define the roles, responsibilities, and qualifications of others who have the authority to direct or supersede the specific technical actions of controllers as required by § 195.446(b)(5). The CRM procedures require amendment to adequately address direct and supersede in section 3 and to adequately cover how restarts are achieved after a potential release. CRM procedure section 3 indicated that only the lead controller can direct or supersede a controller action if they are qualified as a controller to the specific console. Section 3 also noted that a record of Lead Controllers with this ability will be listed in the “Domain of Responsibility Table” (located in CSC’s CRM procedure Exhibit B). However, Exhibit B’s “Domain of Responsibility Table” did not address directing the controller’s actions. Also Exhibit B included names of individuals that were no longer qualified controllers on Console 8. Exhibit B requires amendment to correctly identify who can direct and supersede, not just supersede, controllers’ actions. Additionally, CSC’s CRM procedure in section 3.1, “Authority and Responsibility,” stated “Authority to restart a pipeline that has been shut down due to a potential release should follow the Asset Specific O&M Manuals or Emergency Response Procedures (ERP) Manuals.” However, Caliber’s O&M and ERP Manuals did not indicate who would approve a restart beyond O&M sections 3.4, “Maintaining Design Limits,” and 4.6.2, “Wildfires.” This did not adequately address who from Caliber or the control room would approve a restart and if communication with the Lead Controller would be required so that a controller could be directed to restart the pipeline. At the time of the inspection, when discussing this with Caliber and control room personnel, it was not clear if any employee from Caliber (including any specific field personnel) would provide approval for a restart or how the Lead Controller would be made aware of this information. Caliber’s O&M and CSC’s CRM procedures require amendment to adequately define the roles, responsibilities, and qualifications of others who have the authority to direct or supersede the specific technical actions of controllers per § 195.446(b)(5). 8. § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (c) Provide adequate information. Each operator must provide its controllers with the information, tools, processes and procedures necessary for the controllers to carry out the roles and responsibilities the operator has defined by performing each of the following: (1) Implement API RP 1165 (incorporated by reference, see § 195.3) whenever a SCADA system is added, expanded or replaced, unless the operator demonstrates that certain provisions of API RP 1165 are not practical for the SCADA system used; CSC’s CRM procedures were not adequate to determine when implementing API RP 1165 would be required, as per § 195.446(c)(1). Specifically, CSC’s CRM procedures and referenced procedures (such as the Everline SCADA Functional Technical Specification) did not define a SCADA system addition, expansion, or replacement. The CRM procedures were not clear on whether API RP 1165 would be implemented when hardware or software (such as server architecture changes, console additions, software and hardware revisions including upgrades, assets being added or moved, a separate SCADA system being added to this control room or a SCADA system change) changes were made to the SCADA system. CSC’s CRM procedures referenced API RP 1165 in various ways (for example the procedures did state “1165,” “recommended practice RP 1165,” etc. in sections 1.8, “Reference Documents,” 1.10, “Definitions,” 2.4, “Record Keeping,” 3.8, “SCADA/IT/Network Operations,” 4.5, “Pipeline SCADA Displays and API RP-1165,” and in Form 11-17, “Pipeline SCADA Displays 1165 Checklist.”). CSC’s CRM section 4.5, “Pipeline SCADA Displays and API RP-1165,” stated “[t]he requirements shall be waived if it is demonstrated that the API RP- 1165 requirements are not practical to implement. This may be demonstrated by a description, in memorandum or other form, of why API RP-1165 is not appropriate for a given system change.” However, “Not appropriate” is not equivalent to “not being practical” for implementation and the language requires revision. CSC’s CRM procedure referenced the SCADA Functional Technical Specification; the CRM procedure did not reference the NuGen Display Guide. At the time of the inspection, PHMSA requested a copy of the SCADA Functional Technical Specification, but the document provided to PHMSA as a result of this request was titled “NuGen OCC Display Standard” (referred to below as NuGen Display Guide). CSC’s CRM procedure and SCADA Functional Technical Specification (called the “NuGen Display Guide” during the inspection) were inadequate and require the following amendments: 1) The NuGen Display Guide, section 4.0, “Display Hardware,” needs to be updated to include the primary and backup center locations (as different configurations were identified for these locations: 12 monitors per console in the primary control room, 9 monitors per console in the back up location, 8 monitors were listed in the NuGen Display Guide, CRM 4.5.2 referenced 10-14 monitors depending on workload). 2) The NuGen Display Guide, section 5.0, “Display Layout and Organization (PHMSA c1-5),” requires multiple amendments. Subsection 5.1, “General Design Considerations,” was not clear that section 2, “General Considerations (PHMSA C1- 3),” of the Nugen Display Guide was relevant, not section 2 of API RP1165, entitled “References.” Section 5.1 should also reference section 3 in this same document, entitled “Human Factors Engineering (PHMSA C1-4),” and it did not. Subsection 5.2, “Display Hierarchy,” was not clear regarding API RP1165 applicability and needs to be changed to start with Level 1 and work through Level 4. Section 5.3, “Window Management,” referenced screen savers however at the time of the inspection screen savers were disabled for CSC. Additionally, this same section did not address the number of windows being open at one time as limited to 100 (the control room personnel indicated this verbally during the inspection, but this was not clarified in specifications or procedures). During the inspection, PHMSA observed that Object Menus and other display features had various permission levels for access (such as SCADA versus that for the controller), but this was not described at any location in the procedures or specifications and would impact various sections such as section 5.3.2, “Toolbars.” It was not clear for each of the menu drop down options what controllers would be allowed to access. 3) The NuGen Display Guide, section 7.0, “Object Characteristics (PHMSA C1-7),” was not clear on how it is employed. Procedures should be clarified on how flow setpoints (or pressure setpoints) would be linked in the display to the corresponding control valve. API RP 1165 states in section 9, entitled “Object Dynamics,” that “Object dynamics refers to changes in an object’s characteristics associated with changes in the 4) 5) 6) 7) database point(s) tied to that object.” However, PHMSA could not determine how a setpoint in the display would be linked to the control valve the setpoint impacted. An example was the flow setpoint associated with Tesoro 12177 34th Street NW would affect the control valve V335 on this display. However, the valve (object) was not linked to the flow setpoint on the display. The NuGen Display Guide section 7.1.1, “Color,” utilized incorrect colors for what was used in the system, as observed by PHMSA during the inspection, and referenced in other sections of the same document and found in the CSC CRM procedures. The colors defined in section 7.1.10, “Alarm Colors,” differed from the colors described within the same document in section 7.1.1. Further, the colors referenced in the CRM procedures sections 8.4, “Alarm Design,” and 8.5, “Alarm Priority,” conflicted with the colors referenced the NuGen Display Guide section 7.1.1. Clarification is required to describe whether a loss of communication and a loss of the equipment is displayed differently than a pink color as referenced in section 7.1.1 versus that of section 7.1.11, “Reserved Colors.” The NuGen Display Guide in section 7.1.3, “Dynamic Lines and Symbols,” referenced Appendix A, “Symbols.” However, Appendix A did not reference symbols or objects that are being used in displays. Manual valve symbols were not defined, valves in travel status were not described as to how this would be displayed, tanks needed to be included, and 2- and 3-way valves should be added. The check valve symbol requires updating as the symbol in the procedures was not the symbol that was being used in the displays observed during the inspection. Lease Automatic Custody Transfer (LACT) units need to be added. Information provided should reflect what would occur or change on the symbol or object should the element go into alarm or change status (such as a pump changing from local to remote, manual to automatic, or Hand to Auto). The NuGen Display Guide, section 8.3.1, “Data States,” needs to be updated to match the data states identified in CSC’s CRM procedure section 8.10.6.3, “Data Quality & Configuration Issues,” and to clarify what color will be used. The information in the procedure did not address how calculated points will be displayed if one of the required inputs to the calculation is out of communication or in alarm status. Error coding and associated color also needs to be clarified. The NuGen Display Guide section 9, “Control and Selection Techniques (PHMSA C1- 9),” did not clearly define what happens when a valve is commanded to a state (such as open or closed) and the state was not achieved in a certain amount of time. The procedure did not clarify what happens and how this would be displayed. Similarly, it was not clear what would be displayed if a valve shows open and closed at the same time. Section 9 also needs further amendment to define each step required to execute a command (for example, click on the symbol/object, view pop-up menu, identify what is selected, and then include what is done to execute the command). Section 9’s error management was inadequate as described and should be amended to address other elements such as maximum keystrokes, clamped values on setpoint maximums or minimums. It was also not clear how valve status and pumps would be shown, such as 8) 9) 10) 11) open color green with an O, on for pumps with green, or R, along with green, for running. Additionally, section 9.3, “Error Management,” stated “Errors are logged in the log files stored in predefined locations. If errors are found, they are addressed.” However, the procedures did not define what is done with the log files nor did it identify where the logs reside (on the share drive, in SCADA, reviewed Monthly, etc.). During the inspection, PHMSA determined that flatlining is an option within the SCADA system. Thus, flatlining should be referenced and identified when it will be used and how it will be represented to the controller in procedures. The NuGen Display Guide, section 10.2, “Consistency between control centers and remote locations” stated “[t]he redundant systems are monitored monthly to be ready for operation if needed.” During the inspection PHMSA determined that monthly monitoring is no longer practiced. The procedures should be amended to reflect the updated process. CSC’s CRM procedures, section 4.5.3, “Consistency,” and the NuGen Display Guide section 3.5, “Consistency Across the Company,” both addressed consistency in a way that was inconsistent. The CRM procedures stated, “Display designs are documented in Everline SCADA Functional Technical Specification or the Client’s SCADA standard. For consoles where a Client’s SCADA system is displayed, the control room provides a copy of the SCADA standard at the console for reference, conducts additional training on the Client’s SCADA standard and provides a summary template identifying key differences between Everline SCADA Functional Technical Specification and Client’s SCADA standard. CSC’s CRM procedures indicated that all SCADA standards referenced in this manual are listed in Exhibit Z. However, Exhibit Z is the “Alarm System Specifics” and did not contain anything relevant to displays and consistency in standard or practice. Thus, CSC’s CRM procedures require amendment to address this issue. Additionally, NuGen Display Guide stated, “Consistency is important from not only a specific coding aspect but also when looking at the system. Ensuring consistency across the entire company and system ensures that personnel can deal with each other efficiently and effectively when working on the system whether it is in the same location or in-between the control center and remote locations. Every attempt has been made to ensure that NuGen’ s SCADA system is as consistent as possible and that all displays follow standardized rules for size, location, colors, symbols, text formatting, etc. as described in this document.” However, this document did not describe how consistency between remote locations and the SCADA system is achieved or reviewed. Section 3.5 did meet the intention of section 11 in API RP 1165; however, it is in direct conflict with CSC’s CRM section 4.5.3, as there could be two different display standards, or more being used even within the same console. CSC’s CRM procedures and the NuGen Display Guide were inadequate when describing how LACT status would be presented to controllers. Procedures did not describe Local/Remote status, applicable text to be presented and location of that text, or color to be used for specific backlighting and meaning of those colors. 9. For these reasons, CSC’s CRM procedures were not adequate and require amendment to determine when implementing API RP 1165 would be required as per § 195.446(c)(1). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (c) Provide adequate information. Each operator must provide its controllers with the information, tools, processes and procedures necessary for the controllers to carry out the roles and responsibilities the operator has defined by performing each of the following: (1) . . . . (2) Conduct a point-to-point verification between SCADA displays and related field equipment when field equipment is added or moved and when other changes that affect pipeline safety are made to field equipment or SCADA displays; Caliber’s O&M and CSC’s CRM procedures were not adequate to define point-to-point procedures, per § 195.446(c)(2), which requires an operator to conduct point-to-point verification between SCADA displays and related field equipment when field equipment is added or moved and when other changes that affect pipeline safety are made to field equipment or SCADA displays. Specifically, CSC’s point-to-point procedures did not address alarms or adequately define points that are necessary for the controller to carry out the roles and responsibilities and points that, when changed, could affect pipeline safety. CSC’s CRM procedure, section 8.6, “Safety-related Alarms and Points,” stated that “Everline defines safety related points as SCADA points that are necessary to maintain pipeline integrity or that could lead to the recognition of a condition that could impact the integrity of the pipeline, or a developing abnormal or emergency situation.” Section 8.6 also indicated that “Everline defines safety related alarms as alarms that are necessary to maintain pipeline integrity or that could lead to the recognition of a condition that could impact the integrity of the pipeline, or a developing abnormal or emergency situation.” However, Section 8.9.7, “Point-to-Point Verification,” indicated that: “Safety related points often, but do not necessarily, have alarms associated with them. Examples of points that may be considered safety related (and therefore would need to be verified when changes are made to field equipment or SCADA displays) can include, but may not be limited to: • Status of mainline valves • Mainline pressures and flow rates • Tank Levels • Station in local Control • Personnel in normally unmanned station • Station inlet and discharge pressures • Pump/compressor status • Leak Detection • Pressure Regulator inlet and outlet pressures • PLC/RTU Communications Status • Emergency Shutdown Status • Odorant alarms • Composition alarms, such as H2S and Water content • Filtering equipment levels-scrubbers/dehy • Flame, gas and vapor detectors • Power supply indications (Low Battery, AC power failure, UPS failure) • Security monitoring.” However, section 8.9.7 was inadequate as it did not address what specifically was relevant to CSC. Additionally, while pressure and flow alarms may be included per procedure, LoLo pressure alarms (versus HiHi pressure alarms only) and flow alarms (HiHi and LoLo) had not been clarified for implementation on CSC assets. Inclusion of such alarms is required for abnormal operations and leak detection. Further, loss of communication alarms were not addressed, yet each of these points and alarms would be impacted by a loss of communication for that input to the SCADA system. Each of these elements impact safety, assist with abnormal operating conditions detection, or are needed to assist controllers with carrying out their roles and responsibilities, and thus are required to be defined. Caliber's O&M procedure, section 3, “Abnormal Operations,” identified high temperature as an indicator for abnormal operation. However, CSC’s CRM procedure section 8.9.7 did not identify temperatures that are necessary for controllers to carry out their roles and responsibilities and thus was not adequately integrated with Caliber’s O&M. CSC’s CRM procedure, section 8.9.7, “Point-to-Point Verification,” did not clarify that records would include: (1) a listing of all displays that were reviewed; (2) the alarm setpoint values and alarm descriptors checked; (3) confirmation that the alarm and event logging function has been implemented; (4) verification of appropriate display response to various alarm conditions including priority (such as flashing or a certain color appearing); (5) simulation locations and the reason for the simulation being used; and (6) scaling values (the procedures did not explain that unless scaling is set to "no," the display will use the device engineering units scaled range automatically). For these reasons, Caliber’s O&M and CSC’s CRM procedures were not adequate and require amendment to address conducting a point-to-point verification between SCADA displays and related field equipment when field equipment is added or moved and when other changes that affect pipeline safety are made to field equipment or SCADA displays, per § 195.446(c)(2). 10. § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (c) Provide adequate information. Each operator must provide its controllers with the information, tools, processes and procedures necessary for the controllers to carry out the roles and responsibilities the operator has defined by performing each of the following: (1) . . . . (3) Test and verify an internal communication plan to provide adequate means for manual operation of the pipeline safely, at least once each calendar year, but at intervals not to exceed 15 months; Caliber’s O&M and CSC’s CRM procedure section 4.6, entitled “Internal Communications,” were inadequate to properly test and verify an internal communication plan to provide adequate means for manual operation of the pipeline. It is not clear from Caliber’s O&M or CSC’s CRM procedures how an internal communication plan would be tested and verified to provide a means for manual operation of the pipeline safely. Neither Caliber’s emergency procedures (part of the O&M) nor the control room’s emergency procedures addressed communication with the control room except during wildfires. Additionally, Caliber’s O&M procedures did not reference or identify an internal communication plan. Further, CSC’s CRM procedures require amendment to clarify that the internal communication plan for manual operations can be required for reasons other than loss of communication, such as control room evacuations when the back-up is not yet online, or a cyber-attack impacting operations from the control room. CSC’s CRM procedures did not clarify that upon a partial or total communication loss that exceeded 30 minutes, activating the internal communications plan would occur. This was verbally indicated by control room personnel during the inspection as what would happen but was not described in procedures. Caliber’s O&M and CSC’s CRM procedures were inadequate to test and verify an internal communication plan to provide adequate means for manual operation of the pipeline safely, at least once each calendar year, but at intervals not to exceed 15 months as required by § 195.446(c)(3). 11. §195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section . . . . 12. (b) . . . . (c) Provide adequate information. Each operator must provide its controllers with the information, tools, processes and procedures necessary for the controllers to carry out the roles and responsibilities the operator has defined by performing each of the following: (1) . . . . (4) Test any backup SCADA systems at least once each calendar year, but at intervals not to exceed 15 months; and CSC’s CRM procedures were inadequate to properly test any backup SCADA systems at least once each calendar year, but at intervals not to exceed 15 months, per the requirements of § 195.446(c)(4). Specifically, CSC’s CRM procedure section 4 did not address how servers will be switched in order to adequately test all servers associated with the backup SCADA system. Without this aspect of the test, the documentation did not confirm that all servers are experiencing extended run time. Section 4 did not reference the SCADA process or SOP used for switching servers and did not indicate what will be used as the record to document that this occurred (such as using the event log or other record described in procedures). Additionally, section 4 did not clarify the set of information and tools that would be confirmed to be working at the primary control room when returning from the backup location. CSC’s CRM procedures were not adequate and require amendment to properly test any backup SCADA systems at least once each calendar year, but at intervals not to exceed 15 months, per the requirements of § 195.446(c)(4). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (d) Fatigue mitigation. Each operator must implement the following methods to reduce the risk associated with controller fatigue that could inhibit a controller's ability to carry out the roles and responsibilities the operator has defined: CSC’s CRM procedures were inadequate to evaluate controller fatigue risks for the control room, per § 195.446(d), which requires each operator to implement methods to reduce the risk associated with controller fatigue that could inhibit a controller's ability to carry out the roles and responsibilities the operator has defined. Specifically, CSC’s CRM procedures require amendment to identify the actual risks of fatigue relevant to the control room and individual consoles. During the inspection, PHMSA determined that the control room shift schedule had been changed from that of the Dupont schedule to a schedule of 4 on and 2 off. However, nothing in the procedures or records clarified how the risk of fatigue had been accounted for when 13. 14. implementing this change. CSC’s CRM procedures did not adequately define a process that would identify how the risk of fatigue would be reduced in advance of a controller revised schedule being implemented. CSC’s CRM procedures were inadequate and require amendment to evaluate controller fatigue risks for the control room, per § 195.446(d). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (d) Fatigue mitigation. Each operator must implement the following methods to reduce the risk associated with controller fatigue that could inhibit a controller's ability to carry out the roles and responsibilities the operator has defined: (1) Establish shift lengths and schedule rotations that provide controllers off-duty time sufficient to achieve eight hours of continuous sleep; CSC’s CRM procedures were inadequate to define a process establishing shift-lengths and schedule rotations that provided off-duty controllers sufficient time to achieve eight hours of continuous sleep, per §§ 195.446(d)(1) and (j)(1), which require the operator to maintain records that show compliance with § 195.446. Specifically, CSC’s CRM procedures did not have a process that required actual hours worked to be recorded for all individuals that are qualified as controllers, including those working only Monday through Friday. Lead Controllers or Supervisors actual hours worked were not included in a process, yet these individuals, when qualified, could work as a controller on CSC assets. Further, informed fatigue mitigation related decisions need to be made when filling the console for an unplanned schedule modification. CSC’s CRM procedures did not describe how the risk of fatigue is considered when determining the replacement controller. CSC’s CRM procedures were inadequate and require amendment to define a process establishing shift-lengths and schedule rotations that provided off-duty controllers sufficient time to achieve eight hours of continuous sleep, per §§ 195.446(d)(1) and (j)(1). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (d) Fatigue mitigation. Each operator must implement the following methods to reduce the risk associated with controller fatigue that could inhibit a controller’s ability to carry out the roles and responsibilities the operator has defined: (1) . . . . (4) Establish a maximum limit on controller hours-of-service, which may provide for an emergency deviation from the maximum limit if necessary for the safe operation of a pipeline facility. CSC’s CRM procedures were inadequate as a maximum limit on controller’s hour of service was not clearly established, which may provide for an emergency deviation from the maximum limit if necessary for the safe operation of a pipeline facility, per § 195.446(d)(4). Specifically, CSC’s CRM procedure section 6.3, “Maximum Hours of Service,” required a 35 hour reset when 65 hours in any sliding 7 days have been met. However, the 35-hour reset was again referenced under the note for this section and it was listed as "may be used." The procedure has an internal conflict and thus requires amendment. In addition, during PHMSA’s inspection, the control room personnel conveyed that an hours-of- service deviation will not be required for working 72 hours, instead it will be required for working 65 hours without a 35-hour reset. This process was not detailed in the procedures, and thus CSC’s CRM procedures require amendment to clarify this aspect. CSC’s CRM procedure sections 4.9.1, “Planned Evacuation – Deployment to Backup Facility,” and 4.9.2, “Planned Evacuation – Re-Deployment to Primary Facility,” were not adequate to clarify how a reset or rest period of 35 hours would be implemented for CSC controllers when traveling to the back-up facility for a planned test. During the inspection, control room personnel explained that 3.5 hours would be allowed for controllers to travel to the back-up location for the test. It was not clear how the hours required to travel to the back-up control room would be included in the total hours worked by the controller and when the rest period of 35 hours would be implemented (such as 35 hours rest before performing the back-up test, 35 hours rest upon returning to the primary control room location and after test completion, or the 35 hours rest was implemented when the total hours worked in any sliding 7 days exceeded 65 hours on an individual controller basis regardless of back-up control room test status). Sections 4.9.1 and 4.9.2 need to be amended to clarify when the 35 hour reset period will be implemented for back- up control room tests and describe how travel time to or from the back-up control room will factor into total hours worked. Further, CSC’s CRM procedures were not clear on how the Control Room Manager would select a replacement controller in an emergency situation in section 5.4, “On-Call Controllers.” The CRM procedures require amendment to clarify the control room process. CSC’s CRM procedure, section 6.16, “Investigating Possible Contribution of Fatigue to Pipeline Incident/Accident,” Item 2, in the section entitled “Initial Screening Questions,” did not adequately consider elevated fatigue risk that occurs for time worked after the third consecutive night. This section in the procedures requires clarification. CSC’s CRM procedures need to include fatigue related informed decisions that clarify where the 15. list of persons qualified to each console can be found and identify what data systems would be used to review current actual hours worked (timesheet tool only indicated the previous week for actual hours and did not include current values for the week). This was inadequate to confirm reset periods and off-duty activities to achieve 8-hours of sleep had been implemented. For these reasons, CSC’s CRM procedures were inadequate and require amendment to demonstrate compliance with § 195.446(d)(4). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (e) Alarm management. Each operator using a SCADA system must have a written alarm management plan to provide for effective controller response to alarms. An operator's plan must include provisions to: (1) Review SCADA safety-related alarm operations using a process that ensures alarms are accurate and support safe pipeline operations; CSC’s CRM procedures were not adequate to review SCADA safety-related alarm operations using a process that ensures alarms are accurate and support safe pipeline operations, per § 195.446(e)(1). CSC’s CRM procedure section 8.5, “Alarm Priority,” was in conflict with Everline’s SCADA Functional Technical Specification document, section 7.1.10, “Alarm Colors” (cover page is called NuGen Automation SCADA Display Standard). The CSC CRM procedures identified three priorities of alarms and three associated colors: Critical is red; High is yellow; and Low is Cyan. The Everline SCADA Functional Technical Specification identified four priorities of alarms and four associated colors: Critical is Brown; High is Red; Medium is Orange; and Low is Yellow. This conflict requires correction and resolution for all alarm priorities in order to ensure alarms are accurate and support safe operations. Additionally, CSC’s CRM procedure, section 8.5, “Alarm Priority,” indicated that communication is identified as a High priority. However, in section 8.6, “Safety Related Alarms and Points,” the CRM procedures stated that Critical priorities were safety related. Section 8.6 also indicated that communication status was a point that impacts safety. The procedures require amendment to indicate that safety related alarms can occur in Critical and other priorities (such as High). CSC’s CRM procedures require amendment to adequately address how false alarms will be reviewed to ensure alarms are accurate and support safe operations. False alarms are a type of inaccurate alarms. False alarms can be discovered in multiple ways (monthly alarm reviews, leak detection review and testing, alarm management effectiveness review, system performance reviews, etc.) and not just as identified by controllers. However, this was not adequately addressed in procedures. 16. CSC’s CRM procedures did not adequately address bad data and associated alarming functions. It was not clear from CSC’s CRM procedures or Everline SCADA functional technical standards how certain bad data indications would be presented to controllers and whether or not this would result in an alarm. For example, PHMSA could not determine from CRM procedures whether out-of-range alarms were present in the system, or whether a loss of pressure transmitter or other end device would result in an alarm. Additionally, PHMSA could not determine how flatlining would be used and alarmed (a SCADA parameter indicating that data was not changing but communication was fine) based on CRM procedures. This information is necessary in CRM procedures to ensure that alarms are accurate and support safe operations. CSC’s CRM procedures did not define a process that addressed how alarms would be reviewed to ensure alarms were accurate and supported safe operations when different alarm management processes were used within an existing control room. The control room operating CSC assets had combined two different control rooms that used different alarm management processes. At the time of the inspection, the control room did not have consistent alarm priorities implemented across all consoles. During the inspection, control room personnel verbally explained that the Remote Operations Center (ROC) prior control room used different systems and tools than that of the NuGen. CSC’s CRM procedures did not clarify how alarms would be reviewed to ensure alarms are accurate and support safe operations for the control room regardless of systems/tools utilized. For these reasons, CSC’s CRM procedures and Everline’s SCADA functional technical standard were not adequate and require amendment to review SCADA safety-related alarm operations using a process that ensures alarms are accurate and support safe pipeline operations, per § 195.446(e)(1). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (e) Alarm management. Each operator using a SCADA system must have a written alarm management plan to provide for effective controller response to alarms. An operator's plan must include provisions to: (1) . . . . (2) Identify at least once each calendar month points affecting safety that have been taken off scan in the SCADA host, have had alarms inhibited, generated false alarms, or that have had forced or manual values for periods of time exceeding that required for associated maintenance or operating activities; CSC’s CRM procedure was inadequate to describe how the monthly review process for points affecting safety that have been taken off scan in the SCADA host, have had alarms inhibited, generated false alarms, or that have had forced or manual values for periods of time exceeding that required for associated maintenance or operating activities was performed, per § 195.446(e)(2). Specifically, CSC’s CRM procedure section 8.10.3 did not clarify how certain aspects of this review would be performed, including identifying the records that will demonstrate compliance. Section 8.10.3 requires amendment as follows: 1) 2) 3) 4) 5) The automated reports that are run by the control room’s Applications Manager need to be identified in the procedures, and a description included of how each report will be used and what it includes defined. a) The Off Scan report incorrectly referenced Programable Logic Controller (PLCs) that are offline, not points that are off-scan in the SCADA host. This is problematic as points being off-scan (active, inactive) can also occur at the SCADA level and are not identified in this report or other reports. b) The Enabled and Disabled Points report identified points that are no longer being polled. This is similar to being off scan in the SCADA system but not sufficiently clarified as to how the report and data will be used. c) The Manual Override report identified data that has been forced in the SCADA system but not at the PLC or instrument level (which is needed to be adequate). d) The Nuisance Alarm reports identified the data that has alarms three or more times in a minute and those points that are in flooding events (per API RP 1167). This is not all types of false alarms, and other types must also be addressed. During the inspection, the control room personnel verbally communicated that the Dashboard is captured at a given date in the month and does identify total alarms on a given console by critical priority. This was then compared to the prior month’s data. However, this was not included in the procedures as part of the monthly review process described in section 8.10.3. The top 20 alarms were reviewed for total alarm counts and reported on, but it was not identified in section 8.10.3. This was done monthly and was part of the review for false alarms but had not been added to procedures. Section 8.10.3 did not sufficiently address loss of communication. The process needs to clarify that loss of communication was included in the monthly alarm review. Section 8.10.3 also needs to clarify where monthly review records and the substantiating data were stored, identify who performed the review, who pulled the information, describe how the information was pulled, and identify how information was communicated to the team performing the review (such as an email from the Everline’s Applications Manager to an email group if the review was performed by those outside of the immediate SCADA support team as described during the inspection). 17. 6) Section 8.10.3 did not include how false alarms during the monthly review will be identified by looking through Elogger or log book entries. False alarms were indicated as section 8.10.3 to be placed in the log book by controllers, however, nothing in the procedures addressed the review of this information as part of the monthly process. Additionally, false alarms also need to include leak detection indications when no actual leak was found. 7) Section 8.10.3 did not include how alarms that are inhibited were reviewed and reported on a monthly basis regardless of who implemented the inhibit (SCADA, analyst, manager, field). Procedures did not result in this information being reviewed as part of the monthly alarm review. This was also not clarified or defined in the Everline SCADA Display Technical Standard (SCADA Functional Technical Standard). Section 8.10.3 requires clarification that the SCADA system does not allow for individual alarm parameters to be inhibited. 8) Section 8.10.3 did not clarify how points that are forced in the RTU/PLCs or at the individual instrument were determined and reported on. This was not covered by the manual override process already looked at for forcing values in the SCADA system. 9) Section 8.10.3 did not define how equipment that has been kept in manual (local versus remote, field versus remote, manual versus automatic, remote versus manual, hand versus remote, etc.) will be reviewed as part of the monthly alarm process. 10) Procedures did not provide clarity on how to consistently populate Form 11-10 Monthly Alarm Review. For example, the number of points taken off scan and recorded in Form 11-10 is the sum of the number of points affected by a PLC being off scan and the number of points that have been disabled in that month. For these reasons, CSC's CRM procedures were inadequate and require amendment regarding the monthly review process per § 195.446(e)(2). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (e) Alarm management. Each operator using a SCADA system must have a written alarm management plan to provide for effective controller response to alarms. An operator's plan must include provisions to: (1) . . . . (4) Review the alarm management plan required by this paragraph at least once each calendar year, but at intervals not exceeding 15 months, to determine the effectiveness of the plan; CSC’s CRM procedures were inadequate to define a process to review the alarm management plan at least once a calendar year, but at intervals not exceeding 15 months, to determine the 18. effectiveness of the plan, per § 195.446(e)(4). The CRM procedure section 8.10.2, “Alarm Management Plan Review,” required an alarm management plan review at the required interval and also identified “monitoring the volume of alarms” as part of that review. Monitoring the volume of alarms was further discussed in section 8.10.5,. “Controller Workload Review.” Section 8.10.5 stated that “Part of this review will also include the analysis of volume of alarms and the controller’s ability to react to those alarms efficiently. Everline utilized API 1167 to establish metrics for each Controller position to measure the frequency of alarms received per Console with “Acceptable Target Values and Maximum Target Values….” However, the process for recording these metrics versus actual performance and how this would lead to determining the effectiveness of the plan was not clarified in procedures. During the inspection it was discovered that Form 11-11, “Alarm Performance Metrics Per Controller KPI (API RP1167),” existed in the CRM procedures but was not included or referenced in a specific area of the procedures to explain how it would be used and when. CRM Procedures need to be modified to include how and when Form 11-11 will be used. Additionally, the time required to acknowledge an alarm by priority was not reviewed. Alarm management plans cannot be effective if the time for which controllers are expected to respond to alarms based on priority cannot be met. However, this was not included as part of the effectiveness review process. CSC’s CRM procedures were inadequate and require amendment to define a process to review the alarm management plan at least once a calendar year, but at intervals not exceeding 15 months, to determine the effectiveness of the plan, per § 195.446(e)(4). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (e) Alarm management. Each operator using a SCADA system must have a written alarm management plan to provide for effective controller response to alarms. An operator's plan must include provisions to: (1) . . . . (5) Monitor the content and volume of general activity being directed to and required of each controller at least once each calendar year, but at intervals not exceeding 15 months, that will assure controllers have sufficient time to analyze and react to incoming alarms; and CSC’s CRM procedures did not adequately detail a process to monitor the content and volume of general activity being directed to, and required of, each controller at least once each calendar year, but at intervals not exceeding 15 months, that will assure controllers have sufficient time to analyze and react to incoming alarms, per § 195.446(e)(5). Specifically, CSC’s CRM procedures did not define “volume of general activity for workload” sufficiently to determine that a 19. controller had sufficient time to respond to alarms. CSC’s CRM procedures require amendment to include the following available records regarding general activity or sufficient controller response: (1) the total alarm count for the year for each console by priority, as priority defines the speed of response required by the controller; (2) the total number of phone calls for the console. While phone calls were listed, phone calls were only being looked at currently through an approach that addresses an hour observation or a survey question of the controller. Actual phone logs in this third-party control room application were necessary to understand how total workload on the console impacted CSC controllers; (3) the total number of maneuvers for a console as can be determined from the event/alarm log commands or setpoint changes, commands that did not execute properly, or response to valves that were moved without advance notification which were not considered but impact the controllers from having sufficient time to respond; (4) the number of MOCs that are associated with the console which impacted the controller’s time; (5) metrics as identified by API RP 1167 and as may be completed for Form 11-11; and (6) the time required to acknowledge alarms by priority and a review of whether or not this meets the original system design and alarm rationalization process was not included but required to know if the controller has adequate time to respond to incoming alarms by console. CSC’s CRM procedures were inadequate and require amendment to detail a process to monitor the content and volume of general activity being directed to, and required of, each controller at least once each calendar year, but at intervals not exceeding 15 months, that will assure controllers have sufficient time to analyze and react to incoming alarms, per § 195.446(e)(5). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (e) Alarm management. Each operator using a SCADA system must have a written alarm management plan to provide for effective controller response to alarms. An operator's plan must include provisions to: (1) . . . . (6) Address deficiencies identified through the implementation of paragraphs (e)(1) through (e)(5) of this section. CSC’s CRM procedure section 8.10.6, “Deficiencies,” did not adequately detail a process addressing deficiencies identified through the implementation of paragraphs (e)(1) through (e)(5), per § 195.446(e)(6). Section 8.10.6 was inadequate and requires amendment because the procedure did not provide for a creation of a record that identified how and when an issue had been addressed from the monthly alarm reviews (such as chattering or nuisance alarms), or as a result of any point-to-point verification process finding incorrect alarm setpoints or descriptors. CSC’s CRM procedures were inadequate and require amendment to address deficiencies 20. 21. identified through the implementation of paragraphs (e)(1) through (e)(5), per § 195.446(e)(6). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (f) Change management. Each operator must assure that changes that could affect control room operations are coordinated with the control room personnel by performing each of the following: (1) Implement section 7 of API RP 1168 (incorporated by reference, see § 195.3) for control room management change and require coordination between control room representatives, operator's management, and associated field personnel when planning and implementing physical changes to pipeline equipment or configuration; and CSC’s CRM procedures inadequately implemented API RP 1168, section 7, for control room management change and coordination between control room representatives, operator's management, and associated field personnel when planning and implementing physical changes to pipeline equipment or configuration, per § 195.446(f). Specifically, CSC’s CRM procedures did not describe how the Caliber MOC process was coordinated with the control room and how changes were tracked to demonstrate that the control room was made aware of the changes with enough time to enact training as identified in API RP 1168, section 7.1. Additionally, CSC’s CRM procedure, section 7.1, “Management of Change,” did not clarify other types of changes, beyond that of just hydraulic impact, that require coordination with the control room. CSC’s CRM procedures were inadequate and require amendment to implement API RP 1168, section 7, regarding control room management change and coordination between control room representatives, operator's management, and associated field personnel when planning and implementing physical changes to the pipeline equipment or configuration, per § 195.446(f). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (f) Change management. Each operator must assure that changes that could affect control room operations are coordinated with the control room personnel by 22. performing each of the following: (1) . . . . (2) Require its field personnel to contact the control room when emergency conditions exist and when making field changes that affect control room operations. Caliber’s O&M procedures (including emergency procedures) and CSC’s CRM procedures did not adequately assure changes that could affect control room operations were coordinated with control room personnel by requiring field personnel to contact the control room when emergency conditions exist and when making field changes that affect control room operations, per §§ 195.446(f)(2) and 195.446(j)(1), which requires the operator to maintain records that show compliance with § 195.446. CSC’s CRM procedure section 3.7, “Field Operations,” detailed emergency notification requirements for the field operations teams. However, the control room has no field operations personnel working for Caliber; only Caliber has field operations personnel. Caliber’s O&M and emergency operations procedures did not require elements from CSC’s CRM procedure section 3.7 to be implemented for field operation personnel, did not describe who from Caliber would contact the control room, nor did it describe how this contact would be documented. Additionally, section 3.7 did not adequately detail a process for documenting (such as through the Elogger tool) contact with the control room made in advance as part of the required coordination when making field changes that affect the control room operations. Caliber’s O&M and CSC’s CRM procedures were inadequate and require amendment to address coordination by field personnel in emergency conditions and when field changes could affect control room operations (and associated records), per §§ 195.446(f)(2) and 195.446(j)(1). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (g) Operating experience. Each operator must assure that lessons learned from its operating experience are incorporated, as appropriate, into its control room management procedures by performing each of the following: (1) Review accidents that must be reported pursuant to § 195.50 and 195.52 to determine if control room actions contributed to the event and, if so, correct, where necessary, deficiencies related to: (i) Controller fatigue; (ii) Field equipment; (iii) The operation of any relief device; (iv) Procedures; (v) SCADA system configuration; and (vi) SCADA system performance. Caliber’s O&M and CSC’s CRM procedures were inadequate because they not describe how a review of a reportable accident, per §§ 195.50 and 195.52, would determine if the control room actions contributed to the events and, if so, correct, where necessary, deficiencies related to controller fatigue, field equipment, the operation of any relief device, procedures, SCADA system configuration, and SCADA system performance, per §§ 195.446(g)(1) and 195.446(j)(1), which requires the operator to maintain records that show compliance with § 195.446. Firstly, Caliber’s O&M procedures were not sufficiently integrated with CSC's CRM procedure section 9, “Accidents & Incidents,” to require data from the control room that substantiates whether or not the control room contributed to the event. Caliber O&M section 4.12.2, “Accident Investigation,” stated: “[i]t is important that plans for the investigation begin as soon as possible. In this way the information can be protected that will aid in the investigation. Some important items are: Keeping a log of significant events and of actions taken, Preserving the failed facilities or equipment for laboratory analysis as may be appropriate, Obtaining information from automation systems and other operating data that may be available.” While an automation system and other operating data would require information from the control room, what would be needed or required from the third party control room is not defined in procedures. In order to know if the control room contributed to the event, retrieving information from the third-party control room was required. Further, if the control room did contribute to the event, the procedures do not clarify how each of the elements identified in § 195.446 (g)(1) would have information exchanged (such as controller fatigue) between Caliber and the control room. Information exchanged regarding deficiencies identified, where necessary, and associated corrective measures to meet the requirements of § 195.446 (g)(1) were not adequately included in Caliber’s O&M or CSC’s CRM procedures. Secondly, Caliber’s O&M section 4.12, “Accident Review and Failure Analysis,” did not clarify that any investigation made by Caliber on a reportable release would be shared with the control room so that lessons learned, deficiencies or corrective measures can be shared or implemented. Thirdly, CSC’s CRM procedure section 9, “Accidents & Incidents Review,” did not clarify what information would be reviewed and provided to the operator to determine if the controller had properly responded or if a control room issue occurred and contributed to the event. The outcome of this review including those elements in § 195.446(g)(1) and CSC’s CRM procedures Form 11- 26, “Investigating the Possible Contribution of Fatigue to Pipeline Incidents/Accidents,” were not communicated to Caliber pursuant to delineated procedures. For these reasons, Caliber’s O&M and CSC’s CRM procedures were inadequate and require amendment to describe a review of a reportable accident to determine if the control room actions contributed to the events, and correct, where necessary, deficiencies, per §§ 195.446(g)(1) and 195.446(j)(1). 23. 24. § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (g) Operating experience. Each operator must assure that lessons learned from its operating experience are incorporated, as appropriate, into its control room management procedures by performing each of the following: (1) . . . . (2) Include lessons learned from the operator’s experience in the training program required by this section. CSC’s CRM procedures were inadequate for including lessons learned from the operator’s experience in the training program, per § 195.446(g)(2). Specifically, during the inspection, control room personnel confirmed that safety stand down reports could be shared with the control room or the console as one type of lessons learned documentation. However, CSC’s CRM procedures did not clarify how this type of information would be communicated to the controllers and documented as part of the training program. CSC’s CRM procedures were inadequate and require amendment for including lessons learned from the operator’s experience in the training program, per § 195.446(g)(2). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (h) Training. Each operator must establish a controller training program and review the training program content to identify potential improvements at least once each calendar year, but at intervals not to exceed 15 months. An operator’s program must provide for training each controller to carry out the roles and responsibilities defined by the operator. In addition, the training program must include the following elements: . . . . CSC’s CRM procedures were inadequate to establish a training program that met the requirements of §§ 195.446(h) and 195.446(j)(1). Section 195.446(h) requires operators to create a program that trains the controllers to carry out their roles and responsibilities as defined by the operator. CRM procedure, section 5.2.1, “Return from Absence,” explained individuals that have been away from the console for a time were required to review significant shift logs to understand changes that have occurred during the individual absence. However, this is 25. inadequate as the procedures did not address specifics regarding training that may be required for new procedures, operational procedure changes, or equipment changes that may be noted in the shift log. CRM procedures did not clarify who would identify the significant shift logs needed for review. At the time of the inspection, the control room cross-trained controllers on console 7 and console 8 and this impacts CSC controllers. Cross-training affects the roles and responsibilities of a controller. However, cross-training was not addressed in CSC’s CRM procedures. Further, during the inspection it was determined that a consistent alarm management process has not been implemented for all consoles in the control room. This was also not addressed in CSC’s CRM procedures. This difference would impact cross-training content. CRM procedures require amendment to include cross-training and describe what the cross-training process includes. Additionally, while training procedures did describe an On-the-Job Training (OJT) process, CSC’s CRM procedures did not define a documentation process that records (1) who performed the OJT training per topical area, and (2) when (the date) the OJT per topical area was covered with the individual in training. CSC’s CRM procedures were inadequate to establish a training program that met the requirements of §§ 195.446(h) and 195.446(j)(1). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (h) Training. Each operator must establish a controller training program and review the training program content to identify potential improvements at least once each calendar year, but at intervals not to exceed 15 months. An operator’s program must provide for training each controller to carry out the roles and responsibilities defined by the operator. In addition, the training program must include the following elements: (1) Responding to abnormal operating conditions likely to occur simultaneously or in sequence; Caliber’s O&M and CSC’s CRM procedures were inadequate to establish a controller training program on abnormal operating conditions (AOCs), per the requirements of § 195.446(h)(1). Specifically, CSC’s CRM procedures did not include any abnormal operating conditions unique to CSC that are likely to occur simultaneously or in sequence. At the time of the inspection, CSC was recovering from a massive power outage (caused by a weather event) resulting in a loss of communication for many of the points monitored and controlled from the control room. PHMSA observed the status and recovery of portions of the 26. pipeline system during the inspection. PHMSA determined from the overview screen and associated information that flow could be moving in two different directions on the CSC system on a planned or unplanned basis. PHMSA confirmed with CSC and the control room that multiple flow directions could occur at the same time. PHMSA asked which valve or valves would be used to separate the two portions of the system. The valve or valve numbers and locations that would be used to separate these two flow directions was not known by Caliber or control room personnel. Loss of communication and an unplanned reverse flow condition are abnormal operating conditions. However, these two abnormal operating conditions occurring simultaneously or in sequence was not included in training described in the CRM procedures. CSC’s CRM procedures did not define how abnormal operations likely to occur simultaneously or in sequence were determined for this control room or for CSC assets. CRM procedures and training reviewed (CRM sections 10.2, “Onboarding Training for New Controllers,” and 10.4, “New Customer Integration Training,” along with training content provided during the inspection) were not specific to CSC assets. Further, Caliber's operating procedures (specifically Caliber’s O&M, Section 3, “Abnormal Operations”) identified high temperature as an indicator for abnormal operation, yet nothing had been developed for training associated with temperature and any other abnormal operation occurring simultaneously or in sequence. Caliber’s O&M and CSC’s CRM procedures were inadequate and require amendment to establish a controller training program on abnormal operating conditions (AOCs), per the requirements of §195.446(h)(1). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (h) Training. Each operator must establish a controller training program and review the training program content to identify potential improvements at least once each calendar year, but at intervals not to exceed 15 months. An operator's program must provide for training each controller to carry out the roles and responsibilities defined by the operator. In addition, the training program must include the following elements: (1) . . . . (3) Training controllers on their responsibilities for communication under the operator's emergency response procedures; Caliber’s O&M and CSC’s CRM procedures were inadequate regarding training controllers on responsibilities for communication under the operator’s emergency response procedures, per the requirements of § 195.446(h)(3). Specifically, CSC’s CRM, “System Specific Operating 27. Procedures (SSOP),” “Gathering Pipeline Systems,” version 3, dated February 25, 2022, referred to as Exhibit S for CSC, was inadequate and needs amendment. The SSOP section called “Emergency Response” directed the controller to use the Caliber Emergency Response Procedure and Everline Leak Response Procedures. The SSOP had a list of eight steps to follow. Steps 1 and 3 stated: “1. Shutdown the system and Isolate facilities a. Notify customers to shutdown deliveries b. Notify Delivery Interconnects of Emergency, shutdown, and isolation c. Close all system valves; and …. 3. Ensure Caliber Management and Regulatory will notify all Regulatory Agencies.” However, at the time of the inspection it was not clear from these statements who was performing all of these actions and whether all system valves and interconnects could be shutdown, isolated, or operated remotely (especially in light of an excess of 98 LACTs coming into the system) by the control room. Caliber’s O&M and CSC’s CRM procedures require clarification to describe who will perform each of the requirements in Steps 1 and 3. Neither the CRM procedures SSOP (also referred to as Exhibit S) or CRM procedure Exhibit B, “Domain of Responsibility Table,” clarified whether 800 calls or 811 calls for CSC were handled by the control room. Additionally, Caliber’s O&M procedures did not clarify who was responsible for addressing 800 calls. Caliber has multiple assets, only one of which is operated by the control room, that are located in the same counties of North Dakota. When 800 calls would be received relevant to Caliber, it was not clear what or who would determine if the call was relevant only to CSC assets. Caliber did not clarify in the O&M procedures specific to CSC assets how 800-line calls would be handled, and once confirmed an 800 call was relevant to CSC, what would be communicated to the control room and how this communication would occur. During the inspection, who was receiving 800 and 811 (emergency locates) calls for CSC assets was not clear by either Caliber or the control room personnel (this does affect the required communication between Caliber and the control room during emergencies). Caliber O&M and CSC’s CRM procedures were inadequate and require amendment regarding the training of controllers on responsibilities for communication under the operator’s emergency response procedures, per the requirements of § 195.446(h)(3). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (h) Training. Each operator must establish a controller training program and review the training program content to identify potential improvements at least once each calendar year, but at intervals not to exceed 15 months. An operator's program must provide for training each controller to carry out the roles and responsibilities defined by the operator. In addition, the training program must include the following elements: (1) . . . . (4) Training that will provide a controller a working knowledge of the pipeline system, especially during the development of abnormal operating conditions; Caliber’s O&M and CSC’s CRM procedures did not adequately provide each controller training regarding the working knowledge of the pipeline system, especially during the development of abnormal operating conditions, per § 195.446(h)(4). The abnormal operations training content had not been properly integrated between Caliber’s O&M procedure section 3, “Abnormal Operations,” and the CSC CRM procedures. Caliber’s O&M procedure section 3 defined types of AOCs as, “[i]ncrease or decrease in pressure, temperature, or flow rate outside the normal operating limits.” CSC’s CRM procedures did not address all of those abnormal operations identified in Caliber’s O&M procedures section 3. This was also not addressed in the leak detection training for controllers. Additionally, CSC’s CRM procedures and training content were not in agreement. A decrease in pressure with a corresponding increase in flow upstream of the location was not found in the CRM procedures as an AOC, but was identified in one of the controller training documents and in a leak detection definition. CRM procedures need to include this AOC as it is provided as part of the controller training and would impact the working knowledge of the pipeline system. Caliber’s O&M and CSC’s CRM procedures were inadequate and require amendment to provide each controller training regarding the working knowledge of the pipeline system, especially during the development of abnormal operating conditions, per § 195.446(h)(4). 28. §195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (h) Training. Each operator must establish a controller training program and review the training program content to identify potential improvements at least once each calendar year, but at intervals not to exceed 15 months. An operator’s program must provide for training each controller to carry out the roles and responsibilities defined by the operator. In addition, the training program must include the following elements: (1) . . . . 29. (5) For pipeline operating setups that are periodically, but infrequently used, providing an opportunity for controllers to review relevant procedures in advance of their application; and CSC’s CRM procedures were inadequate to ensure the training program included an opportunity for controller review of procedures for pipeline operating setups that are periodically, but infrequently used, in advance of their application, per § 195.446(h)(5). Specifically, CSC’s CRM procedure section 10, “Annual Training of Existing Controllers,” stated “Pipeline Operating Set- Ups that are Infrequently, but periodically used providing the controller an opportunity to review relevant procedures in advance of the application.” Pipeline operating set-ups that are periodically but infrequently used can be required at any time and this would not always align with annual training activities. CRM procedures did not provide a clear process on how an opportunity to review the relevant procedures in advance of their application at any time, not just during annual training would be implemented. Additionally, during the inspection, when asked to identify pipeline operating setups that are periodically but infrequently used, both Caliber and control room personnel indicated that there were none. PHMSA asked if smart pigs would be run on the pipeline periodically and Caliber answered that this could occur. Similarly, PHMSA asked if the pipeline was frequently shutdown entirely such as was experienced during the inspection due to the severe weather event. It was explained that the pipeline would seldom be shutdown entirely. As a result of these discussions, it was clear that Caliber and the control room had not adequately identified procedures for training that could be periodically or infrequently used. While some of these periodically but infrequently used procedures had been implemented, whether or not controllers had an opportunity to review the relevant procedures in advance of their application was not clear. CSC’s CRM procedures require modification to adequately identify procedures that would be periodically or infrequently used and provide a clear process on how this would be implemented to assure controllers have an opportunity to review the relevant procedures in advance of their application, per § 195.446(h)(5). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (h) Training. Each operator must establish a controller training program and review the training program content to identify potential improvements at least once each calendar year, but at intervals not to exceed 15 months. An operator’s program must provide for training each controller to carry out the roles and responsibilities defined by the operator. In addition, the training program must include the following elements: (1) . . . . 30. (6) Control room team training and exercises that include both controllers and other individuals, defined by the operator, who would reasonably be expected to operationally collaborate with controllers (control room personnel) during normal, abnormal or emergency situations. Operators must comply with the team training requirements under this paragraph no later than January 23, 2018. CSC’s CRM procedures did not clearly describe when team training would occur, who should have the training, or adequately define those who would be expected to operationally collaborate with controllers during normal, abnormal, or emergency situations, per the requirements of § 195.446(h)(6). During the inspection, Caliber and the control room personnel indicated that a team training event occurred every other year and each controller was expected to be present. However, CSC’s CRM procedure, section 10.3.1, “Team Training,” indicated that this was annual training. Further in this same procedure section, Item 2, “Frequency of Training,” training was identified as having occurred once every two years. CRM procedures require clarification regarding the frequency of team training. CSC’s CRM procedure section 10.3.1 did not include other departments or divisions besides field operations that operationally collaborate with the control room (such as engineering, hydraulics, SCADA/IT/Network Operations, commercial & scheduling, compliance, leak detection, measurement personnel, or others as maybe applicable). CSC’s CRM procedures were inadequate and require amendment to clearly describe team training, per the requirements of § 195.446(h)(6). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (i) Compliance validation. Upon request, operators must submit their procedures to PHMSA or, in the case of an intrastate pipeline facility regulated by a State, to the appropriate State agency. Caliber’s O&M procedures were inadequate to implement proper compliance validation, per §§ 195.446(i) and 195.446(j)(1), which require the operator to maintain records that show compliance with § 195.446. Specifically, Caliber’s O&M procedures did not include the requirement to submit procedures to PHMSA, or other appropriate State agency, upon request. Caliber’s O&M procedures also need to define the process of submitting a request to the control room for procedures and include what record will be used to demonstrate compliance with this requirement. Section 2.3 of CSC’s CRM procedure described this requirement; however Caliber’s O&M had not been properly integrated and did not adequately address it. 31. 32. Caliber’s O&M procedures were inadequate and require amendment to address compliance validation and identify associated records, per § 195.446(i) and 195.446(j)(1). § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (b) . . . . (j) Compliance and deviations. An operator must maintain for review during inspection: (1) Records that demonstrate compliance with the requirements of this section;…. CSC’s CRM procedures were inadequate to properly define records that will be retained to demonstrate compliance, per the requirements of § 195.446(j)(1). Specifically, CSC’s CRM procedure section 2.4, “CRM Record Keeping Documentation,” did not include detail on what records were to be retained. This would also include records required by CRM procedures called Exhibits. Records that demonstrate compliance have substantiation data that is also retained, and this was not adequately addressed in procedures. Additionally, the records generated following the CRM annual review (Form 11-4a) detailed in CSC’s CRM procedure section 2.1 did not adequately address that documents reviewed were to be maintained and the CRM procedures did not address discharge pressure records required by § 195.404(b)(1). CSC’s CRM procedures were inadequate and require amendment to adequately address records to demonstrate compliance, per the requirements of § 195.446(j)(1). § 195.452 Pipeline integrity management in high consequence areas. (a) Which pipelines are covered by this section? This section applies to each hazardous liquid pipeline and carbon dioxide pipeline that could affect a high consequence area, including any pipeline located in a high consequence area unless the operator effectively demonstrates by risk assessment that the pipeline could not affect the area. . . . (b) . . . . (i) What preventive and mitigative measures must an operator take to protect the high consequence area? (1) . . . . (3) Leak detection. An operator must have a means to detect leaks on its pipeline system. An operator must evaluate the capability of its leak detection means and modify, as necessary, to protect the high consequence area. An operator’s evaluation must, at least, consider, the following factors—length and size of the pipeline, type of product carried, the pipeline’s proximity to the high consequence area, the swiftness of leak detection, location of nearest response personnel, leak history, and risk assessment results. Caliber’s O&M, CSC’s CRM procedures, and the control room Line Balance-Leak Response procedures, were inadequate to evaluate the capability of its leak detection means and consider the swiftness of leak detection as required by § 195.452(i)(3). The CSC system has 4.46 miles of pipeline located in “Could Affect HCA” areas according to its annual report relevant to 2022. The procedure entitled “Everline Control Center, Caliber Spring Creek Pipeline System, Line Balance-Leak Response” (Line Balance – Leak Response procedure) did not adequately define how the leak detection capabilities, including swiftness, would be evaluated. Caliber’s O&M procedures did not describe how information on swiftness of leak detection and capability evaluations (flowing and non-flowing conditions) would be completed between Caliber and the control room. Nothing provided during the inspection indicated that this had been adequately integrated with the Caliber’s O&M or Caliber’s integrity management procedures. Additionally, during the inspection it was determined that a daily imbalance is calculated manually by Caliber for the CSC system and sent to the control room. While this information could be used as part of evaluation of leak detection capability, nothing in CSC’s CRM procedures, the Line Balance – Leak Response procedure, or Caliber’s O&M addressed this process, or how it could be used to assist with determining a leak. Caliber’s O&M, CSC’s CRM, and the Line Balance- Leak Response procedures require amendment to identify which types of changes would result in a revision to the leak detection imbalance process and impact the evaluation of leak detection capability and swiftness. For these reasons, Caliber’s O&M, CSC’s CRM, and the Line Balance-Leak Response procedures require amendment to meet the requirement of § 195.452(i)(3). Response to this Notice This Notice is provided pursuant to 49 U.S.C. § 60108(a) and 49 C.F.R. § 190.206. Enclosed as part of this Notice is a document entitled Response Options for Pipeline Operators in Enforcement Proceedings. Please refer to this document and note the response options. Be advised that all material you submit in response to this enforcement action is subject to being made publicly available. If you believe that any portion of your responsive material qualifies for confidential treatment under 5 U.S.C. § 552(b), along with the complete original document you must provide a second copy of the document with the portions you believe qualify for confidential treatment redacted and an explanation of why you believe the redacted information qualifies for confidential treatment under 5 U.S.C. § 552(b). Following the receipt of this Notice, you have 30 days to submit written comments, revised procedures, or a request for a hearing under § 190.211. If you do not respond within 30 days of receipt of this Notice, this constitutes a waiver of your right to contest the allegations in this Notice and authorizes the Associate Administrator for Pipeline Safety to find facts as alleged in this Notice without further notice to you and to issue an Order Directing Amendment. If your plans or procedures are found inadequate as alleged in this Notice, you may be ordered to amend your plans or procedures to correct the inadequacies (49 C.F.R. § 190.206). If you are not contesting this Notice, we propose that you submit your amended procedures to my office within 90 days of receipt of this Notice. This period may be extended by written request for good cause. Once the inadequacies identified herein have been addressed in your amended procedures, this enforcement action will be closed. It is requested that Caliber maintain documentation of the safety improvement costs associated with fulfilling this Notice of Amendment (preparation/revision of plans, procedures) and submit the total to Gregory A. Ochs, Director, Central, Pipeline and Hazardous Materials Safety Administration. In correspondence concerning this matter, please refer to 3-2024-017-NOA and, for each document you submit, please provide a copy in electronic format whenever possible. Sincerely, Gregory A. Ochs Director, Central Region, Office of Pipeline Safety Pipeline and Hazardous Materials Safety Administration Enclosure: Response Options for Pipeline Operators in Enforcement Proceedings cc: Daniel Werth, Sr. Executive, Caliber (dwerth@calibermidstream.com) Charles Maybee, DOT Compliance Primary, Caliber (cmaybee@calibermidstream.com) ## Provenance - Official: Yes - Source: - Source ID: `phmsa-enforcement` - SHA-256: `61098eb26a76fb97c970fca3a5a715d3580608513d5caf8b04744240aabe4e2a` - Retrieved: 2026-08-20T04:44:44.458Z - Exported: 2026-08-22T13:21:18.171Z - Document slug: `phmsa-enforcement-32024017noa` ### Source metadata ```json { "cpf": "32024017NOA", "operator": "CALIBER SPRING CREEK LLC", "region": "Central", "pipelineType": "INTRASTATE LIQUID", "caseStatus": "CLOSED", "citedSections": [ "195.402(a)", "195.408(b)(3)", "195.446(a)", "195.446(b)(1)", "195.446(b)(2)", "195.446(b)(3)", "195.446(b)(4)", "195.446(b)(5)", "195.446(c)(1)", "195.446(c)(2)", "195.446(c)(3)", "195.446(c)(4)", "195.446(d)", "195.446(d)(1)", "195.446(d)(4)", "195.446(e)(1)", "195.446(e)(2)", "195.446(e)(4)", "195.446(e)(5)", "195.446(e)(6)", "195.446(f)(1)", "195.446(f)(2)", "195.446(g)(1)", "195.446(g)(2)", "195.446(h)", "195.446(h)(1)", "195.446(h)(3)", "195.446(h)(4)", "195.446(h)(5)", "195.446(h)(6)", "195.446(i)", "195.446(j)(1)", "195.452(a)", "195.452(i)(3)" ], "dataAsOf": "08/04/2026 12PM", "caseDataAsOf": "2026-08-04", "attachmentCount": 5, "attachments": [ { "name": "32024017NOA_Closure Letter_05212026_(22-236545).pdf", "url": "https://primis.phmsa.dot.gov/enforcement-documents/32024017NOA/32024017NOA_Closure%20Letter_05212026_(22-236545).pdf", "bytes": 194347, "category": "agency_document" }, { "name": "32024017NOA_Closure Letter_05212026_(22-236545)_text.pdf", "url": "https://primis.phmsa.dot.gov/enforcement-documents/32024017NOA/32024017NOA_Closure%20Letter_05212026_(22-236545)_text.pdf", "bytes": 182935, "category": "agency_document" }, { "name": "32024017NOA_Notice of Amendment_07032024_(22-236545).pdf", "url": "https://primis.phmsa.dot.gov/enforcement-documents/32024017NOA/32024017NOA_Notice%20of%20Amendment_07032024_(22-236545).pdf", "bytes": 452907, "category": "agency_document" }, { "name": "32024017NOA_Notice of Amendment_07032024_(22-236545)_text.pdf", "url": "https://primis.phmsa.dot.gov/enforcement-documents/32024017NOA/32024017NOA_Notice%20of%20Amendment_07032024_(22-236545)_text.pdf", "bytes": 390665, "category": "agency_document" }, { "name": "32024017NOA_Operator Response to Notice and Request Time Extension_07152024_(22-236545).pdf", "url": "https://primis.phmsa.dot.gov/enforcement-documents/32024017NOA/32024017NOA_Operator%20Response%20to%20Notice%20and%20Request%20Time%20Extension_07152024_(22-236545).pdf", "bytes": 160300, "category": "party_submission" } ], "extractedAgencyDocumentCount": 2, "attachmentPolicy": "Official attachment links are retained. Agency-issued documents may also include a verified local PDF and page-level text representation.", "jurisdiction": "US", "operatorName": "CALIBER SPRING CREEK LLC" } ```