CPF 32025010NOA
CPF 32025010NOA
party submissionOfficial PDF32025010NOA_Operator Response to Notice_01152025_(24-297197).pdf#
32025010NOA_Notice of Amendment_01142025_(24-297197)_text.pdf, page 1Official PDFNOTICE OF AMENDMENT VIA ELECTRONIC MAIL TO: hfaulkner@lambdaoil.com; jmcgrath@lambdaenergyllc.com; bberthelot@lambdaenergyllc.com January 14, 2024 Mr. Harry Faulkner President and CEO Lambda Energy Gathering, LLC 12012 Wickchester Lane, Suite 300 Houston, TX 77079 CPF 3-2025-010-NOA Dear Mr. Faulkner: From June 11 to July 25, 2024, of the on-site inspection, a representative of the Pipeline and Hazardous Materials Safety Administration (PHMSA), pursuant to Chapter 601 of 49 United States Code (U.S.C.), inspected Lambda Energy Gathering, LLC’s (Lambda) procedures for Control Room Management (CRM) in Kalkaska, Michigan. As a result of the inspection, PHMSA has identified the apparent inadequacies found within Lambda’s plans or procedures. The items inspected and the inadequacies are described below: 1. § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (c) Provide adequate information. Each operator must provide its controllers with the information, tools, processes and procedures necessary for the controllers to carry out the roles and responsibilities the operator has defined by performing each of the following: (1) . . . . (4) Test any backup SCADA systems at least once each calendar year, but at intervals not to exceed 15 months.#
32025010NOA_Notice of Amendment_01142025_(24-297197)_text.pdf, page 2Lambda’s Control Room Management Program, (version 1.0, effective April 18, 2022) (CRM Plan), section C, subsection 8.4, was not adequate to describe Lambda’s process to test its backup SCADA system and document the test to demonstrate compliance with § 195.446(c)(4). While Lambda did not have a designated back up control room, it did have a backup SCADA system. Lambda has only conducted a backup server test, called a “server swap,” once every six months. The test entailed cold starting1 up the backup secondary server and shutting down the current primary server. The control center operated on the backup secondary server for 6 months. This was a back-and-forth process every six months. Lambda’s documentation of the swap consisted of writing on a chalk board the date of the “swap” and which server was the current primary for operations. This documentation via chalk board is obviously not sufficient to meet [the documentation requirements of § 195.446(j)(1) because of its temporary nature.]. Lambda had a very detailed procedure, outside the CRM plan, developed by its predecessor company, Merit Energy, entitled “Server Swap Operations Procedures Version 11.28.2018.” This process was not described nor cross-referenced in its CRM plan, and it failed to include a documentation requirement. Lambda’s CRM Plan did not include a process requiring the documentation and verification of the functionality of the “swapped server,” once in service. This process should include a form or checklist to verify critical functions were working. CRM Plan section 8.4 contained statements prescribing the elements of the SCADA operation that should be tested and operations verified after the server swap. The procedure designated the, “[a]larm and event logs from the backup SCADA system to help demonstrate adequate functioning during back up operations.” While an alarm-event log can be adequate to document functionality of the server, it is not adequate to demonstrate compliance. This is because Lambda’s alarm-event log did not include information such as: (1) who conducted the test, (2) who verified the information, (3) which server was engaged during the test, and (4) whether there were any failures, if so, what were the follow up actions to correct. A checklist or form including this information would support constancy and provide instruction to the relevant controller for test expectations. If Lambda decides to stand up a backup control room, the required documentation would need to be expanded to include: (1) verification that monitors work, (2) whether IT systems are functional, (3) whether phones are working and transfer, (4) whether printers are working (if applicable), etc. The CRM Plan did provide guidance for manual control and monitoring of the system while the SCADA system is being swapped. However, the procedure did not reference Lambda’s Internal Communication Plan. During the inspection, Lambda indicated that this plan was employed during the swap, and therefore it should be cross-referenced in the CRM Plan. Additionally, during the COVID-19 pandemic, Lambda created a second control room, in the plant facility, to support social distancing. While post COVID-19 the control room is no longer in use, it can still function, and therefore it should be clarified whether Lambda considered this control room as its designated backup control room. If so, then this control room needs to be included in Lambda’s CRM Plan and be tested once each calendar year not to exceed 15 months, in addition to, or in conjunction with the “server swap.” 1 Cold starts occur when an operator completely shuts down the server and then re-boots the system.#
32025010NOA_Notice of Amendment_01142025_(24-297197)_text.pdf, page 3The CRM Plan needs to be amended to include: (1) reference to the detailed swap plan, (2) a formalized swap plan within the Lambda Procedure Library, (3) a checklist or form used for documenting tests and for ensuring the consistency of testing, (4) a plan to test the back up control room, if the decision is made to stand that up as a backup control center, and (5) cross reference to the Internal Communication Plan for manual operation and monitoring during the server swap. 2. § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (e) Alarm management. Each operator using a SCADA system must have a written alarm management plan to provide for effective controller response to alarms. An operator's plan must include provisions to: (1) Review SCADA safety-related alarm operations using a process that ensures alarms are accurate and support safe pipeline operations. Lambda’s CRM Plan, section E, subsection 3.3, was not adequate to describe the current practice in place for identifying and correcting inaccurate or malfunctioning alarms, per the requirements of § 195.446(e)(1). Specifically, Lambda employed a ticket process to document, assign, and dispatch work orders to the field when inaccurate or malfunctioning alarms were identified. Subsection 3.3 required controllers to “identify, report, and correct inaccurate or malfunctioning alarms based on alarm priority and safety related status to maintain the safe operation of the pipeline. The controller will make notification of any identified alarm points to the control room manager for corrective action.” It also stated, “reporting and the correction of inaccurate or malfunctioning alarms will be documented using Form CRM-26B, “SCADA Failure Review of Crude Oil Pipeline.” Form CRM-26B is different than the tickets used for documenting and reporting. Lambda’s ticket process was effective for communicating issues and assigning work to field personnel to correct the condition. However, Lambda’s procedures did not include the management control aspect of the practice. The ticket process did not include a tracking system to ensure that the work was completed, beyond the second copy of the ticket maintained in the control room. The ticket process did not require Lambda’s personnel to log the tickets, follow progress, or match completed work tickets to the log. While Lambda was able to make corrections through this process, it did not have any way of tracking what work was not completed and how long it has been dispatched. Additionally, Lambda did not have any established criteria to prioritize repair. Lambda’s CRM Plan needs to be amended to provide a description of the process, a method of logging and tracking work tickets and criteria for prioritizing work, criteria for completion, and follow up for review and escalation.#
32025010NOA_Notice of Amendment_01142025_(24-297197)_text.pdf, page 43. § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (e) Alarm management. Each operator using a SCADA system must have a written alarm management plan to provide for effective controller response to alarms. An operator's plan must include provisions to: (1) . . . . (3) Verify the correct safety-related alarm set-point values and alarm descriptions when associated field instruments are calibrated or changed and at least once each calendar year, but at intervals not to exceed 15 months. Lambda's CRM Plan, section E, subsection 3.3, was not adequate to provide a process to verify the correct safety-related alarm set-point values and alarm descriptions when associated field instruments are calibrated or changed and at least once each calendar year, but at intervals not to exceed 15 months, per the requirements of § 195.446(e)(3). Specifically, the CRM Plan only paraphrased the regulatory requirements and directed the use of Form CRM-16, “Annual Safety Related Alarm Review,” to support and document the review. While Form CRM-16 was adequate for the annual set point and description verification, it did not require signature of who completed the review and who accepted the results of the review, which is necessary to demonstrate compliance. Form CRM-16 also did not include space for comments in the event the review produced findings that the set points and descriptions were not adequate and associated follow up for correction and reverification. Additionally, the CRM Plan did not include instructions for how the form was to be completed, who was to complete the form, how to document and correct deficiencies (errors between the master database and SCADA values), and reverification. The CRM Plan also did not include a requirement to verify safety related set points when field instruments are calibrated or changed. Lambda has a form entitled “Instrument Calibration Report,” that was used to verify field set points and descriptions that is well suited for this requirement. The CRM Plan needs to be amended to include the following elements: (1) documentation of comments when a deficiency is found; (2) documentation of the name and signature of who completed the review; (3) documentation of the date the review was finalized; and (4) instruction on how to complete Form CRM-16, including (a) the requirement to verify the correct safety- related alarm set-point values and alarm descriptions when associated field instruments are calibrated or changed, and (b) reference to the current process and forms. Form CRM-16 also needs to be amended to reflect all requirements specified in the procedure. For both the CRM Plan and Form CRM-16, Lambda must include a process detailing required actions when deficiencies are identified, such as required documentation of the identified deficiency, required investigations to determine the correct information, and required documentation of the correction.#
32025010NOA_Notice of Amendment_01142025_(24-297197)_text.pdf, page 54. § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section… (e) Alarm management. Each operator using a SCADA system must have a written alarm management plan to provide for effective controller response to alarms. An operator's plan must include provisions to: (1) . . . . (4) Monitor the content and volume of general activity being directed to and required of each controller at least once each calendar year, but at intervals not exceeding 15 months, that will assure controllers have sufficient time to analyze and react to incoming alarms; and Lambda’s CRM Plan, section E, subsection 3.3, “Safety Reviews,” was not adequate to provide a process that demonstrated compliance to monitor the content and volume of general activity being directed to and required of each controller that will assure controllers have sufficient time to analyze and react to incoming alarms, per the requirements of § 195.446(e)(4). Specifically, the CRM Plan simply paraphrased the regulatory requirements and directed controllers to use Form CRM-18, “Alarm Content and Volume Review,” to support and document the review. The CRM Plan did not contain any instructions on how to complete the review utilizing the form. The CRM Plan did not include detail on who was responsible to initiate, conduct, and analyze the workload review. Additionally, the CRM Plan also did not include the criteria for determining controllers have sufficient time to analyze and react to incoming alarms. Form CRM-18 provided documentation spaces for a variety of time periods for activity review (day of the week, time of day, season, etc.) without any direction on how these should be evaluated; for example, each console, every year, a different activity each year. While Form CRM-18 did have a table to gather data, it was a table to present summarized data, rather than raw data. Also, it was unclear if the data was collected from Lambda’s data bases, such as SCADA, or from phone records, or if controllers were estimating their time spent on an activity. Form CRM-18 also did not require the documentation of the identify the console related to the data. The CRM Plan must be amended to specify the time frame an activity review will be conducted and how often for each console. This should also include conditions outside of this time frequency that may be related to acquisitions or divestitures that may support adding or reducing consoles. The CRM Plan must also be amended to include detail on how data will be gathered, such as from SCADA data, phone records, or controller document task frequency. The CRM Plan must include detail on who will initiate the review, provide the data, analyze the data, and determine if controllers have sufficient time to analyze and react to incoming alarms using the established criteria in the procedure. If the review identifies deficiencies, the CRM Plan needs to provide instruction on how those deficiencies will be documented and require an action plan developed and implemented to correct.#
32025010NOA_Notice of Amendment_01142025_(24-297197)_text.pdf, page 65. § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (f) Change management. Each operator must assure that changes that could affect control room operations are coordinated with the control room personnel by performing each of the following: (1) Implement section 7 of API RP 1168 (incorporated by reference, see §195.3) for control room management change and require coordination between control room representatives, operator's management, and associated field personnel when planning and implementing physical changes to pipeline equipment or configuration. Lambda’s CRM Plan was not adequate to demonstrate compliance with § 195.446(f)(1). CRM Plan, section F, subsection 3.1, stated, “Lambda Energy will ensure that changes that could affect control room operations are coordinated with control room personnel.” However, the CRM Plan did not include detail on how this change and coordination would be accomplished. The CRM Plan lacked a mechanism to notify the control room of changes, so the control room representative could review the changes, provide appropriate feedback, represent the control room’s perspective and needs, initiate internal control room management of changes (MOC), update procedures, and training. Lambda utilized a Process Safety Management (PSM) system that had a form—entitled “Management of Change Authorization” (MOCA Form)—that was used exclusively throughout the plant and pipeline for documenting changes. The MOCA Form was quite thorough in its considerations. However, the CRM Plan, section F, did not cross-reference the MOCA Form nor did it require its use for documenting control room management change. Instead, CRM Plan, section F, subsection 3.4 identified Form CRM-11, “Management of Change Record,” to be used to record changes in control room management. It appeared this form should have been used for all changes, but during the inspection Lambda indicated that controllers, in practice used the PSM system’s MOCA Form to process changes, not Form CRM-11. If Form CRM-11 was intended to be used for smaller control room changes, it was not adequate because it did not identify all the parties or systems impacted by the change and verification that stake holders were made aware of the change and approved the change. The form was very generic and lacked sufficient detail. Subsection 3.4 also indicated that Form CRM-11 should be used to record temporary changes. Subsection 3.4 stated “[a] time restriction must be given in the ‘Summary of Change’ section.” However, Form CRM-11 did not address whether the change was temporary or permanent. Additionally, Form CRM-11 only required the time restriction be recorded in the “Summary of Change” section and did not have a section to call out the specific date the change occurred on, for tracking purposes. Subsection 4.2 stated, “control room personnel must establish a time limit for temporary changes and monitor them closely.” The CRM Plan did not contain a mechanism#
32025010NOA_Notice of Amendment_01142025_(24-297197)_text.pdf, page 7for tracking these time limit dates or who was responsible for that task. Form CRM-11 was also confusing due to the various dates provided. It was unclear if the “Completed/Reviewed/Approved By” section was intended to indicate the dates the MOC was initiated, when it was reviewed, or when it was completed. The CRM Plan must be amended to include (1) reference to the PSM procedure and identify who will be the control room representative to review the changes, (2) a requirement to provide appropriate feedback, (3) a requirement to document the control room’s perspective and needs, and (4) a requirement to initiate internal control room management of changes (MOC), update procedures, and training. The CRM Plan also needs to further define whether Form CRM-11 should be used for smaller control room MOCs such as alarm, SCADA, or procedure changes. Lambda must also revise Form CRM-11 to better accommodate the procedure requirement as well as address tracking changes for temporary MOCs and a method to track all MOCs. 6. § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (f) Change management. Each operator must assure that changes that could affect control room operations are coordinated with the control room personnel by performing each of the following: (1) . . . . (2) Require its field personnel to contact the control room when emergency conditions exist and when making field changes that affect control room operations. Lambda’s field Standard Operating Procedures (SOP) were not adequate to demonstrate compliance with § 195.446(f)(2), which requires field personnel to contact the control room when emergency conditions exist and when making field changes that affect control room operations, and § 195.402(c)(15) which requires the implementation of the applicable control room management procedures required by § 195.446. Specifically, Lambda’s valve inspection SOP 1052022 and Tank Float Inspection SOP did not require field personnel to contact the control room when making field changes that affect control room operations. Additionally, the Tank Float Inspection SOP was maintained separately and outside of the formalized and approved Operations and Maintenance (O&M) Manual and SOP library. The field SOPs and procedures must be amended to include a step to contact the control room when making field changes that affect control room operations which includes, but not limited to maintenance, calibration, inspection and modification activities. Lambda also needs to amend its procedures to require the Tank Float Inspection SOP be incorporated into its O&M Manual and SOP library.#
32025010NOA_Notice of Amendment_01142025_(24-297197)_text.pdf, page 87. § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (g) Operating experience. Each operator must assure that lessons learned from its operating experience are incorporated, as appropriate, into its control room management procedures by performing each of the following: (1) Review accidents that must be reported pursuant to §§ 195.50 and 195.52 to determine if control room actions contributed to the event and, if so, correct, where necessary, deficiencies related to: (i) Controller fatigue. Lambda’s CRM Plan, section D, subsection 6.4, was not adequate to describe its practice for reviewing accidents that must be reported pursuant to §§ 195.50 and 195.52 to determine if control room actions contributed to the event and, if so, correct, where necessary, deficiencies related to controller fatigue, per the requirements of § 195.446(g)(1)(i). Specifically, Lambda’s CRM Plan, which referenced a Lambda Form 130 “Incident Investigation,” to be utilized in determining if fatigue contributed to an accident, did not include the “[q]uantitative controller fatigue information to be collected,” as stated in the procedure. In practice, the control room used Form CRM – 20, “Event Cause Analysis,” to document controller shift and sleep hours to support determination of whether fatigue contributed to the event. However, Form CRM – 20 was not referenced in either CRM Plan section D, subsection 6.4, or in Form 130. While Form CRM – 20 captured information related to shift pattern, shifts worked, hours of sleep etc., it did not require documentation of the analysis/process used to determine, from the collected quantitative controller fatigue information, whether fatigue contributed to an event. Lambda’s CRM Plan must be amended to include a detailed process to evaluate the controller’s level of fatigue and whether it contributed to the event and requiring completion of a form to document this process. The process should include detail on the data required to perform the analysis, the analysis process, and criteria for determining both controller fatigue and whether fatigue contributed to the event. This process should also include clarity on which form will be used for the analysis, either Form CRM-20, Form 130, or a combination of both and providing, in procedures, the appropriate form reference. 8. § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. . . . (h) Training. Each operator must establish a controller training program and review#
32025010NOA_Notice of Amendment_01142025_(24-297197)_text.pdf, page 9the training program content to identify potential improvements at least once each calendar year, but at intervals not to exceed 15 months. An operator's program must provide for training each controller to carry out the roles and responsibilities defined by the operator. In addition, the training program must include the following elements: (1) . . . . (6) Control room team training and exercises that include both controllers and other individuals, defined by the operator, who would reasonably be expected to operationally collaborate with controllers (control room personnel) during normal, abnormal or emergency situations. Operators must comply with the team training requirements under this paragraph no later than January 23, 2018. CRM Plan, section H, subsection 5.8, “Team Training,” was not adequate to demonstrate compliance with § 195.446(h)(6). Specifically, the CRM Plan did not require training to include the three different operational modes of normal, abnormal and emergency conditions. The CRM Plan did not provide detail on the content to be considered for Team Training, nor did it include a requirement for soft skills to be included in the training. Additionally, while the procedure indicated team members that “[o]perationally collaborate with the control room have been defined,” there was no list of which job titles had been considered to operationally collaborate with the control room, or a reference of where to find the list, nor was there a description of job roles that Lambda considered to be operationally collaborating with the control room. The CRM Plan must be amended to include training on the three different operational modes of normal, abnormal and emergency conditions, and detail content in the Team Training on soft skills. Also, the CRM Plan must address those, by title or job group, who can be expected operationally collaborate with the control room to be included in Team Training, either through a list in the procedure or reference to where the information can be found. Response to this Notice This Notice is provided pursuant to 49 U.S.C. § 60108(a) and 49 C.F.R. § 190.206. Enclosed as part of this Notice is a document entitled Response Options for Pipeline Operators in Enforcement Proceedings. Please refer to this document and note the response options. Be advised that all material you submit in response to this enforcement action is subject to being made publicly available. If you believe that any portion of your responsive material qualifies for confidential treatment under 5 U.S.C. § 552(b), along with the complete original document you must provide a second copy of the document with the portions you believe qualify for confidential treatment redacted and an explanation of why you believe the redacted information qualifies for confidential treatment under 5 U.S.C. § 552(b). Following the receipt of this Notice, you have 30 days to submit written comments, revised procedures, or a request for a hearing under § 190.211. If you do not respond within 30 days of receipt of this Notice, this constitutes a waiver of your right to contest the allegations in this#
32025010NOA_Notice of Amendment_01142025_(24-297197)_text.pdf, page 10Notice and authorizes the Associate Administrator for Pipeline Safety to find facts as alleged in this Notice without further notice to you and to issue an Order Directing Amendment. If your plans or procedures are found inadequate as alleged in this Notice, you may be ordered to amend your plans or procedures to correct the inadequacies (49 C.F.R. § 190.206). If you are not contesting this Notice, we propose that you submit your amended procedures to my office within 30 days of receipt of this Notice. This period may be extended by written request for good cause. Once the inadequacies identified herein have been addressed in your amended procedures, this enforcement action will be closed. It is requested (not mandated) that Lambda maintain documentation of the safety improvement costs associated with fulfilling this Notice of Amendment (preparation/revision of plans, procedures) and submit the total to Gregory A. Ochs, Director, Central Region, Pipeline and Hazardous Materials Safety Administration. In correspondence concerning this matter, please refer to CPF 3-2024-079-NOA and, for each document you submit, please provide a copy in electronic format whenever possible. Sincerely, Gregory A. Ochs Director, Central Region, Office of Pipeline Safety Pipeline and Hazardous Materials Safety Administration cc: Ben Berthelot, Regulatory Compliance Manager, Lambda, bberthelot@lambdaenergyllc.com James McGrath, Michigan Plant and Pipeline Manager, Lambda, jmcgrath@lamdaenergyllc.com Enclosure: Response Options for Pipeline Operators in Enforcement Proceedings#
32025010NOA_Closure Letter_11182025_(24-297197)_text.pdf, page 1Official PDFU.S. Department of Transportation Pipeline and Hazardous Materials Safety 901 Locust Street, Suite 480 Kansas City, MO 64106 VIA ELECTRONIC MAIL TO: hfaulkner@lambdaoil.com; jmcgrath@lambdaenergyllc.com; bberthelot@lambdaenergyllc.com November 18, 2025 Mr. Harry Faulkner President and CEO Lambda Energy Gathering, LLC 12012 Wickchester Lane, Suite 300 Houston, TX 77079 RE: CPF 3-2025-010-NOA Dear Mr. Faulkner: From June 11 to July 25, 2024, a representative of the Pipeline and Hazardous Materials Safety Administration (PHMSA), pursuant to Chapter 601 of 49 United States Code (U.S.C.), conducted an inspection of Lambda Energy Gathering, LLC’s (Lambda) procedures for Control Room Management (CRM) in Kalkaska, Michigan. On January 14, 2025, pursuant to 49 CFR § 190.206, PHMSA issued a Notice of Amendment which proposed amendment of Lambda’s procedures. Lambda submitted its amended procedures, with its final submission on July 2, 2025. PHMSA has reviewed the amended procedures, and it appears that the inadequacies outlined in the Notice of Amendment have been corrected. This letter is to inform you that no further action is necessary, and this case is now closed. Thank you for your cooperation. Sincerely, David Barrett Acting Director, Central Region, Office of Pipeline Safety Pipeline and Hazardous Materials Safety Administration cc: Ben Berthelot, Regulatory Compliance Manager, Lambda, bberthelot@lambdaenergyllc.com James McGrath, Michigan Plant and Pipeline Manager, Lambda, jmcgrath@lamdaenergyllc.com#
This material provides agency context. It does not replace binding regulatory text, and its legal effect depends on the underlying authority and facts.