SUNOCO PIPELINE L.P. — Notice of Probable Violation

party submissionOfficial PDF120135021__Operator_Response_and_Request_for_Hearing_10312013.pdf#

120135021_NOPV PCP PCO_09302013_text.pdf, page 1Official PDFU.S. Department Of Transportation Pipeline and 609.989.2171 Hazardous Materials Safety Administration 820 Bear Tavern Road, Suite 103 West Trenton, NJ 08628 NOTICE OF PROBABLE VIOLATION PROPOSED CIVIL PENALTY and PROPOSED COMPLIANCE ORDER OVERNIGHT EXPRESS DELIVERY September 30, 2013 David Chalson Vice President, Operations Sunoco Pipeline L.P. 4041 Market Street Aston, PA 19014 CPF 1-2013-5021 Dear Mr. Chalson: From December 7 - 9, 2011, representatives of the Pipeline and Hazardous Materials Safety Administration (PHMSA), Office of Pipeline Safety (OPS), pursuant to Chapter 601 of 49 United States Code inspected Sunoco Pipeline L.P.’s (Sunoco) control room management procedures titled Sunoco Pipeline L.P. & Affiliates Control Room Management Manual, in Montello, Pennsylvania.1 These procedures apply to Operator Identification (OPID) 18718 Sunoco Pipeline LP – Eastern area and Magtex System and OPID 07063 Harbor Pipeline. As a result of the inspection, it appears that you have committed a probable violation of the Pipeline Safety Regulations, Title 49, Code of Federal Regulations. The item inspected and the probable violation is: 1 The deadlines for pipeline operators to implement certain control room management procedures are prescribed in§195.446(a). At the time of this inspection, pipeline operators had to have implemented most procedures. Implementation of all related requirements was due no later than August 1, 2012.#

120135021_NOPV PCP PCO_09302013_text.pdf, page 2CPF 1-2013-5021 1. §195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. The procedures required by this section must be integrated, as appropriate, with the operator's written procedures required by § 195.402. An operator must develop the procedures no later than August 1, 2011, and must implement the procedures according to the following schedule. The procedures required by paragraphs (b), (c)(5), (d)(2) and (d)(3), (f) and (g) of this section must be implemented no later than October 1, 2011. The procedures required by paragraphs (c)(1) through (4), (d)(1), (d)(4), and (e) must be implemented no later than August 1, 2012. The training procedures required by paragraph (h) must be implemented no later than August 1, 2012, except that any training required by another paragraph of this section must be implemented no later than the deadline for that paragraph. The Sunoco Pipeline L.P. & Affiliates Control Room Management Manual that was issued on October 1, 2011, updated on November 15, 2011, failed to have a detailed written process on how training scenarios for recognizing and responding to abnormal operating conditions that are likely to occur simultaneously or in sequence are determined, in accordance with §195.446(h)(1). Specifically, the CRM Training Rule procedure did not address reviewing historical alarm logs to identify candidate scenarios for training. Proposed Civil Penalty Under 49 United States Code, § 60122, you are subject to a civil penalty not to exceed $200,000 per violation per day the violation persists up to a maximum of $2,000,000 for a related series of violations. For violations occurring prior to January 4, 2012, the maximum penalty may not exceed $100,000 per violation per day, with a maximum penalty not to exceed $1,000,000 for a related series of violations. The Compliance Officer has reviewed the circumstances and supporting documentation involved in the above probable violation and has recommended that you be preliminarily assessed a civil penalty of $30,200 as follows: Item number PENALTY 1 $30,200 Proposed Compliance Order With respect to item 1 pursuant to 49 United States Code § 60118, the Pipeline and Hazardous Materials Safety Administration proposes to issue a Compliance Order to Sunoco. Please refer to the Proposed Compliance Order, which is enclosed and made a part of this Notice. Response to this Notice Enclosed as part of this Notice is a document entitled Response Options for Pipeline Operators in Compliance Proceedings. Please refer to this document and note the response options. All material you submit in response to this enforcement action may be made publicly available. If you believe that any portion of your responsive material qualifies for confidential treatment under 5 U.S.C. 552(b), along with the complete original document you must provide a second copy of the document with the portions you believe qualify for confidential treatment redacted and an explanation of why you believe the redacted information qualifies for confidential treatment under 5 U.S.C. 552(b). If you do not respond within 30 days of receipt of this Notice, this constitutes a waiver of your right to contest the allegations in this Notice and authorizes the Associate Administrator for Pipeline Safety to find facts as alleged in this Notice without further notice to you and to issue a Final Order.120135021_NOPV_PCP_PCO_09302013 Page 2 of 4#

120135021_NOPV PCP PCO_09302013_text.pdf, page 3CPF 1-2013-5021 Please submit all correspondence in this matter to Byron Coy, PE, Director, PHMSA Eastern Region, 820 Bear Tavern Road, Suite 103, W. Trenton, NJ 08628. Please refer to CPF 1-2013-5021 on each document you submit, and please whenever possible provide a signed PDF copy in electronic format. Smaller files may be emailed to Byron.Coy@dot.gov. Larger files should be sent on a CD accompanied by the original paper copy to the Eastern Region Office. Sincerely, Byron Coy, PE Director, Eastern Region Pipeline and Hazardous Materials Safety Administration Enclosures: Proposed Compliance Order Response Options for Pipeline Operators in Compliance Proceedings 120135021_NOPV_PCP_PCO_09302013 Page 3 of 4#

120135021_NOPV PCP PCO_09302013_text.pdf, page 4CPF 1-2013-5021 PROPOSED COMPLIANCE ORDER Pursuant to 49 United States Code § 60118, the Pipeline and Hazardous Materials Safety Administration (PHMSA) proposes to issue to Sunoco Pipeline L.P. (Sunoco) a Compliance Order incorporating the following remedial requirements to ensure the compliance of Sunoco with the pipeline safety regulations: 1. With respect to Item Number 1 of the Notice, Sunoco must amend its Control Room Management Manual to include a detailed written process for reviewing historical alarm logs to identify training scenarios, in accordance with §195.446(h)(1). Sunoco must submit its amended procedure to Byron Coy, Director, Eastern Region, Pipeline and Hazardous Materials Safety Administration within 60 days of receipt of the Final Order. 2. It is requested (not mandated) that Sunoco maintain documentation of the safety improvement costs associated with fulfilling this Compliance Order and submit the total to Byron Coy, Director, Eastern Region, Pipeline and Hazardous Materials Safety Administration. It is requested that these costs be reported in two categories: 1) total cost associated with preparation/revision of plans, procedures, studies and analyses, and 2) total cost associated with replacements, additions and other changes to pipeline infrastructure. 120135021_NOPV_PCP_PCO_09302013 Page 4 of 4#

120135021_Final Order_12022016_text.pdf, page 1Official PDFDecember 2, 2016 Mr. Michael J. Hennigan President & CEO Sunoco Pipeline L.P. 1801 Market Street Suite 1500 Philadelphia, PA 19103 Re: CPF No. 1-2013-5021 Dear Mr. Hennigan: Enclosed please find the Final Order issued in the above-referenced case. It makes findings of violation, assesses a reduced civil penalty of $25,900, and finds that Sunoco Pipeline L.P. has completed the actions specified in the Notice to comply with the pipeline safety regulations. The penalty payment terms are set forth in the Final Order. This enforcement action closes automatically upon receipt of payment. Service of the Final Order by certified mail is effective as provided under 49 C.F.R. § 190.5. Thank you for your cooperation in this matter. Sincerely, Alan K. Mayberry Acting Associate Administrator for Pipeline Safety Enclosure cc: Mr. Byron Coy, Director, Eastern Region, OPS Mr. Kevin Dunleavy, Chief Counsel, Sunoco Pipeline L.P. 3801 West Chester Pike, Newtown Square, PA 19073 CERTIFIED MAIL - RETURN RECEIPT REQUESTED#

120135021_Final Order_12022016_text.pdf, page 2U.S. DEPARTMENT OF TRANSPORTATION PIPELINE AND HAZARDOUS MATERIALS SAFETY ADMINISTRATION OFFICE OF PIPELINE SAFETY WASHINGTON, D.C. 20590 ___________________________________ In the Matter of ) Sunoco Pipeline L.P., ) CPF No. 1-2013-5021 ) ) ) Respondent. ) ___________________________________ ) FINAL ORDER On December 7-9, 2011, pursuant to 49 U.S.C. § 60117, representatives of the Pipeline and Hazardous Materials Safety Administration (PHMSA), Office of Pipeline Safety (OPS), conducted an inspection of the control room management procedures of Sunoco Pipeline L.P. (Sunoco or Respondent) in Montello, Pennsylvania.1 As a result of the inspection, the Director, Eastern Region, OPS (Director), issued a Notice of Probable Violation, Proposed Civil Penalty, and Proposed Compliance Order (Notice) to Respondent on September 30, 2013. In accordance with 49 C.F.R. § 190.207, the Notice alleged that Sunoco committed a single violation of the control room management regulations and proposed a civil penalty of $30,200. The Notice also proposed corrective action to remediate the alleged violation. Sunoco responded to the Notice and requested a hearing by letter dated October 31, 2013. Respondent submitted written exhibits on May 8, 2014. In accordance with § 190.211, a hearing was held on May 22, 2014, in Trenton, New Jersey, before a Presiding Official from the Office of Chief Counsel, PHMSA. After the hearing, Respondent submitted a post-hearing brief on June 23, 2014 (Brief). Pursuant to § 190.209(b)(7), the Director submitted a written evaluation of Respondent’s response material on September 30, 2015. FINDING OF VIOLATION Item 1: The Notice alleged that Respondent violated 49 C.F.R. § 195.446(a), which states: 1 Sunoco is a subsidiary of Sunoco Logistics Partners L.P. and operates approximately 6,000 miles of pipeline transporting primarily crude oil and refined products in Texas, Oklahoma, and several other states. This information is reported by Sunoco for calendar year 2015 pursuant to 49 C.F.R. § 195.49.#

120135021_Final Order_12022016_text.pdf, page 3CPF No. 1-2013-5021 Page 2 § 195.446 Control room management. (a) General. This section applies to each operator of a pipeline facility with a controller working in a control room who monitors and controls all or part of a pipeline facility through a SCADA system. Each operator must have and follow written control room management procedures that implement the requirements of this section. The procedures required by this section must be integrated, as appropriate, with the operator’s written procedures required by § 195.402. An operator must develop the procedures no later than August 1, 2011, and must implement the procedures according to the following schedule. The procedures required by paragraphs (b), (c)(5), (d)(2) and (d)(3), (f) and (g) of this section must be implemented no later than October 1, 2011. The procedures required by paragraphs (c)(1) through (4), (d)(1), (d)(4), and (e) must be implemented no later than August 1, 2012. The training procedures required by paragraph (h) must be implemented no later than August 1, 2012, except that any training required by another paragraph of this section must be implemented no later than the deadline for that paragraph . . . . (h) Training. Each operator must establish a controller training program and review the training program content to identify potential improvements at least once each calendar year, but at intervals not to exceed 15 months. An operator’s program must provide for training each controller to carry out the roles and responsibilities defined by the operator. In addition, the training program must include the following elements: (1) Responding to abnormal operating conditions likely to occur simultaneously or in sequence . . . . The Notice alleged that Respondent violated § 195.446(a) by failing to have control room management procedures for implementing the requirements of paragraph (h)(1) of that section related to controller training. Paragraph (h)(1) requires an operator to have procedures for training controllers on responding to abnormal operating conditions (AOCs) “likely to occur simultaneously or in sequences.” The Notice alleged that Respondent’s procedures did not have a detailed written process for how Respondent would determine training scenarios for recognizing and responding to AOCs likely to occur simultaneously or in sequence. Specifically, the Notice alleged the procedures did not address reviewing historical alarm logs to identify candidate scenarios for training. Respondent argued that even though its written procedures did not contain the words “simultaneously or in sequence,” the Company’s actual training program trains controllers to recognize and respond to AOCs likely to occur simultaneously or in sequence, and in fact, Respondent uses historical alarm logs. Moreover, Respondent argued that although the regulation requires an operator to have a training program, the regulation does not require an operator to have written procedures for the training program. Finally, Respondent argued OPS is impermissibly attempting to enforce a guidance document that merely recommends, but does not#

120135021_Final Order_12022016_text.pdf, page 4CPF No. 1-2013-5021 Page 3 require, operators to review historical alarm logs to identify candidate scenarios for AOC training. Analysis Section 195.446(a) requires a pipeline operator to have written control room management procedures. At a minimum, the procedures must implement the requirements of § 195.446, including § 195.446(h), which requires an operator to have a training program for controllers. Each operator must therefore have written procedures that implement its controller training program. The procedures for the training program must include, among other things, training controllers on responding to AOCs “likely to occur simultaneously or in sequence” as stated in § 195.446(h)(1). I have reviewed the record to determine whether Respondent had written procedures for training controllers to respond to AOCs likely to occur simultaneously or in sequence. In the record is Respondent’s manual of written control room management procedures.2 Section H of the manual is titled “CRM Training Rule.” This section documents Respondent’s procedures for training controllers. Both OPS and Respondent submitted for the record copies of Section H with various passages highlighted that are relevant to AOCs. OPS highlighted provisions in Section H that reference the class (or module) number for training controllers on “Control Center Abnormal Operating Conditions.”3 Also, there is a provision that references initial training for controllers on a number of subjects, including “Abnormal Operating Conditions.”4 OPS highlighted a provision concerning tests for console qualification, including a “Written AOC Test” and a skill demonstration test that includes “3 random AOC/Emergency scenarios and how to react.”5 Respondent highlighted those plus a few additional provisions. Respondent pointed to a procedure that references a computer based “AOC Exam” that “Covers Abnormal Operating Conditions.”6 The initial testing procedure references computer based training on topics including “Dispatching; Normal, Abnormal, and Emergency Procedures.”7 Finally, Respondent highlighted a provision on refresher training that references desktop drills “related to recognition of and response to AOC/Emergency situations.”8 2 OPS Violation Report, Exhibit A-1; Respondent Prehearing Submission, Exhibit 5. The procedures were issued October 1, 2011. Despite issuance after the regulatory deadline of August 1, 2011, the Notice did not allege any violation with regard to Respondent exceeding the deadline. 3 OPS Violation Report, Ex. A-1, Sec. II.B.2. 4 OPS Violation Report, Ex. A-1, Sec. III.C.24. 5 OPS Violation Report, Ex. A-1, Sec. IV.E. 6 Respondent Prehearing Submission, Ex. 5, Sec. II.C.2. 7 Respondent Prehearing Submission, Ex. 5, Sec. III.D.3. 8 Respondent Prehearing Submission, Ex. 5, Sec. VI.C.#

120135021_Final Order_12022016_text.pdf, page 5CPF No. 1-2013-5021 Page 4 All of the procedures highlighted by OPS and Respondent in Section H reference training of controllers on AOCs, but none of them include any provisions for training controllers on recognizing and responding to AOCs likely to occur simultaneously or in sequence. The procedures only discuss training controllers about AOCs generally, not training on the possibility of multiple AOCs occurring at the same time or immediately following one another. Respondent argued that even though it might not be detailed in its procedures, the Company actually trains controllers on AOCs likely to occur simultaneously or in sequence. Respondent argued the regulation only requires the operator to have a training “program” and does not otherwise require written procedures for the program. I disagree. Section 195.446(a) requires a pipeline operator to have written procedures that implement the requirements in § 195.446(h) for a training program. Accordingly, Respondent’s written procedures must provide for the implementation of a controller training program that includes training controllers on responding to AOCs likely to occur simultaneously or in sequence pursuant to § 195.446(h)(1). To the extent Respondent actually trained controllers on AOCs likely to occur simultaneously or in sequence, it does not demonstrate compliance with the requirement to have written procedures. Respondent argued that OPS alleged a violation in this case solely because the words “simultaneously or in sequence” are missing from its written procedures. I disagree. The absence of verbatim regulatory text in Respondent’s procedures is not the basis for the allegation. The basis is the evidence that Respondent did not have written procedures that met the requirements of §§ 195.446(a) and (h)(1). Respondent also argued that OPS is attempting to enforce guidance that merely recommends, but does not require that operators review historical alarm logs to identify candidate scenarios for training. The guidance in question is a set of inspection questions, or protocols, that OPS has made publically available and that the Agency uses to conduct control room management safety inspections. The protocol regarding controller training states the following: H1-1: Does the operator’s program provide controller training on recognizing and responding to abnormal operating conditions that are likely to occur simultaneously or in sequence?  Operator must establish a list of foreseeable operating scenarios that are more likely to cause simultaneous AOCs, or multiple AOCs in sequence, and train controllers on how to recognize and handle them.  Operators must include training on lessons learned from the review of operating experience, in accordance with (g)(2), including critiques of all recent accidents/incidents.  Operators should review historical alarm logs to identify candidate scenarios for training. 9 9 Respondent Prehearing Submission, Exhibit 4, PHMSA Control Room Management: Inspection Questions at 53 (Sept. 30, 2011) (emphasis added).#

120135021_Final Order_12022016_text.pdf, page 6CPF No. 1-2013-5021 Page 5 Respondent is correct that the protocol is not adopted into the pipeline safety regulations and therefore does not, on its own, form the basis for a regulatory violation. The guidance does, however, provide information to the regulated community regarding how OPS evaluates an operator’s control room management program under the regulations. For example, with respect to the regulation cited in this case, the Agency believes the best way to identify AOCs likely to occur simultaneously or in sequence is to review historical alarm logs. This is because an operator’s alarm logs document actual occurrences of AOCs on the operator’s pipeline system. The alarm logs also demonstrate when AOCs have occurred simultaneously or in sequence. It may be possible for an operator to use an alternative method to identify AOCs that are likely to occur simultaneously or in sequence on its system, but in most cases the Agency believes an operator will at least need to review actual historical AOCs documented in its historical alarm logs. In the present case, Respondent has not demonstrated that its procedures provided for the review of historical alarm logs or any other method for identifying and training controllers on AOCs likely to occur simultaneously or in sequence on its pipeline system. Accordingly, having reviewed the record, I find that Respondent violated § 195.446 by failing to have control room management procedures for implementing the requirements of § 195.446(h)(1) related to controller training. ASSESSMENT OF PENALTY Under 49 U.S.C. § 60122, Respondent is subject to an administrative civil penalty not to exceed $200,000 per violation for each day of the violation, up to a maximum of $2,000,000 for any related series of violations.10 In determining the amount of a civil penalty under 49 U.S.C. § 60122 and 49 C.F.R. § 190.225, I must consider the following criteria: the nature, circumstances, and gravity of the violation, including adverse impact on the environment; the degree of Respondent’s culpability; the history of Respondent’s prior offenses; and any effect that the penalty may have on its ability to continue doing business; and the good faith of Respondent in attempting to comply with the pipeline safety regulations. In addition, I may consider the economic benefit gained from the violation without any reduction because of subsequent damages, and such other matters as justice may require. 10 On June 30, 2016, PHMSA adjusted the maximum penalties for inflation (81 Fed. Reg. 42564). Pursuant to § 190.223, any person found to have committed a violation on or after August 1, 2016, is subject to an administrative civil penalty not to exceed $205,638 for each violation for each day the violation continues, with the maximum administrative civil penalty not to exceed $2,056,380 for any related series of violations.#

120135021_Final Order_12022016_text.pdf, page 7CPF No. 1-2013-5021 Page 6 Item 1: The Notice proposed a civil penalty of $30,200 for Respondent’s violation of 49 C.F.R. § 195.446(a), for failing to have control room management procedures that implemented the requirements of § 195.446(h)(1) related to controller training. The proposed penalty was based on assertions in the Notice and Violation Report relevant to the penalty assessment criteria in § 190.225. With regard to the nature of the violation, the Violation Report noted this violation concerned Respondent’s failure to have procedures related to controller training. With regard to circumstances, the Violation Report noted the violation was discovered by OPS. With regard to gravity, the Violation Report suggested pipeline safety was minimally affected. With regard to the degree of Respondent’s good faith, the Violation Report suggested no credit under this factor. Respondent has a history of nine prior offenses within the last five years. Respondent argued that the proposed penalty should be reduced because it had written procedures for training controllers on AOCs and followed them. Respondent also offered evidence that AOCs were listed as an element of the CRM Training Program,11 and that the training materials and content addressed the possible occurrence of AOCs likely to occur simultaneously or in sequence.12 Respondent also provided examples of historic AOCs that occurred simultaneously or in sequence, which were used for training.13 Further, a narrative summary of the event, historic alarm logs, and historian trend printouts were used to illustrate the examples.14 With regard to the culpability of the Respondent, I find that although the Company’s procedures did not meet the requirement of the regulation, Respondent has taken significant steps towards compliance with the cited regulation by ensuring controllers were trained regarding AOCs likely to occur simultaneously or in sequence. Therefore, I find a reduction to the proposed penalty is appropriate. Based upon the foregoing, I assess Respondent a reduced civil penalty of $25,900 for violation of 49 C.F.R. § 195.446(a). Payment of the civil penalty must be made within 20 days of service. Federal regulations (49 C.F.R. § 89.21(b)(3)) require such payment to be made by wire transfer through the Federal Reserve Communications System (Fedwire), to the account of the U.S. Treasury. Detailed instructions are contained in the enclosure. Questions concerning wire transfers should be directed to: Financial Operations Division (AMK-325), Federal Aviation Administration, 6500 S MacArthur Blvd., Oklahoma City, Oklahoma 79169. The Financial Operations Division telephone number is (405) 954-8845. 11 Respondent Prehearing Submission, Ex. 5, Section III. Initial training C. 12 Respondent Prehearing Submission, Exs. 6 and 10. 13 Respondent Prehearing Submission, Ex. 10. 14 Closing at 7.#

120135021_Final Order_12022016_text.pdf, page 8CPF No. 1-2013-5021 Page 7 Failure to pay the $25,900 civil penalty will result in accrual of interest at the current annual rate in accordance with 31 U.S.C. § 3717, 31 C.F.R. § 901.9 and 49 C.F.R. § 89.23. Pursuant to those same authorities, a late penalty charge of six percent (6%) per annum will be charged if payment is not made within 110 days of service. Furthermore, failure to pay the civil penalty may result in referral of the matter to the Attorney General for appropriate action in a district court of the United States. COMPLIANCE ORDER The Notice proposed a compliance order with respect to Items 1 in the Notice for violation of 49 C.F.R. § 195.446 respectively. Under 49 U.S.C. § 60118(a), each person who engages in the transportation of hazardous liquids or who owns or operates a pipeline facility is required to comply with the applicable safety standards established under chapter 601. The Director indicates that Respondent has taken the following actions specified in the proposed compliance order: With respect to the violation of § 195.446 (Item 1), Respondent amended its Control Room Management Manual to include a detailed written process for reviewing historical alarm logs to identify training scenarios, in accordance with §195.446(h)(1). Accordingly, I find that compliance has been achieved with respect to this violation. It is not necessary to include compliance terms in this Order. Under 49 C.F.R. § 190.243, Respondent may submit a petition for reconsideration of this Final Order to the Associate Administrator for Pipeline Safety, PHMSA, 1200 New Jersey Avenue SE, East Building, 2nd Floor, Washington, DC 20590, no later than 20 days after receipt of the Final Order by the Respondent. Any petition submitted must contain a brief statement of the issue(s) and meet all other requirements of 49 C.F.R. § 190.243. The filing of a petition automatically stays the payment of any civil penalty assessed, however, the other terms of the order, including the corrective action, remain in effect unless the Associate Administrator, upon request, grants a stay. The terms and conditions of this Final Order are effective upon service in accordance with 49 C.F.R. § 190.5. December 2, 2016 _______________________________ _______________________ Alan K. Mayberry Date Issued Acting Associate Administrator for Pipeline Safety#