NTSB Safety Recommendation P-11-024

NTSB safety recommendation P-11-024.

TO PACIFIC GAS AND ELECTRIC COMPANY: Revise your work clearance procedures to include requirements for identifying the likelihood and consequence of failure associated with the planned work and for developing contingency plans.

Priority: CLASS II

Overall Status: Closed - Acceptable Action

Issued Date: 2011-09-26

Adopted Date: 2011-09-12

Overall Date Closed: 2013-03-14

Synopsis: On September 9, 2010, about 6:11 p.m. Pacific daylight time,1 a 30-inch-diameter underground natural gas transmission pipeline (which is identified by the company as Line 132), owned and operated by Pacific Gas and Electric Company (PG&E), ruptured in a residential area in San Bruno, California. The accident killed eight people, injured many more, and caused substantial property damage. The rupture on Line 132 occurred near mile point 39.28, at the intersection of Earl Avenue and Glenview Drive in San Bruno. About 47.6 million standard cubic feet of natural gas were released as a result of the rupture. The released natural gas was ignited after the rupture; the subsequent explosion created a crater about 72 feet long by 26 feet wide, and the resulting fire destroyed 38 homes and damaged 70. A ruptured pipe segment about 28 feet long was found about 100 feet away from the crater.

Probable Cause: The National Transportation Safety Board determines that the probable cause of the accident was the Pacific Gas and Electric Company's (PG&E) (1) inadequate quality assurance and quality control in 1956 during its Line 132 relocation project, which allowed the installation of a substandard and poorly welded pipe section with a visible seam weld flaw that, over time grew to a critical size, causing the pipeline to rupture during a pressure increase stemming from poorly planned electrical work at the Milpitas Terminal; and (2) inadequate pipeline integrity management program, which failed to detect and repair or remove the defective pipe section.

Ntsbnumber: DCA10MP008

Report Number: PAR-11-01

Addressee Name: Pacific Gas and Electric Company

Addressee Status: Closed - Acceptable Action

Addressee Date Closed: 2013-03-14

Addressee Acronym: PG&E

Addressee Organization Type: P-Private Industry

Communication Date: 2012-03-13

Communication Type: Official Correspondence

Communication Contents: The NTSB understands that PG&E is currently evaluating approaches to revise work clearance procedures and to update its SCADA leak detection capabilities (such as information technology solutions, Lean Six Sigma improvement processes, emergency backup relocation exercises, and a new Data Historian system to assist operators and planning teams). Pending receipt of further information from PG&E on the completion of these efforts, Safety Recommendations P-11-24 and -26 are classified OPEN—ACCEPTABLE RESPONSE.

Addressee Acronym: PG&E

Addressee Organization Type: P-Private Industry

Communication Date: 2012-08-29

Communication Type: Official Correspondence

Communication Contents: The NTSB understands that PG&E is nearing completion of its work clearance procedure and will issue the revised procedure to all employees involved in the gas clearance process before the end of 2012. PG&E will also improve its clearance work processes by creating a distribution control center by the end of 2012. The center will oversee a uniform distribution clearance process nearly identical to the transmission process. In addition, PG&E’s utility performance improvement team (Lean Six Sigma experts), in conjunction with gas control, engineering, and field maintenance, are now writing the distribution clearance process, which is expected to be completed in the third quarter of 2012. Pending completion of these efforts, Safety Recommendation P-11-24 is classified OPEN—ACCEPTABLE RESPONSE.

Addressee Acronym: PG&E

Addressee Organization Type: P-Private Industry

Communication Date: 2012-02-06

Communication Type: Official Correspondence

Communication Contents: Correspondence control 201200077 was closed administratively and it was combined with 201100506. The reply to 201100506/ 201200077 was mailed on 3/13/2012.

Addressee Acronym: PG&E

Addressee Organization Type: P-Private Industry

Communication Date: 2013-03-14

Communication Type: Official Correspondence

Communication Contents: The revisions that PG&E has made to its work clearance procedures and other PG&E actions discussed in your letter satisfy the intent of Safety Recommendation P-11-24. Accordingly, this recommendation is classified CLOSED—ACCEPTABLE ACTION.

Addressee Acronym: PG&E

Addressee Organization Type: P-Private Industry

Communication Date: 2012-01-27

Communication Type: Official Correspondence

Communication Contents: -From Christopher P. Johns, President: Pacific Gas and Electric Company (PG&E) fully embraces the safety recommendations outlined by the National Transportation Safety Board as part of the agencies review of the 2010 San Bruno pipeline accident. We recognize the importance of preventing a tragedy like this from occurring again and that is why we are committed to successfully implementing the NTSB's recommendations. To date, PG&E has taken numerous actions to make fundamental changes to our operations and management -all with a focus on putting public and employee safety first. In response to your September 26, 2011 letter, PG&E sent to you on December 22, 2011, an update on the progress we have made toward implementing the recommendations and the plans we have in place. This letter included an update on activities related to integrity management, emergency response, public awareness, threat assessment and recordkeeping, among other aspects of our operations and management. Since that time, members of our team have had the opportunity to meet with NTSB technical staff to discuss our submission and review our activities. We are grateful for the guidance and feedback provided at that meeting. As a result of those discussions, we recognize that it is incumbent upon us to provide a greater level of detail than was previously provided in the December 22, 2011 update. Therefore, we would like to take the opportunity to supplement our December 22, 2011 response by providing an amended response with the requisite detail within 45 days. PG&E will continue to meet with NTSB staff in the coming weeks to seek additional guidance to ensure that the update we provide is fully responsive and will allow the NTSB to more ably assess our progress and plans.

Addressee Acronym: PG&E

Addressee Organization Type: P-Private Industry

Communication Date: 2013-01-31

Communication Type: Official Correspondence

Communication Contents: -From Christopher P. Johns, President: Pacific Gas and Electric Company (PG&E) continues to make substantial progress implementing the safety recommendations outlined by the NTSB's investigation of the September 2010 San Bruno pipeline accident. This status report provides details on the actions we are taking to assure public safety remains the company's highest priority. In 2012, the NTSB evaluated PG&E's progress and closed four recommendations: 1. pe10-2: Search for Records 2. P-11-3: 911 Notifications 3. P-11-25: Emergency Response Procedures 4. P-11-28: Toxicology Testing In this report, we are submitting three additional recommendations for closure consideration by the NTSB: 1. P-10-3: MAOP Validation 2. P-11-24: Work Clearance Procedures 3. P-11-31: Public Awareness Program Continuous Improvement For recommendation P-10-3 (MAOP Validation), PG&E has completed the determination of the valid maximum allowable operating pressure (MAOP), based on the weakest section of the pipeline or component. The purpose of the MAOP validation is to ensure safe operation of natural gas transmission lines in class 3 and class 4 locations and class 1 and class 2 high consequence areas (HCA) that have not had a MAOP established through prior hydrostatic testing. In total, MAOP validation was performed for all 2,088 miles of these transmission pipelines. In addition to completing NTSB Recommendation P-10-3, PG&E is validating all remaining transmission lines in non-HCAs by mid- 2013. In 2012, PG&E completed the MAOP validation of 4, 199 miles of non-HCA pipelines. For recommendation P-11-24 (Work Clearance Procedures), PG&E has completed the revision and issuance of work clearance procedures that include requirements for identifying the likelihood and consequences of failure associated with planned work. The development of contingency plans is now a part of this process. PG&E's new procedure ensures accurate and completed clearance forms and requires field crews, control room operators and individuals who have been assigned the clearance supervisor role to have complete knowledge of the intended work and written clearance procedure. PG&E has completed recommendation P-11-31 (Public Awareness Program Continuous Improvement) through the development and incorporation of written performance measurements and guidelines into our Public Awareness Plan (PAP) for evaluating the plan and for continuous program improvement. The primary objectives include awareness, damage prevention and emergency response readiness. PG&E has also completed two portions of recommendation P-11-29 (Integrity Management Program): Revisions to PG&E's Risk Model and Risk Analysis Methodology. Other recommendations with significant progress highlighted in the attachment include: • (P-10-4)-ln 2012, PG&E strength tested or verified an additional 202 miles for a total of 417 miles since 2011 • (P-11-2)-PG&E installed 46 valves in 2012 (for a total of 59 valves since 2010) • (P-11-29)-ln addition to revising the Integrity Management Risk Model and Risk Analysis Methodology, PG&E is continuing to revise other portions of its integrity management program PG&E thanks the NTSB for both its continuing guidance and leadership as the company works to address the remaining safety recommendations. Please contact me directly if you have any questions. PG&E's has completed the implementation of a revised work clearance procedure to include requirements for identifying the likelihood and consequences of failure associated with the planned work and developing contingency plans (Attachment P-11-24 Clearance). PG&E has clarified and underscored the following in its revised clearance procedure: • All sections and fields contained in the clearance form must be filled out completely. • Individuals assigned the clearance supervisor role must have complete knowledge of the intended work and written clearance procedure before accepting this role. - Field crew and control room operators must have clear and complete understanding of the scope and details of the clearance, including consequences and contingency plans. The understanding of the clearance will be gained through a crew tailboard and phone calls to the control room. PG&E's Control Room Management process includes a change management procedure that requires commissioning and functional check out testing (end to end testing) of all components at the field level connected to SCADA. Commissioning and functional check out testing is now being completed for all new and rebuilt installations in conjunction with work clearance activities. In addition to completing NTSB Recommendation P-11-24 for the gas transmission system, PG&E is implementing new electronic tools by the end of 02 in 2013. In November 2012, the new Distribution Control Center opened for training in San Francisco; monitoring will begin in 01 of 2013 and full operation is targeted for 04 of 2013.

Addressee Acronym: PG&E

Addressee Organization Type: P-Private Industry

Communication Date: 2012-05-23

Communication Type: Official Correspondence

Communication Contents: -From Christopher P. Johns, President: Pacific Gas and Electric Company (PG&E) is making substantial progress to implement the safety recommendations outlined by the National\ Transportation Safety Board's (NTSB) investigation of the September 2010 San Bruno pipeline accident. This status report details the actions PG&E has taken and will continue to take to assure public safety remains the company's highest priority in the operation of our natural gas transmission system. We have made many fundamental changes to the operations and management practices throughout our gas organization. Some of the more critical actions include: completion of the Maximum Allowable Operating Pressure validation of all 2,088 miles of high consequence area (HCA) pipelines; validation of 1,032 miles of non -HCA pipelines through April 2012; and automation of 36 valves. In 2011, PG&E conducted strength tests and verified strength test pressure records for a total of approximately 214.5 miles of pipeline. Additionally, from January through April 2012, an additional 37.6 miles have been tested or verified. PG&E's Pipeline Safety Enhancement Plan (PSEP), currently before the CPUC for approval in R.11-02-019, outlines our plan for testing or verifying records of 547 miles in 2012 -2014. PG&E has modified its 911 Notification Process to respond to the NTSB's suggestion that SCADA real time operating data and alarms serve as triggers for 911 notifications. We have developed a comprehensive emergency response procedure for large-scale emergencies on transmission lines, which identifies a single person in charge, outlines specific protocols and provides for drills and training. PG&E has also incorporated performance measures and guidelines to assure continuous improvement in our public awareness program. PG&E has initiated a complete assessment of every aspect of our transmission integrity management program including threat identification and assessment. Much more work lays ahead, but the progress to date gives us confidence that PG&E is on track to achieving the goal of regaining the trust of the public and our regulators by demonstrating our steadfast commitment to safety. The attachment to this letter provides a more detailed summary of each NTSB recommendation and the steps we have taken and continue to take to act upon them. If you have any questions, please contact me directly. PG&E is nearing finalization of the Work Clearance Procedure and is committed to issuing the revised procedure to all employees involved in the gas clearance process before the end of the 2nd quarter of 2012. The working draft of the revised procedure is shown in Attachment P-11-24A. Attachments P-1124B through P-11 24H are the job aids that support the work clearance procedure. PG&E recognizes the importance of requiring a well-analyzed, ful1y and properly completed clearance form including risk assessment and contingency planning for al1 work that could potentially impact the system. Currently, work that has been identified as potentially impacting a station (or valve) control system or electrical supply is routed to the local facility/controls engineer for review to ensure the identified work will not pose a risk to the normal operations of the facility. Gas Control's final approval process verifies that all work associated with control systems or electrical supplies has been properly reviewed and if not, routes the draft clearance to proper reviewers before issuing final approval. PG&E has also clarified and underscored the following in its revised clearance procedure: • All sections and fields contained in the clearance form must be filled out completely. PG&E is bUilding an electronic tool that will prevent the clearance form from moving forward for approval if it is incomplete. • Individuals assigned the clearance supervisor role must have complete knowledge of the intended work and written clearance procedure before accepting this role. • Field crew and control room operators must have clear and complete understanding of the scope and details of the clearance. The understanding of the clearance will be gained through a crew tailboard and phone calls to the control room. PG&E's Control Room Management process includes a change management procedure that requires commissioning and functional check out testing (end to end testing) of all components at the field level connected to SCADA. Commissioning and 'functional check out testing is now being completed for all new and rebuilt installations in conjunction with work clearance activities. The requirement to write the step by step test sequence forces the engineering, field and Gas Control crews to proactively review the impact of potential equipment failure. These activities are additional measures PG&E is taking to ensure that facility equipment is capable of meeting normal operating requirements at all times. As noted above, PG&E expects to issue the finalized gas clearance process before the end of the 2nd quarter of 2012. Web based training will be completed by all employees involved in the gas clearance process upon rollout of the revised procedure. Through industry level benchmarking (site visits to more than a dozen major North American gas and electric utilities), PG&E has learned that the best in practice clearance processes utilize an electronic tool accessible to all participants involved in a clearance. Use of the electronic tool will ensure sustained conformance with the clearance procedure requirements and the completion of appropriate levels of review by engineering, maintenance, and Gas Control before clearance work begins. PG&E has committed funding to build an electronic clearance writing, calendaring, routing and approval tool. The electronic clearance tool will also allow enhanced Control Room visibility through the use of large video wall screens. It is anticipated the new electronic tool development and prototype rollout to field organizations by the end of 1st quarter 2013. PG&E plans to further improve its clearance work processes by creating a Distribution Control Center by the end of 2012. The Distribution Control Center will oversee a uniform distribution clearance process nearly identical to the transmission process. PG&E's Utility Performance Improvement team (Lean Six Sigma experts) in conjunction with Gas Control, engineering, and field maintenance has undertaken the effort to write the distribution clearance process and it is on track to be completed by 3rd quarter of 2012.

Addressee Acronym: PG&E

Addressee Organization Type: P-Private Industry

Communication Date: 2011-12-22

Communication Type: Official Correspondence

Communication Contents: -From Christopher P. Johns, President: Pacific Gas and Electric Company (PG&E) has fully embraced the safety recommendations outlined by the National Transportation Safety Board (NTSB) as part of the agency's review of the September 2010 San Bruno pipeline accident. We at PG&E recognize the importance of preventing a tragedy like this from ever happening again, and we are grateful for the NTSB's guidance toward that goal. As requested in your September 26, 2011 letter, we are providing an update on the actions PG&E is taking, and the plans that are in place to implement the NTSB's thoughtful and thorough recommendations. These recommendations have prompted PG&E to take significant actions on many fronts, including pipeline leak and break identification, emergency response, integrity management, threat assessments, public awareness of pipeline safety measures, and recordkeeping. The NTSB's recommendations have helped propel PG&E to make fundamental changes to our operations and management-all intended to underscore our commitment to putting public and employee safety first. For example, we've completed Maximum Allowable Operating Pressure validation of more than 1,600 miles of high consequence area pipelines, with plans to complete all 2,000 miles by the end of January 2012. We are implementing a new data management system intended to address the NTSB's recommendations for traceable, verifiable and complete records. Additionally, PG&E has automated 11 shutoff valves in 2011. As part of the company's Pipeline Safety Enhancement Plan, which was presented for approval to the California Public Utilities Commission this year, PG&E has proposed automating a total of 228 valves through 2014. We've also updated our emergency response plans to reflect industry best practices and have begun training employees, public officials and first responders. In an effort to improve public gas safety awareness, we've mailed more than two million safety information letters to customers who live within 2,000 feet of a transmission pipeline. We realize we still have much to learn and do in the wake of the tragic San Bruno accident. As you'll see in the attachment, we have completed several of the recommendations and are in mid-stream on the remainder. We're working with urgency to complete all of the recommendations and we remain steadfast in our commitment to safety and to regaining the trust of the public and our regulators. The attachment to this letter provides a more detailed summary of each NTSB recommendation and the steps we are taking to act upon them. If you have any questions, please contact me directly. • Developing Comprehensive Controls Framework PG&E is currently evaluating information technology solutions to deliver the right information to operators to allow them to make prompt, informed decisions related to pipeline safety. PG&E is incorporating Lean Six Sigma improvement processes from a variety of internal stakeholders and industry consultants to ensure a solution focused on interoperability and usability. Examples include: o Beginning the establishment of a new Distribution Control Center to expand visibility and control capability of the distribution system. o The new Distribution Control Center will work in tandem with the existing Transmission Control Center function. o PG&E will take best practices and findings from internal assessments and external benchmarking to create a leading class distribution control center that oversees planning and execution of critical distribution system enhancements, and monitors work on the distribution system. It will also include low-power consumption solutions for gathering SCADA data from remote gas distribution assets. o Ensuring that both the Transmission and Distribution control centers will be supported by a common SCADA and real-time Data Historian platform, an enhanced clearance process, and integration with the Gas Dispatch and Emergency Response organizations. o Nearing completion of a SCADA modification project as a major step in implementing an alarm management process. o This SCADA enhancement will prioritize alarms for appropriate operator action upon activation. o In conjunction with the new alarm management strategy, PG&E is working with human factors consultants to develop a new SCADA visual coding design, including use of color, text and symbols in graphic displays to present alarm status and data quality. The new design will meet the requirements of API 1165 (Graphic Standard, Recommended Practice for Pipeline SCADA Display) by August 2012. This modification will prioritize alarms on Operators' operating screens.