Application No.: A.19-08-013Exhibit No.: SCE-06, Vol. 1 Part 1A EWitnesses: L. Garris

J. IshiguroA. MaR. NandaE. Roddick

(U 338-E)

2021 General Rate Case

Amended Testimony on Enterprise Technology

Before the

Public Utilities Commission of the State of California

Rosemead, California February 20, 2020

ERRATA

1

1 I.

2 INTRODUCTION

3 A. Content and Organization of Volume

4 This Volume of amended testimony presents Southern California Edison's (SCE) 2021 Test Year

5 forecast of Operations and Maintenance (O&M) expenses and 2019-2023 capital expenditures forecast

6 for the Enterprise Technology Business Planning Element (BPE).1 This BPE includes the activities and

7 infrastructure to support SCE's broader Information Technology (IT) needs which are foundational to

8 the operation of SCE. SCE's Enterprise Technology forecast of $217 million (Constant 2018 dollars) in

9 O&M expenses for Test Year 2021 and capital expenditures of $538 million for 2019-2023 will allow

10 SCE to continue necessary work to manage our increasingly complex technology environment,

11 including over 7,500 midrange servers (UNIX, Linux, and Wintel), over 2,000 terabytes of datastorage,

12 700 miles of data network routing and switching infrastructure inclusive of copper and fiber-optic

13 cabling, 400 appliances to support over 500 large data repository solutions, and operations of SCE's

14 three primary data centers.2 In addition, SCE-06, Volume 1, Pt. 2 of this Exhibit presents SCE's request

15 of $506 million in forecast capitalized software expenditures for 2019-2023. As a result of SCE's

16 reorganization of its 2021 GRC presentation,3 Cybersecurity & Compliance and Grid Services, which

17 were previously presented within the IT Organizational Unit (OU) testimony in SCE's 2018 GRC, are

18 now presented in SCE-04, Volume 3 and SCE-02, Volume 3, respectively, even though organizationally

19 they still reside within IT. SCE's presentation of the remaining IT costs (collectively, Enterprise

20 Technology) are structured consistent with our Plan, Deliver (or build), Run operating model, which

21 SCE implemented in 2017 to help streamline processes and reduce inefficiencies across technology

22 domains. The Plan function includes the identification, prioritization, and design of technology

23 investments to enable SCE to meet customers' evolving needs, address regulatory compliance

24 objectives, and support SCE's operational needs. The Deliver function is the construction and oversight

25 of product development, configuration, and deployment of technology solutions including project

1 This testimony amendment reflects the incorporation of additional IT costs to support the continued operation of legacy systems through 2021, and the realignment of post-CSRP implementation costs with the new CSRP go-live in early 2021. For administrative convenience, this testimony also incorporates SCE's previous errata submitted on November 22, 2019.

2 SCE's three primary data centers include the Alhambra Data Center, Irvine Data Center, and Grid Data Center. Expenditures related to SCE's Grid Data Center are addressed in Exhibit SCE-02, Volume 3.

3 See Exhibit SCE-07, Volume 1, Chapter X.

535

2

1 management. The Run function includes management of software products, application maintenance

2 and enhancements, the operation and replacement of IT infrastructure (hardware), and oversight over the

3 Managed Services Providers (MSPs). The recorded history and forecasts for these activities have been

4 sequenced according to this work flow and are described in detail in the following chapters:

5 Chapter I - Introduction

6 Chapter II - Technology Planning, Design & Support (Plan)

7 Chapter III - Technology Delivery (Deliver)

8 Chapter IV - Service Management Office & Operations(Run)

9 B. Summary of O&M and Capital Request

10 SCE's Enterprise Technology forecast of $217 million in O&M expense for the Test Year 2021

11 represents a $25 million increase over 2018 recorded expenses, which is necessary to support SCE's

12 growing IT portfolio. This includes incremental costs to accommodate:

13 - Maintenance, replacement, and modernization of grid technologies;

14 - Post-CSRP implementation costs, including decommissioning of Customer Service System

15 (CSS) mainframe and non-mainframe applications;

16 - Growth in software license and maintenance agreements; and

17 - Incremental MSP support for major programs such as Grid Modernization and Digital

18 Managed Services.

19 - Growth in cloud-based services and solutions

20 - Enablement of more digital technologies and data analytics to address needs of customers

21 and SCE operations4

22 SCE's Enterprise Technology capital expenditures of $538 million for 2019-2023 include

23 expenditures for technology infrastructure maintenance and replacement that are necessary to support

24 SCE's data center infrastructure and applications, including disaster recovery, server refreshes, storage

25 refreshes, and application upgrades. In addition, the Enterprise Technology capital expenditures provide

26 end-user computing equipment such as desktop computers, laptops, phones, and printing for our

27 approximately 12,500 SCE employees across our 50,000 square miles of service territory. IT capital

28 software expenditures for OU and IT projects are discussed in SCE-06, Volume 1, Pt. 2. This Volume

29 includes SCE's 2019-2023 Enterprise Technology capital expenditure forecast, which reflects:

4 Refer to Exhibit SCE-06, Volume 2, Chapter V testimony entitled "Digital and Process Transformation."

535

4

Figure I-2Enterprise Technology Expenditures 2019-2023

(Total Company - $Million)

535

52

1 In 2023, there is a projected increase of $6.8 million. 52

2 Examples of applications being upgraded to retain vendor supportability include Power Plan and

3 Electronic Document Management and Records Management (eDMRM).

4 C. Technology Infrastructure Maintenance & Replacement

5 Figure IV-12 provides 2014-2018 recorded and 2019-2021 forecast O&M expenses, broken

6 down by labor and non-labor, for the Technology Infrastructure Maintenance & Replacement work

7 activity and its sub-work activities. Table IV-16 provides this information prior to being normalizedfor

8 ratemaking purposes.

Figure IV-12Technology Infrastructure Maintenance & Replacement

O&M Recorded / Forecast53

(Constant $000)

52 See WPSCE-06V01P01 pp. 68 - 74.53 See WPSCE-06V01P01 pp. 75 - 81.

$13,906 $16,714$24,954

$21,411

53

$21,411 $24,954

Table IV-16Technology Infrastructure Maintenance &Replacement

O&M Recorded / Forecast by Sub Work Activity (Constant $000)

1 Figure IV-13 and Table IV-17 provide 2014-2018 recorded and 2019-2023 forecast capital

2 expenditures for the Technology Infrastructure Maintenance & Replacement work activity and its sub-

3 work activities.

Figure IV-13Technology Infrastructure Maintenance & Replacement

Capital Recorded / Forecast(Nominal $000)

$9,979 $10,827

$49,205 $65,328

54

(Nominal $000) $32,826 $47,688

$49,205$65,328

Table IV-17Technology Infrastructure Maintenance & Replacement

Capital Recorded / Forecast

1 1. Overview of Work Activity

2 The Technology Infrastructure Maintenance and Replacement work activity is critical to

3 the support of the business applications and services that allow SCE to safely deliver reliable, clean, and

4 affordable energy for customers. This activity consists of three sub-work activities: (1) Data Center

5 Infrastructure; (2) End User Computing Maintenance, Services & Replacement; and (3) Technology

6 Adoption.

7 Data Center Infrastructure covers the compute, storage, and network infrastructure

8 housed in three SCE enterprise data centers: Alhambra Data Center (ADC), Irvine Operations Center

9 (IOC), and Rancho Customer Contact Center (RCCC). This sub-work activity involves the procuring,

10 installing, and maintenance of all enterprise data center hardware infrastructure. This infrastructure is

11 inclusive of a mainframe platform (primary and disaster recovery), over 7,500 midrange servers (UNIX,

12 Linux, and Wintel), over 2,000 terabytes of data storage, 700 data network routing and switching

13 infrastructure inclusive of copper and fiber-optic cabling, and 400 appliances to support over 500 large

14 data repository solutions.

15 The End User Computing Maintenance, Services & Replacement sub-work activity

16 covers the performance management of SCE's Service Desk and maintenance of devices. SCE's Service

17 Desk handles approximately 17,000 calls per month. SCE has a system to funnel calls into different tiers

18 based on the type and complexity of the issue. The Service Desk also resolves about 204,000 service

19 tickets per year (about 60% are restoration services, while 40% are service requests). SCE employs

20 certain SLAs that govern response times for service requests and service restoration incidents. For

21 example, the Service Desk must respond to 98% of electronic requests within 20 minutes from the time

22 of submission. In addition, End User Computing Maintenance, Services and Replacement includes

23 management of approximately 7,500 smart phone plans, 1,500 tablet cellular data and Apple care, 4,500

55

1 air cards, 1,256 printers, 225 plotters, 16,000 laptops and desktops, 108 teleconference rooms with AV

2 equipment across the company, and 19 Mauell Walls (monitors).

3 Lastly, the Technology Adoption sub-work activity consists of the retirement of

4 computer, storage, network, and operating software assets and the replacement of these assets with

5 hardware and operating software that may be more operationally efficient with improved price

6 performance to leverage new and emerging technologies such as the cloud. As SCE continues its

7 adoption of cloud technologies utilizing the Microsoft Azure Cloud computing platform and services,

8 the focus over the 2019-2023 time frame will be in the following areas:

9 Reliability and Business Resiliency

10 Backup and Recovery

11 Operational Reliability

12 Security

13 Identity Management

14 Overall, for Technology Infrastructure Maintenance and Replacement, SCEforecasts

15 2019-2023 capital expenditures of $340.84 million and test year 2021 O&M expenses of $21.78 million.

16 2. Comparison of Authorized 2018 to Recorded-O&M

17 SCE was authorized $14.0 million in O&M expenses for Technology Infrastructure

18 Maintenance & Replacement in the 2018 GRC. This work activity's recorded 2018 O&M expenses were

19 approximately $26.7 million, which was $12.6 million above authorized.54

20 This spending above authorized was primarily due to two reasons: (1) an accounting

21 methodology change for IT products and services wherein O&M costs that previously were billed to

22 SCE OUs for these products and services are now recording directly to IT O&M, and (2) a change in

23 capitalization rules for hardware maintenance resulting in a transfer of $3.9 million from capital to

24 O&M cost accounting.

25 3. Comparison of Authorized 2018 to Recorded-Capital

26 SCE was authorized $52.4 million in capital expenditures for Technology Infrastructure

27 Maintenance & Replacement in the 2018 GRC. This work activity's recorded 2018 capital expenditures

28 were approximately $52.2 million, which was only $0.2 million below authorized, which is within

29 normal operating expectations.

54 Refer to WP SCE-07 Vol. 1 Authorized to Recorded.

$338.20

60

(Nominal $000)

2014 2015 2016

$774,749

2017 2018 2019 2020 2021 2022 2023

3,221,412 2,132,608 $14,071,1107,000,000

20,723,000 $10,886,06711,047,173

44,124,193 212,914,996

$47,688$32,826

$6,547,164 $17,116,628 $9,112,000 $6,430,000$337,172$7,472,690 $7,189,302

$32,826,488 $47,688,479

Table IV-18Data Center Infrastructure

O&M Labor/Non Labor Recorded / Forecast(Constant $000)

1 Table IV-19 below provides 2014-2018 recorded and 2019-2023 forecast

2 capital expenditures for the Data Center Infrastructure sub-work activity and further breaks these

3 expenditures down into the categories discussed in the Work Description section above.

Table IV-19Data Center Infrastructure Capital Recorded / Forecast

Mainframe Replacement 1,943,628 6,775,287 578,085 Server Replacement 18,680,699 9,141,670 8,826,355 8,771,722 12,338,495 17,217,758 9,200,000 5,021,695 3,862,711 Storage Replacemet 15,438,946 7,633,637 3,685,352 15,719,934 979,184 1,010,604 6,500,000 3,316,295 2,584,419 Data Center Network Replacement 4,851,257 1,630,018 7,707,452 9,634,957 7,223,944 7,235,990 8,000,000 7,087,500 8,662,500 Appliance Replacement 14,461,250 30,169,575 10,890,675 Organic Growth 681,623 2,417,177 1,646,801 15,417,471 9,370,370 7,955,679 10,996,067 6,372,000 7,668,000

Subtotals 39,652,525 20,822,502 23,809,588 49,544,084 36,687,280 33,998,116 49,157,317 51,967,065 33,668,305 Totals 170,515,979

4 (a) Historical Variance Analysis

5 (i) Labor

6 From 2014-2015, there was a reduction in workforce that

7 resulted in a decline in labor costs from $19.1 million to $4.6 million as SCE made the transition to

8 MSPs. The SCE workforce for the Data Center Infrastructure sub-work activity then continued to

9 decline from 2015 through 2018 due to subsequent re-organizations of SCE IT.

$5,811 $6,115

$9,979 $10,827

64

1 and/or network equipment was added to address

2 organic growth, a proportional or, in some cases, an

3 exact matching of infrastructure was provisioned and

4 added to the appropriate disaster recovery data center

5 location.

6 (b) Basis of Forecast

7 Table IV-20 provides 2014-2018 recorded and 2019-2023 forecast

8 O&M expenses, broken down by labor and non-labor for the Data Center Infrastructure sub work

9 activity.

Table IV-20Data Center Infrastructure O&M

O&M Labor/Non Labor Recorded / Forecast(Constant $000)

10 Table IV-21 provides 2014-2018 recorded and 2019-2023 forecast

11 capital expenditures for the Data Center Infrastructure sub-work activity and further breaks these

12 expenditures down into the categories discussed in the Work Description section above.

$9,979 $5,811 $6,115 $10,827

65

t

$14,071,110

$10,886,067

$32,826,488

Table IV-21Data Center Infrastructure Capital Recorded / Forecast

(Nominal $000)

$7,472,690 $337,172

2014

$6,547,164

2015

$774,749

2016

$17,116,62

2017

8

2018

$9,112,000

2019 2020

6,43

2021

0,000

2022

$7,189,302

2023

Mainframe Replacement 1,943,628 6,775,287 578,085 Server Replacement 18,680,699 9,141,670 8,826,355 8,771,722 12,338,495 17,217,758 9,200,000 5,021,695 3,862,711 3,221,412 Storage Replacemet 15,438,946 7,633,637 3,685,352 15,719,934 979,184 1,010,604 6,500,000 3,316,295 2,584,419 2,132,608 Data Center Network Replacemen 4,851,257 1,630,018 7,707,452 9,634,957 7,223,944 7,235,990 8,000,000 7,087,500 8,662,500 7,000,000 Appliance Replacement 14,461,250 30,169,575 10,890,675 20,723,000 Organic Growth 681,623 2,417,177 1,646,801 15,417,471 9,370,370 7,955,679 10,996,067 6,372,000 7,668,000 11,047,173

Subtotals 39,652,525 20,822,502 23,809,588 49,544,084 36,687,280 33,998,116 49,157,317 51,967,065 33,668,305 44,124,193 Totals 170,515,979 212,914,996

1 (i) Labor

2 The 2021 test year labor forecast for this sub-work activity

3 is $5.0 million. As shown in Table IV-20, SCE's 2016-2018 recorded expenses have trended downward.

4 Therefore, SCE utilizes the last recorded year estimating methodology to form the basis of our test year

5 forecast. This is also consistent with Commission guidance to use the last recorded year when historical

6 recorded costs exhibit a downward trend or are relatively stable for three or more years. From the 2018

7 recorded labor amount of $3.5 million, SCE forecasts an increase of $1.5 million, which results in a Test

8 Year 2021 forecast of $5.0 million.

9 This forecast increase over last year recorded is primarily

10 due to the increased staff needed to provide cybersecurity operational support. As outlined in the

11 Cybersecurity Testimony, there will be an increase in Data Center Infrastructure staff required to

12 provide operational support for programs such as Perimeter Defense, Interior Defense, and Data

13 Protection. This is reflected in the incremental labor increase forecast from 2018-2021, with staffing

14 requirements normalized in 2022-2023. The incremental budget request of $1.5 million over the 2019-

15 2023 timeframe will be inclusive of $1.0 million for SCE staff to support Cybersecurity program

16 operational requirements. The remaining $0.4 million is due to CSRP resources returning to SMOO after

17 CSRP stabilization and additional service manager vacancy.59

59 See WPSCE-06V01P01 pp. 89 - 91.

$32,826 $47,688

$47,688,479

66

1 (ii) Non-Labor

2 The 2021 test year non-labor forecast for this sub-work

3 activity is $ 5.260 million. The actual spend for 2014-2018 was $49.4 million whereas the forecast spend

4 for 2019-2023 is $25.3 million. In 2014-2018, SCE implemented several capitalized hardware purchases

5 with prepaid maintenance agreements that will end during 2019-2023.61 Unless this hardware is

6 refreshed on the standard five-year refresh cycle, O&M costs are incurred. The forecast period reflects

7 relatively flat expenditures year-over-year to cover the maintenance costs for IT hardware infrastructure

8 which has been extended beyond its standard five-year refresh life cycle. The forecast reflected in this

9 GRC period is derived from an itemized list of hardware assets that go beyond the five-year refresh life

10 cycle.62

11 (iii) Capital Expenditures

12 The forecast capital expenditures for 2019-2023 are

13 $212.91 million. Forecast expenditures for 2019 are flat compared to 2018, with significant increases in

14 appliance and storage replacement expenditures forecast in 2020 and additional increases in appliance

15 replacement expenditures forecast in 2021 and 2023. See below for the basis of this forecast broken

16 down into the Data Center Infrastructure categories described in the work activities section:

17 Mainframe Replacement: The only forecast capital

18 expenditures in the 2019-2023 time frame are for a disk

19 library for the mainframe in 2019. With implementation

20 of the CSRP program, the mainframe environment will

21 be decommissioned beginning in 2021, with much of

22 the decommissioning work occurring in years 2022-

23 2023. Description of the decommissioning activities are

24 summarized in the Application Refresh63 section of the

60 An additional $250,000 for one-year extension of application server maintenance is reflected in the forecast to align with the CSRP implementation in early 2021.

61 Refer to SCE 07 Vol. 2 for accounting practices.

62 Note that pre-paid hardware maintenance for 2021 will be considered capital, similar to what is reflected inthe 2018 GRC testimony. Similar to capitalized software projects, in which SCE capitalizes prepaid license and maintenance

agreements and amortizes them over the life of the software asset, SCE will also capitalizes and amortizes prepaid hardware maintenance agreements for regulatory recovery if they meet the capitalization threshold. SCE follows this recovery practice when the maintenance agreements coincide with the associated hardware asset life.

63 Refer to Section IV. B. 3. B section Application Refresh Non-Labor (O&M) for references and work papers related to Customer Service Applications Decommissioning.

$210.27

67

$38.3

1 testimony. For more detail on these calculations, refer

2 to the work papers for this section.64

3 Server Replacement: Our forecast expenditures of

4 $38.5 million for 2019-2023 include costs for

5 acquiring, building, configuring, and implementing

6 midrange servers to replace existing servers at the end

7 of their useful life. There is an increase in 2019 funding

8 for the investment in a software-based hardware

9 virtualization technology, Hyper Converged

10 Infrastructure (HCI), and for the replacement of the

11 current Flexpod technology (hardware-based

12 component integration). As mentioned previously, SCE

13 expects improved operational efficiencies with HCI

14 and, in order to better track implementation of this new

15 technology, has created the Appliance Replacement

16 sub-work activity to capture forecast expenditures

17 going forward from 2020. This transition aligns with

18 our cloud migration strategy, which will in turn

19 improve reliability and availability through innovation.

20 The expenditures in the forecast period will replace

21 midrange servers and associated converged

22 infrastructure appliances for mission-critical systems

23 such as SAP Enterprise Resource Planning, T&D

24 applications, and Outage Management System. SCE

25 developed this midrange server forecast through a

26 detailed analysis of existing midrange server assets,

27 their useful lives, and the expected midrange server

28 needs in this GRC period. Once SCE quantified the

29 need (number of units) for each type of midrange server

64 See WPSCE-06V01P01 pp. 92 - 97.

68

$15.5$15.2

1 equipment category, SCE applied a unit cost to derive

2 the total forecast for that equipment category. These

3 unit costs were based on actual vendor quotes or an

4 extrapolation of historical cost data for each equipment

5 category.65

6 Storage Replacement: Our forecast expenditures of

7 million for 2019-2023 include costs for

8 acquiring, building, configuring, and testing storage

9 arrays to replace existing systems at the end of their

10 useful life. Expenditures in the forecast period will

11 replace storage devices and associated infrastructure

12 appliances for mission-critical systems such as Edison

13 SmartConnect®, Outage Management System,and

14 Customer Service System (CSS). SCE developed this

15 storage forecast through a detailed analysis of existing

16 storage assets, their useful lives, and the expected data

17 growth needs in this GRC period. Once SCE quantified

18 the need (number of units) for each type of storage

19 equipment category, SCE applied a unit cost to derive

20 the total forecast for that equipment category. These

21 unit costs were based on actual vendor quotes or an

22 extrapolation of historical cost data for each equipment

23 category.66 The forecast accounts for the future

24 migration from the traditional storage architecture to

25 the Hyper Converged Infrastructure (HCI), which will

26 drive down storage costs over time.

27 Data Center Network Replacement: The forecast

28 expenditures for 2019-2023, totaling $38.0 million, are

65 Id.66 Id.

$36.5

70

1 (NetApp). SCE developed this appliance replacement

2 forecast through a detailed analysis of existing assets,

3 their useful lives, and the expected needs in this GRC

4 period. Higher spend is expected for 2021 as the

5 majority of the SAP solutions are due for a refresh with

6 newer HANA backend hardware. Because SAP

7 solutions are integrated, the SAP landscape (multiple

8 applications) is required to be refreshed together at the

9 same time resulting in the increase in expenditures. In

10 2022 and 2023, there is an increased expenditure on

11 Exadata supporting our Meter Data Management

12 System (MDMS), Outage Management System (OMS),

13 and Comprehensive Geographic Information System

14 (cGIS). Once SCE quantified the need (number of

15 units) for each type of appliance category, SCE applied

16 a unit cost to derive the total forecast for that equipment

17 category. These unit costs were based on actual vendor

18 quotes or an extrapolation of historical cost data for

19 each equipment category.68

20 Organic Growth: Our forecast expenditures of $44.0

21 million for 2019-2023 include costs for acquiring,

22 building, configuring, and testing midrange servers,

23 appliances, and datacenter network components

24 required to meet the organic growth. Due to the nature

25 of these appliance models, such as HANA, Big Data

26 Appliance (BDA), and Exadata, capacities are now

27 procured in pre-configured increments and therefore

28 organic growth hardware outlays are increased.

29 Consequently, the increases for HANA and Oracle

68 See WPSCE-06V01P01 pp. 103 - 105.

$43.5