Enabling software defined networking for electric power utilities

Introduction

To effectively manage the increasingly dynamic power grid, an equally dynamic and agile approach in architecting power utility communication networks is required.
Communication networks are a crucial backbone of power utilities, and are used to provide fundamental capabilities to support applications such as protection, SCADA, engineering access, field maintenance and asset management. For power utilities, the communication networks are required to be resilient against failures and increasingly from cyber-attacks and threats.
The CIGRE Study Committee D2 Working Group D2.43 examines the opportunities that SDN and NFV provide to power utilities in the above context. The result is a Technical Brochure, titled Enabling Software Defined Network for Electric Power Utilities.
The Technical Brochure provides an overview of SDN and NFV from its technological evolution, and provides a description of the building blocks of SDN and NFV, in the context of power utilities.

Utility Use Cases

The use cases examined by the Working Group are as follows:

Substation Virtualisation

Due to the increasingly dynamic nature of the power grid, the information and data exchange no longer follow the traditional hub-spoke pattern commonly encountered in the conventional utility applications (for example, SCADA typically follows the hub-to-spoke pattern, where the remote terminal units in remote sites communicate with one or two SCADA control centres in the utility’s network).

With distributed energy resources (DER) and the use of high-speed monitoring applications (for example, PMUs), there is an increasing requirement for utilities to equip their remote sites and substations with information processing capabilities. Figure 1 shows a use case where NFV and SDN can be deployed as an integral part of the intelligent edge. Implemented alongside the virtualisation technology with ruggedised hardware, NFV and SDN have the potential of realising the intelligent utility edge.

Other aspects, such as recommendations in integration with the utility’s existing telecommunication networks (SDN/SONET/MPLS) are discussed in the Technical Brochure.

Network Modelling and OT Network Digital Twin

The networks and applications in power utilities are becoming more efficient, automated and intelligent, and will become increasingly so as the industry is under pressure to modernise its networks and applications, to support the transition towards renewables, and to meet the cybersecurity threats and challenges – all these factors contribute to additional applications and solutions and their underlying networks, many of which are complex.

To maintain reliability and resilience, especially in the utility’s operational technology (OT) networks, power utilities would benefit by having an OT network digital twin to model the state of the network.

NFV and SDN can be used to provide a virtualised network model to represent the state (or a subset) of the production network. The benefits of having a representation of the OT network in a self-contained and non-production environment are as follows:

1. Efficiency and rapid setup - Faster setup of the digital twin due to the use of virtual components, all of which are software components run within hypervisor machines. 
2. Ease of simulation - Being implemented within a virtualisation environment enables automation and orchestration tools to quickly change the simulation scenario to validate proposed changes to the system. For example, if IPSEC encryption is required on the DNP3 communication between the RTU and the master station, configuration changes can be designed and tested in the digital twin prior to updating the physical components. Multiple iterations of tests can be done more easily within the digital twin due common toolset in the digital twin environment (for example, the ability to quickly rollback changes).
3. Training environment - The digital twin is a functional replica of the physical system, and hence provides a useful training environment.

The discussions around which OT components can be virtualised (for example, MPLS routers, firewalls, switches, IEDs, RTUs) are further discussed in the Technical Brochure.

Micro-segmentation

The concept of micro-segmentation is based on the notion that every host on an Ethernet or IP network should be segmented - this concept relates closely to the Zero Trust cyber security model, where hosts should not be implicitly trusted.

In many OT networks today, the edge of the network (for example the border between the OT network and the Corporate network) provide stronger security controls than the core within the OT network.

Micro-segmentation relates closely to the Zero Trust cyber security model, where hosts and devices should not be implicitly trusted, even when they reside within the same internal OT network. This improves security, but should be carefully designed and implemented to prevent reducing the availability of the OT network.

Figure 2 and Figure 3 show the contrast between a non-micro-segemented substation Ethernet network and a micro-segmented substation Ethernet network using SDN.

It should be noted that further work needs to be carried out to assess OT-SDN issues, such as the potential impact on the availability and operation on the OT network when the SDN controller is offline.

OT Cloud Service Integration

Public cloud and advanced cyber security capabilities built into the cloud are becoming the cornerstones of the future utility infrastructure, and an increasing proportion of power utilities will rely on the cloud services to meet the needs of the customers and infrastructures.

Figure 4 shows the overview of the integration, and is further discussed in the Technical Brochure.

IEC 61850

A novel use case for SDN is in pre-configuring IEC 61850 switches.

Conventional Ethernet switches communicate using the flood-and-learn mechanism, and are based on the notion that all ports in the same segment are trusted by default. In contrast, SDN switches differ from conventional Ethernet switches in that SDN switches can be configured to block all communication by default, without the usual Ethernet flood-and-learn process enabled. Instead the required communication between ports on an SDN switch can be pre-programmed.

Since the IEC 61850 environment is a statically defined environment, i.e. the substation automation system communication between IEDs are well defined, SDN can be used to pre-define the flows required to communicate between devices.

The result is that the configuration of the IEC 61850 LAN becomes a fully automated and tightly-coupled process by programming a rigid and secure set of behaviour on the switches based on current IEC 61850 provisioning processes.

The Technical Brochure describes some example scenarios, including setting up logical signal exchange via Generic Object-Oriented Substation Events (GOOSE) and Sample Value (SV) messages, and proposes that the pre-configuration required on the SDN switches can be achieved as part of the Substation Configuration Language (SCL) definitions.

Survey Results

A survey was undertaken in 2018 with the CIGRE representatives from member countries as respondents. Some results are as shown in the graphs below:

Most respondents indicated that SDN and NFV are relevant or somewhat relevant to their organisation, as shown in Figure 5.

Case Study

A Japanese electric power company use case on SDN is described in the Technical Brochure, where the network provisioning using SDN improves the efficiency of workflows in the network operations team. Further details are described in the Technical Brochure.

Future Work

There is significant potential for 5G to improve the operations for power utilities.

5G uses the concepts of SDN and NFV extensively. We propose for future studies to include the how SDN and NFV play their role in meeting the requirements such as the architecture, implementations, cyber security issues, and other challenges of 5G. A specific area that may be worth venturing into is the area of private 5G networks provisioned by the power utility to support its mission critical applications such as protection and SCADA, and how a private utility 5G solution with SDN and NFV components can successfully interoperate and be integrated into the utility's existing telecommunication networks and information systems infrastructure.

Another area of interest is the concept of the intelligent edge of the power utility. Due to the demands of renewables and DER, with the significant increase of telemetry and flow of information from various points in the power network, the intelligent substation or intelligent remote sites and asset may not stay conceptual, as utilities increasingly require methods in providing services and intelligent processing capabilities at various points of the distributed network - for a transmission utility, these may be the transmission-level substations and communication sites; for a distribution utility, these may be street-level cabinets or power poles distributed throughout a vast geographical area. We believe SDN and NFV, along with virtualisation, would provide the technology required to meet the demands of the intelligent power utility edge. 

The use cases of extending the utility's OT networks into the cloud, forming a hybrid environment, may also be an area of interest. This is due to the ever-converging IT and OT environments and systems within the utility, and the requirements for utilities to scale services in an agile method.

Finally, deterministic network technologies based on Time-Sensitive Networking (TSN) and Detnet provide the capabilities to meet essential requirements for protection, automation and control within a substation environment and over the WAN (for example, inter-substation communication). Time-Sensitive Networking is a set of standards specified by the IEEE 802. DetNet is being developed by the IETF in the Deterministic Networking (DetNet) Working Group. A stringent controller concept allows flexibility, programmability, and automation, and it would be of interest to investigate SDN’s applicability to TSN in the context of power utility applications.