SCADA In The Data Center

Bringing IT and SCADA in Sync

Historically Human Machine Interface (HMI) and Supervisory Control and Data Acquisition (SCADA) systems were not included in the scope of the charter for Information Technology (IT). Process engineering or the plant maintenance organization were normally tasked with the responsibility for the SCADA/HMI systems. The responsibility included the selection and management of the computing platform and the industrial control network (ICN). In the many cases, a lone electrician was responsible for the entire system, including the workstation, the ICN and the Programmable Logic Controller (PLC). In this era of SCADA/HMI the ICN was rarely interconnected with the IT enterprise network or the Internet. Remote access was typically done using phone modems connected directly to the SCADA host workstation.

Bringing IT and SCADA in Sync

Two trends in the second half of the 1990’s brought IT and SCADA closer together. The first was the so-called “Y2K scare” that affected all enterprise computing systems. This included computing systems found in production management software such as SCADA and HMI. IT departments included production system in the scope of their audits, looking at production systems to ensure that they were ready to handle the new century date formats.

This alone, did not bring about the integration of production systems with IT architecture. For the most part these systems continued to not be connected to the rest of the corporate IT networks. Given the “air gap” security combined with the unique requirements for managing traffic on the ICN, centralized IT network management tools simply could not be effectively used.

A second trend that coincided with the Y2K focus was the deployment of Enterprise Resource Management (ERP) systems in many organizations. The data demands of the ERP and its uncertain connection to the shop floor meant that either a parallel system had to be deployed or SCADA would become the bridge. As a result, SCADA and other production system like Manufacturing Execution Systems (MES) were used for tracking production and feeding ERP’s shop floor data appetite by connecting to both IT networks and Industrial control networks.

During the past 15 years we have seen continued integration of IT and production systems. SCADA systems are now for the most part connected to the enterprise networks. Remote access is usually provided via VPNs so the SCADA system can be reached from anywhere on the Internet.

There has been some organizational friction in bringing these two very different cultures together. At the top level IT generally looks at their systems as dynamic and have a focus on scalability, performance and cybersecurity. They may be more willing to apply patches and updates, as long as they know they have the ability to roll back to previous configurations if problems occur. The focus of the industrial network is to deliver extremely high reliability and safety. Due to the critical nature of the real-time data, even short term disruptions can impact production rates and the safety of the workers and equipment that rely on it. For this reason, software patches and firmware upgrades are more thoroughly tested before they are applied to production systems.

SCADA’s Place in the Control Center vs. Data Center

Today we have a world in which the systems are more and more under the protection and management of IT and IT practices. This is particularly true for cyber security and virtualization, which is becoming widely adopted. This leads to the question of whether the SCADA host computer should be removed from the control room or the shop floor and moved into the data center.

We were recently asked to provide guidance to a medium-sized organization that was considering this issue. They have a large warehouse, housing materials that must be controlled to strict environmental conditions. In order to automatically maintain the integrity of the environment of the warehouse, they purchased a system from a vendor that uses our SCADA platform for the HMI. The vendor delivered the HMI on a single user workstation connected with an industrial network to several PLCs in the warehouse that controlled the heating and cooling equipment along with other environmental control equipment. The workstation was physically located in a desk located in the warehouse.

Management was concerned that the workstation was exposed to physical damage from normal warehouse material handling activities such as forklift movements. They were also concerned about the time required to respond to either an accident of this sort or even normal maintenance issues with the workstation, given its physical location. For these reasons, the company wanted to move the workstation from the warehouse floor to the enterprise data center.

Newer Protocols Open Up Possibilities

This was possible because the PLC communication was an Internet Protocol (IP), which have generally replaced the older serial protocols in modern ICNs. The IP protocol created the possibility to have a virtual private network (VPN) established from the data center to allow the HMI to connect to the PLCs on the ICN.

In order to provide the warehouse operator access to the Graphical User Interface (GUI) of the HMI, the IT department preferred to use Microsoft Remote Desktop Applications (RDA). RDA is a subset of the widely used Remote Desktop Services (RDS). RDA allows the HMI GUI to appear on the operator’s normal desktop as an icon without having the RDS requirement to overlay the entire desktop. As a standard Microsoft solution, the use of RDS or RDA is transparent Microsoft compliant SCADA software platforms.

The IT department also wanted to host the HMI workstation on a virtual machine (VM) in the data center rather than have it on a dedicated workstation, as was the case when it was in the warehouse. Once again, with a standard Microsoft application, running on a VM is transparent, as designed. The only issue that was out of the ordinary for IT was the need to map a USB port to the host. Use of a USB dongle for licensing is a common practice found in SCADA and HMI platforms, but less common for enterprise software. In this case, it was a simply a mapping of the USB port to the VM hosting the SCADA. A network USB device may be required in more complicated multi-station networks.

Successful Outcome in the Data Center – for Operations, Warehouse and Facilities Managers

The move from the warehouse floor to the data center was judged a success from many perspectives. Some of the many benefits include increased transparency of the technology, tightened cybersecurity and improvement in system reliability including major reduction to disaster recovery estimates. Both IT and the warehouse management organization judged the performance to be equal to what they had when the workstation was located on-site.

The program had another effect. The facilities maintenance team realized they could improve their performance if they had ready access to the environmental monitoring information. Subsequently to roll-out of the virtualized workstation, the maintenance department requested separate and unique access to the SCADA/HMI system.

The project had originally been licensed and configured as standalone system supporting a single user at a time. A new user profile for maintenance could be set up under the existing license and the RDA connection could be shared with the maintenance organization. With the new architecture, anyone using the system installs RDA. After it is installed, it is simply a matter of opening the RDA icon on their desktop, logging into their session and they are in a window that behaves exactly as it did on the dedicated workstation. The downside was when the operations were logged in, the maintenance people would get a “connection refused error” when they tried to log in and vice versa when maintenance was logged in, operations could not access the GUI.

In order to provide for multiple users to access a standalone HMI, it simply needed to be reconfigured to a SCADA system. Some modifications were required to the license and the configuration of the SCADA project. A standalone HMI license can be converted to a multi-user client server SCADA license in most systems with various degrees of difficulty. In this case, it was simply a matter of the exchange of a license file.

There are also configuration changes required to reflect the new users in the system. For example, the configuration to ensure that each user is sharing the same communication channel to the PLC and not impacting the overall level of traffic on the ICN with parallel communications requests.

With these architectural and configuration changes in place, it was now possible to expand the number of concurrent users. The customer considered how many users would need to access the system simultaneously. In this case the answer came back as three. The on-shift operator, the maintenance manager and the plant manager. There are multiple people in the operations and maintenance department, but for sizing the platform, it was decided that it would support a maximum of three users simultaneously.

Virtual Machines

Virtual machines were provisioned for the additional client workstations supporting the architecture. The VM hosting the SCADA server software was also designated as the license manager. The network license enables the client station VMs to be created on-demand rather than consuming resources during idle times. Client access licenses were acquired from Microsoft for the new RDA sessions required for this project.

The system is now a multiuser system residing in the IT department’s data center. There have been no issues from the warehouse operations end users or the new maintenance department and management end users about performance or accessibility of the system.

Bearing in mind that the environmental controls that are being monitored and applied here do not change rapidly, the migration to the data center was well advised. If any additional latency was introduced by the more complex ICN routing, it was not noticeable. More importantly, the risk to losing the SCADA server from physical damage from a forklift strike was eliminated.

Disaster Recovery Plan

The IT department did call one last time when they were reviewing their overall Disaster Recovery plans. Protection and recovery of historical alarm and event database information was discussed, which they maintain in a Microsoft SQL Server database. Second, consideration was given as to how to best protect the project configuration directory and the platform installation directory. Finally, a review was done of the steps to put it all together into a clear plan of action in the case of an unexpected loss of the server hosting the virtual machines or even the data center itself.

The interesting part of this disaster recovery discussion is that it was done entirely within the comfort zone of the IT professionals that I was talking to. They are used to thinking about minimizing the time required for recovering servers and databases. They know they have to plan to achieve minimal downtime and they know the value of that downtime.

The acceptance of SCADA and HMI in the data center has already occurred. As virtualization becomes the standard deployment of computing in an enterprise setting, we will see continued growth of the percentage of customers choosing this approach. Cybersecurity concerns will also push the SCADA center into the safety of the data center. For most SCADA and many HMI applications of the future, this will likely be the standard deployment strategy.

Ed Nugent is COO of PcVue.