Los Angeles County Sanitation Districts (LACSD) are a public agency that converts waste into resources, such as recycled water and energy, for over 5 million people across 24 independent special districts in Los Angeles County. The Sanitation Districts build, operate, and maintain facilities for the collection, treatment, recycling, and disposal of wastewater and industrial waste.
LACSD’s Warren Facility is one of the largest wastewater treatment plants in the world, providing primary and secondary treatment for 260 MGD daily. The Warren Facility works with the Joint Outfall System (JOS) network of treatment plants and trunk sewers, providing centralized processing of solids removed during wastewater treatments for all JOS plants and producing electricity and reusable biosolids in the process. This facility’s digester gas fired turbines and steam turbine generate 21-22 megawatts of electricity for on-site use, making the site entirely self-reliant with respect to power generation and saving them $20 million in annual electricity purchases.
FactoryTalk View SE version 7 facilitated the energy production of the Warren Facility through the control and monitoring of systems related to energy generation. The Warren Facility’s FactoryTalk version 7 was nearing its end of life, which could threaten the operational integrity of the SCADA system. Although the system could technically continue to operate, this would be inadvisable due to the security risks associated with the end of vendor support and updates. They contracted Enterprise Automation to migrate the SCADA software from version 7 to version 12 of FactoryTalk before the version 7 support was discontinued. Our goal was to migrate FactoryTalk while keeping all functionality and minimizing disruption to ongoing operations.
Energy production at the Warren Facility became more reliable through the elimination of security vulnerabilities associated with using outdated software and by upgrading the hardware and software.
Although a “software upgrade” can sometimes be a simple overnight task, this upgrade process was not straightforward. The Warren Facility machines ran FactoryTalk version 7 on Windows 7 while FactoryTalk version 12 can only run on Windows 10 operating system. These two versions were wholly incompatible with each other. This necessitated an intermediate upgrade from FactoryTalk version 7 to version 10, which could also run on Windows 7. After transferring data to FactoryTalk version 10, we were able to upgrade the OS to Windows 10 and as a result transition FactoryTalk 10 to version 12.
We worked closely with our partner Rockwell Automation to develop this upgrade procedure, which included conducting trial upgrades to identify potential issues and ensure that the process was well-documented. Prior to the actual deployment, we tested the upgrade process in a controlled environment to identify and address any gaps. Our test platform was essentially a digital twin of the Warren Facility’s SCADA system where we verified that all configurations could successfully migrate to the new system and that the upgraded system would function as intended.
Project implementation involved several stages. We first set up the new HP servers and client workstations. This included configuring the servers to run VMware ESXi as the hypervisor, which allowed for the virtualization of the HMI servers. We also selected VMware vCenter as the platform to manage the ESXi hosts and virtual machines. One of the key benefits of virtualization is redundancy, which minimizes the risk of downtime by running the server’s replicated backup during an incident of failure. Following the hardware configuration was the installation of necessary software, including FactoryTalk, on the new VMs. We timed the installation to coincide with the joining of the servers and clients to LACSD’s domain and ensure proper integration into the existing network.
One week prior to deployment, LACSD provided us with a final backup of the existing FactoryTalk configuration. We executed the upgrade procedure, restored backups onto the new HMI server hardware, and verified the success of the upgrade using tools included with the Rockwell Automation software.
After the upgrade, we conducted a site acceptance test (SAT) with the LACSD operators to ensure that the system functioned as expected. This included checking the graphics and confirming that the end-user experience from version 7 to version 12 remained unchanged.
This project is part of an ongoing collaboration with LACSD. Given the typical lifecycle of software and hardware, and the continued evolution of LACSD’s operations, EA is positioned to remain a partner in their technological advancements.
Enterprise Automation
9050 Irvine Center Drive, Ste 200
Irvine, CA 92618
Phone 949.769.6000
Fax 949.769.6005
