Rapid scaling
Customize and adapt for “last-mile” implementation
Build capabilities
Design for the operator experience
Create machine learning operations (MLOps) infrastructure
Turn each successful proof of concept (or MVP) into a product
Vistra implemented an MLOps approach to create a “factory” that standardized the deployment and maintenance of more than 400 AI models. This enabled the team to bring live data from each of Vistra’s power units into a single database; use Gitlab software to manage version control for code; containerize the code so it could be easily deployed to any environment; set up a scheduler (using AirFlow) to make sure recommendations and actions were delivered on time; create dashboards to monitor model performance and usage; and manage the continuous improvement of each model to make sure the plants were sustaining the value they captured. Teams also incorporated multiple approaches to reduce risk by building functional limits (e.g. maximum throttle pressure or heat levels) into the code and putting all code through biweekly peer reviews and multi-week testing. McKinsey risk dynamics experts worked with the team to test assumptions, review code, and ensure that all risk best practices were reflected in the models.
Create machine learning operations (MLOps) infrastructure
Next
Back
Next
Back
Machine Learning Operations (MLOps) is the set of practices and infrastructure to manage the production and deployment of analytics solutions or products. Improvements in AI tooling and technologies have dramatically transformed AI workflows, expediting the AI application life cycle and enabling consistent and reliable scaling of AI across business domains. This framework can underpin a company AI “factory” to achieve scale.
What is MLOps?
400+
Turn each successful proof of concept (or MVP) into a product
AI models developed
When a solution has proven value at a pilot site and is approved for scaling, a team of software and machine learning engineers immediately takes over to refactor, modularize, and containerize the code.
That way there is a single software ‘core’ package for each deployment that can be updated and improved. A product owner manages the overall process and takes ownership for use and adoption.
Over time, the team developed seven solutions archetypes, which provided consistent approaches, logic, assumptions, and algorithmic elements as a basis for each new application being developed. This gave each new solution a big head start when development began. It took 10-12 weeks to build the first HRO. Rolling the HRO out to each new plant now takes just 2-3 weeks.
50-70
Customize and adapt for “last-mile” implementation
percent of each product is reusable
Because each product is designed to be modular and reusable, 50 to 70 percent of each is ready to be used from day one. But customization is always needed for each plant’s characteristics. Take the maintenance solution we developed to understand the best time to replace inlet filters at gas turbines (each filter costs $150K, and a plant needs to be shut down for two days to replace one).
At the Moss Landing Power Plant in California,
the unit that faces the ocean had to deal with more moisture and salt than the one that faces inland,
so the degradation profile for each filter is different. So dedicated customization teams made up of data scientists, engineers, operators, and power generation experts worked with each power plant to tailor the solution to the unique conditions of that particular plant.
Build capabilities
Building on Vistra’s culture of continuous improvement, McKinsey worked to help Vistra to get the most from AI. That included a program for operators to explain the models and how they were developed, as well as how to use the tool itself.
As Rachit Gupta, Vistra Energy’s vice president
for generation and wholesale applications, said: “People had to know what the model was doing and learn to trust that it was right. Once they saw that the models were generating recommendations that made sense and helping lower heat rate, they were ready to start using them more.” Additional training for the core tech team covered how to build and maintain models, resolve issues, and deeper machine learning and analytics skills
to understand how models work in detail. This happened by working alongside McKinsey
AI experts. Vistra has also partnered with the University of Texas to offer a range of analytics courses for employees.
Once they saw that the models were generating recommendations
that made sense and helping lower heat rate, they were ready to start using them more.”
“
—Rachit Gupta,
Vistra VP for generation and wholesale
Design for the operator experience
From the beginning of the solution development process, McKinsey designers worked with operators to understand their day-to-day activities. What soon became clear was that plant operators had constraints on their time and had to manage dozens of inputs that were tracked on an array
of screens in the control room. Adding to that workload would overwhelm operators and reduce the solutions’ effectiveness. The tools had to make operators’ lives easier.
For this reason, the screen that displayed the AI solutions and recommendations were integrated into PowerSuite, an interface that operators already used so they didn’t need to monitor another screen. The displays themselves were designed to be easy to read. A solution displays
a green signal if the plant is running optimally for the given conditions, and red when it isn’t. A brief recommended action accompanies any red display, with the value attached to implementing that recommendation.
—Lloyd Hughes,
Vistra operations manager
There’s a simple screen that not only shows you what needs to be optimized but how much it’s worth. That really registers with people.”
“
Design for the operator experience
Build capabilities
Customize and adapt for “last-mile” implementation
Create machine learning operations (MLOps) infrastructure
Turn each successful proof of concept (or MVP) into a product