Project overview

A large energy company contacted Kforce Consulting Solutions for help in designing and building data streams to monitor and administer natural gas pipelines. The client wanted to improve their ability to predict demand, direct their supply and foresee outages and other problems. Before creating the proper AI solution, they first needed to establish best practices in data quality and governance.

Business case

Implementation

• Real-time statistical analysis and anomaly detection for data quality assurance • Python modules in Sagemaker (XG Boost and Gradient Boosted Trees) to predict demand and station pressures in distribution • Real-time data ingestion (Kafka/SparkFlow) from sensor data combined into 5-minute batch intervals for dashboarding/reporting • Continuous model evaluation, retraining using AWS technologies

to screen and onboard a team of ml engineers & aws full stack developers

From strategy to implementation, we provide the knowledge and leadership our clients rely on to accelerate their business. Our proven team takes a unified approach to driving large-scale change and unlocking new opportunities for growth and success.

This Ruler is for the Space Between the Bottom of the Hero/Header Text and the Subheader Copy. Check Your Spacing for Consistency.

This Ruler is for the Space Between Text/Buttons/Callouts. Check Your Spacing for Consistency.

High-level architecture

28 days

Energy industry: Using AI to build an autonomous pipeline

25%

reduction in computational power required to process the data pipeline etl daily

20%

greater accuracy in demand predictions

Discover the power of partnership

INSTITUTING DATA QUALITY AND GOVERNANCE PRACTICES TO IMPROVE STATISTICAL METHODS IN THE MACHINE LEARNING PROCESS

Foundational data work included: • Implementing anomaly detection for sensor data to prevent data artifacts from impacting prediction accuracy • Instituting data normalization and other data quality protocols for the AI/ML data pipeline • Creating source and transformation documentation standards for the data pipeline • Automating and standardizing all data governance and quality processes for data related to predictive models • Creating and documenting data formatting and ingestion protocols for the predictive models Ultimately, this AI solution operated on batched data from more than thirty sensors. The predictive model enabled client stakeholders to see usage, supply predictions and assess outage risks in near real-time.

This Ruler is for the Space Between Modules/Macros/Sections. It Should Reach from the Bottom of the Previous Module/Macro to the Top of the Next Section.

Explore more stories

Revolutionizing team performance: Kforce's AI chatbot empowers team performance with dynamic real-time sales insights

Whitepaper: Preparing your organization for generative AI

What AI solution is best for your company?

• Establish data quality, architecture and governance to build effective machine learning data dreams • Use machine learning models to gain insight into sensor data • Leverage data to generate reliable predictive analytics • Provide the client with confidence that the reports and KPIs they are seeing are verifiable and reliable

Solution

Our Data and Analytics Practice partnered with the client to develop a data and AI architecture that provided the necessary foundation for effective AI and machine learning elements. This architecture included: • Data ingestion • Feature engineering • Automated model training/storage • Inference pipelines for predictive analytics

The initial plan was for the Kforce Consulting Solutions Practice Lead to work closely with the Kforce machine learning engineering team on feature engineering and model architecture. However, through data and process exploration, the practice lead discovered numerous data quality and governance problems around data ingestion into the client data stores, as well as numerical and statistical problems around the machine learning models themselves. As a result, the practice lead spent much of his time instituting data quality and governance practices with client ETL engineers and consulting with client data scientists to improve the statistical methods in the machine learning processes. The machine learning engineering team conducted much of their work independently, while full stack developers embedded in client teams and worked to iteratively improve dashboards and analytic displays based on stakeholder feedback.

Data Sources: Sensor data Weather data Satellite images Customer data Station data

Data quality checks: Anomaly detection real-time data collation (sagemaker)

ML PROCESSES: XG BOOST MODELS GRADIENT BOOSTED TREES (SAGEMAKER)

Data warehouse (aws)

ml evaluation / retraining processes: (redshift/sagemaker)

Dashboard / report generation (tableau)

USERS

Machine learning platform: AWS Sagemaker Data platform: SQL server, SSAS, AWS Redshift Machine learning models: Gradient Boosted Trees, XG Boost

Our collaboration with a global networking technology company aimed at the design and implementation of a generative AI chatbot. This innovative tool was created to empower team and regional leaders with immediate, comprehensive insights into their teams’ performance across a broad range of data metrics.

A comprehensive review of the data foundations and AI governance needed to take your company to the next level.

Nearly any business can profit from thoughtful AI integration so long as they are prepared to create the data engineering an governance foundations necessary to use the technology effectively. From models to infrastructure, learn how to successfully navigate the AI market.

Custom solutions to achieve your goals

SPACE IN BETWEEN BUTTONS

SPACE IN BETWEEN MODULES/ Macros/ Sections

SPACE IN BETWEEN HEADER AND SUBHEAD + Subhead & Body

Implementation

Establishing proper data quality and governance was essential to creating this machine-learning solution

Together Toward Tomorrow

creating the proper AI solution

Establish data quality, architecture and governance