Project synopsis

Our collaboration with a global networking technology company aimed to design and implement a generative AI chatbot. This innovative tool was created to empower team and regional leaders with immediate, comprehensive insights into their teams’ performance across a broad range of data metrics. The company had numerous dashboards with relevant metrics. However, configuring these dashboards to highlight specific required data series and interpreting graphs was proving to be a challenge. The goal was to provide data summaries to leaders on demand, ensuring maximum flexibility with minimal dashboard tweaks.

This Ruler is for the Space Between the Bottom of the Hero/Header Text and the Subheader Copy. Check Your Spacing for Consistency.

This Ruler is for the Space Between Text/Buttons/Callouts. Check Your Spacing for Consistency.

Revolutionizing team performance: Kforce’s AI chatbot empowers team performance with dynamic, real-time sales insights

Our approach

Given the client’s initial uncertainty about the possible functionalities of the large language model (LLM) chatbot, our KCS Data and Analytics Practice Leader provided initial recommendations and conducted a workshop with client stakeholders. The goal of this workshop was to better understand the types of data features and inferences that would be most beneficial in the initial deployment. After identifying the needs and desires of end-users and stakeholders and considering the client’s institutional rules around LLM API use and development, Kforce developed a plan that outlined the creation of a chatbot MVP for testing by client users for later iteration.

Project scoping

Connect with me

Brad Boyd

MEET THE TEAM

PRACTICE LEADER Data & analytics

Connect with me

Andrew Gracyzk

PRACTICE DIRECTOR Data & analytics

Connect with me

Salman Kabir

SERVICE DELIVERY MANAGER

Connect with me

Joe Mahaffey

CLIENT SOLUTIONS SR. DIRECTOR

• The chatbot is only accessible through existing client company internal portals • This both ensures access security and allows user credentials to be easily verified and passed via API to the chatbot service environment

Process stages

• User credentials are used to ensure only the proper users get access to any given data through the client’s data entitlement layer

• Classification was initially done using the LLM through prompting and examples • Once more sample questions were amassed in logs, a simple classification model was trained from Google t5-small

• Simple chain-based prompting strategy

• Used for both ease of logging and because agent-based methods (e.g., LangChain) were incompatible with the rate limits imposed upon the API institutionally by the client • Simple to set up guardrails around improper API uses (e.g., asking about unrelated topics)

• Different query/topic classifications lead to different prompt sequences within the chatbot architecture

• For some simple and common queries, SQL queries are stored to retrieve data without LLM input • For most queries and topics, classification allows for targeted prompting and passing only essential table metadata relevant to the topic(s) involved in the user’s query. This both improves performance (in accuracy and time) and reduces costs from excess tokens sent to the LLM

1. Users issue a query through an API to the chatbot service

2. The chatbot service categorizes the questions into several topics which pertain to different data topics and/or query templates

3. Prompting sequences instruct the LLM to create or use SQL queries based on the classification of the query/topic

• All data passes through the client’s entitlement layer

4. SQL queries retrieve relevant data from the Snowflake data warehouse

• This ensures all data access protocols are followed and users only see data they are entitled to access

• Formatting/inference instructions are passed to the LLM alongside data according to the topic classification of the initial query

5. Retrieved data is passed back to the LLM for formatting and inference

• Responses are, depending on the query and user preference settings, displayed in bullet points, text and/or table(s)

6. Responses are passed back to the chatbot process and sent along to the user’s UI application

Key Features

1. Trained logistic regression models in the Snowflake environment to create columns for some inference topics • Customer retention risk • Cross-selling/up-selling opportunity prediction

2. Creation of logging for user and session-based analysis of chatbot performance • Logging also allows for history context implementations in future iterations • User feedback features implemented and incorporated into session and user logging for process analysis and improvement

3. Classification process and modular prompting strategy • Provides futureproofing against changes in data warehousing or additions/changes to desired question scope

Team composition

We built a custom-designed team from our portfolio of proven technologists to address the customer's specific needs and integrate successfully within their environment. The initial team composition for product development included: • Lead Data Scientist This role served as the primary communication and planning touchpoint interacting with client teams and resources, determining and enacting a prompting strategy in consultation with Kforce Consulting Solutions Practice Leadership and client teams. • Data Scientist This role was responsible for creating the code and prompting framework for the prompting strategy and conducting feature testing for chatbot features.

Technical breakdown

The primary technical requirement was to leverage the client’s API to GPT 3.5 Turbo (licensed through Microsoft/OpenAI) alongside existing data warehousing of sales data in Snowflake. This setup allowed for on-demand data retrieval and basic inference for Sales Leadership. Other technical requirements and limitations that shaped the needs of the project included: • No retrieval augmented generation (RAG) as the client’s internal AI rules and administration had not yet approved the use of the Microsoft Azure GPT API for use in RAG or RAG-assisted applications. • All data must be accessed through existing data entitlement processes. • Tokenization costs of using the LLM API meant tokens sent to the LLM should be minimized to reduce production and testing costs. • Query rates to the LLM API were limited on a per-team basis.

For the purposes of development and initial MVP deployment, a cloud based VM was created to host the chatbot code, which runs as Python Flask applications within containers on the VM. The chatbot service is accessible as a service through a secure API through which queries and metadata (username, session ID if applicable, etc.) are passed to the chatbot system. The chatbot service ensures that the LLM can retrieve the right data to answer user questions and provide summary and/or inference while abiding by technical and institutional constraints. The KCS team created a methodology that ensured sufficient flexibility to answer free-form questions while also limiting computational costs and ensuring adherence to security measures.

Project outcomes

Kforce’s Data and Analytics Practice partnered with the client to develop a generative AI framework using the client’s internal, licensed instance of OpenAI’s GPT 4 and their internal sales and customer data to serve as a live chatbot. The final product was delivered on schedule for MVP testing internally by the client with all agreed upon features succeeding all tests the Kforce team and initial users could devise. The product also included several more features than what was initially planned (for example, adding more than present quarter data to allow for some time series analysis of sales data).

CLIENT PILOTS DEDICATED CHATGPT CHATBOT TO EXPLORE PRACTICAL, REAL-TIME USES & DRIVE BETTER SALES, RECURRING REVENUE AND CLIENT SATISFACTION

• AI/ML Engineer This role worked with the client’s data engineering team to validate data going into the AI environment, maintain and manage CI/CD pipelines for development and ensure security measures at all stages of development. With an efficient technical discovery and immersion, actual work on the project began quickly after the initial scoping workshops. The custom nature of this team enabled them to quickly deliver outcomes. Delivery Performance developed a Playbook to introduce and acclimate the team with the project scope and team structure before the formal workshops kicked-off. Team leadership worked in tandem with the client supervisor to enable accesses to tools, applications and documentations that assisted the team ramp up and deep dive into the existing infrastructure.

The chatbot was used to answer questions and provide data summaries pertaining to team and regional sales performance, customer/opportunity status and other necessary queries. Sales leadership is now able to make queries pertaining to any data present in the customer or sales team databases and is also provided sample prompts for quick access to answers to frequently asked questions.

Our Kforce Consulting Solutions experts provide the knowledge and leadership our clients rely on to accelerate their business. Our proven team takes a unified approach to driving large-scale change and unlocking new opportunities for growth and success.

Looking to optimize your generative AI capabilities?

SPACE IN BETWEEN BUTTONS

SPACE IN BETWEEN MODULES/ Macros/ Sections

SPACE IN BETWEEN HEADER AND SUBHEAD + Subhead & Body

Together Toward Tomorrow

generative AI