In the realm of natural language processing, domain prediction is a type of sentence classification problem where a user's request is categorized into the most relevant domain.
For instance, if a user asks, "Why is my Okta login stuck in a loop?" it would be classified under an IT domain. Similarly, a question like "How do I get reimbursed for transit expenses?" would be categorized as HR domain.
As Senior Machine Learning Engineers on Moveworks’ Natural Language Understanding (NLU) team, domain prediction is a challenge we've been exploring for many years, guided by our customers' dynamic needs as Moveworks has expanded into a comprehensive enterprise AI Assistant.
Now we want to share our learnings with you.
So let's explore the definition of domain prediction, explain why it matters for enterprise AI assistants, and share how we managed to build a smarter approach to domain prediction.
What is domain prediction in AI?
A domain often refers to a distinct area of expertise or business context, such as a department, functional area, or industry. Domain prediction identifies the domain, or subject area of a query, enabling AI systems to recognize the topic of a user's issue and identify the appropriate expertise needed for resolution.
With domain prediction, AI is able to deliver more precise, context-aware responses. For enterprise AI assistants, this understanding of domain nuances leads to more relevant answers and improved efficiency.
At Moveworks, we use the term 'domain prediction' more narrowly to describe the functional areas of employee service, such as IT, HR, facilities, payroll, and more.
Why do AI assistants need domain prediction?
Domain prediction is critical for an enterprise AI assistant because it acts as an upstream signal to downstream systems, guiding how employee support issues are managed. Many businesses have specific protocols for handling requests based on their domains, making accurate domain prediction essential for efficient and effective support.
How (and why) we built domain prediction for Moveworks AI Assistant
The Moveworks NLU team's mission is to ensure that the Moveworks AI Assistant provides high utility to users by enhancing its ability to understand user intents and improving its reasoning quality.
As Senior Machine Learning Engineers on the team, domain prediction is a challenge we've been exploring for many years, driven by our customers' evolving needs as Moveworks expanded into a comprehensive enterprise AI Assistant.
With the advent of LLMs, we recognized the potential for improved accuracy, customizability, and generalizability in domain prediction and query classification, motivating us to develop a more effective approach to deliver faster, more reliable support outcomes for our customers.
The business problem of query classification
At the heart of the business problem is the challenge of enabling Moveworks AI Assistant to effectively categorize employee queries, ensuring they are directed to the correct department, such as IT, HR, or Payroll.
The complexity arises from the fact that the scope of these domains can vary significantly from one organization to another. Furthermore, companies often need to create new, custom domains to reflect their unique organizational structures and business needs.
Accurate domain prediction is essential for the AI Assistant to function optimally, offering precise support and valuable insights.
Traditional challenges in domain classification
Historically, domain prediction relied on traditional natural language processing (NLP) methods. These methods required annotating millions of user queries to train a classification model—a process that was time-consuming, costly, and not easily extensible.
Further, finetuning domain definition often involved a heuristic layer (i.e. a component within the model that employs predefined rules to guide decision-making with ambiguity) based on keywords and entities, however this was not robust and demanded substantial support and operational work.
Adding on top of all these challenges, for enterprise assistants, it can be difficult to meet every customer’s need for customization in scale because the feedback loop is too long. In the traditional NLP world, there were usually lengthy and resource intensive processes that involved customer stakeholders, CSEs and ML engineers whenever the customer wanted to create a new domain.
Building a new approach to domain prediction
We have developed a solution to address these challenges in scale, focusing on making domain prediction both robust and maintainable in the long term. Here's how we've transformed the process.
Customer self-service configuration
We introduced a UI-based configuration tool in Moveworks Setup that allows customers to create and edit domains effortlessly. Users simply need to provide a domain name, description, positive examples, and negative examples. This self-service approach empowers stakeholders to customize domains without extensive technical knowledge.
See our public documentation for reference
Real-time configuration updates
Once a customer saves their configuration, our configuration hot-reloading infrastructure swiftly reloads the configurations across our microservices within seconds. This process generates embeddings and updates our vector database, ensuring that the AI assistant is always working with the most current data.
The configuration and testing experience for a stakeholder is:
(In MWSetup) Create a new domain config or editing an existing config by providing the following data
Domain Name: The specific name assigned to the custom domain, such as "Sales Enablement" or "Manufacturing".
Domain Description: Detailed description that defines the scope of the custom domain.
Positive Examples: Specific utterances that should be classified under this custom domain, which help to strengthen domain matching.
Negative Examples: Specific utterances that should not be classified under this custom domain, which help to prevent misclassification.
(In MWSetup) Define the Handoff help option associated with this domain in Handoff Config
Examples: File a ticket, point users to a slack channel, provide a support email address
(In AI Assistant) After a few seconds of config change, send test requests to AI Assistant.
Advanced domain prediction techniques
When a user interacts with the Moveworks AI Assistant, the system leverages a semantic retriever to identify the top relevant domains. It then dynamically generates prompts for a Large Language Model (LLM) using in-context learning.
This approach allows the LLM to determine the most relevant domain with high accuracy. Additionally, we have implemented a laser system that overrides the LLM's decision if a user's query closely matches a predefined example.
Benefits of the new approach
- Generalizability: Unlike the previous method, there is no need to retrain a model when supporting additional domains.
- Customizability: Stakeholders can easily update configurations and test changes immediately, allowing for quick iterations and improvements.
- Accuracy: The use of LLMs and semantic retrieval enhances prediction accuracy, as reflected in our quantitative analysis.
Quantitative analysis
Our experiments have demonstrated a significant improvement in domain prediction performance using our ML system. Our online metrics show >90% accuracy on domain selection using human annotation.
While keeping high performance on common domains like IT and HR, we see notable improvements across several low-resource domains on our offline evaluation dataset, which traditional ML models struggle to perform well on due to limited training data:
- Payroll: Accuracy increased from 56% to 89%
- Travel & Expense: Accuracy increased from 54% to 80%
- Facility: Accuracy increased from 55% to 79%
These results indicate that our new method not only enhances accuracy, but also provides a more flexible and user-friendly solution for domain prediction.
Domain prediction is critical for improved employee support
The evolution of domain prediction in enterprise AI assistants marks a pivotal advancement in employee support technology. By leveraging LLMs and enabling customer self-service, Moveworks has created a system that is able to be both adaptable and precise.
For Moveworks AI Assistant, this capability is invaluable for offering relevant help options, such as directly connecting employees to live agents or facilitating ticket submissions.
Additionally, it provides users with detailed insights into employee issues by categorizing and tracking problems across business domains through comprehensive analytics dashboards, enabling proactive issue management and improved decision-making.
This innovation not only reduces the operational burden on engineering and customer support teams but also empowers organizations to tailor their support systems to their unique needs.As we continue to refine our approach, we look forward to further enhancing the capabilities of AI assistants in the workplace.
Watch this webinar to learn more about how you can unify your techstack with Moveworks AI Assistant.
Table of contents
