In a world increasingly shaped by AI, the power of data annotation cannot be underestimated. Accuracy, nuance, and ethics in AI fundamentally rely on human insight during training and evaluation.
At Moveworks, we recognize that data annotation is a pivotal pillar upon which AI stands. This process of labeling and contextualizing data is crucial for training machine learning models to succeed in an enterprise environment. Meticulous data annotation is key not just for the success of our AI-driven solutions but also for the broader ethical landscape of AI.
In this post, we will explore Moveworks’ rigorous and thoughtful approach to data annotation. We aim to provide transparency into the techniques and principles that guide our annotation practices — with the ultimate goal of demonstrating our commitment to developing AI that responsibly augments human intelligence.
Annotation enables natural, intelligent conversations
Annotation is the essential process of labeling the conversational data that powers AI's ability to engage in natural dialogues. Our team of expert annotators methodically analyzes and enriches hundreds of thousands of real-world and synthetic (created by generative large language models) employee service conversations, identifying topics, intents, sentiment, entities, and more. This annotated data trains our models to understand even the most subtle user queries and service tickets and determine appropriate responses.
Our annotators compare and contrast. They rank and evaluate. They score. They attach tags and approve or modify machine-generated pre-tagging, thereby giving feedback to algorithmic output. They discuss. They ask pertinent and probing questions, resulting in improvements to our annotation guidelines. They classify, summarize, and describe. Some of them translate or edit translations. They think, ponder, and judge. They are experts at what they do.
For example, our annotators attach intents to utterances like "ask for order status" or "complain about the broken door." They use positive and negative sentiment categories to label tones like frustrated and grateful. They mark key entities like product names, cities, and dates. These detailed human insights teach our AI to pick up on cues, which can at times be subtle, and react appropriately.
Our commitment to expert annotation enables our AI to handle the complex conversations that drive real business value — from resolving IT issues to answering location-specific HR policy questions. Annotation helps our AI-powered copilot understand each query in the context of the conversation and larger organization and provide instant, effective solutions. This human-annotated data is indispensable for powering AI that feels natural, responsive, and intelligent.
Annotation permeates Moveworks' AI lifecycle
At Moveworks, expert annotation powers our conversational AI and enables us to deliver standout customer experiences.
We begin by leveraging annotation to build foundational datasets for our conversational AI. As we iterate, annotators continuously validate our AI's responses and provide contextual feedback to enhance performance. We also tap annotation to swiftly customize our models for new customers and industries by adapting to unique vocabularies and terminologies.
Annotation powers a constant feedback loop that acts like a performance review for our AI. For instance, one key annotator task involves assessing our AI's responses to novel user queries, tagging any areas for improvement and allowing our models to rapidly learn on the job. Annotators also compile impressive databases of entities, in the process teaching our AIs industry- and organization-specific terminology. And just to make this abundantly clear — organization-specific entities are accessible to the bot of that organization only to ensure data privacy and security.
At Moveworks, annotation serves as a versatile asset that enables Moveworks to deliver an exceptional solution for our customers. Our team of experts supplies the human insight that powers our product from start to finish. Annotation is woven into the very fabric of our conversational AI.
The spectrum of data annotation approaches
Data annotation isn't a one-size-fits-all solution. It spans a spectrum of approaches, each with its own advantages and drawbacks. Here, we'll explore two primary models — crowdsourcing and expert annotation — shedding light on the trade-offs, risks, and benefits associated with each.
Crowdsourcing annotation
Crowdsourced annotation distributes simple tagging tasks to a very large, nearly anonymous pool of online workers, enabling high-volume annotation at a low cost. However, this approach comes with concerning ethical implications surrounding worker treatment and compensation. Crowdworkers are often paid meager wages to rapidly complete microtasks while working in complete isolation, a model some argue is exploitative.
Crowdsourcing also introduces data security risks, as tasks are handled by an uncontrolled, transient crowd. Quality can suffer as well since anonymous workers lack specialized expertise and the ability to provide nuanced judgments. The speed and cost efficiencies of crowdsourcing present potential tradeoffs in ethics, privacy, security, and annotation quality.
Expert annotation
Expert annotation engages experienced professionals to handle intricate annotation tasks requiring specialized domain expertise. For example, Moveworks' annotators over the years have included customer service agents and IT service desk specialists, legal experts and HR professionals, and representatives from various industries such as healthcare, insurance, and manufacturing.
This approach ensures annotators possess the contextual knowledge to make nuanced judgments on complex data. Expert annotation provides premium quality training data that powers advanced AI capabilities. Since experts understand the intricacies of the data, they can minimize biases and errors.
It is also worth mentioning that our expert annotators are curious and analytically-minded people who have collectively studied a wide range of subjects: linguistics and logic, library science and philosophy of mind, psychology and journalism, computer science and mathematics, chemical engineering, history, political science, and more, and who possess a multitude of skills such as editing and copyediting, content moderation, research, technical writing, data analysis, database administration, teaching and curriculum development, technical and literary translation, among others. Some of our annotators are Ph.D. holders or candidates, and a few are professors. Several annotators have two decades of annotation experience each and were there when the field of human annotation was still in its nascency.
Of course, expert annotation is not without its downsides. It moves more slowly, given the limited talent pool. Securing, training, and compensating expert annotators also incur substantially higher costs, particularly for multi-lingual support. But for projects dealing with sensitive data or requiring deep domain knowledge as well as a rigorous approach and very fine linguistic and analytical acumen, expert annotation provides invaluable human insight.
Why Moveworks chooses expert annotation
At Moveworks, we firmly believe the benefits of expert annotation outweigh the costs. Although slower and pricier, expert annotation upholds our commitments to data security, integrity, and privacy.
Our annotators sign non-disclosure agreements, undergo extensive training on privacy and security, and receive schooling in bias mitigation. This precaution ensures responsible and conscientious data labeling, which is imperative when dealing with sensitive employee service data.
Expert annotation also provides the high-quality training data that powers the advanced natural language capabilities of our conversational AI. The rich insights of industry specialists and other professional annotators, including individuals with the ability to be infinitely trainable and attentive to detail to a fault, follow complex and nuanced annotation guidelines, and exercise superior judgment, enable our AI to handle complex queries and deliver exceptional employee experiences.
There are also important changes in the annotation landscape as a result of the rise of large language models, LLMs. We have been closely following academic research in this area, inviting guest speakers to come to talk to us and share their findings: Fabrizio Gilardi, Meysam Alizadeh, and Mael Kubli, authors of the recent article in the Proceedings of the National Academy of Sciences journal, “ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks,” spoke about demonstrating that LLMs now are powerful enough to do a lot of tasks that crowdworkers used to do.
Additionally, Veniamin Veselovsky, the lead author of “Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks,” presented work showing that crowdworkers on Amazon Mechanical Turk, one of the most popular crowdsourcing platforms, actually use LLMs to complete their tasks. This research demonstrates that it is even more important to have expert annotators helping to align LLMs with our values by catching nuanced mistakes and biases.
Given the risks of crowdsourcing, we view expert annotation as a worthwhile investment. While costlier, expert annotation delivers premium quality and ethics, both non-negotiable priorities for Moveworks. We willingly take on the higher overhead of expert annotation to deliver secure, trustworthy AI capable of nuanced conversations.
Our diverse and accomplished team
At Moveworks, our annotation team is a point of pride. We invest in recruiting and retaining specialized talent through:
- Fair compensation and wellness policies: We provide generous wages and flexible hours to facilitate the long-term retention of annotators. Ten minutes of every hour is a paid break that we call “stretch” internally. For every fifty minutes of work or screen time, our annotators take a ten-minute break, getting up from their seats, focusing away from the computer, especially by looking at distant objects, stretching, and going outside so they can reboot. We firmly believe that this “stretch” is necessary to prevent burnout and repetitive stress injuries, as well as ensure data quality on this highly repetitive task that requires concentration at speed.
- Rigorous selection: Candidates undergo tests, interviews, and background checks. We also partner with programs like Year Up to tap overlooked talent pools.
- Ongoing training: Annotators receive regular training in a variety of annotation tasks, complete with quizzes, Google classrooms, and the like.
- Continuous discourse and visibility: We use Slack channels devoted to individual annotation tasks where annotators ask questions related to project documentation and bring up edge cases for discussion and iterative improvements to guidelines. This way, our annotators can feel connected to the final product or teams that use their work and get answers to their questions about guidelines.
Our annotators hail from diverse backgrounds, holding degrees across many fields and bringing unique professional experiences. Our talent is also diverse alongside age, ethnicity, and gender identity axes. This accomplished team supplies the invaluable human insight that sets our AI solutions apart. Their expertise powers our commitment to premium quality data annotation.
The critical role of human judgement in the creation of MoveLM™
MoveLM, Moveworks' specialized enterprise language model, demonstrates AI's potential to understand workplace conversations. Importantly, its success stems from our annotation team.
One important role our annotators play in the training of MoveLM is in the clean-up of public, externally available datasets used to train the model to reason, take sequential steps, strategize, and plan action. We have found that external datasets that typically have been compiled by crowd workers are often lacking in quality and accuracy and, importantly, exhibit identity biases such as gender, national identity, and religious bias. Clearly, we do not want to ingrain biases in our model, so we reached out to our trusted annotators to rewrite or outright remove examples from externally derived datasets that exhibited biases. Our annotators also ensured that upon editing, externally derived examples used in the model training rigorously follow formal logic and demonstrate a well-diversified set of speech patterns.
Further, MoveLM is tuned on industry terminology and workflows thanks to our annotators' expertise. Annotators carefully labeled massive datasets of simulated conversations to teach MoveLM the nuances of enterprise language.
Our annotators also continuously supply MoveLM with feedback on its predictions. Among other things, this reinforcement allows it to improve comprehension of ambiguous queries and provide sensitive solutions.
In essence, MoveLM's conversational intelligence is unlocked by human contextual knowledge. Our annotators' diligent efforts in annotating and guiding MoveLM's training enable its specialized understanding of the language of work.
MoveLM shows machines still require human expertise and oversight to reach their full potential. At Moveworks, expert annotation is the indispensable ingredient, bringing conversational intelligence to our AI innovations.
Moveworks is committed to responsible annotation
At Moveworks, we uphold rigorous practices to ensure responsible, ethical data annotation:
- Expert annotators: We invest in and develop specialized talent rather than utilize transient crowdsourcing. Our experts provide premium quality labeling grounded in deep domain knowledge.
- Rigorous hiring and training: We thoroughly vet annotators and conduct extensive ongoing training on privacy and security.
- Data integrity: Skilled annotators and strict protocols maintain end-to-end data integrity throughout the annotation process. Our annotators use secure company laptops and VPN to access internal systems, enhancing data security compared to crowdsourced workers' dependence on personal unsecured devices.
Annotation directly impacts model performance and ethics. By prioritizing responsible practices, Moveworks produces trustworthy AI while safeguarding sensitive customer data. Our commitment to premium annotation underscores our broader pledge to AI ethics.
Setting the standard for responsible enterprise annotation
At Moveworks, expert annotation sits at the core of our AI development and business operations. We firmly believe this human-powered approach is fundamental to creating principled, highly-performant enterprise AI.
By investing in specialized annotator talent and upholding rigorous data practices, we produce premium quality training data to power our AI innovations responsibly. Our annotation team is the engine behind our ability to understand workplace conversations and swiftly resolve employee needs.
Moveworks offers our approach as a blueprint for enterprise annotation done right. With expertise, diligence, and humanity, annotation can transform AI into a collaborative tool for empowering employees.
We're teaching AI the language of work. Find out more about the AI behind our platform.
Table of contents
