Exploring the Intricacies of Chatbots: From the Lens of a Machine Learning Engineer
Hello to all tech enthusiasts! Today, we have an exciting discussion coming straight from the experiences of Tana, a pioneering woman machine learning engineer at HTE. Here, she uncovers the complexities surrounding chatbots and the hurdles they face, particularly from a data perspective - and boy, do we have a lot to delve into!
Chatbots: From Traditional Rule-based Systems to Conversational AI
Chatbots have been making rounds in the industry for some time now. Initially, they were rule-based systems with clear boundaries on acceptable answers (here, rules point to predefined options like new policy, filing a claim, accident or theft, etc.). However, these chatbots weren't particularly popular among customers due to their lack of flexibility in conversations.
In response to the necessity for a more human-like interaction, modern-day chatbots or IvaS provide enhanced user experiences with their Conversational AI. These bots have the liberty to reflect informal, unstructured conversations filled with irrelevant or assumed information (since the dialogue partner is expected to already possess some knowledge). As a result, conversational AI is gradually finding its way into various domains, including e-commerce, customer support, lead generation, and even Covid-related services.
- Traditional Chatbots: Rule-based systems with stringent conversational boundaries.
- Conversational AI: Unstructured, human-like dialogue systems for enriched user experience.
The Role of Dialogue Systems in Chatbots
Dialog systems play a pivotal role in these human-focused tasks. Typically made to complete specific user tasks, these systems use sophisticated Natural Language Understanding techniques. They aim to discern the user's intent from their unstructured message, retain the conversational context, ask follow-up questions, and steer the user to complete the desired task.
Research Differences: Academia vs Industry
The research surrounding these dialogue systems varies significantly in the realms of academia and industry.
Academic research generally aims to create an impact through novelty, striving to solve original problems with fresh solutions. It's a constant race to outperform the state-of-the-art benchmarks. On the other hand, industry research focuses on issues faced by live, deployed systems. Influenced heavily by business decisions, industry research emphasizes fast throughput, optimized resources, and code maintainability.
- Academic Research: Focuses on novelty and beating state-of-the-art metrics.
- Industry Research: Targets issues in live systems, business decisions, and faster throughput.
Data Challenges in Dialogue Systems
Data forms the backbone of any machine learning model, but it presents its own set of complications. They can be generally categorized into three areas:
- Lack of Data: Data acquisition and pre-processing can be a long and meticulous process. Real-life user conversations, historical data, and academic datasets need to be annotated and validated before being used. If data falls short, augmentation techniques are used to synthesize it.
- Need for Normalization: Elements like spellings, acronyms, colloquialisms, and code-switching need to be standardized for the model to understand.
- Context Handling: The system requires understanding previous dialogues to infer the context of a conversation, managing ambiguities, and identifying sub-dialogues (short conversations within a larger discussion).
The multifaceted world of data challenges presents countless hurdles for dialogue systems. However, understanding these hurdles can help us appreciate the intricacies these chatbots battle, benefiting users and developers alike.
Feel free to reach out for further enquiries or clarifications. Happy learning!