What is unsupervised learning?

Unsupervised learning is a machine learning technique where a model is trained using large datasets without any human guidance.

Unsupervised learning operates differently to supervised learning in that the model is trained on unlabeled data and is left to identify patterns and relationships within the data on its own.

While this can lead to the model discovering natural distributions in the data, it also means that there is no expert guidance to align the model's performance with what the end user is looking for. This can lead to the model making inaccurate predictions that are not in line with the intended outcome, which is problematic for applications where the model's outputs have real-world implications.

The drawbacks of unsupervised learning are:

• Unsupervised learning can produce inconsistently accurate results: With unsupervised learning, the model is trained to discover relationships naturally, so it may result in what’s known as overfitting. Overfitting is when a model is trained too well on the training data to the point where it interprets noise or random fluctuations in the data instead of keying in on the underlying pattern. As a result, the model may perform well on the training data but poorly on new, unseen data. 

• Unsupervised learning requires a large dataset: Unsupervised learning requires massive training sets with a bare minimum of several thousands of data points to produce a desired outcome. For example, GPT-3, the model that powers ChatGPT, is trained on a whopping 45 terabytes of text data from different datasets.

In the world of conversational AI, the debate between supervised and unsupervised learning has been ongoing. While unsupervised models may seem like the self-sufficient, mature plant in the garden, the truth is that a well-tended and nurtured garden with supervised learning can yield the most beautiful and abundant blooms.

The importance of supervised learning for companies like Moveworks lies in the fact that it allows us to fine-tune our language models and bring them to perform at a high level of precision.

By leveraging the skill of over one hundred annotators to label training data and evaluate live performance, we can ensure that our conversational AI models are aligned with human expectations and can effectively handle complex, specific tasks — such as intent and entity mining — as well as rating the quality of answers and actions. This supervised approach allows us to continuously improve and meet the needs of our customers. 

The power of AI lies in the combination of unsupervised and supervised learning, where the human element adds the necessary understanding to take on more specific use cases.

Unsupervised learning provides a valuable tool for uncovering hidden patterns and insights within large datasets, which can lead to new discoveries and opportunities. In certain scenarios, especially when dealing with vast amounts of unstructured data, unsupervised learning can offer unique insights that might not be apparent through other methods. It can help companies identify trends, clusters, or anomalies that could inform business strategies, product development, and decision-making processes.

Unsupervised learning can be particularly useful when exploring unknown territories or when there are no preconceived notions about the data. It allows companies to extract value from data without the need for extensive manual labeling or expert guidance, making it a cost-effective and scalable approach in some contexts.

However, it's essential to recognize that unsupervised learning is not without its challenges, and its outputs may lack the precision and alignment with specific business objectives that supervised learning can provide. Therefore, the importance of unsupervised learning for companies lies in its ability to complement other machine learning techniques and serve as a tool for uncovering hidden knowledge within data.