Overview
I made a fine-tuned machine learning model to accurately answer customer live chats as well as a trained human with up-to-date company policy and product information that was performant enough to converse with real customers in real time. I did this work over the summer of 2024.
Explore the project
Training-data preparation
Reviewing the transcripts, I identified three distinct chat formats, each corresponding to a different date range.
Quality control & curation
In this section I’ve described the filtering process that went into curating the training data for the deployed model, including the problems with the dataset to motivate these choices.
Retrieval & the vector database
I’ve described how I got the model to know up-to-date company policy and product information.
Testing & live-chat integration
I built a testing application for trained models to evaluate training and variations on the RAG (vector DB).
The application
I built a Python application that combined the fine-tuned model with retrieval of product and policy information. We then commissioned the developer of the company’s live chat software to integrate my application into the existing system, allowing sales staff to use AI suggestions within their usual chat interface.
Evaluation and outcome
The head of sales reviewed the first fine-tuned model, and judged its responses to be below the standard of trained sales employees, making it unsuitable for deployment. After I manually curated the training data and retrained the model, he judged its conversational performance to be good relative to that of a trained sales employee. The model already demonstrated general knowledge of diamonds and jewellery but showed improved alignment to the sales process after retraining. Company-specific product information and policies were introduced later through a vector database so it would always know the latest product and policy information.