Large Language Models (LLMs) significantly improve intent classification accuracy in career guidance, paving the way for AI-enabled educational support.
Author
Mohit Bhatnagar, Associate Professor, Jindal Global Business School, O.P. Jindal Global University, Sonipat, Haryana, India
Summary
This study uses Large Language Models (LLMs) for extracting intents within the field of career guidance. Collating discussions from a popular social media platform, BERTopic, a state-of-the-art topic modeling technique leveraging Bidirectional Encoder Representations from Transformers (BERT) embeddings is employed to extract key career-related themes.
Our analysis evaluates BERTopic’s proficiency, particularly its integration with LLMs, to refine topic modeling and automate the intent extraction process. A central focus lies on the application of Generative AI for automatic topic labeling, contrasting the performance of proprietary models like OpenAI’s GPT-3.5 with open-source models such as Llama-2. Subsequently, the study uses these mined intents to fine-tune a BERT based LLM, scrutinizing its efficacy in intent classification against a Random Forest baseline model.
The BERT model demonstrates a remarkable improvement in multi-classification accuracy of 0.92. Our results underscore the profound and emerging capabilities of LLMs to integrate in task based chatbots that can offer nuanced, tailored career guidance, heralding potentially a new era of AI-enabled educational support. To ensure the reproducibility of our results and foster further research, the dataset and code is publicly made available.
Published in: Lecture Notes on Data Engineering and Communications Technologies
To read the full chapter, please click here.
