A Fine-Tuned GPT-2 For Intelligent Auto-Response System Using Transfer Learning with Contrived Email Data

Abstract

This study involves improving intelligent auto-response systems by fine-tuning GPT-2 using a contrived email dataset through transfer learning. Existing natural language models lacks the contextual specificity and understanding necessary for generating coherent and relevant email responses. Initial experiments using the standard Enron corpus revealed that general-purpose datasets often lack the contextual specificity required for high-quality email generation. To tackle this gap, GPT-2 was fine-tuned on a specialised dataset developed to mimic diverse email interactions to adapt its generative capabilities for various email contexts, such as professional communication, non-academic questions, and academic inquiries. The results of experiments show significant improvements with an increase in BLEU score from 0.53 to 0.66 compared to the baseline model, in response quality. Also, the contrived fine-tuned model achieved substantial gains in ROUGE scores across all variants indicating better contextual understanding and relevance (e.g., ROUGE-1 F1: 0.580) over the baseline. The human evaluations and qualitative analysis conducted support these findings, confirming the ability of the model to generate relevant replies that is largely close to expected human behaviour. The potential of transfer learning in fine-tuning large language models for domain-specific applications is illustrated. This study provides a robust framework for deploying intelligent auto-response systems in real-life. In the future, reinforcement learning techniques to mitigate current limitations will be explored, such as handling ambiguous prompts and improving response diversity.Transfer Learning, Domain-Specific Language Models, Fine Tuning, Generative Pre-trained Transformer, Automated Response