Title
Retrieval Augmented Generation for Domain-specific Question Answering
Research Question
How can we design a retrieval-augmented generation (RAG) framework that understands domainspecific knowledge and terminology, effectively retrieves data from dynamically updated
databases, maintains user privacy, and presents relevant information while reducing
hallucinations in generated answers?
Year
2024
Model Used
•
Retrieval Model
•
Finetuned LLM
•
QA Generation Module
•
Product Intent Extraction Model
•
En_core_web_lg (NER Model)
Approach
This paper introduces a two-stage retrieval-augmented generation (RAG) framework—consisting of
a click-behavior-trained retriever and a retrieval-aware finetuned LLM—designed to accurately
understand domain-specific knowledge, retrieve relevant content from dynamic databases, and
generate grounded, low-hallucination answers while preserving user privacy
Metrics
• Normalized Discounted Cumulative Gain (nDCG)
• Relevancy Metrics
Findings
•
o
Domain Understanding – One of the key attributes gained is the ability to
understand product names and place importance on them.
o
Ability to comprehend shorter sentences.
o
Query Augmentation via Product Identification
Challenges:
o
Generator underperformance — Finetuned LLM performed worse than GPT-4
due to stylistic gap between LLM-generated training data and human-written
answers.
o
Product ambiguity — Similar Adobe products (e.g., Photoshop vs.
Photoshop Express) make query disambiguation difficult
o
Dynamic data — Frequently changing product features/pricing require
constant database updates.
Research Gap
•
No use of behavioral signals — Prior studies rarely leverage real user click data to
train or improve retrieval relevance.
• Poor performance of the finetuned generator on the test set
Accuracy %
•
Since this is a retrieval and generation task, performance is evaluated using nDCG for
retrieval and a GPT-4 relevance score (0–1) for generation instead of traditional
accuracy. The finetuned retriever achieved the best retrieval performance with nDCG up
to 0.8221, outperforming larger pretrained models. Combined with GPT-4, it also
produced the highest generation relevance score (0.7242 vs. 0.1705 for GPT-4 alone),
showing that retrieval-grounded generation significantly improves answer quality.
Dataset
•
Helpx Articles
•
Community Questions
•
Generated Helpx QA
•
Generated AdobeCare Video QA