Skip to content

feat: add hallucination grader and fix web search dead branch#123

Open
bhavyakeerthi3 wants to merge 3 commits intoreactome:mainfrom
bhavyakeerthi3:feat/hallucination-grader
Open

feat: add hallucination grader and fix web search dead branch#123
bhavyakeerthi3 wants to merge 3 commits intoreactome:mainfrom
bhavyakeerthi3:feat/hallucination-grader

Conversation

@bhavyakeerthi3
Copy link
Copy Markdown

Summary

This PR introduces two significant improvements to the cross_database agent graph: a new Hallucination Grader for RAG quality control and a fix for the Web Search dead branch bug.


Changes

1. Hallucination Grader (New Feature)

While the existing pipeline had a completeness_grader to assess if answers were detailed, it lacked a check for factual grounding. This PR adds a "Hallucination Grader" to verify that generated answers are strictly supported by the retrieved context.

Implementation Details:

  • New Task: Added src/agent/tasks/hallucination_grader.py which uses structured LLM output to return a binary Yes/No score and a reason for any detected hallucinations.
  • State Updates: CrossDatabaseState now tracks retrieved search results (reactome_context and uniprot_context) to provide the grader with the necessary evidence.
  • Graph Logic: Two new parallel nodes (check_reactome_hallucination and check_uniprot_hallucination) have been added to the graph. They run immediately after each database generates an answer and before the completeness check.

2. Fix: Web Search Dead Branch (Bug Fix)

Problem: When both Reactome and UniProt returned incomplete answers, the graph was short-circuiting directly to the final response, completely skipping the intended Tavily web search.

The Fix:

  • Registered a functional perform_web_search graph node backed by TavilyWrapper.
  • Fixed the conditional edge logic in decide_next_steps to route to the actual search node.
  • Updated summarize_reactome_uniprot.py to include a {web_results} placeholder so the final response can incorporate and cite external sources when relevant.

Files Changed

File Type Description
src/agent/tasks/hallucination_grader.py New Hallucination grader task logic
src/agent/profiles/cross_database.py Modified Added state fields, hallucination nodes, and fixed search edges
src/agent/tasks/cross_database/summarize_reactome_uniprot.py Modified Updated prompt template to support citation of web results

Testing

Verified via static code review and graph wiring validation. Full end-to-end verification requires OpenAI and Tavily API keys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant