DiffSense
A self-hostable GitHub App that autonomously reviews pull requests using an agentic Claude AI loop with RAG-based learning and adaptive feedback.
Overview
DiffSense is a self-hostable GitHub App / AI Agent that automatically reviews pull requests using an agentic Claude AI loop. When a PR is opened, the bot queues a review job, then runs a multi-step reasoning cycle — listing changed files, reading diffs, searching for historically similar review patterns, and posting inline comments directly on the PR diff. It uses OpenAI embeddings and pgvector to perform semantic similarity search over past review comments, and learns from developer responses: when a review thread is resolved or dismissed on GitHub, the bot updates the feedback weight of that comment in the database, making similar suggestions more or less prominent on future PRs.
What Does it Do?
- Automatically trigger a PR review when a pull request is opened or updated on GitHub
- Run a multi-iteration Claude agentic loop dispatching 5 tools per review cycle
- Post inline review comments directly on specific lines of the PR diff
- Search past review comments using semantic vector similarity to surface relevant patterns
- Adapt future reviews by adjusting comment weights when GitHub threads are resolved or dismissed
- Queue and process multiple PR review jobs asynchronously without blocking webhook responses
- Authenticate as a GitHub App with scoped installation tokens per repository
Why I Built this
- To explore agentic AI patterns with multi-step tool dispatch and iterative LLM reasoning
- To implement RAG with pgvector similarity search for context-aware, repo-specific AI responses
- To build a self-improving feedback loop that adapts to each team's review preferences over time
- To practice GitHub App development, HMAC webhook security, and event-driven backend architecture
- To learn BullMQ job queues and async processing patterns for reliable background task execution
Tech Stack
- TypeScript
- Node.js
- Express.js
- Claude AI
- Anthropic SDK
- OpenAI Embeddings
- BullMQ
- Redis
- PostgreSQL
- pgvector
- Prisma
- GitHub App
- Octokit
- Docker