ToS Risk Guard — Quote-Backed Contract Risk Analysis
A Chrome extension that flags risky clauses in terms of service and privacy policies only with verbatim quotes: "an AI system that doesn't trust AI." 95% precision on 6 real contracts; 4 out of 4 hidden prompt injection attempts caught.
- Period
- 2026
- Stack
- Chrome Extension (MV3) · Gemini API · JSON Schema · Cypress
Problem
Very few people read the contract before checking the "I agree" box. When a terms of service, privacy policy, or subscription terms page is opened, ToS Risk Guard captures the text and explains the risky clauses in plain Turkish.
Design principle: an LLM system that doesn't trust the LLM
- Every claim backed by a quote: Each risky clause is shown with a verbatim quote from the contract. Any claim whose quote can't be found in the original text is discarded before the user ever sees it.
- Code computes the score: The risk score (0–100) doesn't come from the LLM; it is calculated with fixed weights from verified clauses only.
- No free text: LLM output is obtained through a JSON schema and deterministically re-validated.
- Prompt injection filtered out: "Instructions to the AI" hidden in the page are stripped out before they reach the LLM and shown to the user as a warning.
- Says so when unsure: When the system isn't sure, it says so and explains why: "Low confidence — the system is not sure."
Evaluation
A mini evaluation on 6 real contracts (Netflix, Spotify, YouTube, Zoom, GitHub, Dropbox):
| Metric | Result |
|---|---|
| Precision | 95% — 95% of the clauses shown are correct |
| Recall | 51% — roughly half of the clauses are missed |
| Recall with a stronger model | 62%, but analysis is 10–60× slower |
| Hidden prompt injection | 4 out of 4 attempts caught; the score could not be manipulated |
I listed every missed clause and every false alarm one by one in the repository; I wanted the system's weaknesses to be visible too.
Built with
Chrome Extension (Manifest V3), Gemini API, Alibaba's open-source page-agent library (read-only, following the principle of least privilege), JSON schema validation, and Cypress for end-to-end testing and demo recording.
The app does not provide legal advice; it is only a first-pass reading tool.
I would like to thank Asst. Prof. Dr. Nurettin Şenyer for the guidance.