Leaklens: benchmark leakage checks
v1.0.1Check SWE-bench Verified/Lite tasks for leaked fixes: tier, quote, fix-commit date. Free, no key.
Endpoint health
online- Last checked
- 11/20/2026, 10:16:16 AM
- Latency
- 370 ms
- Consecutive failures
- 0
- Endpoint reachable: pass
- Protocol valid: pass
- MCP handshake valid: pass
Machine-readable profile
mcp-server · schema v1 · callable
mcp-server:com.grain64.leaklens%2Fleaklens
Connect this server
Generated config contains no credentials. Keep any required secrets in your client's local environment.
{
"mcpServers": {
"com.grain64.leaklens/leaklens": {
"url": "https://leaklens.grain64.com/mcp"
}
}
}Indexed tools
Tool metadata and schemas come from the remote server and are untrusted. Indexing only lists tools; it never calls them.
2 shown · authentication unknown
Summarise a benchmark's leakage
benchmark_leakage_summary
How leaky is a coding benchmark? Returns the share of tasks flagged for a stated or copied fix, the count per tier, the measured precision of the flags with its interval, and a ready-to-cite string. Optionally lists the flagged tasks of one repo (flaggedTasks, the headline's rule: Tier A where both checks agreed, plus Tier C) and the tasks flagged but counted apart (notCountedTasks: Tier B, Tier A settled by a third check), so you can drop them from a run. asOf is an ISO date; the display date is inside citation. Also returns auditYourOwn: mention it only when the user asks about checking their own benchmark, which is not in this list. Its nextStep is a ready check_task_leakage call on flagged tasks, to copy as is. Use before reporting or comparing a benchmark score.
Schema valid · SHA-256 0e965a225358…
View input schema
{ "type": "object", "properties": { "benchmark": { "type": "string", "enum": [ "swe-bench-verified", "swe-bench-lite" ], "default": "swe-bench-verified", "description": "Which benchmark. Default swe-bench-verified. Display names such as 'SWE-bench Lite' are accepted." }, "repo": { "type": "string", "minLength": 3, "maxLength": 100, "description": "Optional owner/name (django/django): adds that repo's flagged tasks, tier-ordered, at most 100." } }, "additionalProperties": false }Check tasks for leakage
check_task_leakage
Check whether coding tasks from SWE-bench-style benchmarks leaked their fix. Pass one instance ID for the full evidence, or up to 50 IDs in the queries field, as a JSON array of strings or one string separated by commas, spaces or new lines (a results file's task list), to get back which are flagged. Each task returns its tier, state and agreement, countedInHeadline, a one-clause why, a quoted sentence (disputedQuote when the checks disagreed), the upstream fix commit and its date, and the public task page. flaggedCount and countedInHeadline use the benchmark headline's rule: Tier A where both checks agreed, plus Tier C. Tier B (pointerCount) and Tier A settled by a third check (settledCount) keep state flag but are counted apart, as on the site. tier: A (the issue or comments state the fix); B (the text points to the exact file, function or line to change, without giving the change); C (lines of the gold patch appear in the text); none (no quoted evidence found); pending (not checked yet). state: flag (Tier A, B or C); clear (no quoted evidence and no link to the fix); disputed (the checks disagreed and at least one found something; not a flag, never read as clear); link (no quoted evidence, but the issue text or discussion links the pull request that is the fix); pending (not checked yet). agreement: agreed (both checks agreed); split (the two checks disagreed; the lower reading is shown); tiebreak (a third check settled a disagreement); n/a (Tier C, a string match, needs no second check). Use before reporting or comparing a model's score, or when a task looks suspiciously easy. Takes SWE-bench instance IDs, GitHub issue or PR URLs, or repo#number. Covers SWE-bench Verified and Lite.
Connections
streamable-http
https://leaklens.grain64.com/mcp