\\n>

⚙️ Run configuration

\\n> \\n> **Configuration used**: defaults\\n> \\n> **Review profile**: CHILL\\n> \\n> **Plan**: Pro Plus\\n> \\n> **Run ID**: `8bfa6dc8-47da-41f1-bf4d-13450e13e468`\\n> \\n>

\\n> \\n>

\\n>

📥 Commits

\\n> \\n> Reviewing files that changed from the base of the PR and between d4fa05d977a0d3d568a81b3881a7c33f5bc98b84 and b3f045a189414fc69cf6a07b6f58a1b37657a36e.\\n> \\n>

\\n> \\n>

\\n>

📒 Files selected for processing (4)

\\n> \\n> * `docs/leakfix-mine.md`\\n> * `leakmine/bigquery.py`\\n> * `leakmine/cli.py`\\n> * `leakmine/tests/test_leakmine.py`\\n> \\n>

\\n> \\n> ```ascii\\n> _________________________________________\\n> < Fly, you fools! I'll hold off the bugs. >\\n> -----------------------------------------\\n> \\\\\\n> \\\\ (\\\\__/)\\n> (•ㅅ•)\\n> / 　づ\\n> ```\\n\\n\\n\\n\\n\\n

\\n

📝 Walkthrough

\\n\\n\\n\\n## Walkthrough\\n\\nAdds a new `leakmine/bigquery.py` module with two BigQuery SQL generators (GH-Archive PR discovery and contents sweep), metadata scoring, and NDJSON ingestion into the leakmine store. Two new CLI subcommands (`bq-sql`, `bq-ingest`) are wired into `leakmine/cli.py`, the module is exported from `__init__.py`, tests are added, and a documentation section is appended to `docs/leakfix-mine.md`.\\n\\n## Changes\\n\\n**BigQuery Discovery Arm**\\n\\n| Layer / File(s) | Summary |\\n|---|---|\\n| **SQL generators and sweep pairs**
`leakmine/bigquery.py` | Defines `SWEEP_PAIRS` (ecosystem acquire/cleanup regex pairs), `_LANGS` (language filters), `gharchive_discovery_sql` (GH-Archive merged-PR query with keyword/size/dedup), and `contents_sweep_sql` (zero-fetch syntactic sweep over repo contents tables, sample or full). |\\n| **Metadata scoring and NDJSON ingestion**
`leakmine/bigquery.py` | Adds `MetaResult` dataclass with `summary_md()`, `metadata_score` (keyword + changed-files heuristic), `ingest_rows` (dedup by `(repo, number)`, score filter, optional DB write via `schema.insert_candidate`/`insert_label`), and `read_ndjson` parser. |\\n| **CLI subcommands and package export**
`leakmine/__init__.py`, `leakmine/cli.py` | Exports `bigquery` from `__all__`, imports it in CLI, adds `cmd_bq_sql` and `cmd_bq_ingest` handlers, and wires `bq-sql`/`bq-ingest` subcommands with their full argument sets. |\\n| **Tests**
`leakmine/tests/test_leakmine.py` | Adds `test_bq_gharchive_sql`, `test_bq_contents_sweep_sql`, `test_bq_metadata_score`, and `test_bq_ingest_ndjson` covering SQL shape, scoring behavior, deduplication, and DB storage counts. |\\n| **Documentation**
`docs/leakfix-mine.md` | Adds section 14 describing GH-Archive and contents-sweep modes, the NDJSON→`bq-ingest`→per-PR-fetch pipeline flow, CLI examples, and cost/honesty caveats. |\\n\\n## Estimated code review effort\\n\\n🎯 3 (Moderate) | ⏱️ ~20 minutes\\n\\n## Possibly related PRs\\n\\n- [PhysShell/OwnAudit#24](https://github.com/PhysShell/OwnAudit/pull/24): Introduced the original leakmine pipeline, CLI scaffolding, and test structure that this PR extends with the BigQuery arm.\\n\\n## Poem\\n\\n> 🐇 A rabbit hops to BigQuery's gate,\\n> SQL in paw, partition dates straight.\\n> GH-Archive scanned, sweep pairs aligned,\\n> NDJSON rows neatly assigned.\\n> The fetch wall stands, but discovery's free —\\n> metadata scored, candidates in spree! 🌟\\n\\n

\\n\\n\\n\\n\\n

\\n

🚥 Pre-merge checks | ✅ 4 | ❌ 1

\\n\\n### ❌ Failed checks (1 warning)\\n\\n| Check name | Status | Explanation | Resolution |\\n| :----------------: | :--------- | :------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------- |\\n| Docstring Coverage | ⚠️ Warning | Docstring coverage is 26.32% which is insufficient. The required threshold is 80.00%. | Write docstrings for the functions missing them to satisfy the coverage threshold. |\\n\\n

\\n

✅ Passed checks (4 passed)

\\n\\n

\\n\\n\\n\\n\\n

\\n

✨ Finishing Touches

\\n\\n

\\n

📝 Generate docstrings

\\n\\n- [ ] Create stacked PR\\n- [ ] Commit on current branch\\n\\n

\\n

🧪 Generate unit tests (beta)

\\n\\n- [ ] Create PR with unit tests\\n- [ ] Commit unit tests in branch `claude/own-language-market-research-jgrrgm`\\n\\n

\\n\\n

\\n\\n\\n\\n\\n---\\n\\n\\n\\n\\n_{Comment `@coderabbitai help` to get the list of available commands.}\\n\\n\",\"path\":null,\"side\":null,\"line\":null}{\"type\":\"pull_request_comment\",\"body\":\"\\n### 💡 Codex Review\\n\\nHere are some automated review suggestions for this pull request.\\n\\n**Reviewed commit:** `d4fa05d977`\\n \\n\\n

ℹ️ About Codex in GitHub

\\n
\\n\\n[Your team has set up Codex to review pull requests in this repo](https://chatgpt.com/codex/cloud/settings/general). Reviews are triggered when you\\n- Open a pull request for review\\n- Mark a draft as ready\\n- Comment \\\"@codex review\\\".\\n\\nIf Codex has suggestions, it will comment; otherwise it will react with 👍.\\n\\n\\n\\n\\nCodex can also answer questions or update the PR. Try commenting \\\"@codex address that feedback\\\".\\n \\n

\",\"path\":null,\"side\":null,\"line\":null}{\"type\":\"pull_request_comment\",\"body\":\"**_{_{![P2 Badge](https://img.shields.io/badge/P2-yellow?style=flat)}} Filter out balanced files from contents sweep**\\n\\nWhen a file contains both an acquire and its cleanup, such as `addEventListener(` plus `removeEventListener`, this `WHERE` clause still admits it because it only checks for the acquire pattern; the `CASE` then falls through to `signal = 'other'`, so `bq-sql --kind contents` exports balanced files as candidate sites. On large sweeps this pollutes the zero-fetch candidate set with non-leaks; filter on `(acquire AND NOT cleanup)` for at least one pair, or exclude the `other` rows before returning results.\\n\\nUseful? React with 👍 / 👎.\",\"path\":\"leakmine/bigquery.py\",\"side\":\"RIGHT\",\"line\":null}{\"type\":\"pull_request_comment\",\"body\":\"_📐 Maintainability & Code Quality_ | _🟡 Minor_ | _⚡ Quick win_\\n\\n**Fix the inline-call example.**\\n\\n`addEventListener(` is missing the closing `)`, which reads awkwardly and matches the static warning. Rephrase the examples as balanced call forms.\\n\\n

\\n

Proposed fix

\\n\\n```diff\\n-`addEventListener(` есть, `removeEventListener` нет; `setInterval(` без\\n+`addEventListener()` есть, `removeEventListener()` нет; `setInterval()` без\\n```\\n

\\n\\n\\n\\n

\\n

📝 Committable suggestion

\\n\\n> ‼️ **IMPORTANT**\\n> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.\\n\\n```suggestion\\n**B. Contents sweep (снапшот кода, zero-fetch) — `contents_sweep_sql`.**\\nSQL по `bigquery-public-data.github_repos`: синтаксический тир прямо в SQL — acquire без\\ncleanup (`addEventListener()` есть, `removeEventListener()` нет; `setInterval()` без\\n`clearInterval`; и т.д., см. `SWEEP_PAIRS`). **Ноль fetch**, масштаб до миллионов файлов.\\n**Дорого**: любое обращение к `contents.content` сканирует весь столбец ~2.7 ТБ (~$13,\\nсжигает free-tier) — поэтому по умолчанию бьёт по `sample_files`/`sample_contents` (дёшево);\\n`--full` включай осознанно.\\n```\\n\\n

\\n\\n\\n\\n

\\n

🧰 Tools

\\n\\n

\\n

🪛 LanguageTool

\\n\\n[typographical] ~262-~262: Непарный символ: «)» скорей всего пропущен\\nContext: ...й тир прямо в SQL — acquire без cleanup (`addEventListener(` есть, `removeEventLi...\\n\\n(RU_UNPAIRED_BRACKETS)\\n\\n

\\n\\n

\\n

🤖 Prompt for AI Agents

\\n\\n```\\nVerify each finding against current code. Fix only still-valid issues, skip the\\nrest with a brief reason, keep changes minimal, and validate.\\n\\nIn `@docs/leakfix-mine.md` around lines 260 - 266, The inline-call examples in the\\n`contents_sweep_sql` description are unbalanced and should be rewritten as\\nproper call forms. Update the examples around `SWEEP_PAIRS` to show complete\\nbalanced invocations, using the existing symbols like `addEventListener`,\\n`removeEventListener`, `setInterval`, and `clearInterval`, so the text reads\\ncleanly and matches the static warning.\\n```\\n\\n

\\n\\n\\n\\n\\n\\n\\n\\n_Source: Linters/SAST tools_\\n\\n\",\"path\":\"docs/leakfix-mine.md\",\"side\":\"RIGHT\",\"line\":266}{\"type\":\"pull_request_comment\",\"body\":\"_📐 Maintainability & Code Quality_ | _🟡 Minor_ | _⚡ Quick win_\\n\\n**Add language tags to the fenced blocks.**\\n\\nBoth fences currently trigger MD040. Use `text` for the flow diagram and `bash` for the CLI example so the docs render and lint cleanly.\\n\\n

\\n

Proposed fix

\\n\\n```diff\\n-```\\n+```text\\n BigQuery (SQL) -> экспорт NDJSON -> leakmine bq-ingest -> store + candidates.json\\n |\\n узкий per-PR fetch + signals.classify + confirm\\n-```\\n+```\\n@@\\n-```\\n+```bash\\n # A) сгенерить discovery-SQL, выполнить в BigQuery, выгрузить как NDJSON:\\n python3 -m leakmine.cli bq-sql --kind gharchive --ecosystem dotnet_wpf --from 20240101 --to 20241231\\n # -> bq query ... | bq extract ... > rows.ndjson (в своём GCP)\\n@@\\n-```\\n+```\\n```\\n

\\n\\n\\n\\n\\nAlso applies to: 283-294\\n\\n

\\n

🧰 Tools

\\n\\n

\\n

🪛 markdownlint-cli2 (0.22.1)

\\n\\n[warning] 270-270: Fenced code blocks should have a language specified\\n\\n(MD040, fenced-code-language)\\n\\n

\\n\\n

\\n

🤖 Prompt for AI Agents

\\n\\n```\\nVerify each finding against current code. Fix only still-valid issues, skip the\\nrest with a brief reason, keep changes minimal, and validate.\\n\\nIn `@docs/leakfix-mine.md` around lines 270 - 274, Add explicit language tags to\\nthe fenced code blocks in the docs so Markdown lint stops flagging MD040. Update\\nthe diagram fence to use text and the CLI example fence to use bash, keeping the\\nsurrounding content in the same sections of docs/leakfix-mine.md. Use the\\nexisting fenced blocks around the BigQuery flow diagram and the bq-sql example\\nto locate them, and preserve the current content while only changing the fence\\nannotations.\\n```\\n\\n

\\n\\n\\n\\n\\n\\n\\n\\n_Source: Linters/SAST tools_\\n\\n\",\"path\":\"docs/leakfix-mine.md\",\"side\":\"RIGHT\",\"line\":null}{\"type\":\"pull_request_comment\",\"body\":\"_📐 Maintainability & Code Quality_ | _🟡 Minor_ | _⚡ Quick win_\\n\\n**Fix the Ruff errors in this module.**\\n\\nLine 58 uses the ambiguous name `l`, and Lines 192/194/197/199 use multiple statements per line. Those are real Ruff violations in the added code. \\n \\n\\n

\\n

Proposed fix

\\n\\n```diff\\n langs = \\\" OR \\\".join(\\n- f\\\"LOWER(JSON_EXTRACT_SCALAR(payload,'$.pull_request.base.repo.language')) = '{l}'\\\"\\n- for l in _LANGS.get(eco_key, ())\\n+ f\\\"LOWER(JSON_EXTRACT_SCALAR(payload,'$.pull_request.base.repo.language')) = '{lang}'\\\"\\n+ for lang in _LANGS.get(eco_key, ())\\n ) or \\\"TRUE\\\"\\n@@\\n tl, bl = (title or \\\"\\\").lower(), (body or \\\"\\\").lower()\\n if any(k in tl for k in eco.keywords):\\n- score += 3; ev.append(\\\"title:leak-keyword\\\")\\n+ score += 3\\n+ ev.append(\\\"title:leak-keyword\\\")\\n if any(k in bl for k in eco.keywords):\\n- score += 2; ev.append(\\\"body:leak-keyword\\\")\\n+ score += 2\\n+ ev.append(\\\"body:leak-keyword\\\")\\n if changed_files is not None:\\n if changed_files <= 10:\\n- score += 2; ev.append(\\\"small-pr\\\")\\n+ score += 2\\n+ ev.append(\\\"small-pr\\\")\\n elif changed_files > 200:\\n- score -= 3; ev.append(\\\"penalty:mega-pr\\\")\\n+ score -= 3\\n+ ev.append(\\\"penalty:mega-pr\\\")\\n```\\n

\\n\\n\\nAlso applies to: 191-199\\n\\n

\\n

🧰 Tools

\\n\\n

\\n

🪛 Ruff (0.15.18)

\\n\\n[error] 58-58: Ambiguous variable name: `l`\\n\\n(E741)\\n\\n

\\n\\n

\\n

🤖 Prompt for AI Agents

\\n\\n```\\nVerify each finding against current code. Fix only still-valid issues, skip the\\nrest with a brief reason, keep changes minimal, and validate.\\n\\nIn `@leakmine/bigquery.py` around lines 56 - 58, The new code in bigquery.py has\\nRuff violations: rename the ambiguous loop variable used in the language filter\\nconstruction to a clearer name in the logic around the langs query, and split\\nthe multiple statements on each of the added lines in the helper/processing\\nblock around the referenced 191-199 range into one statement per line. Use the\\naffected symbols building the BigQuery filter and the nearby processing\\nfunctions to update the implementation without changing behavior.\\n```\\n\\n

\\n\\n\\n\\n\\n\\n\\n\\n_Source: Linters/SAST tools_\\n\\n\\n\\n✅ Addressed in commit b3f045a\",\"path\":\"leakmine/bigquery.py\",\"side\":\"RIGHT\",\"line\":null}{\"type\":\"pull_request_comment\",\"body\":\"_🎯 Functional Correctness_ | _🟠 Major_ | _⚡ Quick win_\\n\\n**Filter out the non-leak `other` rows.**\\n\\nThe current `WHERE` only requires an acquire match, so files that also contain the matching cleanup still come back as `signal='other'`. That turns the sweep into “files with acquire calls” instead of “acquire without cleanup”, which will flood the result set with valid code. \\n \\n\\n

\\n

Proposed fix

\\n\\n```diff\\n cases = \\\"\\\\n\\\".join(\\n f\\\" WHEN REGEXP_CONTAINS(c.content, r'{acq}') \\\"\\n f\\\"AND NOT REGEXP_CONTAINS(c.content, r'{cln}') THEN '{label}'\\\"\\n for acq, cln, label in pairs\\n )\\n- any_acquire = \\\" OR \\\".join(f\\\"REGEXP_CONTAINS(c.content, r'{acq}')\\\" for acq, _, _ in pairs)\\n+ candidate_pred = \\\" OR \\\".join(\\n+ f\\\"(REGEXP_CONTAINS(c.content, r'{acq}') \\\"\\n+ f\\\"AND NOT REGEXP_CONTAINS(c.content, r'{cln}'))\\\"\\n+ for acq, cln, _ in pairs\\n+ )\\n return (\\n f\\\"-- LeakFixMine contents sweep: ecosystem={eco_key}, table=\\\"\\n f\\\"{'sample_' if sample else ''}(files/contents). \\\"\\n f\\\"{'CHEAP sample.' if sample else 'WARNING: full content scan ~2.7TB (~$13).'}\\\\n\\\"\\n \\\"SELECT f.repo_name AS repo, f.path AS path,\\\\n\\\"\\n \\\" CASE\\\\n\\\"\\n f\\\"{cases}\\\\n\\\"\\n- \\\" ELSE 'other' END AS signal\\\\n\\\"\\n+ \\\" END AS signal\\\\n\\\"\\n f\\\"FROM `bigquery-public-data.github_repos.{files_tbl}` f\\\\n\\\"\\n f\\\"JOIN `bigquery-public-data.github_repos.{contents_tbl}` c USING (id)\\\\n\\\"\\n f\\\"WHERE ({exts})\\\\n\\\"\\n \\\" AND NOT c.binary\\\\n\\\"\\n- f\\\" AND ({any_acquire})\\\\n\\\"\\n+ f\\\" AND ({candidate_pred})\\\\n\\\"\\n f\\\"LIMIT {limit}\\\\n\\\"\\n )\\n```\\n

\\n\\n\\n\\n

\\n

📝 Committable suggestion

\\n\\n> ‼️ **IMPORTANT**\\n> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.\\n\\n```suggestion\\n cases = \\\"\\\\n\\\".join(\\n f\\\" WHEN REGEXP_CONTAINS(c.content, r'{acq}') \\\"\\n f\\\"AND NOT REGEXP_CONTAINS(c.content, r'{cln}') THEN '{label}'\\\"\\n for acq, cln, label in pairs\\n )\\n candidate_pred = \\\" OR \\\".join(\\n f\\\"(REGEXP_CONTAINS(c.content, r'{acq}') \\\"\\n f\\\"AND NOT REGEXP_CONTAINS(c.content, r'{cln}'))\\\"\\n for acq, cln, _ in pairs\\n )\\n return (\\n f\\\"-- LeakFixMine contents sweep: ecosystem={eco_key}, table=\\\"\\n f\\\"{'sample_' if sample else ''}(files/contents). \\\"\\n f\\\"{'CHEAP sample.' if sample else 'WARNING: full content scan ~2.7TB (~$13).'}\\\\n\\\"\\n \\\"SELECT f.repo_name AS repo, f.path AS path,\\\\n\\\"\\n \\\" CASE\\\\n\\\"\\n f\\\"{cases}\\\\n\\\"\\n \\\" END AS signal\\\\n\\\"\\n f\\\"FROM `bigquery-public-data.github_repos.{files_tbl}` f\\\\n\\\"\\n f\\\"JOIN `bigquery-public-data.github_repos.{contents_tbl}` c USING (id)\\\\n\\\"\\n f\\\"WHERE ({exts})\\\\n\\\"\\n \\\" AND NOT c.binary\\\\n\\\"\\n f\\\" AND ({candidate_pred})\\\\n\\\"\\n f\\\"LIMIT {limit}\\\\n\\\"\\n )\\n```\\n\\n

\\n\\n\\n\\n

\\n

🤖 Prompt for AI Agents

\\n\\n```\\nVerify each finding against current code. Fix only still-valid issues, skip the\\nrest with a brief reason, keep changes minimal, and validate.\\n\\nIn `@leakmine/bigquery.py` around lines 139 - 158, The BigQuery sweep query in the\\ncontent-building logic is still returning rows labeled as 'other' because the\\nWHERE clause only filters for acquire matches; update the query assembled in the\\nfunction that builds the SELECT so it excludes rows where the CASE falls through\\nto 'other' and only returns acquire hits that do not also match the cleanup\\nregex. Keep the existing pair-based CASE logic, but tighten the filter around\\nthe same symbols (`cases`, `any_acquire`, and the returned SQL string) so the\\nresult set contains only true leak candidates.\\n```\\n\\n

\\n\\n\\n\\n\\n\\n\\n\\n\",\"path\":\"leakmine/bigquery.py\",\"side\":\"RIGHT\",\"line\":165}{\"type\":\"pull_request_comment\",\"body\":\"_🚀 Performance & Scalability_ | _🟠 Major_ | _⚡ Quick win_\\n\\n**Keep the cheaper 5k default for `--kind contents`.**\\n\\n`contents_sweep_sql()` defaults to `limit=5000`, but this CLI hardcodes `--limit 50000` for both kinds. So `leakmine bq-sql --kind contents` silently emits a query that is 10x broader than the helper contract, and `--full` makes that the expensive path. Use a per-kind default instead of a shared 50k default. \\n \\n\\n

\\n

Suggested fix

\\n\\n```diff\\n def cmd_bq_sql(a) -> int:\\n if a.kind == \\\"gharchive\\\":\\n print(bigquery.gharchive_discovery_sql(\\n a.ecosystem, date_from=getattr(a, \\\"from\\\"), date_to=a.to,\\n- max_changed_files=a.max_changed_files, limit=a.limit))\\n+ max_changed_files=a.max_changed_files,\\n+ limit=50000 if a.limit is None else a.limit))\\n else:\\n- print(bigquery.contents_sweep_sql(a.ecosystem, sample=not a.full, limit=a.limit))\\n+ print(bigquery.contents_sweep_sql(\\n+ a.ecosystem, sample=not a.full, limit=5000 if a.limit is None else a.limit))\\n return 0\\n@@\\n- bq.add_argument(\\\"--limit\\\", type=int, default=50000)\\n+ bq.add_argument(\\\"--limit\\\", type=int)\\n```\\n

\\n\\n\\nAlso applies to: 192-200\\n\\n

\\n

🤖 Prompt for AI Agents

\\n\\n```\\nVerify each finding against current code. Fix only still-valid issues, skip the\\nrest with a brief reason, keep changes minimal, and validate.\\n\\nIn `@leakmine/cli.py` around lines 108 - 114, The `cmd_bq_sql` CLI path is\\noverriding the cheaper `contents_sweep_sql()` default by using a shared 50k\\nlimit for both query kinds. Update `cmd_bq_sql` so the `gharchive` branch can\\nkeep its larger limit behavior if needed, but the `contents` branch uses the\\nhelper’s cheaper 5k default unless an explicit `a.limit` is provided; check the\\nargument handling around `a.limit`, `a.full`, and the\\n`bigquery.contents_sweep_sql` call to make the default per-kind rather than\\nshared.\\n```\\n\\n

\\n\\n\\n\\n\\n\\n\\n\\n\",\"path\":\"leakmine/cli.py\",\"side\":\"RIGHT\",\"line\":null}{\"type\":\"pull_request_comment\",\"body\":\"_🩺 Stability & Availability_ | _🟠 Major_ | _🏗️ Heavy lift_\\n\\n**Stream the NDJSON ingest path instead of loading the whole export.**\\n\\nLine 119 reads the entire file into memory, `read_ndjson()` builds a second full list, and `ingest_rows()` then walks that list. That breaks the PR’s stated 10k–millions scale target; large BigQuery exports will hit high RSS or OOM before dedup/scoring runs. Please refactor this path so the file is consumed line-by-line and `ingest_rows` accepts an iterable.\\n\\n

\\n

🤖 Prompt for AI Agents

\\n\\n```\\nVerify each finding against current code. Fix only still-valid issues, skip the\\nrest with a brief reason, keep changes minimal, and validate.\\n\\nIn `@leakmine/cli.py` around lines 118 - 122, The NDJSON ingest path in\\ncmd_bq_ingest currently materializes the full export before ingestion, which\\nprevents scaling to large BigQuery dumps. Refactor _read/read_ndjson usage so\\nthe file is consumed as a streaming iterable line-by-line, and update\\nbigquery.ingest_rows to accept and process an iterator instead of a prebuilt\\nlist. Keep the behavior of dedup/scoring intact while ensuring cmd_bq_ingest\\npasses the streamed rows through without building an in-memory copy.\\n```\\n\\n

\\n\\n\\n\\n\\n\\n\\n\\n\",\"path\":\"leakmine/cli.py\",\"side\":\"RIGHT\",\"line\":null}{\"type\":\"pull_request_comment\",\"body\":\"_🎯 Functional Correctness_ | _🟡 Minor_ | _⚡ Quick win_\\n\\n**Assert the title/body `OR` semantics, not just field presence.**\\n\\nLine 462 passes as long as both field names appear anywhere in the SQL, so a regression from `...title...) OR (...body...)` to `...title...) AND (...body...)` would still be missed. Tighten this to assert the boolean fragment you actually rely on.\\n\\n

\\n

🤖 Prompt for AI Agents

\\n\\n```\\nVerify each finding against current code. Fix only still-valid issues, skip the\\nrest with a brief reason, keep changes minimal, and validate.\\n\\nIn `@leakmine/tests/test_leakmine.py` at line 462, The test assertion in the\\nLeakmine SQL check is too weak because it only verifies that both field names\\nappear somewhere in the generated query. Update the assertion in the relevant\\ntest around the SQL built by the leakmine matcher to verify the actual boolean\\nstructure for the title/body search, ensuring the query contains the expected OR\\nfragment rather than just presence of pull_request.title and pull_request.body.\\nUse the existing test helper and the SQL string from the same test to tighten\\nthe check against regressions in the query builder.\\n```\\n\\n

\\n\\n\\n\\n\\n\\n\\n\\n\",\"path\":\"leakmine/tests/test_leakmine.py\",\"side\":\"RIGHT\",\"line\":null}{\"type\":\"pull_request_comment\",\"body\":\"_📐 Maintainability & Code Quality_ | _🟡 Minor_ | _⚡ Quick win_\\n\\n**Use the penalized score in the assertion or drop the binding.**\\n\\n`lo` is never used here, so Ruff will keep flagging this test. The cleaner fix is to assert on the score as well as the evidence so the mega-PR penalty is covered behaviorally.\\n\\n

\\n

🧰 Tools

\\n\\n

\\n

🪛 Ruff (0.15.18)

\\n\\n[warning] 493-493: Unpacked variable `lo` is never used\\n\\nPrefix it with an underscore or any other dummy variable pattern\\n\\n(RUF059)\\n\\n

\\n\\n

\\n

🤖 Prompt for AI Agents

\\n\\n```\\nVerify each finding against current code. Fix only still-valid issues, skip the\\nrest with a brief reason, keep changes minimal, and validate.\\n\\nIn `@leakmine/tests/test_leakmine.py` around lines 493 - 495, The test in the\\nmetadata_score call binds lo but never uses it, which will keep triggering Ruff.\\nUpdate the assertion in this test to also verify the penalized score returned by\\nbigquery.metadata_score alongside the existing evidence check, or remove the\\nunused binding if you only intend to assert on ev2; use the metadata_score\\nresult tuple in this test to cover the mega-PR penalty behaviorally.\\n```\\n\\n

\\n\\n\\n\\n\\n\\n\\n\\n_Source: Linters/SAST tools_\\n\\n\",\"path\":\"leakmine/tests/test_leakmine.py\",\"side\":\"RIGHT\",\"line\":null}{\"type\":\"pull_request_comment\",\"body\":\"**Actionable comments posted: 8**\\n\\n

\\n

🤖 Prompt for all review comments with AI agents

\\n\\n```\\nVerify each finding against current code. Fix only still-valid issues, skip the\\nrest with a brief reason, keep changes minimal, and validate.\\n\\nInline comments:\\nIn `@docs/leakfix-mine.md`:\\n- Around line 270-274: Add explicit language tags to the fenced code blocks in\\nthe docs so Markdown lint stops flagging MD040. Update the diagram fence to use\\ntext and the CLI example fence to use bash, keeping the surrounding content in\\nthe same sections of docs/leakfix-mine.md. Use the existing fenced blocks around\\nthe BigQuery flow diagram and the bq-sql example to locate them, and preserve\\nthe current content while only changing the fence annotations.\\n- Around line 260-266: The inline-call examples in the `contents_sweep_sql`\\ndescription are unbalanced and should be rewritten as proper call forms. Update\\nthe examples around `SWEEP_PAIRS` to show complete balanced invocations, using\\nthe existing symbols like `addEventListener`, `removeEventListener`,\\n`setInterval`, and `clearInterval`, so the text reads cleanly and matches the\\nstatic warning.\\n\\nIn `@leakmine/bigquery.py`:\\n- Around line 56-58: The new code in bigquery.py has Ruff violations: rename the\\nambiguous loop variable used in the language filter construction to a clearer\\nname in the logic around the langs query, and split the multiple statements on\\neach of the added lines in the helper/processing block around the referenced\\n191-199 range into one statement per line. Use the affected symbols building the\\nBigQuery filter and the nearby processing functions to update the implementation\\nwithout changing behavior.\\n- Around line 139-158: The BigQuery sweep query in the content-building logic is\\nstill returning rows labeled as 'other' because the WHERE clause only filters\\nfor acquire matches; update the query assembled in the function that builds the\\nSELECT so it excludes rows where the CASE falls through to 'other' and only\\nreturns acquire hits that do not also match the cleanup regex. Keep the existing\\npair-based CASE logic, but tighten the filter around the same symbols (`cases`,\\n`any_acquire`, and the returned SQL string) so the result set contains only true\\nleak candidates.\\n\\nIn `@leakmine/cli.py`:\\n- Around line 118-122: The NDJSON ingest path in cmd_bq_ingest currently\\nmaterializes the full export before ingestion, which prevents scaling to large\\nBigQuery dumps. Refactor _read/read_ndjson usage so the file is consumed as a\\nstreaming iterable line-by-line, and update bigquery.ingest_rows to accept and\\nprocess an iterator instead of a prebuilt list. Keep the behavior of\\ndedup/scoring intact while ensuring cmd_bq_ingest passes the streamed rows\\nthrough without building an in-memory copy.\\n- Around line 108-114: The `cmd_bq_sql` CLI path is overriding the cheaper\\n`contents_sweep_sql()` default by using a shared 50k limit for both query kinds.\\nUpdate `cmd_bq_sql` so the `gharchive` branch can keep its larger limit behavior\\nif needed, but the `contents` branch uses the helper’s cheaper 5k default unless\\nan explicit `a.limit` is provided; check the argument handling around `a.limit`,\\n`a.full`, and the `bigquery.contents_sweep_sql` call to make the default\\nper-kind rather than shared.\\n\\nIn `@leakmine/tests/test_leakmine.py`:\\n- Line 462: The test assertion in the Leakmine SQL check is too weak because it\\nonly verifies that both field names appear somewhere in the generated query.\\nUpdate the assertion in the relevant test around the SQL built by the leakmine\\nmatcher to verify the actual boolean structure for the title/body search,\\nensuring the query contains the expected OR fragment rather than just presence\\nof pull_request.title and pull_request.body. Use the existing test helper and\\nthe SQL string from the same test to tighten the check against regressions in\\nthe query builder.\\n- Around line 493-495: The test in the metadata_score call binds lo but never\\nuses it, which will keep triggering Ruff. Update the assertion in this test to\\nalso verify the penalized score returned by bigquery.metadata_score alongside\\nthe existing evidence check, or remove the unused binding if you only intend to\\nassert on ev2; use the metadata_score result tuple in this test to cover the\\nmega-PR penalty behaviorally.\\n```\\n\\n

\\n\\n

\\n

🪄 Autofix (Beta)

\\n\\nFix all unresolved CodeRabbit comments on this PR:\\n\\n- [ ] Push a commit to this branch (recommended)\\n- [ ] Create a new PR with the fixes\\n\\n

\\n\\n---\\n\\n

\\n

ℹ️ Review info

\\n\\n

\\n

⚙️ Run configuration

\\n\\n**Configuration used**: defaults\\n\\n**Review profile**: CHILL\\n\\n**Plan**: Pro Plus\\n\\n**Run ID**: `f73ee075-699d-4f5c-bcf5-6cab3b76f19b`\\n\\n

\\n\\n

\\n

📥 Commits

\\n\\nReviewing files that changed from the base of the PR and between ca31a664ff61f654d68db49b46cb46c1cda8443a and d4fa05d977a0d3d568a81b3881a7c33f5bc98b84.\\n\\n

\\n\\n

\\n

📒 Files selected for processing (5)

\\n\\n* `docs/leakfix-mine.md`\\n* `leakmine/__init__.py`\\n* `leakmine/bigquery.py`\\n* `leakmine/cli.py`\\n* `leakmine/tests/test_leakmine.py`\\n\\n

\\n\\n

\\n\\n\",\"path\":null,\"side\":null,\"line\":null}","text_elements",[],{"_394":558,"_375":376,"_396":382},{"_397":559,"_399":-5,"_42":-5,"_400":-5,"_401":-5,"_96":326,"_92":403},[560],{"_46":554,"_92":561},"input_text",{"_394":563,"_375":376,"_396":382},{"_397":564,"_399":-5,"_42":-5,"_400":-5,"_401":-5,"_96":326,"_92":403},[565],{"_46":566,"_92":561},"\n /workspace/OwnAudit\n bash\n 2026-06-28\n Etc/UTC\n",{"_394":568,"_375":376,"_396":382},{"_397":569,"_399":-5,"_42":-5,"_400":-5,"_401":-5,"_96":570,"_92":403},[571,572],"developer",{"_46":574,"_92":561},{"_46":573,"_92":561},"\n## Skills\nA skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.\n### Available skills\n- imagegen: Generate or edit raster images when the task benefits from AI-created bitmap visuals such as photos, illustrations, textures, sprites, mockups, or transparent-background cutouts. Use when Codex should create a brand-new image, transform an existing image, or derive visual variants from references, and the output should be a bitmap asset rather than repo-native code or vector. Do not use when the task is better handled by editing existing SVG/vector/code-native assets, extending an established icon or logo system, or building the visual directly in HTML/CSS/canvas. (file: /opt/codex/skills/.system/imagegen/SKILL.md)\n- openai-docs: Use when the user asks how to build with OpenAI products or APIs and needs up-to-date official documentation with citations, help choosing the latest model for a use case, or model upgrade and prompt-upgrade guidance; prioritize OpenAI docs MCP tools, use bundled references only as helper context, and restrict any fallback browsing to official OpenAI domains. (file: /opt/codex/skills/.system/openai-docs/SKILL.md)\n- plugin-creator: Create and scaffold plugin directories for Codex with a required `.codex-plugin/plugin.json`, optional plugin folders/files, and baseline placeholders you can edit before publishing or testing. Use when Codex needs to create a new local plugin, add optional plugin structure, or generate or update repo-root `.agents/plugins/marketplace.json` entries for plugin ordering and availability metadata. (file: /opt/codex/skills/.system/plugin-creator/SKILL.md)\n- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /opt/codex/skills/.system/skill-creator/SKILL.md)\n- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /opt/codex/skills/.system/skill-installer/SKILL.md)\n### How to use skills\n- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.\n- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.\n- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.\n- How to use a skill (progressive disclosure):\n 1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.\n 2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.\n 3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.\n 4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.\n 5) If `assets/` or templates exist, reuse them instead of recreating from scratch.\n- Coordination and sequencing:\n - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.\n - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.\n- Context hygiene:\n - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.\n - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.\n - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.\n- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.\n","\nFilesystem sandboxing defines which files can be read or written. `sandbox_mode` is `danger-full-access`: No filesystem sandboxing - all commands are permitted. Network access is enabled.\nApproval policy is currently never. Do not provide the `sandbox_permissions` for any reason, commands will be rejected.\n","turn/started",{"_375":376,"_87":577},{"_103":-5,"_42":382,"_383":578,"_385":386,"_387":579},[],"inProgress",{"_387":581,"_375":376},{"_582":583,"_92":584},"activeFlags",[],"active","thread/started",{"_587":588},"thread",{"_589":-5,"_590":-5,"_591":592,"_593":386,"_442":443,"_594":15,"_595":-5,"_596":-5,"_597":-5,"_598":599,"_42":376,"_600":601,"_181":-5,"_602":-5,"_161":603,"_604":199,"_605":-5,"_447":606,"_387":607,"_608":-5,"_77":609,"_610":386},"agentNickname","agentRole","cliVersion","0.128.0","createdAt","ephemeral","extra","forkedFromId","gitInfo","historyMode","legacy","modelProvider","openai","parentThreadId","/opt/codex/sessions/2026/06/28/rollout-2026-06-28T04-14-34-019f0c6f-b28e-7581-b44b-cb3a22d13824.jsonl","preview","recencyAt","vscode",{"_92":391},"threadSource",[],"updatedAt",[],"messages",[614],{"_42":615,"_616":617,"_618":619,"_620":619,"_397":621,"_387":622,"_399":-5,"_623":32,"_624":625,"_626":627,"_628":-5},"7e1093c0-8887-4304-9d1f-4766feb232fa","author",{"_96":326,"_181":-5,"_624":637},"create_time",1782620073.969425,"update_time",{"_633":46,"_634":635},"finished_successfully","weight","metadata",{"_629":630},"recipient","all","channel","l1239dk1",{"_631":632},"inline_comments",[],"content_type","parts",[636],"Thanks @coderabbitai @chatgpt-codex-connector — all 9 addressed in b3f045a:\n\n**Functional**\n- **Contents sweep returning balanced files (Major, flagged twice)**: the `WHERE` now requires `(acquire AND NOT cleanup)` per pair and the `CASE` drops its `ELSE 'other'`, so a file with both `addEventListener(` and `removeEventListener` is no longer emitted as a candidate. The sweep is now genuinely \"acquire without cleanup.\"\n- **Per-kind `--limit` (Major)**: default is now `None` → 50000 for `gharchive`, 5000 for `contents`, so the pricier content scan isn't silently widened 10×.\n- **Streaming ingest (Major)**: added `iter_ndjson` (line-by-line); `bq-ingest` streams it and `ingest_rows` takes any `Iterable`, so a multi-GB export ingests at bounded memory (only the dedup set is held).\n\n**Lint / docs / tests**\n- Renamed ambiguous `l` → `lang`; split the multi-statement lines in `metadata_score` (Ruff E741 / multiple-statements).\n- Docs: balanced call forms (`addEventListener()` …) and `text`/`bash` language tags on the fenced blocks (MD040).\n- Tests: assert the actual title-**OR**-body fragment (so an AND regression fails), and assert the mega-PR penalty score (dropped the unused `lo` binding).\n\n30 tests pass under `python3` and `-O`.\n\n---\n_Generated by [Claude Code](https://claude.ai/code)_\nleakmine: BigQuery arm — firehose-scale discovery + zero-fetch contents sweep\n\n## What\n\nAdds `leakmine/bigquery.py` — the firehose-scale arm for the **10k–millions** tier (the \"not 10 repos, 10k\" question). docs/leakfix-mine.md §14.\n\n**Honest framing up front:** BigQuery removes the *discovery* wall, not the *diff-fetch* wall. GH Archive has PR metadata (title/body/repo/number), not diffs — so this yields a large, **pre-filtered** candidate queue that the existing `fetch → signals.classify → confirm` path then narrows. It doesn't magic away fetching diffs.\n\n## Two products (very different cost)\n\n**A. GH-Archive discovery — `gharchive_discovery_sql`** (events, `githubarchive.day.*`)\nMerged PRs with a leak keyword in **title OR body**, language-filtered, with a `changed_files` size cap to drop mega-refactors, deduped via `QUALIFY ROW_NUMBER()`. **Cheap** — `_TABLE_SUFFIX BETWEEN` scopes the scan to the chosen day-partitions (inside the 1 TB/mo free tier). This is the real \"10k, not 10\" discovery.\n\n**B. Contents sweep — `contents_sweep_sql`** (code snapshot, `github_repos`, **zero fetch**)\nRuns the **syntactic tier in SQL** — acquire-without-cleanup regex (e.g. `addEventListener(` with no `removeEventListener`, `setInterval(` without `clearInterval`). Scales to millions of files with no API calls. **Expensive**: any reference to `contents.content` scans the full ~2.7 TB column regardless of filters (~$13), so it defaults to the cheap `sample_*` tables; `--full` is opt-in and cost-warned in the emitted SQL.\n\n## Closing the loop\n\n`ingest_rows` / `metadata_score`: a BigQuery NDJSON export flows back into the leakmine store, scored at the **metadata tier** (keyword strength + PR-size heuristic — deliberately *not* the patch signal, which needs the diff). This ranks the narrow-fetch queue; survivors then get the real `signals.classify` / `confirm`.\n\n## CLI\n\n```\npython3 -m leakmine.cli bq-sql --kind gharchive --ecosystem dotnet_wpf --from 20240101 --to 20241231\npython3 -m leakmine.cli bq-ingest --rows rows.ndjson --ecosystem dotnet_wpf --store bq-out/corpus.db\npython3 -m leakmine.cli bq-sql --kind contents --ecosystem react_ts # --full = full ~2.7TB scan\n```\n\n## Tests / CI\n\n30 tests pass under `python3` and `python3 -O` (+4: gharchive SQL gen incl. partition/keyword/lang/size-cap/dedup; contents sweep sample-vs-full + missing-pairs error; metadata score; NDJSON ingest with dedup + store writes). SQL generation and ingest are fully offline-testable. Existing suite stays green.\n\n> Note: the SQL is generated here but run in your own GCP (no creds in CI by design). The cost model is documented in §14 and embedded as comments in the emitted SQL.\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)\n\n---\n_Generated by [Claude Code](https://claude.ai/code/session_01W6Lbeeoqy7v3wjzV43iLun)_\n\n\n\n## Summary by CodeRabbit\n\n* **New Features**\n * Added BigQuery-based discovery and ingest options for large-scale scans, including commands to generate queries and import results into the pipeline.\n * Added a faster zero-fetch sweep mode with sampling by default and an optional full scan.\n\n* **Documentation**\n * Expanded guidance with example workflows, limits, and cost/accuracy notes for large BigQuery runs.\n\n* **Bug Fixes**\n * Improved candidate deduplication and ranking so weaker matches are handled more consistently during ingest.\n\n\n\n\n\n[![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/PhysShell/OwnAudit/pull/26?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)\n\n\n\n\n> [!NOTE]\n> Currently processing new changes in this PR. This may take a few minutes, please wait...\n> \n>

\n>

⚙️ Run configuration

\n> \n> **Configuration used**: defaults\n> \n> **Review profile**: CHILL\n> \n> **Plan**: Pro Plus\n> \n> **Run ID**: `8bfa6dc8-47da-41f1-bf4d-13450e13e468`\n> \n>

\n> \n>

\n>

📥 Commits

\n> \n> Reviewing files that changed from the base of the PR and between d4fa05d977a0d3d568a81b3881a7c33f5bc98b84 and b3f045a189414fc69cf6a07b6f58a1b37657a36e.\n> \n>

\n> \n>

\n>

📒 Files selected for processing (4)

\n> \n> * `docs/leakfix-mine.md`\n> * `leakmine/bigquery.py`\n> * `leakmine/cli.py`\n> * `leakmine/tests/test_leakmine.py`\n> \n>

\n> \n> ```ascii\n> _________________________________________\n> < Fly, you fools! I'll hold off the bugs. >\n> -----------------------------------------\n> \\\n> \\ (\\__/)\n> (•ㅅ•)\n> / 　づ\n> ```\n\n\n\n\n\n

📝 Walkthrough

\n\n\n\n## Walkthrough\n\nAdds a new `leakmine/bigquery.py` module with two BigQuery SQL generators (GH-Archive PR discovery and contents sweep), metadata scoring, and NDJSON ingestion into the leakmine store. Two new CLI subcommands (`bq-sql`, `bq-ingest`) are wired into `leakmine/cli.py`, the module is exported from `__init__.py`, tests are added, and a documentation section is appended to `docs/leakfix-mine.md`.\n\n## Changes\n\n**BigQuery Discovery Arm**\n\n| Layer / File(s) | Summary |\n|---|---|\n| **SQL generators and sweep pairs**
`leakmine/bigquery.py` | Defines `SWEEP_PAIRS` (ecosystem acquire/cleanup regex pairs), `_LANGS` (language filters), `gharchive_discovery_sql` (GH-Archive merged-PR query with keyword/size/dedup), and `contents_sweep_sql` (zero-fetch syntactic sweep over repo contents tables, sample or full). |\n| **Metadata scoring and NDJSON ingestion**
`leakmine/bigquery.py` | Adds `MetaResult` dataclass with `summary_md()`, `metadata_score` (keyword + changed-files heuristic), `ingest_rows` (dedup by `(repo, number)`, score filter, optional DB write via `schema.insert_candidate`/`insert_label`), and `read_ndjson` parser. |\n| **CLI subcommands and package export**
`leakmine/__init__.py`, `leakmine/cli.py` | Exports `bigquery` from `__all__`, imports it in CLI, adds `cmd_bq_sql` and `cmd_bq_ingest` handlers, and wires `bq-sql`/`bq-ingest` subcommands with their full argument sets. |\n| **Tests**
`leakmine/tests/test_leakmine.py` | Adds `test_bq_gharchive_sql`, `test_bq_contents_sweep_sql`, `test_bq_metadata_score`, and `test_bq_ingest_ndjson` covering SQL shape, scoring behavior, deduplication, and DB storage counts. |\n| **Documentation**
`docs/leakfix-mine.md` | Adds section 14 describing GH-Archive and contents-sweep modes, the NDJSON→`bq-ingest`→per-PR-fetch pipeline flow, CLI examples, and cost/honesty caveats. |\n\n## Estimated code review effort\n\n🎯 3 (Moderate) | ⏱️ ~20 minutes\n\n## Possibly related PRs\n\n- [PhysShell/OwnAudit#24](https://github.com/PhysShell/OwnAudit/pull/24): Introduced the original leakmine pipeline, CLI scaffolding, and test structure that this PR extends with the BigQuery arm.\n\n## Poem\n\n> 🐇 A rabbit hops to BigQuery's gate,\n> SQL in paw, partition dates straight.\n> GH-Archive scanned, sweep pairs aligned,\n> NDJSON rows neatly assigned.\n> The fetch wall stands, but discovery's free —\n> metadata scored, candidates in spree! 🌟\n\n

\n\n\n\n\n

🚥 Pre-merge checks | ✅ 4 | ❌ 1

\n\n### ❌ Failed checks (1 warning)\n\n| Check name | Status | Explanation | Resolution |\n| :----------------: | :--------- | :------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------- |\n| Docstring Coverage | ⚠️ Warning | Docstring coverage is 26.32% which is insufficient. The required threshold is 80.00%. | Write docstrings for the functions missing them to satisfy the coverage threshold. |\n\n

✅ Passed checks (4 passed)

\n\n

\n\n\n\n\n

✨ Finishing Touches

\n\n

📝 Generate docstrings

\n\n- [ ] Create stacked PR\n- [ ] Commit on current branch\n\n

🧪 Generate unit tests (beta)

\n\n- [ ] Create PR with unit tests\n- [ ] Commit unit tests in branch `claude/own-language-market-research-jgrrgm`\n\n

\n\n

\n\n\n\n\n---\n\n\n\n\n_{Comment `@coderabbitai help` to get the list of available commands.}\n\n\n\n### 💡 Codex Review\n\nHere are some automated review suggestions for this pull request.\n\n**Reviewed commit:** `d4fa05d977`\n \n\n

ℹ️ About Codex in GitHub

\n
\n\n[Your team has set up Codex to review pull requests in this repo](https://chatgpt.com/codex/cloud/settings/general). Reviews are triggered when you\n- Open a pull request for review\n- Mark a draft as ready\n- Comment \"@codex review\".\n\nIf Codex has suggestions, it will comment; otherwise it will react with 👍.\n\n\n\n\nCodex can also answer questions or update the PR. Try commenting \"@codex address that feedback\".\n \n

\n**_{_{![P2 Badge](https://img.shields.io/badge/P2-yellow?style=flat)}} Filter out balanced files from contents sweep**\n\nWhen a file contains both an acquire and its cleanup, such as `addEventListener(` plus `removeEventListener`, this `WHERE` clause still admits it because it only checks for the acquire pattern; the `CASE` then falls through to `signal = 'other'`, so `bq-sql --kind contents` exports balanced files as candidate sites. On large sweeps this pollutes the zero-fetch candidate set with non-leaks; filter on `(acquire AND NOT cleanup)` for at least one pair, or exclude the `other` rows before returning results.\n\nUseful? React with 👍 / 👎.\n_📐 Maintainability & Code Quality_ | _🟡 Minor_ | _⚡ Quick win_\n\n**Fix the inline-call example.**\n\n`addEventListener(` is missing the closing `)`, which reads awkwardly and matches the static warning. Rephrase the examples as balanced call forms.\n\n

Proposed fix

\n\n```diff\n-`addEventListener(` есть, `removeEventListener` нет; `setInterval(` без\n+`addEventListener()` есть, `removeEventListener()` нет; `setInterval()` без\n```\n

\n\n\n\n

📝 Committable suggestion

\n\n> ‼️ **IMPORTANT**\n> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.\n\n```suggestion\n**B. Contents sweep (снапшот кода, zero-fetch) — `contents_sweep_sql`.**\nSQL по `bigquery-public-data.github_repos`: синтаксический тир прямо в SQL — acquire без\ncleanup (`addEventListener()` есть, `removeEventListener()` нет; `setInterval()` без\n`clearInterval`; и т.д., см. `SWEEP_PAIRS`). **Ноль fetch**, масштаб до миллионов файлов.\n**Дорого**: любое обращение к `contents.content` сканирует весь столбец ~2.7 ТБ (~$13,\nсжигает free-tier) — поэтому по умолчанию бьёт по `sample_files`/`sample_contents` (дёшево);\n`--full` включай осознанно.\n```\n\n

\n\n\n\n

🧰 Tools

\n\n

🪛 LanguageTool

\n\n[typographical] ~262-~262: Непарный символ: «)» скорей всего пропущен\nContext: ...й тир прямо в SQL — acquire без cleanup (`addEventListener(` есть, `removeEventLi...\n\n(RU_UNPAIRED_BRACKETS)\n\n

\n\n

🤖 Prompt for AI Agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nIn `@docs/leakfix-mine.md` around lines 260 - 266, The inline-call examples in the\n`contents_sweep_sql` description are unbalanced and should be rewritten as\nproper call forms. Update the examples around `SWEEP_PAIRS` to show complete\nbalanced invocations, using the existing symbols like `addEventListener`,\n`removeEventListener`, `setInterval`, and `clearInterval`, so the text reads\ncleanly and matches the static warning.\n```\n\n

\n\n\n\n\n\n\n\n_Source: Linters/SAST tools_\n\n\n_📐 Maintainability & Code Quality_ | _🟡 Minor_ | _⚡ Quick win_\n\n**Add language tags to the fenced blocks.**\n\nBoth fences currently trigger MD040. Use `text` for the flow diagram and `bash` for the CLI example so the docs render and lint cleanly.\n\n

Proposed fix

\n\n```diff\n-```\n+```text\n BigQuery (SQL) -> экспорт NDJSON -> leakmine bq-ingest -> store + candidates.json\n |\n узкий per-PR fetch + signals.classify + confirm\n-```\n+```\n@@\n-```\n+```bash\n # A) сгенерить discovery-SQL, выполнить в BigQuery, выгрузить как NDJSON:\n python3 -m leakmine.cli bq-sql --kind gharchive --ecosystem dotnet_wpf --from 20240101 --to 20241231\n # -> bq query ... | bq extract ... > rows.ndjson (в своём GCP)\n@@\n-```\n+```\n```\n

\n\n\n\n\nAlso applies to: 283-294\n\n

🧰 Tools

\n\n

🪛 markdownlint-cli2 (0.22.1)

\n\n[warning] 270-270: Fenced code blocks should have a language specified\n\n(MD040, fenced-code-language)\n\n

\n\n

🤖 Prompt for AI Agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nIn `@docs/leakfix-mine.md` around lines 270 - 274, Add explicit language tags to\nthe fenced code blocks in the docs so Markdown lint stops flagging MD040. Update\nthe diagram fence to use text and the CLI example fence to use bash, keeping the\nsurrounding content in the same sections of docs/leakfix-mine.md. Use the\nexisting fenced blocks around the BigQuery flow diagram and the bq-sql example\nto locate them, and preserve the current content while only changing the fence\nannotations.\n```\n\n

\n\n\n\n\n\n\n\n_Source: Linters/SAST tools_\n\n\n_📐 Maintainability & Code Quality_ | _🟡 Minor_ | _⚡ Quick win_\n\n**Fix the Ruff errors in this module.**\n\nLine 58 uses the ambiguous name `l`, and Lines 192/194/197/199 use multiple statements per line. Those are real Ruff violations in the added code. \n \n\n

Proposed fix

\n\n```diff\n langs = \" OR \".join(\n- f\"LOWER(JSON_EXTRACT_SCALAR(payload,'$.pull_request.base.repo.language')) = '{l}'\"\n- for l in _LANGS.get(eco_key, ())\n+ f\"LOWER(JSON_EXTRACT_SCALAR(payload,'$.pull_request.base.repo.language')) = '{lang}'\"\n+ for lang in _LANGS.get(eco_key, ())\n ) or \"TRUE\"\n@@\n tl, bl = (title or \"\").lower(), (body or \"\").lower()\n if any(k in tl for k in eco.keywords):\n- score += 3; ev.append(\"title:leak-keyword\")\n+ score += 3\n+ ev.append(\"title:leak-keyword\")\n if any(k in bl for k in eco.keywords):\n- score += 2; ev.append(\"body:leak-keyword\")\n+ score += 2\n+ ev.append(\"body:leak-keyword\")\n if changed_files is not None:\n if changed_files <= 10:\n- score += 2; ev.append(\"small-pr\")\n+ score += 2\n+ ev.append(\"small-pr\")\n elif changed_files > 200:\n- score -= 3; ev.append(\"penalty:mega-pr\")\n+ score -= 3\n+ ev.append(\"penalty:mega-pr\")\n```\n

\n\n\nAlso applies to: 191-199\n\n

🧰 Tools

\n\n

🪛 Ruff (0.15.18)

\n\n[error] 58-58: Ambiguous variable name: `l`\n\n(E741)\n\n

\n\n

🤖 Prompt for AI Agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nIn `@leakmine/bigquery.py` around lines 56 - 58, The new code in bigquery.py has\nRuff violations: rename the ambiguous loop variable used in the language filter\nconstruction to a clearer name in the logic around the langs query, and split\nthe multiple statements on each of the added lines in the helper/processing\nblock around the referenced 191-199 range into one statement per line. Use the\naffected symbols building the BigQuery filter and the nearby processing\nfunctions to update the implementation without changing behavior.\n```\n\n

\n\n\n\n\n\n\n\n_Source: Linters/SAST tools_\n\n\n\n✅ Addressed in commit b3f045a\n_🎯 Functional Correctness_ | _🟠 Major_ | _⚡ Quick win_\n\n**Filter out the non-leak `other` rows.**\n\nThe current `WHERE` only requires an acquire match, so files that also contain the matching cleanup still come back as `signal='other'`. That turns the sweep into “files with acquire calls” instead of “acquire without cleanup”, which will flood the result set with valid code. \n \n\n

Proposed fix

\n\n```diff\n cases = \"\\n\".join(\n f\" WHEN REGEXP_CONTAINS(c.content, r'{acq}') \"\n f\"AND NOT REGEXP_CONTAINS(c.content, r'{cln}') THEN '{label}'\"\n for acq, cln, label in pairs\n )\n- any_acquire = \" OR \".join(f\"REGEXP_CONTAINS(c.content, r'{acq}')\" for acq, _, _ in pairs)\n+ candidate_pred = \" OR \".join(\n+ f\"(REGEXP_CONTAINS(c.content, r'{acq}') \"\n+ f\"AND NOT REGEXP_CONTAINS(c.content, r'{cln}'))\"\n+ for acq, cln, _ in pairs\n+ )\n return (\n f\"-- LeakFixMine contents sweep: ecosystem={eco_key}, table=\"\n f\"{'sample_' if sample else ''}(files/contents). \"\n f\"{'CHEAP sample.' if sample else 'WARNING: full content scan ~2.7TB (~$13).'}\\n\"\n \"SELECT f.repo_name AS repo, f.path AS path,\\n\"\n \" CASE\\n\"\n f\"{cases}\\n\"\n- \" ELSE 'other' END AS signal\\n\"\n+ \" END AS signal\\n\"\n f\"FROM `bigquery-public-data.github_repos.{files_tbl}` f\\n\"\n f\"JOIN `bigquery-public-data.github_repos.{contents_tbl}` c USING (id)\\n\"\n f\"WHERE ({exts})\\n\"\n \" AND NOT c.binary\\n\"\n- f\" AND ({any_acquire})\\n\"\n+ f\" AND ({candidate_pred})\\n\"\n f\"LIMIT {limit}\\n\"\n )\n```\n

\n\n\n\n

📝 Committable suggestion

\n\n> ‼️ **IMPORTANT**\n> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.\n\n```suggestion\n cases = \"\\n\".join(\n f\" WHEN REGEXP_CONTAINS(c.content, r'{acq}') \"\n f\"AND NOT REGEXP_CONTAINS(c.content, r'{cln}') THEN '{label}'\"\n for acq, cln, label in pairs\n )\n candidate_pred = \" OR \".join(\n f\"(REGEXP_CONTAINS(c.content, r'{acq}') \"\n f\"AND NOT REGEXP_CONTAINS(c.content, r'{cln}'))\"\n for acq, cln, _ in pairs\n )\n return (\n f\"-- LeakFixMine contents sweep: ecosystem={eco_key}, table=\"\n f\"{'sample_' if sample else ''}(files/contents). \"\n f\"{'CHEAP sample.' if sample else 'WARNING: full content scan ~2.7TB (~$13).'}\\n\"\n \"SELECT f.repo_name AS repo, f.path AS path,\\n\"\n \" CASE\\n\"\n f\"{cases}\\n\"\n \" END AS signal\\n\"\n f\"FROM `bigquery-public-data.github_repos.{files_tbl}` f\\n\"\n f\"JOIN `bigquery-public-data.github_repos.{contents_tbl}` c USING (id)\\n\"\n f\"WHERE ({exts})\\n\"\n \" AND NOT c.binary\\n\"\n f\" AND ({candidate_pred})\\n\"\n f\"LIMIT {limit}\\n\"\n )\n```\n\n

\n\n\n\n

🤖 Prompt for AI Agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nIn `@leakmine/bigquery.py` around lines 139 - 158, The BigQuery sweep query in the\ncontent-building logic is still returning rows labeled as 'other' because the\nWHERE clause only filters for acquire matches; update the query assembled in the\nfunction that builds the SELECT so it excludes rows where the CASE falls through\nto 'other' and only returns acquire hits that do not also match the cleanup\nregex. Keep the existing pair-based CASE logic, but tighten the filter around\nthe same symbols (`cases`, `any_acquire`, and the returned SQL string) so the\nresult set contains only true leak candidates.\n```\n\n

\n\n\n\n\n\n\n\n\n_🚀 Performance & Scalability_ | _🟠 Major_ | _⚡ Quick win_\n\n**Keep the cheaper 5k default for `--kind contents`.**\n\n`contents_sweep_sql()` defaults to `limit=5000`, but this CLI hardcodes `--limit 50000` for both kinds. So `leakmine bq-sql --kind contents` silently emits a query that is 10x broader than the helper contract, and `--full` makes that the expensive path. Use a per-kind default instead of a shared 50k default. \n \n\n

Suggested fix

\n\n```diff\n def cmd_bq_sql(a) -> int:\n if a.kind == \"gharchive\":\n print(bigquery.gharchive_discovery_sql(\n a.ecosystem, date_from=getattr(a, \"from\"), date_to=a.to,\n- max_changed_files=a.max_changed_files, limit=a.limit))\n+ max_changed_files=a.max_changed_files,\n+ limit=50000 if a.limit is None else a.limit))\n else:\n- print(bigquery.contents_sweep_sql(a.ecosystem, sample=not a.full, limit=a.limit))\n+ print(bigquery.contents_sweep_sql(\n+ a.ecosystem, sample=not a.full, limit=5000 if a.limit is None else a.limit))\n return 0\n@@\n- bq.add_argument(\"--limit\", type=int, default=50000)\n+ bq.add_argument(\"--limit\", type=int)\n```\n

\n\n\nAlso applies to: 192-200\n\n

🤖 Prompt for AI Agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nIn `@leakmine/cli.py` around lines 108 - 114, The `cmd_bq_sql` CLI path is\noverriding the cheaper `contents_sweep_sql()` default by using a shared 50k\nlimit for both query kinds. Update `cmd_bq_sql` so the `gharchive` branch can\nkeep its larger limit behavior if needed, but the `contents` branch uses the\nhelper’s cheaper 5k default unless an explicit `a.limit` is provided; check the\nargument handling around `a.limit`, `a.full`, and the\n`bigquery.contents_sweep_sql` call to make the default per-kind rather than\nshared.\n```\n\n

\n\n\n\n\n\n\n\n\n_🩺 Stability & Availability_ | _🟠 Major_ | _🏗️ Heavy lift_\n\n**Stream the NDJSON ingest path instead of loading the whole export.**\n\nLine 119 reads the entire file into memory, `read_ndjson()` builds a second full list, and `ingest_rows()` then walks that list. That breaks the PR’s stated 10k–millions scale target; large BigQuery exports will hit high RSS or OOM before dedup/scoring runs. Please refactor this path so the file is consumed line-by-line and `ingest_rows` accepts an iterable.\n\n

🤖 Prompt for AI Agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nIn `@leakmine/cli.py` around lines 118 - 122, The NDJSON ingest path in\ncmd_bq_ingest currently materializes the full export before ingestion, which\nprevents scaling to large BigQuery dumps. Refactor _read/read_ndjson usage so\nthe file is consumed as a streaming iterable line-by-line, and update\nbigquery.ingest_rows to accept and process an iterator instead of a prebuilt\nlist. Keep the behavior of dedup/scoring intact while ensuring cmd_bq_ingest\npasses the streamed rows through without building an in-memory copy.\n```\n\n

\n\n\n\n\n\n\n\n\n_🎯 Functional Correctness_ | _🟡 Minor_ | _⚡ Quick win_\n\n**Assert the title/body `OR` semantics, not just field presence.**\n\nLine 462 passes as long as both field names appear anywhere in the SQL, so a regression from `...title...) OR (...body...)` to `...title...) AND (...body...)` would still be missed. Tighten this to assert the boolean fragment you actually rely on.\n\n

🤖 Prompt for AI Agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nIn `@leakmine/tests/test_leakmine.py` at line 462, The test assertion in the\nLeakmine SQL check is too weak because it only verifies that both field names\nappear somewhere in the generated query. Update the assertion in the relevant\ntest around the SQL built by the leakmine matcher to verify the actual boolean\nstructure for the title/body search, ensuring the query contains the expected OR\nfragment rather than just presence of pull_request.title and pull_request.body.\nUse the existing test helper and the SQL string from the same test to tighten\nthe check against regressions in the query builder.\n```\n\n

\n\n\n\n\n\n\n\n\n_📐 Maintainability & Code Quality_ | _🟡 Minor_ | _⚡ Quick win_\n\n**Use the penalized score in the assertion or drop the binding.**\n\n`lo` is never used here, so Ruff will keep flagging this test. The cleaner fix is to assert on the score as well as the evidence so the mega-PR penalty is covered behaviorally.\n\n

🧰 Tools

\n\n

🪛 Ruff (0.15.18)

\n\n[warning] 493-493: Unpacked variable `lo` is never used\n\nPrefix it with an underscore or any other dummy variable pattern\n\n(RUF059)\n\n

\n\n

🤖 Prompt for AI Agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nIn `@leakmine/tests/test_leakmine.py` around lines 493 - 495, The test in the\nmetadata_score call binds lo but never uses it, which will keep triggering Ruff.\nUpdate the assertion in this test to also verify the penalized score returned by\nbigquery.metadata_score alongside the existing evidence check, or remove the\nunused binding if you only intend to assert on ev2; use the metadata_score\nresult tuple in this test to cover the mega-PR penalty behaviorally.\n```\n\n

\n\n\n\n\n\n\n\n_Source: Linters/SAST tools_\n\n\n**Actionable comments posted: 8**\n\n

🤖 Prompt for all review comments with AI agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nInline comments:\nIn `@docs/leakfix-mine.md`:\n- Around line 270-274: Add explicit language tags to the fenced code blocks in\nthe docs so Markdown lint stops flagging MD040. Update the diagram fence to use\ntext and the CLI example fence to use bash, keeping the surrounding content in\nthe same sections of docs/leakfix-mine.md. Use the existing fenced blocks around\nthe BigQuery flow diagram and the bq-sql example to locate them, and preserve\nthe current content while only changing the fence annotations.\n- Around line 260-266: The inline-call examples in the `contents_sweep_sql`\ndescription are unbalanced and should be rewritten as proper call forms. Update\nthe examples around `SWEEP_PAIRS` to show complete balanced invocations, using\nthe existing symbols like `addEventListener`, `removeEventListener`,\n`setInterval`, and `clearInterval`, so the text reads cleanly and matches the\nstatic warning.\n\nIn `@leakmine/bigquery.py`:\n- Around line 56-58: The new code in bigquery.py has Ruff violations: rename the\nambiguous loop variable used in the language filter construction to a clearer\nname in the logic around the langs query, and split the multiple statements on\neach of the added lines in the helper/processing block around the referenced\n191-199 range into one statement per line. Use the affected symbols building the\nBigQuery filter and the nearby processing functions to update the implementation\nwithout changing behavior.\n- Around line 139-158: The BigQuery sweep query in the content-building logic is\nstill returning rows labeled as 'other' because the WHERE clause only filters\nfor acquire matches; update the query assembled in the function that builds the\nSELECT so it excludes rows where the CASE falls through to 'other' and only\nreturns acquire hits that do not also match the cleanup regex. Keep the existing\npair-based CASE logic, but tighten the filter around the same symbols (`cases`,\n`any_acquire`, and the returned SQL string) so the result set contains only true\nleak candidates.\n\nIn `@leakmine/cli.py`:\n- Around line 118-122: The NDJSON ingest path in cmd_bq_ingest currently\nmaterializes the full export before ingestion, which prevents scaling to large\nBigQuery dumps. Refactor _read/read_ndjson usage so the file is consumed as a\nstreaming iterable line-by-line, and update bigquery.ingest_rows to accept and\nprocess an iterator instead of a prebuilt list. Keep the behavior of\ndedup/scoring intact while ensuring cmd_bq_ingest passes the streamed rows\nthrough without building an in-memory copy.\n- Around line 108-114: The `cmd_bq_sql` CLI path is overriding the cheaper\n`contents_sweep_sql()` default by using a shared 50k limit for both query kinds.\nUpdate `cmd_bq_sql` so the `gharchive` branch can keep its larger limit behavior\nif needed, but the `contents` branch uses the helper’s cheaper 5k default unless\nan explicit `a.limit` is provided; check the argument handling around `a.limit`,\n`a.full`, and the `bigquery.contents_sweep_sql` call to make the default\nper-kind rather than shared.\n\nIn `@leakmine/tests/test_leakmine.py`:\n- Line 462: The test assertion in the Leakmine SQL check is too weak because it\nonly verifies that both field names appear somewhere in the generated query.\nUpdate the assertion in the relevant test around the SQL built by the leakmine\nmatcher to verify the actual boolean structure for the title/body search,\nensuring the query contains the expected OR fragment rather than just presence\nof pull_request.title and pull_request.body. Use the existing test helper and\nthe SQL string from the same test to tighten the check against regressions in\nthe query builder.\n- Around line 493-495: The test in the metadata_score call binds lo but never\nuses it, which will keep triggering Ruff. Update the assertion in this test to\nalso verify the penalized score returned by bigquery.metadata_score alongside\nthe existing evidence check, or remove the unused binding if you only intend to\nassert on ev2; use the metadata_score result tuple in this test to cover the\nmega-PR penalty behaviorally.\n```\n\n

\n\n

🪄 Autofix (Beta)

\n\nFix all unresolved CodeRabbit comments on this PR:\n\n- [ ] Push a commit to this branch (recommended)\n- [ ] Create a new PR with the fixes\n\n

\n\n---\n\n

ℹ️ Review info

\n\n

⚙️ Run configuration

\n\n**Configuration used**: defaults\n\n**Review profile**: CHILL\n\n**Plan**: Pro Plus\n\n**Run ID**: `f73ee075-699d-4f5c-bcf5-6cab3b76f19b`\n\n

\n\n

📥 Commits

\n\nReviewing files that changed from the base of the PR and between ca31a664ff61f654d68db49b46cb46c1cda8443a and d4fa05d977a0d3d568a81b3881a7c33f5bc98b84.\n\n

\n\n

📒 Files selected for processing (5)

\n\n* `docs/leakfix-mine.md`\n* `leakmine/__init__.py`\n* `leakmine/bigquery.py`\n* `leakmine/cli.py`\n* `leakmine/tests/test_leakmine.py`\n\n

\n\n

\n\n",{},"created",1782620114.9143612,"Completing the task","event_type","info",{"_92":403,"_96":93,"_397":1696},{"_92":645,"_646":647},"partial_repo_snapshot","files",[648,649,650,651],{"_161":1464,"_653":-5,"_654":1465,"_656":-5},{"_161":1300,"_653":-5,"_654":1301,"_656":-5},{"_161":1070,"_653":-5,"_654":1071,"_656":-5},{"_161":652,"_653":-5,"_654":655,"_656":-5},"leakmine/tests/test_leakmine.py","contents","line_range_contents",[657],"missing_reason",{"_658":659,"_660":661,"_397":662,"_663":59},"line_range_start",1,"line_range_end",530,[664,199,665,199,666,667,668,669,670,671,672,673,674,675,676,199,677,678,679,199,680,681,682,199,199,683,684,685,199,199,686,199,687,688,689,690,691,692,693,694,695,696,697,698,699,700,670,199,199,701,702,703,704,705,706,707,708,709,710,711,199,199,712,713,714,715,716,717,718,719,720,199,199,721,722,723,724,725,717,726,727,199,199,728,199,729,730,731,732,733,734,199,199,735,736,737,738,739,740,199,199,741,742,743,744,745,732,746,199,199,747,748,749,750,751,752,753,199,199,754,199,755,756,199,199,757,758,759,760,199,761,762,763,764,199,199,765,766,767,768,769,770,199,771,772,773,774,775,776,199,199,777,778,779,780,781,782,693,694,783,784,700,670,199,199,785,786,787,788,789,790,791,792,793,199,199,794,199,795,796,797,798,799,800,801,199,199,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,807,818,819,199,820,821,822,823,824,199,199,825,199,826,827,828,829,830,831,832,833,834,835,836,837,838,839,199,199,840,841,842,843,844,845,828,846,847,848,833,849,850,851,852,199,199,853,854,855,856,857,858,859,860,833,861,862,863,864,859,865,833,866,199,199,867,868,828,869,870,833,834,871,872,873,199,199,874,199,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,199,199,895,199,896,897,898,899,900,901,902,903,904,199,905,906,907,908,909,910,911,912,913,914,915,199,199,916,917,918,906,907,919,911,920,921,199,199,922,199,923,924,925,926,199,927,928,929,930,199,199,931,932,933,934,935,936,937,938,199,199,939,940,941,942,943,944,945,946,947,948,949,950,951,952,199,199,953,954,955,956,957,958,885,959,960,961,199,962,963,199,964,965,199,966,967,968,969,970,971,972,973,974,199,199,975,199,976,966,977,978,979,980,981,982,983,984,985,199,199,986,987,988,966,989,990,991,992,993,994,995,199,199,996,997,966,998,999,982,1000,1001,199,199,1002,1003,966,1004,1005,1006,1007,1008,1009,1010,199,199,1011,1012,1013,928,1014,1015,1016,1017,1018,1019,1020,1021,1022,1023,1024,199,199,1025,1026,1027,1028,1029,1030,1031,1032,1033,1034,1005,1035,1036,1009,1037,199,199,1038,1039,1040,1041,1042,1043,1044,1045,1046,1047,199,199,1048,1049,1050,1051,1052,1053,1054,1055,1056,966,1057,1058,1059,1060,1061,1062,199,199,1063,1064,1065,1066,1067,199,199,1068,1069],"contains_end_of_file","\"\"\"LeakFixMine pipeline tests (docs/leakfix-mine.md). Bare python3 or pytest:"," PYTHONPATH=. python3 leakmine/tests/test_leakmine.py","Covers the honest core: diff parsing, patch-signal classification (incl. penalties), SZZ","attribution + before/after correspondence + lead-time over a REAL temp git repo, the","verdict combiner (unique catch / unique miss), metrics aggregation, the over-vetted sweep","selection, and deterministic query/SQL generation. -O-safe (explicit raises, no bare assert).","\"\"\"","import json","import os","import sqlite3","import subprocess","import sys","import tempfile","HERE = os.path.dirname(os.path.abspath(__file__))","ROOT = os.path.dirname(os.path.dirname(HERE))","sys.path.insert(0, ROOT)","from leakmine import ( # noqa: E402"," bigquery, collect, confirm, diffparse, metrics, mine, schema, signals, sweep, szz,",")","def _expect(cond, msg):"," if not cond:"," raise AssertionError(msg)","# ---- diffparse -------------------------------------------------------------------","REACT_PATCH = \"\"\"diff --git a/src/Widget.tsx b/src/Widget.tsx","--- a/src/Widget.tsx","+++ b/src/Widget.tsx","@@ -10,5 +10,9 @@ export function Widget() {"," useEffect(() => {"," const onResize = () => setW(window.innerWidth);"," window.addEventListener('resize', onResize);","- }, []);","+ return () => {","+ window.removeEventListener('resize', onResize);","+ };","+ }, []);","+ // cleanup added"," }","def test_diffparse():"," fds = diffparse.parse_patch(REACT_PATCH)"," _expect(len(fds) == 1, \"one file diff\")"," fd = fds[0]"," _expect(fd.path == \"src/Widget.tsx\", f\"path {fd.path}\")"," added = \"\\n\".join(fd.added_text())"," _expect(\"removeEventListener\" in added, \"added removeEventListener\")"," # the fix removed old line 13 (`}, []);`) — it must register as touched."," _expect(fd.touches_old_line(13, window=0), \"old line 13 removed/replaced\")"," _expect(not fd.touches_old_line(10, window=0), \"context line 10 untouched\")"," _expect(fd.touches_old_line(11, window=2), \"window admits near line\")","def test_diffparse_body_lines_with_header_prefixes():"," # a removed SQL comment renders as \"--- ...\" at column 0; inside the hunk budget it"," # must be body content, not misread as a file header."," patch = (\"diff --git a/q.sql b/q.sql\\n--- a/q.sql\\n+++ b/q.sql\\n\""," \"@@ -1,2 +1,2 @@\\n--- old comment\\n+-- new comment\\n SELECT 1;\\n\")"," fds = diffparse.parse_patch(patch)"," _expect(len(fds) == 1, f\"one file, got {len(fds)}\")"," _expect(\"-- old comment\" in fds[0].removed_text(), \"removed comment captured as body\")"," _expect(\"-- new comment\" in fds[0].added_text(), \"added comment captured as body\")","def test_diffparse_bare_multifile():"," # bare patch (no `diff --git`): the second file's \"--- \" must start a NEW FileDiff,"," # not overwrite the first."," patch = (\"--- a/one.py\\n+++ b/one.py\\n@@ -1,1 +1,1 @@\\n-a\\n+b\\n\""," \"--- a/two.py\\n+++ b/two.py\\n@@ -1,1 +1,1 @@\\n-c\\n+d\\n\")"," _expect(len(fds) == 2, f\"two files, got {len(fds)}\")"," _expect({f.path for f in fds} == {\"one.py\", \"two.py\"}, f\"paths {[f.path for f in fds]}\")","# ---- signals ---------------------------------------------------------------------","def test_signals_react():"," cls = signals.classify(\"react_ts\", title=\"Fix memory leak in Widget\","," body=\"useEffect never removed the resize listener\", patch=REACT_PATCH)"," _expect(cls.category == signals.SUBSCRIPTION, f\"category {cls.category}\")"," _expect(cls.is_likely_fix, f\"likely fix, score={cls.score}\")"," _expect(\"title:leak-keyword\" in cls.evidence, \"title keyword scored\")","def test_signals_docs_penalty():"," docs = (\"diff --git a/README.md b/README.md\\n--- a/README.md\\n+++ b/README.md\\n\""," \"@@ -1,1 +1,2 @@\\n context\\n+a note about leaks\\n\")"," cls = signals.classify(\"react_ts\", title=\"docs: mention memory leak\", body=\"\", patch=docs)"," _expect(\"penalty:docs-only\" in cls.evidence, \"docs-only penalised\")"," _expect(not cls.is_candidate, f\"docs-only not a candidate, score={cls.score}\")","def test_signals_dotnet_event():"," patch = (\"diff --git a/A.xaml.cs b/A.xaml.cs\\n--- a/A.xaml.cs\\n+++ b/A.xaml.cs\\n\""," \"@@ -5,3 +5,4 @@\\n void Wire() {\\n- svc.Tick += OnTick;\\n\""," \"+ svc.Tick += OnTick;\\n+ Unloaded += (s,e) => svc.Tick -= OnTick;\\n }\\n\")"," cls = signals.classify(\"dotnet_wpf\", title=\"Fix event handler leak\", body=\"\", patch=patch)"," _expect(cls.is_candidate, f\"candidate, score={cls.score}\")","def test_signals_java_executor_single_form():"," # a fix that adds ONLY .shutdown() (not both forms) must still clear the threshold."," patch = (\"diff --git a/Svc.java b/Svc.java\\n--- a/Svc.java\\n+++ b/Svc.java\\n\""," \"@@ -5,2 +5,3 @@\\n void stop() {\\n+ pool.shutdown();\\n }\\n\")"," cls = signals.classify(\"java_spring\", title=\"fix ExecutorService memory leak\", body=\"\", patch=patch)"," _expect(cls.category == signals.TASK, f\"category {cls.category}\")"," _expect(cls.is_candidate, f\"single-form shutdown is a candidate, score={cls.score}\")","# ---- szz attribution + correspondence --------------------------------------------","def _finding(tool, rule, file, line, **kw):"," return szz.Finding(tool=tool, rule=rule, file=file, line=line, **kw)","def test_attribution_causal_vs_coincidental():"," f_on_fix = _finding(\"ownaudit\", \"OWN-EFFECT\", \"src/Widget.tsx\", 13)"," att = szz.attribute(REACT_PATCH, f_on_fix, window=0)"," _expect(att.fix_touches, \"finding on removed line is causal\")"," f_elsewhere = _finding(\"eslint\", \"no-leak\", \"src/Widget.tsx\", 200)"," att2 = szz.attribute(REACT_PATCH, f_elsewhere, window=2)"," _expect(not att2.fix_touches, \"far-away finding not causal\")"," _expect(att2.in_changed_file, \"but still in the changed file\")","def test_correspondence_confirmed_vs_lucky():"," target = _finding(\"ownaudit\", \"OWN-EFFECT\", \"src/Widget.tsx\", 13, resolution=\"semantic\")"," before = [target]"," after = [] # gone after"," corr = szz.correspond(before, after, REACT_PATCH, target, window=0)"," _expect(corr.confirmed_catch, \"flagged before, gone after, causal -> confirmed\")"," # coincidental: finding far from the fix also 'disappears' (e.g. file reshuffled)."," lucky = _finding(\"eslint\", \"x\", \"src/Widget.tsx\", 999)"," corr2 = szz.correspond([lucky], [], REACT_PATCH, lucky, window=2)"," _expect(corr2.detected_before and corr2.gone_after, \"before/after set\")"," _expect(not corr2.causal, \"not on fixed lines\")"," _expect(not corr2.confirmed_catch, \"gone-but-not-causal is NOT a confirmed catch\")","RENAME_PATCH = \"\"\"diff --git a/src/Old.tsx b/src/New.tsx","rename from src/Old.tsx","rename to src/New.tsx","--- a/src/Old.tsx","+++ b/src/New.tsx","@@ -10,3 +10,4 @@ export function Widget() {","+ }, []); // touched, but the leak is NOT actually fixed","+ // (rule still fires on the new path below)","def test_correspondence_rename_not_confirmed():"," # fix renames Old.tsx -> New.tsx and edits the line, but the SAME rule still fires on"," # the renamed file. That is a file-move, not a fix: it must NOT count as confirmed."," before = [_finding(\"ownaudit\", \"OWN-EFFECT\", \"src/Old.tsx\", 11)]"," after = [_finding(\"ownaudit\", \"OWN-EFFECT\", \"src/New.tsx\", 11)] # persists under new path"," corr = szz.correspond(before, after, RENAME_PATCH, before[0], window=2)"," _expect(corr.detected_before, \"flagged before\")"," _expect(not corr.gone_after, \"rule still fires on the renamed path -> not gone\")"," _expect(not corr.confirmed_catch, \"rename+persisting finding is NOT a confirmed catch\")","# ---- lead-time over a real temp git repo -----------------------------------------","def _git(repo, *args, date=\"2024-01-01T00:00:00\"):"," env = {**os.environ,"," \"GIT_AUTHOR_NAME\": \"t\", \"GIT_AUTHOR_EMAIL\": \"t@t\","," \"GIT_COMMITTER_NAME\": \"t\", \"GIT_COMMITTER_EMAIL\": \"t@t\","," \"GIT_AUTHOR_DATE\": date, \"GIT_COMMITTER_DATE\": date}"," return subprocess.run([\"git\", \"-C\", repo, *args], env=env,"," capture_output=True, text=True, check=True)","def test_lead_time():"," with tempfile.TemporaryDirectory() as repo:"," _git(repo, \"init\", \"-q\", \"-b\", \"main\")"," path = os.path.join(repo, \"svc.cs\")"," # commit 1: introduce a leaking line."," with open(path, \"w\") as fh:"," fh.write(\"class S {\\n void M() { Tick += OnTick; }\\n}\\n\")"," _git(repo, \"add\", \".\")"," _git(repo, \"commit\", \"-qm\", \"intro\", date=\"2024-01-01T00:00:00\")"," leak_sha = _git(repo, \"rev-parse\", \"HEAD\").stdout.strip()"," # a couple of unrelated commits."," for i, d in enumerate((\"2024-02-01T00:00:00\", \"2024-03-01T00:00:00\")):"," with open(os.path.join(repo, f\"f{i}.txt\"), \"w\") as fh:"," fh.write(\"x\\n\")"," _git(repo, \"add\", \".\"); _git(repo, \"commit\", \"-qm\", f\"noise{i}\", date=d)"," # the human fix: change line 2."," fh.write(\"class S {\\n void M() { Tick += OnTick; Unloaded += () => Tick -= OnTick; }\\n}\\n\")"," _git(repo, \"add\", \".\"); _git(repo, \"commit\", \"-qm\", \"fix leak\", date=\"2024-04-01T00:00:00\")"," lt = szz.lead_time(repo, leak_sha, \"svc.cs\", 2)"," _expect(lt.found, \"fix located in history\")"," # 2 intervening commits (noise0, noise1); the fix commit itself is excluded."," _expect(lt.commits_between == 2, f\"2 intervening commits, got {lt.commits_between}\")"," _expect(lt.fix_date.startswith(\"2024-04-01\"), f\"fix date {lt.fix_date}\")","# ---- confirm (verdict combiner) --------------------------------------------------","def test_judge_unique_catch_and_miss():"," own_before = [_finding(\"ownaudit\", \"OWN-EFFECT\", \"src/Widget.tsx\", 13, resolution=\"interproc\")]"," cand = confirm.Candidate("," id=\"c1\", ecosystem=\"react_ts\", title=\"fix memory leak\", body=\"listener leak\","," patch=REACT_PATCH,"," before={\"ownaudit\": own_before, \"eslint\": []},"," after={\"ownaudit\": [], \"eslint\": []},"," )"," v = confirm.judge(cand, ownaudit_tool=\"ownaudit\", baseline_tools=(\"eslint\",))"," _expect(v.is_real_fix, \"real fix\")"," _expect(\"ownaudit\" in v.caught_by, \"ownaudit caught\")"," _expect(v.unique_to_ownaudit, \"unique to ownaudit (eslint missed)\")"," _expect(\"eslint\" in v.missed_by, \"eslint in missed_by\")"," _expect(v.own_resolution == \"interproc\", f\"resolution {v.own_resolution}\")","def test_judge_tool_catch_does_not_define_real_fix():"," # a borderline patch (candidate but score < 10) that a tool confirm-catches must NOT be"," # promoted to a \"real fix\" — ground truth is the patch signal, not the tool's catch."," patch = (\"diff --git a/a.tsx b/a.tsx\\n--- a/a.tsx\\n+++ b/a.tsx\\n\""," \"@@ -1,3 +1,3 @@\\n function setup() {\\n- window.addEventListener('x', f);\\n\""," \"+ window.removeEventListener('x', f);\\n }\\n\")"," id=\"b1\", ecosystem=\"react_ts\", title=\"improve effect cleanup\", body=\"\", patch=patch,"," before={\"ownaudit\": [_finding(\"ownaudit\", \"OWN-X\", \"a.tsx\", 2)]},"," after={\"ownaudit\": []},"," v = confirm.judge(cand, ownaudit_tool=\"ownaudit\")"," _expect(\"ownaudit\" in v.caught_by, \"catch still recorded\")"," _expect(not v.is_real_fix, \"tool catch must not define a real fix\")"," _expect(\"borderline-send-to-review\" in v.notes, \"routed to review instead\")","def test_fp_after_matches_rule_and_file():"," patch = (\"diff --git a/a.cs b/a.cs\\n--- a/a.cs\\n+++ b/a.cs\\n\""," \"@@ -5,2 +5,3 @@\\n void M() {\\n+ DoThing();\\n }\\n\")"," # same (rule, file) survives on a touched file -> precision smell."," same = confirm.Candidate("," id=\"f1\", ecosystem=\"dotnet_wpf\", title=\"t\", body=\"\", patch=patch,"," before={\"ownaudit\": [_finding(\"ownaudit\", \"R1\", \"a.cs\", 5)]},"," after={\"ownaudit\": [_finding(\"ownaudit\", \"R1\", \"a.cs\", 5)]},"," _expect(confirm._fp_after(same, \"ownaudit\"), \"surviving same rule+file is a smell\")"," # a DIFFERENT rule on the same file must not count."," diff = confirm.Candidate("," id=\"f2\", ecosystem=\"dotnet_wpf\", title=\"t\", body=\"\", patch=patch,"," after={\"ownaudit\": [_finding(\"ownaudit\", \"R2\", \"a.cs\", 9)]},"," _expect(not confirm._fp_after(diff, \"ownaudit\"), \"unrelated rule must not inflate fp_after\")","def test_judge_unique_miss():"," # a real fix (strong patch signal) that NO tool caught -> the precious blind-spot bucket."," id=\"c2\", ecosystem=\"react_ts\", title=\"fix memory leak\", body=\"listener leak\","," patch=REACT_PATCH, before={\"ownaudit\": [], \"eslint\": []}, after={\"ownaudit\": [], \"eslint\": []},"," _expect(v.is_real_fix, \"still a real fix by patch signal\")"," _expect(not v.caught_by, \"no tool caught it\")"," _expect(\"real-fix-no-tool-caught\" in v.notes, \"flagged as shared blind spot\")","# ---- metrics ---------------------------------------------------------------------","def test_metrics_aggregate():"," vs = ["," confirm.Verdict(\"a\", \"react_ts\", \"subscription-leak\", 12, True,"," caught_by=[\"ownaudit\"], missed_by=[\"eslint\"], unique_to_ownaudit=True,"," own_resolution=\"interproc\"),"," confirm.Verdict(\"b\", \"react_ts\", \"timer-leak\", 11, True,"," caught_by=[\"ownaudit\", \"eslint\"], unique_to_ownaudit=False,"," own_resolution=\"syntactic\"),"," confirm.Verdict(\"c\", \"react_ts\", \"subscription-leak\", 9, True, caught_by=[],"," missed_by=[\"ownaudit\", \"eslint\"], notes=[\"real-fix-no-tool-caught\"]),"," ]"," rep = metrics.aggregate(vs, ownaudit_tool=\"ownaudit\", baseline_tools=(\"eslint\",))"," _expect(rep.n_real_fixes == 3, \"3 real fixes\")"," _expect(rep.catches[\"ownaudit\"] == 2, \"ownaudit caught 2\")"," _expect(rep.unique_to_ownaudit == 1, \"1 unique to ownaudit\")"," _expect(rep.unique_miss == 1, \"1 shared blind spot\")"," _expect(abs(rep.recall_on_corpus[\"ownaudit\"] - 2 / 3) < 1e-3, \"ownaudit recall 2/3\")"," _expect(rep.by_tier.get(\"interproc\") == 1 and rep.by_tier.get(\"syntactic\") == 1, \"tier split\")"," md = metrics.render_markdown(rep)"," _expect(\"LeakFixMine\" in md and \"by analysis tier\" in md, \"markdown renders\")","# ---- sweep (over-vetted selection) -----------------------------------------------","def test_sweep_over_vetted():"," polished = sweep.Package(\"Serilog\", \"nuget\", downloads=10**8, stars=6000, maintainers=8,"," open_issues=10, age_days=3000, has_ci=True, shape=\"library\")"," appish = sweep.Package(\"acme-dashboard\", \"npm\", downloads=10**6, stars=400, maintainers=2,"," open_issues=120, age_days=400, has_ci=False, shape=\"application\")"," sp = sweep.over_vetted_score(polished)"," sa = sweep.over_vetted_score(appish)"," _expect(sp.vetted > sa.vetted, \"polished lib scores more vetted\")"," _expect(\"high-stars-per-issue\" in sp.reason, \"stars/issue signal fires\")"," # many polished libs + a few apps; cap must let apps through despite higher downloads."," pkgs = [sweep.Package(f\"lib{i}\", \"nuget\", downloads=10**8, stars=8000, maintainers=9,"," open_issues=5, age_days=3000, has_ci=True, shape=\"library\")"," for i in range(8)]"," pkgs += [appish, sweep.Package(\"ctrl-plane\", \"npm\", downloads=5 * 10**5, stars=200,"," open_issues=90, age_days=300, shape=\"application\")]"," chosen = sweep.select_targets(pkgs, n=5, max_vetted_fraction=0.4)"," names = {c.pkg.name for c in chosen}"," n_vetted = sum(1 for c in chosen if c.vetted >= 0.6)"," _expect(n_vetted <= 2, f\"vetted cap (<=40% of 5) respected, got {n_vetted}\")"," _expect(names & {\"acme-dashboard\", \"ctrl-plane\"}, \"an app-shaped repo got in\")","def test_sweep_underfill_is_intentional():"," # an all-vetted pool can't fill n without breaching the cap: underfill is intentional,"," # NOT a backfill with the very libs the cap excludes."," for i in range(10)]"," _expect(len(chosen) == 2, f\"hard cap -> 2 of 5 (intentional underfill), got {len(chosen)}\")"," _expect(all(c.vetted >= 0.6 for c in chosen), \"the two admitted are the vetted libs\")","# ---- collect (deterministic query/SQL gen) ---------------------------------------","def test_queries_and_sql():"," qs = collect.github_search_queries(\"react_ts\", merged_after=\"2023-01-01\")"," _expect(all(\"merged:>=2023-01-01\" in q for q in qs), \"date qualifier appended\")"," _expect(any(\"useEffect\" in q for q in qs), \"useEffect query present\")"," sql = collect.gharchive_sql(\"dotnet_wpf\", date_from=\"20240101\", date_to=\"20241231\")"," _expect(\"_TABLE_SUFFIX BETWEEN '20240101' AND '20241231'\" in sql, \"partition-scoped\")"," _expect(\"PullRequestEvent\" in sql and \"memory leak\" in sql, \"PR + keyword filter\")"," _expect(\"pull_request.title\" in sql and \"pull_request.body\" in sql, \"title AND body matched\")","def test_fetch_search_with_fake_http():"," fake = json.dumps({\"items\": ["," {\"number\": 7, \"title\": \"fix leak\", \"html_url\": \"https://x/pr/7\","," \"repository_url\": \"https://api.github.com/repos/acme/widget\", \"pull_request\": {}},"," ]}).encode()"," got = collect.fetch_search(\"q\", http=lambda url, token: fake)"," _expect(len(got) == 1 and got[0].kind == \"pr\", \"parsed one PR\")"," _expect(got[0].repo == \"acme/widget\", f\"repo parsed: {got[0].repo}\")","def test_fetch_patch_with_fake_http():"," calls = {}"," def fake_http(url, token):"," calls[\"url\"] = url"," calls[\"token\"] = token"," return \"diff --git a/a b/a\\n@@ -1 +1 @@\\n-x\\n+y\\n\""," out = collect.fetch_patch(\"acme/w\", 7, token=\"T\", http=fake_http)"," _expect(out.startswith(\"diff --git\"), \"returns diff text\")"," _expect(calls[\"url\"].endswith(\"/repos/acme/w/pulls/7\"), f\"url {calls['url']}\")"," _expect(calls[\"token\"] == \"T\", \"token threaded through\")"," # a fetch error must degrade to \"\" (one bad PR can't abort a mining run), not raise."," def boom(url, token):"," raise RuntimeError(\"network\")"," _expect(collect.fetch_patch(\"a/b\", 1, http=boom) == \"\", \"fetch error -> empty string\")","def test_mine_run_orchestration():"," cands = ["," collect.Candidate(\"\", \"acme/w\", 1, \"pr\", \"fix memory leak\", \"u1\", \"listener leak\"),"," collect.Candidate(\"\", \"acme/w\", 1, \"pr\", \"dup of #1\", \"u1\", \"\"), # dedup by (repo,#)"," collect.Candidate(\"\", \"acme/x\", 2, \"pr\", \"chore: tidy\", \"u2\", \"\"), # weak -> dropped"," collect.Candidate(\"\", \"acme/y\", 3, \"issue\", \"an issue\", \"u3\", \"\"), # not a PR -> skip"," weak = (\"diff --git a/x.tsx b/x.tsx\\n--- a/x.tsx\\n+++ b/x.tsx\\n\""," \"@@ -1,1 +1,1 @@\\n-const a = 1;\\n+const a = 2;\\n\")"," patches = {(\"acme/w\", 1): REACT_PATCH, (\"acme/x\", 2): weak}"," def fake_search(query, *, token=\"\", per_page=50):"," return cands # query pack repeats the same hits; dedup must collapse them"," def fake_patch(repo, number, *, token=\"\"):"," return patches.get((repo, number), \"\")"," conn = schema.connect(\":memory:\")"," res = mine.run(\"react_ts\", search=fake_search, fetch_patch=fake_patch, conn=conn, min_score=7)"," _expect(res.seen == 2, f\"PR-only + deduped examined, got {res.seen}\")"," _expect(res.kept == 1, f\"only the real leak fix kept, got {res.kept}\")"," _expect(res.rows and res.rows[0][\"repo\"] == \"acme/w\", \"kept the leak fix\")"," _expect(res.rows[0][\"category\"] == signals.SUBSCRIPTION, \"classified as subscription\")"," _expect(schema.count(conn, \"candidates\") == 1, \"kept candidate stored\")"," _expect(schema.count(conn, \"labels\") == 1, \"kept label stored\")"," _expect(\"mining run\" in res.summary_md(), \"summary renders\")","# ---- schema (store round-trip) ---------------------------------------------------","def test_schema_store():"," schema.insert_candidate(conn, {\"id\": \"c1\", \"ecosystem\": \"react_ts\", \"repo\": \"a/b\","," \"number\": 1, \"kind\": \"pr\", \"title\": \"fix leak\", \"merged\": 1})"," schema.insert_label(conn, \"c1\", \"subscription-leak\", 12, [\"patch:subscription-leak\"], \"patch\")"," v = confirm.Verdict(\"c1\", \"react_ts\", \"subscription-leak\", 12, True,"," caught_by=[\"ownaudit\"], unique_to_ownaudit=True)"," schema.insert_verdict(conn, v)"," _expect(schema.count(conn, \"candidates\") == 1, \"candidate stored\")"," _expect(schema.count(conn, \"labels\") == 1, \"label stored\")"," _expect(schema.count(conn, \"verdicts\") == 1, \"verdict stored\")","def test_candidate_upsert_preserves_children():"," # re-inserting a candidate must UPDATE in place, not DELETE+reinsert — otherwise (with"," # FK on) it would orphan/cascade the child label below."," schema.insert_candidate(conn, {\"id\": \"c1\", \"ecosystem\": \"react_ts\", \"title\": \"old\", \"merged\": 1})"," schema.insert_label(conn, \"c1\", \"subscription-leak\", 12, [\"e\"], \"patch\")"," schema.insert_candidate(conn, {\"id\": \"c1\", \"ecosystem\": \"react_ts\", \"title\": \"new\", \"merged\": 1})"," _expect(schema.count(conn, \"candidates\") == 1, \"still one candidate\")"," _expect(schema.count(conn, \"labels\") == 1, \"child label survived the re-insert\")"," title = conn.execute(\"SELECT title FROM candidates WHERE id='c1'\").fetchone()[0]"," _expect(title == \"new\", f\"row updated in place, title={title}\")","def test_schema_verdict_idempotent():"," # re-confirming the same candidate upserts, so resume never double-counts verdicts."," schema.insert_candidate(conn, {\"id\": \"c1\", \"ecosystem\": \"react_ts\", \"title\": \"t\", \"merged\": 1})"," v = confirm.Verdict(\"c1\", \"react_ts\", \"subscription-leak\", 12, True, caught_by=[\"ownaudit\"])"," schema.insert_verdict(conn, v) # second confirm of the same candidate"," _expect(schema.count(conn, \"verdicts\") == 1, \"verdict upserts, not duplicated\")","def test_schema_foreign_keys_enforced():"," # FK enforcement is ON: a verdict for an unknown candidate must be rejected."," v = confirm.Verdict(\"ghost\", \"react_ts\", \"subscription-leak\", 12, True)"," try:"," schema.insert_verdict(conn, v)"," raised = False"," except sqlite3.IntegrityError: # specifically the FK rejection, not any error"," raised = True"," _expect(raised, \"orphan verdict rejected by foreign-key constraint\")","def test_bq_gharchive_sql():"," sql = bigquery.gharchive_discovery_sql(\"dotnet_wpf\", date_from=\"20240101\", date_to=\"20241231\","," max_changed_files=80)"," # assert the actual OR boolean, not mere field presence (an AND regression must fail)."," _expect((\"title')) LIKE '%memory leak%' OR \""," \"LOWER(COALESCE(JSON_EXTRACT_SCALAR(payload,'$.pull_request.body'),'')) \""," \"LIKE '%memory leak%'\") in sql, \"title OR body, not AND\")"," _expect(\"base.repo.language')) = 'c#'\" in sql, \"language filter\")"," _expect(\"changed_files') AS INT64) <= 80\" in sql, \"size cap applied\")"," _expect(\"QUALIFY ROW_NUMBER()\" in sql, \"deduped to one row per PR\")"," # size cap can be disabled."," nocap = bigquery.gharchive_discovery_sql(\"react_ts\", date_from=\"20240101\", date_to=\"20240131\","," max_changed_files=0)"," _expect(\"changed_files') AS INT64) <=\" not in nocap, \"cap omitted when 0\")","def test_bq_contents_sweep_sql():"," s = bigquery.contents_sweep_sql(\"react_ts\", sample=True)"," _expect(\"sample_files\" in s and \"sample_contents\" in s, \"uses cheap sample tables by default\")"," _expect(\"REGEXP_CONTAINS(c.content, r'addEventListener\\\\(')\" in s, \"acquire regex present\")"," _expect(\"NOT REGEXP_CONTAINS(c.content, r'removeEventListener')\" in s, \"cleanup-absence test\")"," full = bigquery.contents_sweep_sql(\"react_ts\", sample=False)"," _expect(\"github_repos.files`\" in full and \"github_repos.contents`\" in full, \"full tables\")"," _expect(\"2.7TB\" in full, \"full scan is cost-warned\")"," # an ecosystem without sweep pairs raises rather than emit garbage."," raised = False"," bigquery.contents_sweep_sql(\"nim\")"," except ValueError:"," _expect(raised, \"missing sweep pairs -> explicit error\")","def test_bq_metadata_score():"," hi, ev = bigquery.metadata_score(\"dotnet_wpf\", title=\"Fix memory leak in view\","," body=\"event handler retained\", changed_files=4)"," _expect(hi >= 4 and \"small-pr\" in ev, f\"focused leak fix ranks high, {hi}\")"," lo, ev2 = bigquery.metadata_score(\"dotnet_wpf\", title=\"Fix memory leak\", body=\"\","," changed_files=500)"," # +3 title keyword, -3 mega penalty -> 0; assert the score, not just the evidence."," _expect(lo == 0 and \"penalty:mega-pr\" in ev2, f\"mega-PR penalty zeroes the score, {lo}\")"," none, _ = bigquery.metadata_score(\"dotnet_wpf\", title=\"add feature\", body=\"\", changed_files=3)"," _expect(none < 4, \"no-keyword PR below threshold\")","def test_bq_ingest_ndjson():"," ndjson = \"\\n\".join(["," json.dumps({\"repo\": \"a/b\", \"number\": 1, \"title\": \"Fix memory leak\", \"body\": \"x\","," \"changed_files\": 3, \"html_url\": \"u1\"}),"," json.dumps({\"repo\": \"a/b\", \"number\": 1, \"title\": \"dup\", \"body\": \"\", \"changed_files\": 3}),"," json.dumps({\"repo\": \"c/d\", \"number\": 9, \"title\": \"add feature\", \"body\": \"\","," \"changed_files\": 2}), # no keyword -> dropped"," ])"," rows = bigquery.read_ndjson(ndjson)"," res = bigquery.ingest_rows(rows, \"dotnet_wpf\", min_meta_score=4, conn=conn)"," _expect(res.seen == 2, f\"deduped by (repo,number), got {res.seen}\")"," _expect(res.kept == 1, f\"only the keyword'd small PR kept, got {res.kept}\")"," _expect(res.rows[0][\"repo\"] == \"a/b\", \"kept the leak PR\")"," _expect(schema.count(conn, \"candidates\") == 1 and schema.count(conn, \"labels\") == 1, \"stored\")"," _expect(\"metadata tier\" in res.summary_md(), \"summary flags the tier\")","def main():"," tests = [v for k, v in sorted(globals().items()) if k.startswith(\"test_\") and callable(v)]"," for t in tests:"," t()"," print(f\"ok - {len(tests)} leakmine tests passed\")","if __name__ == \"__main__\":"," main()","docs/leakfix-mine.md",[1072],{"_658":659,"_660":1073,"_397":1074,"_663":59},302,[1075,199,1076,1077,1078,1079,1080,199,1081,199,1082,199,1083,1084,1085,1086,199,1087,1088,199,1089,199,1090,1091,1092,1093,199,1094,1095,1096,1097,1098,199,1099,1100,199,1101,199,1102,1103,1104,1105,199,1106,199,1107,1108,199,1109,1110,1111,1112,1113,1114,1115,1109,199,1116,1117,1118,1119,1120,1121,199,1122,199,1123,1124,199,1125,1126,1127,1128,1129,1130,1131,1132,1133,199,1134,1135,1136,199,1137,199,1138,1139,199,1140,1141,1142,199,1143,1144,1145,1146,199,1109,1147,1109,199,1148,1149,1150,1151,1152,199,1153,1154,1155,199,1156,199,1157,1158,1159,1160,1161,199,1162,199,1163,199,1164,1165,1166,1167,1168,1169,1170,1171,199,1172,199,1173,1174,199,1175,1165,1176,1177,1178,199,1179,1180,1181,1182,1183,1184,1185,1186,1187,199,1188,199,1189,1190,1191,199,1192,1193,1194,199,1195,1196,1197,1198,1199,1200,1201,199,1202,1203,199,1204,199,1205,1206,1207,1208,1209,1210,199,1211,199,1109,1212,1213,1214,1215,1216,1217,1218,1109,199,1219,1220,199,1221,199,1222,1223,1224,1225,1226,1227,1228,199,1229,199,1230,1231,1232,1233,1234,1235,199,1236,199,1237,1238,1239,1240,1241,1242,199,1243,1109,1244,1245,1109,199,1246,1247,1248,1249,199,1250,1251,1252,1253,1254,199,1255,199,1256,1257,1258,1259,1260,199,1261,199,1262,1263,1264,1265,1266,199,1267,1268,1269,1270,1271,1272,1273,199,1274,199,1275,1276,1277,1278,1109,199,1279,1280,1281,1282,199,1283,199,1284,1285,1286,1287,199,1288,1289,1290,199,1291,1292,1109,199,1293,1294,1295,1296,1297,1298,1299],"# LeakFixMine — методика майнинга leak-фиксов и честной оценки OwnAudit","Статус: реализованный research-пайплайн (`leakmine/`) + методика. Код — чистый stdlib,","тесты офлайн (`leakmine/tests/test_leakmine.py`). Это **исследовательская рука** датасет-","истории (см. `audit-data-leverage.md §3`): корпус из исправленных багов — это","регрессионный набор и материал для статьи, **а не** знаменатель recall. Поэтому половина","работы тут — отделять сигнал от везения.","## 0. Зачем","Два результата за одну работу:","1. **Практика для OwnAudit** — реальный корпус «было/стало», регрессионные тесты,"," приоритизация правил по данным, таблица «мы vs baseline-линтеры».","2. **Публикация** — «прогнали на N реальных leak-фиксах в .NET/React/Android/Java;"," baseline ловит X%, OwnAudit — Y%, и Z уникальных, с объяснением пути удержания».","Главный принцип: **не переобещать**. Цифры ниже честны ровно в тех границах, в которых их","можно защитить.","## 1. Три независимых эксперимента (не один)","Корпус исправленных багов смещён by construction — в него попадают только баги, которые","(а) заметили, (б) диагностировали, (в) починили и (г) описали leak-словами. Поэтому recall","по нему — это recall **по корпусу**, а не «по всем течам». Чтобы закрыть смещение, у нас","три дизайна, и они дополняют друг друга:","| Дизайн | Что меряет | Защита от смещения | Модули |","|---|---|---|---|","| **Fixed-bug corpus** | regression-recall, уникальные находки | SZZ-привязка + before/after | `collect` → `signals` → `szz` → `confirm` → `metrics` |","| **Prospective sweep** | precision в дикой природе | сэмплируем *весь живой код*, не «уже починенное» | `sweep` |","| **Time-travel** | lead time («поймали бы раньше людей») | смотрим вперёд по истории | `szz.lead_time` |","Самый сильный для adoption — **prospective sweep**: он уже сработал в зародыше на OwnTS","(реальные useEffect-течки в популярных npm). Fixed-bug corpus понижен до роли регрессий.","## 2. Хранилище","Один SQLite-файл (`schema.py`), пять таблиц-спина: `candidates → patches → labels →","tool_runs → verdicts`. Находки и вердикты лежат JSON-блобами (они вложенные), реляционная","часть — только чтобы возобновлять прогон и джойнить стадии. SQLite, не DuckDB — stdlib, CI","ничего не ставит.","## 3. Классификация фиксов — дёшево и детерминированно сначала","Сначала **не** LLM. Сначала patch-сигналы (`signals.py`): что патч *добавил* и *удалил*.","Скоринг как в плане:","```","+3 leak-keyword в title, +2 в body","+weight за каждый сматченный patch-сигнал (настоящая улика)","+2 changed-file с релевантным расширением","-3 только docs, -4 только bump зависимостей","score >= 7 -> кандидат","score >= 10 -> вероятный фикс","Категории — **тот же таксон, что в `report/sarif.py`** (`subscription-leak`,","`idisposable-leak`, `timer-leak`, …), чтобы майненый фикс и живая находка лежали в одних","бакетах. Сигналы заведены по экосистемам: .NET/WPF, React/TS, Android/Kotlin, Java/Spring,","плюс appendix Zig/Nim. LLM подключается **после** patch-сигналов и только на borderline","(7 ≤ score < 10), и всё равно с ручной выборкой — иначе получится «датасет, размеченный","воображением».","## 4. Сбор кандидатов — два бэкенда по масштабу","`collect.py` генерирует запросы **детерминированно** (тестируются офлайн), сам fetch","изолирован за инъектируемым HTTP-геттером (CI в сеть не ходит).","- **GitHub Search API** — норм для первого среза, но душит: ~30 req/min, hard-cap 1000"," результатов на запрос, и `language:` — это язык **репозитория**, не диффа. Поэтому язык"," диффа **обязательно** перепроверяем по changed-files (расширения из query-pack).","- **GH Archive** (`gharchive.org`, зеркало — публичный BigQuery-датасет `githubarchive`) —"," весь firehose событий. Фильтруем PR/issue по leak-ключевикам прямо в SQL, без rate-limit."," Данные **открыты** (качай .gz и парь локально бесплатно); запрос через BigQuery требует"," GCP-аккаунта (free tier 1 TB/мес), поэтому SQL **всегда** скоупит по дневной партиции"," (`_TABLE_SUFFIX BETWEEN`) — разница между «пара центов» и «съели весь терабайт». GH"," Archive заменяет **только discovery** — патч всё равно тянется per-PR через API/clone.","Связка issue↔PR: `linked:issue` покрывает плохо — надёжнее парсить `Fixes/Closes #N` из","тела PR и GraphQL `closingIssuesReferences`. На практике начинаем **с PR**, потому что нам","нужен патч.","## 5. Доказать, что это был фикс, а не везение (ядро)","`szz.py` реализует **SZZ-критерий** (Śliwerski–Zimmermann–Zeller, 2005, *When do changes","induce fixes?*). Две ловушки, которые без него убивают корпус:","1. находка baseline-тула «исчезла после фикса» по **неправильной** причине — файл"," переехал/переименован/массово переформатирован, а не течь закрыли;","2. PR с заголовком «fix memory leak» чинит другое, а течь жива.","Защита: находка на `(file, line)` pre-fix ревизии **причинно** связана с фиксом только если","фикс-дифф **удалил или заменил эту строку**. Мы накладываем находку на removed-line-диапазоны","патча (`diffparse.touches_old_line`), поэтому «gone after» засчитывается **только** когда","это «gone *потому что изменились именно те строки*».","confirmed_catch = detected_before AND gone_after AND causal","- `detected_before` — тул пометил на pre-fix ревизии;","- `gone_after` — той же находки (tool, rule, file) нет на post-fix (по строке не матчим —"," она легально съезжает);","- `causal` — SZZ-пересечение: фикс тронул именно строку находки (с маленьким `window` под"," off-by-N; широкий `window` возвращает то самое везение).","`window` держим тесным сознательно. Для тяжёлого (трекинг переименований, отделение","рефакторинга от поведения) методически опираемся на **PyDriller + RefactoringMiner**; в коде","оставляем stdlib + `git`, чтобы CI был герметичным.","### Time-travel / lead time","`szz.lead_time(repo, leak_sha, file, line)`: от течи, которую OwnAudit пометил на старой","ревизии, идём **вперёд** по истории (`git log -L` по одной строке, фильтр по потомкам","`leak_sha`) до первого коммита, который тронул эту строку — это человеческий фикс. Отдаём","`commits_between` и дату. Маркетинговый payoff: «OwnAudit пометил бы это за N коммитов /","D дней до фикса».","## 6. Метрики — и что они честно НЕ значат","`metrics.py`, всё с оговорками:","| Метрика | Что говорит | Чем **не** является |","|---|---|---|","| `recall_on_corpus` (tool) | caught / real-fixes **в этом корпусе** | НЕ recall по «всем течам» — корпус смещён |","| `unique_to_ownaudit` | реальные фиксы, что поймал OwnAudit и не поймал ни один baseline | заголовочное число |","| `unique_miss` | реальные фиксы, что **никто** не поймал | общая слепая зона — следующий раунд работы |","| `fp_after_rate` (tool) | тул всё ещё фаерит на починенном файле | precision **smell**, не precision |","| `by_tier` | catches по тиру анализа (см. §7) | сколько ловится без сборки |","| `by_category` | распределение по таксону | — |","## 7. Тир анализа = ось зависимости от сборки (важная поправка)","Мы **статический** анализ, полная сборка проекта не нужна. Но честная граница тоньше, чем","«собирать/не собирать», поэтому каждая находка несёт `resolution`:","| Тир | Что нужно | Примеры |","| `syntactic` | сырой текст/AST, **ноль** сборки и референсов | `+=` без `-=`, `addEventListener` без `removeEventListener`, `setInterval` без `clearInterval` |","| `semantic` | разрешение символов / референс-сборки | «это *реально* `IDisposable`», «это именно event-подписка», DI lifetime |","| `interproc` | межпроцедурное ownership | retention path, XAML↔C# join — **край OwnAudit** |","Следствия:","- `by_tier` отвечает на вопрос «насколько проблема сборки вообще актуальна» — **какой %"," фиксов ловится с нулевой сборкой**. Это публикуемая цифра сама по себе.","- Сравнение «OwnAudit vs ESLint» держим **внутри тира** (ESLint без type-info — это"," `syntactic`), а суперсилу показываем отдельно на `interproc`-подвыборке. Иначе сравнение"," едет.","- Restore-гниль (старый WPF не восстановит NuGet, старый React не поставит node_modules)"," бьёт по `semantic`/`interproc`-тиру. Поэтому **build-success rate — явная метрика отбора**,"," иначе корпус схлопнется молча.","## 8. Prospective sweep — и проблема «слишком вылизанных» библиотек","`sweep.py`. Берём top-N пакетов по загрузкам, гоним OwnAudit на **текущем** HEAD, триажим,","репортим в апстрим. Acceptance rate = precision в дикой природе — смещения «только","починенное» тут нет вовсе.","Ловушка: самый верх npm/NuGet — это фундаментальные, **вылизанные донельзя** библиотеки","(Dapper, Polly, Serilog), которые *обязаны* быть без течей и дают floor-эффект. Сэмплировать","только их — узнать ничего. Поэтому отбор:","- `over_vetted_score(pkg)` — эвристика 0..1 «насколько уже вылизан»: много мейнтейнеров,"," CI, зрелый возраст, высокий stars/issue ratio, foundational-library. Это профиль"," Dapper/Polly/Serilog.","- `select_targets(..., max_vetted_fraction=0.3)` — жадно по весу отбора (загрузки,"," дисконтированные вылизанностью), но **капает долю вылизанных** и тем самым проталкивает"," application-shaped репозитории (дашборды, control-plane'ы, CLI, sample-приложения), где"," течи реально выживают.","Registry-доступ (npm downloads API, NuGet search) — за инъектируемыми геттерами, логика","отбора тестируется офлайн. Для NuGet та же схема, что для npm; query-builder'ы готовы.","## 9. MVP-объём","Не 1000 PR сразу. Phase 1: **4 экосистемы × 25 подтверждённых фиксов = 100**. И **сначала","одна** (.NET/WPF) от и до — включая before/after прогоны тулов и SZZ-привязку — потом","веер. Риск весь в сантехнике (checkout старых ревизий, разрешение референсов на","`semantic`-тире), а не в «поймаем ли течь». Zig/Nim — отдельный appendix по 10 фиксов;","Gleam пропускаем, пока не наберётся actor/process-lifecycle (и это всё равно ниша **BEAM**,","т.е. целиться надо в Elixir, а не Gleam).","## 10. CLI","python3 -m leakmine.cli queries --ecosystem react_ts --merged-after 2023-01-01","python3 -m leakmine.cli sql --ecosystem dotnet_wpf --from 20240101 --to 20241231","python3 -m leakmine.cli classify --ecosystem react_ts --patch fix.diff --title \"fix leak\"","python3 -m leakmine.cli confirm --candidate cand.json --baseline eslint","python3 -m leakmine.cli metrics --verdicts verdicts.json --baseline eslint --markdown","python3 -m leakmine.cli sweep --packages pkgs.json --n 50 --max-vetted 0.3","python3 -m leakmine.cli leadtime --repo PATH --sha SHA --file svc.cs --line 42","Каждая стадия читает/пишет JSON — шаги склеиваются в shell-пайплайн; SQLite-стор —","долговременный спин для настоящего многодневного прогона.","## 11. Что это даёт практически","- реальный corpus «плохо/починено» → регрессионные тесты OwnAudit;","- приоритет правил из данных, а не из головы;","- таблица «мы vs baseline» с разбивкой по тиру;","- материал для arXiv/HN/README без позора:"," «На 100 реальных lifetime/resource-фиксах в .NET, React, Android и Java OwnAudit поймал"," Y pre-fix багов, включая Z, не пойманных baseline-линтерами; X% фиксов ловятся вообще без"," сборки».","## 12. Границы честности (повторить вслух)","- `recall_on_corpus` — по смещённому корпусу, не популяционный.","- `fp_after_rate` — smell, не precision; настоящий FP-rate требует стратифицированной"," выборки + ручной разметки (см. `audit-data-leverage.md §3`).","- «gone after» без `causal` — **не** засчитывается (это и есть антивезение).","- lead-time «исправлено по истории» ≠ «исправлено именно эту течь» на 100% — `git log -L`"," следит за строкой, не за семантикой; на сильных заявлениях подтверждать вручную.","## 13. Майнинг в CI (GitHub Action)","`.github/workflows/leakmine-mine.yml` — `leakmine mine` как self-service сборщик корпуса","на hosted-раннере. `workflow_dispatch` с инпутами (`ecosystem`, `merged_after`,","`per_query`, `min_score`) + опциональный недельный `schedule` (закомментирован). Авторизация","— штатный `GITHUB_TOKEN` (read/search публичных PR; для приватных репо или поднятия лимита","Search API — подставить PAT-секрет). На выходе артефакты: `corpus.db` (SQLite-стор),","`dataset.json` (размеченные кандидаты), `summary.md` (он же уходит в `$GITHUB_STEP_SUMMARY`).","Команда:","python3 -m leakmine.cli mine --ecosystem dotnet_wpf --merged-after 2024-01-01 \\"," --per-query 50 --min-score 7 --sleep 2 --out-dir leakmine-out --store leakmine-out/corpus.db","**Что делает оркестратор** (`leakmine/mine.py`): query-pack → `fetch_search` → дедуп по","`(repo, number)` → `fetch_patch` (diff медиа-тип) → `signals.classify` → фильтр по","`min_score` → стор + dataset + summary. Сеть изолирована за инъектируемыми `search`/","`fetch_patch`, поэтому оркестрация юнит-тестится офлайн (CI наружу не ходит).","**Сознательно только discovery + classification** — это часть, которая чисто гоняется в CI.","Стадия before/after (`confirm`/`metrics`) требует checkout каждого репо и сборки анализаторов","(см. §7), поэтому остаётся локальной / self-hosted. Ограничения CI-версии: Search API","~30 req/min и cap 1000 (отсюда `--sleep` и `per_query`); одна страница на запрос (пагинация —","на потом); `language:` репо-уровневый — язык диффа всё равно перепроверяется по расширениям.","## 14. BigQuery — массовость (10k–миллионы), `leakmine/bigquery.py`","Search-API путь упирается в cap 1000, ~30 req/min и стенку per-PR fetch (~1–2k/прогон).","Для массовости backend — BigQuery. **Главное: BigQuery снимает стенку discovery, но не","стенку fetch** — GH Archive отдаёт метаданные PR (title/body/repo/number), а не дифф.","Поэтому роль BigQuery: дать огромный, уже отфильтрованный список кандидатов, который потом","узко дофетчивается и классифицируется существующим пайплайном.","### Два продукта (с РАЗНОЙ стоимостью)","**A. GH-Archive discovery (события) — `gharchive_discovery_sql`.**","SQL по `githubarchive.day.*`: merged-PR, где leak-ключевик в title ИЛИ body, нужный язык,","с отсечкой мега-PR (`changed_files <= N`, чтобы убрать «течь сбоку от рефакторинга»),","дедуп `QUALIFY ROW_NUMBER()`. **Дёшево**: скан ограничен дневными партициями (`_TABLE_SUFFIX","BETWEEN`) — сотни МБ–единицы ГБ, внутри free 1 ТБ/мес. Это и есть discovery «10k, не 10».","**B. Contents sweep (снапшот кода, zero-fetch) — `contents_sweep_sql`.**","SQL по `bigquery-public-data.github_repos`: синтаксический тир прямо в SQL — acquire без","cleanup (`addEventListener()` есть, `removeEventListener()` нет; `setInterval()` без","`clearInterval()`; и т.д., см. `SWEEP_PAIRS`). **Ноль fetch**, масштаб до миллионов файлов.","**Дорого**: любое обращение к `contents.content` сканирует весь столбец ~2.7 ТБ (~$13,","сжигает free-tier) — поэтому по умолчанию бьёт по `sample_files`/`sample_contents` (дёшево);","`--full` включай осознанно.","### Поток и замыкание петли","```text","BigQuery (SQL) -> экспорт NDJSON -> leakmine bq-ingest -> store + candidates.json"," |"," узкий per-PR fetch + signals.classify + confirm","`ingest_rows` грузит экспорт BigQuery в стор, **скоринг — на metadata-тире** (`metadata_score`:","ключевик в title/body + форма по размеру PR). Это **сознательно слабее** patch-классификатора:","без диффа категорию не присвоить — metadata-скор лишь ранжирует очередь на fetch. Survivors","потом дофетчиваются и проходят настоящий `signals.classify`/`confirm`.","### CLI","```bash","# A) сгенерить discovery-SQL, выполнить в BigQuery, выгрузить как NDJSON:","python3 -m leakmine.cli bq-sql --kind gharchive --ecosystem dotnet_wpf --from 20240101 --to 20241231","# -> bq query ... | bq extract ... > rows.ndjson (в своём GCP)","# B) загрузить результат обратно в пайплайн (metadata-тир):","python3 -m leakmine.cli bq-ingest --rows rows.ndjson --ecosystem dotnet_wpf \\"," --min-meta-score 4 --out-dir bq-out --store bq-out/corpus.db","# zero-fetch syntactic sweep по снапшоту кода (по умолчанию дешёвый sample_*):","python3 -m leakmine.cli bq-sql --kind contents --ecosystem react_ts # --full = весь ~2.7TB","### Стоимость и честность","- discovery (A): партиция-скоуп обязателен; без него скан = вся история (терабайты).","- contents (B): `content` — не партиционирован, фильтры скан не уменьшают → `sample_*` по"," умолчанию; полный скан только когда реально надо и бюджет позволяет.","- metadata-скор ≠ patch-классификация: BigQuery даёт чистую *очередь*, не финальный вердикт.","- стенка fetch остаётся: на 100k+ дофетч диффов делается клонированием репо (`--filter=blob:none`,"," `git show`) вне Actions, либо для проспективного sweep'а полагаемся на contents-тир (без fetch).","leakmine/cli.py",[1302],{"_658":659,"_660":1303,"_397":1304,"_663":59},228,[1305,199,1306,1307,1308,1309,1310,1311,1312,199,1313,1314,670,1315,199,1316,671,672,675,199,1317,199,199,1318,1319,1320,199,199,1321,1322,1323,1324,199,199,1325,1326,1324,199,199,1327,1328,1329,1330,1331,1332,1333,1334,1324,199,199,1335,1336,199,199,1337,1338,828,1339,1340,1341,1342,833,1343,1344,1324,199,199,1345,1346,1347,1348,1349,1350,1351,1352,1324,199,199,1353,1354,1355,1356,1357,1358,1359,1360,1324,199,199,1361,1362,1363,1364,1365,1366,1367,833,1368,1369,1370,1371,1372,1373,1374,1324,199,199,1375,1376,1377,1378,1379,1380,1351,1381,1382,1383,1324,199,199,1384,1363,1364,1385,1386,1387,1368,1369,1388,1371,1372,1373,1374,1324,199,199,1389,1390,1391,1324,199,199,1392,1393,1394,199,1395,1396,1397,1398,199,1399,1400,1401,1402,1403,1404,199,1405,1406,1407,1408,1409,1410,199,1411,1412,1413,1414,1415,199,1416,1417,1418,1419,1420,1421,199,1422,1423,1424,1425,1426,199,1427,1428,1429,1430,1431,1432,1433,1434,1435,1436,199,1437,1438,1439,1440,1441,1442,1443,1444,1445,1446,1447,199,1448,1449,1450,1451,1452,1453,1454,199,1455,1456,1457,1458,1459,1460,199,1461,1462,199,199,1068,1463],"\"\"\"LeakFixMine CLI — drive the research pipeline stage by stage (docs/leakfix-mine.md §12)."," python3 -m leakmine.cli queries --ecosystem react_ts"," python3 -m leakmine.cli sql --ecosystem dotnet_wpf --from 20240101 --to 20241231"," python3 -m leakmine.cli classify --ecosystem react_ts --patch fix.diff --title \"fix leak\""," python3 -m leakmine.cli confirm --candidate cand.json"," python3 -m leakmine.cli metrics --verdicts verdicts.json"," python3 -m leakmine.cli sweep --packages pkgs.json --n 50"," python3 -m leakmine.cli leadtime --repo PATH --sha SHA --file F --line N","Every stage reads/writes JSON so the steps compose in a shell pipeline; the SQLite store","(`schema.py`) is the durable spine for a real multi-day run.","from __future__ import annotations","import argparse","from . import bigquery, collect, confirm, metrics, mine, schema, signals, sweep, szz","def _read(path: str) -> str:"," with open(path, encoding=\"utf-8\") as f:"," return f.read()","def cmd_queries(a) -> int:"," for q in collect.github_search_queries(a.ecosystem, merged_after=a.merged_after or \"\"):"," print(q)"," return 0","def cmd_sql(a) -> int:"," print(collect.gharchive_sql(a.ecosystem, date_from=getattr(a, \"from\"), date_to=a.to, limit=a.limit))","def cmd_classify(a) -> int:"," patch = _read(a.patch) if a.patch else \"\""," cls = signals.classify(a.ecosystem, title=a.title or \"\", body=a.body or \"\", patch=patch)"," print(json.dumps({"," \"category\": cls.category, \"score\": cls.score,"," \"is_candidate\": cls.is_candidate, \"is_likely_fix\": cls.is_likely_fix,"," \"by_category\": cls.by_category, \"evidence\": cls.evidence,"," }, indent=2, ensure_ascii=False))","def _findings(rows) -> list[szz.Finding]:"," return [szz.Finding(**r) for r in rows]","def cmd_confirm(a) -> int:"," raw = json.loads(_read(a.candidate))"," id=raw[\"id\"], ecosystem=raw[\"ecosystem\"], title=raw.get(\"title\", \"\"),"," body=raw.get(\"body\", \"\"), patch=raw.get(\"patch\", \"\"),"," before={k: _findings(v) for k, v in raw.get(\"before\", {}).items()},"," after={k: _findings(v) for k, v in raw.get(\"after\", {}).items()},"," v = confirm.judge(cand, ownaudit_tool=a.ownaudit, baseline_tools=tuple(a.baseline))"," print(json.dumps(v.__dict__, indent=2, ensure_ascii=False))","def cmd_metrics(a) -> int:"," rows = json.loads(_read(a.verdicts))"," verdicts = [confirm.Verdict(**r) for r in rows]"," rep = metrics.aggregate(verdicts, ownaudit_tool=a.ownaudit, baseline_tools=tuple(a.baseline))"," if a.markdown:"," print(metrics.render_markdown(rep))"," else:"," print(json.dumps(rep.as_dict(), indent=2, ensure_ascii=False))","def cmd_sweep(a) -> int:"," pkgs = [sweep.Package(**p) for p in json.loads(_read(a.packages))]"," chosen = sweep.select_targets(pkgs, n=a.n, max_vetted_fraction=a.max_vetted)"," print(json.dumps(["," {\"name\": s.pkg.name, \"registry\": s.pkg.registry, \"vetted\": round(s.vetted, 3),"," \"weight\": round(s.weight, 3), \"reason\": s.reason}"," for s in chosen"," ], indent=2, ensure_ascii=False))","def cmd_mine(a) -> int:"," token = a.token or os.environ.get(\"GITHUB_TOKEN\", \"\")"," os.makedirs(a.out_dir, exist_ok=True)"," conn = schema.connect(a.store) if a.store else None"," res = mine.run("," a.ecosystem, token=token, merged_after=a.merged_after or \"\","," per_query=a.per_query, min_score=a.min_score, sleep=a.sleep, conn=conn,"," if conn is not None:"," conn.commit()"," with open(os.path.join(a.out_dir, \"dataset.json\"), \"w\", encoding=\"utf-8\") as f:"," json.dump(res.rows, f, indent=2, ensure_ascii=False)"," with open(os.path.join(a.out_dir, \"summary.md\"), \"w\", encoding=\"utf-8\") as f:"," f.write(res.summary_md())"," print(res.summary_md())","def cmd_bq_sql(a) -> int:"," if a.kind == \"gharchive\":"," print(bigquery.gharchive_discovery_sql("," a.ecosystem, date_from=getattr(a, \"from\"), date_to=a.to,"," max_changed_files=a.max_changed_files,"," limit=50000 if a.limit is None else a.limit))"," # contents scan is far pricier per row, so keep the helper's cheaper 5k default."," print(bigquery.contents_sweep_sql("," a.ecosystem, sample=not a.full, limit=5000 if a.limit is None else a.limit))","def cmd_bq_ingest(a) -> int:"," # stream the export line-by-line so a multi-GB dump ingests at bounded memory."," res = bigquery.ingest_rows(bigquery.iter_ndjson(a.rows), a.ecosystem,"," min_meta_score=a.min_meta_score, conn=conn)"," with open(os.path.join(a.out_dir, \"candidates.json\"), \"w\", encoding=\"utf-8\") as f:","def cmd_leadtime(a) -> int:"," lt = szz.lead_time(a.repo, a.sha, a.file, a.line)"," print(json.dumps(lt.__dict__, indent=2, ensure_ascii=False))","def main(argv=None) -> int:"," p = argparse.ArgumentParser(prog=\"leakmine\", description=__doc__)"," sub = p.add_subparsers(dest=\"cmd\", required=True)"," q = sub.add_parser(\"queries\", help=\"print GitHub-search query pack\")"," q.add_argument(\"--ecosystem\", required=True, choices=sorted(signals.ECOSYSTEMS))"," q.add_argument(\"--merged-after\", default=\"\")"," q.set_defaults(fn=cmd_queries)"," s = sub.add_parser(\"sql\", help=\"print GH-Archive BigQuery SQL\")"," s.add_argument(\"--ecosystem\", required=True, choices=sorted(signals.ECOSYSTEMS))"," s.add_argument(\"--from\", required=True, help=\"YYYYMMDD partition start\")"," s.add_argument(\"--to\", required=True, help=\"YYYYMMDD partition end\")"," s.add_argument(\"--limit\", type=int, default=2000)"," s.set_defaults(fn=cmd_sql)"," c = sub.add_parser(\"classify\", help=\"patch-signal classify one candidate\")"," c.add_argument(\"--ecosystem\", required=True, choices=sorted(signals.ECOSYSTEMS))"," c.add_argument(\"--patch\", help=\"path to a unified diff\")"," c.add_argument(\"--title\", default=\"\")"," c.add_argument(\"--body\", default=\"\")"," c.set_defaults(fn=cmd_classify)"," cf = sub.add_parser(\"confirm\", help=\"judge a materialised candidate (before/after/patch)\")"," cf.add_argument(\"--candidate\", required=True)"," cf.add_argument(\"--ownaudit\", default=\"ownaudit\")"," cf.add_argument(\"--baseline\", nargs=\"*\", default=[])"," cf.set_defaults(fn=cmd_confirm)"," m = sub.add_parser(\"metrics\", help=\"aggregate verdicts -> report\")"," m.add_argument(\"--verdicts\", required=True)"," m.add_argument(\"--ownaudit\", default=\"ownaudit\")"," m.add_argument(\"--baseline\", nargs=\"*\", default=[])"," m.add_argument(\"--markdown\", action=\"store_true\")"," m.set_defaults(fn=cmd_metrics)"," sw = sub.add_parser(\"sweep\", help=\"select prospective sweep targets\")"," sw.add_argument(\"--packages\", required=True)"," sw.add_argument(\"--n\", type=int, default=50)"," sw.add_argument(\"--max-vetted\", type=float, default=0.3, dest=\"max_vetted\")"," sw.set_defaults(fn=cmd_sweep)"," mn = sub.add_parser(\"mine\", help=\"discover + classify a leak-fix corpus (CI-runnable)\")"," mn.add_argument(\"--ecosystem\", required=True, choices=sorted(signals.ECOSYSTEMS))"," mn.add_argument(\"--merged-after\", default=\"\", help=\"ISO date YYYY-MM-DD; appends merged:>=\")"," mn.add_argument(\"--per-query\", type=int, default=50)"," mn.add_argument(\"--min-score\", type=int, default=7)"," mn.add_argument(\"--sleep\", type=float, default=1.0, help=\"throttle between API calls (s)\")"," mn.add_argument(\"--out-dir\", default=\"leakmine-out\")"," mn.add_argument(\"--store\", default=\"\", help=\"SQLite path; empty = no DB written\")"," mn.add_argument(\"--token\", default=\"\", help=\"GitHub token; falls back to $GITHUB_TOKEN\")"," mn.set_defaults(fn=cmd_mine)"," bq = sub.add_parser(\"bq-sql\", help=\"emit BigQuery SQL (GH-Archive discovery / contents sweep)\")"," bq.add_argument(\"--kind\", choices=(\"gharchive\", \"contents\"), default=\"gharchive\")"," bq.add_argument(\"--ecosystem\", required=True, choices=sorted(signals.ECOSYSTEMS))"," bq.add_argument(\"--from\", default=\"20240101\", help=\"YYYYMMDD partition start (gharchive)\")"," bq.add_argument(\"--to\", default=\"20241231\", help=\"YYYYMMDD partition end (gharchive)\")"," bq.add_argument(\"--max-changed-files\", type=int, default=80, dest=\"max_changed_files\")"," bq.add_argument(\"--full\", action=\"store_true\","," help=\"contents sweep: scan full ~2.7TB tables instead of sample_* (costs $$)\")"," bq.add_argument(\"--limit\", type=int, default=None,"," help=\"row cap (default: 50000 gharchive, 5000 contents)\")"," bq.set_defaults(fn=cmd_bq_sql)"," bi = sub.add_parser(\"bq-ingest\", help=\"ingest BigQuery NDJSON export -> store (metadata tier)\")"," bi.add_argument(\"--rows\", required=True, help=\"path to NDJSON exported from BigQuery\")"," bi.add_argument(\"--ecosystem\", required=True, choices=sorted(signals.ECOSYSTEMS))"," bi.add_argument(\"--min-meta-score\", type=int, default=4, dest=\"min_meta_score\")"," bi.add_argument(\"--out-dir\", default=\"leakmine-out\")"," bi.add_argument(\"--store\", default=\"\", help=\"SQLite path; empty = no DB written\")"," bi.set_defaults(fn=cmd_bq_ingest)"," lt = sub.add_parser(\"leadtime\", help=\"time-travel: commits between a leak and its human fix\")"," lt.add_argument(\"--repo\", required=True)"," lt.add_argument(\"--sha\", required=True)"," lt.add_argument(\"--file\", required=True)"," lt.add_argument(\"--line\", type=int, required=True)"," lt.set_defaults(fn=cmd_leadtime)"," a = p.parse_args(argv)"," return a.fn(a)"," sys.exit(main())","leakmine/bigquery.py",[1466],{"_658":659,"_660":1467,"_397":1468,"_663":59},284,[1469,199,1470,1471,1472,199,1473,1474,1475,1476,1477,1478,199,1479,1480,1481,1482,1483,1484,199,1485,1486,1487,670,1315,199,671,1488,1489,199,1490,199,199,1491,199,1492,1493,1494,1495,1496,1497,1498,1499,1500,199,1501,1502,1503,1504,1505,1506,1507,1508,1509,1510,833,1511,1512,1513,1514,1515,1516,1517,1518,833,1519,1520,1521,1522,1523,1524,1525,1526,1527,1528,1529,1530,1531,1532,1533,1534,1535,1536,1537,1538,1539,1540,1541,1542,1543,1544,1545,1546,1547,833,199,199,1548,199,1549,1550,1551,1552,1553,1554,1555,1556,1557,1558,1559,1556,1560,1561,1562,1556,1563,1564,1556,1565,199,199,1566,1493,1494,1567,1568,1499,1569,199,1570,1571,1572,1504,1573,1574,1575,1576,1577,1578,1579,1580,1581,1582,833,1583,1584,1585,1586,1587,1588,833,1519,1589,1590,1591,1592,1593,1594,1595,1596,1597,1598,1599,1600,1547,833,199,199,1601,199,1602,1603,1604,1605,1606,1607,199,1608,1609,1610,1611,1612,1613,1614,199,199,1615,1616,1617,199,1618,1619,1620,1504,1505,1621,1622,1623,1624,1625,1626,1627,1628,1629,1630,1631,1632,1633,1634,1635,1636,199,199,1637,1638,1639,199,1640,1641,1642,1643,1504,1644,1645,1646,1647,1648,1649,1650,1651,1649,1652,1653,1654,1655,1656,1657,1658,1649,1659,1660,1661,1662,1663,1664,1665,1666,1667,1668,1669,1670,1671,1672,1673,1674,1675,1676,199,199,1677,1678,1679,199,199,1680,1681,1682,1683,1319,1684,1685,1686,1687,199,199,1688,1689,1690,1691,1692,1693,1694,1695,1565],"\"\"\"BigQuery arm — discovery (and a zero-fetch syntactic sweep) at firehose scale.","The Search-API path (`collect`/`mine`) tops out around a few thousand candidates per run","(1000-result cap, ~30 req/min, the per-PR diff-fetch wall). For 10k–millions, BigQuery is","the backend (docs/leakfix-mine.md §4, §14). Two products, with very different cost:"," 1. GH Archive discovery (`gharchive_discovery_sql`) — the public `githubarchive.day.*`"," event firehose. One partition-scoped query returns every merged PR whose title OR body"," carries a leak keyword, in the target language, optionally size-capped to drop mega-"," refactors. CHEAP: scanning a bounded date range is a few hundred MB–few GB (well inside"," the 1 TB/month free tier). This REPLACES discovery — but GH Archive has no diffs, so the"," output is *metadata candidates* you then narrow-fetch + classify with `mine`/`collect`."," 2. Contents sweep (`contents_sweep_sql`) — `bigquery-public-data.github_repos`, a snapshot"," of file *contents*. Runs the SYNTACTIC tier (acquire-without-cleanup, e.g. an"," `addEventListener(` with no `removeEventListener`) as SQL regex over millions of files,"," ZERO fetch. EXPENSIVE: any query touching `contents.content` scans the full ~2.7 TB"," column regardless of filters (~$13, blows the free tier), so this defaults to the much"," smaller `sample_contents`/`sample_files` tables. Use the full tables only deliberately.","`ingest_rows` closes the loop: BigQuery results exported as NDJSON flow back into the leakmine","store, scored at the *metadata tier* (keyword strength + PR-size heuristic — NOT the patch","signal, which needs the diff). Pure stdlib; SQL generation + ingest unit-test offline.","from collections.abc import Iterable, Iterator","from dataclasses import dataclass, field","from . import schema, signals","# ---- GH Archive discovery (events) -----------------------------------------------","def gharchive_discovery_sql("," eco_key: str,"," *,"," date_from: str,"," date_to: str,"," max_changed_files: int = 80,"," limit: int = 50000,",") -> str:"," \"\"\"SQL over `githubarchive.day.*` for merged leak-fix PR candidates in one ecosystem."," `date_from`/`date_to` are YYYYMMDD `day`-table suffixes; the `_TABLE_SUFFIX BETWEEN`"," bound is what keeps the scan cheap — never run this unbounded. `max_changed_files` drops"," mega-PRs where a leak keyword is incidental (set 0 to disable). Dedups to one row per PR."," \"\"\""," eco = signals.ECOSYSTEMS[eco_key]"," kw = \" OR \".join("," \"(LOWER(JSON_EXTRACT_SCALAR(payload,'$.pull_request.title')) LIKE '%{k}%' OR \""," \"LOWER(COALESCE(JSON_EXTRACT_SCALAR(payload,'$.pull_request.body'),'')) \""," \"LIKE '%{k}%')\".format(k=k)"," for k in eco.keywords"," langs = \" OR \".join("," f\"LOWER(JSON_EXTRACT_SCALAR(payload,'$.pull_request.base.repo.language')) = '{lang}'\""," for lang in _LANGS.get(eco_key, ())"," ) or \"TRUE\""," size_cap = ("," f\"\\n AND SAFE_CAST(JSON_EXTRACT_SCALAR(payload,'$.pull_request.changed_files') \""," f\"AS INT64) <= {max_changed_files}\""," if max_changed_files else \"\""," return ("," f\"-- LeakFixMine GH-Archive discovery: ecosystem={eco_key}, \""," f\"partitions {date_from}..{date_to}. Cost = bytes in those day-partitions only.\\n\""," \"SELECT\\n\""," \" repo.name AS repo,\\n\""," \" JSON_EXTRACT_SCALAR(payload,'$.number') AS number,\\n\""," \" JSON_EXTRACT_SCALAR(payload,'$.pull_request.title') AS title,\\n\""," \" JSON_EXTRACT_SCALAR(payload,'$.pull_request.body') AS body,\\n\""," \" JSON_EXTRACT_SCALAR(payload,'$.pull_request.merged_at') AS merged_at,\\n\""," \" JSON_EXTRACT_SCALAR(payload,'$.pull_request.base.repo.language') AS lang,\\n\""," \" SAFE_CAST(JSON_EXTRACT_SCALAR(payload,'$.pull_request.changed_files') AS INT64) \""," \"AS changed_files,\\n\""," \" SAFE_CAST(JSON_EXTRACT_SCALAR(payload,'$.pull_request.additions') AS INT64) \""," \"AS additions,\\n\""," \" SAFE_CAST(JSON_EXTRACT_SCALAR(payload,'$.pull_request.deletions') AS INT64) \""," \"AS deletions,\\n\""," \" JSON_EXTRACT_SCALAR(payload,'$.pull_request.html_url') AS html_url,\\n\""," \" JSON_EXTRACT_SCALAR(payload,'$.pull_request.diff_url') AS diff_url\\n\""," \"FROM `githubarchive.day.*`\\n\""," f\"WHERE _TABLE_SUFFIX BETWEEN '{date_from}' AND '{date_to}'\\n\""," \" AND type = 'PullRequestEvent'\\n\""," \" AND JSON_EXTRACT_SCALAR(payload,'$.action') = 'closed'\\n\""," \" AND JSON_EXTRACT_SCALAR(payload,'$.pull_request.merged') = 'true'\\n\""," f\" AND ({kw})\\n\""," f\" AND ({langs}){size_cap}\\n\""," \"-- one row per PR (the firehose repeats events):\\n\""," \"QUALIFY ROW_NUMBER() OVER (PARTITION BY repo.name, \""," \"JSON_EXTRACT_SCALAR(payload,'$.number') ORDER BY created_at DESC) = 1\\n\""," f\"LIMIT {limit}\\n\"","# ---- Contents sweep (code snapshot, zero fetch) ----------------------------------","# acquire-without-cleanup pairs: a file containing `acquire` but not `cleanup` is a","# candidate syntactic-tier leak site. Regexes are RE2 (BigQuery REGEXP_CONTAINS).","SWEEP_PAIRS: dict[str, tuple[tuple[str, str, str], ...]] = {"," \"react_ts\": ("," (r\"addEventListener\\(\", r\"removeEventListener\", \"listener\"),"," (r\"setInterval\\(\", r\"clearInterval\", \"timer\"),"," (r\"\\.subscribe\\(\", r\"unsubscribe\", \"subscription\"),"," ),"," \"dotnet_wpf\": ("," (r\"new DispatcherTimer\", r\"\\.Stop\\(\", \"timer\"),"," (r\"new FileSystemWatcher\", r\"\\.Dispose\\(\", \"idisposable\"),"," \"java_spring\": ("," (r\"newFixedThreadPool|newCachedThreadPool\", r\"\\.shutdown\", \"executor\"),"," (r\"new ThreadLocal\", r\"\\.remove\\(\", \"static-retention\"),"," \"android_kotlin\": ("," (r\"registerReceiver\\(\", r\"unregisterReceiver\", \"subscription\"),","}","def contents_sweep_sql("," sample: bool = True,"," limit: int = 5000,"," \"\"\"SQL over `github_repos` that flags syntactic-tier leak sites with zero fetch."," `sample=True` (default) targets `sample_files`/`sample_contents` — far cheaper. The full"," `files`/`contents` tables scan the ~2.7 TB content column on any reference (≈$13/query),"," so flip `sample` off only when you mean it. Returns (repo, path, signal) candidate sites."," pairs = SWEEP_PAIRS.get(eco_key, ())"," if not pairs:"," raise ValueError(f\"no contents-sweep pairs defined for ecosystem {eco_key!r}\")"," files_tbl = \"sample_files\" if sample else \"files\""," contents_tbl = \"sample_contents\" if sample else \"contents\""," exts = \" OR \".join(f\"f.path LIKE '%{e}'\" for e in signals.ECOSYSTEMS[eco_key].file_ext)"," cases = \"\\n\".join("," f\" WHEN REGEXP_CONTAINS(c.content, r'{acq}') \""," f\"AND NOT REGEXP_CONTAINS(c.content, r'{cln}') THEN '{label}'\""," for acq, cln, label in pairs"," # candidate = acquire present AND its cleanup absent — for at least one pair. Without the"," # NOT-cleanup half a *balanced* file (acquire + cleanup) would still match and fall through"," # the CASE to 'other', flooding the sweep with non-leaks. So the WHERE mirrors the CASE."," candidate_pred = \" OR \".join("," f\"(REGEXP_CONTAINS(c.content, r'{acq}') AND NOT REGEXP_CONTAINS(c.content, r'{cln}'))\""," for acq, cln, _ in pairs"," f\"-- LeakFixMine contents sweep: ecosystem={eco_key}, table=\""," f\"{'sample_' if sample else ''}(files/contents). \""," f\"{'CHEAP sample.' if sample else 'WARNING: full content scan ~2.7TB (~$13).'}\\n\""," \"SELECT f.repo_name AS repo, f.path AS path,\\n\""," \" CASE\\n\""," f\"{cases}\\n\""," \" END AS signal\\n\""," f\"FROM `bigquery-public-data.github_repos.{files_tbl}` f\\n\""," f\"JOIN `bigquery-public-data.github_repos.{contents_tbl}` c USING (id)\\n\""," f\"WHERE ({exts})\\n\""," \" AND NOT c.binary\\n\""," f\" AND ({candidate_pred})\\n\"","# ---- ingest: BigQuery NDJSON -> leakmine store (metadata tier) --------------------","@dataclass","class MetaResult:"," ecosystem: str"," rows: list[dict] = field(default_factory=list)"," seen: int = 0"," kept: int = 0"," def summary_md(self) -> str:"," return ("," f\"# LeakFixMine — BigQuery ingest (`{self.ecosystem}`, metadata tier)\\n\\n\""," f\"- rows seen: **{self.seen}**\\n- kept (>= meta threshold): **{self.kept}**\\n\\n\""," \"NOTE: metadata-tier score (keyword + PR size), NOT the patch signal. \""," \"Narrow-fetch the kept PRs and run `mine`/`confirm` for the real classification.\\n\""," )","def metadata_score(eco_key: str, *, title: str = \"\", body: str = \"\","," changed_files: int | None = None) -> tuple[int, list[str]]:"," \"\"\"Discovery-tier score for a PR we have metadata-only for (no diff yet)."," +3 leak keyword in title, +2 in body; PR-size shaping so a focused fix outranks a mega-"," refactor where the keyword is incidental. This is deliberately weaker than the patch"," classifier — it ranks the fetch queue, it does not assign a leak category."," score, ev = 0, []"," tl, bl = (title or \"\").lower(), (body or \"\").lower()"," if any(k in tl for k in eco.keywords):"," score += 3"," ev.append(\"title:leak-keyword\")"," if any(k in bl for k in eco.keywords):"," score += 2"," ev.append(\"body:leak-keyword\")"," if changed_files is not None:"," if changed_files <= 10:"," score += 2"," ev.append(\"small-pr\")"," elif changed_files > 200:"," score -= 3"," ev.append(\"penalty:mega-pr\")"," return score, ev","def ingest_rows(rows: Iterable[dict], eco_key: str, *, min_meta_score: int = 4,"," conn=None) -> MetaResult:"," \"\"\"Ingest exported BigQuery rows (dicts with the SELECT-alias keys) into the store."," `rows` is any iterable — pass `iter_ndjson(path)` to stream a multi-GB export without"," loading it into memory. Dedups by (repo, number), scores at the metadata tier, keeps"," >= `min_meta_score`. With a `conn` it writes candidates + a 'bigquery-meta' label so the"," rest of the pipeline (fetch + `signals.classify` + `confirm`) can pick the survivors up."," res = MetaResult(ecosystem=eco_key)"," seen: set[tuple[str, str]] = set()"," for r in rows:"," repo, number = r.get(\"repo\", \"\"), str(r.get(\"number\", \"\"))"," if not repo or not number:"," continue"," key = (repo, number)"," if key in seen:"," seen.add(key)"," res.seen += 1"," cf = r.get(\"changed_files\")"," cf = int(cf) if cf not in (None, \"\") else None"," score, ev = metadata_score(eco_key, title=r.get(\"title\", \"\"), body=r.get(\"body\", \"\"),"," changed_files=cf)"," if score < min_meta_score:"," res.kept += 1"," row = {"," \"ecosystem\": eco_key, \"repo\": repo, \"number\": int(number),"," \"url\": r.get(\"html_url\") or r.get(\"diff_url\") or \"\","," \"title\": r.get(\"title\", \"\"), \"meta_score\": score, \"evidence\": ev,"," \"changed_files\": cf,"," }"," res.rows.append(row)"," if conn is not None:"," schema.insert_candidate(conn, {"," \"id\": f\"{repo}#{number}\", \"ecosystem\": eco_key, \"query\": \"bigquery:gharchive\","," \"repo\": repo, \"number\": int(number), \"kind\": \"pr\","," \"title\": r.get(\"title\", \"\"), \"body\": r.get(\"body\", \"\"),"," \"state\": \"merged\", \"merged\": 1, \"url\": row[\"url\"],"," })"," schema.insert_label(conn, f\"{repo}#{number}\", \"metadata-candidate\", score, ev,"," \"bigquery-meta\")"," return res","def read_ndjson(text: str) -> list[dict]:"," \"\"\"Parse an in-memory NDJSON string (one JSON object per line; blanks ignored).\"\"\""," return [json.loads(ln) for ln in text.splitlines() if ln.strip()]","def iter_ndjson(path: str) -> Iterator[dict]:"," \"\"\"Stream an NDJSON export file line-by-line — never materialises the whole dump, so a"," multi-GB BigQuery export ingests at bounded memory (the only in-memory state downstream"," is the (repo, number) dedup set).\"\"\""," for line in f:"," line = line.strip()"," if line:"," yield json.loads(line)","# repo-language filter values (lowercased) per ecosystem for the GH-Archive query.","_LANGS: dict[str, tuple[str, ...]] = {"," \"dotnet_wpf\": (\"c#\",),"," \"react_ts\": (\"typescript\", \"javascript\"),"," \"android_kotlin\": (\"kotlin\", \"java\"),"," \"java_spring\": (\"java\",),"," \"zig\": (\"zig\",),"," \"nim\": (\"nim\",),",[1697,1698,1699,1700,1701,1702,1703,1704,1705,1706,1707,1708,1709,1710,1711,1712],{"_633":46,"_46":1738},{"_633":1714,"_161":1464,"_658":1736,"_660":1737},{"_633":46,"_46":1735},{"_633":1714,"_161":1300,"_658":1733,"_660":1734},{"_633":1714,"_161":1300,"_658":1731,"_660":1732},{"_633":46,"_46":1730},{"_633":1714,"_161":1300,"_658":1728,"_660":1729},{"_633":1714,"_161":1464,"_658":1726,"_660":1727},{"_633":1714,"_161":1464,"_658":1725,"_660":1720},{"_633":46,"_46":1724},{"_633":1714,"_161":1070,"_658":1722,"_660":1723},{"_633":1714,"_161":1070,"_658":1720,"_660":1721},{"_633":46,"_46":1719},{"_633":1714,"_161":652,"_658":1717,"_660":1718},{"_633":1714,"_161":652,"_658":1715,"_660":1716},{"_633":46,"_46":1713},"\n\n**Testing**\n* ✅ `git status --short`\n* ✅ `python3 -m pytest -q leakmine/tests/test_leakmine.py`\n* ✅ `python3 -O -m pytest -q leakmine/tests/test_leakmine.py`","repo_file_citation",492,499,460,465,"\n* Confirmed the tests assert title/body `OR` semantics and the mega-PR penalty score. ",270,294,260,266,"\n* Confirmed the docs have balanced inline call examples and language-tagged fenced blocks. ",265,214,221,121,126,"\n* Confirmed BigQuery ingest now streams NDJSON via `iter_ndjson()` into `ingest_rows()`, and `ingest_rows()` accepts an iterable. ",196,205,108,117,"\n* Confirmed the CLI uses per-kind defaults: `50000` for `gharchive` and `5000` for `contents`, with `--limit` defaulting to `None`. ",140,165,"### Summary\n* Verified the current branch is already at `b3f045a`, with the review fixes present and no uncommitted changes, so I did **not** create an additional commit or PR.\n* Confirmed the contents sweep now mirrors the `CASE` logic in its `WHERE` predicate, requiring acquire-without-cleanup and avoiding fallthrough `other` rows. ",{"_92":326,"_42":84,"_96":326,"_1741":1742,"_1743":1744,"_1745":1746,"_132":-5,"_119":1747},[83],"input_items",[1766,1767,1768,1769,1770,1771,1772,1773,1774,1775,1776,1777,1778,1779],"external_post_config",{"_1748":1749,"_263":196,"_1750":305,"_274":-5,"_1751":257,"_1752":-5,"_1753":-5,"_1754":1755,"_1756":1757,"_1758":-5,"_1759":-5,"_1760":1761,"_1762":15,"_1763":59,"_1764":54,"_1765":15},"user_model_slug","slug",1782620067.566243,"installation_id",131220309,"pr_number","repo_id","gitlab_service_account_project_id","review_trigger","comment_id",4824740929,"comment_type","issue","pr_write_guard_token","pending_pr_reaction_id","pending_comment_reaction_id",375626084,"comment_if_no_findings","comment_on_error","share_url","is_parallel_review",{"_92":403,"_96":326,"_397":1800},{"_312":313,"_314":1796,"_1797":319,"_1798":129,"_322":-5,"_92":1799},{"_92":1780,"_314":1795,"_161":-5,"_1782":-5,"_1783":-5},{"_92":1780,"_314":1794,"_161":-5,"_1782":-5,"_1783":-5},{"_92":1780,"_314":1793,"_161":1464,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1792,"_161":1070,"_1782":1785,"_1783":1723},{"_92":1780,"_314":1791,"_161":1070,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1790,"_161":1464,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1789,"_161":1464,"_1782":1785,"_1783":1737},{"_92":1780,"_314":1788,"_161":1300,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1787,"_161":1300,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1786,"_161":652,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1784,"_161":652,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1781,"_161":-5,"_1782":-5,"_1783":-5},"pull_request_comment","**Actionable comments posted: 8**\n\n

🤖 Prompt for all review comments with AI agents

\n\n

🪄 Autofix (Beta)

\n\nFix all unresolved CodeRabbit comments on this PR:\n\n- [ ] Push a commit to this branch (recommended)\n- [ ] Create a new PR with the fixes\n\n

\n\n---\n\n

ℹ️ Review info

\n\n

⚙️ Run configuration

\n\n**Configuration used**: defaults\n\n**Review profile**: CHILL\n\n**Plan**: Pro Plus\n\n**Run ID**: `f73ee075-699d-4f5c-bcf5-6cab3b76f19b`\n\n

\n\n

📥 Commits

\n\nReviewing files that changed from the base of the PR and between ca31a664ff61f654d68db49b46cb46c1cda8443a and d4fa05d977a0d3d568a81b3881a7c33f5bc98b84.\n\n

\n\n

📒 Files selected for processing (5)

\n\n* `docs/leakfix-mine.md`\n* `leakmine/__init__.py`\n* `leakmine/bigquery.py`\n* `leakmine/cli.py`\n* `leakmine/tests/test_leakmine.py`\n\n

\n\n

\n\n","side","line","_📐 Maintainability & Code Quality_ | _🟡 Minor_ | _⚡ Quick win_\n\n**Use the penalized score in the assertion or drop the binding.**\n\n`lo` is never used here, so Ruff will keep flagging this test. The cleaner fix is to assert on the score as well as the evidence so the mega-PR penalty is covered behaviorally.\n\n

🧰 Tools

\n\n

🪛 Ruff (0.15.18)

\n\n[warning] 493-493: Unpacked variable `lo` is never used\n\nPrefix it with an underscore or any other dummy variable pattern\n\n(RUF059)\n\n

\n\n

🤖 Prompt for AI Agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nIn `@leakmine/tests/test_leakmine.py` around lines 493 - 495, The test in the\nmetadata_score call binds lo but never uses it, which will keep triggering Ruff.\nUpdate the assertion in this test to also verify the penalized score returned by\nbigquery.metadata_score alongside the existing evidence check, or remove the\nunused binding if you only intend to assert on ev2; use the metadata_score\nresult tuple in this test to cover the mega-PR penalty behaviorally.\n```\n\n

\n\n\n\n\n\n\n\n_Source: Linters/SAST tools_\n\n","RIGHT","_🎯 Functional Correctness_ | _🟡 Minor_ | _⚡ Quick win_\n\n**Assert the title/body `OR` semantics, not just field presence.**\n\nLine 462 passes as long as both field names appear anywhere in the SQL, so a regression from `...title...) OR (...body...)` to `...title...) AND (...body...)` would still be missed. Tighten this to assert the boolean fragment you actually rely on.\n\n

🤖 Prompt for AI Agents

\n\n\n\n\n\n\n\n","_🩺 Stability & Availability_ | _🟠 Major_ | _🏗️ Heavy lift_\n\n**Stream the NDJSON ingest path instead of loading the whole export.**\n\nLine 119 reads the entire file into memory, `read_ndjson()` builds a second full list, and `ingest_rows()` then walks that list. That breaks the PR’s stated 10k–millions scale target; large BigQuery exports will hit high RSS or OOM before dedup/scoring runs. Please refactor this path so the file is consumed line-by-line and `ingest_rows` accepts an iterable.\n\n

🤖 Prompt for AI Agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nIn `@leakmine/cli.py` around lines 118 - 122, The NDJSON ingest path in\ncmd_bq_ingest currently materializes the full export before ingestion, which\nprevents scaling to large BigQuery dumps. Refactor _read/read_ndjson usage so\nthe file is consumed as a streaming iterable line-by-line, and update\nbigquery.ingest_rows to accept and process an iterator instead of a prebuilt\nlist. Keep the behavior of dedup/scoring intact while ensuring cmd_bq_ingest\npasses the streamed rows through without building an in-memory copy.\n```\n\n

\n\n\n\n\n\n\n\n","_🚀 Performance & Scalability_ | _🟠 Major_ | _⚡ Quick win_\n\n**Keep the cheaper 5k default for `--kind contents`.**\n\n`contents_sweep_sql()` defaults to `limit=5000`, but this CLI hardcodes `--limit 50000` for both kinds. So `leakmine bq-sql --kind contents` silently emits a query that is 10x broader than the helper contract, and `--full` makes that the expensive path. Use a per-kind default instead of a shared 50k default. \n \n\n

Suggested fix

\n\n\nAlso applies to: 192-200\n\n

🤖 Prompt for AI Agents

\n\n\n\n\n\n\n\n","_🎯 Functional Correctness_ | _🟠 Major_ | _⚡ Quick win_\n\n**Filter out the non-leak `other` rows.**\n\nThe current `WHERE` only requires an acquire match, so files that also contain the matching cleanup still come back as `signal='other'`. That turns the sweep into “files with acquire calls” instead of “acquire without cleanup”, which will flood the result set with valid code. \n \n\n

Proposed fix

\n\n\n\n

📝 Committable suggestion

\n\n> ‼️ **IMPORTANT**\n> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.\n\n```suggestion\n cases = \"\\n\".join(\n f\" WHEN REGEXP_CONTAINS(c.content, r'{acq}') \"\n f\"AND NOT REGEXP_CONTAINS(c.content, r'{cln}') THEN '{label}'\"\n for acq, cln, label in pairs\n )\n candidate_pred = \" OR \".join(\n f\"(REGEXP_CONTAINS(c.content, r'{acq}') \"\n f\"AND NOT REGEXP_CONTAINS(c.content, r'{cln}'))\"\n for acq, cln, _ in pairs\n )\n return (\n f\"-- LeakFixMine contents sweep: ecosystem={eco_key}, table=\"\n f\"{'sample_' if sample else ''}(files/contents). \"\n f\"{'CHEAP sample.' if sample else 'WARNING: full content scan ~2.7TB (~$13).'}\\n\"\n \"SELECT f.repo_name AS repo, f.path AS path,\\n\"\n \" CASE\\n\"\n f\"{cases}\\n\"\n \" END AS signal\\n\"\n f\"FROM `bigquery-public-data.github_repos.{files_tbl}` f\\n\"\n f\"JOIN `bigquery-public-data.github_repos.{contents_tbl}` c USING (id)\\n\"\n f\"WHERE ({exts})\\n\"\n \" AND NOT c.binary\\n\"\n f\" AND ({candidate_pred})\\n\"\n f\"LIMIT {limit}\\n\"\n )\n```\n\n

\n\n\n\n

🤖 Prompt for AI Agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nIn `@leakmine/bigquery.py` around lines 139 - 158, The BigQuery sweep query in the\ncontent-building logic is still returning rows labeled as 'other' because the\nWHERE clause only filters for acquire matches; update the query assembled in the\nfunction that builds the SELECT so it excludes rows where the CASE falls through\nto 'other' and only returns acquire hits that do not also match the cleanup\nregex. Keep the existing pair-based CASE logic, but tighten the filter around\nthe same symbols (`cases`, `any_acquire`, and the returned SQL string) so the\nresult set contains only true leak candidates.\n```\n\n

\n\n\n\n\n\n\n\n","_📐 Maintainability & Code Quality_ | _🟡 Minor_ | _⚡ Quick win_\n\n**Fix the Ruff errors in this module.**\n\nLine 58 uses the ambiguous name `l`, and Lines 192/194/197/199 use multiple statements per line. Those are real Ruff violations in the added code. \n \n\n

Proposed fix

\n\n\nAlso applies to: 191-199\n\n

🧰 Tools

\n\n

🪛 Ruff (0.15.18)

\n\n[error] 58-58: Ambiguous variable name: `l`\n\n(E741)\n\n

\n\n

🤖 Prompt for AI Agents

\n\n\n\n\n\n\n\n_Source: Linters/SAST tools_\n\n\n\n✅ Addressed in commit b3f045a","_📐 Maintainability & Code Quality_ | _🟡 Minor_ | _⚡ Quick win_\n\n**Add language tags to the fenced blocks.**\n\nBoth fences currently trigger MD040. Use `text` for the flow diagram and `bash` for the CLI example so the docs render and lint cleanly.\n\n

Proposed fix

\n\n\n\n\nAlso applies to: 283-294\n\n

🧰 Tools

\n\n

🪛 markdownlint-cli2 (0.22.1)

\n\n[warning] 270-270: Fenced code blocks should have a language specified\n\n(MD040, fenced-code-language)\n\n

\n\n

🤖 Prompt for AI Agents

\n\n```\nVerify each finding against current code. Fix only still-valid issues, skip the\nrest with a brief reason, keep changes minimal, and validate.\n\nIn `@docs/leakfix-mine.md` around lines 270 - 274, Add explicit language tags to\nthe fenced code blocks in the docs so Markdown lint stops flagging MD040. Update\nthe diagram fence to use text and the CLI example fence to use bash, keeping the\nsurrounding content in the same sections of docs/leakfix-mine.md. Use the\nexisting fenced blocks around the BigQuery flow diagram and the bq-sql example\nto locate them, and preserve the current content while only changing the fence\nannotations.\n```\n\n

\n\n\n\n\n\n\n\n_Source: Linters/SAST tools_\n\n","_📐 Maintainability & Code Quality_ | _🟡 Minor_ | _⚡ Quick win_\n\n**Fix the inline-call example.**\n\n`addEventListener(` is missing the closing `)`, which reads awkwardly and matches the static warning. Rephrase the examples as balanced call forms.\n\n

Proposed fix

\n\n```diff\n-`addEventListener(` есть, `removeEventListener` нет; `setInterval(` без\n+`addEventListener()` есть, `removeEventListener()` нет; `setInterval()` без\n```\n

\n\n\n\n

📝 Committable suggestion

\n\n\n\n

🧰 Tools

\n\n

🪛 LanguageTool

\n\n

🤖 Prompt for AI Agents

\n\n\n\n\n\n\n\n_Source: Linters/SAST tools_\n\n","**_{_{![P2 Badge](https://img.shields.io/badge/P2-yellow?style=flat)}} Filter out balanced files from contents sweep**\n\nWhen a file contains both an acquire and its cleanup, such as `addEventListener(` plus `removeEventListener`, this `WHERE` clause still admits it because it only checks for the acquire pattern; the `CASE` then falls through to `signal = 'other'`, so `bq-sql --kind contents` exports balanced files as candidate sites. On large sweeps this pollutes the zero-fetch candidate set with non-leaks; filter on `(acquire AND NOT cleanup)` for at least one pair, or exclude the `other` rows before returning results.\n\nUseful? React with 👍 / 👎.","\n### 💡 Codex Review\n\nHere are some automated review suggestions for this pull request.\n\n**Reviewed commit:** `d4fa05d977`\n \n\n

ℹ️ About Codex in GitHub

","\n\n\n[![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/PhysShell/OwnAudit/pull/26?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)\n\n\n\n\n> [!NOTE]\n> Currently processing new changes in this PR. This may take a few minutes, please wait...\n> \n>

\n>

⚙️ Run configuration

\n> \n> **Configuration used**: defaults\n> \n> **Review profile**: CHILL\n> \n> **Plan**: Pro Plus\n> \n> **Run ID**: `8bfa6dc8-47da-41f1-bf4d-13450e13e468`\n> \n>

\n> \n>

\n>

📥 Commits

\n> \n> Reviewing files that changed from the base of the PR and between d4fa05d977a0d3d568a81b3881a7c33f5bc98b84 and b3f045a189414fc69cf6a07b6f58a1b37657a36e.\n> \n>

\n> \n>

\n>

📒 Files selected for processing (4)

\n> \n> * `docs/leakfix-mine.md`\n> * `leakmine/bigquery.py`\n> * `leakmine/cli.py`\n> * `leakmine/tests/test_leakmine.py`\n> \n>

📝 Walkthrough

\n\n\n\n\n

🚥 Pre-merge checks | ✅ 4 | ❌ 1

✅ Passed checks (4 passed)

\n\n

\n\n\n\n\n

✨ Finishing Touches

\n\n

📝 Generate docstrings

\n\n- [ ] Create stacked PR\n- [ ] Commit on current branch\n\n

🧪 Generate unit tests (beta)

\n\n- [ ] Create PR with unit tests\n- [ ] Commit unit tests in branch `claude/own-language-market-research-jgrrgm`\n\n

\n\n

\n\n\n\n\n---\n\n\n\n\n_{Comment `@coderabbitai help` to get the list of available commands.}\n\n","## What\n\nAdds `leakmine/bigquery.py` — the firehose-scale arm for the **10k–millions** tier (the \"not 10 repos, 10k\" question). docs/leakfix-mine.md §14.\n\n**Honest framing up front:** BigQuery removes the *discovery* wall, not the *diff-fetch* wall. GH Archive has PR metadata (title/body/repo/number), not diffs — so this yields a large, **pre-filtered** candidate queue that the existing `fetch → signals.classify → confirm` path then narrows. It doesn't magic away fetching diffs.\n\n## Two products (very different cost)\n\n**A. GH-Archive discovery — `gharchive_discovery_sql`** (events, `githubarchive.day.*`)\nMerged PRs with a leak keyword in **title OR body**, language-filtered, with a `changed_files` size cap to drop mega-refactors, deduped via `QUALIFY ROW_NUMBER()`. **Cheap** — `_TABLE_SUFFIX BETWEEN` scopes the scan to the chosen day-partitions (inside the 1 TB/mo free tier). This is the real \"10k, not 10\" discovery.\n\n**B. Contents sweep — `contents_sweep_sql`** (code snapshot, `github_repos`, **zero fetch**)\nRuns the **syntactic tier in SQL** — acquire-without-cleanup regex (e.g. `addEventListener(` with no `removeEventListener`, `setInterval(` without `clearInterval`). Scales to millions of files with no API calls. **Expensive**: any reference to `contents.content` scans the full ~2.7 TB column regardless of filters (~$13), so it defaults to the cheap `sample_*` tables; `--full` is opt-in and cost-warned in the emitted SQL.\n\n## Closing the loop\n\n`ingest_rows` / `metadata_score`: a BigQuery NDJSON export flows back into the leakmine store, scored at the **metadata tier** (keyword strength + PR-size heuristic — deliberately *not* the patch signal, which needs the diff). This ranks the narrow-fetch queue; survivors then get the real `signals.classify` / `confirm`.\n\n## CLI\n\n```\npython3 -m leakmine.cli bq-sql --kind gharchive --ecosystem dotnet_wpf --from 20240101 --to 20241231\npython3 -m leakmine.cli bq-ingest --rows rows.ndjson --ecosystem dotnet_wpf --store bq-out/corpus.db\npython3 -m leakmine.cli bq-sql --kind contents --ecosystem react_ts # --full = full ~2.7TB scan\n```\n\n## Tests / CI\n\n30 tests pass under `python3` and `python3 -O` (+4: gharchive SQL gen incl. partition/keyword/lang/size-cap/dedup; contents sweep sample-vs-full + missing-pairs error; metadata score; NDJSON ingest with dedup + store writes). SQL generation and ingest are fully offline-testable. Existing suite stays green.\n\n> Note: the SQL is generated here but run in your own GCP (no creds in CI by design). The cost model is documented in §14 and embedded as comments in the emitted SQL.\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)\n\n---\n_Generated by [Claude Code](https://claude.ai/code/session_01W6Lbeeoqy7v3wjzV43iLun)_\n\n\n\n## Summary by CodeRabbit\n\n* **New Features**\n * Added BigQuery-based discovery and ingest options for large-scale scans, including commands to generate queries and import results into the pipeline.\n * Added a faster zero-fetch sweep mode with sampling by default and an optional full scan.\n\n* **Documentation**\n * Expanded guidance with example workflows, limits, and cost/accuracy notes for large BigQuery runs.\n\n* **Bug Fixes**\n * Improved candidate deduplication and ranking so weaker matches are handled more consistently during ingest.\n\n","base_ref","head_ref","pull_request_info",[1801],{"_633":46,"_46":1802},"Thanks @coderabbitai @chatgpt-codex-connector — all 9 addressed in b3f045a:\n\n**Functional**\n- **Contents sweep returning balanced files (Major, flagged twice)**: the `WHERE` now requires `(acquire AND NOT cleanup)` per pair and the `CASE` drops its `ELSE 'other'`, so a file with both `addEventListener(` and `removeEventListener` is no longer emitted as a candidate. The sweep is now genuinely \"acquire without cleanup.\"\n- **Per-kind `--limit` (Major)**: default is now `None` → 50000 for `gharchive`, 5000 for `contents`, so the pricier content scan isn't silently widened 10×.\n- **Streaming ingest (Major)**: added `iter_ndjson` (line-by-line); `bq-ingest` streams it and `ingest_rows` takes any `Iterable`, so a multi-GB export ingests at bounded memory (only the dedup set is held).\n\n**Lint / docs / tests**\n- Renamed ambiguous `l` → `lang`; split the multi-statement lines in `metadata_score` (Ruff E741 / multiple-statements).\n- Docs: balanced call forms (`addEventListener()` …) and `text`/`bash` language tags on the fenced blocks (MD040).\n- Tests: assert the actual title-**OR**-body fragment (so an AND regression fails), and assert the mega-PR penalty score (dropped the unused `lo` binding).\n\n30 tests pass under `python3` and `-O`.\n\n---\n_Generated by [Claude Code](https://claude.ai/code)_",[1989,1990],[],{"_638":639,"_46":640,"_641":642},{"_612":1980},{"_337":1979},{"_337":1838},{"_304":305,"_295":306,"_307":308,"_309":59,"_310":15,"_311":15,"_312":313,"_314":315,"_316":267,"_317":127,"_318":319,"_320":321,"_322":323,"_324":-5,"_325":-5,"_326":1837,"_328":329,"_330":331,"_332":333,"_334":335},{"_42":123,"_193":194,"_195":196,"_197":15,"_198":199,"_200":201,"_202":1826,"_204":-5,"_205":206,"_207":15,"_208":1827,"_210":199,"_211":199,"_212":1828,"_214":1829,"_216":1830,"_218":-5,"_219":220,"_221":222,"_223":224,"_225":1831,"_55":-5,"_119":227,"_228":1832,"_230":1833,"_232":233,"_234":15,"_235":15},{"_133":1819},[1813,1814,1815,1816,1817,1818],{"_148":171,"_150":165,"_152":166,"_154":163,"_156":172,"_158":159,"_160":32,"_161":-5},{"_148":170,"_150":165,"_152":166,"_154":163,"_156":167,"_158":159,"_160":32,"_161":-5},{"_148":168,"_150":165,"_152":166,"_154":163,"_156":169,"_158":159,"_160":32,"_161":-5},{"_148":164,"_150":165,"_152":166,"_154":163,"_156":167,"_158":159,"_160":32,"_161":-5},{"_148":162,"_150":151,"_152":153,"_154":163,"_156":157,"_158":159,"_160":32,"_161":-5},{"_148":149,"_150":151,"_152":153,"_154":155,"_156":157,"_158":159,"_160":32,"_161":-5},[1820,1821,1822],{"_177":1825,"_179":190},{"_177":1824,"_179":186},{"_177":1823,"_179":180},{"_181":182,"_92":183,"_119":184},{"_181":187,"_92":183,"_119":188},{"_181":191,"_92":183,"_119":192},{"_297":298,"_299":-5,"_300":-5,"_301":-5,"_302":-5,"_303":-5},[257],{"_257":1834},{"_239":240,"_241":242,"_243":244,"_245":246,"_247":248,"_249":250,"_251":252,"_253":254,"_255":256},{},[],{"_238":59},{"_236":59,"_237":199},{"_42":259,"_181":260,"_261":1835,"_263":196,"_264":265,"_266":267,"_268":269,"_270":271,"_272":273,"_274":64,"_275":15,"_276":15,"_277":59,"_278":59,"_279":59,"_280":15,"_281":-5,"_160":282,"_283":-5,"_55":1836},{"_289":290,"_42":291,"_92":292,"_293":294,"_295":296},{"_285":59,"_286":59,"_287":59,"_288":59},{"_289":290,"_181":290,"_336":-5,"_293":294,"_42":291},[1839,1840,1841,1842,1843,1844,1845,1846,1847,1848,1849,1850,1851,1852,1853,1854,1855,1856,1857,1858,1859,1860,1861,1862,1863,1864,1865,1866,1867,1868,1869,1870,1871],{"_154":585,"_373":1975},{"_154":388,"_373":1972},{"_154":575,"_373":1969},{"_154":392,"_373":1964},{"_154":392,"_373":1960},{"_154":392,"_373":1956},{"_154":407,"_373":1951},{"_154":407,"_373":1947},{"_154":392,"_373":1943},{"_154":392,"_373":1941},{"_154":407,"_373":1937},{"_154":392,"_373":1935},{"_154":392,"_373":1933},{"_154":407,"_373":1929},{"_154":392,"_373":1927},{"_154":407,"_373":1923},{"_154":392,"_373":1919},{"_154":392,"_373":1917},{"_154":392,"_373":1915},{"_154":392,"_373":1913},{"_154":392,"_373":1911},{"_154":407,"_373":1907},{"_154":407,"_373":1903},{"_154":392,"_373":1899},{"_154":392,"_373":1897},{"_154":407,"_373":1893},{"_154":392,"_373":1891},{"_154":407,"_373":1887},{"_154":392,"_373":1883},{"_154":407,"_373":1881},{"_154":392,"_373":1877},{"_154":388,"_373":1875},{"_154":372,"_373":1872},{"_375":376,"_87":1873},{"_378":379,"_380":381,"_103":-5,"_42":382,"_383":1874,"_385":386,"_387":113},[],{"_387":1876,"_375":376},{"_92":391},{"_394":1878,"_375":376,"_396":382},{"_397":1879,"_399":-5,"_42":-5,"_400":-5,"_401":402,"_96":93,"_92":403},[1880],{"_46":405,"_92":406},{"_394":1882,"_375":376,"_396":382},{"_42":410,"_401":402,"_46":405,"_92":411},{"_394":1884,"_375":376,"_396":382},{"_397":-5,"_414":415,"_42":-5,"_400":-5,"_416":1885,"_92":418},[1886],{"_46":420,"_92":421},{"_394":1888,"_375":376,"_396":382},{"_397":1889,"_42":425,"_416":1890,"_92":418},[],[420],{"_394":1892,"_375":376,"_396":382},{"_429":430,"_42":-5,"_400":-5,"_431":432,"_92":433},{"_394":1894,"_375":376,"_396":382},{"_436":437,"_438":439,"_440":1895,"_442":443,"_380":32,"_444":32,"_42":430,"_445":446,"_447":448,"_387":113,"_92":449},[1896],{"_438":451,"_92":452},{"_394":1898,"_375":376,"_396":382},{"_455":456,"_429":430,"_42":-5,"_400":-5,"_181":457,"_458":-5,"_92":459},{"_394":1900,"_375":376,"_396":382},{"_397":-5,"_414":462,"_42":-5,"_400":-5,"_416":1901,"_92":418},[1902],{"_46":465,"_92":421},{"_394":1904,"_375":376,"_396":382},{"_397":1905,"_42":469,"_416":1906,"_92":418},[],[465],{"_394":1908,"_375":376,"_396":382},{"_436":473,"_438":474,"_440":1909,"_442":443,"_380":476,"_444":32,"_42":477,"_445":478,"_447":448,"_387":113,"_92":449},[1910],{"_438":480,"_92":452},{"_394":1912,"_375":376,"_396":382},{"_429":483,"_42":-5,"_400":-5,"_431":484,"_92":433},{"_394":1914,"_375":376,"_396":382},{"_455":487,"_429":483,"_42":-5,"_400":-5,"_181":488,"_458":-5,"_92":459},{"_394":1916,"_375":376,"_396":382},{"_429":477,"_42":-5,"_400":-5,"_431":491,"_92":433},{"_394":1918,"_375":376,"_396":382},{"_455":494,"_429":477,"_42":-5,"_400":-5,"_181":457,"_458":-5,"_92":459},{"_394":1920,"_375":376,"_396":382},{"_397":-5,"_414":497,"_42":-5,"_400":-5,"_416":1921,"_92":418},[1922],{"_46":500,"_92":421},{"_394":1924,"_375":376,"_396":382},{"_397":1925,"_42":504,"_416":1926,"_92":418},[],[500],{"_394":1928,"_375":376,"_396":382},{"_429":508,"_42":-5,"_400":-5,"_431":509,"_92":433},{"_394":1930,"_375":376,"_396":382},{"_436":512,"_438":513,"_440":1931,"_442":443,"_380":32,"_444":32,"_42":508,"_445":515,"_447":448,"_387":113,"_92":449},[1932],{"_438":517,"_92":452},{"_394":1934,"_375":376,"_396":382},{"_455":520,"_429":508,"_42":-5,"_400":-5,"_181":457,"_458":-5,"_92":459},{"_394":1936,"_375":376,"_396":382},{"_429":523,"_42":-5,"_400":-5,"_431":524,"_92":433},{"_394":1938,"_375":376,"_396":382},{"_436":527,"_438":528,"_440":1939,"_442":443,"_380":32,"_444":32,"_42":523,"_445":530,"_447":448,"_387":113,"_92":449},[1940],{"_438":532,"_92":452},{"_394":1942,"_375":376,"_396":382},{"_455":535,"_429":523,"_42":-5,"_400":-5,"_181":457,"_458":-5,"_92":459},{"_394":1944,"_375":376,"_396":382},{"_397":-5,"_414":538,"_42":-5,"_400":-5,"_416":1945,"_92":418},[1946],{"_46":541,"_92":421},{"_394":1948,"_375":376,"_396":382},{"_397":1949,"_42":545,"_416":1950,"_92":418},[],[541],{"_394":1952,"_375":376,"_396":382},{"_549":-5,"_397":1953,"_42":551,"_92":552},[1954],{"_46":554,"_555":1955,"_92":46},[],{"_394":1957,"_375":376,"_396":382},{"_397":1958,"_399":-5,"_42":-5,"_400":-5,"_401":-5,"_96":326,"_92":403},[1959],{"_46":554,"_92":561},{"_394":1961,"_375":376,"_396":382},{"_397":1962,"_399":-5,"_42":-5,"_400":-5,"_401":-5,"_96":326,"_92":403},[1963],{"_46":566,"_92":561},{"_394":1965,"_375":376,"_396":382},{"_397":1966,"_399":-5,"_42":-5,"_400":-5,"_401":-5,"_96":570,"_92":403},[1967,1968],{"_46":574,"_92":561},{"_46":573,"_92":561},{"_375":376,"_87":1970},{"_103":-5,"_42":382,"_383":1971,"_385":386,"_387":579},[],{"_387":1973,"_375":376},{"_582":1974,"_92":584},[],{"_587":1976},{"_589":-5,"_590":-5,"_591":592,"_593":386,"_442":443,"_594":15,"_595":-5,"_596":-5,"_597":-5,"_598":599,"_42":376,"_600":601,"_181":-5,"_602":-5,"_161":603,"_604":199,"_605":-5,"_447":606,"_387":1977,"_608":-5,"_77":1978,"_610":386},{"_92":391},[],[],[1981],{"_42":615,"_616":1982,"_618":619,"_620":619,"_397":1983,"_387":622,"_399":-5,"_623":32,"_624":1984,"_626":627,"_628":-5},{"_96":326,"_181":-5,"_624":1988},{"_633":46,"_634":1987},{"_629":1985},{"_631":1986},[],[636],{},{"_92":403,"_96":93,"_397":2008},{"_92":645,"_646":1991},[1992,1993,1994,1995],{"_161":1464,"_653":-5,"_654":2005,"_656":-5},{"_161":1300,"_653":-5,"_654":2002,"_656":-5},{"_161":1070,"_653":-5,"_654":1999,"_656":-5},{"_161":652,"_653":-5,"_654":1996,"_656":-5},[1997],{"_658":659,"_660":661,"_397":1998,"_663":59},[664,199,665,199,666,667,668,669,670,671,672,673,674,675,676,199,677,678,679,199,680,681,682,199,199,683,684,685,199,199,686,199,687,688,689,690,691,692,693,694,695,696,697,698,699,700,670,199,199,701,702,703,704,705,706,707,708,709,710,711,199,199,712,713,714,715,716,717,718,719,720,199,199,721,722,723,724,725,717,726,727,199,199,728,199,729,730,731,732,733,734,199,199,735,736,737,738,739,740,199,199,741,742,743,744,745,732,746,199,199,747,748,749,750,751,752,753,199,199,754,199,755,756,199,199,757,758,759,760,199,761,762,763,764,199,199,765,766,767,768,769,770,199,771,772,773,774,775,776,199,199,777,778,779,780,781,782,693,694,783,784,700,670,199,199,785,786,787,788,789,790,791,792,793,199,199,794,199,795,796,797,798,799,800,801,199,199,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,807,818,819,199,820,821,822,823,824,199,199,825,199,826,827,828,829,830,831,832,833,834,835,836,837,838,839,199,199,840,841,842,843,844,845,828,846,847,848,833,849,850,851,852,199,199,853,854,855,856,857,858,859,860,833,861,862,863,864,859,865,833,866,199,199,867,868,828,869,870,833,834,871,872,873,199,199,874,199,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,199,199,895,199,896,897,898,899,900,901,902,903,904,199,905,906,907,908,909,910,911,912,913,914,915,199,199,916,917,918,906,907,919,911,920,921,199,199,922,199,923,924,925,926,199,927,928,929,930,199,199,931,932,933,934,935,936,937,938,199,199,939,940,941,942,943,944,945,946,947,948,949,950,951,952,199,199,953,954,955,956,957,958,885,959,960,961,199,962,963,199,964,965,199,966,967,968,969,970,971,972,973,974,199,199,975,199,976,966,977,978,979,980,981,982,983,984,985,199,199,986,987,988,966,989,990,991,992,993,994,995,199,199,996,997,966,998,999,982,1000,1001,199,199,1002,1003,966,1004,1005,1006,1007,1008,1009,1010,199,199,1011,1012,1013,928,1014,1015,1016,1017,1018,1019,1020,1021,1022,1023,1024,199,199,1025,1026,1027,1028,1029,1030,1031,1032,1033,1034,1005,1035,1036,1009,1037,199,199,1038,1039,1040,1041,1042,1043,1044,1045,1046,1047,199,199,1048,1049,1050,1051,1052,1053,1054,1055,1056,966,1057,1058,1059,1060,1061,1062,199,199,1063,1064,1065,1066,1067,199,199,1068,1069],[2000],{"_658":659,"_660":1073,"_397":2001,"_663":59},[1075,199,1076,1077,1078,1079,1080,199,1081,199,1082,199,1083,1084,1085,1086,199,1087,1088,199,1089,199,1090,1091,1092,1093,199,1094,1095,1096,1097,1098,199,1099,1100,199,1101,199,1102,1103,1104,1105,199,1106,199,1107,1108,199,1109,1110,1111,1112,1113,1114,1115,1109,199,1116,1117,1118,1119,1120,1121,199,1122,199,1123,1124,199,1125,1126,1127,1128,1129,1130,1131,1132,1133,199,1134,1135,1136,199,1137,199,1138,1139,199,1140,1141,1142,199,1143,1144,1145,1146,199,1109,1147,1109,199,1148,1149,1150,1151,1152,199,1153,1154,1155,199,1156,199,1157,1158,1159,1160,1161,199,1162,199,1163,199,1164,1165,1166,1167,1168,1169,1170,1171,199,1172,199,1173,1174,199,1175,1165,1176,1177,1178,199,1179,1180,1181,1182,1183,1184,1185,1186,1187,199,1188,199,1189,1190,1191,199,1192,1193,1194,199,1195,1196,1197,1198,1199,1200,1201,199,1202,1203,199,1204,199,1205,1206,1207,1208,1209,1210,199,1211,199,1109,1212,1213,1214,1215,1216,1217,1218,1109,199,1219,1220,199,1221,199,1222,1223,1224,1225,1226,1227,1228,199,1229,199,1230,1231,1232,1233,1234,1235,199,1236,199,1237,1238,1239,1240,1241,1242,199,1243,1109,1244,1245,1109,199,1246,1247,1248,1249,199,1250,1251,1252,1253,1254,199,1255,199,1256,1257,1258,1259,1260,199,1261,199,1262,1263,1264,1265,1266,199,1267,1268,1269,1270,1271,1272,1273,199,1274,199,1275,1276,1277,1278,1109,199,1279,1280,1281,1282,199,1283,199,1284,1285,1286,1287,199,1288,1289,1290,199,1291,1292,1109,199,1293,1294,1295,1296,1297,1298,1299],[2003],{"_658":659,"_660":1303,"_397":2004,"_663":59},[1305,199,1306,1307,1308,1309,1310,1311,1312,199,1313,1314,670,1315,199,1316,671,672,675,199,1317,199,199,1318,1319,1320,199,199,1321,1322,1323,1324,199,199,1325,1326,1324,199,199,1327,1328,1329,1330,1331,1332,1333,1334,1324,199,199,1335,1336,199,199,1337,1338,828,1339,1340,1341,1342,833,1343,1344,1324,199,199,1345,1346,1347,1348,1349,1350,1351,1352,1324,199,199,1353,1354,1355,1356,1357,1358,1359,1360,1324,199,199,1361,1362,1363,1364,1365,1366,1367,833,1368,1369,1370,1371,1372,1373,1374,1324,199,199,1375,1376,1377,1378,1379,1380,1351,1381,1382,1383,1324,199,199,1384,1363,1364,1385,1386,1387,1368,1369,1388,1371,1372,1373,1374,1324,199,199,1389,1390,1391,1324,199,199,1392,1393,1394,199,1395,1396,1397,1398,199,1399,1400,1401,1402,1403,1404,199,1405,1406,1407,1408,1409,1410,199,1411,1412,1413,1414,1415,199,1416,1417,1418,1419,1420,1421,199,1422,1423,1424,1425,1426,199,1427,1428,1429,1430,1431,1432,1433,1434,1435,1436,199,1437,1438,1439,1440,1441,1442,1443,1444,1445,1446,1447,199,1448,1449,1450,1451,1452,1453,1454,199,1455,1456,1457,1458,1459,1460,199,1461,1462,199,199,1068,1463],[2006],{"_658":659,"_660":1467,"_397":2007,"_663":59},[1469,199,1470,1471,1472,199,1473,1474,1475,1476,1477,1478,199,1479,1480,1481,1482,1483,1484,199,1485,1486,1487,670,1315,199,671,1488,1489,199,1490,199,199,1491,199,1492,1493,1494,1495,1496,1497,1498,1499,1500,199,1501,1502,1503,1504,1505,1506,1507,1508,1509,1510,833,1511,1512,1513,1514,1515,1516,1517,1518,833,1519,1520,1521,1522,1523,1524,1525,1526,1527,1528,1529,1530,1531,1532,1533,1534,1535,1536,1537,1538,1539,1540,1541,1542,1543,1544,1545,1546,1547,833,199,199,1548,199,1549,1550,1551,1552,1553,1554,1555,1556,1557,1558,1559,1556,1560,1561,1562,1556,1563,1564,1556,1565,199,199,1566,1493,1494,1567,1568,1499,1569,199,1570,1571,1572,1504,1573,1574,1575,1576,1577,1578,1579,1580,1581,1582,833,1583,1584,1585,1586,1587,1588,833,1519,1589,1590,1591,1592,1593,1594,1595,1596,1597,1598,1599,1600,1547,833,199,199,1601,199,1602,1603,1604,1605,1606,1607,199,1608,1609,1610,1611,1612,1613,1614,199,199,1615,1616,1617,199,1618,1619,1620,1504,1505,1621,1622,1623,1624,1625,1626,1627,1628,1629,1630,1631,1632,1633,1634,1635,1636,199,199,1637,1638,1639,199,1640,1641,1642,1643,1504,1644,1645,1646,1647,1648,1649,1650,1651,1649,1652,1653,1654,1655,1656,1657,1658,1649,1659,1660,1661,1662,1663,1664,1665,1666,1667,1668,1669,1670,1671,1672,1673,1674,1675,1676,199,199,1677,1678,1679,199,199,1680,1681,1682,1683,1319,1684,1685,1686,1687,199,199,1688,1689,1690,1691,1692,1693,1694,1695,1565],[2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024],{"_633":46,"_46":1738},{"_633":1714,"_161":1464,"_658":1736,"_660":1737},{"_633":46,"_46":1735},{"_633":1714,"_161":1300,"_658":1733,"_660":1734},{"_633":1714,"_161":1300,"_658":1731,"_660":1732},{"_633":46,"_46":1730},{"_633":1714,"_161":1300,"_658":1728,"_660":1729},{"_633":1714,"_161":1464,"_658":1726,"_660":1727},{"_633":1714,"_161":1464,"_658":1725,"_660":1720},{"_633":46,"_46":1724},{"_633":1714,"_161":1070,"_658":1722,"_660":1723},{"_633":1714,"_161":1070,"_658":1720,"_660":1721},{"_633":46,"_46":1719},{"_633":1714,"_161":652,"_658":1717,"_660":1718},{"_633":1714,"_161":652,"_658":1715,"_660":1716},{"_633":46,"_46":1713},[2027,2028,2029,2030,2031,2032,2033,2034,2035,2036,2037,2038,2039,2040],{"_1748":1749,"_263":196,"_1750":305,"_274":-5,"_1751":257,"_1752":-5,"_1753":-5,"_1754":1755,"_1756":1757,"_1758":-5,"_1759":-5,"_1760":1761,"_1762":15,"_1763":59,"_1764":54,"_1765":15},{"_92":403,"_96":326,"_397":2041},{"_312":313,"_314":1796,"_1797":319,"_1798":129,"_322":-5,"_92":1799},{"_92":1780,"_314":1795,"_161":-5,"_1782":-5,"_1783":-5},{"_92":1780,"_314":1794,"_161":-5,"_1782":-5,"_1783":-5},{"_92":1780,"_314":1793,"_161":1464,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1792,"_161":1070,"_1782":1785,"_1783":1723},{"_92":1780,"_314":1791,"_161":1070,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1790,"_161":1464,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1789,"_161":1464,"_1782":1785,"_1783":1737},{"_92":1780,"_314":1788,"_161":1300,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1787,"_161":1300,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1786,"_161":652,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1784,"_161":652,"_1782":1785,"_1783":-5},{"_92":1780,"_314":1781,"_161":-5,"_1782":-5,"_1783":-5},[2042],{"_633":46,"_46":1802},"task_e_6a409fa22978832081ecc6b84d8ad183",1782620066.162235,"GitHub Mention: leakmine: BigQuery arm — firehose-scale discovery + zero-fetch contents sweep","has_generated_title","has_unread_turn","denormalized_metadata",{"_2074":32,"_2075":2076,"_2077":-5,"_2078":-5,"_2069":2079,"_2080":-5,"_2081":-5,"_2082":-5,"_2083":-5},"task_status_display",{"_2062":2063,"_2064":196,"_130":127,"_2065":-5,"_2066":2067},"creator_workspace","external_pull_requests",[2055],{"_42":2056,"_2057":83,"_2058":2059,"_2060":129},"github-1280321876-26","assistant_turn_id","pull_request",{"_304":305,"_295":306,"_307":308,"_309":59,"_310":15,"_311":15,"_312":313,"_314":315,"_316":267,"_317":127,"_318":319,"_320":321,"_322":323,"_324":-5,"_325":-5,"_326":2061,"_328":329,"_330":331,"_332":333,"_334":335},"codex_updated_sha",{"_289":290,"_181":290,"_336":-5,"_293":294,"_42":291},"latest_turn_status_display",{"_2068":83,"_112":113,"_2069":-5,"_94":2070,"_2071":-5,"_2072":-5,"_2073":2067,"_114":-5},"environment_label","fix_task_id","initial_intent","pr","turn_id","diff_stats",[],"image_asset_pointer_citations","issue_counts","intent","num_follow_ups","repo_name_map",{},"last_user_turn_created_at","last_turn_status",{"_2084":-5,"_2085":-5,"_2086":-5},"last_pushed_turn_id","start_branch_name","pushed_branch_name","cached_pull_request_data","files_modified","lines_added","lines_removed","dd",{"_2098":2099,"_2100":2101},"disablePrefetch","shouldPrefetchAccount","shouldPrefetchUser","shouldPrefetchSystemHints","promoteCss","disableSSR","stripModulepreloadImports","statsigGateEvaluationsPromise",["P",2097],"traceId","8941737934027938287","traceTime",1785125326800]