ℹ️ About Codex in GitHub

\n
\n\n[Your team has set up Codex to review pull requests in this repo](https://chatgpt.com/codex/cloud/settings/general). Reviews are triggered when you\n- Open a pull request for review\n- Mark a draft as ready\n- Comment \"@codex review\".\n\nIf Codex has suggestions, it will comment; otherwise it will react with 👍.\n\n\n\n\nCodex can also answer questions or update the PR. Try commenting \"@codex address that feedback\".\n \n

","**_{_{![P2 Badge](https://img.shields.io/badge/P2-yellow?style=flat)}} Resolve parser regex fixture path from test file location**\n\nThis test reads `backend/src/...` via a cwd-relative path, which fails when pytest is run from the `backend/` directory (for example with the project’s backend-local pytest config). In that context the path resolves to `backend/backend/src/...` and the new apostrophe tests fail with `FileNotFoundError`; deriving the path from `__file__` avoids cwd-dependent failures.\n\nUseful? React with 👍 / 👎.","tests/test_apostrophe_normalization_and_tokenization.py","**_{_{![P1 Badge](https://img.shields.io/badge/P1-orange?style=flat)}} Preserve hyphenated compounds in backend tokenizer**\n\nRestore hyphen handling in this regex: removing `-` from the word branch now splits compounds like `state-of-the-art` into `['state', '-', 'of', '-', 'the', '-', 'art']`, so the classifier emits a punctuation bone for each hyphen and inflates K counts. This contradicts the canon rule that hyphenated compounds emit one K per compound word, and it changes structural metrics for any hyphen-rich input.\n\nUseful? React with 👍 / 👎.","## Pull request overview\n\nThis PR tightens affix detection by validating stripped residuals against canon stems, while also adjusting apostrophe normalization and test organization.\n\n**Changes:**\n- Adds canon-backed residual validation for backend parser affix emission and optional stem gating in the core matcher.\n- Normalizes smart apostrophes to ASCII and adds regression tests for contractions and affix false positives.\n- Moves pytest collection toward lowercase `tests/` and removes older uppercase/Backend artifacts.\n\n### Reviewed changes\n\nCopilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.\n\n

Show a summary per file

\n\n| File | Description |\\n| ---- | ----------- |\\n| `backend/src/edcmbone/parser/turns_rounds.py` | Updates tokenizer and affix residual validation in the backend parser. |\\n| `core/operator/matcher.py` | Adds optional `valid_stems` gating to affix matching. |\\n| `core/parsing/normalizer.py` | Normalizes smart apostrophe variants to ASCII apostrophes. |\\n| `tests/test_affix_residual_validation.py` | Adds regression coverage for affix residual validation. |\\n| `tests/test_apostrophe_normalization_and_tokenization.py` | Adds regression coverage for apostrophe normalization/tokenization. |\\n| `backend/src/edcmbone/ucns/ucns_v04.py` | Adds packaged UCNS implementation under backend source tree. |\\n| `backend/src/edcmbone/ucns/closed_tokens.py` | Updates UCNS import to package-relative form. |\\n| `backend/pyproject.toml` | Changes pytest collection path to lowercase `tests`. |\\n| `Tests/test_backend.py` | Removes old uppercase backend regression suite. |\\n| `Tests/test_closed_tokens.py` | Removes old uppercase closed-token test copy. |\\n| `Documentation/spec.md` | Removes old documentation file. |\\n| `Backend/requirements.txt` | Removes old uppercase Backend requirements file. |\\n| `Backend/LICENSE` | Removes old uppercase Backend license copy. |\n

\n\n\n\n\n\n\n---\n\n💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.","Switching collection to `../tests` drops the existing backend regression coverage because `Tests/test_backend.py` is deleted in this PR and the corresponding `tests/test_backend.py` in the updated tree is empty. The parser, metrics, compression, and canon assertions from the deleted suite should be migrated before changing the configured test path, otherwise configured pytest runs no longer exercise those backend behaviors.\n","backend/pyproject.toml","This test reads `backend/src/...` relative to the current working directory, but `backend/pyproject.toml` is configured to collect `../tests` when pytest is run from the `backend/` directory. In that common invocation this resolves to `backend/backend/src/...` and fails before testing the regex; build the path from `__file__`/repo root instead of CWD.\n","This suffix-side gate has the same intermediate-residual problem for tokens that still include a valid prefix: a word such as `redoing` has final root `do`, but the suffix pass sees `redo` and rejects it before the prefix can be accounted for. Validate against `valid_stems` after considering the complete affix combination rather than after each single strip.","Validating the stem immediately after a prefix strip prevents multi-affix words from matching even when the final residual is valid. For example, with `valid_stems={\"do\"}`, `redoing` cannot emit `re` plus `ing` because the intermediate residual `doing` is rejected before the suffix pass. Defer validation until a full prefix/suffix strip candidate has produced its final root, or otherwise allow intermediate residuals that can still be stripped to a valid stem.","This tokenizer no longer preserves hyphenated compounds as one surface token: `state-of-the-art` is split into words plus three `-` punctuation tokens, so the parser can emit three K bones even though the frozen canon requires exactly one K per hyphenated compound. Preserve `[A-Za-z0-9]+(?:-[A-Za-z0-9]+)+` as a word token before the single-hyphen punctuation path (see canon_eng/spec_hard_freeze_appendix_v1.md:32-38 and backend/src/edcmbone/canon/data/bones_punct_v1.json:87-92).","base_ref","head_ref","pull_request_info",[720],{"_628":57,"_57":721},"@codex[agent] apply repair (path=backend/src/edcmbone/parser/turns_rounds.py line=None side=RIGHT)",[750,751],[],{"_320":749},{"_317":748},{"_317":747},{"_280":281,"_270":282,"_283":284,"_285":70,"_286":18,"_287":18,"_288":289,"_290":291,"_292":241,"_293":137,"_294":295,"_296":139,"_297":-5,"_298":-5,"_299":300,"_301":-5,"_302":-5,"_303":746,"_305":306,"_307":308,"_309":310,"_311":312},{"_53":133,"_166":-5,"_167":168,"_169":170,"_171":18,"_172":173,"_174":175,"_176":735,"_178":-5,"_179":180,"_181":18,"_182":736,"_184":173,"_185":173,"_186":737,"_188":738,"_190":739,"_192":-5,"_193":194,"_195":196,"_197":198,"_199":740,"_66":-5,"_129":201,"_202":741,"_204":742,"_206":207,"_208":18,"_209":18},{"_143":730},[731,732],{"_154":734,"_156":163},{"_154":733,"_156":157},{"_158":159,"_103":160,"_129":161},{"_158":164,"_103":160,"_129":165},{"_272":273,"_274":275,"_276":173,"_277":-5,"_278":-5,"_279":18},[231],{"_231":743},{"_213":214,"_215":216,"_217":218,"_219":220,"_221":222,"_223":224,"_225":226,"_227":228,"_229":230},{},[],{"_212":70},{"_210":70,"_211":173},{"_53":233,"_158":234,"_235":744,"_237":170,"_238":239,"_240":241,"_242":243,"_244":245,"_246":247,"_248":75,"_249":18,"_250":70,"_251":70,"_252":70,"_253":70,"_254":70,"_255":-5,"_256":257,"_258":-5,"_66":745},{"_264":265,"_53":266,"_103":267,"_268":269,"_270":271},{"_260":70,"_261":18,"_262":70,"_263":70},{"_264":313,"_158":313,"_314":-5,"_268":315,"_53":316},[],[],[],{"_103":623,"_107":104,"_339":757},{"_103":324,"_325":752},[753],{"_328":329,"_330":-5,"_331":754,"_333":-5},[755],{"_335":336,"_337":338,"_339":756,"_341":70},[342,343,344,345,173,346,347,348,173,349,350,351,173,352,353,354,342,173,355,173,356,357,358,173,359,173,360,361,360,173,362,363,173,364,173,365,366,367,368,369,370,371,173,372,373,173,173,374,375,173,376,173,377,378,173,372,379,173,173,380,381,173,382,383,384,385,173,386,387,173,388,389,390,391,173,392,393,173,372,394,173,173,395,396,173,397,398,399,173,400,401,402,389,403,404,405,406,407,173,392,408,409,173,372,410,411,412,173,173,413,414,173,415,416,399,417,173,418,173,419,420,173,421,422,423,424,425,173,372,426,427,412,173,173,360,428,360,173,429,430,431,432,433,434,435,436,437,438,439,173,173,440,441,442,443,444,445,173,446,447,448,449,450,173,173,360,451,360,173,452,453,173,454,455,456,457,458,459,460,461,173,462,463,464,465,466,173,467,468,469,470,471,173,472,473,474,475,476,477,478,479,480,173,481,482,173,483,173,173,360,484,360,173,485,486,487,173,173,488,489,490,491,173,173,360,492,360,173,493,494,173,495,496,173,497,498,499,500,501,412,502,503,504,501,412,173,505,506,507,508,509,510,511,512,513,173,514,515,516,517,518,519,520,521,412,173,522,523,173,524,525,526,527,528,529,530,531,532,533,534,535,173,536,537,538,539,540,541,542,543,544,545,173,546,547,548,549,550,551,173,552,553,554,555,556,557,558,559,560,561,562,563,564,565,566,567,568,550,551,173,569,570,571,572,558,559,573,561,562,563,564,565,566,567,568,550,551,173,574,575,576,577,550,551,173,578,579,580,173,581,173,173,360,582,360,173,583,584,173,585,586,587,588,589,590,591,592,593,594,595,596,597,598,599,173,600,601,602,459,603,604,173,605,173,606,607,608,609,610,611,173,612,613,173,173,360,614,360,173,615,616,617,618,619,620,621,622],[758,759,760],{"_628":57,"_57":633},{"_628":630,"_328":329,"_335":631,"_337":632},{"_628":57,"_57":629},[763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791],{"_643":644,"_237":170,"_645":281,"_248":-5,"_646":231,"_647":18,"_648":-5,"_649":-5,"_650":651,"_652":653,"_654":-5,"_655":-5,"_656":657,"_658":18,"_659":70,"_660":65,"_661":18,"_662":-5},{"_103":623,"_107":303,"_339":792},{"_288":289,"_290":291,"_716":295,"_717":139,"_299":-5,"_103":718},{"_103":692,"_290":715,"_328":329,"_695":696,"_697":-5},{"_103":692,"_290":714,"_328":694,"_695":696,"_697":-5},{"_103":692,"_290":713,"_328":694,"_695":696,"_697":-5},{"_103":692,"_290":712,"_328":707,"_695":696,"_697":-5},{"_103":692,"_290":710,"_328":711,"_695":696,"_697":-5},{"_103":692,"_290":709,"_328":-5,"_695":-5,"_697":-5},{"_103":692,"_290":708,"_328":329,"_695":696,"_697":-5},{"_103":692,"_290":706,"_328":707,"_695":696,"_697":-5},{"_103":692,"_290":705,"_328":-5,"_695":-5,"_697":-5},{"_103":692,"_290":699,"_328":329,"_695":696,"_697":-5},{"_103":692,"_290":699,"_328":329,"_695":696,"_697":-5},{"_103":692,"_290":698,"_328":694,"_695":696,"_697":-5},{"_103":692,"_290":693,"_328":694,"_695":696,"_697":-5},{"_103":692,"_290":693,"_328":694,"_695":696,"_697":-5},{"_103":692,"_290":699,"_328":329,"_695":696,"_697":-5},{"_103":692,"_290":704,"_328":-5,"_695":-5,"_697":-5},{"_103":692,"_290":693,"_328":694,"_695":696,"_697":-5},{"_103":692,"_290":703,"_328":329,"_695":696,"_697":-5},{"_103":692,"_290":702,"_328":694,"_695":696,"_697":-5},{"_103":692,"_290":701,"_328":694,"_695":696,"_697":-5},{"_103":692,"_290":700,"_328":-5,"_695":-5,"_697":-5},{"_103":692,"_290":699,"_328":329,"_695":696,"_697":-5},{"_103":692,"_290":698,"_328":694,"_695":696,"_697":-5},{"_103":692,"_290":693,"_328":694,"_695":696,"_697":-5},{"_103":692,"_290":699,"_328":329,"_695":696,"_697":-5},{"_103":692,"_290":698,"_328":694,"_695":696,"_697":-5},{"_103":692,"_290":693,"_328":694,"_695":696,"_697":-5},[793],{"_628":57,"_57":721},"task_e_6a07fe1168d08320aae8d814d56453a7",1778908689.409678,"Github Mention: Fix #49: block false-positive affix bones with canon stem validation","has_generated_title","has_unread_turn","denormalized_metadata",{"_824":43,"_825":826,"_827":-5,"_828":-5,"_819":829,"_830":-5,"_831":-5,"_832":-5,"_833":-5},"task_status_display",{"_813":814,"_815":170,"_140":137,"_816":-5,"_817":653},"creator_workspace","external_pull_requests",[806],{"_53":807,"_808":94,"_809":810,"_811":139},"github-1158893709-65","assistant_turn_id","pull_request",{"_280":281,"_270":282,"_283":284,"_285":70,"_286":18,"_287":18,"_288":289,"_290":291,"_292":241,"_293":137,"_294":295,"_296":139,"_297":-5,"_298":-5,"_299":300,"_301":-5,"_302":-5,"_303":812,"_305":306,"_307":308,"_309":310,"_311":312},"codex_updated_sha",{"_264":313,"_158":313,"_314":-5,"_268":315,"_53":316},"latest_turn_status_display",{"_818":94,"_122":123,"_819":-5,"_105":820,"_821":-5,"_822":-5,"_823":653,"_124":-5},"environment_label","fix_task_id","initial_intent","turn_id","diff_stats",[],"image_asset_pointer_citations","issue_counts","intent","num_follow_ups","repo_name_map",{},"last_user_turn_created_at","last_turn_status",{"_834":-5,"_835":-5,"_836":-5},"last_pushed_turn_id","start_branch_name","pushed_branch_name","cached_pull_request_data","files_modified","lines_added","lines_removed","dd",{"_851":852,"_853":854},"disablePrefetch","shouldPrefetchAccount","shouldPrefetchUser","shouldPrefetchSystemHints","manageRootRouteStylesheet","mobileWebHandoffEnabled","promoteCss","disableSSR","stripModulepreloadImports","paidCampaignAppRedirect","statsigGateEvaluationsPromise",["P",850],"traceId","1006062447579876000","traceTime",1789509551298]