Repository navigation
Fix Italian electronic tests for raw and post-processed punctuation - #533
Open
janbernloehr wants to merge 1 commit into
Open
janbernloehr wants to merge 1 commit into
janbernloehr wants to merge 1 commit into
Conversation
Signed-off-by: Jan Bernloehr <jbernloehr@nvidia.com>
10 of 14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Fix the punctuation mode used by Italian electronic normalization tests and cover optional punctuation cleanup separately.
The electronic fixture is shared with Sparrowhawk and expects raw grammar output, including a space before a final period (
punto com .). The Python test enabledpunct_post_process, which correctly removes that space (punto com.), causing four failures. Run the shared fixture without punctuation cleanup and add eight explicit cleanup cases covering those four sentences, a URL path, an email address, a social handle, and a URL without sentence punctuation. Existing fixture expectations and production grammars are unchanged.Before your PR is "Ready for review"
Pre checks:
git commit -sto sign.pytestor (if your machine does not have GPU)pytest --cpufrom the root folder (given you marked your test cases accordingly@pytest.mark.run_only_on('CPU')).bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...pytestand Sparrowhawk here. (Not applicable: test correction.)__init__.pyfor every folder and subfolder, includingdatafolder which has .TSV files? (Not applicable: no new folders.)Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.to all newly added Python files? (Not applicable: no new Python files.)Copyright 2015 and onwards Google, Inc.. See an example here. (Not applicable.)try import: ... except: ...) if not already done. (Not applicable: no import changes.)Validation:
ddadfb2a38d2bc6b8cc6232c4f915eb60f500688with Python 3.13.12 / Pynini 2.1.6.post1 and Python 3.14.3 / Pynini 2.1.7: 48 passed, 4 failed on each.12:30mismatch: the shell compares onlydodici e trenta minuti, while Sparrowhawk returns the fixture's other valid option,dodici e mezza. A supplemental batch check of all 52 Italian deployment fixtures accepts only their explicitly listed alternatives and passes on both versions, including all 13 electronic cases. The unmodified shell suite's failure keeps the all-tests checkbox unchecked.PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.