Skip to content

Handle empty input in normalize_list and normalize_manifest - #535

Open
0z5a wants to merge 1 commit into
NVIDIA:mainfrom
0z5a:fix/normalize-list-empty
Open

0z5a wants to merge 1 commit into
NVIDIA:mainfrom
0z5a:fix/normalize-list-empty

Conversation

@0z5a

@0z5a 0z5a commented Nov 13, 2026

Copy link
Copy Markdown

What does this PR do ?

normalize_list([]), inverse_normalize_list([]) and normalize_manifest on an empty manifest now return [] / write an empty output file instead of raising ValueError: range() arg 3 must not be zero.

Fixes #534

Changes

  • normalize_list: return [] for empty input. This also covers inverse_normalize_list and the normalize.py / inverse_normalize.py scripts with an empty --input_file.
  • normalize_manifest: use batch = max(1, min(len(lines), batch_size)), so an empty manifest gives an empty output file.
  • Two CPU unit tests in tests/nemo_text_processing/en/test_cardinal.py: test_normalize_list_empty and test_normalize_manifest_empty.

Testing (macOS arm64, Python 3.11, pynini 2.1.6.post1)

  • pytest --cpu tests/nemo_text_processing/en/test_cardinal.py -k empty: 2 failed on main (ValueError: range() arg 3 must not be zero at normalize.py:271 and :541), 2 passed with this change.
  • pytest --cpu tests/nemo_text_processing/en/test_{cardinal,money,ordinal,decimal}.py: 580 passed.
  • The scripts with an empty --input_file write an empty output file instead of raising.
  • black (-l 119 -S) and isort report no changes for the two files.

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

I ran the tests listed above, not the full pytest --cpu suite, and not the Sparrowhawk tests (the change is in the Python batching code, not in the grammars). The other unchecked items do not apply (no new feature, files, language or import guards).

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.

normalize_list() computed batch = min(len(texts), batch_size), which is
0 for an empty list, and range(0, 0, 0) raised "ValueError: range() arg
3 must not be zero". inverse_normalize_list() and the normalize.py and
inverse_normalize.py scripts with an empty --input_file failed the same
way. Return [] for empty input.

normalize_manifest() had the same batch computation and failed the same
way for an empty manifest. Use a batch size of at least 1, so an empty
manifest gives an empty output file.

Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

normalize_list / inverse_normalize_list raise "range() arg 3 must not be zero" on an empty list

1 participant