Skip to content

perf(dbf): reduce per-value decoding overhead - #337

Open
devdudumuniz wants to merge 1 commit into
AlertaDengue:mainfrom
devdudumuniz:performance/dbf-decoding
Open

perf(dbf): reduce per-value decoding overhead#337
devdudumuniz wants to merge 1 commit into
AlertaDengue:mainfrom
devdudumuniz:performance/dbf-decoding

Conversation

@devdudumuniz

Copy link
Copy Markdown
Contributor

Summary

  • decode NumPy byte-string columns with a list comprehension
  • centralize the optimized path in _decode_column
  • use the same implementation for full-file and streaming reads

Performance

A local microbenchmark on representative fixed-width NumPy byte strings showed approximately 1.51x faster decoding at both 1,000 and 100,000 values.

Validation

  • pytest -q pysus/tests/data/test_dbf_reader.py pysus/tests/data/test_dbf_reader_parity.py (40 passed)
  • black --check pysus/data/dbf_reader.py
  • flake8 pysus/data/dbf_reader.py

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@2102ba2). Learn more about missing BASE report.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #337   +/-   ##
=======================================
  Coverage        ?   95.23%           
=======================================
  Files           ?      169           
  Lines           ?    20016           
  Branches        ?        0           
=======================================
  Hits            ?    19063           
  Misses          ?      953           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants