clean_path() normalises URL paths with os.path.normpath, breaking every URL-rooted dataset on Windows #1333

clean_path() normalises the path component of a URL with os.path.normpath, which on Windows rewrites every / as \:

# pyhealth/datasets/base_dataset.py
def clean_path(path: str) -> str:
    """Clean a path string."""
    if is_url(path):
        parsed = urlparse(path)
        cleaned_path = os.path.normpath(parsed.path)
        # Rebuild the full URL
        return urlunparse(parsed._replace(path=cleaned_path))

urlunparse then puts those backslashes back into the URL, so no dataset with a URL root can be read on Windows.

Reproduction

Checked on master at 4a173f7, Windows 11, Python 3.13.5.

>>> from pyhealth.datasets.base_dataset import clean_path
>>> clean_path("https://storage.googleapis.com/pyhealth/Synthetic_MIMIC-III/PATIENTS.csv.gz")
'https://storage.googleapis.com/\\pyhealth\\Synthetic_MIMIC-III\\PATIENTS.csv.gz'

End to end, any URL-rooted dataset fails. The three tests in tests/core/test_processor_schemas.py, which build a MIMIC3Dataset on https://storage.googleapis.com/pyhealth/Synthetic_MIMIC-III, all fail the same way:

FileNotFoundError: Neither path exists:
  https://storage.googleapis.com/\pyhealth\Synthetic_MIMIC-III\PATIENTS.csv.gz or
  https://storage.googleapis.com/\pyhealth\Synthetic_MIMIC-III\PATIENTS.csv
pyhealth\datasets\base_dataset.py:203: FileNotFoundError

The error surfaces in _csv_tsv_gz_path, but the mangling happens earlier, in clean_path.

Reach

clean_path has three call sites, and the bug is not limited to reading:

1. _scan_table, line 847 — the main read path for every table.
2. the join path, line 867 — same, for joined tables.
3. _resolve_cache_dir, line 540 — _source_fingerprint(clean_path(f"{self.root}/{file_path}")) goes into key["sources"], which is hashed into the dataset's cache UUID.

The third one has a consequence beyond Windows being broken: for the same dataset, the cache key is platform-dependent. With the synthetic MIMIC-III root and tables=["patients"]:

So a cache directory built on Linux is not found by a Windows run of the same code, and vice versa. That matters for anyone sharing a cache across machines, and it is invisible: nothing reports a miss, the dataset is simply rebuilt under another UUID.

The same substitution also applies to non-HTTP schemes, since is_url only requires a scheme and a netloc: s3://bucket/key/file.csv.gz becomes s3://bucket/\key\file.csv.gz.

Fix

A URL path is always POSIX, so it should be normalised with posixpath, which behaves identically on every platform:

import posixpath

def clean_path(path: str) -> str:
    """Clean a path string."""
    if is_url(path):
        parsed = urlparse(path)
        # A URL path is POSIX whatever the host OS: os.path.normpath would
        # rewrite the separators as backslashes on Windows.
        cleaned_path = posixpath.normpath(parsed.path)
        return urlunparse(parsed._replace(path=cleaned_path))
    else:
        return str(Path(path).expanduser().resolve())

Verified on the same machine:

Note that the fix changes the cache key of URL-rooted datasets on Windows (to the POSIX one, which is the correct and portable value), so existing Windows caches will be rebuilt once. That seems preferable to keeping them keyed on a malformed URL, but say so if you would rather handle it differently.

Two questions before I send a PR:

1. Would you like the fix to come with a unit test over clean_path for the four cases above? It needs no network, unlike test_processor_schemas.py.
2. line 113 dates from abc57230 (1 May 2025), so the behaviour has been there for a while, and CI apparently runs on Linux only. Would a Windows job on a small subset of the test suite be welcome, or is that out of scope here?

Happy to open the PR either way.