Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/compatibility.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ["3.12", "3.13", "3.14"]
python-version: ["3.12", "3.13", "3.14", "3.15"]
env:
UV_PYTHON: ${{ matrix.python-version }}
UV_NO_MANAGED_PYTHON: "1"
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ecosystem.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
qualification:
permissions:
contents: read
uses: ml4t/ecosystem/.github/workflows/qualify-library.yml@322df8513e6f7dceae8eab9d9697b3428ee62af6 # 2026-10-25 scoped policy
uses: ml4t/ecosystem/.github/workflows/qualify-library.yml@2bfa8eb353b0f16bdaae0506c38b8c44d0819298 # Stable Python 3.15 qualification
with:
import-package: ml4t.data
prerelease-exception: python-315-data-databento-030
fetch-depth: 0
53 changes: 0 additions & 53 deletions .github/workflows/python315-canary.yml

This file was deleted.

5 changes: 3 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,11 @@ jobs:
needs: preflight
permissions:
contents: read
uses: ml4t/ecosystem/.github/workflows/qualify-library.yml@322df8513e6f7dceae8eab9d9697b3428ee62af6 # 2026-10-25 scoped policy
uses: ml4t/ecosystem/.github/workflows/qualify-library.yml@2bfa8eb353b0f16bdaae0506c38b8c44d0819298 # Stable Python 3.15 qualification
with:
import-package: ml4t.data
prerelease-exception: python-315-data-databento-030
candidate-version: ${{ inputs.version }}
fetch-depth: 0

compatibility:
name: Release Compatibility
Expand Down
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# ml4t-data

[![PyPI](https://img.shields.io/pypi/v/ml4t-data)](https://pypi.org/project/ml4t-data/)
[![Python 3.12-3.14](https://img.shields.io/badge/python-3.12--3.14-blue.svg)](https://www.python.org/downloads/)
[![Python 3.12-3.15](https://img.shields.io/badge/python-3.12--3.15-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

Market data acquisition, storage, and update workflows for machine learning for trading.
Expand All @@ -12,8 +12,7 @@ foreign exchange, crypto assets, macroeconomic series, factors, and prediction m

## Installation

`ml4t-data` supports CPython 3.12, 3.13, and 3.14 on Linux, macOS, and Windows. Python 3.15 is
temporarily excluded while upstream dependencies complete their compatibility work.
`ml4t-data` supports CPython 3.12 through 3.15 on Linux, macOS, and Windows.

```bash
pip install ml4t-data
Expand Down
3 changes: 1 addition & 2 deletions docs/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,7 @@

## Requirements

`ml4t-data` supports CPython 3.12, 3.13, and 3.14 on Linux, macOS, and Windows.
Python 3.15 is temporarily excluded while upstream dependencies complete compatibility work.
`ml4t-data` supports CPython 3.12 through 3.15 on Linux, macOS, and Windows.

## Install from PyPI

Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ locally generated data.

## Install the package

Use CPython 3.12, 3.13, or 3.14.
Use CPython 3.12 through 3.15.

```bash
pip install ml4t-data
Expand Down
2 changes: 1 addition & 1 deletion docs/providers/coinmetrics.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Coin Metrics spot-market candles

`CoinMetricsProvider` acquires OHLCV candles for one explicit Coin Metrics spot market through
[`/v4/timeseries/market-candles`](https://docs.coinmetrics.io/api/v4/#operation/getTimeseriesMarketCandles).
[`/v4/timeseries/market-candles`](https://docs.coinmetrics.io/market-data/market-data-overview/market-candles#historical-candles-http).
It implements the standard provider API and the exact `TimestampRangeProvider` capability used by
[stored-gap recovery](../user-guide/incremental-updates.md#download-real-observations-for-a-selected-report).

Expand Down
8 changes: 4 additions & 4 deletions docs/user-guide/sessions.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,10 +191,10 @@ or real contract identity. It is not a substitute for historical NQU25 observati
The sample contains nine irregular, close-labeled bars across the March 7 and March 10, 2025
`CME_Equity` sessions, including Sunday evening trading after the US daylight-saving change.
All `datetime` values explicitly use UTC (`Z`). Friday's 16:00 Chicago close is 22:00 UTC;
Monday's is 21:00 UTC. A close-labeled observation at the session close or break start belongs to
the preceding open interval. The sample includes those boundaries and observations after the
scheduled break. The selected calendar supplies the actual intervals; a fixed UTC offset or
weekday rule does not determine the session date.
Monday's is 21:00 UTC. A close-labeled observation at the session close belongs to that session.
[CME removed the 15:15 to 15:30 Chicago time pause](https://www.cmegroup.com/notices/electronic-trading/2021/06/20210614.html)
for these contracts starting with the June 28, 2021 trade date. The calendar still applies the
historical pause before the cutover and the daily 16:00 to 17:00 maintenance period.

These rows illustrate an already aggregated input. They are not generated from ticks, and no
volume or trade-count threshold, partial-bar policy, or cleaning pipeline is claimed. The example
Expand Down
20 changes: 15 additions & 5 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -69,17 +69,19 @@ classifiers = [
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Programming Language :: Python :: 3.15",
"Topic :: Office/Business :: Financial :: Investment",
"Topic :: Scientific/Engineering :: Information Analysis",
"Typing :: Typed",
]
requires-python = ">=3.12,<3.15"
requires-python = ">=3.12,<3.16"

# Core dependencies
dependencies = [
"ml4t-specs>=0.2.0",
# Data processing
"polars>=1.36.1",
"polars>=2.0.0; python_version >= '3.15'",
"pandas>=2.0.0",
"numpy>=1.24.0",
# HTTP and networking
Expand All @@ -93,14 +95,16 @@ dependencies = [
"python-dotenv>=1.0.0",
"pydantic-settings>=2.0.0",
"pydantic>=2.12,<3",
"pydantic>=2.14.1; python_version >= '3.15'",
# Utilities
"structlog>=23.0.0",
"platformdirs>=4.0.0",
"filelock>=4.2.1",
"rich>=13.0.0",
"aiofiles>=23.0.0", # Async file I/O for async storage backends
# Exchange calendars for session-aware operations
"pandas-market-calendars>=5.4.0",
"pandas-market-calendars>=5.5.0",
"pandas-market-calendars>=5.6.0; python_version >= '3.15'",
"openpyxl>=3.1.5",
]

Expand All @@ -111,12 +115,15 @@ yahoo = [
]
databento = [
"databento>=0.38.0",
"databento>=0.90.0; python_version >= '3.15'",
"pyarrow>=14.0.0",
"pyarrow>=26.0.0; python_version >= '3.15'",
]
oanda = [
"oandapyV20>=0.7.0",
# oandapyV20 imports requests but does not declare it.
# oandapyV20 imports requests and six without declaring them.
"requests>=2.32.0",
"six>=1.17.0",
]
cot = [
"cot-reports>=0.1.0", # CFTC Commitment of Traders data
Expand All @@ -141,6 +148,7 @@ dev = [
"pre-commit>=3.3.0",
"xlsxwriter>=3.1.0", # Excel export testing
"oandapyV20>=0.7.0", # OANDA provider tests/imports
"six>=1.17.0",
]

# Documentation dependencies (MkDocs Material with Picasso theme)
Expand Down Expand Up @@ -178,15 +186,17 @@ test = [
"xlsxwriter>=3.1.0",
"oandapyV20>=0.7.0",
"requests>=2.32.0",
"six>=1.17.0",
"databento>=0.38.0",
"databento>=0.90.0; python_version >= '3.15'",
"yfinance>=1.5.2",
]
dev = [
{ include-group = "test" },
"ruff>=0.8.0",
"pre-commit>=3.3.0",
"twine>=6.0.0",
"pip-audit>=2.10.1",
"databento>=0.38.0",
"yfinance>=1.5.2",
]

[tool.uv]
Expand Down
6 changes: 4 additions & 2 deletions scripts/release_candidate.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,15 @@
"Issues": "https://github.com/ml4t/data/issues",
"Changelog": "https://github.com/ml4t/data/releases",
}
EXPECTED_PYTHON = ">=3.12,<3.16"
EXPECTED_KEYWORDS = {"finance", "quantitative-finance", "algorithmic-trading", "market-data"}
EXPECTED_CLASSIFIERS = {
"Development Status :: 5 - Production/Stable",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Programming Language :: Python :: 3.15",
"Typing :: Typed",
}

Expand Down Expand Up @@ -97,8 +99,8 @@ def metadata_failures(metadata: Message, *, expected_version: str) -> list[str]:
for field, expected in expected_scalars.items():
if metadata.get(field) != expected:
failures.append(f"{field} is {metadata.get(field)!r}, expected {expected!r}")
if SpecifierSet(str(metadata.get("Requires-Python", ""))) != SpecifierSet(">=3.12,<3.15"):
failures.append("Requires-Python differs from >=3.12,<3.15")
if SpecifierSet(str(metadata.get("Requires-Python", ""))) != SpecifierSet(EXPECTED_PYTHON):
failures.append(f"Requires-Python differs from {EXPECTED_PYTHON}")
project_urls = {}
for value in metadata.get_all("Project-URL", []):
label, separator, url = value.partition(", ")
Expand Down
12 changes: 10 additions & 2 deletions scripts/verify_distribution.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,8 @@ def validate_distributions(
for field, expected in expected_scalars.items():
if item[field] != expected:
raise ValueError(f"{field} is {item[field]!r}, expected {expected!r}")
if SpecifierSet(str(item["Requires-Python"])) != SpecifierSet(">=3.12,<3.15"):
raise ValueError("Requires-Python differs from >=3.12,<3.15")
if SpecifierSet(str(item["Requires-Python"])) != SpecifierSet(">=3.12,<3.16"):
raise ValueError("Requires-Python differs from >=3.12,<3.16")
project_urls = {}
for value in item.get_all("Project-URL", []):
label, separator, url = value.partition(", ")
Expand Down Expand Up @@ -176,6 +176,14 @@ def verify_install(archive: Path, version: str) -> None:
subprocess.run(
[str(python), str(Path(__file__).with_name("run_readme_quickstart.py"))], check=True
)
subprocess.run(
["uv", "pip", "install", "--python", str(python), f"{archive}[oanda]"],
check=True,
)
subprocess.run(
[str(python), "-I", "-c", "from ml4t.data.providers.oanda import OandaProvider"],
check=True,
)


def main() -> None:
Expand Down
4 changes: 3 additions & 1 deletion scripts/verify_published_release.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
EXPECTED_DESCRIPTION,
EXPECTED_KEYWORDS,
EXPECTED_MAINTAINER,
EXPECTED_PYTHON,
EXPECTED_URLS,
)
except ModuleNotFoundError: # Direct execution adds scripts/, not the repository root, to sys.path.
Expand All @@ -30,6 +31,7 @@
EXPECTED_DESCRIPTION,
EXPECTED_KEYWORDS,
EXPECTED_MAINTAINER,
EXPECTED_PYTHON,
EXPECTED_URLS,
)

Expand Down Expand Up @@ -81,7 +83,7 @@ def published_release_failures(
and info.get("author_email") == EXPECTED_AUTHOR
and info.get("maintainer_email") == EXPECTED_MAINTAINER
and info.get("license_expression") == "MIT"
and SpecifierSet(str(info.get("requires_python", ""))) == SpecifierSet(">=3.12,<3.15")
and SpecifierSet(str(info.get("requires_python", ""))) == SpecifierSet(EXPECTED_PYTHON)
and info.get("project_urls") == EXPECTED_URLS
and keywords >= EXPECTED_KEYWORDS
and set(info.get("classifiers", [])) >= EXPECTED_CLASSIFIERS
Expand Down
3 changes: 2 additions & 1 deletion tests/test_distribution_verification.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,14 @@ def _write_distributions(
"Summary: Market data acquisition, storage, and update workflows for machine learning for trading.",
"Author-email: Stefan Jansen <stefan@applied-ai.com>",
"Maintainer-email: Stefan Jansen <pm@ml4trading.io>",
"Requires-Python: >=3.12,<3.15",
"Requires-Python: >=3.12,<3.16",
"Keywords: finance,quantitative-finance,algorithmic-trading,market-data",
"Classifier: Development Status :: 5 - Production/Stable",
"Classifier: License :: OSI Approved :: MIT License",
"Classifier: Programming Language :: Python :: 3.12",
"Classifier: Programming Language :: Python :: 3.13",
"Classifier: Programming Language :: Python :: 3.14",
"Classifier: Programming Language :: Python :: 3.15",
"Classifier: Typing :: Typed",
"Project-URL: Homepage, https://www.ml4trading.io/",
"Project-URL: Documentation, https://www.ml4trading.io/docs/data/",
Expand Down
26 changes: 21 additions & 5 deletions tests/test_nasdaq_bars_example.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,17 @@ def test_external_naive_bars_require_timezone_instead_of_silently_assuming_utc(t
assert not (tmp_path / "nasdaq_bars_with_sessions.parquet").exists()


@pytest.mark.parametrize("instant", ["2025-03-08T12:00:00Z", "2025-03-10T20:20:00Z"])
def test_example_rejects_weekend_and_break_observations_without_silent_filtering(tmp_path, instant):
@pytest.mark.parametrize(
("instant", "session"),
[
("2025-03-08T12:00:00Z", None),
("2025-03-10T21:20:00Z", None),
("2021-06-25T20:20:00Z", None),
("2021-06-28T20:20:00Z", date(2021, 6, 28)),
("2025-03-10T20:20:00Z", date(2025, 3, 10)),
],
)
def test_example_respects_the_cme_equity_pause_cutover(tmp_path, instant, session):
bars = (
pl.read_csv(SAMPLE, try_parse_dates=True)
.head(1)
Expand All @@ -73,9 +82,16 @@ def test_example_rejects_weekend_and_break_observations_without_silent_filtering
source = tmp_path / "bars.parquet"
bars.write_parquet(source)
result = run_example(tmp_path, "--input", source)
assert result.returncode != 0
assert "Observations outside a trading session" in result.stderr
assert not (tmp_path / "nasdaq_bars_with_sessions.parquet").exists()
output = tmp_path / "nasdaq_bars_with_sessions.parquet"
if session is None:
assert result.returncode != 0
assert "Observations outside a trading session" in result.stderr
assert not output.exists()
else:
assert result.returncode == 0, result.stdout + result.stderr
assigned = pl.read_parquet(output)
assert assigned["session_date"].to_list() == [session]
assert_frame_equal(assigned.drop("session_date"), bars)


def test_open_labels_include_sunday_open_and_reject_monday_close(tmp_path):
Expand Down
10 changes: 7 additions & 3 deletions tests/test_package_metadata.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ def test_supported_python_and_platform_metadata() -> None:
project = load_project()
classifiers = set(project["classifiers"])

assert project["requires-python"] == ">=3.12,<3.15"
assert project["requires-python"] == ">=3.12,<3.16"
assert "Development Status :: 5 - Production/Stable" in classifiers
assert "Operating System :: OS Independent" not in classifiers
assert {
Expand All @@ -37,15 +37,19 @@ def test_supported_python_and_platform_metadata() -> None:
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Programming Language :: Python :: 3.15",
} <= classifiers
assert "Programming Language :: Python :: 3.11" not in classifiers
assert "Programming Language :: Python :: 3.15" not in classifiers
assert "Programming Language :: Python :: 3.16" not in classifiers
pydantic_requirements = {
item
for item in project["dependencies"]
if item.startswith("pydantic>=") or item.startswith("pydantic==")
}
assert pydantic_requirements == {"pydantic>=2.12,<3"}
assert pydantic_requirements == {
"pydantic>=2.12,<3",
"pydantic>=2.14.1; python_version >= '3.15'",
}


def test_resolver_uses_prereleases_only_when_explicitly_required() -> None:
Expand Down
Loading
Loading