[DaScient]

Summary

Replaces the placeholder RLStrategy, which returned hold on every bar, with a working reinforcement-learning agent. Also fixes how strategy settings reach strategies, and makes a few backtester changes the agent needs.

RL strategy (strategies/rl/, numpy only)

• features.py: turns each bar into one of 36 discrete market states, built from:

  • SuperTrend direction
  • RSI zone
  • momentum relative to ATR
  • whether volatility is above its rolling median

  Each state uses only past data; a test checks that a prefix of a series produces identical states.

• environment.py: actions are target positions (flat, long or short). The reward is the log return of the position held into the next bar, minus transaction costs, which matches how the backtester fills at the bar close.

• agent.py: tabular Q-learning. Confidence is the margin between the best and second-best action, and a state the agent never saw in training gets confidence 0. The agent saves to and loads from JSON, writing the file atomically.

• trainer.py: RLConfig collects every RL setting with validation:

  • learning rate, gamma, episodes
  • exploration schedule and seed
  • transaction cost and shorting
  • train/test split
  • persistence
  • feature definitions

  train_agent() is reproducible for a given seed.

• RLStrategy:

  • Maps the agent's target position to buy, sell or exit (or hold when already flat), using the position the backtester passes in.
  • backtest() trains on the first 70% of the data and trades only the rest.
  • An agent trained earlier in memory is retrained for every backtest. One loaded from disk is used as-is.
  • A saved model is loaded only if its feature settings match the current config.

Supporting changes

• Backtester: asks the strategy once per bar, passes the current position in the window, and supports an exit action. SuperTrend results are unchanged (same trades and final capital).
• Strategy parameters: strategies created by the engine never received their strategies.parameters.<name> section, so parameters was always {}. That's fixed. set_parameters now also refreshes derived attributes (for example SuperTrend's atr_period), and an invalid update leaves the previous settings in place.
• calculate_supertrend moved to utils/indicators.py so the RL features can reuse it.
• The rl_agent section of default_config.yaml now documents every RL setting. The README has an RL section and an updated status table.

Known limitations

• Market data is still mock. On the random-walk mock data the agent gains on the bars it trained on and loses on the unseen ones, which is the expected result when there is nothing real to learn.
• ATR stops apply but aren't modelled in training. The backtester's stop-loss and take-profit still apply to RL positions, but the training environment doesn't include them.

Testing

• pytest tests/: 85 passed, about 5 seconds, no network calls.
• 30 new tests in tests/test_rl.py. They cover:
  • features that use only past data
  • reward and cost calculation
  • learning to stay long in a steady uptrend
  • reproducibility for a given seed
  • shorting turned off
  • save/load and rejecting a mismatched saved model
  • config validation
  • trading only after the train/test split
  • the backtester's one-decision-per-bar and exit behaviour

🤖 Generated with Claude Code

https://claude.ai/code/session_015rhcFXLJHUCwo4mn74RAHy

---

Generated by Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

Terminal cost accounting, model compatibility, configuration validation, and parameter refresh behavior have unresolved correctness issues.

Review effort: Balanced
Findings: 4

Open (4)

• Parameter updates leave cached strategy fields stale · New
• Transaction cost is missing from model compatibility checks · New
• Final position liquidation cost is omitted · New
• Validate ATR and RSI periods before feature calculation · New

What changed in this PR

Implements a tabular Q-learning strategy and integrates position-aware RL trading into the backtester.

Changes:

• Adds causal features, training environment, Q-learning agent, and persistence.
• Adds position-aware signals, exits, and strategy preparation hooks.
• Expands configuration, documentation, and tests.

File	Description
utils/​indicators.py	Extracts reusable SuperTrend calculation.
utils/​backtest.py	Adds position-aware decisions and exit actions.
tests/​test_rl.py	Tests the RL stack and backtester contract.
strategies/​rl/​trainer.py	Adds RL configuration and training.
strategies/​rl/​features.py	Builds causal discrete market states.
strategies/​rl/​environment.py	Models positions, returns, and costs.
strategies/​rl/​agent.py	Implements Q-learning and JSON persistence.
strategies/​rl/​__init__.py	Exports RL components.
strategies/​__init__.py	Integrates RL and strategy parameter handling.
README.md	Documents RL behavior and usage.
configs/​default_config.yaml	Defines RL defaults and feature settings.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.