📁 ICT ML Trading System

Complete Project Structure & File Organization

Production-grade architecture with comprehensive logging, error handling, and best practices

Complete Directory Structure

🎯 Architecture Philosophy: This project follows enterprise-grade Python best practices with clear separation of concerns, comprehensive error handling, extensive logging, and modular design for maintainability and scalability.
ict-ml-trading/ │ ├── .env.example # Environment variables template ├── .env # Actual environment variables (git-ignored) ├── .gitignore # Git ignore patterns ├── .pre-commit-config.yaml # Pre-commit hooks configuration ├── README.md # Project documentation ├── requirements.txt # Production dependencies ├── requirements-dev.txt # Development dependencies ├── setup.py # Package installation configuration ├── pyproject.toml # Modern Python project configuration ├── Makefile # Common commands automation ├── CHANGELOG.md # Version history and changes ├── LICENSE # Software license │ ├── config/ # Configuration files │ ├── config.yaml # Main application configuration │ ├── logging.yaml # Logging configuration │ ├── detectors.yaml # Pattern detector parameters │ ├── models.yaml # ML model hyperparameters │ ├── trading.yaml # Trading strategy parameters │ ├── alerts.yaml # Alert system configuration │ └── database.yaml # Database connection settings │ ├── src/ # Main source code │ ├── __init__.py │ │ │ ├── core/ # Core business logic │ │ ├── __init__.py │ │ ├── exceptions.py # Custom exception classes │ │ ├── constants.py # Application-wide constants │ │ ├── enums.py # Enumerations (PatternType, Grade, etc.) │ │ └── base_classes.py # Abstract base classes │ │ │ ├── config/ # Configuration management │ │ ├── __init__.py │ │ ├── config_loader.py # Load and validate configurations │ │ ├── settings.py # Settings dataclasses │ │ └── validators.py # Configuration validation logic │ │ │ ├── logging/ # Logging system │ │ ├── __init__.py │ │ ├── logger.py # Logger setup and configuration │ │ ├── formatters.py # Custom log formatters │ │ ├── handlers.py # Custom log handlers │ │ ├── filters.py # Log filters │ │ └── decorators.py # Logging decorators (@log_execution) │ │ │ ├── data/ # Data management │ │ ├── __init__.py │ │ ├── database.py # Database connection and ORM │ │ ├── models.py # SQLAlchemy data models │ │ ├── repositories.py # Data access layer (repository pattern) │ │ ├── loaders.py # Data loading utilities │ │ ├── preprocessors.py # Data preprocessing and cleaning │ │ ├── validators.py # Data validation │ │ └── schemas.py # Pydantic schemas for validation │ │ │ ├── detectors/ # Pattern detection │ │ ├── __init__.py │ │ ├── base_detector.py # Abstract base detector class │ │ ├── fvg_detector.py # Fair Value Gap detection │ │ ├── order_block_detector.py # Order Block detection │ │ ├── liquidity_detector.py # Liquidity sweep detection │ │ ├── premium_discount.py # Premium/Discount calculator │ │ ├── structure_detector.py # Market structure (BOS, CHoCH) │ │ ├── scanner.py # Orchestrates all detectors │ │ └── utils.py # Detector utility functions │ │ │ ├── features/ # Feature engineering │ │ ├── __init__.py │ │ ├── base_features.py # Base feature engineering class │ │ ├── pattern_features.py # Pattern-specific features │ │ ├── technical_features.py # Technical indicators │ │ ├── temporal_features.py # Time-based features │ │ ├── context_features.py # Market context features │ │ ├── orderflow_features.py # Order flow features │ │ └── feature_store.py # Feature storage and retrieval │ │ │ ├── models/ # Machine learning models │ │ ├── __init__.py │ │ ├── base_model.py # Abstract base model class │ │ ├── pattern_grader.py # Individual pattern grading models │ │ ├── setup_classifier.py # Complete setup classification │ │ ├── training.py # Model training pipeline │ │ ├── evaluation.py # Model evaluation metrics │ │ ├── tuning.py # Hyperparameter tuning │ │ ├── inference.py # Real-time prediction │ │ └── registry.py # Model versioning and registry │ │ │ ├── labeling/ # Labeling system │ │ ├── __init__.py │ │ ├── interface.py # Labeling interface orchestrator │ │ ├── notion_integration.py # Notion API integration │ │ ├── web_ui.py # Flask/FastAPI web interface │ │ ├── quality_checker.py # Label quality validation │ │ ├── anchors.py # Anchor example management │ │ └── export.py # Export labels to training format │ │ │ ├── alerts/ # Alert system │ │ ├── __init__.py │ │ ├── base_alert.py # Abstract alert handler │ │ ├── telegram_bot.py # Telegram bot integration │ │ ├── slack_integration.py # Slack integration │ │ ├── email_alerts.py # Email notifications │ │ ├── alert_manager.py # Alert orchestration and routing │ │ └── templates.py # Alert message templates │ │ │ ├── trading/ # Trading execution │ │ ├── __init__.py │ │ ├── broker_interface.py # Broker API abstraction │ │ ├── order_manager.py # Order execution and tracking │ │ ├── risk_manager.py # Position sizing and risk controls │ │ ├── portfolio.py # Portfolio state management │ │ ├── execution_engine.py # Trade execution logic │ │ └── validators.py # Pre-trade validation │ │ │ ├── backtesting/ # Backtesting framework │ │ ├── __init__.py │ │ ├── engine.py # Backtest execution engine │ │ ├── metrics.py # Performance metrics calculation │ │ ├── reports.py # Backtest report generation │ │ ├── visualizations.py # Performance visualizations │ │ └── validators.py # Backtest result validation │ │ │ ├── monitoring/ # System monitoring │ │ ├── __init__.py │ │ ├── metrics_collector.py # Collect system metrics │ │ ├── health_checker.py # System health checks │ │ ├── performance_tracker.py # Track model performance │ │ ├── alerting.py # Monitoring alerts │ │ └── dashboard.py # Monitoring dashboard │ │ │ ├── visualization/ # Chart and data visualization │ │ ├── __init__.py │ │ ├── chart_generator.py # Generate trading charts │ │ ├── pattern_plotter.py # Plot detected patterns │ │ ├── performance_plots.py # Performance visualizations │ │ └── screenshot.py # Screenshot capture for labeling │ │ │ └── utils/ # Utility functions │ ├── __init__.py │ ├── datetime_utils.py # Date/time helpers │ ├── file_utils.py # File operations │ ├── string_utils.py # String manipulation │ ├── math_utils.py # Mathematical utilities │ ├── decorators.py # Utility decorators │ ├── retry.py # Retry logic for API calls │ └── validators.py # General validation functions │ ├── data/ # Data storage │ ├── raw/ # Raw downloaded data │ │ ├── ohlcv/ # OHLCV market data │ │ │ ├── 1min/ │ │ │ ├── 5min/ │ │ │ └── 15min/ │ │ └── orderflow/ # Order flow data (if available) │ │ │ ├── processed/ # Cleaned and processed data │ │ ├── features/ # Computed features │ │ ├── patterns/ # Detected patterns │ │ └── snapshots/ # OHLC snapshots for patterns │ │ │ ├── labels/ # Labeled data │ │ ├── individual_patterns/ # Pattern-level labels │ │ ├── complete_setups/ # Setup-level labels │ │ ├── anchors/ # Anchor examples │ │ └── label_metadata.json # Label versioning and metadata │ │ │ ├── screenshots/ # Chart screenshots for labeling │ │ ├── patterns/ # Pattern screenshots │ │ └── setups/ # Complete setup screenshots │ │ │ └── external/ # External data sources │ ├── economic_calendar/ # News events │ └── reference/ # Reference data (holidays, etc.) │ ├── models/ # Trained models storage │ ├── pattern_graders/ # Individual pattern models │ │ ├── fvg/ │ │ │ ├── v1.0.0/ # Versioned models │ │ │ ├── v1.1.0/ │ │ │ └── latest/ # Symlink to latest version │ │ ├── order_block/ │ │ └── liquidity/ │ │ │ ├── strategy_models/ # Complete setup models │ │ ├── continuation/ # 3:00-3:15 continuation model │ │ └── reversal/ # 3:30-3:50 reversal model │ │ │ ├── metadata/ # Model metadata and configs │ │ ├── model_registry.json # Track all model versions │ │ └── performance_history.json # Historical performance │ │ │ └── artifacts/ # Training artifacts │ ├── feature_importances/ │ ├── training_curves/ │ └── validation_reports/ │ ├── logs/ # Application logs │ ├── application/ # General application logs │ │ ├── app.log # Current log file │ │ └── archive/ # Rotated logs │ │ │ ├── detectors/ # Pattern detection logs │ │ ├── fvg_detector.log │ │ ├── order_block_detector.log │ │ └── scanner.log │ │ │ ├── models/ # ML model logs │ │ ├── training.log # Training sessions │ │ ├── inference.log # Real-time predictions │ │ └── evaluation.log # Model evaluation │ │ │ ├── trading/ # Trading execution logs │ │ ├── orders.log # Order execution │ │ ├── risk.log # Risk management decisions │ │ └── portfolio.log # Portfolio changes │ │ │ ├── alerts/ # Alert system logs │ │ ├── telegram.log │ │ └── notifications.log │ │ │ ├── errors/ # Error logs │ │ ├── exceptions.log # All exceptions │ │ └── critical.log # Critical errors only │ │ │ ├── performance/ # Performance monitoring │ │ ├── metrics.log # System metrics │ │ └── profiling.log # Performance profiling │ │ │ └── audit/ # Audit trail │ ├── data_access.log # Data access audit │ ├── config_changes.log # Configuration changes │ └── model_deployments.log # Model deployment history │ ├── tests/ # Test suite │ ├── __init__.py │ ├── conftest.py # Pytest configuration and fixtures │ │ │ ├── unit/ # Unit tests │ │ ├── test_detectors/ │ │ ├── test_features/ │ │ ├── test_models/ │ │ └── test_utils/ │ │ │ ├── integration/ # Integration tests │ │ ├── test_pipeline.py # End-to-end pipeline │ │ ├── test_database.py # Database integration │ │ └── test_alerts.py # Alert system integration │ │ │ ├── validation/ # Validation tests │ │ ├── test_model_accuracy.py │ │ ├── test_backtest_results.py │ │ └── test_data_quality.py │ │ │ ├── fixtures/ # Test data and fixtures │ │ ├── sample_data/ │ │ ├── mock_responses/ │ │ └── expected_outputs/ │ │ │ └── performance/ # Performance tests │ ├── test_speed.py # Speed benchmarks │ └── test_memory.py # Memory usage tests │ ├── scripts/ # Utility scripts │ ├── download_data.py # Download historical data │ ├── process_data.py # Batch data processing │ ├── run_detection.py # Run pattern detection │ ├── train_models.py # Train ML models │ ├── evaluate_models.py # Evaluate model performance │ ├── backtest.py # Run backtests │ ├── live_trading.py # Live trading execution │ ├── setup_database.py # Initialize database │ ├── migrate_data.py # Data migration │ ├── cleanup.py # Cleanup old logs/data │ └── export_labels.py # Export labels from Notion │ ├── notebooks/ # Jupyter notebooks │ ├── 01_data_exploration.ipynb │ ├── 02_pattern_analysis.ipynb │ ├── 03_feature_engineering.ipynb │ ├── 04_model_training.ipynb │ ├── 05_model_evaluation.ipynb │ ├── 06_backtest_analysis.ipynb │ └── 07_label_quality_check.ipynb │ ├── docs/ # Documentation │ ├── architecture.md # System architecture │ ├── api_reference.md # API documentation │ ├── user_guide.md # User guide │ ├── deployment.md # Deployment guide │ ├── troubleshooting.md # Troubleshooting guide │ ├── contributing.md # Contribution guidelines │ ├── diagrams/ # Architecture diagrams │ └── examples/ # Code examples │ ├── deployment/ # Deployment configuration │ ├── Dockerfile # Docker container │ ├── docker-compose.yml # Multi-container setup │ ├── .dockerignore # Docker ignore patterns │ ├── kubernetes/ # Kubernetes manifests │ │ ├── deployment.yaml │ │ ├── service.yaml │ │ └── configmap.yaml │ ├── systemd/ # Systemd service files │ │ └── ict-trading.service │ └── supervisor/ # Supervisor configuration │ └── ict-trading.conf │ ├── monitoring/ # Monitoring configuration │ ├── prometheus.yml # Prometheus config │ ├── grafana_dashboards/ # Grafana dashboards │ └── alerts.yml # Alert rules │ └── backups/ # Backup directory ├── database/ # Database backups ├── models/ # Model backups └── config/ # Configuration backups

Key File Descriptions

📝 Logging System (src/logging/)

Purpose: Comprehensive logging infrastructure for debugging, monitoring, and auditing

  • logger.py: Central logger setup - creates module-specific loggers with configured handlers and formatters
  • formatters.py: JSON formatter for machine parsing, colored console formatter, detailed/compact formatters
  • handlers.py: Rotating file handlers (size/time-based), database handler for critical events, alert handler for CRITICAL logs
  • filters.py: Sensitive data redaction (API keys, passwords), rate limiting for repeated messages
  • decorators.py: @log_execution (entry/exit/timing), @log_exceptions (catch and log errors), @log_performance

⚠️ Error Handling (src/core/exceptions.py)

Custom Exception Hierarchy:

  • ICTTradingException - Base exception with error codes and context data
  • DataError - Data loading/validation failures
  • DetectorError - Pattern detection failures
  • ModelError - ML training/inference errors
  • ConfigurationError - Invalid configuration
  • TradingError - Order execution failures

Every exception is logged with full context and stack trace

📊 Data Management (data/)

Directory Purpose Retention
data/raw/ Downloaded OHLCV data (Parquet format) 24 months
data/processed/ Cleaned data, detected patterns, features 12 months
data/labels/ User labels (versioned), anchor examples Permanent
data/screenshots/ Chart images for labeling 6 months

🤖 Model Versioning (models/)

Every model version includes:

  • model.pkl - Trained model file
  • hyperparameters.yaml - All configuration used
  • features.json - Feature list and scaling parameters
  • metrics.json - Performance metrics (accuracy, precision, recall)
  • training_log.txt - Full training session log
  • feature_importance.csv - Which features matter most

Symlink "latest/" always points to production model - easy rollback

📈 Logging Categories (logs/)

Log Type Contents Rotation
application/ Startup, shutdown, config loading Daily or 10MB
detectors/ Pattern detection runs, counts, timing Daily or 10MB
models/ Training, inference, evaluation Per session
trading/ Orders, fills, risk decisions, P&L Never rotate (compliance)
errors/ All exceptions with stack traces Daily or 10MB
audit/ Data access, config changes, deployments Monthly

All logs: 30-day retention, compressed archives for older logs

Best Practices Implementation

🔒 Security

  • All secrets in .env (never committed)
  • Sensitive data filtered from logs automatically
  • Database credentials from environment variables only
  • API keys validated at startup, logged failures
  • All external calls use HTTPS

⚡ Performance

  • Async logging (non-blocking for high-volume logs)
  • Parquet format for data (10x faster than CSV)
  • Database indexes on frequently queried columns
  • Connection pooling for database
  • Retry logic with exponential backoff for API calls
  • Performance profiling logs (identify bottlenecks)

🧪 Testing

  • Unit tests for all detectors, features, models
  • Integration tests for end-to-end pipeline
  • Validation tests ensure model accuracy thresholds met
  • Performance tests benchmark speed and memory
  • Fixtures provide realistic test data
  • Target: 80%+ code coverage

🚀 Deployment

  • Docker for reproducible environments
  • docker-compose for multi-container setup
  • Kubernetes manifests for cloud deployment
  • systemd service for Linux servers
  • Health checks and auto-restart
  • Blue-green deployment support

🤖 AI Agent Implementation Guide

📋 Purpose: This guide provides step-by-step instructions for AI agents (like Claude, GPT, etc.) to build the ICT ML Trading System. Each version is a stable checkpoint that can be committed to Git, tested, and rolled back if needed.

🎯 For AI Agents: Code Quality Standards

When implementing ANY file in this project, ALWAYS follow these rules:

  • Type Hints: Add type hints to all function signatures (PEP 484)
  • Docstrings: Write Google-style docstrings for all classes and functions
  • Error Handling: Wrap risky operations in try-except, use custom exceptions from src/core/exceptions.py
  • Logging: Log entry/exit of major functions, all errors, and important state changes using src/logging/logger.py
  • Validation: Validate all inputs (function args, config values, data)
  • Constants: No magic numbers - define constants in src/core/constants.py
  • Configuration: Read parameters from config files, never hardcode
  • Testing: Write at least one test for each public function
  • Comments: Explain WHY, not WHAT (code should be self-explanatory)
  • Dependencies: Minimize external dependencies, document all additions to requirements.txt

🌳 Git Branching Strategy

Recommended Branch Model (GitHub Flow - Simple & Effective):

  • main: Production-ready code, always stable
  • develop: Integration branch for completed features
  • feature/VERSION-feature-name: Feature development branches
  • hotfix/issue-description: Emergency fixes

Workflow:

# Start new version
git checkout develop
git checkout -b feature/v0.1.0-project-setup

# Work on version, commit frequently
git add .
git commit -m "feat: add project structure and config files"

# Complete version
git checkout develop
git merge feature/v0.1.0-project-setup
git tag v0.1.0
git push origin develop --tags

# If stable, merge to main
git checkout main
git merge develop
git push origin main

📦 Version Roadmap

Version Milestone Duration Deliverables
v0.1.0 Project Foundation 1-2 days Project structure, configs, logging
v0.2.0 Data Pipeline 2-3 days Data loading, preprocessing, database
v0.3.0 Pattern Detectors 3-5 days FVG, OB, Liquidity detectors
v0.4.0 Feature Engineering 2-3 days Feature extraction pipeline
v0.5.0 Labeling System 2-3 days Labeling interface, quality checks
v0.6.0 ML Models 3-4 days Training pipeline, evaluation
v0.7.0 Backtesting 2-3 days Backtest engine, metrics
v0.8.0 Alert System 1-2 days Telegram bot, notifications
v0.9.0 Paper Trading 2-3 days Live detection, simulated execution
v1.0.0 Production Ready 1-2 days Deployment, monitoring, docs

🔨 Version 0.1.0 - Project Foundation

Branch: feature/v0.1.0-project-setup

✅ Todo List:

  • ☐ Create directory structure (all folders as shown above)
  • ☐ Initialize Git repository: git init
  • ☐ Create .gitignore file with Python, data, logs, models patterns
  • ☐ Create requirements.txt with core dependencies
  • ☐ Create requirements-dev.txt with testing/linting tools
  • ☐ Create setup.py for package installation
  • ☐ Create pyproject.toml with tool configurations
  • ☐ Create Makefile with common commands
  • ☐ Create README.md with project overview
  • ☐ Create .env.example with required environment variables

📝 Configuration Files to Create:

  • ☐ config/config.yaml - Main configuration
  • ☐ config/logging.yaml - Logging setup (JSON formatter, rotating handlers)
  • ☐ config/detectors.yaml - Pattern detector parameters (placeholder values)
  • ☐ config/models.yaml - ML hyperparameters (placeholder values)
  • ☐ config/trading.yaml - Trading parameters (risk limits, time windows)
  • ☐ config/alerts.yaml - Alert configuration (Telegram tokens from env)
  • ☐ config/database.yaml - Database settings

🔧 Core Infrastructure Files:

  • ☐ src/__init__.py - Package marker
  • ☐ src/core/__init__.py
  • ☐ src/core/constants.py - Define TIMEFRAMES, SESSION_TIMES, PATHS
  • ☐ src/core/enums.py - PatternType, Grade, SetupType, TimeWindow enums
  • ☐ src/core/exceptions.py - Custom exception hierarchy (7-8 exception classes)
  • ☐ src/core/base_classes.py - Abstract base classes (BaseDetector, BaseModel, etc.)

📊 Logging System:

  • ☐ src/logging/__init__.py
  • ☐ src/logging/logger.py - get_logger() function, load from logging.yaml
  • ☐ src/logging/formatters.py - JSONFormatter, ColoredFormatter, DetailedFormatter
  • ☐ src/logging/handlers.py - RotatingFileHandler, ErrorFileHandler, DatabaseHandler
  • ☐ src/logging/filters.py - SensitiveDataFilter (redact API keys), RateLimitFilter
  • ☐ src/logging/decorators.py - @log_execution, @log_exceptions, @log_performance

⚙️ Configuration Management:

  • ☐ src/config/__init__.py
  • ☐ src/config/config_loader.py - Load YAML configs, merge with env vars, singleton pattern
  • ☐ src/config/settings.py - Pydantic dataclasses for type-safe config
  • ☐ src/config/validators.py - Validate config ranges, file paths, consistency

🧪 Initial Tests:

  • ☐ tests/__init__.py
  • ☐ tests/conftest.py - Pytest fixtures (temp directories, mock configs)
  • ☐ tests/unit/test_core/test_exceptions.py - Test exception hierarchy
  • ☐ tests/unit/test_logging/test_logger.py - Test logger initialization
  • ☐ tests/unit/test_config/test_config_loader.py - Test config loading

🔍 Validation Steps:

  • ☐ Run: python -m pytest tests/ - All tests pass
  • ☐ Run: python -c "from src.logging import get_logger; logger = get_logger('test'); logger.info('Test')"
  • ☐ Check logs/ directory has app.log file created
  • ☐ Run: python -c "from src.config import config_loader; config = config_loader.load_config()"
  • ☐ Verify no import errors, no missing dependencies

📦 Commit Checklist:

  • ☐ All files have docstrings and type hints
  • ☐ All tests pass
  • ☐ No hardcoded secrets in code
  • ☐ README.md updated with setup instructions
  • ☐ Git commit: git commit -m "feat(v0.1.0): project foundation with logging and config"
  • ☐ Git tag: git tag v0.1.0

🔨 Version 0.2.0 - Data Pipeline

Branch: feature/v0.2.0-data-pipeline

Prerequisites:

  • ☐ v0.1.0 completed and tagged
  • ☐ Branch from develop: git checkout -b feature/v0.2.0-data-pipeline develop

✅ Todo List:

📦 Database Setup:

  • ☐ src/data/__init__.py
  • ☐ src/data/database.py - SQLAlchemy engine, session management, connection pooling
  • ☐ src/data/models.py - ORM models (OHLCVData, DetectedPattern, Label, Trade, etc.)
  • ☐ src/data/repositories.py - Repository classes (OHLCVRepository, PatternRepository)
  • ☐ scripts/setup_database.py - Create tables, indexes, seed data

📥 Data Loaders:

  • ☐ src/data/loaders.py - CSVLoader, ParquetLoader, DatabaseLoader classes
  • ☐ src/data/preprocessors.py - handle_missing_data(), remove_duplicates(), filter_session()
  • ☐ src/data/validators.py - validate_ohlcv(), check_continuity(), detect_outliers()
  • ☐ src/data/schemas.py - Pydantic schemas (OHLCVSchema, PatternSchema)

🛠️ Utility Scripts:

  • ☐ scripts/download_data.py - Download DAX OHLCV data (CLI with argparse)
  • ☐ scripts/process_data.py - Batch preprocessing (clean, filter 3-4am, save)

📁 Data Directory Structure:

  • ☐ Create data/raw/ohlcv/1min/ (with .gitkeep)
  • ☐ Create data/raw/ohlcv/5min/ (with .gitkeep)
  • ☐ Create data/raw/ohlcv/15min/ (with .gitkeep)
  • ☐ Create data/processed/features/ (with .gitkeep)
  • ☐ Create data/processed/patterns/ (with .gitkeep)
  • ☐ Create data/labels/individual_patterns/ (with .gitkeep)

🧪 Tests:

  • ☐ tests/unit/test_data/test_database.py - Test connection, session management
  • ☐ tests/unit/test_data/test_loaders.py - Test loading from various sources
  • ☐ tests/unit/test_data/test_preprocessors.py - Test data cleaning functions
  • ☐ tests/unit/test_data/test_validators.py - Test validation logic
  • ☐ tests/integration/test_database.py - Test full database workflow
  • ☐ tests/fixtures/sample_data/ - Add sample OHLCV CSV for testing

🔍 Validation Steps:

  • ☐ Run: python scripts/setup_database.py - Database created
  • ☐ Run: python scripts/download_data.py --symbol DAX --start 2024-01-01 --end 2024-01-31 --output data/raw/ohlcv/1min/
  • ☐ Verify data file created in data/raw/
  • ☐ Run: python scripts/process_data.py --input data/raw/ --output data/processed/
  • ☐ Verify processed data created
  • ☐ All tests pass: pytest tests/

📦 Commit Checklist:

  • ☐ All repository methods have proper error handling and logging
  • ☐ Database connection uses environment variables
  • ☐ All SQL queries use parameterized statements (no SQL injection)
  • ☐ Data validation catches common issues (nulls, duplicates, outliers)
  • ☐ Git commit: git commit -m "feat(v0.2.0): data pipeline with loaders and database"
  • ☐ Merge to develop, tag: git tag v0.2.0

🔨 Version 0.3.0 - Pattern Detectors

Branch: feature/v0.3.0-pattern-detectors

Prerequisites:

  • ☐ v0.2.0 completed and tagged
  • ☐ Sample OHLCV data available in data/processed/

✅ Todo List:

🔍 Detector Implementation:

  • ☐ src/detectors/__init__.py
  • ☐ src/detectors/base_detector.py - Abstract BaseDetector class with detect() method
  • ☐ src/detectors/fvg_detector.py - FVGDetector class (bullish/bearish gap detection)
  • ☐ src/detectors/order_block_detector.py - OrderBlockDetector (find BOS, last opposite candle)
  • ☐ src/detectors/liquidity_detector.py - LiquidityDetector (swing sweeps, reversals)
  • ☐ src/detectors/premium_discount.py - PremiumDiscountCalculator (equilibrium levels)
  • ☐ src/detectors/structure_detector.py - StructureDetector (BOS, CHoCH, trend)
  • ☐ src/detectors/utils.py - Helper functions (find_swing_high, find_swing_low, calculate_atr)
  • ☐ src/detectors/scanner.py - Scanner class orchestrating all detectors

📊 Visualization:

  • ☐ src/visualization/__init__.py
  • ☐ src/visualization/chart_generator.py - Generate charts with matplotlib/plotly
  • ☐ src/visualization/pattern_plotter.py - Annotate FVG zones, OB levels on charts
  • ☐ src/visualization/screenshot.py - Capture chart screenshots for labeling

🛠️ Scripts:

  • ☐ scripts/run_detection.py - CLI to run detectors on historical data
  • ☐ Add --detector flag (fvg, ob, liquidity, all)
  • ☐ Add --visualize flag to generate screenshots
  • ☐ Save detections to database and data/processed/patterns/

🧪 Tests:

  • ☐ tests/unit/test_detectors/test_fvg_detector.py - Test FVG detection logic
  • ☐ tests/unit/test_detectors/test_order_block_detector.py - Test OB detection
  • ☐ tests/unit/test_detectors/test_liquidity_detector.py - Test liquidity sweeps
  • ☐ tests/unit/test_detectors/test_scanner.py - Test scanner orchestration
  • ☐ tests/fixtures/sample_data/patterns/ - Known patterns for validation

🔍 Validation Steps:

  • ☐ Run: python scripts/run_detection.py --input data/processed/ --detector fvg
  • ☐ Check logs/detectors/fvg_detector.log for detection counts
  • ☐ Verify detections saved to database and data/processed/patterns/
  • ☐ Run with --visualize flag, check screenshots in data/screenshots/
  • ☐ Manually review 10-20 screenshots - are detections correct?
  • ☐ All tests pass: pytest tests/

📦 Commit Checklist:

  • ☐ Each detector logs entry, parameters, detection count, execution time
  • ☐ Detectors handle edge cases (insufficient data, no patterns found)
  • ☐ Scanner catches detector exceptions, continues with other detectors
  • ☐ Detection metadata includes all context (price levels, timestamp, features)
  • ☐ Git commit: git commit -m "feat(v0.3.0): pattern detectors with FVG, OB, liquidity"
  • ☐ Merge to develop, tag: git tag v0.3.0

🔨 Version 0.4.0 - Feature Engineering

Branch: feature/v0.4.0-feature-engineering

✅ Todo List:

🎯 Feature Extractors:

  • ☐ src/features/__init__.py
  • ☐ src/features/base_features.py - BaseFeatureEngineering abstract class
  • ☐ src/features/pattern_features.py - Extract FVG size, OB range, liquidity sweep distance
  • ☐ src/features/technical_features.py - ATR, momentum, volume metrics
  • ☐ src/features/temporal_features.py - Time since open, day of week, session phase
  • ☐ src/features/context_features.py - HTF alignment, premium/discount position
  • ☐ src/features/orderflow_features.py - CVD, delta (if available)
  • ☐ src/features/feature_store.py - Save/load computed features (Parquet)

🧪 Tests:

  • ☐ tests/unit/test_features/ - Test each feature extractor
  • ☐ Verify feature ranges (no NaNs, no infinities)
  • ☐ Test feature reproducibility (same input → same output)

🔍 Validation Steps:

  • ☐ Run feature extraction on sample patterns
  • ☐ Verify feature values are reasonable (no extreme outliers)
  • ☐ Check feature correlation matrix (detect redundant features)
  • ☐ Save features to data/processed/features/

📦 Commit & Tag v0.4.0

🔨 Version 0.5.0 - Labeling System

Branch: feature/v0.5.0-labeling-system

✅ Todo List:

🏷️ Labeling Components:

  • ☐ src/labeling/__init__.py
  • ☐ src/labeling/interface.py - Orchestrator for labeling workflow
  • ☐ src/labeling/notion_integration.py - Notion API client (optional)
  • ☐ src/labeling/web_ui.py - Flask/FastAPI web interface (optional)
  • ☐ src/labeling/quality_checker.py - Inter-session consistency, drift detection
  • ☐ src/labeling/anchors.py - Manage anchor examples for calibration
  • ☐ src/labeling/export.py - Export labels to training format

🛠️ Scripts:

  • ☐ scripts/export_labels.py - Export from Notion or web UI to CSV/database

📊 Grading Rubrics:

  • ☐ docs/grading_rubric.md - Document FVG, OB, Liquidity grading criteria (1-5 scale)
  • ☐ Create 20-30 anchor examples per pattern type
  • ☐ Save anchors in data/labels/anchors/

🔍 Validation Steps:

  • ☐ Test labeling workflow end-to-end
  • ☐ Label 50 patterns, check quality metrics
  • ☐ Re-label 10 anchors after 1 day - check consistency

📦 Commit & Tag v0.5.0

🔨 Version 0.6.0 - ML Models

Branch: feature/v0.6.0-ml-models

✅ Todo List:

🤖 Model Components:

  • ☐ src/models/__init__.py
  • ☐ src/models/base_model.py - Abstract BaseModel with train(), predict(), evaluate()
  • ☐ src/models/pattern_grader.py - RandomForest/XGBoost for individual patterns
  • ☐ src/models/setup_classifier.py - Meta-model for complete setups
  • ☐ src/models/training.py - Training pipeline (load labels, extract features, train, save)
  • ☐ src/models/evaluation.py - Calculate metrics (accuracy, precision, recall, F1)
  • ☐ src/models/tuning.py - Hyperparameter search (GridSearchCV or Optuna)
  • ☐ src/models/inference.py - Real-time prediction with loaded model
  • ☐ src/models/registry.py - Track model versions, metadata

🛠️ Scripts:

  • ☐ scripts/train_models.py - CLI for training (--model-type, --pattern, --tune)
  • ☐ scripts/evaluate_models.py - Generate evaluation reports

📁 Model Storage:

  • ☐ Save trained models to models/pattern_graders/fvg/v1.0.0/
  • ☐ Include model.pkl, hyperparameters.yaml, features.json, metrics.json
  • ☐ Create models/metadata/model_registry.json
  • ☐ Create symlink models/pattern_graders/fvg/latest/ → v1.0.0/

🧪 Tests:

  • ☐ tests/unit/test_models/ - Test training, prediction, evaluation
  • ☐ tests/validation/test_model_accuracy.py - Assert accuracy >= 75%

🔍 Validation Steps:

  • ☐ Train on 300 labeled patterns
  • ☐ Evaluate on held-out test set
  • ☐ Check train/test gap < 10%
  • ☐ Generate confusion matrix, feature importance

📦 Commit & Tag v0.6.0

🔨 Version 0.7.0 - Backtesting

Branch: feature/v0.7.0-backtesting

✅ Todo List:

📈 Backtesting Components:

  • ☐ src/backtesting/__init__.py
  • ☐ src/backtesting/engine.py - Backtest execution loop
  • ☐ src/backtesting/metrics.py - Win rate, Sharpe, max drawdown, expectancy
  • ☐ src/backtesting/reports.py - HTML/PDF report generation
  • ☐ src/backtesting/visualizations.py - Equity curve, drawdown chart

🛠️ Scripts:

  • ☐ scripts/backtest.py - Run strategy backtest (--strategy, --start, --end, --capital)

🔍 Validation Steps:

  • ☐ Backtest continuation strategy on 6 months data
  • ☐ Generate performance report
  • ☐ Verify metrics match manual backtest (sanity check)

📦 Commit & Tag v0.7.0

🔨 Version 0.8.0 - Alert System

Branch: feature/v0.8.0-alerts

✅ Todo List:

🔔 Alert Components:

  • ☐ src/alerts/__init__.py
  • ☐ src/alerts/base_alert.py - Abstract BaseAlertHandler
  • ☐ src/alerts/telegram_bot.py - Telegram bot with python-telegram-bot
  • ☐ src/alerts/slack_integration.py - Slack webhook integration (optional)
  • ☐ src/alerts/email_alerts.py - Email via SMTP (optional)
  • ☐ src/alerts/alert_manager.py - Route alerts to appropriate channels
  • ☐ src/alerts/templates.py - Message templates with pattern details

🔍 Validation Steps:

  • ☐ Send test alert to Telegram
  • ☐ Verify message formatting, pattern details included
  • ☐ Test alert rate limiting (no spam)

📦 Commit & Tag v0.8.0

🔨 Version 0.9.0 - Paper Trading

Branch: feature/v0.9.0-paper-trading

✅ Todo List:

💼 Trading Components:

  • ☐ src/trading/__init__.py
  • ☐ src/trading/broker_interface.py - Abstract broker API (paper trading mode)
  • ☐ src/trading/order_manager.py - Order submission, tracking
  • ☐ src/trading/risk_manager.py - Position sizing, max loss checks
  • ☐ src/trading/portfolio.py - Track positions, P&L
  • ☐ src/trading/execution_engine.py - Live detection → prediction → order execution
  • ☐ src/trading/validators.py - Pre-trade validation (risk checks)

🛠️ Scripts:

  • ☐ scripts/live_trading.py - Run live (paper) trading session
  • ☐ Add --dry-run flag for simulation
  • ☐ Add --auto-grade flag (automate only grade 5 predictions)

🔍 Validation Steps:

  • ☐ Run paper trading for 1 week (5 sessions)
  • ☐ Manually validate each alert before execution
  • ☐ Track model grade vs. your judgment (correlation)
  • ☐ Log all decisions to logs/trading/orders.log

📦 Commit & Tag v0.9.0

🔨 Version 1.0.0 - Production Ready

Branch: feature/v1.0.0-production

✅ Todo List:

🚀 Production Readiness:

  • ☐ deployment/Dockerfile - Create production Docker image
  • ☐ deployment/docker-compose.yml - Multi-container setup (app + database + monitoring)
  • ☐ deployment/systemd/ict-trading.service - Systemd service file
  • ☐ monitoring/prometheus.yml - Prometheus metrics collection
  • ☐ monitoring/grafana_dashboards/ - Create dashboards for key metrics
  • ☐ src/monitoring/metrics_collector.py - Expose Prometheus metrics
  • ☐ src/monitoring/health_checker.py - Health check endpoint

📚 Documentation:

  • ☐ docs/architecture.md - Complete system architecture
  • ☐ docs/api_reference.md - Generate from docstrings (Sphinx)
  • ☐ docs/user_guide.md - Setup, configuration, usage
  • ☐ docs/deployment.md - Deployment instructions
  • ☐ docs/troubleshooting.md - Common issues and solutions
  • ☐ README.md - Update with badges, quick start, features

🧪 Final Testing:

  • ☐ Run full test suite: pytest tests/ --cov=src --cov-report=html
  • ☐ Verify code coverage >= 80%
  • ☐ Run linting: flake8 src/
  • ☐ Run type checking: mypy src/
  • ☐ Test Docker build: docker build -t ict-trading .
  • ☐ Test deployment in staging environment

🔒 Security Audit:

  • ☐ Verify no secrets in code (use .env only)
  • ☐ Check all API keys are redacted in logs
  • ☐ Verify database uses parameterized queries
  • ☐ Run security scan: bandit -r src/

📦 Release Checklist:

  • ☐ Update CHANGELOG.md with all changes
  • ☐ Update version in setup.py to 1.0.0
  • ☐ Merge to main branch
  • ☐ Git tag: git tag v1.0.0 -m "Release version 1.0.0"
  • ☐ Push tags: git push origin main --tags
  • ☐ Create GitHub release with notes

🔄 Post-Release: Continuous Improvement

Ongoing Maintenance Versions

v1.1.0 - Performance Optimization:

  • Profile code with cProfile, identify bottlenecks
  • Optimize hot loops with Numba JIT compilation
  • Add caching for expensive computations
  • Database query optimization (indexes, query plans)

v1.2.0 - Enhanced Features:

  • Add more order flow features (if ATAS API available)
  • Implement model retraining pipeline (monthly)
  • Add regime detection features
  • Enhance visualization dashboard

v1.3.0 - Advanced ML:

  • Experiment with XGBoost/LightGBM vs. RandomForest
  • Implement ensemble models
  • Add SHAP explainability
  • Hyperparameter optimization with Optuna

v2.0.0 - Multi-Instrument Support:

  • Extend to trade multiple instruments (NQ, ES)
  • Shared feature engineering pipeline
  • Instrument-specific models
  • Portfolio-level risk management

⚠️ Troubleshooting Guide for AI Agents

Common Issues & Solutions

❌ Tests Failing:

  • Issue: Import errors
  • Solution: Ensure all __init__.py files exist, check PYTHONPATH
  • Issue: Database connection errors
  • Solution: Check .env file has correct DATABASE_URL, test connection manually

❌ Logging Not Working:

  • Issue: No logs appearing
  • Solution: Check logging.yaml loaded correctly, verify logs/ directory exists with write permissions

❌ Pattern Detection Produces No Results:

  • Issue: Detectors find 0 patterns
  • Solution: Check data date range (3-4 AM only?), verify detection thresholds not too strict, add debug logging

❌ Model Training Fails:

  • Issue: Insufficient labeled data
  • Solution: Need minimum 200-300 labels per pattern type, verify labels exported correctly
  • Issue: Feature extraction errors
  • Solution: Check for NaN values, validate feature ranges, log feature statistics

💡 Best Practice: Always Check Logs First

Every component logs extensively. Before debugging code, check:

  • logs/application/app.log - General flow
  • logs/errors/exceptions.log - Errors with stack traces
  • logs/[component]/*.log - Component-specific logs

📝 AI Agent Final Checklist

Before marking any version complete, verify:
  • ✅ All files have proper docstrings (Google style)
  • ✅ All functions have type hints
  • ✅ All risky operations wrapped in try-except
  • ✅ All errors logged with context
  • ✅ All tests pass: pytest tests/
  • ✅ No hardcoded secrets (use .env)
  • ✅ No TODO comments left in code
  • ✅ Git commit messages follow format: type(scope): description
  • ✅ Version tagged: git tag vX.Y.Z
  • ✅ Merged to develop branch