## Complete Directory Structure

🎯 Architecture Philosophy: This project follows enterprise-grade Python best practices with clear separation of concerns, comprehensive error handling, extensive logging, and modular design for maintainability and scalability.
ict-ml-trading/ │ ├── .env.example # Environment variables template ├── .env # Actual environment variables (git-ignored) ├── .gitignore # Git ignore patterns ├── .pre-commit-config.yaml # Pre-commit hooks configuration ├── README.md # Project documentation ├── requirements.txt # Production dependencies ├── requirements-dev.txt # Development dependencies ├── setup.py # Package installation configuration ├── pyproject.toml # Modern Python project configuration ├── Makefile # Common commands automation ├── CHANGELOG.md # Version history and changes ├── LICENSE # Software license │ ├── config/ # Configuration files │ ├── config.yaml # Main application configuration │ ├── logging.yaml # Logging configuration │ ├── detectors.yaml # Pattern detector parameters │ ├── models.yaml # ML model hyperparameters │ ├── trading.yaml # Trading strategy parameters │ ├── alerts.yaml # Alert system configuration │ └── database.yaml # Database connection settings │ ├── src/ # Main source code │ ├── __init__.py │ │ │ ├── core/ # Core business logic │ │ ├── __init__.py │ │ ├── exceptions.py # Custom exception classes │ │ ├── constants.py # Application-wide constants │ │ ├── enums.py # Enumerations (PatternType, Grade, etc.) │ │ └── base_classes.py # Abstract base classes │ │ │ ├── config/ # Configuration management │ │ ├── __init__.py │ │ ├── config_loader.py # Load and validate configurations │ │ ├── settings.py # Settings dataclasses │ │ └── validators.py # Configuration validation logic │ │ │ ├── logging/ # Logging system │ │ ├── __init__.py │ │ ├── logger.py # Logger setup and configuration │ │ ├── formatters.py # Custom log formatters │ │ ├── handlers.py # Custom log handlers │ │ ├── filters.py # Log filters │ │ └── decorators.py # Logging decorators (@log_execution) │ │ │ ├── data/ # Data management │ │ ├── __init__.py │ │ ├── database.py # Database connection and ORM │ │ ├── models.py # SQLAlchemy data models │ │ ├── repositories.py # Data access layer (repository pattern) │ │ ├── loaders.py # Data loading utilities │ │ ├── preprocessors.py # Data preprocessing and cleaning │ │ ├── validators.py # Data validation │ │ └── schemas.py # Pydantic schemas for validation │ │ │ ├── detectors/ # Pattern detection │ │ ├── __init__.py │ │ ├── base_detector.py # Abstract base detector class │ │ ├── fvg_detector.py # Fair Value Gap detection │ │ ├── order_block_detector.py # Order Block detection │ │ ├── liquidity_detector.py # Liquidity sweep detection │ │ ├── premium_discount.py # Premium/Discount calculator │ │ ├── structure_detector.py # Market structure (BOS, CHoCH) │ │ ├── scanner.py # Orchestrates all detectors │ │ └── utils.py # Detector utility functions │ │ │ ├── features/ # Feature engineering │ │ ├── __init__.py │ │ ├── base_features.py # Base feature engineering class │ │ ├── pattern_features.py # Pattern-specific features │ │ ├── technical_features.py # Technical indicators │ │ ├── temporal_features.py # Time-based features │ │ ├── context_features.py # Market context features │ │ ├── orderflow_features.py # Order flow features │ │ └── feature_store.py # Feature storage and retrieval │ │ │ ├── models/ # Machine learning models │ │ ├── __init__.py │ │ ├── base_model.py # Abstract base model class │ │ ├── pattern_grader.py # Individual pattern grading models │ │ ├── setup_classifier.py # Complete setup classification │ │ ├── training.py # Model training pipeline │ │ ├── evaluation.py # Model evaluation metrics │ │ ├── tuning.py # Hyperparameter tuning │ │ ├── inference.py # Real-time prediction │ │ └── registry.py # Model versioning and registry │ │ │ ├── labeling/ # Labeling system │ │ ├── __init__.py │ │ ├── interface.py # Labeling interface orchestrator │ │ ├── notion_integration.py # Notion API integration │ │ ├── web_ui.py # Flask/FastAPI web interface │ │ ├── quality_checker.py # Label quality validation │ │ ├── anchors.py # Anchor example management │ │ └── export.py # Export labels to training format │ │ │ ├── alerts/ # Alert system │ │ ├── __init__.py │ │ ├── base_alert.py # Abstract alert handler │ │ ├── telegram_bot.py # Telegram bot integration │ │ ├── slack_integration.py # Slack integration │ │ ├── email_alerts.py # Email notifications │ │ ├── alert_manager.py # Alert orchestration and routing │ │ └── templates.py # Alert message templates │ │ │ ├── trading/ # Trading execution │ │ ├── __init__.py │ │ ├── broker_interface.py # Broker API abstraction │ │ ├── order_manager.py # Order execution and tracking │ │ ├── risk_manager.py # Position sizing and risk controls │ │ ├── portfolio.py # Portfolio state management │ │ ├── execution_engine.py # Trade execution logic │ │ └── validators.py # Pre-trade validation │ │ │ ├── backtesting/ # Backtesting framework │ │ ├── __init__.py │ │ ├── engine.py # Backtest execution engine │ │ ├── metrics.py # Performance metrics calculation │ │ ├── reports.py # Backtest report generation │ │ ├── visualizations.py # Performance visualizations │ │ └── validators.py # Backtest result validation │ │ │ ├── monitoring/ # System monitoring │ │ ├── __init__.py │ │ ├── metrics_collector.py # Collect system metrics │ │ ├── health_checker.py # System health checks │ │ ├── performance_tracker.py # Track model performance │ │ ├── alerting.py # Monitoring alerts │ │ └── dashboard.py # Monitoring dashboard │ │ │ ├── visualization/ # Chart and data visualization │ │ ├── __init__.py │ │ ├── chart_generator.py # Generate trading charts │ │ ├── pattern_plotter.py # Plot detected patterns │ │ ├── performance_plots.py # Performance visualizations │ │ └── screenshot.py # Screenshot capture for labeling │ │ │ └── utils/ # Utility functions │ ├── __init__.py │ ├── datetime_utils.py # Date/time helpers │ ├── file_utils.py # File operations │ ├── string_utils.py # String manipulation │ ├── math_utils.py # Mathematical utilities │ ├── decorators.py # Utility decorators │ ├── retry.py # Retry logic for API calls │ └── validators.py # General validation functions │ ├── data/ # Data storage │ ├── raw/ # Raw downloaded data │ │ ├── ohlcv/ # OHLCV market data │ │ │ ├── 1min/ │ │ │ ├── 5min/ │ │ │ └── 15min/ │ │ └── orderflow/ # Order flow data (if available) │ │ │ ├── processed/ # Cleaned and processed data │ │ ├── features/ # Computed features │ │ ├── patterns/ # Detected patterns │ │ └── snapshots/ # OHLC snapshots for patterns │ │ │ ├── labels/ # Labeled data │ │ ├── individual_patterns/ # Pattern-level labels │ │ ├── complete_setups/ # Setup-level labels │ │ ├── anchors/ # Anchor examples │ │ └── label_metadata.json # Label versioning and metadata │ │ │ ├── screenshots/ # Chart screenshots for labeling │ │ ├── patterns/ # Pattern screenshots │ │ └── setups/ # Complete setup screenshots │ │ │ └── external/ # External data sources │ ├── economic_calendar/ # News events │ └── reference/ # Reference data (holidays, etc.) │ ├── models/ # Trained models storage │ ├── pattern_graders/ # Individual pattern models │ │ ├── fvg/ │ │ │ ├── v1.0.0/ # Versioned models │ │ │ ├── v1.1.0/ │ │ │ └── latest/ # Symlink to latest version │ │ ├── order_block/ │ │ └── liquidity/ │ │ │ ├── strategy_models/ # Complete setup models │ │ ├── continuation/ # 3:00-3:15 continuation model │ │ └── reversal/ # 3:30-3:50 reversal model │ │ │ ├── metadata/ # Model metadata and configs │ │ ├── model_registry.json # Track all model versions │ │ └── performance_history.json # Historical performance │ │ │ └── artifacts/ # Training artifacts │ ├── feature_importances/ │ ├── training_curves/ │ └── validation_reports/ │ ├── logs/ # Application logs │ ├── application/ # General application logs │ │ ├── app.log # Current log file │ │ └── archive/ # Rotated logs │ │ │ ├── detectors/ # Pattern detection logs │ │ ├── fvg_detector.log │ │ ├── order_block_detector.log │ │ └── scanner.log │ │ │ ├── models/ # ML model logs │ │ ├── training.log # Training sessions │ │ ├── inference.log # Real-time predictions │ │ └── evaluation.log # Model evaluation │ │ │ ├── trading/ # Trading execution logs │ │ ├── orders.log # Order execution │ │ ├── risk.log # Risk management decisions │ │ └── portfolio.log # Portfolio changes │ │ │ ├── alerts/ # Alert system logs │ │ ├── telegram.log │ │ └── notifications.log │ │ │ ├── errors/ # Error logs │ │ ├── exceptions.log # All exceptions │ │ └── critical.log # Critical errors only │ │ │ ├── performance/ # Performance monitoring │ │ ├── metrics.log # System metrics │ │ └── profiling.log # Performance profiling │ │ │ └── audit/ # Audit trail │ ├── data_access.log # Data access audit │ ├── config_changes.log # Configuration changes │ └── model_deployments.log # Model deployment history │ ├── tests/ # Test suite │ ├── __init__.py │ ├── conftest.py # Pytest configuration and fixtures │ │ │ ├── unit/ # Unit tests │ │ ├── test_detectors/ │ │ ├── test_features/ │ │ ├── test_models/ │ │ └── test_utils/ │ │ │ ├── integration/ # Integration tests │ │ ├── test_pipeline.py # End-to-end pipeline │ │ ├── test_database.py # Database integration │ │ └── test_alerts.py # Alert system integration │ │ │ ├── validation/ # Validation tests │ │ ├── test_model_accuracy.py │ │ ├── test_backtest_results.py │ │ └── test_data_quality.py │ │ │ ├── fixtures/ # Test data and fixtures │ │ ├── sample_data/ │ │ ├── mock_responses/ │ │ └── expected_outputs/ │ │ │ └── performance/ # Performance tests │ ├── test_speed.py # Speed benchmarks │ └── test_memory.py # Memory usage tests │ ├── scripts/ # Utility scripts │ ├── download_data.py # Download historical data │ ├── process_data.py # Batch data processing │ ├── run_detection.py # Run pattern detection │ ├── train_models.py # Train ML models │ ├── evaluate_models.py # Evaluate model performance │ ├── backtest.py # Run backtests │ ├── live_trading.py # Live trading execution │ ├── setup_database.py # Initialize database │ ├── migrate_data.py # Data migration │ ├── cleanup.py # Cleanup old logs/data │ └── export_labels.py # Export labels from Notion │ ├── notebooks/ # Jupyter notebooks │ ├── 01_data_exploration.ipynb │ ├── 02_pattern_analysis.ipynb │ ├── 03_feature_engineering.ipynb │ ├── 04_model_training.ipynb │ ├── 05_model_evaluation.ipynb │ ├── 06_backtest_analysis.ipynb │ └── 07_label_quality_check.ipynb │ ├── docs/ # Documentation │ ├── architecture.md # System architecture │ ├── api_reference.md # API documentation │ ├── user_guide.md # User guide │ ├── deployment.md # Deployment guide │ ├── troubleshooting.md # Troubleshooting guide │ ├── contributing.md # Contribution guidelines │ ├── diagrams/ # Architecture diagrams │ └── examples/ # Code examples │ ├── deployment/ # Deployment configuration │ ├── Dockerfile # Docker container │ ├── docker-compose.yml # Multi-container setup │ ├── .dockerignore # Docker ignore patterns │ ├── kubernetes/ # Kubernetes manifests │ │ ├── deployment.yaml │ │ ├── service.yaml │ │ └── configmap.yaml │ ├── systemd/ # Systemd service files │ │ └── ict-trading.service │ └── supervisor/ # Supervisor configuration │ └── ict-trading.conf │ ├── monitoring/ # Monitoring configuration │ ├── prometheus.yml # Prometheus config │ ├── grafana_dashboards/ # Grafana dashboards │ └── alerts.yml # Alert rules │ └── backups/ # Backup directory ├── database/ # Database backups ├── models/ # Model backups └── config/ # Configuration backups

## Key File Descriptions

### 📝 Logging System (src/logging/)

Purpose: Comprehensive logging infrastructure for debugging, monitoring, and auditing

- logger.py: Central logger setup - creates module-specific loggers with configured handlers and formatters
- formatters.py: JSON formatter for machine parsing, colored console formatter, detailed/compact formatters
- handlers.py: Rotating file handlers (size/time-based), database handler for critical events, alert handler for CRITICAL logs
- filters.py: Sensitive data redaction (API keys, passwords), rate limiting for repeated messages
- decorators.py: @log_execution (entry/exit/timing), @log_exceptions (catch and log errors), @log_performance

### ⚠️ Error Handling (src/core/exceptions.py)

Custom Exception Hierarchy:

- `ICTTradingException` - Base exception with error codes and context data
- `DataError` - Data loading/validation failures
- `DetectorError` - Pattern detection failures
- `ModelError` - ML training/inference errors
- `ConfigurationError` - Invalid configuration
- `TradingError` - Order execution failures

Every exception is logged with full context and stack trace

### 📊 Data Management (data/)

Directory | Purpose | Retention
--- | --- | ---
data/raw/ | Downloaded OHLCV data (Parquet format) | 24 months
data/processed/ | Cleaned data, detected patterns, features | 12 months
data/labels/ | User labels (versioned), anchor examples | Permanent
data/screenshots/ | Chart images for labeling | 6 months

### 🤖 Model Versioning (models/)

Every model version includes:

- `model.pkl` - Trained model file
- `hyperparameters.yaml` - All configuration used
- `features.json` - Feature list and scaling parameters
- `metrics.json` - Performance metrics (accuracy, precision, recall)
- `training_log.txt` - Full training session log
- `feature_importance.csv` - Which features matter most

Symlink "latest/" always points to production model - easy rollback

### 📈 Logging Categories (logs/)

Log Type | Contents | Rotation
--- | --- | ---
application/ | Startup, shutdown, config loading | Daily or 10MB
detectors/ | Pattern detection runs, counts, timing | Daily or 10MB
models/ | Training, inference, evaluation | Per session
trading/ | Orders, fills, risk decisions, P&L | Never rotate (compliance)
errors/ | All exceptions with stack traces | Daily or 10MB
audit/ | Data access, config changes, deployments | Monthly

All logs: 30-day retention, compressed archives for older logs

## Best Practices Implementation

### 🔒 Security

- All secrets in .env (never committed)
- Sensitive data filtered from logs automatically
- Database credentials from environment variables only
- API keys validated at startup, logged failures
- All external calls use HTTPS

### ⚡ Performance

- Async logging (non-blocking for high-volume logs)
- Parquet format for data (10x faster than CSV)
- Database indexes on frequently queried columns
- Connection pooling for database
- Retry logic with exponential backoff for API calls
- Performance profiling logs (identify bottlenecks)

### 🧪 Testing

- Unit tests for all detectors, features, models
- Integration tests for end-to-end pipeline
- Validation tests ensure model accuracy thresholds met
- Performance tests benchmark speed and memory
- Fixtures provide realistic test data
- Target: 80%+ code coverage

### 🚀 Deployment

- Docker for reproducible environments
- docker-compose for multi-container setup
- Kubernetes manifests for cloud deployment
- systemd service for Linux servers
- Health checks and auto-restart
- Blue-green deployment support

## 🤖 AI Agent Implementation Guide

📋 Purpose: This guide provides step-by-step instructions for AI agents (like Claude, GPT, etc.) to build the ICT ML Trading System. Each version is a stable checkpoint that can be committed to Git, tested, and rolled back if needed.

### 🎯 For AI Agents: Code Quality Standards

When implementing ANY file in this project, ALWAYS follow these rules:

- Type Hints: Add type hints to all function signatures (PEP 484)
- Docstrings: Write Google-style docstrings for all classes and functions
- Error Handling: Wrap risky operations in try-except, use custom exceptions from src/core/exceptions.py
- Logging: Log entry/exit of major functions, all errors, and important state changes using src/logging/logger.py
- Validation: Validate all inputs (function args, config values, data)
- Constants: No magic numbers - define constants in src/core/constants.py
- Configuration: Read parameters from config files, never hardcode
- Testing: Write at least one test for each public function
- Comments: Explain WHY, not WHAT (code should be self-explanatory)
- Dependencies: Minimize external dependencies, document all additions to requirements.txt

### 🌳 Git Branching Strategy

Recommended Branch Model (GitHub Flow - Simple & Effective):

- main: Production-ready code, always stable
- develop: Integration branch for completed features
- feature/VERSION-feature-name: Feature development branches
- hotfix/issue-description: Emergency fixes

Workflow:

`# Start new version
git checkout develop
git checkout -b feature/v0.1.0-project-setup
# Work on version, commit frequently
git add .
git commit -m "feat: add project structure and config files"
# Complete version
git checkout develop
git merge feature/v0.1.0-project-setup
git tag v0.1.0
git push origin develop --tags
# If stable, merge to main
git checkout main
git merge develop
git push origin main`

## 📦 Version Roadmap

Version | Milestone | Duration | Deliverables
--- | --- | --- | ---
v0.1.0 | Project Foundation | 1-2 days | Project structure, configs, logging
v0.2.0 | Data Pipeline | 2-3 days | Data loading, preprocessing, database
v0.3.0 | Pattern Detectors | 3-5 days | FVG, OB, Liquidity detectors
v0.4.0 | Feature Engineering | 2-3 days | Feature extraction pipeline
v0.5.0 | Labeling System | 2-3 days | Labeling interface, quality checks
v0.6.0 | ML Models | 3-4 days | Training pipeline, evaluation
v0.7.0 | Backtesting | 2-3 days | Backtest engine, metrics
v0.8.0 | Alert System | 1-2 days | Telegram bot, notifications
v0.9.0 | Paper Trading | 2-3 days | Live detection, simulated execution
v1.0.0 | Production Ready | 1-2 days | Deployment, monitoring, docs

## 🔨 Version 0.1.0 - Project Foundation

### Branch: feature/v0.1.0-project-setup

#### ✅ Todo List:

- ☐ Create directory structure (all folders as shown above)
- ☐ Initialize Git repository: `git init`
- ☐ Create .gitignore file with Python, data, logs, models patterns
- ☐ Create requirements.txt with core dependencies
- ☐ Create requirements-dev.txt with testing/linting tools
- ☐ Create setup.py for package installation
- ☐ Create pyproject.toml with tool configurations
- ☐ Create Makefile with common commands
- ☐ Create README.md with project overview
- ☐ Create .env.example with required environment variables

#### 📝 Configuration Files to Create:

- ☐ config/config.yaml - Main configuration
- ☐ config/logging.yaml - Logging setup (JSON formatter, rotating handlers)
- ☐ config/detectors.yaml - Pattern detector parameters (placeholder values)
- ☐ config/models.yaml - ML hyperparameters (placeholder values)
- ☐ config/trading.yaml - Trading parameters (risk limits, time windows)
- ☐ config/alerts.yaml - Alert configuration (Telegram tokens from env)
- ☐ config/database.yaml - Database settings

#### 🔧 Core Infrastructure Files:

- ☐ src/__init__.py - Package marker
- ☐ src/core/__init__.py
- ☐ src/core/constants.py - Define TIMEFRAMES, SESSION_TIMES, PATHS
- ☐ src/core/enums.py - PatternType, Grade, SetupType, TimeWindow enums
- ☐ src/core/exceptions.py - Custom exception hierarchy (7-8 exception classes)
- ☐ src/core/base_classes.py - Abstract base classes (BaseDetector, BaseModel, etc.)

#### 📊 Logging System:

- ☐ src/logging/__init__.py
- ☐ src/logging/logger.py - get_logger() function, load from logging.yaml
- ☐ src/logging/formatters.py - JSONFormatter, ColoredFormatter, DetailedFormatter
- ☐ src/logging/handlers.py - RotatingFileHandler, ErrorFileHandler, DatabaseHandler
- ☐ src/logging/filters.py - SensitiveDataFilter (redact API keys), RateLimitFilter
- ☐ src/logging/decorators.py - @log_execution, @log_exceptions, @log_performance

#### ⚙️ Configuration Management:

- ☐ src/config/__init__.py
- ☐ src/config/config_loader.py - Load YAML configs, merge with env vars, singleton pattern
- ☐ src/config/settings.py - Pydantic dataclasses for type-safe config
- ☐ src/config/validators.py - Validate config ranges, file paths, consistency

#### 🧪 Initial Tests:

- ☐ tests/__init__.py
- ☐ tests/conftest.py - Pytest fixtures (temp directories, mock configs)
- ☐ tests/unit/test_core/test_exceptions.py - Test exception hierarchy
- ☐ tests/unit/test_logging/test_logger.py - Test logger initialization
- ☐ tests/unit/test_config/test_config_loader.py - Test config loading

#### 🔍 Validation Steps:

- ☐ Run: `python -m pytest tests/` - All tests pass
- ☐ Run: `python -c "from src.logging import get_logger; logger = get_logger('test'); logger.info('Test')"`
- ☐ Check logs/ directory has app.log file created
- ☐ Run: `python -c "from src.config import config_loader; config = config_loader.load_config()"`
- ☐ Verify no import errors, no missing dependencies

#### 📦 Commit Checklist:

- ☐ All files have docstrings and type hints
- ☐ All tests pass
- ☐ No hardcoded secrets in code
- ☐ README.md updated with setup instructions
- ☐ Git commit: `git commit -m "feat(v0.1.0): project foundation with logging and config"`
- ☐ Git tag: `git tag v0.1.0`

## 🔨 Version 0.2.0 - Data Pipeline

### Branch: feature/v0.2.0-data-pipeline

#### Prerequisites:

- ☐ v0.1.0 completed and tagged
- ☐ Branch from develop: `git checkout -b feature/v0.2.0-data-pipeline develop`

#### ✅ Todo List:

#### 📦 Database Setup:

- ☐ src/data/__init__.py
- ☐ src/data/database.py - SQLAlchemy engine, session management, connection pooling
- ☐ src/data/models.py - ORM models (OHLCVData, DetectedPattern, Label, Trade, etc.)
- ☐ src/data/repositories.py - Repository classes (OHLCVRepository, PatternRepository)
- ☐ scripts/setup_database.py - Create tables, indexes, seed data

#### 📥 Data Loaders:

- ☐ src/data/loaders.py - CSVLoader, ParquetLoader, DatabaseLoader classes
- ☐ src/data/preprocessors.py - handle_missing_data(), remove_duplicates(), filter_session()
- ☐ src/data/validators.py - validate_ohlcv(), check_continuity(), detect_outliers()
- ☐ src/data/schemas.py - Pydantic schemas (OHLCVSchema, PatternSchema)

#### 🛠️ Utility Scripts:

- ☐ scripts/download_data.py - Download DAX OHLCV data (CLI with argparse)
- ☐ scripts/process_data.py - Batch preprocessing (clean, filter 3-4am, save)

#### 📁 Data Directory Structure:

- ☐ Create data/raw/ohlcv/1min/ (with .gitkeep)
- ☐ Create data/raw/ohlcv/5min/ (with .gitkeep)
- ☐ Create data/raw/ohlcv/15min/ (with .gitkeep)
- ☐ Create data/processed/features/ (with .gitkeep)
- ☐ Create data/processed/patterns/ (with .gitkeep)
- ☐ Create data/labels/individual_patterns/ (with .gitkeep)

#### 🧪 Tests:

- ☐ tests/unit/test_data/test_database.py - Test connection, session management
- ☐ tests/unit/test_data/test_loaders.py - Test loading from various sources
- ☐ tests/unit/test_data/test_preprocessors.py - Test data cleaning functions
- ☐ tests/unit/test_data/test_validators.py - Test validation logic
- ☐ tests/integration/test_database.py - Test full database workflow
- ☐ tests/fixtures/sample_data/ - Add sample OHLCV CSV for testing

#### 🔍 Validation Steps:

- ☐ Run: `python scripts/setup_database.py` - Database created
- ☐ Run: `python scripts/download_data.py --symbol DAX --start 2024-01-01 --end 2024-01-31 --output data/raw/ohlcv/1min/`
- ☐ Verify data file created in data/raw/
- ☐ Run: `python scripts/process_data.py --input data/raw/ --output data/processed/`
- ☐ Verify processed data created
- ☐ All tests pass: `pytest tests/`

#### 📦 Commit Checklist:

- ☐ All repository methods have proper error handling and logging
- ☐ Database connection uses environment variables
- ☐ All SQL queries use parameterized statements (no SQL injection)
- ☐ Data validation catches common issues (nulls, duplicates, outliers)
- ☐ Git commit: `git commit -m "feat(v0.2.0): data pipeline with loaders and database"`
- ☐ Merge to develop, tag: `git tag v0.2.0`

## 🔨 Version 0.3.0 - Pattern Detectors

### Branch: feature/v0.3.0-pattern-detectors

#### Prerequisites:

- ☐ v0.2.0 completed and tagged
- ☐ Sample OHLCV data available in data/processed/

#### ✅ Todo List:

#### 🔍 Detector Implementation:

- ☐ src/detectors/__init__.py
- ☐ src/detectors/base_detector.py - Abstract BaseDetector class with detect() method
- ☐ src/detectors/fvg_detector.py - FVGDetector class (bullish/bearish gap detection)
- ☐ src/detectors/order_block_detector.py - OrderBlockDetector (find BOS, last opposite candle)
- ☐ src/detectors/liquidity_detector.py - LiquidityDetector (swing sweeps, reversals)
- ☐ src/detectors/premium_discount.py - PremiumDiscountCalculator (equilibrium levels)
- ☐ src/detectors/structure_detector.py - StructureDetector (BOS, CHoCH, trend)
- ☐ src/detectors/utils.py - Helper functions (find_swing_high, find_swing_low, calculate_atr)
- ☐ src/detectors/scanner.py - Scanner class orchestrating all detectors

#### 📊 Visualization:

- ☐ src/visualization/__init__.py
- ☐ src/visualization/chart_generator.py - Generate charts with matplotlib/plotly
- ☐ src/visualization/pattern_plotter.py - Annotate FVG zones, OB levels on charts
- ☐ src/visualization/screenshot.py - Capture chart screenshots for labeling

#### 🛠️ Scripts:

- ☐ scripts/run_detection.py - CLI to run detectors on historical data
- ☐ Add --detector flag (fvg, ob, liquidity, all)
- ☐ Add --visualize flag to generate screenshots
- ☐ Save detections to database and data/processed/patterns/

#### 🧪 Tests:

- ☐ tests/unit/test_detectors/test_fvg_detector.py - Test FVG detection logic
- ☐ tests/unit/test_detectors/test_order_block_detector.py - Test OB detection
- ☐ tests/unit/test_detectors/test_liquidity_detector.py - Test liquidity sweeps
- ☐ tests/unit/test_detectors/test_scanner.py - Test scanner orchestration
- ☐ tests/fixtures/sample_data/patterns/ - Known patterns for validation

#### 🔍 Validation Steps:

- ☐ Run: `python scripts/run_detection.py --input data/processed/ --detector fvg`
- ☐ Check logs/detectors/fvg_detector.log for detection counts
- ☐ Verify detections saved to database and data/processed/patterns/
- ☐ Run with --visualize flag, check screenshots in data/screenshots/
- ☐ Manually review 10-20 screenshots - are detections correct?
- ☐ All tests pass: `pytest tests/`

#### 📦 Commit Checklist:

- ☐ Each detector logs entry, parameters, detection count, execution time
- ☐ Detectors handle edge cases (insufficient data, no patterns found)
- ☐ Scanner catches detector exceptions, continues with other detectors
- ☐ Detection metadata includes all context (price levels, timestamp, features)
- ☐ Git commit: `git commit -m "feat(v0.3.0): pattern detectors with FVG, OB, liquidity"`
- ☐ Merge to develop, tag: `git tag v0.3.0`

## 🔨 Version 0.4.0 - Feature Engineering

### Branch: feature/v0.4.0-feature-engineering

#### ✅ Todo List:

#### 🎯 Feature Extractors:

- ☐ src/features/__init__.py
- ☐ src/features/base_features.py - BaseFeatureEngineering abstract class
- ☐ src/features/pattern_features.py - Extract FVG size, OB range, liquidity sweep distance
- ☐ src/features/technical_features.py - ATR, momentum, volume metrics
- ☐ src/features/temporal_features.py - Time since open, day of week, session phase
- ☐ src/features/context_features.py - HTF alignment, premium/discount position
- ☐ src/features/orderflow_features.py - CVD, delta (if available)
- ☐ src/features/feature_store.py - Save/load computed features (Parquet)

#### 🧪 Tests:

- ☐ tests/unit/test_features/ - Test each feature extractor
- ☐ Verify feature ranges (no NaNs, no infinities)
- ☐ Test feature reproducibility (same input → same output)

#### 🔍 Validation Steps:

- ☐ Run feature extraction on sample patterns
- ☐ Verify feature values are reasonable (no extreme outliers)
- ☐ Check feature correlation matrix (detect redundant features)
- ☐ Save features to data/processed/features/

#### 📦 Commit & Tag v0.4.0

## 🔨 Version 0.5.0 - Labeling System

### Branch: feature/v0.5.0-labeling-system

#### ✅ Todo List:

#### 🏷️ Labeling Components:

- ☐ src/labeling/__init__.py
- ☐ src/labeling/interface.py - Orchestrator for labeling workflow
- ☐ src/labeling/notion_integration.py - Notion API client (optional)
- ☐ src/labeling/web_ui.py - Flask/FastAPI web interface (optional)
- ☐ src/labeling/quality_checker.py - Inter-session consistency, drift detection
- ☐ src/labeling/anchors.py - Manage anchor examples for calibration
- ☐ src/labeling/export.py - Export labels to training format

#### 🛠️ Scripts:

- ☐ scripts/export_labels.py - Export from Notion or web UI to CSV/database

#### 📊 Grading Rubrics:

- ☐ docs/grading_rubric.md - Document FVG, OB, Liquidity grading criteria (1-5 scale)
- ☐ Create 20-30 anchor examples per pattern type
- ☐ Save anchors in data/labels/anchors/

#### 🔍 Validation Steps:

- ☐ Test labeling workflow end-to-end
- ☐ Label 50 patterns, check quality metrics
- ☐ Re-label 10 anchors after 1 day - check consistency

#### 📦 Commit & Tag v0.5.0

## 🔨 Version 0.6.0 - ML Models

### Branch: feature/v0.6.0-ml-models

#### ✅ Todo List:

#### 🤖 Model Components:

- ☐ src/models/__init__.py
- ☐ src/models/base_model.py - Abstract BaseModel with train(), predict(), evaluate()
- ☐ src/models/pattern_grader.py - RandomForest/XGBoost for individual patterns
- ☐ src/models/setup_classifier.py - Meta-model for complete setups
- ☐ src/models/training.py - Training pipeline (load labels, extract features, train, save)
- ☐ src/models/evaluation.py - Calculate metrics (accuracy, precision, recall, F1)
- ☐ src/models/tuning.py - Hyperparameter search (GridSearchCV or Optuna)
- ☐ src/models/inference.py - Real-time prediction with loaded model
- ☐ src/models/registry.py - Track model versions, metadata

#### 🛠️ Scripts:

- ☐ scripts/train_models.py - CLI for training (--model-type, --pattern, --tune)
- ☐ scripts/evaluate_models.py - Generate evaluation reports

#### 📁 Model Storage:

- ☐ Save trained models to models/pattern_graders/fvg/v1.0.0/
- ☐ Include model.pkl, hyperparameters.yaml, features.json, metrics.json
- ☐ Create models/metadata/model_registry.json
- ☐ Create symlink models/pattern_graders/fvg/latest/ → v1.0.0/

#### 🧪 Tests:

- ☐ tests/unit/test_models/ - Test training, prediction, evaluation
- ☐ tests/validation/test_model_accuracy.py - Assert accuracy >= 75%

#### 🔍 Validation Steps:

- ☐ Train on 300 labeled patterns
- ☐ Evaluate on held-out test set
- ☐ Check train/test gap < 10%
- ☐ Generate confusion matrix, feature importance

#### 📦 Commit & Tag v0.6.0

## 🔨 Version 0.7.0 - Backtesting

### Branch: feature/v0.7.0-backtesting

#### ✅ Todo List:

#### 📈 Backtesting Components:

- ☐ src/backtesting/__init__.py
- ☐ src/backtesting/engine.py - Backtest execution loop
- ☐ src/backtesting/metrics.py - Win rate, Sharpe, max drawdown, expectancy
- ☐ src/backtesting/reports.py - HTML/PDF report generation
- ☐ src/backtesting/visualizations.py - Equity curve, drawdown chart

#### 🛠️ Scripts:

- ☐ scripts/backtest.py - Run strategy backtest (--strategy, --start, --end, --capital)

#### 🔍 Validation Steps:

- ☐ Backtest continuation strategy on 6 months data
- ☐ Generate performance report
- ☐ Verify metrics match manual backtest (sanity check)

#### 📦 Commit & Tag v0.7.0

## 🔨 Version 0.8.0 - Alert System

### Branch: feature/v0.8.0-alerts

#### ✅ Todo List:

#### 🔔 Alert Components:

- ☐ src/alerts/__init__.py
- ☐ src/alerts/base_alert.py - Abstract BaseAlertHandler
- ☐ src/alerts/telegram_bot.py - Telegram bot with python-telegram-bot
- ☐ src/alerts/slack_integration.py - Slack webhook integration (optional)
- ☐ src/alerts/email_alerts.py - Email via SMTP (optional)
- ☐ src/alerts/alert_manager.py - Route alerts to appropriate channels
- ☐ src/alerts/templates.py - Message templates with pattern details

#### 🔍 Validation Steps:

- ☐ Send test alert to Telegram
- ☐ Verify message formatting, pattern details included
- ☐ Test alert rate limiting (no spam)

#### 📦 Commit & Tag v0.8.0

## 🔨 Version 0.9.0 - Paper Trading

### Branch: feature/v0.9.0-paper-trading

#### ✅ Todo List:

#### 💼 Trading Components:

- ☐ src/trading/__init__.py
- ☐ src/trading/broker_interface.py - Abstract broker API (paper trading mode)
- ☐ src/trading/order_manager.py - Order submission, tracking
- ☐ src/trading/risk_manager.py - Position sizing, max loss checks
- ☐ src/trading/portfolio.py - Track positions, P&L
- ☐ src/trading/execution_engine.py - Live detection → prediction → order execution
- ☐ src/trading/validators.py - Pre-trade validation (risk checks)

#### 🛠️ Scripts:

- ☐ scripts/live_trading.py - Run live (paper) trading session
- ☐ Add --dry-run flag for simulation
- ☐ Add --auto-grade flag (automate only grade 5 predictions)

#### 🔍 Validation Steps:

- ☐ Run paper trading for 1 week (5 sessions)
- ☐ Manually validate each alert before execution
- ☐ Track model grade vs. your judgment (correlation)
- ☐ Log all decisions to logs/trading/orders.log

#### 📦 Commit & Tag v0.9.0

## 🔨 Version 1.0.0 - Production Ready

### Branch: feature/v1.0.0-production

#### ✅ Todo List:

#### 🚀 Production Readiness:

- ☐ deployment/Dockerfile - Create production Docker image
- ☐ deployment/docker-compose.yml - Multi-container setup (app + database + monitoring)
- ☐ deployment/systemd/ict-trading.service - Systemd service file
- ☐ monitoring/prometheus.yml - Prometheus metrics collection
- ☐ monitoring/grafana_dashboards/ - Create dashboards for key metrics
- ☐ src/monitoring/metrics_collector.py - Expose Prometheus metrics
- ☐ src/monitoring/health_checker.py - Health check endpoint

#### 📚 Documentation:

- ☐ docs/architecture.md - Complete system architecture
- ☐ docs/api_reference.md - Generate from docstrings (Sphinx)
- ☐ docs/user_guide.md - Setup, configuration, usage
- ☐ docs/deployment.md - Deployment instructions
- ☐ docs/troubleshooting.md - Common issues and solutions
- ☐ README.md - Update with badges, quick start, features

#### 🧪 Final Testing:

- ☐ Run full test suite: `pytest tests/ --cov=src --cov-report=html`
- ☐ Verify code coverage >= 80%
- ☐ Run linting: `flake8 src/`
- ☐ Run type checking: `mypy src/`
- ☐ Test Docker build: `docker build -t ict-trading .`
- ☐ Test deployment in staging environment

#### 🔒 Security Audit:

- ☐ Verify no secrets in code (use .env only)
- ☐ Check all API keys are redacted in logs
- ☐ Verify database uses parameterized queries
- ☐ Run security scan: `bandit -r src/`

#### 📦 Release Checklist:

- ☐ Update CHANGELOG.md with all changes
- ☐ Update version in setup.py to 1.0.0
- ☐ Merge to main branch
- ☐ Git tag: `git tag v1.0.0 -m "Release version 1.0.0"`
- ☐ Push tags: `git push origin main --tags`
- ☐ Create GitHub release with notes

## 🔄 Post-Release: Continuous Improvement

### Ongoing Maintenance Versions

#### v1.1.0 - Performance Optimization:

- Profile code with cProfile, identify bottlenecks
- Optimize hot loops with Numba JIT compilation
- Add caching for expensive computations
- Database query optimization (indexes, query plans)

#### v1.2.0 - Enhanced Features:

- Add more order flow features (if ATAS API available)
- Implement model retraining pipeline (monthly)
- Add regime detection features
- Enhance visualization dashboard

#### v1.3.0 - Advanced ML:

- Experiment with XGBoost/LightGBM vs. RandomForest
- Implement ensemble models
- Add SHAP explainability
- Hyperparameter optimization with Optuna

#### v2.0.0 - Multi-Instrument Support:

- Extend to trade multiple instruments (NQ, ES)
- Shared feature engineering pipeline
- Instrument-specific models
- Portfolio-level risk management

## ⚠️ Troubleshooting Guide for AI Agents

### Common Issues & Solutions

#### ❌ Tests Failing:

- Issue: Import errors
- Solution: Ensure all __init__.py files exist, check PYTHONPATH
- Issue: Database connection errors
- Solution: Check .env file has correct DATABASE_URL, test connection manually

#### ❌ Logging Not Working:

- Issue: No logs appearing
- Solution: Check logging.yaml loaded correctly, verify logs/ directory exists with write permissions

#### ❌ Pattern Detection Produces No Results:

- Issue: Detectors find 0 patterns
- Solution: Check data date range (3-4 AM only?), verify detection thresholds not too strict, add debug logging

#### ❌ Model Training Fails:

- Issue: Insufficient labeled data
- Solution: Need minimum 200-300 labels per pattern type, verify labels exported correctly
- Issue: Feature extraction errors
- Solution: Check for NaN values, validate feature ranges, log feature statistics

#### 💡 Best Practice: Always Check Logs First

Every component logs extensively. Before debugging code, check:

- logs/application/app.log - General flow
- logs/errors/exceptions.log - Errors with stack traces
- logs/[component]/*.log - Component-specific logs

## 📝 AI Agent Final Checklist

Before marking any version complete, verify:

- ✅ All files have proper docstrings (Google style)
- ✅ All functions have type hints
- ✅ All risky operations wrapped in try-except
- ✅ All errors logged with context
- ✅ All tests pass: `pytest tests/`
- ✅ No hardcoded secrets (use .env)
- ✅ No TODO comments left in code
- ✅ Git commit messages follow format: `type(scope): description`
- ✅ Version tagged: `git tag vX.Y.Z`
- ✅ Merged to develop branch