1372 lines
70 KiB
HTML
1372 lines
70 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="UTF-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
<title>ICT ML Trading System - Project Structure</title>
|
|
<style>
|
|
* { margin: 0; padding: 0; box-sizing: border-box; }
|
|
body {
|
|
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
|
|
line-height: 1.6;
|
|
color: #333;
|
|
background: #f5f5f5;
|
|
padding: 20px;
|
|
}
|
|
.container {
|
|
max-width: 1400px;
|
|
margin: 0 auto;
|
|
background: white;
|
|
box-shadow: 0 0 20px rgba(0,0,0,0.1);
|
|
border-radius: 10px;
|
|
}
|
|
header {
|
|
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
|
color: white;
|
|
padding: 40px;
|
|
border-radius: 10px 10px 0 0;
|
|
}
|
|
h1 { font-size: 2.5em; margin-bottom: 10px; }
|
|
.subtitle { font-size: 1.2em; opacity: 0.9; }
|
|
.content { padding: 40px; }
|
|
h2 {
|
|
color: #667eea;
|
|
margin: 40px 0 20px 0;
|
|
padding-bottom: 10px;
|
|
border-bottom: 3px solid #667eea;
|
|
font-size: 1.8em;
|
|
}
|
|
.tree {
|
|
font-family: 'Courier New', monospace;
|
|
background: #f8f9fa;
|
|
padding: 30px;
|
|
border-radius: 10px;
|
|
margin: 20px 0;
|
|
overflow-x: auto;
|
|
border-left: 4px solid #667eea;
|
|
white-space: pre;
|
|
font-size: 14px;
|
|
line-height: 1.4;
|
|
}
|
|
.alert {
|
|
padding: 15px 20px;
|
|
border-radius: 8px;
|
|
margin: 20px 0;
|
|
border-left: 5px solid;
|
|
}
|
|
.alert-info {
|
|
background: #d1ecf1;
|
|
border-color: #17a2b8;
|
|
color: #0c5460;
|
|
}
|
|
.section-card {
|
|
background: #f8f9fa;
|
|
padding: 25px;
|
|
margin: 20px 0;
|
|
border-radius: 10px;
|
|
border-top: 4px solid #667eea;
|
|
}
|
|
table {
|
|
width: 100%;
|
|
border-collapse: collapse;
|
|
margin: 20px 0;
|
|
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
|
|
}
|
|
th {
|
|
background: #667eea;
|
|
color: white;
|
|
padding: 15px;
|
|
text-align: left;
|
|
font-weight: 600;
|
|
}
|
|
td {
|
|
padding: 12px 15px;
|
|
border-bottom: 1px solid #ddd;
|
|
}
|
|
tr:hover { background: #f8f9fa; }
|
|
ul { margin: 10px 0 10px 25px; }
|
|
li { margin: 5px 0; }
|
|
code {
|
|
background: #f4f4f4;
|
|
padding: 2px 6px;
|
|
border-radius: 3px;
|
|
font-family: 'Courier New', monospace;
|
|
color: #c7254e;
|
|
}
|
|
footer {
|
|
margin-top: 60px;
|
|
padding: 30px 40px;
|
|
background: #f8f9fa;
|
|
border-radius: 0 0 10px 10px;
|
|
text-align: center;
|
|
color: #666;
|
|
}
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<div class="container">
|
|
<header>
|
|
<h1>📁 ICT ML Trading System</h1>
|
|
<p class="subtitle">Complete Project Structure & File Organization</p>
|
|
<p style="margin-top: 20px; opacity: 0.9;">Production-grade architecture with comprehensive logging, error handling, and best practices</p>
|
|
</header>
|
|
|
|
<div class="content">
|
|
<section>
|
|
<h2>Complete Directory Structure</h2>
|
|
|
|
<div class="alert alert-info">
|
|
<strong>🎯 Architecture Philosophy:</strong> This project follows enterprise-grade Python best practices with clear separation of concerns, comprehensive error handling, extensive logging, and modular design for maintainability and scalability.
|
|
</div>
|
|
|
|
<div class="tree">ict-ml-trading/
|
|
│
|
|
├── .env.example # Environment variables template
|
|
├── .env # Actual environment variables (git-ignored)
|
|
├── .gitignore # Git ignore patterns
|
|
├── .pre-commit-config.yaml # Pre-commit hooks configuration
|
|
├── README.md # Project documentation
|
|
├── requirements.txt # Production dependencies
|
|
├── requirements-dev.txt # Development dependencies
|
|
├── setup.py # Package installation configuration
|
|
├── pyproject.toml # Modern Python project configuration
|
|
├── Makefile # Common commands automation
|
|
├── CHANGELOG.md # Version history and changes
|
|
├── LICENSE # Software license
|
|
│
|
|
├── config/ # Configuration files
|
|
│ ├── config.yaml # Main application configuration
|
|
│ ├── logging.yaml # Logging configuration
|
|
│ ├── detectors.yaml # Pattern detector parameters
|
|
│ ├── models.yaml # ML model hyperparameters
|
|
│ ├── trading.yaml # Trading strategy parameters
|
|
│ ├── alerts.yaml # Alert system configuration
|
|
│ └── database.yaml # Database connection settings
|
|
│
|
|
├── src/ # Main source code
|
|
│ ├── __init__.py
|
|
│ │
|
|
│ ├── core/ # Core business logic
|
|
│ │ ├── __init__.py
|
|
│ │ ├── exceptions.py # Custom exception classes
|
|
│ │ ├── constants.py # Application-wide constants
|
|
│ │ ├── enums.py # Enumerations (PatternType, Grade, etc.)
|
|
│ │ └── base_classes.py # Abstract base classes
|
|
│ │
|
|
│ ├── config/ # Configuration management
|
|
│ │ ├── __init__.py
|
|
│ │ ├── config_loader.py # Load and validate configurations
|
|
│ │ ├── settings.py # Settings dataclasses
|
|
│ │ └── validators.py # Configuration validation logic
|
|
│ │
|
|
│ ├── logging/ # Logging system
|
|
│ │ ├── __init__.py
|
|
│ │ ├── logger.py # Logger setup and configuration
|
|
│ │ ├── formatters.py # Custom log formatters
|
|
│ │ ├── handlers.py # Custom log handlers
|
|
│ │ ├── filters.py # Log filters
|
|
│ │ └── decorators.py # Logging decorators (@log_execution)
|
|
│ │
|
|
│ ├── data/ # Data management
|
|
│ │ ├── __init__.py
|
|
│ │ ├── database.py # Database connection and ORM
|
|
│ │ ├── models.py # SQLAlchemy data models
|
|
│ │ ├── repositories.py # Data access layer (repository pattern)
|
|
│ │ ├── loaders.py # Data loading utilities
|
|
│ │ ├── preprocessors.py # Data preprocessing and cleaning
|
|
│ │ ├── validators.py # Data validation
|
|
│ │ └── schemas.py # Pydantic schemas for validation
|
|
│ │
|
|
│ ├── detectors/ # Pattern detection
|
|
│ │ ├── __init__.py
|
|
│ │ ├── base_detector.py # Abstract base detector class
|
|
│ │ ├── fvg_detector.py # Fair Value Gap detection
|
|
│ │ ├── order_block_detector.py # Order Block detection
|
|
│ │ ├── liquidity_detector.py # Liquidity sweep detection
|
|
│ │ ├── premium_discount.py # Premium/Discount calculator
|
|
│ │ ├── structure_detector.py # Market structure (BOS, CHoCH)
|
|
│ │ ├── scanner.py # Orchestrates all detectors
|
|
│ │ └── utils.py # Detector utility functions
|
|
│ │
|
|
│ ├── features/ # Feature engineering
|
|
│ │ ├── __init__.py
|
|
│ │ ├── base_features.py # Base feature engineering class
|
|
│ │ ├── pattern_features.py # Pattern-specific features
|
|
│ │ ├── technical_features.py # Technical indicators
|
|
│ │ ├── temporal_features.py # Time-based features
|
|
│ │ ├── context_features.py # Market context features
|
|
│ │ ├── orderflow_features.py # Order flow features
|
|
│ │ └── feature_store.py # Feature storage and retrieval
|
|
│ │
|
|
│ ├── models/ # Machine learning models
|
|
│ │ ├── __init__.py
|
|
│ │ ├── base_model.py # Abstract base model class
|
|
│ │ ├── pattern_grader.py # Individual pattern grading models
|
|
│ │ ├── setup_classifier.py # Complete setup classification
|
|
│ │ ├── training.py # Model training pipeline
|
|
│ │ ├── evaluation.py # Model evaluation metrics
|
|
│ │ ├── tuning.py # Hyperparameter tuning
|
|
│ │ ├── inference.py # Real-time prediction
|
|
│ │ └── registry.py # Model versioning and registry
|
|
│ │
|
|
│ ├── labeling/ # Labeling system
|
|
│ │ ├── __init__.py
|
|
│ │ ├── interface.py # Labeling interface orchestrator
|
|
│ │ ├── notion_integration.py # Notion API integration
|
|
│ │ ├── web_ui.py # Flask/FastAPI web interface
|
|
│ │ ├── quality_checker.py # Label quality validation
|
|
│ │ ├── anchors.py # Anchor example management
|
|
│ │ └── export.py # Export labels to training format
|
|
│ │
|
|
│ ├── alerts/ # Alert system
|
|
│ │ ├── __init__.py
|
|
│ │ ├── base_alert.py # Abstract alert handler
|
|
│ │ ├── telegram_bot.py # Telegram bot integration
|
|
│ │ ├── slack_integration.py # Slack integration
|
|
│ │ ├── email_alerts.py # Email notifications
|
|
│ │ ├── alert_manager.py # Alert orchestration and routing
|
|
│ │ └── templates.py # Alert message templates
|
|
│ │
|
|
│ ├── trading/ # Trading execution
|
|
│ │ ├── __init__.py
|
|
│ │ ├── broker_interface.py # Broker API abstraction
|
|
│ │ ├── order_manager.py # Order execution and tracking
|
|
│ │ ├── risk_manager.py # Position sizing and risk controls
|
|
│ │ ├── portfolio.py # Portfolio state management
|
|
│ │ ├── execution_engine.py # Trade execution logic
|
|
│ │ └── validators.py # Pre-trade validation
|
|
│ │
|
|
│ ├── backtesting/ # Backtesting framework
|
|
│ │ ├── __init__.py
|
|
│ │ ├── engine.py # Backtest execution engine
|
|
│ │ ├── metrics.py # Performance metrics calculation
|
|
│ │ ├── reports.py # Backtest report generation
|
|
│ │ ├── visualizations.py # Performance visualizations
|
|
│ │ └── validators.py # Backtest result validation
|
|
│ │
|
|
│ ├── monitoring/ # System monitoring
|
|
│ │ ├── __init__.py
|
|
│ │ ├── metrics_collector.py # Collect system metrics
|
|
│ │ ├── health_checker.py # System health checks
|
|
│ │ ├── performance_tracker.py # Track model performance
|
|
│ │ ├── alerting.py # Monitoring alerts
|
|
│ │ └── dashboard.py # Monitoring dashboard
|
|
│ │
|
|
│ ├── visualization/ # Chart and data visualization
|
|
│ │ ├── __init__.py
|
|
│ │ ├── chart_generator.py # Generate trading charts
|
|
│ │ ├── pattern_plotter.py # Plot detected patterns
|
|
│ │ ├── performance_plots.py # Performance visualizations
|
|
│ │ └── screenshot.py # Screenshot capture for labeling
|
|
│ │
|
|
│ └── utils/ # Utility functions
|
|
│ ├── __init__.py
|
|
│ ├── datetime_utils.py # Date/time helpers
|
|
│ ├── file_utils.py # File operations
|
|
│ ├── string_utils.py # String manipulation
|
|
│ ├── math_utils.py # Mathematical utilities
|
|
│ ├── decorators.py # Utility decorators
|
|
│ ├── retry.py # Retry logic for API calls
|
|
│ └── validators.py # General validation functions
|
|
│
|
|
├── data/ # Data storage
|
|
│ ├── raw/ # Raw downloaded data
|
|
│ │ ├── ohlcv/ # OHLCV market data
|
|
│ │ │ ├── 1min/
|
|
│ │ │ ├── 5min/
|
|
│ │ │ └── 15min/
|
|
│ │ └── orderflow/ # Order flow data (if available)
|
|
│ │
|
|
│ ├── processed/ # Cleaned and processed data
|
|
│ │ ├── features/ # Computed features
|
|
│ │ ├── patterns/ # Detected patterns
|
|
│ │ └── snapshots/ # OHLC snapshots for patterns
|
|
│ │
|
|
│ ├── labels/ # Labeled data
|
|
│ │ ├── individual_patterns/ # Pattern-level labels
|
|
│ │ ├── complete_setups/ # Setup-level labels
|
|
│ │ ├── anchors/ # Anchor examples
|
|
│ │ └── label_metadata.json # Label versioning and metadata
|
|
│ │
|
|
│ ├── screenshots/ # Chart screenshots for labeling
|
|
│ │ ├── patterns/ # Pattern screenshots
|
|
│ │ └── setups/ # Complete setup screenshots
|
|
│ │
|
|
│ └── external/ # External data sources
|
|
│ ├── economic_calendar/ # News events
|
|
│ └── reference/ # Reference data (holidays, etc.)
|
|
│
|
|
├── models/ # Trained models storage
|
|
│ ├── pattern_graders/ # Individual pattern models
|
|
│ │ ├── fvg/
|
|
│ │ │ ├── v1.0.0/ # Versioned models
|
|
│ │ │ ├── v1.1.0/
|
|
│ │ │ └── latest/ # Symlink to latest version
|
|
│ │ ├── order_block/
|
|
│ │ └── liquidity/
|
|
│ │
|
|
│ ├── strategy_models/ # Complete setup models
|
|
│ │ ├── continuation/ # 3:00-3:15 continuation model
|
|
│ │ └── reversal/ # 3:30-3:50 reversal model
|
|
│ │
|
|
│ ├── metadata/ # Model metadata and configs
|
|
│ │ ├── model_registry.json # Track all model versions
|
|
│ │ └── performance_history.json # Historical performance
|
|
│ │
|
|
│ └── artifacts/ # Training artifacts
|
|
│ ├── feature_importances/
|
|
│ ├── training_curves/
|
|
│ └── validation_reports/
|
|
│
|
|
├── logs/ # Application logs
|
|
│ ├── application/ # General application logs
|
|
│ │ ├── app.log # Current log file
|
|
│ │ └── archive/ # Rotated logs
|
|
│ │
|
|
│ ├── detectors/ # Pattern detection logs
|
|
│ │ ├── fvg_detector.log
|
|
│ │ ├── order_block_detector.log
|
|
│ │ └── scanner.log
|
|
│ │
|
|
│ ├── models/ # ML model logs
|
|
│ │ ├── training.log # Training sessions
|
|
│ │ ├── inference.log # Real-time predictions
|
|
│ │ └── evaluation.log # Model evaluation
|
|
│ │
|
|
│ ├── trading/ # Trading execution logs
|
|
│ │ ├── orders.log # Order execution
|
|
│ │ ├── risk.log # Risk management decisions
|
|
│ │ └── portfolio.log # Portfolio changes
|
|
│ │
|
|
│ ├── alerts/ # Alert system logs
|
|
│ │ ├── telegram.log
|
|
│ │ └── notifications.log
|
|
│ │
|
|
│ ├── errors/ # Error logs
|
|
│ │ ├── exceptions.log # All exceptions
|
|
│ │ └── critical.log # Critical errors only
|
|
│ │
|
|
│ ├── performance/ # Performance monitoring
|
|
│ │ ├── metrics.log # System metrics
|
|
│ │ └── profiling.log # Performance profiling
|
|
│ │
|
|
│ └── audit/ # Audit trail
|
|
│ ├── data_access.log # Data access audit
|
|
│ ├── config_changes.log # Configuration changes
|
|
│ └── model_deployments.log # Model deployment history
|
|
│
|
|
├── tests/ # Test suite
|
|
│ ├── __init__.py
|
|
│ ├── conftest.py # Pytest configuration and fixtures
|
|
│ │
|
|
│ ├── unit/ # Unit tests
|
|
│ │ ├── test_detectors/
|
|
│ │ ├── test_features/
|
|
│ │ ├── test_models/
|
|
│ │ └── test_utils/
|
|
│ │
|
|
│ ├── integration/ # Integration tests
|
|
│ │ ├── test_pipeline.py # End-to-end pipeline
|
|
│ │ ├── test_database.py # Database integration
|
|
│ │ └── test_alerts.py # Alert system integration
|
|
│ │
|
|
│ ├── validation/ # Validation tests
|
|
│ │ ├── test_model_accuracy.py
|
|
│ │ ├── test_backtest_results.py
|
|
│ │ └── test_data_quality.py
|
|
│ │
|
|
│ ├── fixtures/ # Test data and fixtures
|
|
│ │ ├── sample_data/
|
|
│ │ ├── mock_responses/
|
|
│ │ └── expected_outputs/
|
|
│ │
|
|
│ └── performance/ # Performance tests
|
|
│ ├── test_speed.py # Speed benchmarks
|
|
│ └── test_memory.py # Memory usage tests
|
|
│
|
|
├── scripts/ # Utility scripts
|
|
│ ├── download_data.py # Download historical data
|
|
│ ├── process_data.py # Batch data processing
|
|
│ ├── run_detection.py # Run pattern detection
|
|
│ ├── train_models.py # Train ML models
|
|
│ ├── evaluate_models.py # Evaluate model performance
|
|
│ ├── backtest.py # Run backtests
|
|
│ ├── live_trading.py # Live trading execution
|
|
│ ├── setup_database.py # Initialize database
|
|
│ ├── migrate_data.py # Data migration
|
|
│ ├── cleanup.py # Cleanup old logs/data
|
|
│ └── export_labels.py # Export labels from Notion
|
|
│
|
|
├── notebooks/ # Jupyter notebooks
|
|
│ ├── 01_data_exploration.ipynb
|
|
│ ├── 02_pattern_analysis.ipynb
|
|
│ ├── 03_feature_engineering.ipynb
|
|
│ ├── 04_model_training.ipynb
|
|
│ ├── 05_model_evaluation.ipynb
|
|
│ ├── 06_backtest_analysis.ipynb
|
|
│ └── 07_label_quality_check.ipynb
|
|
│
|
|
├── docs/ # Documentation
|
|
│ ├── architecture.md # System architecture
|
|
│ ├── api_reference.md # API documentation
|
|
│ ├── user_guide.md # User guide
|
|
│ ├── deployment.md # Deployment guide
|
|
│ ├── troubleshooting.md # Troubleshooting guide
|
|
│ ├── contributing.md # Contribution guidelines
|
|
│ ├── diagrams/ # Architecture diagrams
|
|
│ └── examples/ # Code examples
|
|
│
|
|
├── deployment/ # Deployment configuration
|
|
│ ├── Dockerfile # Docker container
|
|
│ ├── docker-compose.yml # Multi-container setup
|
|
│ ├── .dockerignore # Docker ignore patterns
|
|
│ ├── kubernetes/ # Kubernetes manifests
|
|
│ │ ├── deployment.yaml
|
|
│ │ ├── service.yaml
|
|
│ │ └── configmap.yaml
|
|
│ ├── systemd/ # Systemd service files
|
|
│ │ └── ict-trading.service
|
|
│ └── supervisor/ # Supervisor configuration
|
|
│ └── ict-trading.conf
|
|
│
|
|
├── monitoring/ # Monitoring configuration
|
|
│ ├── prometheus.yml # Prometheus config
|
|
│ ├── grafana_dashboards/ # Grafana dashboards
|
|
│ └── alerts.yml # Alert rules
|
|
│
|
|
└── backups/ # Backup directory
|
|
├── database/ # Database backups
|
|
├── models/ # Model backups
|
|
└── config/ # Configuration backups</div>
|
|
|
|
<h2>Key File Descriptions</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>📝 Logging System (src/logging/)</h3>
|
|
<p><strong>Purpose:</strong> Comprehensive logging infrastructure for debugging, monitoring, and auditing</p>
|
|
<ul>
|
|
<li><strong>logger.py:</strong> Central logger setup - creates module-specific loggers with configured handlers and formatters</li>
|
|
<li><strong>formatters.py:</strong> JSON formatter for machine parsing, colored console formatter, detailed/compact formatters</li>
|
|
<li><strong>handlers.py:</strong> Rotating file handlers (size/time-based), database handler for critical events, alert handler for CRITICAL logs</li>
|
|
<li><strong>filters.py:</strong> Sensitive data redaction (API keys, passwords), rate limiting for repeated messages</li>
|
|
<li><strong>decorators.py:</strong> @log_execution (entry/exit/timing), @log_exceptions (catch and log errors), @log_performance</li>
|
|
</ul>
|
|
</div>
|
|
|
|
<div class="section-card">
|
|
<h3>⚠️ Error Handling (src/core/exceptions.py)</h3>
|
|
<p><strong>Custom Exception Hierarchy:</strong></p>
|
|
<ul>
|
|
<li><code>ICTTradingException</code> - Base exception with error codes and context data</li>
|
|
<li><code>DataError</code> - Data loading/validation failures</li>
|
|
<li><code>DetectorError</code> - Pattern detection failures</li>
|
|
<li><code>ModelError</code> - ML training/inference errors</li>
|
|
<li><code>ConfigurationError</code> - Invalid configuration</li>
|
|
<li><code>TradingError</code> - Order execution failures</li>
|
|
</ul>
|
|
<p><strong>Every exception is logged with full context and stack trace</strong></p>
|
|
</div>
|
|
|
|
<div class="section-card">
|
|
<h3>📊 Data Management (data/)</h3>
|
|
<table>
|
|
<tr>
|
|
<th>Directory</th>
|
|
<th>Purpose</th>
|
|
<th>Retention</th>
|
|
</tr>
|
|
<tr>
|
|
<td>data/raw/</td>
|
|
<td>Downloaded OHLCV data (Parquet format)</td>
|
|
<td>24 months</td>
|
|
</tr>
|
|
<tr>
|
|
<td>data/processed/</td>
|
|
<td>Cleaned data, detected patterns, features</td>
|
|
<td>12 months</td>
|
|
</tr>
|
|
<tr>
|
|
<td>data/labels/</td>
|
|
<td>User labels (versioned), anchor examples</td>
|
|
<td>Permanent</td>
|
|
</tr>
|
|
<tr>
|
|
<td>data/screenshots/</td>
|
|
<td>Chart images for labeling</td>
|
|
<td>6 months</td>
|
|
</tr>
|
|
</table>
|
|
</div>
|
|
|
|
<div class="section-card">
|
|
<h3>🤖 Model Versioning (models/)</h3>
|
|
<p><strong>Every model version includes:</strong></p>
|
|
<ul>
|
|
<li><code>model.pkl</code> - Trained model file</li>
|
|
<li><code>hyperparameters.yaml</code> - All configuration used</li>
|
|
<li><code>features.json</code> - Feature list and scaling parameters</li>
|
|
<li><code>metrics.json</code> - Performance metrics (accuracy, precision, recall)</li>
|
|
<li><code>training_log.txt</code> - Full training session log</li>
|
|
<li><code>feature_importance.csv</code> - Which features matter most</li>
|
|
</ul>
|
|
<p><strong>Symlink "latest/" always points to production model - easy rollback</strong></p>
|
|
</div>
|
|
|
|
<div class="section-card">
|
|
<h3>📈 Logging Categories (logs/)</h3>
|
|
<table>
|
|
<tr>
|
|
<th>Log Type</th>
|
|
<th>Contents</th>
|
|
<th>Rotation</th>
|
|
</tr>
|
|
<tr>
|
|
<td>application/</td>
|
|
<td>Startup, shutdown, config loading</td>
|
|
<td>Daily or 10MB</td>
|
|
</tr>
|
|
<tr>
|
|
<td>detectors/</td>
|
|
<td>Pattern detection runs, counts, timing</td>
|
|
<td>Daily or 10MB</td>
|
|
</tr>
|
|
<tr>
|
|
<td>models/</td>
|
|
<td>Training, inference, evaluation</td>
|
|
<td>Per session</td>
|
|
</tr>
|
|
<tr>
|
|
<td>trading/</td>
|
|
<td>Orders, fills, risk decisions, P&L</td>
|
|
<td>Never rotate (compliance)</td>
|
|
</tr>
|
|
<tr>
|
|
<td>errors/</td>
|
|
<td>All exceptions with stack traces</td>
|
|
<td>Daily or 10MB</td>
|
|
</tr>
|
|
<tr>
|
|
<td>audit/</td>
|
|
<td>Data access, config changes, deployments</td>
|
|
<td>Monthly</td>
|
|
</tr>
|
|
</table>
|
|
<p><strong>All logs: 30-day retention, compressed archives for older logs</strong></p>
|
|
</div>
|
|
|
|
<h2>Best Practices Implementation</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>🔒 Security</h3>
|
|
<ul>
|
|
<li>All secrets in .env (never committed)</li>
|
|
<li>Sensitive data filtered from logs automatically</li>
|
|
<li>Database credentials from environment variables only</li>
|
|
<li>API keys validated at startup, logged failures</li>
|
|
<li>All external calls use HTTPS</li>
|
|
</ul>
|
|
</div>
|
|
|
|
<div class="section-card">
|
|
<h3>⚡ Performance</h3>
|
|
<ul>
|
|
<li>Async logging (non-blocking for high-volume logs)</li>
|
|
<li>Parquet format for data (10x faster than CSV)</li>
|
|
<li>Database indexes on frequently queried columns</li>
|
|
<li>Connection pooling for database</li>
|
|
<li>Retry logic with exponential backoff for API calls</li>
|
|
<li>Performance profiling logs (identify bottlenecks)</li>
|
|
</ul>
|
|
</div>
|
|
|
|
<div class="section-card">
|
|
<h3>🧪 Testing</h3>
|
|
<ul>
|
|
<li>Unit tests for all detectors, features, models</li>
|
|
<li>Integration tests for end-to-end pipeline</li>
|
|
<li>Validation tests ensure model accuracy thresholds met</li>
|
|
<li>Performance tests benchmark speed and memory</li>
|
|
<li>Fixtures provide realistic test data</li>
|
|
<li>Target: 80%+ code coverage</li>
|
|
</ul>
|
|
</div>
|
|
|
|
<div class="section-card">
|
|
<h3>🚀 Deployment</h3>
|
|
<ul>
|
|
<li>Docker for reproducible environments</li>
|
|
<li>docker-compose for multi-container setup</li>
|
|
<li>Kubernetes manifests for cloud deployment</li>
|
|
<li>systemd service for Linux servers</li>
|
|
<li>Health checks and auto-restart</li>
|
|
<li>Blue-green deployment support</li>
|
|
</ul>
|
|
</div>
|
|
</section>
|
|
</div>
|
|
|
|
<section>
|
|
<h2>🤖 AI Agent Implementation Guide</h2>
|
|
|
|
<div class="alert alert-info">
|
|
<strong>📋 Purpose:</strong> This guide provides step-by-step instructions for AI agents (like Claude, GPT, etc.) to build the ICT ML Trading System. Each version is a stable checkpoint that can be committed to Git, tested, and rolled back if needed.
|
|
</div>
|
|
|
|
<div class="section-card">
|
|
<h3>🎯 For AI Agents: Code Quality Standards</h3>
|
|
<p><strong>When implementing ANY file in this project, ALWAYS follow these rules:</strong></p>
|
|
<ul>
|
|
<li><strong>Type Hints:</strong> Add type hints to all function signatures (PEP 484)</li>
|
|
<li><strong>Docstrings:</strong> Write Google-style docstrings for all classes and functions</li>
|
|
<li><strong>Error Handling:</strong> Wrap risky operations in try-except, use custom exceptions from src/core/exceptions.py</li>
|
|
<li><strong>Logging:</strong> Log entry/exit of major functions, all errors, and important state changes using src/logging/logger.py</li>
|
|
<li><strong>Validation:</strong> Validate all inputs (function args, config values, data)</li>
|
|
<li><strong>Constants:</strong> No magic numbers - define constants in src/core/constants.py</li>
|
|
<li><strong>Configuration:</strong> Read parameters from config files, never hardcode</li>
|
|
<li><strong>Testing:</strong> Write at least one test for each public function</li>
|
|
<li><strong>Comments:</strong> Explain WHY, not WHAT (code should be self-explanatory)</li>
|
|
<li><strong>Dependencies:</strong> Minimize external dependencies, document all additions to requirements.txt</li>
|
|
</ul>
|
|
</div>
|
|
|
|
<div class="section-card">
|
|
<h3>🌳 Git Branching Strategy</h3>
|
|
<p><strong>Recommended Branch Model (GitHub Flow - Simple & Effective):</strong></p>
|
|
<ul>
|
|
<li><strong>main:</strong> Production-ready code, always stable</li>
|
|
<li><strong>develop:</strong> Integration branch for completed features</li>
|
|
<li><strong>feature/VERSION-feature-name:</strong> Feature development branches</li>
|
|
<li><strong>hotfix/issue-description:</strong> Emergency fixes</li>
|
|
</ul>
|
|
|
|
<p><strong>Workflow:</strong></p>
|
|
<code style="display: block; background: #f4f4f4; padding: 15px; margin: 10px 0; border-radius: 5px;">
|
|
# Start new version<br/>
|
|
git checkout develop<br/>
|
|
git checkout -b feature/v0.1.0-project-setup<br/>
|
|
<br/>
|
|
# Work on version, commit frequently<br/>
|
|
git add .<br/>
|
|
git commit -m "feat: add project structure and config files"<br/>
|
|
<br/>
|
|
# Complete version<br/>
|
|
git checkout develop<br/>
|
|
git merge feature/v0.1.0-project-setup<br/>
|
|
git tag v0.1.0<br/>
|
|
git push origin develop --tags<br/>
|
|
<br/>
|
|
# If stable, merge to main<br/>
|
|
git checkout main<br/>
|
|
git merge develop<br/>
|
|
git push origin main
|
|
</code>
|
|
</div>
|
|
|
|
<h2>📦 Version Roadmap</h2>
|
|
|
|
<table>
|
|
<tr>
|
|
<th>Version</th>
|
|
<th>Milestone</th>
|
|
<th>Duration</th>
|
|
<th>Deliverables</th>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>v0.1.0</strong></td>
|
|
<td>Project Foundation</td>
|
|
<td>1-2 days</td>
|
|
<td>Project structure, configs, logging</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>v0.2.0</strong></td>
|
|
<td>Data Pipeline</td>
|
|
<td>2-3 days</td>
|
|
<td>Data loading, preprocessing, database</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>v0.3.0</strong></td>
|
|
<td>Pattern Detectors</td>
|
|
<td>3-5 days</td>
|
|
<td>FVG, OB, Liquidity detectors</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>v0.4.0</strong></td>
|
|
<td>Feature Engineering</td>
|
|
<td>2-3 days</td>
|
|
<td>Feature extraction pipeline</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>v0.5.0</strong></td>
|
|
<td>Labeling System</td>
|
|
<td>2-3 days</td>
|
|
<td>Labeling interface, quality checks</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>v0.6.0</strong></td>
|
|
<td>ML Models</td>
|
|
<td>3-4 days</td>
|
|
<td>Training pipeline, evaluation</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>v0.7.0</strong></td>
|
|
<td>Backtesting</td>
|
|
<td>2-3 days</td>
|
|
<td>Backtest engine, metrics</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>v0.8.0</strong></td>
|
|
<td>Alert System</td>
|
|
<td>1-2 days</td>
|
|
<td>Telegram bot, notifications</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>v0.9.0</strong></td>
|
|
<td>Paper Trading</td>
|
|
<td>2-3 days</td>
|
|
<td>Live detection, simulated execution</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>v1.0.0</strong></td>
|
|
<td>Production Ready</td>
|
|
<td>1-2 days</td>
|
|
<td>Deployment, monitoring, docs</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<h2>🔨 Version 0.1.0 - Project Foundation</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>Branch: feature/v0.1.0-project-setup</h3>
|
|
|
|
<h4>✅ Todo List:</h4>
|
|
<ul>
|
|
<li>☐ Create directory structure (all folders as shown above)</li>
|
|
<li>☐ Initialize Git repository: <code>git init</code></li>
|
|
<li>☐ Create .gitignore file with Python, data, logs, models patterns</li>
|
|
<li>☐ Create requirements.txt with core dependencies</li>
|
|
<li>☐ Create requirements-dev.txt with testing/linting tools</li>
|
|
<li>☐ Create setup.py for package installation</li>
|
|
<li>☐ Create pyproject.toml with tool configurations</li>
|
|
<li>☐ Create Makefile with common commands</li>
|
|
<li>☐ Create README.md with project overview</li>
|
|
<li>☐ Create .env.example with required environment variables</li>
|
|
</ul>
|
|
|
|
<h4>📝 Configuration Files to Create:</h4>
|
|
<ul>
|
|
<li>☐ config/config.yaml - Main configuration</li>
|
|
<li>☐ config/logging.yaml - Logging setup (JSON formatter, rotating handlers)</li>
|
|
<li>☐ config/detectors.yaml - Pattern detector parameters (placeholder values)</li>
|
|
<li>☐ config/models.yaml - ML hyperparameters (placeholder values)</li>
|
|
<li>☐ config/trading.yaml - Trading parameters (risk limits, time windows)</li>
|
|
<li>☐ config/alerts.yaml - Alert configuration (Telegram tokens from env)</li>
|
|
<li>☐ config/database.yaml - Database settings</li>
|
|
</ul>
|
|
|
|
<h4>🔧 Core Infrastructure Files:</h4>
|
|
<ul>
|
|
<li>☐ src/__init__.py - Package marker</li>
|
|
<li>☐ src/core/__init__.py</li>
|
|
<li>☐ src/core/constants.py - Define TIMEFRAMES, SESSION_TIMES, PATHS</li>
|
|
<li>☐ src/core/enums.py - PatternType, Grade, SetupType, TimeWindow enums</li>
|
|
<li>☐ src/core/exceptions.py - Custom exception hierarchy (7-8 exception classes)</li>
|
|
<li>☐ src/core/base_classes.py - Abstract base classes (BaseDetector, BaseModel, etc.)</li>
|
|
</ul>
|
|
|
|
<h4>📊 Logging System:</h4>
|
|
<ul>
|
|
<li>☐ src/logging/__init__.py</li>
|
|
<li>☐ src/logging/logger.py - get_logger() function, load from logging.yaml</li>
|
|
<li>☐ src/logging/formatters.py - JSONFormatter, ColoredFormatter, DetailedFormatter</li>
|
|
<li>☐ src/logging/handlers.py - RotatingFileHandler, ErrorFileHandler, DatabaseHandler</li>
|
|
<li>☐ src/logging/filters.py - SensitiveDataFilter (redact API keys), RateLimitFilter</li>
|
|
<li>☐ src/logging/decorators.py - @log_execution, @log_exceptions, @log_performance</li>
|
|
</ul>
|
|
|
|
<h4>⚙️ Configuration Management:</h4>
|
|
<ul>
|
|
<li>☐ src/config/__init__.py</li>
|
|
<li>☐ src/config/config_loader.py - Load YAML configs, merge with env vars, singleton pattern</li>
|
|
<li>☐ src/config/settings.py - Pydantic dataclasses for type-safe config</li>
|
|
<li>☐ src/config/validators.py - Validate config ranges, file paths, consistency</li>
|
|
</ul>
|
|
|
|
<h4>🧪 Initial Tests:</h4>
|
|
<ul>
|
|
<li>☐ tests/__init__.py</li>
|
|
<li>☐ tests/conftest.py - Pytest fixtures (temp directories, mock configs)</li>
|
|
<li>☐ tests/unit/test_core/test_exceptions.py - Test exception hierarchy</li>
|
|
<li>☐ tests/unit/test_logging/test_logger.py - Test logger initialization</li>
|
|
<li>☐ tests/unit/test_config/test_config_loader.py - Test config loading</li>
|
|
</ul>
|
|
|
|
<h4>🔍 Validation Steps:</h4>
|
|
<ul>
|
|
<li>☐ Run: <code>python -m pytest tests/</code> - All tests pass</li>
|
|
<li>☐ Run: <code>python -c "from src.logging import get_logger; logger = get_logger('test'); logger.info('Test')"</code></li>
|
|
<li>☐ Check logs/ directory has app.log file created</li>
|
|
<li>☐ Run: <code>python -c "from src.config import config_loader; config = config_loader.load_config()"</code></li>
|
|
<li>☐ Verify no import errors, no missing dependencies</li>
|
|
</ul>
|
|
|
|
<h4>📦 Commit Checklist:</h4>
|
|
<ul>
|
|
<li>☐ All files have docstrings and type hints</li>
|
|
<li>☐ All tests pass</li>
|
|
<li>☐ No hardcoded secrets in code</li>
|
|
<li>☐ README.md updated with setup instructions</li>
|
|
<li>☐ Git commit: <code>git commit -m "feat(v0.1.0): project foundation with logging and config"</code></li>
|
|
<li>☐ Git tag: <code>git tag v0.1.0</code></li>
|
|
</ul>
|
|
</div>
|
|
|
|
<h2>🔨 Version 0.2.0 - Data Pipeline</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>Branch: feature/v0.2.0-data-pipeline</h3>
|
|
|
|
<h4>Prerequisites:</h4>
|
|
<ul>
|
|
<li>☐ v0.1.0 completed and tagged</li>
|
|
<li>☐ Branch from develop: <code>git checkout -b feature/v0.2.0-data-pipeline develop</code></li>
|
|
</ul>
|
|
|
|
<h4>✅ Todo List:</h4>
|
|
|
|
<h4>📦 Database Setup:</h4>
|
|
<ul>
|
|
<li>☐ src/data/__init__.py</li>
|
|
<li>☐ src/data/database.py - SQLAlchemy engine, session management, connection pooling</li>
|
|
<li>☐ src/data/models.py - ORM models (OHLCVData, DetectedPattern, Label, Trade, etc.)</li>
|
|
<li>☐ src/data/repositories.py - Repository classes (OHLCVRepository, PatternRepository)</li>
|
|
<li>☐ scripts/setup_database.py - Create tables, indexes, seed data</li>
|
|
</ul>
|
|
|
|
<h4>📥 Data Loaders:</h4>
|
|
<ul>
|
|
<li>☐ src/data/loaders.py - CSVLoader, ParquetLoader, DatabaseLoader classes</li>
|
|
<li>☐ src/data/preprocessors.py - handle_missing_data(), remove_duplicates(), filter_session()</li>
|
|
<li>☐ src/data/validators.py - validate_ohlcv(), check_continuity(), detect_outliers()</li>
|
|
<li>☐ src/data/schemas.py - Pydantic schemas (OHLCVSchema, PatternSchema)</li>
|
|
</ul>
|
|
|
|
<h4>🛠️ Utility Scripts:</h4>
|
|
<ul>
|
|
<li>☐ scripts/download_data.py - Download DAX OHLCV data (CLI with argparse)</li>
|
|
<li>☐ scripts/process_data.py - Batch preprocessing (clean, filter 3-4am, save)</li>
|
|
</ul>
|
|
|
|
<h4>📁 Data Directory Structure:</h4>
|
|
<ul>
|
|
<li>☐ Create data/raw/ohlcv/1min/ (with .gitkeep)</li>
|
|
<li>☐ Create data/raw/ohlcv/5min/ (with .gitkeep)</li>
|
|
<li>☐ Create data/raw/ohlcv/15min/ (with .gitkeep)</li>
|
|
<li>☐ Create data/processed/features/ (with .gitkeep)</li>
|
|
<li>☐ Create data/processed/patterns/ (with .gitkeep)</li>
|
|
<li>☐ Create data/labels/individual_patterns/ (with .gitkeep)</li>
|
|
</ul>
|
|
|
|
<h4>🧪 Tests:</h4>
|
|
<ul>
|
|
<li>☐ tests/unit/test_data/test_database.py - Test connection, session management</li>
|
|
<li>☐ tests/unit/test_data/test_loaders.py - Test loading from various sources</li>
|
|
<li>☐ tests/unit/test_data/test_preprocessors.py - Test data cleaning functions</li>
|
|
<li>☐ tests/unit/test_data/test_validators.py - Test validation logic</li>
|
|
<li>☐ tests/integration/test_database.py - Test full database workflow</li>
|
|
<li>☐ tests/fixtures/sample_data/ - Add sample OHLCV CSV for testing</li>
|
|
</ul>
|
|
|
|
<h4>🔍 Validation Steps:</h4>
|
|
<ul>
|
|
<li>☐ Run: <code>python scripts/setup_database.py</code> - Database created</li>
|
|
<li>☐ Run: <code>python scripts/download_data.py --symbol DAX --start 2024-01-01 --end 2024-01-31 --output data/raw/ohlcv/1min/</code></li>
|
|
<li>☐ Verify data file created in data/raw/</li>
|
|
<li>☐ Run: <code>python scripts/process_data.py --input data/raw/ --output data/processed/</code></li>
|
|
<li>☐ Verify processed data created</li>
|
|
<li>☐ All tests pass: <code>pytest tests/</code></li>
|
|
</ul>
|
|
|
|
<h4>📦 Commit Checklist:</h4>
|
|
<ul>
|
|
<li>☐ All repository methods have proper error handling and logging</li>
|
|
<li>☐ Database connection uses environment variables</li>
|
|
<li>☐ All SQL queries use parameterized statements (no SQL injection)</li>
|
|
<li>☐ Data validation catches common issues (nulls, duplicates, outliers)</li>
|
|
<li>☐ Git commit: <code>git commit -m "feat(v0.2.0): data pipeline with loaders and database"</code></li>
|
|
<li>☐ Merge to develop, tag: <code>git tag v0.2.0</code></li>
|
|
</ul>
|
|
</div>
|
|
|
|
<h2>🔨 Version 0.3.0 - Pattern Detectors</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>Branch: feature/v0.3.0-pattern-detectors</h3>
|
|
|
|
<h4>Prerequisites:</h4>
|
|
<ul>
|
|
<li>☐ v0.2.0 completed and tagged</li>
|
|
<li>☐ Sample OHLCV data available in data/processed/</li>
|
|
</ul>
|
|
|
|
<h4>✅ Todo List:</h4>
|
|
|
|
<h4>🔍 Detector Implementation:</h4>
|
|
<ul>
|
|
<li>☐ src/detectors/__init__.py</li>
|
|
<li>☐ src/detectors/base_detector.py - Abstract BaseDetector class with detect() method</li>
|
|
<li>☐ src/detectors/fvg_detector.py - FVGDetector class (bullish/bearish gap detection)</li>
|
|
<li>☐ src/detectors/order_block_detector.py - OrderBlockDetector (find BOS, last opposite candle)</li>
|
|
<li>☐ src/detectors/liquidity_detector.py - LiquidityDetector (swing sweeps, reversals)</li>
|
|
<li>☐ src/detectors/premium_discount.py - PremiumDiscountCalculator (equilibrium levels)</li>
|
|
<li>☐ src/detectors/structure_detector.py - StructureDetector (BOS, CHoCH, trend)</li>
|
|
<li>☐ src/detectors/utils.py - Helper functions (find_swing_high, find_swing_low, calculate_atr)</li>
|
|
<li>☐ src/detectors/scanner.py - Scanner class orchestrating all detectors</li>
|
|
</ul>
|
|
|
|
<h4>📊 Visualization:</h4>
|
|
<ul>
|
|
<li>☐ src/visualization/__init__.py</li>
|
|
<li>☐ src/visualization/chart_generator.py - Generate charts with matplotlib/plotly</li>
|
|
<li>☐ src/visualization/pattern_plotter.py - Annotate FVG zones, OB levels on charts</li>
|
|
<li>☐ src/visualization/screenshot.py - Capture chart screenshots for labeling</li>
|
|
</ul>
|
|
|
|
<h4>🛠️ Scripts:</h4>
|
|
<ul>
|
|
<li>☐ scripts/run_detection.py - CLI to run detectors on historical data</li>
|
|
<li>☐ Add --detector flag (fvg, ob, liquidity, all)</li>
|
|
<li>☐ Add --visualize flag to generate screenshots</li>
|
|
<li>☐ Save detections to database and data/processed/patterns/</li>
|
|
</ul>
|
|
|
|
<h4>🧪 Tests:</h4>
|
|
<ul>
|
|
<li>☐ tests/unit/test_detectors/test_fvg_detector.py - Test FVG detection logic</li>
|
|
<li>☐ tests/unit/test_detectors/test_order_block_detector.py - Test OB detection</li>
|
|
<li>☐ tests/unit/test_detectors/test_liquidity_detector.py - Test liquidity sweeps</li>
|
|
<li>☐ tests/unit/test_detectors/test_scanner.py - Test scanner orchestration</li>
|
|
<li>☐ tests/fixtures/sample_data/patterns/ - Known patterns for validation</li>
|
|
</ul>
|
|
|
|
<h4>🔍 Validation Steps:</h4>
|
|
<ul>
|
|
<li>☐ Run: <code>python scripts/run_detection.py --input data/processed/ --detector fvg</code></li>
|
|
<li>☐ Check logs/detectors/fvg_detector.log for detection counts</li>
|
|
<li>☐ Verify detections saved to database and data/processed/patterns/</li>
|
|
<li>☐ Run with --visualize flag, check screenshots in data/screenshots/</li>
|
|
<li>☐ Manually review 10-20 screenshots - are detections correct?</li>
|
|
<li>☐ All tests pass: <code>pytest tests/</code></li>
|
|
</ul>
|
|
|
|
<h4>📦 Commit Checklist:</h4>
|
|
<ul>
|
|
<li>☐ Each detector logs entry, parameters, detection count, execution time</li>
|
|
<li>☐ Detectors handle edge cases (insufficient data, no patterns found)</li>
|
|
<li>☐ Scanner catches detector exceptions, continues with other detectors</li>
|
|
<li>☐ Detection metadata includes all context (price levels, timestamp, features)</li>
|
|
<li>☐ Git commit: <code>git commit -m "feat(v0.3.0): pattern detectors with FVG, OB, liquidity"</code></li>
|
|
<li>☐ Merge to develop, tag: <code>git tag v0.3.0</code></li>
|
|
</ul>
|
|
</div>
|
|
|
|
<h2>🔨 Version 0.4.0 - Feature Engineering</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>Branch: feature/v0.4.0-feature-engineering</h3>
|
|
|
|
<h4>✅ Todo List:</h4>
|
|
|
|
<h4>🎯 Feature Extractors:</h4>
|
|
<ul>
|
|
<li>☐ src/features/__init__.py</li>
|
|
<li>☐ src/features/base_features.py - BaseFeatureEngineering abstract class</li>
|
|
<li>☐ src/features/pattern_features.py - Extract FVG size, OB range, liquidity sweep distance</li>
|
|
<li>☐ src/features/technical_features.py - ATR, momentum, volume metrics</li>
|
|
<li>☐ src/features/temporal_features.py - Time since open, day of week, session phase</li>
|
|
<li>☐ src/features/context_features.py - HTF alignment, premium/discount position</li>
|
|
<li>☐ src/features/orderflow_features.py - CVD, delta (if available)</li>
|
|
<li>☐ src/features/feature_store.py - Save/load computed features (Parquet)</li>
|
|
</ul>
|
|
|
|
<h4>🧪 Tests:</h4>
|
|
<ul>
|
|
<li>☐ tests/unit/test_features/ - Test each feature extractor</li>
|
|
<li>☐ Verify feature ranges (no NaNs, no infinities)</li>
|
|
<li>☐ Test feature reproducibility (same input → same output)</li>
|
|
</ul>
|
|
|
|
<h4>🔍 Validation Steps:</h4>
|
|
<ul>
|
|
<li>☐ Run feature extraction on sample patterns</li>
|
|
<li>☐ Verify feature values are reasonable (no extreme outliers)</li>
|
|
<li>☐ Check feature correlation matrix (detect redundant features)</li>
|
|
<li>☐ Save features to data/processed/features/</li>
|
|
</ul>
|
|
|
|
<h4>📦 Commit & Tag v0.4.0</h4>
|
|
</div>
|
|
|
|
<h2>🔨 Version 0.5.0 - Labeling System</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>Branch: feature/v0.5.0-labeling-system</h3>
|
|
|
|
<h4>✅ Todo List:</h4>
|
|
|
|
<h4>🏷️ Labeling Components:</h4>
|
|
<ul>
|
|
<li>☐ src/labeling/__init__.py</li>
|
|
<li>☐ src/labeling/interface.py - Orchestrator for labeling workflow</li>
|
|
<li>☐ src/labeling/notion_integration.py - Notion API client (optional)</li>
|
|
<li>☐ src/labeling/web_ui.py - Flask/FastAPI web interface (optional)</li>
|
|
<li>☐ src/labeling/quality_checker.py - Inter-session consistency, drift detection</li>
|
|
<li>☐ src/labeling/anchors.py - Manage anchor examples for calibration</li>
|
|
<li>☐ src/labeling/export.py - Export labels to training format</li>
|
|
</ul>
|
|
|
|
<h4>🛠️ Scripts:</h4>
|
|
<ul>
|
|
<li>☐ scripts/export_labels.py - Export from Notion or web UI to CSV/database</li>
|
|
</ul>
|
|
|
|
<h4>📊 Grading Rubrics:</h4>
|
|
<ul>
|
|
<li>☐ docs/grading_rubric.md - Document FVG, OB, Liquidity grading criteria (1-5 scale)</li>
|
|
<li>☐ Create 20-30 anchor examples per pattern type</li>
|
|
<li>☐ Save anchors in data/labels/anchors/</li>
|
|
</ul>
|
|
|
|
<h4>🔍 Validation Steps:</h4>
|
|
<ul>
|
|
<li>☐ Test labeling workflow end-to-end</li>
|
|
<li>☐ Label 50 patterns, check quality metrics</li>
|
|
<li>☐ Re-label 10 anchors after 1 day - check consistency</li>
|
|
</ul>
|
|
|
|
<h4>📦 Commit & Tag v0.5.0</h4>
|
|
</div>
|
|
|
|
<h2>🔨 Version 0.6.0 - ML Models</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>Branch: feature/v0.6.0-ml-models</h3>
|
|
|
|
<h4>✅ Todo List:</h4>
|
|
|
|
<h4>🤖 Model Components:</h4>
|
|
<ul>
|
|
<li>☐ src/models/__init__.py</li>
|
|
<li>☐ src/models/base_model.py - Abstract BaseModel with train(), predict(), evaluate()</li>
|
|
<li>☐ src/models/pattern_grader.py - RandomForest/XGBoost for individual patterns</li>
|
|
<li>☐ src/models/setup_classifier.py - Meta-model for complete setups</li>
|
|
<li>☐ src/models/training.py - Training pipeline (load labels, extract features, train, save)</li>
|
|
<li>☐ src/models/evaluation.py - Calculate metrics (accuracy, precision, recall, F1)</li>
|
|
<li>☐ src/models/tuning.py - Hyperparameter search (GridSearchCV or Optuna)</li>
|
|
<li>☐ src/models/inference.py - Real-time prediction with loaded model</li>
|
|
<li>☐ src/models/registry.py - Track model versions, metadata</li>
|
|
</ul>
|
|
|
|
<h4>🛠️ Scripts:</h4>
|
|
<ul>
|
|
<li>☐ scripts/train_models.py - CLI for training (--model-type, --pattern, --tune)</li>
|
|
<li>☐ scripts/evaluate_models.py - Generate evaluation reports</li>
|
|
</ul>
|
|
|
|
<h4>📁 Model Storage:</h4>
|
|
<ul>
|
|
<li>☐ Save trained models to models/pattern_graders/fvg/v1.0.0/</li>
|
|
<li>☐ Include model.pkl, hyperparameters.yaml, features.json, metrics.json</li>
|
|
<li>☐ Create models/metadata/model_registry.json</li>
|
|
<li>☐ Create symlink models/pattern_graders/fvg/latest/ → v1.0.0/</li>
|
|
</ul>
|
|
|
|
<h4>🧪 Tests:</h4>
|
|
<ul>
|
|
<li>☐ tests/unit/test_models/ - Test training, prediction, evaluation</li>
|
|
<li>☐ tests/validation/test_model_accuracy.py - Assert accuracy >= 75%</li>
|
|
</ul>
|
|
|
|
<h4>🔍 Validation Steps:</h4>
|
|
<ul>
|
|
<li>☐ Train on 300 labeled patterns</li>
|
|
<li>☐ Evaluate on held-out test set</li>
|
|
<li>☐ Check train/test gap < 10%</li>
|
|
<li>☐ Generate confusion matrix, feature importance</li>
|
|
</ul>
|
|
|
|
<h4>📦 Commit & Tag v0.6.0</h4>
|
|
</div>
|
|
|
|
<h2>🔨 Version 0.7.0 - Backtesting</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>Branch: feature/v0.7.0-backtesting</h3>
|
|
|
|
<h4>✅ Todo List:</h4>
|
|
|
|
<h4>📈 Backtesting Components:</h4>
|
|
<ul>
|
|
<li>☐ src/backtesting/__init__.py</li>
|
|
<li>☐ src/backtesting/engine.py - Backtest execution loop</li>
|
|
<li>☐ src/backtesting/metrics.py - Win rate, Sharpe, max drawdown, expectancy</li>
|
|
<li>☐ src/backtesting/reports.py - HTML/PDF report generation</li>
|
|
<li>☐ src/backtesting/visualizations.py - Equity curve, drawdown chart</li>
|
|
</ul>
|
|
|
|
<h4>🛠️ Scripts:</h4>
|
|
<ul>
|
|
<li>☐ scripts/backtest.py - Run strategy backtest (--strategy, --start, --end, --capital)</li>
|
|
</ul>
|
|
|
|
<h4>🔍 Validation Steps:</h4>
|
|
<ul>
|
|
<li>☐ Backtest continuation strategy on 6 months data</li>
|
|
<li>☐ Generate performance report</li>
|
|
<li>☐ Verify metrics match manual backtest (sanity check)</li>
|
|
</ul>
|
|
|
|
<h4>📦 Commit & Tag v0.7.0</h4>
|
|
</div>
|
|
|
|
<h2>🔨 Version 0.8.0 - Alert System</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>Branch: feature/v0.8.0-alerts</h3>
|
|
|
|
<h4>✅ Todo List:</h4>
|
|
|
|
<h4>🔔 Alert Components:</h4>
|
|
<ul>
|
|
<li>☐ src/alerts/__init__.py</li>
|
|
<li>☐ src/alerts/base_alert.py - Abstract BaseAlertHandler</li>
|
|
<li>☐ src/alerts/telegram_bot.py - Telegram bot with python-telegram-bot</li>
|
|
<li>☐ src/alerts/slack_integration.py - Slack webhook integration (optional)</li>
|
|
<li>☐ src/alerts/email_alerts.py - Email via SMTP (optional)</li>
|
|
<li>☐ src/alerts/alert_manager.py - Route alerts to appropriate channels</li>
|
|
<li>☐ src/alerts/templates.py - Message templates with pattern details</li>
|
|
</ul>
|
|
|
|
<h4>🔍 Validation Steps:</h4>
|
|
<ul>
|
|
<li>☐ Send test alert to Telegram</li>
|
|
<li>☐ Verify message formatting, pattern details included</li>
|
|
<li>☐ Test alert rate limiting (no spam)</li>
|
|
</ul>
|
|
|
|
<h4>📦 Commit & Tag v0.8.0</h4>
|
|
</div>
|
|
|
|
<h2>🔨 Version 0.9.0 - Paper Trading</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>Branch: feature/v0.9.0-paper-trading</h3>
|
|
|
|
<h4>✅ Todo List:</h4>
|
|
|
|
<h4>💼 Trading Components:</h4>
|
|
<ul>
|
|
<li>☐ src/trading/__init__.py</li>
|
|
<li>☐ src/trading/broker_interface.py - Abstract broker API (paper trading mode)</li>
|
|
<li>☐ src/trading/order_manager.py - Order submission, tracking</li>
|
|
<li>☐ src/trading/risk_manager.py - Position sizing, max loss checks</li>
|
|
<li>☐ src/trading/portfolio.py - Track positions, P&L</li>
|
|
<li>☐ src/trading/execution_engine.py - Live detection → prediction → order execution</li>
|
|
<li>☐ src/trading/validators.py - Pre-trade validation (risk checks)</li>
|
|
</ul>
|
|
|
|
<h4>🛠️ Scripts:</h4>
|
|
<ul>
|
|
<li>☐ scripts/live_trading.py - Run live (paper) trading session</li>
|
|
<li>☐ Add --dry-run flag for simulation</li>
|
|
<li>☐ Add --auto-grade flag (automate only grade 5 predictions)</li>
|
|
</ul>
|
|
|
|
<h4>🔍 Validation Steps:</h4>
|
|
<ul>
|
|
<li>☐ Run paper trading for 1 week (5 sessions)</li>
|
|
<li>☐ Manually validate each alert before execution</li>
|
|
<li>☐ Track model grade vs. your judgment (correlation)</li>
|
|
<li>☐ Log all decisions to logs/trading/orders.log</li>
|
|
</ul>
|
|
|
|
<h4>📦 Commit & Tag v0.9.0</h4>
|
|
</div>
|
|
|
|
<h2>🔨 Version 1.0.0 - Production Ready</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>Branch: feature/v1.0.0-production</h3>
|
|
|
|
<h4>✅ Todo List:</h4>
|
|
|
|
<h4>🚀 Production Readiness:</h4>
|
|
<ul>
|
|
<li>☐ deployment/Dockerfile - Create production Docker image</li>
|
|
<li>☐ deployment/docker-compose.yml - Multi-container setup (app + database + monitoring)</li>
|
|
<li>☐ deployment/systemd/ict-trading.service - Systemd service file</li>
|
|
<li>☐ monitoring/prometheus.yml - Prometheus metrics collection</li>
|
|
<li>☐ monitoring/grafana_dashboards/ - Create dashboards for key metrics</li>
|
|
<li>☐ src/monitoring/metrics_collector.py - Expose Prometheus metrics</li>
|
|
<li>☐ src/monitoring/health_checker.py - Health check endpoint</li>
|
|
</ul>
|
|
|
|
<h4>📚 Documentation:</h4>
|
|
<ul>
|
|
<li>☐ docs/architecture.md - Complete system architecture</li>
|
|
<li>☐ docs/api_reference.md - Generate from docstrings (Sphinx)</li>
|
|
<li>☐ docs/user_guide.md - Setup, configuration, usage</li>
|
|
<li>☐ docs/deployment.md - Deployment instructions</li>
|
|
<li>☐ docs/troubleshooting.md - Common issues and solutions</li>
|
|
<li>☐ README.md - Update with badges, quick start, features</li>
|
|
</ul>
|
|
|
|
<h4>🧪 Final Testing:</h4>
|
|
<ul>
|
|
<li>☐ Run full test suite: <code>pytest tests/ --cov=src --cov-report=html</code></li>
|
|
<li>☐ Verify code coverage >= 80%</li>
|
|
<li>☐ Run linting: <code>flake8 src/</code></li>
|
|
<li>☐ Run type checking: <code>mypy src/</code></li>
|
|
<li>☐ Test Docker build: <code>docker build -t ict-trading .</code></li>
|
|
<li>☐ Test deployment in staging environment</li>
|
|
</ul>
|
|
|
|
<h4>🔒 Security Audit:</h4>
|
|
<ul>
|
|
<li>☐ Verify no secrets in code (use .env only)</li>
|
|
<li>☐ Check all API keys are redacted in logs</li>
|
|
<li>☐ Verify database uses parameterized queries</li>
|
|
<li>☐ Run security scan: <code>bandit -r src/</code></li>
|
|
</ul>
|
|
|
|
<h4>📦 Release Checklist:</h4>
|
|
<ul>
|
|
<li>☐ Update CHANGELOG.md with all changes</li>
|
|
<li>☐ Update version in setup.py to 1.0.0</li>
|
|
<li>☐ Merge to main branch</li>
|
|
<li>☐ Git tag: <code>git tag v1.0.0 -m "Release version 1.0.0"</code></li>
|
|
<li>☐ Push tags: <code>git push origin main --tags</code></li>
|
|
<li>☐ Create GitHub release with notes</li>
|
|
</ul>
|
|
</div>
|
|
|
|
<h2>🔄 Post-Release: Continuous Improvement</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>Ongoing Maintenance Versions</h3>
|
|
|
|
<h4>v1.1.0 - Performance Optimization:</h4>
|
|
<ul>
|
|
<li>Profile code with cProfile, identify bottlenecks</li>
|
|
<li>Optimize hot loops with Numba JIT compilation</li>
|
|
<li>Add caching for expensive computations</li>
|
|
<li>Database query optimization (indexes, query plans)</li>
|
|
</ul>
|
|
|
|
<h4>v1.2.0 - Enhanced Features:</h4>
|
|
<ul>
|
|
<li>Add more order flow features (if ATAS API available)</li>
|
|
<li>Implement model retraining pipeline (monthly)</li>
|
|
<li>Add regime detection features</li>
|
|
<li>Enhance visualization dashboard</li>
|
|
</ul>
|
|
|
|
<h4>v1.3.0 - Advanced ML:</h4>
|
|
<ul>
|
|
<li>Experiment with XGBoost/LightGBM vs. RandomForest</li>
|
|
<li>Implement ensemble models</li>
|
|
<li>Add SHAP explainability</li>
|
|
<li>Hyperparameter optimization with Optuna</li>
|
|
</ul>
|
|
|
|
<h4>v2.0.0 - Multi-Instrument Support:</h4>
|
|
<ul>
|
|
<li>Extend to trade multiple instruments (NQ, ES)</li>
|
|
<li>Shared feature engineering pipeline</li>
|
|
<li>Instrument-specific models</li>
|
|
<li>Portfolio-level risk management</li>
|
|
</ul>
|
|
</div>
|
|
|
|
<h2>⚠️ Troubleshooting Guide for AI Agents</h2>
|
|
|
|
<div class="section-card">
|
|
<h3>Common Issues & Solutions</h3>
|
|
|
|
<h4>❌ Tests Failing:</h4>
|
|
<ul>
|
|
<li><strong>Issue:</strong> Import errors</li>
|
|
<li><strong>Solution:</strong> Ensure all __init__.py files exist, check PYTHONPATH</li>
|
|
<li><strong>Issue:</strong> Database connection errors</li>
|
|
<li><strong>Solution:</strong> Check .env file has correct DATABASE_URL, test connection manually</li>
|
|
</ul>
|
|
|
|
<h4>❌ Logging Not Working:</h4>
|
|
<ul>
|
|
<li><strong>Issue:</strong> No logs appearing</li>
|
|
<li><strong>Solution:</strong> Check logging.yaml loaded correctly, verify logs/ directory exists with write permissions</li>
|
|
</ul>
|
|
|
|
<h4>❌ Pattern Detection Produces No Results:</h4>
|
|
<ul>
|
|
<li><strong>Issue:</strong> Detectors find 0 patterns</li>
|
|
<li><strong>Solution:</strong> Check data date range (3-4 AM only?), verify detection thresholds not too strict, add debug logging</li>
|
|
</ul>
|
|
|
|
<h4>❌ Model Training Fails:</h4>
|
|
<ul>
|
|
<li><strong>Issue:</strong> Insufficient labeled data</li>
|
|
<li><strong>Solution:</strong> Need minimum 200-300 labels per pattern type, verify labels exported correctly</li>
|
|
<li><strong>Issue:</strong> Feature extraction errors</li>
|
|
<li><strong>Solution:</strong> Check for NaN values, validate feature ranges, log feature statistics</li>
|
|
</ul>
|
|
|
|
<h4>💡 Best Practice: Always Check Logs First</h4>
|
|
<p>Every component logs extensively. Before debugging code, check:</p>
|
|
<ul>
|
|
<li>logs/application/app.log - General flow</li>
|
|
<li>logs/errors/exceptions.log - Errors with stack traces</li>
|
|
<li>logs/[component]/*.log - Component-specific logs</li>
|
|
</ul>
|
|
</div>
|
|
|
|
<h2>📝 AI Agent Final Checklist</h2>
|
|
|
|
<div class="alert alert-info">
|
|
<strong>Before marking any version complete, verify:</strong>
|
|
<ul>
|
|
<li>✅ All files have proper docstrings (Google style)</li>
|
|
<li>✅ All functions have type hints</li>
|
|
<li>✅ All risky operations wrapped in try-except</li>
|
|
<li>✅ All errors logged with context</li>
|
|
<li>✅ All tests pass: <code>pytest tests/</code></li>
|
|
<li>✅ No hardcoded secrets (use .env)</li>
|
|
<li>✅ No TODO comments left in code</li>
|
|
<li>✅ Git commit messages follow format: <code>type(scope): description</code></li>
|
|
<li>✅ Version tagged: <code>git tag vX.Y.Z</code></li>
|
|
<li>✅ Merged to develop branch</li>
|
|
</ul>
|
|
</div>
|
|
</section>
|
|
</div>
|
|
|
|
<footer>
|
|
<h3>📁 Project Structure Complete</h3>
|
|
<p>This structure provides production-grade foundation with enterprise-level logging, error handling, testing, and deployment.</p>
|
|
|
|
<p style="margin-top: 20px;"><strong>Key Principles:</strong></p>
|
|
<ul style="text-align: left; max-width: 800px; margin: 10px auto;">
|
|
<li>Every significant action is logged with context</li>
|
|
<li>Errors are caught, logged, and handled gracefully</li>
|
|
<li>Configuration is externalized for easy tuning</li>
|
|
<li>Data and models are versioned for reproducibility</li>
|
|
<li>Comprehensive test coverage ensures reliability</li>
|
|
</ul>
|
|
|
|
<p style="margin-top: 30px; padding-top: 20px; border-top: 1px solid #ddd;">
|
|
<strong>Created:</strong> January 2026 | <strong>For:</strong> ICT ML Trading System - DAX Futures
|
|
</p>
|
|
</footer>
|
|
</div>
|
|
</body>
|
|
</html>
|