1. Problem
Highway fatalities remain one of the leading causes of preventable death in the US — 38,000 annually. Most crashes are preceded by detectable warning signs: tailgating, sudden braking, and dangerous following distances. This project builds a real-time AI pipeline that detects those warning signs from overhead highway camera footage before they escalate into incidents.
2. Approach
- Dataset Engineering: Diagnosed and programmatically fixed a critical class-index mismatch across 82,085 YOLO label files in the UA-DETRAC benchmark, recovering the full 140,000-frame training set from near-zero effective utilisation.
- Training Strategy: Fine-tuned YOLOv8s on dual T4 GPUs with cosine learning rate decay and AMP; validated against the published YOLOv3 baseline (+7.3 mAP points improvement).
- Tracking Pipeline: Implemented ByteTrack multi-object tracking with Kalman filtering for persistent vehicle IDs and per-frame speed estimation in km/h.
- Incident Detection: TTC-based near-miss classifier fires across three alert levels (WARNING / HIGH / CRITICAL); behaviour classifier detects five incident types from motion history.
- Cloud Integration: Full GCP pipeline — Pub/Sub for real-time streaming, BigQuery for incident warehousing, Cloud Run for containerised API deployment.
- Validation: Thresholds validated against 8.3M real vehicle observations from NGSIM I-80 (US Department of Transportation).
3. Tech Stack
YOLOv8s · PyTorch · OpenCV · ByteTrack · FastAPI · Docker · GCP (Cloud Run · Pub/Sub · BigQuery · Vertex AI · Artifact Registry) · Streamlit · Kalman Filter · Python
4. Results
- Detection accuracy: 0.833 mAP@0.5 — exceeds 0.80 production-viable threshold; +7.3 points over published UA-DETRAC YOLOv3 baseline.
- Incident detection: 1,553 incidents logged across test sequences; minimum TTC recorded: 0.09 seconds.
- Validation: Speed estimates and TTC thresholds confirmed against 8.3M NGSIM I-80 observations — 11.5% of real US highway vehicles operate at CRITICAL TTC levels in peak hour.
- Deployment: Live REST API on Google Cloud Run with interactive Swagger UI; Streamlit dashboard querying BigQuery in real time.
- Inference speed: Sub-second per frame on CPU; container cold start under 3 seconds on Cloud Run.
5. Architecture
Camera frames → YOLOv8s detection → ByteTrack tracking
→ TTC near-miss analysis → Behavior classification
→ GCP Pub/Sub (real-time) → BigQuery (warehouse)
→ FastAPI (Cloud Run) → Streamlit dashboard
6. Next Steps
- Lane segmentation: Add lane boundary detection to restrict TTC calculations to same-lane vehicle pairs, eliminating opposite-carriageway false positives on divided highways.
- Camera auto-calibration: Replace the fixed pixels-per-metre constant with automated calibration from known road markings, making the system portable to new camera installations without manual tuning.
- Edge deployment: Package detection and tracking for Nvidia Jetson deployment at roadside, removing cloud dependency for real-time alert latency.