E2E-Spot - Temporal Event Detection in Sports Videos

🎯 Project Overview

E2E-Spot is an end-to-end architecture for precise temporal event detection in sports videos. The system automatically detects key actions (passes, dribbles, shots) at frame-level in futsal matches.

Team: Hicham EL MEHDI & Mohammed Imrane GRICH
Supervised by: Prof. Noureddine MOHTARAM

Key Achievement: 93% recall for dribble detection with a lightweight architecture (10M parameters).

🔍 Problem Statement

Challenges

• ⏱️ Ultra-precise temporal detection at frame level (±1 second)
• 🎯 Contextual understanding to differentiate similar actions
• 🔄 Temporal relationships across long sequences
• 💰 Computational efficiency for fast video processing

Solution

End-to-end architecture combining:

• RegNet-Y: Efficient visual feature extraction
• Gate Shift Modules (GSM): Local temporal modeling
• Bidirectional GRU: Global temporal reasoning
• Frame-by-frame classification with background detection

🏗️ Technical Architecture

Figure 1: Complete E2E-Spot Pipeline

E2E-Spot Pipeline Architecture

--- ### 🔧 Key Components

Module F: Feature Extraction

RegNet-Y + Gate Shift Modules

• ⚡ Lightweight: 10M parameters (vs 25M ResNet-50)
• 🔄 GSM capture variations between adjacent frames
• 🚀 Ultra-efficient single-pass processing

Module G: Temporal Reasoning

Bidirectional GRU

• 📊 Models past + future context
• 🎯 Global temporal understanding
• 📈 Frame-wise classification with confidence scores

--- ### 📐 Architecture Flow

📦 Dataset & Preprocessing

Source: FIFA Futsal World Cup 2024
Volume: 40 GB video data
Content: 7 complete matches (14 halves)

Preprocessing Pipeline

1. Frame extraction at 30 FPS
2. Temporal normalization (timestamps → seconds)
3. Augmentation: Jittering, cropping, mixup
4. Split: 70% train / 15% val / 15% test

🚀 Training Configurations

Configuration 1: RegNet-Y (clip_len=50)

Model: rny008_gsm
Clip Length: 50 frames (long sequences)
Training Time: 8h 20min

Configuration 2: ResNet-50 (clip_len=16)

Model: rn50
Clip Length: 16 frames
Training Time: 300h (~12.5 days)

Configuration 3: RegNet-Y (clip_len=16) ⭐

Model: rny008_gsm
Clip Length: 16 frames (best tradeoff)
Training Time: 450h (~18.7 days)

📊 Results & Performance

Best Configuration: RegNet-Y (clip_len=16)

Key Insights

• 🏆 93% Recall for DRIVE: Detects almost all dribbles
• 🎯 Good PASS precision (0.76) but low recall (class imbalance)
• ⚡ Lightweight architecture: 2.5x fewer parameters than ResNet-50
• 📈 Best tradeoff performance/efficiency

Benchmark Comparison

🛠️ Challenges & Solutions

Challenge 1: Class Imbalance

Problem: PASS class underrepresented → low recall
Solution: Weighted Cross-Entropy Loss, PASS oversampling

Challenge 2: Training Time

Problem: 450h for optimal config
Solution: Checkpointing, Mixed Precision (FP16), Gradient Accumulation

Challenge 3: Temporal Capture

Problem: clip_len=50 too long, clip_len=16 sometimes insufficient
Future Solution: Multi-scale temporal fusion (8/16/32 frames)

🔮 Future Improvements

Short-Term:

• Focal Loss for class imbalance
• Multi-scale temporal fusion
• Temporal NMS post-processing

Long-Term:

• Transformer architecture for very long-term relationships
• Real-time inference (<100ms latency)
• Multi-sport generalization (basketball, handball)

💼 Impact & Value

For Sports Analytics

✅ 70% reduction in manual annotation time
✅ Frame-precise detection (±1 second)
✅ Lightweight architecture = reduced infrastructure costs

For Coaches & Media

✅ Fast post-match analysis
✅ Automatic highlight generation
✅ Video indexing for event search

🛠️ Technical Stack

Deep Learning: PyTorch, TorchVision, Lightning
Computer Vision: OpenCV, Albumentations
Models: RegNet-Y, Gate Shift Modules, GRU
Tools: Weights & Biases, TensorBoard
Hardware: NVIDIA RTX 4060, AMD Ryzen 7 5700X

🎓 Skills Developed

• Deep Learning: End-to-end architecture, temporal modeling
• Computer Vision: Video understanding, action spotting
• Model Optimization: Hyperparameter tuning, loss functions
• Research: Experimentation, benchmarking, technical writing

📧 Contact

Author: Hicham El Mehdi
Email: mehdihicham736@gmail.com
LinkedIn: linkedin.com/in/elmehdihicham
GitHub: github.com/MehdiHCH

Co-Author: Mohammed Imrane GRICH
Supervisor: Prof. Noureddine MOHTARAM

📚 Key References

1. RegNet (Radosavovic et al., 2020) - Designing Network Design Spaces
2. Gate Shift Modules (Sudhakaran et al., 2020) - CVPR 2020
3. E2E-Spot (Giancola et al., 2022) - End-to-End Spotting in Soccer

Compact and efficient end-to-end architecture for temporal event detection in sports videos, achieving 93% recall with only 10M parameters.