RoboRangers - Autonomous Vehicle with Deep Reinforcement Learning

June 2024 Robotics Academic Project Ibn Tofail University, Kenitra February 2024 - June 2024
RoboRangers - Autonomous Vehicle with Deep Reinforcement Learning
Autonomous Driving ROS2 Gazebo Deep Reinforcement Learning TD3 LIDAR Computer Vision Robotics
View on GitHub

🎯 Project Overview

RoboRangers is an autonomous vehicle project using Deep Reinforcement Learning (DRL) and ROS2 for intelligent navigation in simulated environments. The system integrates LIDAR sensors and cameras for complete perception, and implements the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm for autonomous decision-making.

Team:

  • ALAMI AROUSSI Zineb
  • GRICHE MOHAMMED Imrane
  • HICHAM EL Mehdi
  • MOUTIA Salma

Under the supervision of: Pr. BOUKIR Khaoula

Mission: Develop an autonomous vehicle capable of navigating, avoiding obstacles and respecting traffic signs in a dynamic environment.

🔍 Problem Statement

Challenges

Problem: How to enable a system to learn real-time decision-making based on changes in its environment while ensuring safety and efficiency?

  • 🚗 Traffic accidents: Human errors = main cause of death
  • 🚦 Traffic jams: Economic losses and pollution
  • 👥 Accessibility: Limited mobility for elderly/disabled people
  • 🌍 Environmental impact: High CO2 emissions
  • 🎯 Complex navigation: Dynamic and unpredictable environments

Solution

Autonomous vehicle integrating:

  • ROS2: Distributed inter-module communication
  • Gazebo: Realistic 3D simulation for safe testing
  • DRL (TD3): Autonomous learning through trial-and-error
  • Multi-modal sensors: LIDAR + Cameras for 360° perception

🏗️ Technical Architecture

🔧 System Components

Sensors:

  • 📹 RGB Cameras: Obstacle detection, signs
  • 🔵 Velodyne LIDAR: 360° distance measurement
  • 📊 Point Cloud Processing: Real-time 3D data
  • 🎯 Sensor Fusion: Robust perception

Intelligence:

  • 🧠 TD3 Agent: Reinforcement learning
  • 🎮 Action Selection: Steering, acceleration, braking
  • 📈 Reward Function: Safety + efficiency
  • 🔄 Continuous Learning: Iterative improvement

Sensors Integration

Gate Shift Modules

Camera

Camera

📐 ROS2 Architecture

ROS2 Architecture - RoboRangers Sensor Layer 🎥 Camera Node /camera/image_raw 📡 LIDAR Node /scan (720 points) 📍 Localization /odom, /amcl_pose 🧭 IMU Node /imu/data Perception Layer 🚧 Obstacle Detection LIDAR + Vision 🚦 Sign Detection CNN Classification 🔀 Sensor Fusion Node Multi-modal integration (LIDAR + Camera + IMU) Decision Layer 🧠 TD3 Agent (Deep RL) State → Actor-Critic Networks → Actions 🗺️ Path Planning Node Global: A* | Local: DWA (Dynamic Window Approach) Control Layer 🎮 Vehicle Control Node → /cmd_vel (linear, angular velocity)

Key Topics:

  • /camera/image_raw → RGB images
  • /scan → LIDAR data (720 points)
  • /odom → Odometry
  • /cmd_vel → Velocity commands
  • /obstacle_detected → Obstacle alerts
  • /traffic_sign → Detected signs

🎓 TD3 Algorithm Explained

Twin Delayed DDPG (TD3)

TD3 Algorithm Architecture Twin Delayed Deep Deterministic Policy Gradient for Continuous Control 🌍 Environment Gazebo Simulation State (s) • LIDAR (180 pts) • Position (x,y,θ) • Goal distance 🎭 Actor Network π(s) FC1(256) → ReLU FC2(256) → ReLU FC3(2) → Tanh Action (a) • Linear vel • Angular vel Range: [-1, 1] 🎯🎯 Twin Critic Networks Q₁(s,a) & Q₂(s,a) Critic 1: FC(256)→ReLU→FC(256)→ReLU→FC(1) Critic 2: FC(256)→ReLU→FC(256)→ReLU→FC(1) Output: Q-values (expected cumulative reward) 💾 Replay Buffer Store: (s, a, r, s', done) | Size: 1M 🎯 Target Networks π'(s) - Target Actor Q₁'(s,a) - Target Critic 1 Q₂'(s,a) - Target Critic 2 Soft update: τ = 0.005 🔑 TD3 Key Innovations vs DDPG 1️⃣ Twin Critics: Use min(Q₁, Q₂) to reduce overestimation bias More stable and accurate Q-value estimates 2️⃣ Delayed Policy Updates: Update actor every 2 critic updates Reduces policy errors from bad Q-values 3️⃣ Target Policy Smoothing: Add noise to target actions Smooths Q-function for robustness Result: More stable training, better sample efficiency, superior performance in continuous control tasks

TD3 Components:

  • 🎭 Actor: Learns optimal policy π(s)
  • 🎯 Twin Critics: Q₁(s,a) & Q₂(s,a) for stable learning
  • 💾 Replay Buffer: Experience replay for sample efficiency
  • 🔄 Target Networks: Slowly updated for stability

Key Features:

  • ✅ Continuous action space (steering, velocity)
  • ✅ Reduced overestimation bias
  • ✅ Delayed policy updates
  • ✅ Target policy smoothing

🎯 Reward Function Design

Reward Function Components Multi-objective reward shaping for safe and efficient navigation 1️⃣ Progress Reward (R_progress) Formula: R_progress = -Δd × 10 where Δd = d(t) - d(t-1) Weight: w₁ = 10.0 Rewards: • Moving closer to goal 2️⃣ Collision Penalty (R_collision) Trigger: if min(LIDAR) < 0.3m: reward = -100 Weight: w₂ = 1.0 Penalties: • Collision with obstacles 3️⃣ Trajectory Penalty (R_trajectory) Formula: R_trajectory = -|heading_error| × 0.5 heading_error = θ_goal - θ_robot Weight: w₃ = 0.5 Penalties: • Deviating from optimal path 4️⃣ Goal Reward (R_goal) Formula: R_goal = +200 Trigger: if distance < 0.5m: reward = +100 Weight: w₄ = 1.0 Rewards: • Task completion • Successful navigation • Episode success Typical Episode Reward Range: -100 (collision) to +100 (goal reached) | Average: +5 to +20 per successful step

Components:

  • ✅ Progress: Reward for advancement towards goal
  • ❌ Collisions: High penalty (-100)
  • 📏 Trajectory: Penalty for deviation
  • 🎯 Goal reached: Large bonus (+100)

🛠️ Implementation Details

Project Structure

RoboRangers/
├── src/
│   └── td3/
│       ├── models/              # URDF robot descriptions
│       │   └── velodyne_description/
│       ├── launch/              # ROS2 launch files
│       │   └── launch_sim.launch.py
│       ├── scripts/             # Python implementations
│       │   ├── train_velodyne_node.py
│       │   ├── test_velodyne_node.py
│       │   ├── replay_buffer.py
│       │   └── td3_algorithm.py
│       ├── pytorch_models/      # Trained models
│       │   ├── td3_velodyne_actor.pth
│       │   └── td3_velodyne_critic.pth
│       └── worlds/              # Gazebo simulation worlds
└── package.xml

Training Configuration

TD3 Hyperparameters:

config = {
    'actor_lr': 3e-4,
    'critic_lr': 3e-4,
    'gamma': 0.99,           # Discount factor
    'tau': 0.005,            # Target network update rate
    'policy_noise': 0.2,
    'noise_clip': 0.5,
    'policy_delay': 2,       # Actor update frequency
    'buffer_size': 1e6,
    'batch_size': 256
}

Training Process:

  • Episodes: 1000+
  • Max steps per episode: 500
  • Training time: ~48h on RTX 4060

📊 Functionalities Achieved

✅ Core Features

Functionality Status Description
Autonomous Navigation ✅ Predefined trajectory following
Obstacle Detection ✅ LIDAR + Cameras in real-time
Collision Avoidance ✅ Dynamic reaction to obstacles
Sign Recognition 🟡 Basic detection implemented
DRL Integration 🟡 TD3 partially integrated

🎥 Visual Results

LIDAR Visualization:

  • Blue lines: LIDAR rays
  • Red center: Sensor position
  • Real-time obstacle detection

🛠️ Challenges & Solutions

Challenge 1: DRL Integration

Problem: Environmental difficulties ROS2-Gazebo-TD3
Status: Partially resolved
Applied solution:

  • Module separation (perception / decision / control)
  • Independent unit tests
  • Fallback to simple reactive control

Challenge 2: LIDAR Processing

Problem: High data volume (720 points @ 10Hz)
Solution:

  • Intelligent downsampling (720→180 points)
  • Distance filtering (ROI: 0.5m - 10m)
  • GPU processing for point cloud

Challenge 3: Sim-to-Real Gap

Problem: Simulation ≠ Reality (physics, sensors)
Future solution:

  • Domain randomization (friction, lighting)
  • Sensor noise injection
  • Sim2Real transfer learning

💼 Impact & Value

For Transportation

✅ Accident reduction: -90% human errors
✅ Traffic fluidity: Trajectory optimization
✅ Accessibility: Mobility for all
✅ Environment: -30% CO2 emissions

For Research

✅ Open-source platform for DRL robotics
✅ Benchmark ROS2 + Gazebo + TD3
✅ Reproducibility via simulation
✅ Extensibility to other mobile robots

🔮 Future Improvements

Short-Term:

  • Complete TD3 integration
  • Improve sign recognition (CNN)
  • Multi-agent scenarios (multiple vehicles)

Long-Term:

  • Deployment on physical robot (TurtleBot3)
  • Real-world testing with real sensors
  • V2V Communication (Vehicle-to-Vehicle)
  • End-to-end learning (pixels → actions)

🛠️ Technical Stack

Robotics: ROS2 Humble, Gazebo 11
Deep Learning: PyTorch, Stable-Baselines3
Sensors: Velodyne LIDAR, RGB Cameras
Languages: Python 3.10, C++
Tools: Rviz, rqt, TensorBoard
Hardware: NVIDIA RTX 4060, AMD Ryzen 7

🎓 Skills Developed

  • ROS2 Mastery: Nodes, Topics, Services, Launch files
  • Reinforcement Learning: TD3, DDPG, Actor-Critic
  • Sensor Fusion: LIDAR + Camera integration
  • Robotics Simulation: Gazebo worlds, URDF modeling
  • Autonomous Systems: Path planning, obstacle avoidance

📚 Key References

  1. TD3 Algorithm (Fujimoto et al., 2018) - ICML
  2. ROS2 Documentation - Open Robotics
  3. Gazebo Simulator - OSRF
  4. Deep Reinforcement Learning (Sutton & Barto, 2018)

📧 Contact

Team Lead: El Mehdi Hicham
Email: mehdihicham736@gmail.com
GitHub: github.com/MEHDI57-NRG/RoboRangers
LinkedIn: linkedin.com/in/elmehdihicham

Team Members:

  • ALAMI AROUSSI Zineb
  • GRICHE MOHAMMED Imrane
  • MOUTIA Salma

Supervisor: Pr. BOUKIR Khaoula
Institution: Faculty of Sciences, Ibn Tofail University

🏆 Achievements

✅ Autonomous navigation functional in Gazebo
✅ Obstacle detection real-time (LIDAR + Cameras)
✅ ROS2 architecture modular and extensible
✅ TD3 Agent partially integrated
✅ Open-source on GitHub for the community

Autonomous driving system combining modern robotics (ROS2), realistic simulation (Gazebo) and artificial intelligence (Deep RL) for safe and efficient mobility.