Explore the repository for a cutting-edge project aimed at developing a SLAM-based navigation system using vision-language data inputs. The project integrates natural language vocal instructions and image feeds to guide a differential drive robot equipped with a Kinect V2 sensor through dynamic environments.
- URDF Model: Custom differential drive robot with Kinect V2 sensor.
- ROS2 Control: Differential drive and position controllers.
- ROS SLAM Toolbox: Utilizes Xbox Kinect sensor data for effective mapping.
- Teleoperation: Control via Xbox game controller with seamless multiplexer support.
- Frontend: Next.js interface for visual and natural language input.
- Multi-model approach integrating NLP and computer vision.
- NLP with vocal data using OpenAI Whisper and BLIP for image feeds.
- Mistral 7B large language model quantized to int8 for robot action prediction.
- Utilizes Huggingface Transformers and CTransformers for local deployment.
- AMCL for localization with prerecorded environment maps.
- Nav2 stack for precise navigation to goal destinations.
- ROS 2 control for armature actions.
Acknowledging the support of ROS 2 Iron Irwini, Ubuntu 22.04 Jammy Jellyfish, Gazebo, HuggingFace Transformers, Salesforce, Whisper, Nav2, and Mistral AI for making this project possible.
Check out the GitHub repository for more details and code implementation.
Stay tuned for LUNA, the broader real-world implementation of this SLAM project!
🔗 Video explanation: Explore Here
#Robotics #AI #ROS2 #MachineLearning #SLAM #VLN #Innovation #TechProjects