robot-perception

Implements camera calibration, sensor streaming, and multi-sensor perception pipelines for robotics.

Updated Aug 16, 2026
One-click install
npx skills add https://github.com/three1324/yeonjinautomotive --skill robot-perception-three1324
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: robot-perception
Source: https://github.com/three1324/yeonjinautomotive/tree/main/.claude/skills/robot-perception
Command: npx skills add https://github.com/three1324/yeonjinautomotive --skill robot-perception-three1324

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires opencv, numpy, pyrealsense2.

What problem does it solve? Building reliable robot perception requires solving recurring hard problems: calibrating cameras and LiDARs, streaming sensor data without frame drops or latency, synchronizing multiple sensors, and turning 2D detections into usable 3D positions. This Skill provides production-tested patterns and reference implementations for each of these tasks. ## Core Features & Use Cases - Camera Calibration: Intrinsic, extrinsic (stereo and camera-to-LiDAR), and hand-eye calibration workflows using OpenCV, with quality checklists and reprojection error validation. - Sensor Streaming: Thread-safe capture architecture with bounded buffers, capture-time timestamping, and diagnostics, plus multi-sensor timestamp synchronization and hardware trigger setup (RealSense inter-cam sync, PTP for GigE). - Perception Pipelines: Image undistortion with precomputed maps, object detection with depth-based 3D back-projection and workspace filtering, and fiducial marker (AprilTag/ArUco) pose estimation. - Use Case: When integrating a RealSense or ZED camera into a ROS2 robot, use this Skill to calibrate intrinsics, set up a drop-free streaming thread, and convert detections into 3D points for manipulation or navigation. ## Quick Start Use the robot-perception skill to calibrate my camera intrinsics from checkerboard images and set up a synchronized RGB-depth streaming pipeline.

Frequently Asked Questions about robot-perception

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calibrate camera intrinsics with OpenCV?

Camera intrinsic calibration uses cv2.calibrateCamera with checkerboard corner detections from 20 or more images covering the full image area at varied tilts. Sub-pixel corner refinement via cv2.cornerSubPix is critical, and RMS reprojection error should be below 0.5 pixels.

How to synchronize multiple cameras and sensors in robotics?

Multi-sensor synchronization matches frames by timestamp within a tolerance such as 33ms, using capture-time timestamps rather than processing-time. For stereo or fast scenes, hardware triggers like RealSense inter-cam sync mode or PTP for GigE cameras achieve sub-millisecond accuracy.

What is hand-eye calibration and when do I need it?

Hand-eye calibration solves AX = XB to find the transform between a camera and a robot, either mounted on the end-effector (eye-in-hand) or fixed in the workspace (eye-to-hand). It requires at least 15 diverse robot poses with multiple rotation axes, solved via cv2.calibrateHandEye.

Does this approach work with RealSense, ZED, and OAK-D cameras?

Yes, the streaming and calibration patterns apply to RealSense (pyrealsense2), ZED (pyzed), and OAK-D (depthai), as well as USB webcams via OpenCV VideoCapture. Each maps to a ROS2 driver package such as realsense2_camera or zed_wrapper.

Why is my camera pipeline dropping frames or lagging?

Frame drops usually come from running capture and processing in the same thread, unbounded buffers, or timestamping at processing time. Decouple capture into a dedicated thread with a bounded deque buffer and always consume the latest frame.

How do I convert a 2D detection into a 3D robot position?

Back-projection combines the detection pixel with a median depth sample from a small patch around it, then applies the inverse pinhole model using the camera intrinsics. The result is a 3D point in the camera frame that can be filtered against workspace bounds.