Building a Retroreflective/Color Pipeline¶
To configure a retroreflective/color vision pipeline, you must first access the web interface at <http://IPADDRESS:5801> or http://10.te.am.11:5801 for FRC teams.
The “Tracking” page is comprised of five tuning tabs:
Input Tab¶
The Input Tab hosts controls to change the raw camera image before it is passed through the processing pipeline.
Pipeline Type¶
Controls the desired pipeline type. Change this option to utilize GRIP or Python Pipelines.
Source Image¶
Controls the source of the image that is passed through the pipeline. Switch to “Snapshot” to test your vision pipelines on stored Snapshots.
This control auto-resets to “Camera” when the GUI is closed.
Resolution + Zoom¶
Controls the resolution of the camera and vision pipeline. We recommend using the 320x240 pipeline unless you are utilizing 3D functionality.
320x240 pipelines execute at 90fps, while 960x720 pipelines execute at 22 fps. In 2020, 2x and 3x Hardware Zoom options were added to this field. The zoom options are not digital and use 100% real sensor pixels.
LEDs¶
Controls the default LED mode for this pipeline. This may be overridden during a match with the “LED” network table option.
Limelight 2+ users have access to an “LED Brightness” Slider which allows for LED dimming.
Orientation¶
Controls the orientation of incoming frames. Set it to “inverted” if your camera is mounted upside-down.
Exposure¶
Controls the camera’s exposure setting in .01 millisecond intervals. Think of a camera as a grid of light-collecting buckets - exposure time controls how long your camera’s “buckets” are open per frame. Lowering the exposure time will effectively darken your image. Low and fixed exposure times are crucial in FRC, as they black-out the bulk of incoming image data. Well-lit retroreflective tape will stand out in a mostly black image, turning vision processing into a straightforward process.
Black Level Offset¶
Increasing the black level offset can significantly darken your camera stream. This should be increased to further remove arena lights and bright spots from your image. This is a sensor-level setting, and not a fake digital brightness setting.
Red Balance, Blue Balance¶
Controls the intensity of Red and Blue color components in your image. These collecively control your Limelight’s white balance. We recommend leaving these at their default values of
Thresholding Tab¶
Thresholding is a critical component of most FRC vision tracking algorithms. It is the act of taking an image, and throwing away any pixels that aren’t in a specific color range. The result of thresholding is generally a one-dimensional image in which a pixel is either “on” or “off.
Video Feed (Located beneath stream)¶
Controls which image is streamed from the mjpeg server. You should switch to the “threshold” image if you need to tune your HSV thresholding.
Thresholding Wands¶
- Wands enable users to click on Limelights’s video stream to perform automatic HSV thresholding.
- The “Eyedropper” wand centers HSV parameters around the selected pixel
- The “Add” wand adjusts HSV parameters to include the selected pixel
- The “Subtract” wand adjust HSV parameters to ignore the selected pixel
Hue¶
Describes a “pure” color. A Hue of “0” describes pure red, and a hue of 1/3 (59 on the slider) describes pure green. Hue is useful because it doesn’t change as a pixel “brightens” or “darkens”. This is the most important parameter to tune. If you make your hue range as small as possible, you will have little if any trouble transitioning to an actual FRC field.
Saturation¶
Describes the extent to which a color is “pure”. Another way to think of this is how washed-out a color appears, that is, how much “white” is in a color. Low saturation means a color is almost white, and high saturation means a color is almost “pure”.
Value¶
Describes the darkness of a color, or how much “black” is in a color. A low value corresponds to a near-black color. You should absolutely increase the minimum value from zero, so that black pixels are not passed through the processing pipeline.
Erosion and Dilation¶
Erosion slightly erodes the result of an HSV threshold. This is useful if many objects are passing through a tuned HSV threshold. Dilation slightly inflates the result of an HSV threshold. Use this to patch holes in thresholding results.
Hue Inversion¶
Use hue inversion to track red targets, as red exists at both the beginning and the end of the hue range.
Contour Filtering¶
After thresholding, Limelight generates a list of contours. After that, each contour is wrapped in a bounding rectangle an unrotated rectangle, and a “convex hull”. These are passed through a series of filters to determine the “best” contour. If multiple contours pass through all filters, Limelight chooses the best contour using the “Sort Mode” Control.
Sort Mode¶
Controls how contours are sorted after they are passed through all other filters.
In 2019, the “closest” sort mode was added. This mode will select the target that is closest to the configurable crosshair.
Target Area¶
Controls the range of acceptable bounding-rectangle areas, as percentages of the screen. You can increase the minimum area to help filter-out stadium lights, and decrease the maximum value to help filter-out things like large displays near the field.
Note
The area slider is not linearly scaled, but quarticly scaled. This is done to provide extra precision near the lower-end of area values, where many FRC targets lie. The area of a square scales quadratically with its side length, but x^4 scaling provides even greater precision where it is needed.
Target Fullness¶
Fullness is the percentage of “on” pixels in the chosen contour’s bounding rectangle. A solid rectangle target will have a near-1.0 fullness, while a U-shaped target will have a low fullness.
Target Aspect Ratio¶
Aspect ratio is defined by the width of the bounding rectangle of the chosen contour divided by its height. A low aspect ratio describes a “tall” rectangle, while a high aspect ratio describes a “wide” rectangle.
Note
The aspect ratio slider is also quadratically scaled.
Smart Speckle Rejection¶
Rejects relatively small (as opposed to absolutely small w/ the area filter) contours that have passed through all other filters. This is essential if a target must remain trackable from short-range and long-range. This feature was introduced in the 2019 season to reject Limelight’s LED reflections when robots were very close to targets.
Target Grouping¶
Controls target “grouping”. Set to dual mode to look for “targets” that consist of two shapes, or tri mode to look for targets that consist of three shapes.
Smart Target Grouping can group a variable number of targets and reject outliers. It was added in 2022 to help track the upper hub target.
Intersection Filter (Dual Targets Only)¶
Rejects groups of contours based on how they would intersect if extended to infinity.
Smart Target Grouping¶
- Automatically group targets that pass all individual target filters.
- Will dynamically group any number of targets between -group size slider minimum- and -group size slider maximum-
- Outlier Rejection
- While group targets are more challenging than normal targets, they provide more information and opportunities for filtering. If you know that a goal is comprised of multiple targets that are close to each other, you can actually reject outlier targets that stand on their own.
- You should rely almost entirely on good target filtering, and only use outlier rejection if you see or expect spurious outliers in your camera stream. If you have poor standard target filtering, outlier detection could begin to work against you!
Output¶
This tab controls what happens during the last stage of the vision pipeline
Targeting Region¶
Controls the point of interest of the chosen contour’s bounding rectangle. By default, the tracking parameters tx and ty represent the offsets from your crosshair to the center of the chosen rectangle. You can use another option if a target changes in size, or is comprised of two targets that sometimes blend together.
Send Raw Corners?¶
Set this control to “yes” to submit raw corners over network tables. Tune the number of corners submitted by adjusting the “Contour Simplification” value in the “Contour Filtering” page.
Send Raw Contours?¶
Set this control to “yes” to submit raw contours over network tables. The top 3 passing contours will be submitted.
Crosshair Calibration¶
Controls the “origin” of your targeting values. Let’s say a shooter on your robot needs to be calibrated such that it always points a bit left-of-center. You can line up your robot, click “calibrate,” and all of your targeting values will be sent relative to your new crosshair. See the calibration page for more details!
3D¶
Experiment with PnP point-based pose estimation here.
Compute 3D¶
Controls whether pose estimation is enabled. You must enable the 960x720 high-res mode for this to work.
Force Convex¶
Use this option to select only the “outermost” corners of a target for SolvePnP.
Contour Simplification¶
Use this option to remove small, noisy edges from the target.
Acceptable Error¶
Limelight will only return a target if it passes a reprojection test with a certain score in pixels.
Camera Matricies (Advanced Users)¶
cameraMatrix = cv::Matx33d(
772.53876202, 0., 479.132337442,
0., 769.052151477, 359.143001808,
0., 0., 1.0);
distortionCoefficient = std::vector<double> {
2.9684613693070039e-01, -1.4380252254747885e+00,-2.2098421479494509e-03,
-3.3894563533907176e-03, 2.5344430354806740e+00};
focalLength = 2.9272781257541; //mm
cameraMatrix = cv::Matx33d(
8.8106888208290547e+02, 0., 4.8844767170376019e+02,
0., 8.7832357838726318e+02, 3.5819038625928994e+02,
0., 0., 1.);
distortionCoefficient = std::vector<double> {
1.3861168261860063e-01, -5.4784067711324946e-01,
-2.2878279907387667e-03, -3.8260257487769065e-04,
5.0520158005588123e-01 };
focalLength = 3.3385168390258093; //mm