Accuracy, Resolution, and Precision
What do they Mean and Why do they Matter?
The subsea industry as a whole suffers from a lack of clarity around the performance of the measurement equipment used to provide critical subsea operations. While the resolution of equipment is relatively easily understood and some accuracy information can be determined, it is the combination of a systems accuracy and precision that results in a high-quality measurement system. It is this overall combined performance that allows you to confidently predict performance on future projects.
2G Robotics is now providing an accuracy reports with every one of our ULS-500 PRO (20m Range) systems that are shipped. An example report is linked at the end of this article.
It is the combination of accuracy and precision that results in a high-quality measurement system. A system that is highly accuracy can still miss the target if the resolution of the system is poor and the sensor can’t detect and display measurements that are on the target, which would be an example using a sonar sensor instead of a laser scanner. In the graphic above the average location of each of the measurements is very good, however, due to the poor resolution of the system, the spread or noise of the data is high. This limits the measurement capability of the system, particularly for measuring fine details. Laser data that is captured with an accurate but low-resolution solution will look great from far but noise in the data will become more apparent when moving in close on features.
A precise but inaccurate laser system is able to measure fine details and show great looking imagery but it will struggle to deliver measurements which reflect the actual positions of the targets. Since the system is precise, there will be little noise and points that should be close to each other will be. However, as the sensor looks to other areas it’s inaccuracy will lead to drift in the measurements relative to the actual positions of the objects – this will result in incorrect results over longer measurement ranges.
Ultimately, a system that is accurate and has sufficient resolution to be precise will provide the desired results. Such a system is not only able to crisply see details but it has measurement accuracy over a range of measurements from the system.
The Importance of Range Resolution
The precision of a system is highly related to the resolution of the system. Resolution can also often mean the density of the data on the target. This article specifically focuses on range resolution rather than point / measurement density as it is range resolution that impacts the noisiness of the date.
There are two main styles of underwater laser scanners currently on the market. 2G Robotics manufacturers a laser scanner that is based on a trigonometric approach. One of the main benefits of this system is the high resolution that is achievable at close ranges. The resolution of this style of system is variable based on the range to the target.
The second main type of Underwater Laser Scanner is a time of flight system, where a pulse of light is sent out and, based on the time for the light to return and the speed of light, the distance to the target is calculated. The primary benefit of this system is the long range (though it is still constrained by water clarity). This style of system has a constant range resolution. Consequently, at close ranges, a trigonometric system will have greater resolution than a time of flight system. As the distance to the target increases there comes a point where the range resolution of the trigonometric approach is equivalent to the range resolution of the time of flight approach. For example, a time of flight system with a 5mm range resolution has superior resolution over the 2G Robotics ULS-500 PRO beyond ranges of approximately 15m to 20m. At shorter ranges than this, the 2G Robotics trigonometric system has superior resolution.
Operationally, for large area mapping, dynamic scanning – where the ROV can be maneuvered to be consistently within 10m of the structures being scanned – removes constraints typically associated with scan range and provides extremely high-resolution point clouds, while saving cost and time over static laser scanning.
What makes Underwater Laser Scanners Accurate
To have an accurate sensor, an accurate mathematical model must exist for the sensor to use between what the sensor actually is measuring and what the user is interested in. For example, laser scanners are directly measuring the light signal returning from a target surface. However, the user is interested in the range to the target and the geometry of the environment. By accurately understanding the physical behaviour of the light between the sensor and the target, the laser scanner is able to accurately determine what the user is interested in. This model can include things specific to the laser scanner itself such as the subtle geometry variances between each unique system and environmental variances such as changes in the index of refraction and the speed of light through the water.
Therefore, the robustness of the calibration procedure is crucial to ensure an accurate system. 2G Robotics uses unique techniques which allow each system to be individually calibrated and maintain this calibration in a range of operating environments. From each unique calibration, a calibration file is generated and loaded onto the system. The best way to test the efficacy of the calibration is to verify it. Therefore, upon completion of the calibration, a calibration verification stage is performed to confirm the accuracy of the system. These 2G Robotics accuracy reports are provided to the customer along with their system.
2G Robotics Accuracy Reports
2G Robotics feels it is important when purchasing or renting laser scanning equipment for use subsea to have confidence in the accuracy of the system. In the industry single use studies will often be carried out to test the performance of a system and this study or use case is then universalized, extrapolating the accuracy in this specific case to be the overall stated accuracy for the system. To address this, 2G Robotics is taking the step of performing a calibration verification for every system it creates, over the entire field of view of the system to ensure our customers not only get crisp, high-resolution scans but highly accurate scans also.
Using our in-house test tank, spheres are accurately positioned within the tank as a baseline and scanned with our underwater laser scanner. Based on the point clouds of each target, a sphere is then digitally fit to this cloud as the center of the sphere can be used for comparisons. Multiple sphere locations are used for each report and laser data is compared with the known baselines for each. The report presents both the accuracy and the precision of the measurements.
Click Here for an example accuracy report from a recently built 2G Robotics ULS-500 PRO.
Understanding the Accuracy Report
The accuracy report has 3 main tables:
Absolute Position Difference
The absolute position difference indicates how accurately the measurements reflect the real world positions by showing the distance between the known baseline measurement and the value obtained from the laser scan. It is the standard deviation of this measurement that is of most relevance. Specifically, for the example report linked to, this ULS-500 PRO measures within 3.96mm 68% of the time (1 standard deviation).
Relative Position Difference
The relative position difference reflects the accuracy of measurements within the point cloud. Since users are most often measuring between items in a scan rather than relative to the laser scanner itself, a table of relative errors between spheres is provided. By cumulatively looking at all of these measurements, summary results are provided showing no elongation or contraction bias as the mean difference is negligible and the standard deviation is maintained at approximately 4mm.
Sphere Fit Residuals
The sphere fit residuals indicate how much noise is inherent to the scanner. At each location, the best fit sphere we have generated is compared to the points in the cloud that define that sphere. How well these points make a sphere is measured with the fit residuals. This laser scanner when measuring the 82.5mm spheres has an average error of 0.22mm radius error and an average standard deviation for the points relative to the surface of the sphere of 1.08mm
For further explanation of the differences and combined importance of accuracy and resolution or for further explanation of our Accuracy Reports please reach out to the 2G Team at firstname.lastname@example.org