We will organize what kind of perceptions are performed by mobile robots and what kind of sensors are used for them. Among mobile robots, we will focus on “automated vehicles” and “automatic guided vehicles (AMR)”, which are commonly known.
Sensors for Autonomous Driving Cars
Sensors for autonomous driving cars can be broadly divided into those for perceiving the external environment of the vehicle and those for perceiving the internal state of the vehicle. In this section, we will discuss the sensors for perceiving the external environment, which are often discussed in information sources on autonomous driving. In this section, we will summarize the sensors that perceive obstacles, which are particularly important in automated driving.
The information about obstacles that we want in order to decide how to proceed (move) the car is the presence or absence and the position of the obstacle. If we know the position of the obstacle, we know whether or not there is an obstacle, so a sensor to perceive the position is enough. We also want to know the speed, if possible. If we know the speed, we can predict the future position of the obstacle and decide how to move the car. The speed of the obstacle can be obtained by acquiring positional information chronologically and differentiating the positions before and after the obstacle (differentiating positional information).
There are several possible ways to obtain location information. If we can directly obtain the coordinate positions of our vehicle and the obstacle on the map, we may be able to calculate how our vehicle should avoid the obstacle. The position information of the car can be obtained by using GNSS such as GPS (for now, accuracy aside). Obstacle location information can be obtained if, for example, there are sensors on city streets that detect obstacles and communication devices that transmit the information to the car. However, at the time of writing (2020), such facilities are unfortunately not in place. Such facilities are called Intelligent Transport Systems (ITS), and research is underway to promote their widespread use.
The method of choice for automatic driving or driver assistance systems is to measure (perceive) the “distance” between the car and the obstacle, and recognize the relative position of the car and the obstacle. Measuring the distance of obstacles is called “ranging,” and the sensor for this purpose is called a “ranging sensor. There are several rangefinding sensors that are installed or being considered for installation in automobiles, and the four typical types are stereo cameras, millimeter wave radar, LiDAR, and ultrasonic sensors (sonar).
Stereo Camera
A stereo camera is a combination of two (or more) cameras, and is widely known to be used in Subaru’s “EyeSight” system. Normally, two cameras are arranged horizontally on the left and right sides with a certain distance between them; if there is a distance between the two cameras, the positional relationship between each camera and the obstacle will change, and the images captured by the cameras (images) will be slightly different. This is called “parallax”. By performing a process called “stereo matching” on the parallax information, it is possible to estimate the distance of the object in the image. It is roughly the same principle as the human eye. Stereo cameras for automobiles are those that are compatible with visible light. Also, stereo cameras for automobiles are capable of measuring the distance of obstacles up to about 150 meters away. For more information about stereo cameras, please see the following page.
Millimeter-wave radar
Millimeter-wave radar is a sensor that estimates the distance to an obstacle by emitting millimeter-wave radio waves at the obstacle and detecting and analyzing the reflected radio waves back to the obstacle. It has been adopted in several mass-produced cars equipped with automatic driving and driver assistance systems, and became popular mainly in luxury cars between 2015 and 2020. The name “millimeter wave” comes from “radio waves with a wavelength of a few millimeters. The name “millimeter wave” comes from “radio waves with a wavelength of a few millimeters.” Millimeter wave radar can measure the distance of obstacles up to 200 meters away. The principle is a little too complicated to explain in one page, so it will be described in another page.
LiDAR
LiDAR stands for “Light Detection and Ranging”. As the name suggests, it is a sensor that “measures distance by detecting light,” but to be more precise, the sensor itself emits light to illuminate obstacles and detects the light reflected back. It is similar to millimeter wave radar, but the principle is simpler. Using the fact that the speed of light (light velocity) is finite (not infinite, although it is very large), the system estimates the distance to the obstacle by measuring the time it takes for the light to hit the obstacle and return. The closer the obstacle, the shorter the return time, and the farther the obstacle, the longer the return time. This method of measuring distance is called the “Time of Flight” (TOF) method.
With the above mechanism alone, the distance of an obstacle can only be determined in one direction where the light is irradiated. In order to measure distances in multiple directions, LiDAR uses a mechanism where the “laser” that emits the light is rotated horizontally while continuously emitting and detecting light. For example, if you measure every 1° while rotating 360°, you can get distance data in 1° increments (1° angular resolution) for all directions around you. Since it is not possible to recognize the height of obstacles with this system, most LiDARs for automobiles have a system in which multiple structures that irradiate and detect light are arranged vertically and rotated simultaneously. If 10 structures are lined up vertically on a sphere in 1° increments and rotated 360°, 10 distance data for 360° in the horizontal direction can be obtained in 1° increments. By arranging this distance data in the form of pixels, it is possible to create (albeit coarse) two-dimensional image information of distance.
LiDAR comes in a variety of configurations, but the most common are those in which both the laser and the optical sensor (photo detector) or one of them is physically rotated by a motor. As of 2020, LiDAR is only used in a limited number of luxury cars such as Toyota’s Lexus. As of 2020, only a limited number of high-end cars, such as Toyota’s Lexus, will use LiDAR. Infrared light with a wavelength of around 1000nm is often used for LiDAR.
For more information about LiDAR, please see the following page.
Ultrasonic sensor (Sonar)
The ultrasonic sensor (sonar) is a sensor that outputs ultrasonic waves (high frequency sound waves) and measures the time it takes for the waves to return to the obstacle, and estimates the distance to the obstacle. It is a TOF method similar to LiDAR. The ultrasonic sensor measures the time it takes for sound waves to return. The speed of light (the speed of light) is 300,000 km/second, and the speed of sound (the speed of sound) is 340 m/second (0.34 km/second), which means that sound waves are a million times slower than light. This means that sound waves are a million times slower than light. The slower the speed, the longer the time it takes for a sound wave to return, making it easier to measure. For these reasons, ultrasonic sensors are compact and inexpensive to manufacture. On the other hand, their weakness is that the maximum distance they can measure is only a few meters. In mass-produced vehicles, ultrasonic sensors are actively used in directions that cannot be covered by stereo cameras, millimeter wave radar, and LiDAR. For more information on ultrasonic sensors, please refer to the following page.
Comparison of stereo camera, millimeter wave radar, and LiDAR
The three most common obstacle sensors used in vehicles traveling at normal speeds of several tens of kilometers per hour are stereo cameras, millimeter wave radar, and LiDAR. All of them are capable of detecting obstacles at a distance of 100 meters or more. Although there is a gap in the number of actual sensors used in mass-produced vehicles, a decision has not yet been reached on which sensor is better or what combination of sensors is best. Stereo cameras can acquire images of visible light as well as distance, since each camera can also function as a regular camera. It can recognize the color and pattern of obstacles, which is advantageous for recognizing the type of obstacle. However, the disadvantage is that the ability to detect obstacles is reduced at night or in bad weather (rain, snow, fog). Since LiDAR uses infrared light, which has a wavelength close to that of visible light, it also has the disadvantage of poor detection performance in bad weather. (This is not a problem at night.) On the other hand, the performance of millimeter wave radar does not degrade much at night or in bad weather. On the other hand, millimeter-wave radar does not degrade much at night or in bad weather. However, it is difficult for millimeter-wave radar to recognize the shape of obstacles. Another weakness of millimeter-wave radar is that it cannot detect objects composed of materials with low reflectivity. (Styrofoam and cardboard are often cited as examples of materials with low reflectivity.
Another disadvantage of stereo cameras is that their distance measurement performance is easily affected by the color and pattern of obstacles. Stereo cameras measure distance from the difference between the images of the left and right cameras (parallax). If a nearby obstacle and a distant obstacle appear to overlap, and the two obstacles are both the same color and the boundary between them is difficult to determine, the images seen by the left and right cameras will appear the same. As a result, the parallax is not clear and the distance cannot be detected properly. This is not the case with millimeter wave radar and LiDAR. However, millimeter wave radar and LiDAR have the disadvantage of being expensive and large in size, partly because they are relatively new sensors.
As mentioned above, the advantages and disadvantages of each sensor are mixed up, and the reality is that we have not settled on which one is better.
Sensors for Autonomous Mobile Robot (AMR)
A robot that transports parts and products in a factory or warehouse is called an AMR (Autonomous Mobile Robot). There is also a device with similar functions called an AGV (Automatic Guided Vehicle), but here we will focus on the sensors of the AMR. (The comparison between AGVs and AMRs is organized on a separate page.
AMRs are similar to self-driving cars and share many of the same key sensors. Even in factories and warehouses, there are obstacles for robots, such as shelves, walls, humans, cardboard, etc. AMRs need to be able to recognize the location of obstacles and stop or drive around them before colliding with them.
On the other hand, the difference with automated vehicles is that it is difficult to prepare a map of possible driving routes. Maps exist for general roads, and self-driving cars can use the map information to determine their driving routes. Recently, efforts have been made to create map information for self-driving cars that is more accurate than car navigation systems. In factories and warehouses, map information is not available unless the business creates it on its own. The layouts in factories and warehouses are often changed, and it is time-consuming to create map information for each change.
Also, being indoors, it is not possible to use GNSS systems such as GPS like cars, and it is not easy to recognize where your car is in the map. There are technologies to build GNSS-like systems for indoor use, but they are not easy and costly.
AMR has the ability to perceive its own position in relation to obstacles and to create its own map information from the detected obstacle information. This technology of simultaneously estimating one’s own position and creating map information is called “SLAM (Simultaneous Localization and Mapping)”. The major feature of AMR is that it has the SLAM function.
The sensor used in AMR to realize SLAM is a “laser scanner”. In AMR, it is called a laser scanner, but the content is the same as “LiDAR”. The basic principle is the same as that for self-driving cars, but the maximum distance at which obstacles can be detected is relatively short, a few tens of meters, which makes it more compact and cheaper to manufacture.
The AMR detects obstacles, moves forward to avoid hitting them, and then detects them again. The AMR is further equipped with an accelerometer and a gyroscope (angular rate sensor), and the acceleration and angular rate information can be used to determine the AMR’s own movement history. By integrating this information with the distance to obstacles measured at each point, it is possible to create map information. The AMR uses the map it has created to calculate and determine the path it will take to a given destination.
Other AMRs are equipped with ultrasonic sensors to detect obstacles in directions that are difficult to detect with laser scanners. They may also have a “bumper sensor” to detect that they have hit an obstacle. The bumper sensor is a spring-loaded sensor that shrinks when it hits an obstacle and detects the collision with the obstacle by detecting the shrinkage, and is sometimes used in cleaning robots such as iRobot’s Roomba.