Generating Road Network Graph with Vision-Based Unmanned Vehicle

With the advancement of technology and its cheapness, robotic vehicles have gained a large number of applications. The spread of their use is growing also because they are getting smaller, lighter and easier to build. In this paper we present a simple and effective way to map a road network with the help of a driverless vehicle. Our approach consists of only three parts: vision-segmentation, angle variation and travelled distance. A video camera attached to a Lego® NXT Mindstorm vehicle guides it by image segmentation using Matlab® Image processing toolbox, along a road network, in which is represented by black tape over a white floor. The algorithm makes the vehicle travel all over the road memorizing main coordinates to identify all crossroads by keeping track of the travelled distance and the current angle. The crossroads and road’s end are the nodes of the graph. After several simulations have been performed, the modelling proved to be successful in that small scale approach. Consequently, there are good chances that driverless cars and UAVs also make use of the strategies to map route networks accordingly. The algorithm presented in this paper is useful when there is no localization signal such as GPS, for example, navigation on water, tunnels, inside buildings, among others.


INTRODUCTION
Vision-based systems consist of a sequence of digital images that enable the detection and the identification of objects. Iris and face recognition are examples of vision-based systems applied to security and private access control.
Many other applications can be found in traffic control, search and rescue, border security and aerial surveillance. These tasks are already accomplished by vision-based robots, which are nowadays built with inexpensive equipment and have an easy maintenance. Several techniques to endow vehicles with vision-based control can be found in (Güzel, 2013;Bertozzi & Broggi, 1997;Tisdale, et al., 2009;Carloni, et al., 2013;Yunji, et al., 2013;Ivancsits & Lee, 2013;Lim, et al., 2012;Mahony, et al., 2012;Guillet, et al., 2014;Mcgill & Taylor, 2011). They enable object detection, path orientation and distance control. Vision-based control makes signal reception more robust than other vision guidance methods do. For instance, GPS (Global Positioning System) receives satellite signals with a considerable margin of error, besides the fact that such signals are sometimes blocked, for example, under the water, inside tunnels or inside buildings.
The first mobile robot was built in the 70s in Stanford Research Institute. Another example of mobile robot, now with a video camera for navigation aid, was developed in the 80s at Carnegie Mellon University (Güzel, 2013).
With advanced cameras one can also plot 3D maps. In (Eisenbeiss, 2004) the author shows an example of photogrammetry, in which a helicopter was used to obtain some data from a settlement of the 13th century A.D. in Peru. By flying automatically according to predefined path points these data produced a 3D image of that settlement.
In (Sibley, et al., 2010) it is possible to see a study of robot location on Mars, with an approach of sliding window filter. In (Newcombe, et al., 2011) it provides a very detailed approach to mapping techniques using a Kinect sensor for mapping of the indoor scenes.
In this study it will be performed simulations with an indoor model, where one video camera attached to a robot guides it by image segmentation along a road network, represented by black tape over a white floor. Hence either GPS or any other outdoor equipment cannot be included in our tests. Therefore, we resort to mapping techniques like orientation by angle and travelled distance.

MATERIALS AND METHODS
The main goal of this project is the elaboration of a strategy to automate the mapping of road networks modelled by black tape over a white floor (Figure 1). Automation is done by an unmanned vehicle that travels across the network and is guided only through image measurements. In this case will be used the LEGO NXT Mindstorm vehicle (Figure 1) that drives through the path according to one Matlab® program, transmitted via Bluetooth. The program ensures that our vehicle will never come off the trail, and also its coordinates will be stored continuously. Moreover, the program identifies all the coordinates that represent a crossroad. The crossroads and the road's end are the nodes (or vertices) of the graph, where the edges are pairs of nodes, represented by (x, y).

A. Segmentation of Road Images
The camera calibration is an important activity (Keivan & Sibley, 2015). In this paper, a low-cost camera (2MP and f4.8mm) was fixed on the robot to 8 cm from the floor, (Figures 1 and 10). The black tape attached over a white floor has 8mm wide. According to each captured image the vehicle must proceed with the appropriate command that keeps the next road stretch within the camera range. For this purpose, we use Mathematical Morphology (Corke, 2005;Gonzalez & Woods, 2007;Serra, 1982;Serra, 1988) to analyze the image and then choose a suitable command that will be executed by the steering wheel and the motor. This algorithm uses the function IMGsegm, which give orders to move according to the image of the tape pieces and their arrangement. Figure 2 shows some examples of images and their corresponding commands. In order to choose the suitable command, the image is submitted to a segmentation process that follows a sequence of functions contained in the Matlab® Image processing toolbox. Firstly, the image is converted to binary format and fits into a square of L x L pixels that will be called IMG. Then a square of (L-30) x (L-30) pixels is removed from the center of IMG. What remains are just boundary objects, as illustrated in Figure 3. In Figure 3, right side, there are two white components, in which aiming from the bottom one to the other one determines the direction of the movement. Moreover, the inclination between these two components determines the intensity of the movement. We calibrate the robot rotors using a constant time with the command "pause" the Matlab® as value 1.5 milliseconds. So we do capture a frame, processing the image and the PC sends a command via Bluetooth for the robot rotors, then wait for these 1.5 milliseconds. This type of robot movement is detailed in (Yano & Zampirolli, 2013).
Receive image imgcaptured; imgSegment=Segment(imgcaptured); Degree=Orientation(imgcaptured); Switch imgSegment Case {"straight route"} Order = "go forward"; Case {"road end"} Order = "road end"; Case {"left route"} Order = "turn left"; Case {"right route"} Order = "turn right"; Case {"crossroad"} Order = "crossroad"; The cases of three or more components (when the robot finds a crossroad for example, having more than one option to follow the road network) are treated by pairwise analyses and it will be explained ahead.

B. The Function Map
Our function called Map was implemented with the same logic used by soldiers and scouts when they guide themselves in jungles and fields (Department of The Army, 2007). For that, one only needs to track their travelled distance and to use a compass. They are instructed to walk a distance in the direction of a certain angle referenced by the North magnetic pole. When they reach the correct point, a new distance has to be walked along a new direction. This procedure is simple to follow and discards any elaborate equipment. For this reason, we chose it in order to elaborate our algorithm.

C. Vehicle Movement
Whenever the function IMGsegm gives an order the function LEGO_Movement reads the order and activates the steering wheel and the motors accordingly. This algorithm not only controls the motors but also records the vehicle coordinates and its orientation.
The intensity of the motor rotation is defined by the route angle, namely the variable called Degree, so that it works like a steering wheel. Large angle variations thrust the rotation of the whole vehicle.
We remark that in the function LEGO_Movement there is a special function called Map, which is invoked whenever Order is equal to "crossroad" (when the robot finds a fork or a crossroad, having more than one option to follow the road network). Those two lines of source code do not define the movement itself, which will be in fact established by the new string stored in Order and the Switch command that follows in the sequel. The function Map records the main road coordinates and also makes the appropriate decision on which direction will be taken whenever the vehicle arrives at a crossroad. For example, when choosing a direction with the lowest weight stored in an adjacency matrix, this will be exemplified in the next section.
The main goal of this paper is the implementation of the function Map, which defines a graph in which the vertices are the centroids of each crossroad and of each road end. In the graph edges represent the road stretches that connect these centroids.

D. The Function of Generating Road Network Graph
The following code is the algorithm of the function Map: The mapping coordinates recorded by the function LEGO_Movement are used to distinguish crossroads between tracked and untracked. The tracked crossroads are stored in the coordinate vector {CX, CY}.
When the vehicle arrives at a crossroad its current coordinates (X(end), Y(end)) are compared with each element in {CX, CY}. If sufficiently close coordinate numbers are found this means that the crossroad has already been tracked. Otherwise it has not yet.
In the latter case both the new coordinates and the incident paths are recorded in CA. Afterwards the program chooses one of the paths at random for the vehicle to move on.  This procedure is implemented as the function regionprops, which analyses each object and locates its centroid regarding the image coordinates. The function bwlabel counts the number of isolated objects in the binary image. This number is stored in the vector Ne. For more information about these Matlab functions see www.mathworks.com/ products/image. The incident paths of a crossroad are identified by their relative angles, as illustrated in Figure 6. The relative angles are defined as the sum of the current orientation with each necessary turning angle. In this way the paths are correctly recognized again no matter from which direction the vehicle arrives. In order to find the turning angle, the crossroad image is submitted to the same segmentation process explained beforehand (Figure 3).
The relative angles are listed in the vector CA, of which columns represent the crossroad counter and each line shows the incident paths of the crossroad. If the path has already been tracked its value is set as zero.
If the crossroad is tracked, it means that CA has recorded the incident paths. So the order is defined by the first non-zero direction element.

A. First Experiment
In this first experiment to create a graph from the robot navigation in a row simulating a road network, it will be considered, in simplified form, only two images. The first image has a vertical line, or "straight route". The second image is a "crossroad". These images after segmentation are shown in Figure 7 (as the images used in the experiments have high contrast, this segmentation used a threshold 80 and a morphological opening with a disk of radius 20 pixels in one of the bands RGB). These images of Figure 7 are exchanged, as follows, as if the robot were "seeing" the image of the "straight route" into two consecutive frames, and the third frame is an image of the "crossroad". The starting point is the image of the vertical path. As a validation of movements, the simulator initially returns the following variables: The angle -0.0058 is the vertical slope of the line shown in Figure 7 (left), using the orientation attribute of regionprops function of Matlab. The X and Y values were calculated as follows: where d = 4. This value should be calibrated, with the movements of the robot rotors. The next frame to be processed is also the same "straight route" and the simulator returns: The angle -0.0117 is calculated with the difference (-0.0117=-0.0058-0.0058) of the two vertical slopes of the line shown in Figure 7 (left). The same occurs in X (-0.0699=-0.0233-0.0466) and in Y (7.9996=3.9999+3.9997).
The next frame was a "crossroad", we now have the following variables: In this first "crossroad", we define the first graph node in the coordinate (0,0) of the Cartesian Plane, we have 4 possible directions, with the angles stored in CA. In this first experiment, we will consider always the 4 angle, i.e., the robot will always turn to the right. This angle is also stored in ANG.
The next frame is the "straight route", and we have the following variables: Now another "crossroad" will be processed with the following output: In the latter process, the simulator calculates the second node of the graph. So, we can now use an adjacency matrix to store the neighbourhood of each node of the graph, see the contents of the variable MatrixAdj where node 1 is adjacent to the node 2 (and vice versa). Figure 8 shows this graph with two nodes already processed, one at position (0, 0) and the other at (7.9916, 0.3660), calculated by taking the last element of each coordinate, or CX = X(end) = 7.9916 and CY = Y(end) = 0.3660. This processing (two frames "straight route" and one frame "crossroad") are repeated indefinitely. Considering more three nodes of the graph, we will have an overlap of nodes (within a margin of error), so we do not account for new nodes of the graph, as shown in Figure 9.

B. Second Experiment
In this second experiment, the robot will be shown driving through a simulation of a road network in blocks, steering always to the right when a "crossroad" is found. See Figure 10. The graph produced by this second experiment is shown in Figure 11. In this case it was used to scrolling constant d=1.
Images, videos and data produced in experiments of this paper are available at http://vision.ufabc.edu.br/RobotRoad.

DISCUSSION
The weights in the adjacency matrix indicate how often the robot travelled over the same edge.
The adjacency matrixes used in the experiments had considered graphs of up to 10 nodes, the higher the number of "crossroad" in the road network, the bigger must be the adjacency matrix.
After several iterations of the robot driving through the same edge (consequently, same pairs of nodes), it is recommended to make an adjustment of the coordinates of the nodes, for example, by averaging the coordinates in several visits in the same "crossroad".
In the performed experiments in this paper, always when the robot arrives in a "crossroad" it will turn the rightmost edge or leftmost edge, or it will choose a random edge. With little change in the algorithm, it can be avoided the robot walking the edges which it had already visited. Thus, the robot will walk on all edges and on all nodes of the graph (connected).

CONCLUSION
In this paper it was created an algorithm to identify important coordinates on a road by guiding an unmanned vehicle using image processing. The code written in Matlab sends orders to a LEGO NXT vehicle with an attached camera, from which an image segmentation algorithm determines the movement. This algorithm ensures that the road is kept within the camera range while it travels along the road network, and another algorithm memorizes the coordinates of the crossroads to decide which way the vehicle has to take, ensuring that not a single path will be left untracked. After having implemented these algorithms, several tests were performed. The vehicle moved according to the expected performance and all the crossroad coordinates were recorded correctly. Therefore, we conclude that our visionbased strategy and the LEGO NXT kit have both succeeded in the indoor modelling.
Furthermore, an important extension of our map creation strategy is another algorithm adapted to GPS. This way the vehicle could travel much larger distances. In future works we intend to apply these results to other vehicles, specially to medium range aerial vehicles that fly over streets, rivers or marks on the ground, and use the aforementioned image processing algorithm.