Hexagonal Image Analysis Using Mathematical Morphology

Multi-legged Robot Walking Strategies, with an Emphasis on Image-based Methods

Department of Mechanical & Electro-Mechanical Engineeringby Kazi Mostafa

1

OutlineIntroduction and BackgroundResearch ObjectivesRelated Research WorkMethodologyConclusion

April 03, 20152

2

OutlineIntroduction and BackgroundResearch ObjectivesRelated Research WorkMethodologyConclusion

April 03, 20153

Multi-legged Robot Walking Strategy For low computational power, low resolution images in real time application; efficient edge detection method is crucial Edge detection method for multi legged robot to detect standing zone edges, gap/obstacle Walking strategy for multi legged robot with damaged leg We focused small robot with low computation power4April 03, 2015

4

Edge DetectionEdge detection effectiveness depends on quality of input imageImages are processed in a rectangular gridIncreases cost & decreases performanceNumerous types of sampling systems are feasible5April 03, 2015

5

OutlineIntroductionBackgroundResearch ObjectivesRelated Research WorkMethodologyConclusion

April 03, 20156

Mathematical Morphology Morphological erosions & dilations produce results identical to the nonlinear minimum & maximum filters7

Figural representation of Grayscale Morphology formula (Dilation)DilationThe value of the output pixel is themaximumvalue of all the pixels in the input pixel's neighborhood

Structuring ElementInput ImageOutput Image

7

Classical Sets8Classical sets: either an element belongs to the set or it does notFor example, for the set of integers, either an integer is even or it is not (it is odd)Another example is for black & white photographs, one cannot say either a pixel is white or it is blackWhen we digitize a b/w figure, we turn all the b/w and gray scales into 256 discrete tonesApril 03, 2015

8

What is Fuzzy Set?Fuzzy sets first introduced by Lotfi A. Zadeh[1] as an extension of the classical set theory

In crisp set a pixel is either black or white. For edge detection, its an edge or a no-edge. But the edges are not always precisely defined. Fuzzy images are characterized by the degree to which each pixel belongs to a particular region9Crisp set and fuzzy set for grayscale image[1] Zadeh, L. A. (1965). Fuzzy sets.Information and control,8(3), 338-353.

Crisp setdark gray-levelsfuzzy setdark gray-levelsApril 03, 2015

http://tizhoosh.uwaterloo.ca/Fuzzy_Image_Processing/set.htm9

Fuzzy Morphology The word fuzzy means vague. Fuzziness occurs when the border of information is not clear-cut It can handle the idea of partial truth & falseFuzzy set theory allows the gradual calculation FM uses the concepts of fuzzy set theoryFM is the extension of mathematical morphology to fuzzy sets10April 03, 2015

OutlineIntroductionBackgroundResearch ObjectivesRelated Research WorkMethodologyConclusion11

April 03, 2015

Research ObjectivesThe purpose of this study is toDevelop structuring elements (morphological gradients) for low resolution imagesApply proposed method for noise removal and edge detectionDerive efficient walking strategies by using image based method and with damaged leg 12April 03, 2015

OutlineIntroductionBackgroundResearch ObjectivesRelated Research WorkMethodologyConclusion

13

April 03, 2015

Mathematical Morphology14April 03, 2015Serra (1986)Z. Yu-qian (2006)Y. Zhang (2010)X. Bai (2010)Zhang(2015)

A good review of morphological operator Detect the edge of lungs CT image with salt-and-pepper noiseComparison between Morphological edge detection and traditional edge operator Multiscale top-hat transformations based on SES has been constructedThe Application of Mathematical Morphology & Sobel Operator in Infrared Image Edge Detection

Moreover, some multiscale top-hat transformations [25, 51, 52] based on structuring elements has been constructed for various application.14

Fuzzy Morphology15Publications about Fuzzy Morphology in Image Processing over the last 30 Years

Tree of fuzzy morphology

The most renowned concept of fuzzy morphology is the alpha-morphologyIts founded on the level sets of fuzzy membership degree function and first introduced by Bloch et al

Multi Legged RobotConsiderable research[1,2] done on robot vision and has been primarily based on the rule of forbidden walking regions Inagaki[3] investigated the leg failure situations in hexapod robots of which one leg was damaged. More recently, Yang[4] mentioned that it is helpful to lock the joint associated with a damaged motor. However, locking mechanism always cannot provide support. When collapse occurs at 2nd or 3rd joint in a leg16[1] R. Ponticelli and P. G. de Santos, "Obtaining terrain maps and obstacle contours for terrain-recognition tasks," Mechatronics, vol. 20, 2010.[2] J. Estremera, et. al. "Continuous free-crab gaits for hexapod robots on a natural terrain with forbidden zones: An application to humanitarian demining, Robotics & Autonomous Systems, 2010.[3] K. Inagaki, "Gait study for hexapod walking with disabled leg," in Intelligent Robots and Systems, IROS'97.[4] J.-M. Yang, "Gait synthesis for hexapod robots with a locked joint failure," Robotica, vol. 23, 2005.

OutlineIntroductionBackgroundResearch ObjectivesRelated Research WorkMethodologyConclusion17

April 03, 2015

Binary Morphology18

April 03, 2015

18

Binary Morphology19Simulated Hexagonal Grid

d=3d=2d=1

Many square grids combine together to create a pixel block.For example, d= 7; Combines 120 rectangular grids. Resolution = (Summation of 120 pixels gray values)/120d=4Image Resampling April 03, 2015

19

Image Resampling 20

Different values for simulated hexagonal gridImage conversion from rectangular to simulated hexagonal grid: (a) hexagonal image(D=2) (b) hexagonal image (D=7)April 03, 2015

20

Simulated Hexagonal Image 21

April 03, 2015

21

Structuring elements with various sizes & directions22

April 03, 2015

22

Noise Removal

The proposed noise removal method: (a) Noisy image (a) Noise-free imageThe proposed method applied opening followed by closing with various combinations of SEsThe better combination to remove noise was defined by horizontal, 60, 120, and three-by-three structuring element23

Edge Detection

Edge detection achieved by applying various directional three-by-three SEs on the hexagonal grid24The performances of the three-by-three hexagonal SEs were superior to those of their five-by-fiveThe smaller circular-shape hexagonal SEs achieved superior performance in identifying curved edgesBy contrast, larger SEs introduced unwanted thickness & discontinuitiesApril 03, 2015

Edge Detection

Edge detection achieved by applying various directional three-by-three SEs on the rectangular gridNumerous unwanted discontinuities in the rectangular imagesApril 03, 201525

Performance evaluation26Combination of Structuring ElementsThe Face Test ImageMSERatio of edge pixels to image size (%)Hexagonal ImageHorizontal, 60 and 120 SEHorizontal, Vertical and 120 SEHorizontal, Vertical and 60 SEVertical, 60 and 120 SE0.971.473.025.0515.7815.0214.7914.23Rectangular ImageHorizontal, 60 and 120 SEHorizontal, Vertical and 120 SEHorizontal, Vertical and 60 SE10.5611.3612.0213.0712.9312.71

27

Grayscale & Fuzzy Morphology April 03, 2015

Grayscale Morphology28D=2D=4D=7

April 03, 2015

28

Grayscale MorphologyHexagonal Structuring Elements (3x3, 5x5, 7x7)

Hexagonal Grayscale Morphological Operator

29

(c)(b)(a)

HexagonalimageStructuringelement(c) After DilationApril 03, 2015

29

Grayscale Morphology (Multiscale) Multi scale morphological analysis seems to be more favorable than single-scale analysis

where B is the SE and n is the number of operations Multiscale Dilation = AnB, and Multiscale Erosion = AnB,where A is the input image, B is the SE, and n is the number of operations Multiscale Gradient = (AnB) (AnB) Multiscale Gradient = (AnB) (AonB)

30

April 03, 2015

30

Edge Detection (Morphological Gradient) Multi-scale Gradient = (AnB) (AnB)

31

Edge image by using hexagonal morphological gradient operator (a) 3x3(b) 5x5Multiscale hexagonal morphological gradient 3x3 SE (a) n=1(b) n=2(c) n=3April 03, 2015

31

Edge Enhancement (Top hat transformation)32MW = max (WTH1, WTH2, ); MB = max (BTH1, BTH2, )Enhanced Edges = (Original Image x W1) + (MW x W2) (M B x W3)

April 03, 2015

32

Edge Enhancement Enhanced edges after applying top-hat transform (3x3 SE)33

April 03, 2015

33

Performance EvaluationSELenaPepperBarbaraMSELinear Index of fuzzinessMSELinear Index of fuzzinessMSELinear Index of fuzziness3x352.980.06457.470.06238.560.1165x546.470.13748.770.13731.280.2037x741.240.20142.360.20526.650.2839x937.160.25537.370.26523.350.353

34Comparison of different HSE Edge detection using morphological gradientSELenaPepperBarbaraMSELinear Index of fuzzinessMSELinear Index of fuzzinessMSELinear Index of fuzziness3x356.210.05459.230.05543.410.1015x553.910.12656.360.10542.560.1557x751.090.15353.010.13340.550.1899x946.810.18148.250.17536.410.229

Comparison of different HSE enhanced Edge detection

April 03, 2015

http://tizhoosh.uwaterloo.ca/Fuzzy_Image_Processing/fuzziness.htm34

Fuzzy Morphology35

System diagram for performing fuzzy image processing on hexagonally sampled gridApril 03, 2015

35

Steps of Fuzzy Image Processing

36April 03, 2015

Image FuzzificationS-membership function as given below:(x) = 0x c,where, b is any value between a and c. For a = Xmin, c = Xmax37

Membership degreePixels

April 03, 2015

Image Fuzzification38

Image fuzzification (a) test image pepper, and (b) fuzzified image pepperApril 03, 2015

Fuzzy Structuring Elements 39

April 03, 2015

39

Noise Removal40

The proposed noise removal method: (a) noisy image (10.13 dB), (b) Rectangular grid (19.97 dB), (c) hexagonal grid (27.09 dB)April 03, 2015

dB 40

Edge Detection

Edge detection by applying different directional 3 by 3 fuzzy SE on hexagonal grid41

41

Performance evaluation

Mean Square Error (MSE), Signal to Noise Ratio (SNR) and The ratio of edge pixels to image size42

Standard test image & its Ground Truth prepared manually for evaluationApril 03, 2015

42

Performance evaluation MethodPepper Test ImageMSESNRRatio of edge pixels to image sizeHexagonal ImageCombination of Horizontal, 60 and 120 SE0.7421.9717.8%Combination of Horizontal, Vertical and 120 SE1.9219.8517.01%Combination of Horizontal, Vertical and 60 SE4.0217.6716.61%Combination of Vertical, 60 and 120 SE7.0515.2216.01%Combination of 60 and 120 SE7.3115.0615.85%Rectangular ImageCombination of Horizontal, 60 and 120 SE10.5612.5614.06%Combination of Horizontal, Vertical and 120 SE11.3612.4314.01%Combination of Horizontal, Vertical and 60 SE12.0212.0913.81%Combination of 60 and 120 SE13.0611.0613.61%

43Quantitative measure obtained by edge detectors in hexagonal grid and rectangular grid by using fuzzy hexagonal morphology for real test images Pepper

43

Better Walking Strategies for Multi-legged RobotsMulti legged robots present various limitations, such as traveling on discontinuous terrain and navigation systems Moreover, there are several kinds of damages can be happen in multi legged robot legsThus, we proposed image-based walking strategy 44April 03, 2015

Better Walking Strategies for Multi-legged Robots

An Image-based Walking Strategy45

Geometric model of the proposed hexapod robot

A uniformly distributed random terrain

Terrain edges (black lines)April 03, 2015

45

46

Discontinuous terrain consisting of the standing zone (white), forbidden zone (black), and edges (yellow)7 by 7 structuring element

An Image-based Walking Strategy

46

Hexapod Parameters47

The 2-3-6 gait support pattern of the hexapodHexapod parametersRmax= 60 PixelsRmin= 20 PixelsB = 100 PixelsW= 50 PixelsD = 50 PixelsApril 03, 2015

Sm= Stability margin47

Grayscale Image of a Simplified Forward Gait48

Each step of the simplified forward gaits ranges of movement (Km) was measured and stored in a matrixBrighter zones represent higher Km values, and vice versa. White denotes wide ranges of movement & black represents shortest ranges of movement. April 03, 2015

Gait Selection for Forward Walking49

-15-30-450S111.57101.2183.43112SX-28-50-590SY1088859112

The parameter values of robots forward gait15304560S111.57101.2183.4367.23SX28505958SY108885934

Forward angle ; Stride length S;SX, SY (S in X and Y directions) This study create an adaptive forward gait list by selecting from eight simplified forward gait angles, namely 0, 15, 30, 45, 60, -15, -30, and -45.April 03, 2015

S stride length49

Rotational Gait (Around the CG) The distance between every point of motion range and the CG was measuredThen applied the aforementioned ranges of movement matrix for the hexapods movement. A gray value of 255 was defined for the maximal angle of rotation50

April 03, 2015

http://mathworld.wolfram.com/LawofCosines.html23650

Maximal Angle of Rotation51

SSXSYRxyRotation around the CG30.03000000Max. angle of rotation 37.6740.16133862620

The comparison of rotational gaitRotation around the point OT and after rotation3 as 58.85 2 as 30.03 6 as 30.39Rotated clockwise with 2-3-6 gaitThe hexapod reached the maximal angle of rotation when 2, 3, and 6 were equivalent.

Rotation around Any Point52

For 5 angle of rotation

For 10 angle of rotationSSXSYRXY5101.9028981162113027210932789527.68515115

The Parameters for rotation of 5 and 10

Hexapods destination target point was in front and the CG was on the right side of the rear positionIn each step, the hexapod could rotate degreesHowever, the rotation of the hexapod depended on the condition of the forbidden edges, zones, and target distanceAfter each movement, the hexapods CG and stability margin were measured and updated

The Algorithm for Gait Selection53

Changes in the gait sequence ( 1-4-5 gaits, 2-3-6 gaits, symmetrical gaits)The gray regions indicate the overlapping ranges of motionApril 03, 2015

Gait Selection for Forward Walking54

April 03, 2015

54

Walking strategy for the proposed multi-legged robot55

April 03, 2015

Parameters for hexapod movement56StepsPositionGaitFootholdPosition Angle1(641,1642)302-3-6(742,1512)(540,1584)(742,1712)02(691,1554)Turn51-4-5(640,1396)(764,1525)(608,1687)03(693,1544)-152-3-6(744,1484)(547,1520)(748,1586)0.494(688,1522)-151-4-5(604,1380)(789,1542)(601,1577)0.495(677,1481)Turn102-3-6(739,1340)(565,1523)(753,1528)0.496(680,1467)Turn101-4-5(630,1309)(786,1495)(615,1511)1.947(690,1428)602-3-6(780,1285)(590,1402)(799,1519)6.258(729,1405)Turn51-4-5(653,1280)(865,1358)(664,1477)6.259(744,1346)Turn102-3-6(831,1222)(638,1288)(795,1482)9.2210(750,1320)Turn51-4-5(671,1174)(866,1273)(690,1363)12.0811(768,1252)302-3-6(869,1222)(667,1194)(869,1322)15.5512(818,1164)601-4-5(768,1084)(977,1155)(768,1222)15.5513(859,1141)Turn102-3-6(935,1039)(759,1183)(922,1257)15.5514(862,1126)151-4-5(803,984)(997,1084)(804,1168)17.1315(887,1033)302-3-6(988,903)(787,990)(988,1103)17.1316(933,952)301-4-5(882,794)(1084,922)(878,1009)17.1317(977,875)302-3-6(1076,751)(876,817)(1078,945)17.1318(1025,791)151-4-5(966,633)(1158,746)(968,835)17.1319(1054,683)452-3-6(1136,581)(954,625)(1154,751)17.1320(1085,652)01-4-5(1021,567)(1186,689)(1026,770)17.1321(1085,629)02-3-6(1136,570)(963,593)(1163,691)17.1322(1085,610)-451-4-5(1024,560)(1237,583)(1028,763)17.1323(1083,608)02-3-6(1134,562)(963,593)(1159,693)17.1324(1083,602)-451-4-5(1025,520)(1237,582)(979,680)17.1325(1079,598)02-3-6(1130,554)(959,594)(1155,695)17.1326(1079,594)151-4-5(1023,436)(1236,577)(993,648)17.1327(1096,531)

57April 03, 2015

58

Walking Strategy with Damaged LegThe use of removable sliding legs

59

Fixed Position AdjustmentStep 01: Damaged leg removedStep 02: Middle leg slide into the removed legWhen the position of the leg started to move & remaining leg may not provide stable supportHowever as any of these legs lifts above the ground, the remaining legs fail to provide a usable support polygonThus, firstly let R1 swing backward a distance S, & secondly make R2 also move in the same direction an amount of S, then swing R1 back to original stances. Repeat these stepsApril 03, 2015

No movement59

Axial stability for an adjusted gait60

Maximum stride length is 2S; SL+ & SL are the front and rear axial stability limit respectivelyThe standard stride length is 2S. The maximum swing for both the front & the rear leg is assumed to be SMost insects can walk with the tripod gait, which is a fast and statically balanced gait (SBG) for Hexapods. However, when one or more legs are missing, regular tripod gait is no longer possible. To get around this, a common leg needs to be shared in two tripod groups. April 03, 2015Multi-legged Robot Schematics

Alternative Gait Configuration

The 61R2 caseCommon Leg R3

A hexapod robot which has no R2 leg In (a), hollow arrow indicates half step & solid arrow indicates full step (2S)Dash lines are indicate the support polygonIn each interval the robot travels a distance SRobot schematicsEnhanced Gait Chart (EGC) April 03, 201561L1L2L3R1R3

Comparison of Alternative Gait62Fault type(Severed Legs)d minStride lengthPE1st Tripod2nd Tripod6-1-R20110.50.50.510.4290.80.20.20.56-1-R30210.4290.510.58-3-L1L4R20220.50.251.51.50.50.50.510.20.28-3-L1L4R10120.4290.250.51.50.48-3-L2L3R20110.50.50.51.50.4

April 03, 2015

progressive efficiency 62

Comparison of Alternative GaitFault type(Severed Legs)Axial stabilityPE[SL+]min[SL-]min6-26-2-L3R33.75006-2-L1R223.75