3D Plant Modelling via Hyperspectral Imaging .3D Plant Modelling via Hyperspectral Imaging Jie Liang

download 3D Plant Modelling via Hyperspectral Imaging .3D Plant Modelling via Hyperspectral Imaging Jie Liang

of 6

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of 3D Plant Modelling via Hyperspectral Imaging .3D Plant Modelling via Hyperspectral Imaging Jie Liang

  • 3D Plant Modelling via Hyperspectral Imaging

    Jie LiangAustralian National University


    Ali ZiaGriffith University


    Jun ZhouGriffith University


    Xavier SiraultCSIRO



    Plant phenomics research requires different types of sen-sors be employed to measure the physical traits of plantsurface and to estimate the plant biomass. Of particularinterest is the hyperspectral imaging device which captureswavelength indexed band images that characterise materialproperties of objects under study. In this paper, we intro-duce a proof of concept research that builds 3D plant modeldirectly from hyperspectral images captured in a controlledlab environment. We show that hyperspectral imaging hasshown clear advantages in segmenting plant from its back-ground and is promising in generating comprehensive 3Dplant models.

    1. IntroductionPlant phenomics is an area of plant biology that studies

    the influence of genetics and environment on both the phys-ical and biochemical traits of plant organisms [7]. One ofthe main tasks in this area is dissecting plant function andperformance via measurement of plant appearance. Suchmeasurements provide inputs to other key tasks in plantphenomics, including investigating carbon partitioning andphotosynthesis in plants, as well as finding mechanisms ofdrought tolerance and flowering behavior. Therefore, ro-bust and accurate plant measurement methods are of greatimportance.

    The development of sensing technology has enabledmany measurement tools such as radar, RGB camera, in-frared camera and hyperspectral camera be bedded in plantobservation process. Among them, of particular interest isthe hyperspectral imaging device, which generates tens orhundreds of contiguous narrow spectral band images in-dexed by the light wavelength. These band images con-

    tain rich information on the spectral and spatial distribu-tions of distinct surface materials. They enable more ac-curate and reliable object detection and material classifica-tion than using panchromatic or multispectral imagery. Asa consequence, hyperspectral imaging techniques have beenwidely used in remote sensing, environmental monitoring,and surveillance in agriculture, industry and military [9].When applied to plant research, hyperspectral imaging hasshown success in detecting traits of disease or nutrition de-ficient [6, 1].

    Despite its advantages in object detection and analysis,the research on hyperspectral imaging in computer vision isstill very limited. In recent years, thanks to the productionof relatively low cost hyperspectral imaging devices, com-puter vision researchers have started to explore this area.More understanding of the statistical properties of hyper-spectral imagery have been reached [4], and some tradi-tional computer vision topics have been covered, such ascamera sensitivity analysis [10], feature extraction [13], andillumination estimation [8].

    In this paper, we address one of the fundamental prob-lems of computer vision, 3D reconstruction, in the contextof plant modelling using hyperspectral images. Some re-search have already incorporated hyperspectral data into 3Dmodels. For example, Brusco et al presented an interest-ing work on modeling historical building with multispectraldata, while the depth information was captured by a rangecamera based on laser scanner [2]. Similarly, Nieto et albuilt 3D model based on depth data captured by a laser scan-ner and mapped hyperspectral image to 3D Model to displaygeological mineral information [16]. More recently, Kimet al integrated a hyperspectral camera into a 3D scanningsystem to enable the measurement of the diffuse spectral re-flectance and fluorescence of specimens [12]. However, allof these have not explicitly built 3D models directly from

    2013 IEEE International Conference on Computer Vision Workshops

    978-0-7695-5161-6/13 $31.00 2013 IEEEDOI 10.1109/ICCVW.2013.29


    2013 IEEE International Conference on Computer Vision Workshops

    978-1-4799-3022-7/13 $31.00 2013 IEEEDOI 10.1109/ICCVW.2013.29


  • hyperspectral data.Our method, on the contrary, attempts to build a 3D

    plant model directly from a sequence of hyperspectral im-ages captured in a controlled lab environment. The spec-tral data is first used to segment plant from its background.Then keypoints are extracted from plant, which are usedto find correspondences between a pair of spectral images.Finally a structure from motion based model is developedto reconstruct the 3D plant. The initial results show thatthe spectral data can be used for effective plant segmenta-tion, which is an important step for 3D modelling. Fur-thermore, the 3D models produced from difference bandscontains mostly consistent structural information of plants,and in some cases, complement each other. This impliesthat different band images can capture different propertiesof plant surface. If these models can be properly com-bined, they will lead to promising approach in building a 3Dmodel that reflects more complete structural information ofthe plants than that can be reconstructed by traditional sys-tems [17, 20]. This technique can also be combined withexisting 3D plant modelling methods based on laser scan-ners or Kinect [15] in order to build more accurate plantmodels.

    The rest of paper is organised as follows. Section 2 de-scribes the hyperspectral plant imaging system. Section 3introduces the proposed 3D plant modelling method. Sec-tion 4 presents the experimental results, with conclusionsand future work given in Section 5.

    2. Hyperspectral Imaging of PlantsOur hyperspectral imaging system consists of three main

    components, i.e. objective lens, a hyperspectral filter, and ahigh sensitivity camera, with the hyperspectral filter con-necting the lens and the camera. In this research, we haveused an acousto-optical tunable filter (AOTF) that supportsimaging from 400nm to 1000nm at 10nm in spectral resolu-tion. A control unit is connected to the filter to let the lightin designated wavelength pass through to reach the cam-era. By scanning through the visible to infrared wavelength,grayscale images can be generated to form different bandsof the hyperspectral image. The output of the imaging pro-cess is a data cube with the first two dimensions show thespatial positions of pixels, and the third dimension indexesthe bands. Therefore, each pixel on the image is a vector ofresponses across the visible to infrared spectrum.

    We collected plant data in the High Resolution PlantPhenomics Centre (HRPPC) in the Commonwealth Scien-tific and Industrial Research Organisation (CSIRO) in Can-berra, Australia. HRPPC provides integrated plant mea-surement system that utilises several imaging tools, such aslight detection and ranging sensors, thermal infrared cam-eras, multispectral and RGB cameras to capture high reso-lution plant data. The imaging lab provides consistent illu-

    mination condition to facilitate the imaging process. Dur-ing the data capture, a plant was put on a turntable plat-form and transmitted into the workspace. After the plantwas positioned, the hyperspectral camera captured imagesby scanning through the visible to infrared bands. Then theplatform rotated for three degrees to allow another scan be-ing done. This process continued until the plant had beenrotated for 360 degrees with all views covered. During theimaging process, camera parameters such as focus length,zoom, exposure time remained unchanged. At last, 120 datacubes were obtained for each plant, covering the whole sur-face of the plant. During the image capture process, a whitebalance reflectance target is used to normalised the hyper-spectral data. Figure 1 shows a plant image example. Thefirst row of the figure shows band images captured at differ-ent wavelength from the same angle, while the second rowshows images captured at different angles from the sameband.

    3. Plant 3D ModelingThe proposed 3D modelling method contains three steps,

    which are image quality improvement, plant segmentation,and 3D reconstructing. The first two steps can be consideredas the preprocessing steps.

    3.1. Image Preprocessing

    The hyperspectral images often suffer from noise andcross band misalignment. The noises mainly come from thenarrow band of light that is allowed to pass the hyperspec-tral filter within short period of time. Although our camerais highly sensitive, the signal to noise ratio is still low, espe-cially in the short wavelength range where the light intensityis low. To reduce the influence of these bands, those withvery low signal to noise ratio were removed from the data.Then the rest band images were smoothed using a Gaussianfilter.

    Misalignment of band image can be caused by the chro-matic abberation of camera lens, or the misalignment ofgrating component in the tunable filter. Then light in differ-ent wavelength follows slightly different transmission pathsbefore reaching the camera. In order to reduce the misalign-ment, each band image is calibrated against an anchor bandimage at 790nm. This is done by maximising the mutualinformation of every band to the anchor band, so that thetransformation matrix in the following equation can be op-timised: xy


    =s cos() s sin() txs sin() s cos() ty

    0 0 1


    (1)In this equation,

    [x y 1


    [x y 1

    ]Tare pixels

    before and after transformation, respectively. , s, tx, and


  • (a) (b) (c) (d)

    (e) (f) (g) (h)

    Figure 1. Hyperspectral images: the first row shows band images captured at 600nm, 700nm, 800nm and 900nm from 0 degree; the secondrow shows band images captured at 800nm from 0 degree, 60 degrees, 1