A Cognitive Vision Platform for Semantic Image Understanding Monique THONNAT and Celine HUDELOT...

A Cognitive Vision Platformfor Semantic Image Understanding

Monique THONNAT and Celine HUDELOT

Orion teamINRIA Sophia Antipolis FRANCE

M. Thonnat 2

Introduction Cognitive Vision Platform Application to Plant Disease Recognition Conclusion

Overview

M. Thonnat 3

Introduction: the Problem• Problem:

•What does it mean to perform image understanding ?

semantic image understanding (e.g. object classification)

•What does it mean to associate semantics to a particular image ?

M. Thonnat 4

Different interpretations of this image are possible:

• A light object on a dark background

• An astronomical object

• NGC4473 galaxy

Introduction

Image semantics is not inside the image

Image interpretation depends on a priori knowledge

M. Thonnat 5

IntroductionFocus:

• complex natural objects with existing taxonomy

Proposed approach: Knowledge-based Vision

• formalize the a priori knowledge for image interpretation in knowledge bases

• explicit the reasoning (how to use a priori knowledge) for each subtask of image interpretation

• propose a platform reusable for different applications

M. Thonnat 6

Cognitive Vision Platform

A platform performing 3 subtasks Semantic data interpretation

Application expert knowledge (domain taxonomy and terminology)

Visual data management Matching between numerical image data

and symbols Scene analysis using spatial reasoning

Image processing numerical object description program supervision techniques : to

automate the management of an image processing library

M. Thonnat 7


For each task: An application independent engine A conceptual model for the knowledge

Two ontologies for the interoperability between the different components: Visual Concept Ontology: spatial, color and

texture concepts Image Processing Ontology: image data and

image processing functionality concepts

M. Thonnat 8


M. Thonnat 9

Cognitive Vision: Semantic Interpretation

Goal: Find the semantic class of physical objects

or situations observed on images

How: Perform the interpretation in the same way

experts do: Use a priori knowledge of application

domain terminology and taxonomy Top down strategy

M. Thonnat 10

Cognitive Vision: Interpretation

Knowledge Formalization: Declarative knowledge:

Domain class: application concept (plant leaf, pollen grain) described by visual concepts (green color and oval shape or pink and circular) and subparts

Domain class tree : hierarchy of domain classes Context: explicit representation of current domain

context and acquisition context Domain request: request of an end user

Representation by frames with slots

M. Thonnat 11


Knowledge FormalizationDomain Classname White_FlySuperClass InsectSubPart DescriptionDomain Class Fly_BodyDomain Class 2 Fly_Antenna

Domain Classname Fly_BodyVisual DescriptionST_VisualConcept Shape[oval] Elongation [important]Color_VisualConcept Hue [white]

Domain Classname Fly_AntennaVisual DescriptionST_VisualConcept Shape [line] Thickness [thin]Color_VisualConcept Hue [white]Spatial_Relation Connected [Fly_Body]Spatial_Relation Right_of [Fly_Body]

Sub-part

M. Thonnat 12


Knowledge Formalization: Inferential knowledge:

Context criteria: describe decisions during the semantic interpretation

Initialization interpretation criteria: information on how to initialize the problem using the context

Post interpretation criteria: information to refine the interpretation results according to the context

Implemented by rules Exemple of post interpretation criteriaIf Powdery Mildew detected and temperature < 25 C and Humidity > 80%then Alert “treatment is needed”

M. Thonnat 13


Reasoning Depth-first domain class tree traversal

Visual object hypothesis propagation by building visual data management requests (visual object instance finding)

Matching between visual object instances and predefined domain classes

Classification refinement

M. Thonnat 14

Cognitive Vision: Visual Data Management

Goal: Matching between symbols and sensor data

How: Data management, spatial reasoning, top

down and bottom up strategies Symbol grounding or Anchoring:

Anchoring = « Problem of connecting, inside an artificial system, symbols and sensor data that refer to the same physical objects in the external world » [coradeschi99]

M. Thonnat 15


Knowledge Formalization: Declarative knowledge:

Visual concepts (symbolic data): description of visual concepts and of their grounding relation with image descriptors

Image data concepts (sensor data): primitives (ridge, region, edge), descriptors (area, eccentricity)

Spatial relations : topology (RCC8), distance and orientation

Visual data management requests : express the visual data management problem

Represented by frames with slots

M. Thonnat 16

Cognitive Vision: Data Management

Knowledge Formalization: Inferential knowledge :

Object extraction criteria : how to constrain image processing requests (using visual concepts and spatial relations)

Spatial deduction criteria : how to infer spatial relations from another ones

to diagnose the image processing results Visual evaluation criteria: how to diagnose

image processing results Implemented by rules

M. Thonnat 17

Cognitive Vision: Data Management

Knowledge Formalization: Example of object extraction criteriaLet c a visual content context and O a visual objectIf O.geometry is an Open Curve and O.thickness is {Thin, Very

Thin} then c.ImageEntityType:=Curvilinear Structure “Ridge or Valley” Example of spatial deduction criteriaLet O1, O2, O3 three visual objectsIf NTTP(O1, O2) is true and Left_Of(O2,O3) is true then Left_Of(O1,O3) is true Example of visual evaluation criteriaIf mode is interactive then assess_data_by _user [correct under_segmentation

over_segmentation noisy]

M. Thonnat 18


Reasoning Image processing request building

according to visual object hypotheses (Object extraction criteria)

Matching between image processing results and symbolic data

Instantiation and sending of visual objects to the Interpretation task

Spatial Reasoning: multiple objects (spatial

deduction criteria)

M. Thonnat 19

Cognitive Vision: Image Processing

Goal :Object extraction and numerical description

How: Use of program supervision techniques:

Dynamic configuration and execution of a library of image processing programs (versus fixed procedure)

Explicit formalization of expertise on how to use programs

M. Thonnat 20


Knowledge formalization: Declarative knowledge:

Goals: image processing functionality (thresholding, edge extraction,…)

Operators: knowledge to solve a given problem:

primitive: particular program composite: particular combination of programs

Program supervision requests: instantiations of goals on particular data, under particular constraints

M. Thonnat 21


Knowledge FormalizationPrimitive OperatorName Recursive_Gaussian_DerivationInput data Image name inputOutput data Image name mfxx

Image name mfyy Image name mfxyParameters sigma default 1.0Preconditions valid inputPostconditions valid mfxx, valid mfxyInitialization CriteriaRule init-sigmaLet c a visual content contextIf trueThen sigma := c.objectwidth/sqrt(3)Calling syntax: Gaussian -sigma input mfxx mfyy mfxy

Composite OperatorName Ridge_ExtractionFunctionality Object ExtractionInput data image input_imageOutput data image segmented_imagePreconditions valid inputPostconditions valid mfxxBody “sequential decomposition”Recursive_Gaussian_derivation – Steger_Detector-Ridge-FilteringDistributionRidge_Extraction.input_image / Recursive_Gaussian_Derivation.input…FlowRecursive_Gaussian_Derivation.mfxx / Steger_detector.mfxx

M. Thonnat 22


Reasoning: Planning techniques (HTN)

Program selection in a library of programs Selected programs execution Evaluation and adjustment if needed

Program Supervision Engine

Library of programs

Program Utilisation

KBPlanning Execution

EvaluationRepair

resultsplan

(part of)

judgementsActionsto correct

1 2

3

4

56

7

correct

incorrect

Request + data

M. Thonnat 23


M. Thonnat 24

Application on plant disease diagnosis

Why Image Understanding ? Plant disease diagnosis = visual observation

which aims at inferring disease presence by the observation of signs and symptoms

TO BE ABLE TO REASON : signs and symptoms interpretation in terms of diseases

TO BE ABLE TO SEE : Focusing on relevant criteria

Star shape network of white and thin filaments (5-10 μ)Presence of elliptical white blobs in the centre of the networkClimatic Context: High humidity, Temperature : 25 °C

Early powdery mildew infection in propitious conditions

Early diagnosis:Microscopic image (x64) of rose leaf part

M. Thonnat 25

Application on Plant Disease Diagnosis: Rose Diseases

Powdery mildew :State of infection : earlyVegetal support : red leaf

Powdery mildew :State of infection : very earlyVegetal support : green leaf

Two white flies close to their eggs

Need of domain knowledge

Intelligent management of image processing programs

Complexity and variability of object appearance

Variability of contexts

Scene knowledge and spatial reasoning

Multiple objectsand various object types

M. Thonnat 26

Leaf

Healthy

Non Healthy

Insects

Virus

Fungi

White fly

Penicillium

Powdery mildew

Germinated tubes

Filamentous

Aphid

Vegetal tissue

Veins

red

green

Subpart

SubclassAcarid

Ungerminated

Pellets

Application on plant disease diagnosis

Domain knowledge base : the class tree

Mycelium: • Part of : Fungi• network of at least 2 connected Hyphae•nb_hyphae = {unknown}

Hyphae:• Part of : Mycelium•Geometry: line•Thickness: thin, very thin•Straightness:=almost straight•Luminosity=bright•...

M. Thonnat 27

Input : User Request Fungi infection?

Image + Context

Variety : LeonidasLeaf : youngSeason: summerTemp: 24° CHumidity: 80...

Application : early detection of plant diseases

InterpretationDomain concept tree traversal to build visual object hypotheses

Leaf Scene

VegetalPart

Disease

Insects

Virus

Fungi

White fly

Penicillium

Powdery mildew

Dispersed

Clump

Aphid

Subpart

Subclass

AcaridVery Early

Pellet

1

Data Management Symbolic request to image processing

request

3

Goal: segmentationContraints:Image entity = ridge

Object.width = [1..3]Object.intensity > 150

Input Data: Image : input imageMask : area of interest

Image Processing Request

4

Image Processing:request solving by

program supervision techniques

5Image Data

Ridge 1

numerical descriptors

Ridge 3+ Numerical descriptors

Ridge 2+

NumericalDescriptors

6

InterpretationDomain concept tree traversal to build visual object hypotheses

Visual Object Hypothesis

2

Group of :Geometry: star shape network of { Geometry: line Thickness : thin width [7..10 m] very thin width [5..7 m] Straightness : almost straight Lightness: bright}Spatial Relation: Connected}

1

M. Thonnat 28

Application : early detection of plant diseases

Image Data

Ridge 1

numerical descriptors

Ridge 3+ Numerical descriptors

Ridge 2+

NumericalDescriptors

Data Management Visual object

hypothesis verification and instantiation

7

Interpretation : classificationMatching between visual object instances and domain concepts

9

Interpretation : diagnosis

Post classification rules activation

11

Visual Object InstanceNetwork of lines

Line 1

Line 2 Line 3

Line 5 Line 4

ECEC

ECEC

Line line1Thickness:=thin (0.8)Straightness:= straight (0.5)Lightness:=bright (0.7)Connected (line2)Connected (line4)+ link to image data

8

Diagnosis

Early powdery mildew infection on young leaf

12

Recognised domain concept

10

Freely dispersed mycelium

M. Thonnat 29

Conclusion

A platform for automatic recognition of natural objects Ontology-based formalism for knowledge acquisition 3 dedicated reusable engines for

semantic interpretation visual data management program supervision for image processing

Future works integrate results on :

machine learning for visual concept detection (Nicolas Maillot)

machine learning for image segmentation (Vincent Martin)

A Cognitive Vision Platform for Semantic Image Understanding Monique THONNAT and Celine HUDELOT...

Documents

Transcript of A Cognitive Vision Platform for Semantic Image Understanding Monique THONNAT and Celine HUDELOT...