Smart ContentWhat Business Innovators Need to Know about Content Analytics

Jeff FriedCTO, BA-Insightjeff.fried@ba-insight.net

Examples from:

Opinions from:

Three Views of Content Analytics

Business Strategist End User Research Scientist

It’s about money, business models, advertising, and

money.

It’s about finding things, having

fun, and getting stuff done.

It’s about fast algorithms,

massive scales, and machine

learning.

Content is Exploding“If you think the information

doesn’t exist you’re not looking hard enough”

6

Traffic, Ads and Information Mash-Upsbecoming a part of emerging ecosystems

cloudplatforms

contentplatforms

adplatforms

services

Rethinking the Data Warehouse

Use of Unstructured Data in Information Analysis Applications

Analyst: Mark Beyer

10

Smart Content is Streaming

Different multimedia applications

Education

Entertainment

Archives

13

Social Video Sharing System

Users create, produce, upload, manage and share video within one system

Music Image Face

Query by example: ContentFusion

Smart Content is Mobile

Location and Form factors

Bing Twitter Maps

Smart Content is Social

Many layers of social media

//twitterviz

Publication Platforms

PublicCommunityPrivate

Publication

Priv

ate

Com

mun

ityPu

blic

Acce

ss

Facebook

Email

Answers

Web

Twitt

er

Social Graph

Naturally connected community Spam marketing campaign

Spammy communities are highly visible – don’t be part of one!

Permission

Social Search Needs• Relevance

– Filtering the document web

• Social Media Content– Filtering the social web

• Trends / Group Insight– Tapping Community Knowledge

• Answers– Trusted Advisor

Recommendation

• “Java” (coffee, island, or language?)

• “compliance”

• “What should I do in New York?”• Where are my friends now?

• Why did power go out in Palo Alto?• How does adoption work?

• ( on FB update) anybody give their babies baby Benedryl for travel/jet lag? Want to hear from parents whether they have or not and how it went

Enables 1:1 relevance based on user profile

Complexity

Value

3. Social Recommendations

(users to users)

1. Content or “Related item”Recommendations

(items to item)

2. PersonalizedRecommendations

(items to user)

Enables connections between like users

Drives service stickiness

Enables users to ‘browse sideways’ from any item

Recommendations“Personalized” to “Social”

Virtuous CyclesCreate new patterns with

positive reinforcement

Enhanced Modes of Discovery

Simplified Authoring

& Participation

New Value in Combining Data

CONSUMPTION

CONNECTIONS CREATIONS

The Long-Tail of Online Business

70 %30 %

QUERYTRAFFIC

+70% Y/Y

Virtuous Cycles in FindabilityTuned experienceSocial behavior affects relevance

Socially driven feedback loop People and expertise location are the key ‘lens’

Structure drives exploration Aligned with taxonomy and tags

Refinement

Social

Relevance

Text Analytics Isn’t Perfect

Realistic Expectations for Powerful Technology

Analytics! Semantics! Machine Learning!

36

Grab-Bag of Related Technologies• Problem – linguistic variations in concept expression

– Technology: natural language processing (NLP)

• Problem – huge numbers of documents that are the same or versions of the same– Technologies : text mining, text analytics, normalizing & de-duping

• Problem – amount of content exceeds amount of human expertise to analyze & categorize– Technologies : entity extraction, contextual analysis, auto-

categorization

• Problem – understanding trends and relative values expressed in content– Technology : sentiment analysis

• Problem – retrieving & federating contextually related and relevant content– Technologies – All of the above

3810 Entire contents © 2006 Forrester Research, Inc. All rights reserved.

BPM, Service Orchestration, Workflow

Content, Search, Integration, & Composition technologies

Presentation tier

Middle tier

Repositories

Unstructured Information access

Structured Data AccessDynamic Information Applications

Visualization

Portals, AJAX, Mash-ups, RSS, widgets, gadgets

Enterprise Content Management

ERP, CRM

, PIM, PLM, SCM

, HCM

Pro

du

ctiv

ity A

pp

s (m

ail,

IM,

offi

ce to

ols

)

Co

llab

ora

tion

To

ols

EAI, EII, ESB

Business Intelligence

Databases

ETL, Data Cleansing, Data Quality

Identity

MDM, Data Warehouses

File systemsFile filters

Connectors

Taxonomy Text Mining

Desktop Search

Federated Search

Video/audio

Enterprise Search

39

Linguistics, Statistics, & GymnasticsLexicon Base

Language-specific Common Words

Inflection Dictionaries

Part-of-speech Dictionaries

Synonymy Dictionaries

Subject-specific ontologies

Spellcheck dictionaries

Geographical and people’s names

Special terminology lexica

Basic Linguistic Algorithms

Pattern extraction

Stemming / Lemmatization

Part-of-speech Tagging

Language normalization

Vectorization

Applications

Data Cleansing

Categori-zation

Entity Extraction

Suggest Synonyms

Find similar

Stop word elimination

Spell checking

Machine Translation

Relationship Extraction

From Entity Extraction

Acronym

Person Location End of sentence

End of paragraph

Date Base = 2002-03-XX

To Fact Extraction....

Substance

Base=„Gold“

Class=„Element“

Number=79

Symbol=Au

Location

Base=„Qilian“

Country=„China“

Region=„Asia“

Subregion=„East“

„The Red Valley property lies within the Qilian fold beltwhich is host to gold deposits.“

Qilian is location of gold

Extracted Fact: Substances x Locations

Substance

Base=„Gold“

Class=„Element“

Number=79

Symbol=Au

Location=„Qilian“

Location

Base=„Qilian“

Country=„China“

Region=„Asia“

Subregion=„East“

Substance=„Gold“

Indicates a gold location

Intelligent Answers from TextInternal/external text sources Internal/external text sources

ContentFusion

Scale, Simplicity and ExpressivenessAccelerating Content Enrichment / Fusion

LookingGlass

Semantics means what?

Beware of overhype; seek pragmatic use of semantic tech

Solving the Knife problem

Man Allegedly Attacked Wife With KnifeA Tyler man is awaiting arraignment this afternoon afterallegedly attacking his wife with a knife, said Tyler police.The 41-year-old man will face aggravated assault andaggravated robbery charges, said Don Martin, thedepartment's spokesman.Officers took the man in custody near Garden Valley and Loop 323. He ran from his residence after "assaulting his wife with a knife and taking her purseat knifepoint," said information released by Martin. Thewoman refused medical treatment and did not appearto be seriously injured, the statement said.

Excellent Knives!!!

Mere frequency counting of key words can lead to undesired results......understanding relationships between words can reveal the true topic of the document.

Objective: Automatically insert an advertisement that matches the content best.

Actor Director Movie

TV Show

Adventure Comedy Face Image

Actor 0 0.6 1 1 1 1 0.9

Director 0 1 1 1 1 0.3

Movie 0 0.6 1 1 -1

TVShow 0 1 1 -1

Adventure 0 0.14 -1

Comedy 0 -1

FaceImage 0

DocumentAnalytics

SemanticAggregation/

Analytics

DomainKnowledge

48

Cyc Knowledge Base

ThingThing

IntangibleThing

IntangibleThing IndividualIndividual

TemporalThing

TemporalThing

SpatialThing

SpatialThing

PartiallyTangible

Thing

PartiallyTangible

ThingPathsPaths

SetsRelations

SetsRelations

LogicMathLogicMath

HumanArtifactsHumanArtifacts

SocialRelations,

Culture

SocialRelations,

Culture

HumanAnatomy &Physiology

HumanAnatomy &Physiology

EmotionPerception

Belief

EmotionPerception

Belief

HumanBehavior &

Actions

HumanBehavior &

Actions

ProductsDevices

ProductsDevices

ConceptualWorks

ConceptualWorks

VehiclesBuildingsWeapons

VehiclesBuildingsWeapons

Mechanical& Electrical

Devices

Mechanical& Electrical

Devices

SoftwareLiterature

Works of Art

SoftwareLiterature

Works of ArtLanguageLanguage

AgentOrganizations

AgentOrganizations

OrganizationalActions

OrganizationalActions

OrganizationalPlans

OrganizationalPlans

Types ofOrganizations

Types ofOrganizations

HumanOrganizations

HumanOrganizations

NationsGovernmentsGeo-Politics

NationsGovernmentsGeo-Politics

Business, Military

Organizations

Business, Military

Organizations

LawLaw

Business &CommerceBusiness &Commerce

PoliticsWarfarePoliticsWarfare

ProfessionsOccupationsProfessionsOccupations

PurchasingShopping

PurchasingShopping

TravelCommunication

TravelCommunication

Transportation& Logistics

Transportation& Logistics

SocialActivitiesSocial

ActivitiesEveryday

LivingEveryday

Living

SportsRecreation

Entertainment

SportsRecreation

Entertainment

ArtifactsArtifacts

MovementMovement

State ChangeDynamics

State ChangeDynamics

MaterialsParts

Statics

MaterialsParts

Statics

PhysicalAgents

PhysicalAgents

BordersGeometryBorders

Geometry

EventsScriptsEventsScripts

SpatialPaths

SpatialPaths

ActorsActionsActorsActions

PlansGoalsPlansGoals

TimeTime

AgentsAgents

SpaceSpace

PhysicalObjectsPhysicalObjects

HumanBeingsHumanBeings

Organ-izationOrgan-ization

HumanActivitiesHuman

Activities

LivingThingsLivingThings

SocialBehaviorSocial

Behavior

LifeFormsLife

Forms

AnimalsAnimals

PlantsPlants

EcologyEcology

NaturalGeography

NaturalGeography

Earth &Solar System

Earth &Solar System

PoliticalGeography

PoliticalGeography

WeatherWeather

General Knowledge about Various DomainsGeneral Knowledge about Various Domains

Cyc contains:17,000 Predicates

400,000 Concepts5,000,000 Assertions

Represented in:• First Order Logic• Higher Order Logic• Modal Logic• Context Logic• Micro-theories

Specific data, facts, and observationsSpecific data, facts, and observations

Machine Learning Techniques

Create Examples Model

Trainer

„Let the occurrence of the term ‚is host to‘ between a location and a

substance increase the probability that this is a location x substance relation

by 10%, because we have seen it more often in positive than in negative

examples.“

Good enough

?Deploy

yesno

Example: The Semantic Associative Search Method(MMM: The Mathematical Model of Meaning)

A

B

C

A

B

C

|| A || = || B || = || C ||

A

B

C

|| A || > || C || > || B ||

impression words(as a context):light, bright

impression words(as a context):dark, black

A,B,C: image data vectors

semantic space:2,000 dimensional space(presently)

(retrieval candidate image data)

2 2 2

w w w || C || > || B || > || A ||w w w

A: a sunny imageB: a silent imageC: a shady image

semanticsubspace

semanticprojection

semanticprojection

USP: 6,138,116Yasushi Kiyoki, 2009

Context Matters

Information Overload

Relevancy Overload

What’s important to me

right now

Audience-specific search experiences

User context

Inform-ation

context

Application context

Social context

Renee LoEngineeringContoso Consulting”What should I know about implementing ERP?”

Alan BrewerSales ManagerContoso Consulting”What should I know about selling ERP consulting?”

Username & Group Memberships

LocationLanguages

Business UnitDepartment

TeamTime of Day

Preferred SitesSharePoint Audiences

Interests & Current ProjectsContext of Current Task

53

Time is Money

Data = MetadataContent = Connections

57Image by Richard Cyganiak and Anja Jentzsch

Cloud

On- premise

Client

My Data

Enterprise Data

Social netw

orks

Web Data

TextData

People

Media

Apps

Behavior

Ads

ContentFusion

Contextual Matching

PervasiveSearch

INTENT

CONTENT

Smart Content needs Gardeners

From Documents to KnowledgeVa

lue

DocumentSearch

Finds documents containing terms

RelationshipExtraction

Finds relationships within documents

AssertionClustering

Finds assertions and the evidence for them

Profiling

Summarizes different kinds of information

Join

Creates indirect correlations and connections

Knowledge Management Framework

SocialIndividual

History

Event(transaction)

MappingContent management

Standardization

Findability

Common ground (practices, values, belief)

Typologies

Sense-makingCategory busting

Discovery

Coordination

Based on Organizing Knowledge: Taxonomies, Knowledge and Organizational Effectiveness, Patrick Lambe (not exact reproduction)

Culture

Collaboration

Expertise and learning

Information

Communities of Practice

Summary

Content Analytics involves• Gardeners• Context• Virtuous Cycles• Lots of cool, imperfect

technology

Smart Content is• Social• Mobile• Streaming• Exploding

jeff.fried@ba-insight.net 64

Q&A

jeff.fried@ba-insight.com