Spam in Blogs and Social Media Pranam Kolari, Tim Finin Akshay … · Affiliate Programs Context...
118
Spam in Blogs and Social Media Pranam Kolari, Tim Finin Akshay Java, Anupam Joshi March 25, 2007
Transcript of Spam in Blogs and Social Media Pranam Kolari, Tim Finin Akshay … · Affiliate Programs Context...
Sp
am i
n B
log
s an
d
So
cial
Med
ia
Pra
nam
Ko
lari,
Tim
Fin
inA
ksha
y Ja
va, A
nup
am J
osh
i
March 25, 2007
Tu
tori
al O
utl
ine
•S
pam
on
the
Inte
rnet
–V
aria
nts
–S
oci
al M
edia
Sp
am
•R
easo
n b
ehin
d S
pam
in B
log
s
•D
etec
ting
Sp
am B
log
s
•T
rend
s an
d Is
sues
•H
ow
can
you
hel
p?
•C
on
clu
sio
ns
Pre
sen
ters
Pra
nam
Ko
lari
is a
UM
BC
PhD
stud
ent.
His
dis
sert
atio
n is
on
spam
blo
g de
tect
ion
, with
tool
s d
evel
oped
in u
se b
oth
by
acad
emia
an
d in
dus
try.
He
has
act
ive
rese
arch
inte
rest
in
inte
rnal
cor
pora
te b
logs
, th
e S
eman
tic
Web
and
blo
g a
nal
ytic
s.
Aks
hay
Jav
a is
a U
MB
C P
hD s
tud
ent.
His
dis
sert
atio
n is
on
iden
tify-
ing
influ
ence
an
d op
inio
ns in
soci
al m
edia
. His
res
earc
h in
tere
sts
incl
ud
e bl
og a
nal
ytic
s, in
form
atio
n
retr
ieva
l, n
atur
al la
ngu
age
pro
cess
ing
an
d th
e S
eman
tic W
eb.
Tim
Fin
in is
a U
MB
C P
rofe
sso
rw
ith o
ver
30
year
s of
exp
erie
nce
in th
e ap
ply
ing
AI t
o in
form
atio
n
syst
ems,
inte
llig
ent i
nte
rfac
es a
nd
robo
tics.
Cur
ren
t in
tere
sts
incl
ud
e so
cial
m
edia
, th
e S
eman
tic W
eb a
nd m
ulti
-ag
ent s
yste
ms.
Anu
pam
Jos
hi i
s a
UM
BC
Pro
-fe
ssor
with
res
earc
h in
tere
sts
inth
e br
oad
are
a o
f net
wo
rked
com
pu
ting
an
d in
telli
gen
t sys
tem
s. H
e cu
rren
tly s
erve
s on
the
edito
rial b
oar
d of
th
e In
tern
atio
nal J
our
nal
of
the
Sem
antic
W
eb a
nd In
form
atio
n.
SP
AM
!
E-m
ail
Sp
am
•E
arly
form
see
n a
roun
d 1
992
with
MA
KE
M
ON
EY
FA
ST
•8
0-85
% o
f all
e-m
ail t
raffi
c is
spa
m
•In
num
ber
s
Sou
rces
: Ir
onP
ort,
Wik
iped
iaht
tp:/
/ww
w.ir
onpo
rt.c
om/c
omp
any/
ironp
ort
_pr_
200
6-06-28.
htm
l
200
5 -
(Ju
ne)
30 b
illio
n p
er d
ay2
006
-(J
une
) 55
bill
ion
per
day
200
6 -
(Dec
embe
r) 8
5 b
illio
n pe
r d
ay
200
7 -
(Feb
ruar
y) 9
0 b
illio
n p
er d
ay
Wh
at i
s S
pam
?
•“U
nso
licite
d u
sual
ly c
om
me
rcia
l e-m
ail s
ent t
o a
lar
ge
nu
mb
er o
f ad
dre
sses
” –
Mer
riam
Web
ster
On
line
•“S
pam
min
g is
the
abu
se o
f ele
ctro
nic
mes
sag
ing
sy
stem
s to
sen
d u
nso
licite
d b
ulk
mes
sag
es,
wh
ich
ar
e al
mo
st u
niv
ersa
lly u
nd
esire
d.”
–W
ikip
edia
•A
s th
e In
tern
et h
as s
uppo
rted
new
app
licat
ions
, m
any
othe
r fo
rms
are
com
mon
, req
uirin
g a
muc
h br
oade
r de
finiti
on
Capturing user attention unjustifiably on
the Internet (E-mail, Web, Social Media
etc..)
Direct
Indirect
E-M
ail
Spa
m
Ge
nera
l We
b S
pam
Spa
m B
logs
(S
plog
s)
Tax
on
om
y o
f S
pam
IM S
pam
(S
PIM
)
spa
md
exin
g
Internet Spam
(Fo
rms)
(Mec
han
ism
s)
Com
mun
ity S
pam
Com
me
nt S
pam
Ta
g S
pam
Sp
amd
exin
g
•C
om
pro
mis
es R
elev
ance
an
d Im
po
rtan
ce S
core
s u
sed
by
Sea
rch
En
gin
es•
Mo
stly
eff
ects
the
lon
g ta
il o
f key
wo
rds
(mo
re
susc
eptib
le)
•S
pam
in B
log
s is
a fo
rm o
f sp
amd
exin
g•
20
% o
f th
e in
dex
ed W
eb (
20
05)
“We use the term spamming (also, spamdexing) to
refer to any deliberate human action that is meant to
trigger an unjustifiably favorable relevance or importance
for a page, considering the page’s true value”
–Gyongyiet al
Direct
Indirect
E-M
ail
Spa
m
Ge
nera
l We
b S
pam
Spa
m B
logs
(S
plog
s)
Tax
on
om
y o
f S
pam
IM S
pam
(S
PIM
)
So
cia
l M
edia
Sp
am
Internet Spam
(Fo
rms)
(Mec
han
ism
s)
Com
mun
ity S
pam
Com
me
nt S
pam
Ta
g S
pam
DIG
G S
pam
•S
ybil
atta
ck
–“i
n co
mpu
ter
secu
rity
is a
n at
tack
whe
rein
a r
eput
atio
n sy
stem
is s
ubve
rted
by
forg
ing
iden
titie
s in
pee
r-to
-pee
rne
twor
ks”
–W
ikip
edia
•U
sers
that
sel
l–
http
://us
ersu
bmitt
er.c
om $
20, p
lus
$1 p
er d
igg
–S
pike
the
vote
, site
sol
d on
eB
ay, b
ough
t by
digg
use
r,
final
ly s
hut d
own
by d
igg
–G
eekf
orlif
e, a
top
100
digg
user
, sol
d hi
s ac
coun
t fo
r $8
22
on e
Bay
•In
ad
diti
on
to d
irect
sp
am, D
IGG
po
pu
lar
pag
e al
so
resu
lts in
hig
h r
anki
ng o
n s
earc
h e
ng
ines
(S
pam
dex
ing
)
“Why
Are
Peo
ple
Fas
cin
ated
By
Ph
oto
gra
phs
of C
row
ds
?”•
Au
tho
r S
ub
mits
Sto
ry•
4 h
ou
rs la
ter
on
ly o
ne D
IGG
on
the
sto
ry•
See
ks o
ut U
ser/
Su
bm
itter
•1
2 h
ou
rs la
ter
a D
IGG
Po
pu
lar
Sto
ry
Tag
Sp
am
•S
oci
al B
ook
mar
k T
ools
–d
el.ic
io.u
s sp
am in
lon
g ta
il
–F
url
po
pul
ar p
age
spam
•A
lso
use
d fo
r sp
amde
My
Blo
gL
og
Sp
am
•F
ake
Pro
file
Imag
e
•F
ake
use
rs
•F
ake
site
vis
its
•F
ake
co-a
uth
ors
•A
dd fr
ien
ds
•C
om
men
t Sp
am
Wid
get S
pam
Adm
iratio
n S
pam
!?
My
Sp
ace
Sp
am
•M
ySp
ace
is th
e (5th
) 4th
hig
hes
t vis
ited
site
•D
irect
Use
r T
arg
etin
g–
Fre
e P
rod
uct
s
–C
AM
S
Th
e H
oo
dia
Sca
m•
Cre
ate
an “
inte
rest
ing
” p
rofil
e•
Ag
gre
ssiv
ely
add
frie
nd
s (+
Syb
il A
ttack
s)•
Wal
l Ho
od
iaC
om
men
ts
Ork
ut
Sp
am
•C
om
mu
nity
Orie
nted
Sp
am e
xten
ds
acro
ss
alm
ost
all
too
ls
Oth
er f
orm
s o
f sp
am
It d
oes
not
take
lon
g fo
r sp
amm
ers
to e
xplo
it a
new
fo
rm
of s
oci
al m
edia
–W
iki(p
edia
) S
pam
the
wik
ipe
dia
com
mu
nity
dele
tes
spam
art
icle
s an
d p
ag
es
–G
uest
book
Spa
mye
s, s
om
e s
till h
ave
gu
est
bo
oks
an
d t
he
y attra
ct s
pa
m e
ntr
ies
–IR
C s
pam
bo
ts o
n I
RC
ch
an
nels
sen
d y
ou
un
wa
nte
d m
ess
age
s–
Sec
ond
life
spam
no
teca
rds
po
p u
p w
ith u
nw
an
ted
mess
ag
es
–T
witt
er s
pam
it a
pp
ea
red
ab
ou
t a
we
ek
afte
r T
witt
er
we
nt
big
tim
e–
Sem
antic
web
spa
mth
is o
ne
ha
s n
ot
yet
be
en
se
en
Sp
amd
exin
g a
nd
its
re
lati
on
to
Blo
gs
•Id
entif
y P
rofit
able
Co
nte
xts
•C
reat
e m
ulti
ple
doo
rway
pag
es (
or
blo
gs)
•S
pam
dex
do
orw
ays
–T
arge
t Rel
evan
ce S
core
s•
TF
IDF
•
Met
a-ta
g
–T
arge
t Im
port
ance
Sco
res
•P
ageR
ank
•(O
r) G
am
e S
po
nso
red
Sea
rch
–S
earc
h E
ngin
e A
rbitr
age
•M
on
etiz
e–
Affi
liate
Pro
gram
s–
Con
text
ual A
dver
tisem
ents
An
ato
my
of
Sp
amd
exin
g
Iden
tify
Pro
fita
ble
C
on
tex
ts
Doo
rway
Pag
e fo
r“S
tud
ent L
oan
C
on
solid
atio
n”
10K
in-li
nks
!
Bro
wse
r Ja
vasc
ript
Red
irect
for
Use
rs
eva
l(une
sca
pe("
%64
%6f
%63
%75
%6d
%65
%6e
%74
%2e
%77
%72
%69
%74
%65
%28
%22
%3c
%73
%63
%72
%69
%70
%74
%20
%73
%72
%63
%3d
%27
%68
%74
%74
%2
2%2b
%22
%70
%3a
%2f
%2f
%62
%69
%67
%22
%2b
%22
%68
%71
%2e
%69
%6
e%6
6%6f
%2f
%73
%74
%61
%74
%73
%32
%2e
%70
%68
%70
%3f
%69
%22
%2b
%2
2%6
4%3d
%31
%32
%26
%67
%72
%6f
%75
%70
%3d
%32
%27
%3e
%3c
%2f
%73
%6
3%7
2%69
%70
%74
%22
%2b
%22
%3e
%22
%29
%3b
"))
doc
umen
t.writ
e(“
<sc
ript
src=
'htt"
+"p
://b
ig"+
"hq
.info
/sta
ts2.
php?
i"+
"d=
12&
gro
up=
2'>
");
Cry
pti
c Ja
vaS
crip
t R
edir
ects
“LE
AD
S”
to a
ffilia
tes
hig
hly
pro
fitab
le
An
ato
my
of
Sp
amd
exin
g
Sp
am p
ages
,S
pam
Blo
gs,
Sp
am C
om
men
ts,
Gu
estb
ook
Sp
amW
ikiS
pam
SE
RP
Sea
rch
Eng
ines
Affi
liate
Pro
gram
sC
ont
ext A
ds
ads/
affil
iate
link
sar
bitr
age
in-li
nks
spam
dexJa
vaS
crip
t Red
irec
t[P
revi
ousl
y C
loak
ing]
Affi
liate
Pro
gram
B
uye
rs
Sp
am p
ages
,S
pam
Blo
gs
[DO
OR
WA
Y]
Wh
y B
log
s u
sed
in
Sp
amd
exin
g
•S
pam
Blo
gs
–Q
uick
ly in
dexe
d by
sea
rch
engi
nes
(pin
g se
rver
s)
–C
an b
e ho
sted
on
thir
d pa
rty
serv
ices
with
hig
h au
thor
ity a
nd
trus
t (e.
g. b
logs
pot)
–S
earc
h en
gine
s sh
ow a
hig
her
pref
eren
ce to
blo
gs –
craw
ling
and
rank
ing
•S
pam
Co
mm
ents
–In
-link
s fr
om h
igh
auth
orin
g bl
og s
ites
inde
xed
freq
uent
ly
–“n
o-fo
llow
” no
t uni
vers
ally
use
d
•In
bo
th c
ases
blo
gs p
rovi
de
quic
ker
retu
rn o
n
inve
stm
ent t
o s
pam
mer
s
Co
mm
ent
Sp
am i
n N
um
ber
s
La
nsin
g, M
I5.
Orla
ndo,
FL
4.
San
Fra
ncis
co, C
A3.
Wa
shin
gton
DC
2.
Mou
ntai
n V
iew
, CA
1.
56%
of a
ll bl
ogs
are
splo
gs!
Sili
con
Val
ley
or S
plog
Val
ley?
Sp
log
s in
Nu
mb
ers
•S
am
ple
d w
eblo
gs.
com
du
ring
the
last
wee
k o
f Jan
uar
y
•8
.3 M
illio
n P
ing
s
•U
sed
exi
stin
g s
plog
det
ectio
n to
ols
•A
rou
nd
10%
err
or
rate
auto
buy
calif
orni
aca
ncer
card
casi
noch
eap
cons
olid
atio
ncr
edit
debt
diet
disc
ount
equi
pmen
t
esta
tefin
ance
florid
afo
rexfr
eegi
ftgo
lfhe
alth
hote
linsu
ranc
eje
wel
ryla
wye
rloa
nlo
ans
med
ical
mon
eym
ortg
agene
won
line
phon
epo
kerr
enta
lsal
eso
ftwar
ete
xas
trad
ing
trav
elus
edva
catio
nvi
deo
wed
ding
Hig
h P
PC
con
text
s ar
e pr
imar
y sp
am d
river
s
Sp
log
s –
Co
nte
xt
Th
e B
log
ger
Ep
iso
de
Wh
y a
re p
eop
le c
om
pla
inin
g?
•S
plo
gs
red
uce
use
r tr
ust
on
so
cial
web
ra
nkin
g, m
akin
g th
em v
isit
web
-pag
es th
ey
can
wel
l do
with
out
•S
plo
g c
onte
nt i
s of
ten
pla
gia
rized
•S
plo
gs
dem
ote
val
ue
of a
uth
entic
con
ten
t•
Sp
log
s st
eal a
dve
rtis
ing
(ref
erra
l) re
ven
ue
fro
m a
uth
entic
con
ten
t pro
duc
ers
•S
plo
gs
stre
ss th
e bl
ogo
sph
ere
infr
astr
uct
ure
•S
plo
gs
skew
res
ults
of m
arke
t res
earc
h to
ols
So
urc
e o
f th
e P
rob
lem
“Ho
nest
ly,
Do
yo
u th
ink
peo
ple
wh
o m
ake
$
10
k/m
on
th fro
m a
dse
nse
ma
ke b
log
s m
an
ua
lly?
C
om
e o
n, th
ey
nee
d to
ma
ke t
hem
as
fast
as
po
ssib
le. S
ave
Tim
e =
Mo
re M
on
ey!
It's
C
om
mon
SE
NS
E! H
ow
mu
ch m
on
ey
do
yo
u th
ink
you
will
sa
ve if
yo
u c
an
incr
ea
se y
ou
r w
ork
pa
ce
by
a h
un
dre
d tim
es?
Th
ink
ab
ou
t it…
”
“Dis
cove
r T
he A
ma
zin
g S
tea
lth
Tra
ffic
Secr
ets
In
sid
ers
Use
To
D
rive
Th
ou
san
ds
Of T
arg
ete
d
Vis
itors
To
An
y S
ite T
hey
Desi
re!”
“Ho
ly G
rail
Of A
dve
rtis
ing
... “
“Ea
sily
Do
min
ate
An
y M
ark
et,
An
yS
ea
rch
En
gin
e, A
ny
Keyw
ord
.”
Sp
log
so
ftw
are
?!
$ 19
7
Our
spl
og b
ait
was
pic
ked
up
and
used
by
doze
ns o
f sp
logg
ers
Ho
w d
o t
hes
e to
ols
wo
rk?
RS
S M
agic
ian
•C
on
ten
t S
our
ce–
Pla
gia
rized
–A
rtic
le D
irec
torie
s
•C
on
ten
t Man
ipu
latio
n–
Dic
tion
ary
–W
ord
Sh
uffli
ng
–S
ente
nce
Sh
uff
ling
•M
ulti
-ling
ual S
upp
ort!
•H
ow
ever
, Pro
-sp
logg
ers
don’
t use
thes
e to
ols
Ch
arac
teri
stic
s o
f S
pam
(B
log
s)
Wh
ere
do
th
ey a
pp
ear?
Update Pings
Update Pings
Ping Stream
1
2
Update Stream
Fetch Content
3
4
12
34
()
No
n-B
log
pin
gs
at w
eblo
gs.
com
•N
o R
SS
Fee
ds
•N
o D
ated
En
try
, n
o c
om
men
ts
Key
wo
rd S
tuff
ed B
log
•‘c
ou
po
n c
od
es’,
‘ca
sin
o’
Po
st S
titc
hin
g•
Ex
cerp
ts s
crap
ed f
rom
oth
er s
ou
rces
Po
st W
eav
ing
•S
pam
Lin
ks
con
tex
tual
ly p
lace
d i
n p
ost
Lin
k-r
oll
sp
am
•W
ith
fu
lly
pla
gia
rize
d t
ext
Sp
log
Ex
per
imen
t
•C
aptc
ha
Bre
aker
•C
ompu
ter
•C
onte
nt•
Affi
liate
Acc
oun
ts
Th
e S
plo
g D
etec
tio
n
pro
ble
m
Th
e S
plo
g D
etec
tio
n P
rob
lem
Giv
en a
blo
g id
entif
ied
by
its
UR
L X
, is
it sp
am?
Ho
w is
this
diff
eren
t fro
m o
ther
spam
det
ectio
n p
rob
lem
s?
Wha
t and
Whe
n of
filte
ring
E-m
ail S
pam
Web
Spa
mS
ocia
l Net
wor
kS
pam
Spa
m B
logs
time
tim
e
po
sts
time
frie
nd
s
Con
stra
ints
on
the
Filt
er
•U
sers
•E
-ma
il S
ervi
ceP
rovi
de
r
E-m
ail S
pam
Web
Spa
mS
ocia
l Net
wor
kS
pam
Spa
m B
logs
•S
earc
h E
ng
ines
•P
age
Ho
stin
g
Ser
vice
s (e
g. T
ripo
d)
•W
eb S
earc
h E
ng
ines
•B
log
Sea
rch
En
gin
es•
Blo
g H
ost
ing
Ser
vice
s•
(Pin
g S
erve
rs)
•C
om
mu
nity
Site
s
Who
act
ivel
y us
es it
?
•F
ast
Det
ectio
n•
Low
Ove
rhea
d•
On
line
•B
atch
De
tect
ion
•M
ost
ly O
fflin
e•
Fas
t D
etec
tion
•Lo
w O
verh
ead
•F
ast
Det
ectio
n•
Bat
ch D
ete
ctio
n
Adv
ersa
rial A
ttack
s on
the
Filt
erE
-mai
l Spa
mW
eb S
pam
Soc
ial N
etw
ork
Spa
mS
pam
Blo
gs
•Im
age
Sp
am
•C
har
acte
r R
epla
cem
ent
•S
crip
ts•
Do
orw
ays
•S
crip
ts•
Do
orw
ays
•D
ecep
tive
Beh
avio
r
•D
ecep
tive
Beh
avio
r
Det
ecti
on
Bac
kg
rou
nd
•P
re-in
dex
ing
at P
ing
Str
eam
•P
ost
-inde
•B
lack
lists
/Wh
itelis
ts
•U
RL
Bas
ed F
eatu
res
•H
ome-
Pag
e B
ased
Fea
ture
s
•F
eed
Bas
ed F
eatu
res
(Tem
pora
l)
•Li
nk B
ased
Fea
ture
s
Det
ecti
on
Bac
kg
rou
nd
Filt
ers
dep
loye
d o
n P
ing
Str
eam
s (t
ypic
ally
)..
Det
ecti
on
Ap
pro
ach
esth
at W
ork
IP B
ased
Bla
ckli
sts
•In
tuiti
on
: Mo
st s
elf-
ho
sted
blo
gs
are
tied
to
“sp
amm
er-f
rien
dly”
dom
ain
host
ing
ser
vice
s
•M
ap P
ing
UR
L’s
to IP
Add
ress
es
•R
ank
IP A
ddre
sses
by
ho
sted
blo
gs
•P
ick
the
top
100
(sa
y)–
Ran
do
mly
sam
ple
for
a su
bse
t of b
logs
–V
erify
no
fals
e p
osi
tives
–A
dd
IP to
Bla
cklis
t
•A
ctiv
ely
use
d by
a2
b.cc
and
man
y o
ther
s
Pin
g S
trea
m a
t W
eblo
gs.
com
Mod
els
over
na
me
and
UR
L va
lues
can
be v
ery
effe
ctiv
e
Reg
ex F
ilte
rs o
n (
Nam
e) U
RL
•T
he
Blo
gsp
ot“
?” fi
lter
•T
he
hyp
hen
filte
r
•..
and
man
y m
ore
•V
ery
effe
ctiv
e an
d fa
st
•A
rea
ctiv
e m
easu
re
•A
ctiv
e S
upe
rvis
ion
•H
igh
ly s
usc
eptib
le to
Ad
vers
aria
l Atta
cks
Pin
g S
trea
m a
t W
eblo
gs.
com
Tex
t tok
en b
ase
d fe
atur
es
over
na
me
and
UR
L ca
nbe
ver
y ef
fect
ive
UR
L *
On
ly*
Fea
ture
s
•In
tuiti
on
: Sp
amm
ers
targ
et s
earc
h e
ngin
es
thro
ugh
co
nte
xt r
ich
UR
Ls•
No
pag
e fe
tch
es –
can
be
very
fast
•T
ech
niq
ues
vary
–H
ow
are
UR
Ls s
egm
ente
d
•H
yphe
ns, f
orw
ard
slas
hes
etc.
.•
Sup
ervi
sed
Seg
men
tatio
n T
echn
ique
•C
hara
cter
N-g
ram
s
–H
ow
are
det
ectio
n m
od
els
con
stru
cted
•S
VM
s•
Bay
esia
n F
ilter
s
No
vel
UR
L S
egm
enta
tio
n
•M
otiv
atio
n: S
pam
mer
s g
lue
wo
rds
in U
RL
•Le
arn
Seg
men
tatio
n u
sing
sp
amm
ing
con
text
s as
pro
pose
d b
y S
alve
ttiet
al
Sal
vetti
et a
l
http
://di
etst
hatw
ork.
blog
spot
.com
http diet
stha
twor
k
blog
spot co
mht
tp
diet
sth
atw
ork
blog
spot
com
Naï
ve A
ppro
ach
Enh
ance
d A
ppro
ach
No
vel
UR
L S
egm
enta
tio
n
•T
rain
a B
ayes
ian
Cla
ssifi
er (
10K
+, 1
0K -
)
•1
k te
st s
amp
les
•F
-mea
sure
76
% s
pam
, 79
% a
uth
entic
•H
uman
Bas
elin
e 7
3%
spa
m, 7
8%
au
then
tic
•C
lass
ifier
bea
ts h
uman
s w
hen
iden
tifyi
ng s
pam
u
sing
UR
Ls a
lone
!
UR
L+
Nam
e
blo
gdr
ive
web
log
indi
atim
esp
ublic
nam
ep
odca
stra
dio
topi
xd
irect
ory
info
med
ical
rent
alfo
rex
insu
ran
cew
eddi
ngse
oau
toch
ann
el
•S
VM
bas
ed a
pp
roac
h•
Han
d V
erif
ied
Sam
ple
s o
f 2K
sp
am
UR
Ls a
nd
2K
au
then
tic U
RLs
fro
m
pin
g s
erve
r
•N
-ch
arac
ter
gra
m to
ken
izat
ion
•Li
nea
r K
ern
el•
Bin
ary
Fea
ture
s
•F
-mea
sure
~ 9
0%
pro
gram
bra
nd
con
sum
erto
day
use
rsjo
urn
alb
log
blo
gsp
otty
pep
ad
do
mai
nan
swer
gro
ups
ford
acn
ere
view
slo
ansh
are
hoo
dia
Blo
g H
om
e-P
age
Bas
ed F
eatu
res
•In
tuiti
on
: Hom
e-P
age
can
pro
vid
e a
goo
d
snap
sho
t of a
blo
g (
mo
st r
ecen
t po
sts)
, and
als
o
cap
ture
s b
log-
rolls
and
link
-ro
lls
•O
nly
one
pag
e fe
tch
per
UR
L –
can
be
very
fast
•T
ech
niq
ues
vary
by
sele
cted
feat
ure
s–
Bag
-of-
wo
rds
–B
ag-o
f-to
ken
ized
-ou
tlin
ks
–T
ext C
om
pre
ssio
n
Scr
aped
con
tent
typi
cally
lack
s th
e pe
rson
al n
atur
e of
blo
g po
sts
Bag
-of-
Wo
rds
Fea
ture
•S
VM
bas
ed m
od
el–
Bin
ary
Fea
ture
s–
Line
ar K
erne
l
•H
and
ver
ified
trai
ning
set
of b
log
s an
d s
plo
gs
–70
0 bl
ogs
and
700
splo
gs•
F m
easu
re ~
88
%
we
wh
atw
asm
yo
rgfli
ckr
pap
erw
ord
s
find
info
new
syo
ur
anot
her
web
site
bes
tar
ticle
s
web
log
mo
tion
me
than
kg
oja
nu
ary
trac
kbac
kar
chiv
es
per
fect
pro
duct
su
nca
tego
rize
dh
otre
sou
rces
inc
thre
eco
pyr
ight
Bag
-of-
2-W
ord
s F
eatu
re 2-c
om
men
tsto
-th
eit-
was
also
-ad
ded
1-c
om
men
t3
-co
mm
ents
com
men
ts-o
ffin
-un
cate
goriz
edca
tego
ries-
unca
tego
rized
new
-yo
rk
•F
eatu
res
crea
ted
usi
ng tw
o
adja
cen
t wo
rds
•Le
ss s
usc
eptib
le to
ad
vers
aria
l at
tack
s•
SV
M b
ased
mo
del
–B
inar
y F
eatu
res
–Li
near
Ker
nel
•F
mea
sure
~ 8
6%
Bag
-of-
UR
L T
ok
ens
Fea
ture
•F
eatu
res
con
stru
cted
by
toke
niz
ing
ou
tgo
ing
UR
Ls
•S
imila
r to
(ea
rlier
) U
RL
on
ly m
ode
ls b
ut c
aptu
res
mor
e co
ntex
t–
UR
L to
ken
s in
the b
adge
cr
azy
blo
gosp
her
e
–D
efau
lt W
ord
pre
ss L
inks
•F
-mea
sure
~ 8
2%
tech
nora
tifli
ckr
feed
bur
ner
blo
gro
llin
g… d
oug
alb
oren
zed
1p
hoto
mat
t
Mo
re H
om
e-P
age
Bas
ed F
eatu
res
•W
e ha
ve b
een
expe
rimen
ting
with
mul
tiple
app
roac
hes
•D
atas
et a
vaila
ble
at:
–ht
tp://
ebiq
uity
.um
bc.e
du/r
esou
rce/
htm
l/id/
212
Pos
t 1 fr
om a
Spa
m B
log
onF
orex
.. T
radi
ng
Pos
t 2 fr
om th
e sa
me
Sp
am B
log
on F
orex
.. T
rad
ing
Tem
po
ral
Reg
ula
riti
es
•In
tuiti
on
: Sp
log
s fe
atur
e h
igh
ly c
orre
late
d
con
ten
t acr
oss
mul
tiple
po
sts
(tex
t, ou
t-lin
ks
etc.
.)
•Li
n e
t al h
ave
pro
po
sed
and
an
alyz
ed u
sefu
l m
etric
s to
iden
tify
such
cor
rela
tion
s
•T
wo
fetc
hes
per
UR
L –
can
exp
loit
stru
ctur
ed
feed
en
trie
s
•P
reci
sio
n ~
86
%, R
ecal
l ~ 4
7%
Oth
er f
eatu
res
that
wo
rk
•O
utg
oin
g A
ncho
rs
•C
har
acte
r N
-gra
ms
and
Wor
d N
-gra
ms
•N
amed
En
tity
Rat
io
•T
ext C
ompr
essi
on R
atio
•P
rono
un E
ntit
y R
atio
Det
ecti
on
Ap
pro
ach
esth
at c
ou
ld a
lso
Wo
rk
Pin
g t
imes
–It
alia
n B
log
s
Sp
ing
vs.
Pin
g t
imes
Sp
ing
s v
s. P
ing
s: f
req
uen
cyb
log
s v
s. t
hei
r p
ing
fre
qu
ency
fo
llo
ws
a p
ow
er
law
, bu
t sp
log
s v
s. s
pin
gs
do
es n
ot
http
://w
ww
.en
gad
get.c
om
19
42
http
://w
ww
.hu
ffin
gto
npo
st.c
om
/th
eblo
g9
05
http
://w
ww
.cro
oks
and
liars
.co
m6
37
http
://b
logs
.gu
ard
ian.
co.u
k/n
ew
s
616
http
://w
ww
.littl
egr
ee
nfo
otb
alls
.co
m/w
eblo
g
6
11
http
://sp
ace
s.m
sn.c
om
/me
mb
ers
/po
ny-
girl
50
5ht
tp://
spac
es.
msn
.co
m/m
em
be
rs/b
lack
-pus
s
505
http
://sp
ace
s.m
sn.c
om
/me
mb
ers
/am
put
ee
-wo
me
n
5
05
http
://sp
ace
s.m
sn.c
om
/me
mb
ers
/fre
e-s
tori
es
50
5ht
tp://
spac
es.
msn
.co
m/m
em
be
rs/fi
rst-
time
-gir
l 5
05
Top
5
Top
5
Sp
log
s v
s. B
log
s –
In-d
egre
e
Sp
log
s v
s. B
log
s –
Ou
t-d
egre
eht
tp://
ww
w.x
ang
a.co
m/h
om
e.as
px?
use
r=hi
t_m
e_la
yout
z2
73
http
://w
ww
.xa
nga.
com
/ho
me.
asp
x?u
ser=
i_jo
ck_
layo
uts
27
1ht
tp://
ww
w.x
ang
a.co
m/h
om
e.as
px?
use
r=sl
p_
layo
uts_
slp
19
8ht
tp://
spac
es.
msn
.co
m/m
em
be
rs/c
yru
stse
19
86
1
93
http
://w
ww
.xa
nga.
com
/ho
me.
asp
x?u
ser
=la
yout
s_n_
cod
es2
00
5
18
0
http
://w
orl
dse
ries
ofp
oke
rchi
psc
ard
gu
ard
.blo
gsp
ot.c
om
89
8ht
tp://
rule
-wso
p.b
logs
po
t.co
m8
98ht
tp://
wo
rld
seri
es-o
fpo
ler.
blo
gsp
ot.c
om
89
8ht
tp://
qso
pco
m-1
.blo
gsp
ot.c
om
8
98
http
://w
eop
com
.blo
gsp
ot.c
om
898
Top
5
Top
5
Sp
log
Pro
tect
ion
–G
lob
al
•T
est s
et o
f 70
0 a
uth
entic
blo
gs a
nd
70
0 s
plo
gs
•In
-link
s fr
om
Tec
hn
ora
ti •
Ou
t-lin
ks fr
om
blo
g-h
om
e p
age
•La
bel
ed in
-lin
ks/o
ut-
links
to b
e a
blo
g, s
plo
g, n
ew
s si
te, o
ther
web
site
•3
feat
ure
-typ
es tr
ain
ed u
sing
SV
Ms
–In
-link
dis
trib
utio
n–
Out
-link
dis
trib
utio
n–
Co-
cita
tion
dist
ribut
ion
•F
-mea
sure
~ 8
0%
Su
mm
ary
of
Ex
isti
ng
Tec
hn
iqu
es
•L
oca
l P
rop
erti
es w
ill
con
tin
ue
to h
old
th
e k
ey –F
ast
–L
ow
Co
st
–C
an
be
easi
ly i
nco
rpo
rate
d i
nto
blo
g h
arv
esti
ng
(c
raw
lin
g)
•G
lob
al T
ech
niq
ues
res
tric
ted
by
co
ver
age
–L
ess
Eff
ecti
ve
to B
log
Sea
rch
En
gin
es
–C
on
tin
ue
to b
e ef
fect
ive
for
web
sea
rch
en
gin
es
BLACKLISTS
WHITELISTS
BLACKLISTS
WHITELISTS
TEMPORAL
MODEL
TEMPORAL
MODEL
TEXT
MODEL
TEXT
MODEL
REGEX
REGEX
URL
MODEL
URL
MODEL
Pin
g S
trea
m
Ch
eck
wit
h
know
n b
log
s an
d s
plo
gs
Ch
eck
know
nre
gex
p
atte
rns
Use
mo
del
s o
ver
text
of
UR
L
Use
mo
del
s o
ver
Blo
g
ho
me-
pag
e
Use
mo
del
s o
ver
po
st
corr
elat
ion
Ho
w T
ech
niq
ues
are
Use
d
Incr
easi
ng
co
st
Eli
min
atio
n A
pp
roac
hes
(M
ost
Do
n’t
Wo
rk)
“So
, wh
ats
that
me
an f
or
the
cu
rren
t u
sers
? It
me
ans
th
at w
e h
ave
a fe
w t
hin
gs
to ir
on
ou
t. W
e ar
e w
ork
ing
on
the
m s
afa
st
as p
oss
ible
. It
will
insu
re t
hat
we
do
nth
ave
thes
e is
sues
go
ing
fo
rwar
d in
the
fu
ture
an
d h
ave
a s
olid
so
lutio
n. “
SP
LOG
RE
PO
RT
ER
ST
AT
US
= R
ED
“D
ue
to la
ck o
f res
ourc
es, m
ainl
y tim
e, w
e h
ave
dec
ided
to p
urs
ue
oth
er o
ther
web
2.0
inte
rest
s. W
e h
ad a
goo
d r
un a
nd
app
reci
ate
ever
yon
e w
ho s
upp
orte
d th
e sp
log
fig
htin
g m
ove
men
t. W
e h
ave
dec
ided
to k
eep
the
site
up
so
as
not t
o "
brea
k th
e w
eb"
and
all
the
links
to th
e si
te. I
f any
on
e is
inte
rest
ed in
acq
uirin
g S
plo
g R
epor
ter
ple
ase
cont
act u
s.”
FIG
HT
ING
SP
LO
G
Elim
inat
ing
Sp
logs
dep
ends
dire
ctly
on
b
lack
listin
g by
sea
rch
eng
ines
, co
nte
xtua
l ad
vert
isem
ent a
nd a
ffilia
tes.
Sp
log
gers
hav
e n
o m
ore
ince
ntiv
e to
ho
st s
uch
blo
gs
and
ty
pic
ally
brin
g it
do
wn
.
... a
rat
her
slo
w e
limin
atio
n p
roce
ss.
Ho
w S
plo
gs
Eff
ect
Blo
g A
nal
yti
cs
Nat
ure
of S
plog
s in
TR
EC
200
6
•A
roun
d 83
K id
entif
iab
le b
log
s in
the
colle
ctio
n, w
ith 3
.2M
per
mal
inks
•W
e id
entif
ied
13,5
42
splo
gs
•B
lack
liste
d 5
43K
per
mal
inks
from
thes
e sp
log
s
•T
his
acc
oun
ts to
16
% o
f th
e en
tire
colle
ctio
n
•R
esu
lts ta
lly in
% to
TR
EC
dat
aset
(M
acdo
nald
et a
l 200
6)
Impa
ct o
f Spl
ogs
in T
RE
C Q
uerie
s
Dis
trib
uti
on
of
Sp
log
s th
at
ap
pe
ar
TR
EC
qu
eri
es
020406080100
120
51
01
52
02
53
03
54
04
55
05
56
06
57
07
58
08
59
09
51
00
To
p 1
00
Se
arc
h R
esu
lts
ran
ke
d u
sin
g T
FID
F S
cori
ng
Number of Splogs
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
Am
eric
an I
do
l
Ch
ole
ster
ol
Hy
bri
d C
ars
Hig
her
in S
pam
Pro
ne C
onte
xts
Sp
log
Dis
trib
uti
on
fo
r 'S
pa
m T
erm
s'
020406080100
120
510
1520
2530
3540
4550
5560
6570
7580
8590
9510
0
Se
arc
h R
esu
lt R
an
k
Number of Splogs
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
Sp
am q
uer
y t
erm
s b
ased
on
an
aly
sis
by
McD
on
ald
et
al 2
006
..
Car
d
Inte
rest
Mo
rtg
ag
e
Inp
ut
fro
m G
rou
ps
Dea
lin
g w
ith
Sp
log
s
NE
C –
Blo
g A
nal
yti
cs•
Whe
n di
d yo
u st
art s
eein
g sp
am?
in
200
5 –
on b
log
sear
ch
engi
ne r
esul
ts.
•H
ow
muc
h sp
am?
(in p
erce
nts)
rou
ghly
25%
of t
op
retr
ieva
l res
ults
•Is
it in
crea
sing
/dec
reas
ing?
in
crea
sing
•H
ow
are
you
tack
ling
it? a
nd H
ow m
uch
effo
rt g
oes
into
it?
defin
itely
my
team
con
side
rs th
is a
n im
port
ant r
esear
ch
effo
rt. I
n te
rms
of p
erso
n-hr
s, a
t lea
st 2
080
in 2
006•
Do
you
have
any
new
arc
hite
ctur
es/fr
amew
orks
/initi
ativ
es?
Y
es –
wor
k pu
blic
ly a
vaila
ble
•W
hat w
ould
you
wan
t res
earc
hers
to a
ddre
ss?
te
mpo
ral a
nd
stru
ctur
al p
rope
rtie
s•
Wha
t are
futu
re tr
ends
? colla
bora
tive
splo
g de
tect
ion
Sp
her
e –
Blo
g S
earc
h•
Wh
en d
id y
ou s
tart
see
ing
sp
am?
La
te 2
005.
Lat
e su
mm
er o
f 200
2 co
uple
of y
ears
han
d ed
ited
blac
k lis
ts, U
RL
patte
rn fil
terin
g•
Ho
w m
uch
sp
am
? (in
per
cent
s) b
ig n
on-b
log
cate
gory
, aro
und
70%
fr
om w
eblo
gs.c
om,
•Is
it in
crea
sin
g/d
ecre
asin
g? No
sign
ifica
nt c
hang
e•
Ho
w a
re y
ou
tack
ling
it?
an
d H
ow
mu
ch e
ffort
go
es
into
it?
Aro
und
4160
pe
rson
-hrs
•A
ny
mes
sag
e(b
lurb
s) to
the
ICW
SM
au
die
nce
?
"At S
pher
e, w
e're
an
alyz
ing
the
mec
hani
sms
used
by
splo
gs a
nd fi
ndin
g d
etec
tabl
e si
gnat
ures
that
can
be
appl
ied
to v
ery
larg
e da
tasets
. It'
s ou
rco
nten
tion
that
ava
ilabl
e to
ols
and
prot
ocol
s m
ust n
eces
saril
y lim
it th
e st
rate
gies
ava
ilabl
e to
suc
cess
ful s
plog
gers
. W
hile
that
set
of
stra
tegi
es c
ontin
ues
to g
row
, we'
re c
ontin
ually
inno
vatin
g to
find
ne
w a
nd e
ffect
ive
way
s of
det
ectin
g th
ose
str
ateg
ies
, sup
pres
sing
sp
am, a
nd k
eepi
ng th
e bl
ogos
pher
e a
rich,
eas
ily n
avi
gabl
e un
iver
se
of in
form
atio
n"
Bu
zzM
etri
cs –
Blo
g A
nal
yti
cs•
Wh
en d
id y
ou
star
t see
ing
sp
am?
In e
arly
200
5, w
e st
arte
d to
see
the
#spa
m
blog
s an
d pi
ngs
from
non
-web
logs
sou
rces
ram
p ou
t rapid
ly.
•H
ow
mu
ch s
pam
? (
in p
erce
nts)
The
re's
a b
ig r
ange
, dep
ends
on
the
day.
The
re c
an b
e huge
spa
m a
ttack
s on
som
e da
ys. F
or to
day,
for
exam
ple:
15%
spa
m a
nd 3
0%
non
-blo
g (b
ut
toda
y's
15%
spa
m n
umbe
r is
on
the
low
sid
e) .
Thi
s do
esn't i
nclu
de o
ther
ki
nds
of b
ad p
ings
: dup
licat
es, o
ut-d
ated
, urls
that
are fo
rbid
den
or h
ave
no d
ata
•Is
it in
crea
sing
/dec
reas
ing?
It'
s in
crea
sing
in a
bsol
ute
coun
ts, j
ust l
ike
the
tota
l num
ber
of w
eblo
gs e
ver
crea
ted
is in
crea
sing
.How
ever
, in
term
s of
th
e nu
mbe
r of
act
ive
spam
web
logs
, it's
fairl
y co
nsta
nt, j
ust l
ike
the
num
ber
of a
ctiv
e E
nglis
h-la
ngua
ge w
eblo
gs is
fairl
y fla
t now
.
•D
o y
ou
hav
e an
y n
ew a
rchi
tect
ures
/fram
ewo
rks/
initi
ativ
es?
O
ur a
rchi
tect
ure
is in
pla
ce a
nd w
orki
ng w
ell.
Bu
zzM
etri
cs –
Blo
g A
nal
yti
cs•
Wh
at w
oul
d yo
u w
ant r
esea
rch
ers
to a
ddre
ss?
-
The
re is
a m
inor
ity o
f sop
hist
icat
ed s
pam
mer
s th
at crea
te s
leep
er s
pam
bl
ogs
that
look
just
like
reg
ular
blo
gs a
nd th
en m
onths
late
r st
art l
inki
ng
to a
ffilia
te s
ites.
How
to c
atch
thes
e ea
rly?
-H
ow to
det
ect r
epur
pose
d bl
og c
onte
nt (
dow
nrig
ht p
lagia
rism
eith
er fo
r us
e in
spa
m b
log
or s
impl
y fo
r th
e ag
e-ol
d re
ason
of bu
ildin
g on
e's
own
repu
tatio
n on
fals
e pr
emis
es).
-
Spa
m in
fore
ign
lang
uage
s: s
truc
tura
l sim
ilarit
ies/
diffe
renc
es/e
xten
t of
the
prob
lem
?
•W
hat
are
futu
re tr
end
s?
-P
ayF
orP
ostt
ype
serv
ices
: in-
post
adv
ertis
emen
ts th
at ind
istin
guis
habl
e fr
om b
logg
er c
onte
nt a
nd m
ay o
r m
ay n
ot b
e di
sclo
sed =>
nee
d fo
rsp
am
dete
ctio
n at
the
post
leve
l ins
tead
of s
impl
y at
the b
log
leve
l;ne
ed to
filte
r ou
t/dis
coun
t pos
ts w
heth
er th
an e
ntire
blo
gs.
Wh
at N
ext?
Ad
ver
sari
al
Cla
ssif
ica
tio
n
Ad
ver
sari
al C
lass
ific
atio
n
•A
ll fo
rms
of s
pam
det
ectio
n fi
t in
to a
gen
eral
cl
ass
of c
lass
ifica
tion
prob
lem
–C
lass
ifica
tion
is c
on
sid
ered
a “
gam
e”
bet
wee
n
clas
sifie
r an
d a
dve
rsar
y
–A
dve
rsar
ies
adap
t to
eva
de
filte
rs
•Its
cle
ar th
at a
cla
ssifi
er h
as to
lear
n to
cla
ssi
fy
new
form
s o
f sp
am–
Up
dat
e tr
ain
ing
set
s
–Id
entif
y n
ew d
etec
tion
tech
niq
ues
“New
Nic
he”
Upd
ate
Tra
inin
g S
ets,
Use
La
ngua
ge In
depe
nden
tT
echn
ique
s
Blo
gsp
ot
“1”
tech
niq
ue
•A
form
of s
pam
blo
g th
at s
eem
s to
be
quite
co
mm
on o
n b
log
ger
–C
reat
e 1
po
st–
Use
blo
g a
s d
oo
rway
•F
ilter
s fin
d it
diff
icu
lt to
det
ect s
pam
with
out
avo
idin
g fa
lse
posi
tives
•S
and
boxi
ng
tech
niq
ue u
sed
by
blo
g s
earc
h
eng
ines
•T
ech
niq
ue w
ill g
o o
ut o
f fas
hio
n o
nce
web
se
arch
eng
ines
use
sim
ilar
stra
teg
ies
Blo
gspo
t“1”
tech
niqu
e
No
n-E
ng
lish
Sp
log
s
•A
re s
plo
gs
appe
arin
g in
Non
-Eng
lish
?–
No
t yet
?
–S
een
man
y ca
ses
of s
pam
blo
gs
in J
apan
ese
•C
reat
ing
trai
ning
exa
mp
les
for
mu
ltip
le
lang
uag
es n
ot f
easi
ble
•Id
entif
yin
g la
ngu
age
ind
epen
den
t tec
hniq
ues
th
at c
aptu
re s
tere
oty
pes
in s
plo
g cr
eate
too
ls
will
be
impo
rtan
t
Oth
er R
equ
irem
ents
•E
valu
atio
n o
f effe
ctiv
enes
s o
f filt
ers
–G
ener
ic P
reci
sion
and
Rec
all
–P
reci
sio
n a
nd
Rec
all w
ith a
wh
en?
•C
on
tinu
ed Q
oS
gua
ran
tees
in a
n a
dver
saria
l se
tting
–M
eth
od
olog
y–
Se
mi-a
uto
mat
ic te
chn
iqu
es
•M
eth
odol
ogy
for
ove
rall
mai
nte
nan
ce o
f th
e fil
ter
BLACKLISTS
WHITELISTS
BLACKLISTS
WHITELISTS
RSS
MODEL
RSS
MODEL
TEXT
MODEL
TEXT
MODEL
REGEX
REGEX
URL
MODEL
URL
MODEL
Pin
g S
trea
m
Ch
eck
wit
h
know
n b
log
s an
d s
plo
gs
Ch
eck
know
nre
gex
p
atte
rns
Use
mo
del
s o
ver
text
of
UR
L
Use
mo
del
s o
ver
Blo
g
ho
me-
pag
e
Use
mo
del
s o
ver
po
st
corr
elat
ion
Pro
cess
Lo
gs
Co
mm
un
ity
In
pu
t
LEA
RN
ER
Fee
db
ack
Con
tinui
ng W
ork
Co
ncl
usi
on
Co
ncl
usi
on
•T
rend
s su
gges
t tha
t the
pro
ble
m o
f sp
log
s w
ill
con
tinu
e
•R
esea
rch
nee
ds
to c
ontin
ue
iden
tifyi
ng
new
d
etec
tion
tech
niq
ues
•T
he
com
mun
ity h
as to
inco
rpo
rate
co
llab
orat
ive
tech
niq
ues
for
det
ectio
n
•R
esea
rch
nee
ds
to u
nder
stan
d fil
ter
evo
lutio
n in
a
mo
re p
rinci
ple
d w
ay
Res
ou
rces
•h
ttp://
icw
sm-b
log-
spam
.blo
gsp
ot.c
om/
–F
or
(to
o)
late
bre
akin
g r
eso
urc
es fo
r th
is tu
toria
l
•h
ttp://
pla
net
.so
cial
Med
iaR
esea
rch
.org
/–
A fe
ed a
gg
reg
ato
r fo
r b
log
s re
leva
nt t
o s
oci
al m
edia
–S
ug
ges
t new
blo
gs
to
pla
net-
smr@
cs.u
mb
c.ed
u
•h
ttp://
sus.
pic
iou
s.in
fo/
–P
roto
typ
e sp
am r
epo
rtin
g s
ervi
ce
•h
ttp://
ebiq
uity
.um
bc.
edu
/mem
eta/
reso
urce
s/–
Blo
g a
nd
sp
log
dat
aset
s, b
iblio
gra
ph
y, e
tc.
Dis
cuss
ion
