Fault Management Manual

7/26/2019 Fault Management Manual

http://slidepdf.com/reader/full/fault-management-manual 1/23

Fault Management through @com for NT-HLR

9Fault Management through @com for NT-HLR

Supervision User Manual

Original uthor : Mohamma! "arghouthi

#ate: Mon!a$% &' pril% (&)&

*ersion: )+&

)




Table of Contents

#ocument Histor$,

List of Ta.les, /

List of Figures /

)+ 0ntro!uction, '

(+ Using the 1vent Manager through @com, 2

2.1. Login to @com: 2

2.2. Starting the Event Manager: 3

2.3. Event Manager Menus:9

2.3.1. Edit Menu:9

2.3.2. View Menu: )&

2.3.2.1.Column Visibility: ))

2.4. Event Manager Toolbar:)(

2.5. Event Filters:)

2.5.1. Created Public Filters:)/

2.5.2. System Filters: )'

2.6. Event Information Window: )4

+ 0mportant larms, )9

3.1. Critical Data Base (DSA) Alarms: (&

3.2. Critical PGW Alarms: ()

3.3. HLR-FE Alarm Groups and frequent alarms: ((

(




Document History:

Version Date Author Updates

1.0 28/03/09 Mohammed Barghouthi Initial Document




List of Tables:

Ta.le ), 5olumn *isi.ilit$ ))Ta.le (, 5ritical #ata"ase 6#S7 larms (&

Ta.le , 5ritical 8: larms ()Ta.le /, HLR-F1 main larm roups ((Ta.le ', Fre;uent HLR-F1 larms((

List of Figures

Figure ), @com Login 2Figure (, Opening 1vent Manager3Figure , 1vent Manager 4Figure /, 1!it Menu 9Figure ', *ie< Menu )&Figure 2, 1vent Manager Tool.ar )(Figure 3, 1vent Manager Filters )Figure 4, 5reate! 8u.lic Filters )/Figure 9, S$stem Filters 6)7 )2Figure )&, S$stem Filters 6(7 = N1 )3Figure )), 1vent 0nformation )4

/




1. Introduction:

The @vantage 5omman!er provi!es comple> monitoring an! event processing functionalities for a

centrali?e! Fault Management of all supporte! N1 t$pes+

The FM feature is the central application for <atching alarm situations occurring on the manage! N1s

an! han!ling them in an effective an! economic <a$+

0f an alarm occurs on an$ N1 an event is sent to the FM application of the @5om+ larm events are

!ispla$e! on a central raphical User 0nterface 6U07% .ut can also .e for<ar!e! to various

!estinations lie e>ternal Operations S$stems% alarm panels% alarm printers or email a!!resses+

The 1vent Manager is the central U0 for routine FM operations+ The ne< events are liste! in a ta.le

.$ severit$% time% name of the alarm-initiating N1% affecte! application function% etc+ User has a variet$

of options to customi?e the vie<+ First of all% the user can !efine event filters to !ispla$ onl$ the eventsmatching the criteria+ S$stem-<i!e filters 6pre-configure! an! not changea.le7% private filters 6user-

specific7 an! pu.lic filters are offere!+

t the startup of the 1vent Manager pplication events are rea! from the event !ata.ase 6#"7+ ll

events% confirme! or unconfirme!% are store! in the event !ata.ase+

'




2. Using the !ent "anager through #com:

2.1.Login to @com:

First thing is to login to the @com client 6that shoul! .e installe! on the userAs 85 or Laptop7 .$ t$pingin the uni;ue username an! pass<or!,

Figure 1: @com Login

2




2.2.Starting the Event Manager:

fter that% the 1vent Manager is opene! either through the Fault Management Menu or !irectl$ through

the Event Manager Icon,

Figure 2: Opening Event Manager

3




:hen the event manager is open $ou get the screen .elo<,

Figure 3: Event Manager

4

5urrentl$ active

Filter

Num.er of 1ventscorrespon!ing to active filter

1vent Manager

Menus

1vent Manager

Tool.ar

1vent Manager

Filters




2.3.Event Manager Menus:

There are four Menus availa.le on the event Manager these are as follo<s,

• File Menu: This onl$ gives $ou the option to e>it the 1vent Manager pplication+

• Edit Menu: This has the alarm han!ling functionalit$ 6e>plaine! .elo<7

• View Menu: This helps in controlling the 1vent Manager #ispla$ 6e>plaine! .elo<7

• Help Menu: This gives access to Fault Management Help+

2.3.1. Edit Menu:

The 1!it Menu gives the alarm han!ling options as sho<n .elo<,

Figure 4: Edit Menu

9

Use! to loc theevent for processing

.$ the user+

Use! to confirmevents .$ user+Onl$ first option

6confirm7 isrecommen!e!+

Use! to e!it theevent rules an! the

event filters

Opens the eventinfo screen

Shortcuts tomanagement of

N1sUse! to appl$

5lose! B Not 5lose!Filters




2.3.2.View Menu:

The *ie< Menu ena.les the user to control <hat is !ispla$e! of the 1vent Manager screen% as!escri.e! .elo<,

• Toolbar: Sho<sBHi!es the 1vent Manager Tool.ar+

• Active Filter: Sho<sBHi!es the ctive Filter a.ove the alarm list+

• Freeze Event List: Free?es the sho<n events un!er this filter 6events <ill not .e refreshe!7

• Archived Events: #ispla$s events from rchive 6other<ise current list <ill .e sho<n7

• Copressed ode: Sho<s a summar$ of the alarms on the net<or elements+

• !"ste Filters include the !tate cleared: Sho<sBHi!es cleare! alarms <hen s$stem filters

are applie!+

• !everit" Colored #ow: 5olours the entire 1vent Ro< ccor!ing to Severit$+

• La"out $ptions: 5hanges the or!er an! si?e of event columns+

•

De%ault La"out: Restores the #efault la$out for columns+• Colun Visibilit": llo<s user to !eci!e <hich columns are !ispla$e!+

Figure 5: View Menu

)&




2.3.2.1.Column Visibility:

The 5olumn *isi.ilit$ allo<s the user to !etermine <hich !etails are sho<n for the alarms !ispla$e!+These !ifferent columns are !escri.e! in the ta.le .elo<,

Column $ame Descri%tion

Unique ID:Disla!s the unique identi"cation o# the corresonding e$ent %ithin the%hole administrati$e domain o# an &$antage 'ommander installation.

User: (ame o# the user %ho has rocessed the alarm

)tate: )tate o# the *larm

)e$erit!: Disla!s the se$erit! le$el o# the corresonding e$ent.

+irst ,ime: Disla!s the time stam o# the "rst occurrence o# this e$ent.

-ast ,ime: Disla!s the time stam o# the most recent occurrence o# this e$ent.

( ,!e:Disla!s the t!e o# the net%or element %here the e$ent %asinitiated.

(et%or lement:Disla!s the s!molic name o# the net%or element to %hich the e$entis related. I# the ( is art o# a comound ( its relati$e distinguishedname is disla!ed.

(ode:Disla!s the host name o# the cluster node to %hich the e$ent is

related.

*lication +unction:Disla!s the name o# the alication #unction %here the e$ent %asinitiated. ,his "eld ma! e emt!.

$ent ,!e: Disla!s the e$ent t!e according to the I,U, ecommendation 4.533.

6roale 'ause:Disla!s the roale cause according to the I,U, ecommendation4.533.

$ent ID:Disla!s the e$ent ID o# the corresonding e$ent7 unique #or each (t!e.

'on"rmation ,ime: ,he ,ime %hen the e$ent %as con"rmed

Instance 'ounter:Disla!s ho% o#ten the corresonding e$ent %as reorted ! the (.I.e. indicates the same error condition #or the same #ault! source.

$ent Instance (ote:Indicates %hether an *larm Instance oerator seci"c note is a$ailaleor not

*utomaticall! Indicates %hether an $ent can e automaticall! cleared or not.

))



Fault Management through @com for NT-HLR'learale:

Descrition: Disla!s the short descrition o# the e$ent.

Tabe 1: !oumn Vi"ibiit#

)(




2.4.Event Manager Toolbar:

The 1vent Manager Tool.ar has icons that are shortcuts to some of the functionalities containe! in the

menus% the figure .elo< e>plains the function of the !ifferent 1vent Manager 0cons,

Figure $: Event Manager Toobar

)

Free?e1vents

#ata Management%SS3 Management

FMHelp

5onfirm% 5onfirm *isi.le%5onfirm using Filter+

5ompactMo!e

ppl$ 5lose! BNot 5lose! Filters

1ventFilters

Loc% Unloc%Force Unloc+

Sho< 5leare! 1ventsin S$stem Filters

1vent rchive




2.5.Event Filters:

To simplif$ the han!ling of alarms the alarms can .e filtere! accor!ing to !ifferent 5riteria+ There are

three main categories of filters,

&' (rivate Filters: User create! filters that can onl$ .e use! .$ the user <ho create! it+

)' (ublic Filters:User create! filters that can .e use! .$ all users+

*' !"ste Filters: @com !efine! 1vent Filters+

Figure %: Event Manager Fiter"

)/




2.5.1. Created Public Filters:

fe< pu.lic filters are alrea!$ create! to simplif$ monitoring the open alarms on the s$stem,

&' Adin: Open larms on @com% #"M% #" install an! HLR install Servers+ Faults on these no!es !o nothave a !irect impact on the s$stem or the su.scri.ers

)' Database Alars:Open larms on the F1-#S an! the "1-#Ss+ Since the #" is the most critical part of thes$stem% these alarms shoul! .e continuousl$ monitore!+ MaCor an! 5ritical alarms un!er thisfilter shoul! .e imme!iatel$ reporte!+

*' HL#+FE Alars: Open larms on the HLR-F1s+ MaCor an! 5ritical ones shoul! .e reporte!+

,' Ma-or and Critical:Open alarms on all 1lements of severit$ MaCor an! 5ritical+ These shoul! .e continuousl$monitore!+

.' (rovisionin/:Open alarms on 8: an! 8:-#S+ These !o not have an impact on the su.scri.ers .ut ma$impact provisioning+ MaCor an! 5ritical ones shoul! .e reporte!+

Figure &: !reated 'ubic Fiter"

)'




2.5.2. System Filters:

To simplif$ the monitoring of larms @com has s$stem !efine! filters% these !ispla$ events accor!ing

to one of the follo<ing criteria 6the ones hi/hli/hted in red are the ost si/ni%icant7,

&' !tate: larms are filtere! accor!ing to their states% possi.le states are ll% $pen% Loce!% 5leare!%5lose! an! Not 5lose!+

)' !everit":These Filters sho< alarms accor!ing to the level of severit$+ This coul! .e, Critical% Ma-or % Minor%Normal% :arning an! 5leare!+

*' 0etwor1 Eleent:These filters sho< the alarms correspon!ing to !ifferent Net<or elements+ The availa.le t$pes of

Net<or elements for NT-HLR are, 2E+D!As 32EDs45 FE+D!A 3FEDs45 HL#+FE 3HFE4% #"M%#"0% HF0% 8:-#S 68#7% 8:s an! @comman!er+

,' User:This categor$ onl$ contains the s$stem confirme! alarms+ These are <arnings or events that aresuppresse! .$ internal rules an! normall$ re;uire no operator action

.' Application Function:The criteria for this filter are the application that triggere! the alarm+ These are "RF% HLRF% None%OMF an! 8latform

6' Event T"pe:

These are the event t$pes accor!ing to the 0TU-T D+3 stan!ar!+ The possi.le 1vent T$pes are,5ommunications larm% Eualit$ of Service larm% 8rocessing 1rror larm% 1;uipment larm an!1nvironmental larm+

)2




Figure (: )#"tem Fiter" *1+

)3




Figure 1,: )#"tem Fiter" *2+ - .E

)4




2.6.Event Information Window:

To get more information a.out the event the 1vent information <in!o< is availa.le+ This can .e accesse! through

the e!it Menu or .$ !ou.le-clicing on the event+ 0n a!!ition to the information that can .e !ispla$e! in the<oring area% a long te>t that offers a more !etaile! !escription of the pro.lem an! a repair te>t <ith suggeste!actions are provi!e!+

Figure 11: Event /n0ormation

)9




&. Im%ortant 'larms:

0n general% all alars reporte! .$ @com nee! to .e chece!+ This can .e a !ifficult tas especiall$

given the comple>it$ an! the num.er of N1s an! processes involve! in the NT-HLR+ To .e a.le tohan!le the large num.er of events <ith a certain perspective certain gui!elines can .e helpful,

• #" alarms are the most critical+ larms on the 2E+D!As an! FE+D!As cannot .e ignore!+ 1ven

<ith the high availa.ilit$ an! high re!un!anc$ on the s$stem% certain errors can lea! to seriousservice !egra!ation for millions of su.scri.ers+

• 1rrors on the 8:% an! 8:-#Ss might lea! to a !isruption in the provisioning process+ These

shoul! .e monitore! closel$% especiall$ in the case of provisioning complaints+

• :ith the presence of )( HLR-F1s an! the loa!sharing of transactions .et<een them% failure on

one or more of these is unliel$ to cause an$ serious issues+ Ho<ever% if certain configurations

are change! these can result in a !egra!e! ;ualit$ of service for all su.scri.ers+ 5ertain alarmscan in!icate missing or <rong configurations+

• !ministration elements are non-su.scri.er affecting+ larms on @com 6e>cept for

communications alarm <ith the !ifferent N1s7% #"M% #"0% an! HF0 have no !irect impact on theHLR functionalit$+

• Critical an! Ma-or larms of this severit$ on all 1lements shoul! .e imme!iatel$ reporte!+

• 1;uipment larms nee! to .e reporte! to NSN care team% the$ <ill arrange for fault$ har!<are to

.e replace! an! mae sure har!<are issues <ill not affect su.scri.ers+

:ith these gui!elines in min!% the ne>t sections have a !escription of some of the most critical alarms%an! also a list of other alarms 6most fre;uentl$ recurring7 an! ho< to han!le them+

(&




3.1.Critical Data Base (DSA) Alarms:

"elo< is a list of the most critical alarms on the #Ss 6F1-#S% "1-#S% an! 8:-#S7+

!ent !ent ID (e!erity

Descri%tion Ho) to handle

Dual6rimar!

301213 'ritical

,he 6rimar! )D+ )er$iceData +unction hasdetected the resence o#a second 6rimar!

,his is a Dual 6rimar!)ituation. ,his could leadto serious ser$icedisrution and should ereorted immediatel!.

Isolated

6rimar!

30119 'ritical

* lin #ailure has occurredto a )econdar!)!nchroni;ed )D+ %ith noother )econdar!

)!nchroni;ed )D+slea$ing the 6rimar!isolated.

,his indicates that the6rimar! is isolated and %illossil! lead to a Dual

6rimar! situation. )houlde reorted Immediatel!

6rimar!+ailo$er

301211 'ritical

,he 6rimar! )D+ )er$iceData +unction has #ailed.

,he local )D+ hasassumed the 6rimar! role.

* #ailo$er has occurredand the reasons ehind itneed to e checed toa$oid ha$ing a Dual6rimar! issue.

-in Do%n300

<=<11'ritical< Ma>or

Inter)D+ lin #ailureInterD)* lin +ailure or(DD -in +ailure.

,his indicates a lostconnection and has to echeced.

-D*6'onnection

+ailure

1=91?03<1?

0?

'ritical< Ma>or

,he -D*6 connection tothe D) @44A in the D)*@A +ailed.

-D*6 connection lost.)hould e in$estigatede#ore it leads to moreserious issues.

)econdar!Uns!nc.

30111?<11

Ma>or*n attemt to er#orm as!nchroni;ation has#ailed.

*ll ser$ers ha$e to es!nchroni;ed7 this issueneeds to e checed andresol$ed.

econciliation Issue

3011? 11= 15.

Ma>orC>ect Mismatch or C>ectdoes not eist

Data is not s!nchroni;edacross the D)*. Eas to echeced

Eanginge#erences 3012?1<2?2 Ma>or InterD)* econciliationalarm Missing e#erence.

Eanging e#erences could

harm the er#ormance o#the s!stem and need toe treated.

Bacu+ailure

3018 'ritical

* acu o# the dataasehas #ailed. ,he most liel!elanation is that thereis insuFcient dis sacea$ailale.

,his issue needs to echeced and sol$ed. *#terthat a manual acuneeds to e taen.

()




DisUtili;ation

3010 'ritical

,he indicated threshold$alue #or the disutilisation statistic haseen eceeded ! thedis artition.

,his alarm indicates highdis usage. (eeds to echeced and resol$ede#ore it leads tointerruting rocesses.

Tabe 2: !ritica ataa"e *)+ arm"

3.2.Critical PGW Alarms:

"elo< is a list of the most critical alarms on the 8rovisioning ate<a$+ 1ven though these alarms !onot have a !irect impact on the su.scri.ers the$ coul! lea! to !isrupting the provisioning process thusaffecting several c$cles for !ifferent applications an! the$ shoul! .e reporte! an! solve! ver$ fast+

!ent !entID

(e!erity

Descri%tion Ho) to handle

Bul UserAsDis Usage

120 'ritical ,he dis usage eceedsthe con"gured limit #orthis user.

Unneeded "les #rom theusersA outo < error#olders need to e deletedto resume ulro$isioning.

+ullC$erload < ,hresholdC$erload

1209<10

'ritical

,his alarm indicates ano$erload situation on aseci"c node andinstance the ro$isioningqueue is #ull and requests

%ill e re>ected.

6ro$isioning ,eam needsto e in#ormed in order toreduce the load. ()( careteam need to e in#ormed

to in$estigate the issue.

-D*6*uthenticati

on+ailure

12013 'ritical

,he 6GH cannot connectto the-D*6 dataase.*uthentication %as denied! the 6GHD)*. (ot allrequests ! 6GH users%ill e eecuted.

,his needs to e reorted.* #ailed -D*6authentication %ill lead toro$isioning #ailure.

-D*6'onnection

-oss

1201? 'ritical

,he maimum numer o#-D*6 connections %asreached. ,he -D*6 ser$eris out o# resources.

ecution o# somerequests %ill e re>ected.

,his needs to e reorted.-ac o# -D*6 resourcescould lead to ro$isioning

#ailure.

Dis +ull 1201 'ritical* #ull "le s!stem re$entsthe 6GH #rom running.

Un%anted "les need to edeleted #or the 6GH toresume %oring normall!.

Tabe 3: !ritica ' arm"

((




3.3.HLR-FE Alarm Groups and frequent alarms:

#ue to the e>tremel$ comple> nature of the HLR-F1 application an! platform% it is har! to generate a

list of the most critical alarms+ 0nstea!% .elo< is a small summar$ of the maCor alarm groups an! a list of the highl$ recurring alarms an! ho< to han!le them+

Alar 7roup Description

Alar 7roup &8):

a7 Logs 6Be>portBhomeBrtp99B99Blog7 too large G remove files un!er this !irector$

.7 not enough file s$stem or #" space availa.le+

c7 SNM8 port not configure!+

!7 the name! process is currentl$ not running+

e7 the current memor$ utili?ation is high+

f7 cpu utili?ation is high+

Alar 7roup &8*:

a7 a specific recover$ action coul!nIt .e run successfull$% manual intervention is

necessar$.7 general pro.lems <ith recover$ process+

Alar 7roup &66a7 08 relate! errors

.7 errors !etecte! !uring startup of HLR-F1

Alar 7roup &69 HTT8 Service is not full$ availa.le 6Trigger% Reencr$ption% :e. U07

Alar 7roup &65onnection to event manager <as interrupte! an! alarms coul!nIt .e sent to@com 6can .e !ispla$e! on L1MF7

Alar 7roup ))6HL#F

3ApplicationAlars4

a7 component - 5 T8 !apter G enerating uthentication *ectors not possi.le

.7 component - 5U G pro.lems <ith 5 Utimaco HSM

c7 component - F: G signalling pro.lems

!7 component - HOL 6Han!ling Overloa!7 G either s$stem overloa! or M8reCectionse7 component - JMH 6Je$ Management7 G pro.lems <ith HSM certificates

f7 component - TFR 6Trigger Frame<or7 G connection pro.lems to 8:

Tabe 4: 6L78FE main arm roup"

lso% .elo< is a summar$ of fre;uentl$ recurring alarms an! ho< to han!le them,

Alar Description handlin/

)&2-(93 K)9-))9 K

)&2-')9

#8D))),5oul! not construct communication .loc+#8D))), TT faile!, error returne! from TTrans67

#82))),Sen!ing T58 re;uest to S558 faile!

T translation in the )&2-(93alarm nee!s to .e recor!e! an!

the three alarms are confirme!

)9-))+S558,ROUT0N F0LUR1 - LO5L SSNUN1EU0881#

The SSN an! the 85 in thealarm nee! to .e recor!e!+

)&(-&Rtpu!Mgr&),File group oraclerchiveLog 6pathBe>portBhomeBoracleB!evBarchivelog7 is large, 2/4&/ M"

0gnore+ 0nternal 8rocess <illcontorl the file group+

)&(-(&)(Rtpu!Mgr&),File 8ermission Monitor foun!!iscrepancies+ Loo at 8ath for more !etails

0gnore+ No action nee!s to .etaen

Tabe 5: Fre9uent 6L78FE arm"

(

Fault Management Manual

Documents

Transcript of Fault Management Manual