SGSN Troubleshooting

Troubleshooting Nokia 3G SGSN

dn04151373Issue 1 en

# Nokia CorporationNokia Proprietary and Confidential

1 (50)

SGNDO3001.00Nokia 3G SGSN Rel. 3, Product Documentation

The information in this document is subject to change without notice and describes only theproduct defined in the introduction of this documentation. This document is intended for the useof Nokia's customers only for the purposes of the agreement under which the document issubmitted, and no part of it may be reproduced or transmitted in any form or means without theprior written permission of Nokia. The document has been prepared to be used by professionaland properly trained personnel, and the customer assumes full responsibility when using it.Nokia welcomes customer comments as part of the process of continuous development andimprovement of the documentation.

The information or statements given in this document concerning the suitability, capacity, orperformance of the mentioned hardware or software products cannot be considered binding butshall be defined in the agreement made between Nokia and the customer. However, Nokia hasmade all reasonable efforts to ensure that the instructions contained in the document areadequate and free of material errors and omissions. Nokia will, if necessary, explain issueswhich may not be covered by the document.

Nokia's liability for any errors in the document is limited to the documentary correction of errors.NOKIA WILL NOT BE RESPONSIBLE IN ANY EVENT FOR ERRORS IN THIS DOCUMENTOR FOR ANY DAMAGES, INCIDENTAL OR CONSEQUENTIAL (INCLUDING MONETARYLOSSES), that might arise from the use of this document or the information in it.

This document and the product it describes are considered protected by copyright according tothe applicable laws.

NOKIA logo is a registered trademark of Nokia Corporation.

Other product names mentioned in this document may be trademarks of their respectivecompanies, and they are mentioned for identification purposes only.

Copyright © Nokia Corporation 2005. All rights reserved.

2 (50) # Nokia CorporationNokia Proprietary and Confidential



Contents

Contents 3

1 Changes in troubleshooting 5

2 Troubleshooting overview 7

3 Gathering information about 3G SGSN 93.1 Alarms 103.2 Logs 103.3 Statistics 113.4 Traffic monitoring 113.5 System Level Trace 123.6 Subscriber information 12

4 3G SGSN units and processes 134.1 Central Routing Processor 134.2 Forwarding unit 144.3 Session and Mobility Management unit 144.4 SS7 unit 144.5 Tunnelling unit 15

5 Checking the state of a process 17

6 Switchovers and reset methods 216.1 Switchovers 216.2 Process reset methods 22

7 Resetting processes 25

8 Fault monitoring and observation 27

9 Hardware faults 299.1 Recommended spare units 299.2 Faulty GPLC and CRP cards 309.3 Faulty CPCI interface cards 329.4 Faulty fan tray 329.5 Faulty power supply 33

10 Software faults 3510.1 Checking system integrity 3510.2 Charging fails 3610.3 SMM procedure fails 39

11 Reporting faults to Nokia 47

12 Glossary 49



3 (50)

Contents

1 Changes in troubleshooting

This is the first issue of the Nokia 3G SGSN troubleshooting documentation. Itincludes information previously covered by the Recovery Guide for Nokia 3GSGSN.



5 (50)

Changes in troubleshooting

2 Troubleshooting overview

This troubleshooting documentation provides instructions and information ontracing a problem to its origin and resolve the issue if possible. It provides basicinformation and instructions for both software and hardware issues.

The Nokia 3G SGSN is designed to be fault tolerant. It recovers from hardwarefailures and take actions in case of software faults. Monitor and react to faults asdocumented. If you encounter problems not covered here, contact Nokia.

Use the form in Nokia Online Services (NOLS) to provide feedback aboutproblem situations that should be added to this documentation.



7 (50)

Troubleshooting overview

3 Gathering information about 3G SGSN

The troubleshooting may consist of one or several stages from the systems faultmanagement process.

Detection is the process of discovering that an error exists. Detection is defined asthe time from a failure causing a loss of service, to the system becoming aware ofit.

Location is the process of narrowing down the failure to the defective component.

Isolation takes the defective portion of the system out of service. The region thatis isolated must be bounded at a point where it can be removed from allinteraction with the system.

Recovery is the process of reassigning the necessary resources to restore thesystem to an operating state. Recovery also requires restoring any portions of thesystem that were adversely affected by the failing component. Recovery is thefinal step in the process that contributes to outage time. Once the system isproviding complete service again, the remainder of the process does not directlycontribute to outage time.

Reporting is the process that notifies the outside world that an event has takenplace. This is the first step in the repair process. The repair process is indirectlyrelated to availability. In systems employing redundancy, a statistical possibilityexists that a second failure can occur in the component covering for this failure.This would result in a complete system outage. While this probability is low, theseverity is high enough to make this a factor in the availability equation. It isimportant, even in redundant systems, to keep repair times low.

Repair is the replacement of the defective component. This phase is generallydesignated for the operator-assisted (human) portion of the process. The repairprocess is broken into these phases because, firstly, the repair step is usually themost time consuming portion of the process. Secondly, this is a point in theprocess where mistakes can account for system outages.

The following tools and features are available in the Nokia 3G SGSN:



9 (50)

Gathering information about 3G SGSN

. software alarms - visual warning or fault indicators from the user interfaceor in a network management system

. hardware alarms - visual warning or fault indicators on the IP3400hardware

. SNMP, standard protocol for informing about problems in a system.

. log files - listing of error and info messages from the software

. statistics - packet counters from processes, interfaces, linecards, or thewhole system

. monitoring - additional indicators in the user interface that provide systemstatus and hardware information

. Traffic monitoring

. System Level Trace

3.1 Alarms

In troubleshooting situations it is good to start by viewing the current list of activealarms.

The fault management can either be enabled or disabled in 3G SGSN. If it isenabled, you can set up the 3G SGSN to act upon alarms by logging the alarms.

For a complete collection of alarm descriptions, see Nokia 3G SGSN Alarms.

For instructions on configuring fault management, see Fault management andSNMP management in Operating and Maintaining Nokia 3G SGSN.

3.2 Logs

The alarm history can be followed from the alarm log, which also shows thecleared alarms.

The following platform log files in /var/log can contain additionalinformation:

fmTraps.log Contains information about raised alarms andcancellations.




httpd_access_log Contains information about each transaction in theVoyager interface.

httpd_error_log Contains information about each error in the Voyagerinterface. Whenever the Voyager interface is enabledor disabled, it is logged to this file.

messages General log file of the system (syslog). This logcontains the error logs that 3G SGSN produces. Thisis one of the key sources of information whentroubleshooting.

3.3 Statistics

Statistics information provided by the 3G SGSN can be used in troubleshootingsituations. The following types of statistics are available:

. general statistics (including CDR, GTP, Online Service Controller, andNokia Subscription Manager statistics)

. PDP session management statistics

. QoS statistics

. IP statistics

For more information, see Introduction to Nokia 3G SGSN PerformanceIndicators and Statistics in Nokia 3G SGSN Performance Indicators.

3.4 Traffic monitoring

You can use sgndump to monitor traffic on 3G SGSN interfaces. It is similar tothe UNIX utility tcpdump, but with an extension for decoding GTP messages.Refer to any common UNIX man page of tcpdump for description of thecommand options. They are the same with sgndump.

When using sgndump, it is important to keep the following in mind:

. You need to make sure you are monitoring traffic on the right interface.

. If you want to store the traffic information, you need to first create a filewhere the traffic information can be dumped.



11 (50)

Gathering information about 3G SGSN

. The sgndump command needs to be executed from the command line onthe active CRP.

. When you execute a traffic dump, make sure there is enough free space onthe harddrive of the CRP. You should monitor disk usage during the dump.An alarm (32805) is raised if you use too much memory.

Note

The sgndump utility collects, but does not decode the SIGTRAN traffic.

3.5 System Level Trace

System Level Trace (SLT) helps in detecting problems in the network that affectservice level. When tracing is enabled, each invoking event (GPRS attach/detach,PDP context activation/deactivation/modification) activates the traceautomatically. The trace record is generated and transferred to the networkmanagement system. For more information on how to carry out traces, see Tracehandling in Operating and Maintaining Nokia 3G SGSN.

3.6 Subscriber information

3G SGSN has a utility smmdump that can be used to gather subscriberinformation from the SMM database. This information is also valuable whentrying to isolate problems in 3G SGSN. For more information on handlingsubscriber information, see Fault traceability in Operating and Maintaining Nokia3G SGSN.




4 3G SGSN units and processes

This is an overview of 3G SGSN units (linecards) and their tasks. For moreinformation, refer also to the 3G SGSN product description.

4.1 Central Routing Processor

The Central Routing Processor (CRP) is the operation and management unit ofthe system and has a hard disk for permanent storage. The switch fabric isphysically located on this linecard. Each IP3400 has two CRPs, one acting as theactive unit and the second one in standby mode, ready to take over in case theactive CRP fails. The active CRP manages the installed software, interfaces,collects statistics, and distributes the routing information.

The two CRPs synchronise the system configuration files in real-time andsynchronise other critical files like application packages every 5 minutes.

The following processes run on this unit:

. dbd - handles subscriber database filtering, dumping, and deletion requestsissued from the Voyager management console

. O&M - This is not a process, but a collection of services. It providesconfiguration services through Voyager with xpand, for example, or anexternal interface with daemons. It also offers performance managementand other counters with a pmd process that is in every linecard.

. pmd - collects and saves statistics to a fail-safe disk system. This processalso collects the performance indicators and sends them to NetAct onrequest via the NE3S interface.

. sctptrn - provides the SCTP translation layer. (SCTP itself is part of theplatform functionality.)

. sltd - system level trace daemon. This process performs the system leveltrace functionality of the 3G SGSN.

. sysman - performs the same tasks as the signalling system manager, butfor non-signalling processes and the sigsm process.



13 (50)

3G SGSN units and processes

4.2 Forwarding unit

The Forwarding units in the Nokia 3G SGSN which provide Gn and Ga interfaceconnectivity using Ethernet interfaces. Routing protocols are use on theseinterfaces to provide redundancy. All FUs and their interfaces can be used at alltimes when configured.

4.3 Session and Mobility Management unit

Similair to the CRP, the Session and Mobility Management Units manage theconnections in the IP3400 system. SMM units control all the signalling traffic andmaintain a database of attached subscribers. A Nokia 3G SGSN has two SMMunits, one active and one standby.The SMM database is synchronised to thestandby SMM unit.


. smm - provides the SMM application, including CDR creation (M, SMS,and LCS CDRs). This process handles the protocol users for RANAP,MAP and CAP, in addition to the full GTP-C implementation (protocol anduser).

. cdrsender - sends M-, S-SMO-, S-SMT-, LCS-MT, and LCS-MO-CDRs to Nokia Charging Gateway

4.4 SS7 unit

The Signalling System No. 7 (SS7) units provide connectivity to the Gr interfaceto reach, for example, the HLR. Even though only one SS7 at a time has thecontrol over the connections, all SS7 interfaces are active, meaning the active unitmay use an interface in the standby unit.


. bbsig - provides MTP3, M3UA distributor (M3D), SCCP and RANAPdistributor (RD) in the broadband stack

. m3uasigbb - provides the M3UA protocol layer including the layermanager (m3ualm) for the broadband stack

. m3uasignb - provides the M3UA protocol layer including the layermanager (m3ualm) for the narrowband stack




. nbasig - provides MAP and CAP stacks (and distributor functions) andlayer manager to MAP and CAP. It also takes care of the interface to smm.

. nbsig - provides the M3UA, MTP-3, SCCP, and TCAP stacks, and thelayer manager to these stacks.

. pmcdrv1, pmcdrv2 - manage the E1 card and the MTP2 driver. There isone per MTP2 card. This process also initialises the actual SS7 interfacecards.

. sigsm - signalling system manager process. This process manages faulttolerance operations for signalling related processes, for example,performing monitoring and switchover of fault tolerant and redundantsignalling applications in the 3G SGSN.

4.5 Tunnelling unit

There can be two to nine Tunnelling units (GPLCs) in the 3G SGSN, dependingon the system configuration. The Tunnelling unit provides the Iu interface andhandles the user plane. The following processes run on this unit:

. atmsig - provides SSCOP and SSCF layers, the adaption layers betweenAAL and MTP3. AAL and ATM layers themselves run on the ATM card.

. cdrsender - sends S-CDRs to Nokia Charging Gateway

. kGTP - loadable kernel module that provides the user data handling part ofthe tunnelling, Quality of Service, Charging and Lawful Interceptionfunctions. It also handles part of the GTP-U signalling and statisticsgathering. This process starts automatically and cannot be manuallystopped or restarted. There is still a log for it, which can be seen from thefault traceability pages in Voyager.

. tunnel - provides tunnelling, quality of service, charging, PDP contextdatabase, and lawful interception



15 (50)

3G SGSN units and processes

5 Checking the state of a processSummary

There are three alternatives for checking process status:

1. Check the process states with the Voyager interface

http://<sgsn_ip_address>/opt/cgi-bin/sgsn/processconf.tcl

2. If you do not have a graphical user interface available, browse to theSystem Supervision page using lynx (a text-based browser). To start lynx,first open a telnet connection to the active CRP, then enter lynx at thecommand line prompt.

3. If neither Voyager nor lynx work, open a console connection to the activeCRP and use the command line interface to check process states asinstructed below.

Steps

1. Check if the process is enabled or disabled.

To check the process state (enable/disable) for CRP, enter:

dbget package:sgsn:process:sysman

To check the process state (enable/disable) for GPLCs, enter:

dbget node:<slot no.>:package:sgsn:process:\

<process name>

Where <process name> is one of the following:

. atmsig

. bbsig

. cdrsender

. nbasig

. m3uasigbb



17 (50)

Checking the state of a process

. m3uasignb

. nbsig

. pmcdrv1

. pmcdrv2

. sigsm

. smm

. tunnel

Expected outcome

Return value 't' means that the process is enabled. No return value meansthat the process is disabled.

2. Check process states on CRP.

Use the following commands to check the CRP processes:

dbget sgsn:sgsnCrp:sgsnCrpStat:\

sgsnProcessStateTable:17:sgsnSysmanState


sgsnProcessStateTable:17:sgsnDbdState


sgsnProcessStateTable:17:sgsnSltdState


sgsnProcessStateTable:17:sgsnPmdState

The slot number is 17 if CRP-A is the active CRP, or 18 if CRP-B is active.

3. Check process states on the GPLCs.

Use the following command to check the sate (active, standby, out-of-service, and so on) of processes running on GPLCs.


sgsnProcessStateTable:<slot no>:<process name>

Where <process name> is one of the following:

sgsnSmmState

sgsnCdrSenderState

sgsnNbsigState




sgsnBbsigState

sgsnTunnelState

sgsnAtmsigState

sgsnPmcdrv1State

sgsnPmcdrv2State

sgsnSigsmState

sgsnM3uasignbState

sgsnM3uasigbbState

sgsnNbasigState

Expected outcome

The possible return values of the dbget commands are:

. 0 = unknown

. 1 = disabled

. 2 = init

. 3 = ready

. 4 = active

. 5 = standby

. 6 = oos

. 7 = activeStandalone

. 8 = activeUpdating

. 9 = activeSynchronized

. 11 = standbyUpdating

. 12 = standbySynchronized



19 (50)

Checking the state of a process

Process state definitions:

unknown (0) The state of the process is unknown. Note that when thepmcdrv process is started, it is at first temporarily in this state.

disabled (1) Process is disabled.

init (2) A process is init state when it is started.

ready (3) Process is ready when it is started and has read theconfiguration. It is 'ready' to be the active or standy process.

active (4) Process is actively handling protocol messages.

standby (5) A process is in a standby state when the software is acting as abackup to the corresponding active process on the ActiveNode. When in this state, the process is not actively involvedin protocol message processing.

oos (6) A process is in an out of service (oos) state when the softwareis unable to process protocol messages. Either the software onthe node has not been configured or it has crashed due to afault.

activeStandalone (7)Fault tolerant process is activeStandalone when the process isactively handling protocol messages.

activeUpdating (8)Active fault tolerant process (SMM) is activeUpdating whenthe process is updating databases with the active process.

activeSynchronized (9)Active fault tolerant process (SMM) is activeSynchronizedwhen the software is actively processing protocol messages.

standbyUpdating (11)Standby fault tolerant process (SMM) is standbyUpdatingwhen the process is updating databases with the standbyprocess.

standbySynchronized (12)SMM process is in a standbySynchronized state when thesoftware is acting as a backup to the corresponding activeprocess on the active SMM unit. When in this state, theprocess is not actively involved in handling protocol message.




6 Switchovers and reset methods

6.1 Switchovers

Central Routing Processor

A switchover to the second CRP occurs when the active CRP has a criticalsoftware or hardware fault, or when it is manually initiated by the user. Inaddition, the user may defined in Voyager any of the following events to trigger aswitchover:

. Disk failure

. Active CRPs fabric down

. Overheating

. High/low voltage

The system is interrupted for a short period of time during a CRP switchover.This is due to the handover of the control to the switch fabric in the second CRP,and the linecards reconnecting and exchanging information with the new, activeCRP.

SMM Unit

If the SMM unit fails, a switchover is performed to the standby unit. This shouldnot affect the end users, if the status returns to active after the failure. If there is anSMM switchover, check the following to ensure that everything continuessmoothly:

1. Check the states of the processes running on the SMM unit.

2. Check the Session Management statistics. Ensure that they are increasingby reloading the page a few times on the web browser.



21 (50)

Switchovers and reset methods

3. Check the Mobility Management statistics. Ensure that they are increasingby reloading the page a few times on the web browser.

4. Check that the broadband and narrowband signalling links are up.

Also check from the Voyager interface that the state of the crashed SMM processis now standby. Ensure that the SMM process backs up all MM and PDP contextsfrom the active unit by checking the start and end times of the backup process.These times correspond to the activation and cancellation times of the alarm32385 STANDBY COPYING DB FROM ACTIVE.

6.2 Process reset methods

atmsig

This is a standalone process, and no switchover is performed. When disabled, allbroadband links configured via the affected TU are dropped down and the SCCPconnections are released.

bbsig

This is a fault tolerant process, and a switchover is performed if an active processis reset. Wait for the switchover to finish (PROCESS FAILED alarms arecancelled and the process states are as expected in the process status table inVoyager) before performing another reset.

Note

If both processes are disabled at the same time, all links to the RNCs are droppedand the SCCP connections released.

cdrsender

This is a standalone process, and no switchover is performed. When disabled, allCDRs in the send queue are lost. For this reason, you should never manuallyrestart the process. However, the cdrsender process needs to be restarted if the Gaaddress for the given linecard is changed.




m3uasigbb and m3uasignb


nbasig

All instances run as active, without standby. They operate in a load sharingrelationship. If one process instance fails, the signals handled by process at thetime are lost, but the 3G SGSN maintains the MAP and CAP signallingfunctionality through the other process.

nbsig


pmcdrv1 and pmcdrv2

All instances of this process run as active without standby processes. Whendisabled, all narrowband links configured via the affected process are dropped.

sctptrn

This is a standalone process within the CRP, and a switchover is performed onlyduring a CRP switchover.

sigsm


smm




23 (50)

Switchovers and reset methods

After the SMM switchover or a standby SMM reset, a backup for the MM andPDP contexts is started. Another switchover may not be performed before thebackup is finished and the alarm 32385 STANDBY COPYING DB FROMACTIVE is cancelled. This can take several minutes, depending on the number ofcontexts in the database. Another switchover during the backup results in lostcontexts in the SMM unit.

sysman

This process should always be running on the CRP, and it cannot be controlledfrom Voyager. However, if a restart is required, it can be done from the commandline.

To disable the process, use the command:

dbset package:sgsn:process:sysman

To enable the process, use the command:

dbset package:sgsn:process:sysman t

tunnel

This is a standalone process, and no switchover is performed. When disabled, allPDP contexts on the affected TU are released. Manual CDR generation must beperformed first (Charging basic configuration), otherwise all charging data is lost.

Note

Never restart several TUs or tunnel processes simultaneously if there are manyactive PDP contexts. This could overload the entire network because the 3GSGSN starts to deactivate the related PDP contexts from the GGSN and page idlesubscribers to deliver the Delete PDP Context Request.




7 Resetting processesPurpose

Controlled process resets and switchovers should always be performed bydisabling/enabling the process from the process configuration page in theVoyager.

Processes can also be terminated from the command line in the active CRP, butthis method should only be used if disabling/enabling through Voyager is notsuccessful.

Steps

1. Disable the process.

dbset sgsn:sgsnCrp:sgsnCrpConf:sgsnProcessTable:\

<slot no>:sgsnProcess<process name> 2

Example 1.

The process name must start with a capital letter!

dbset sgsn:sgsnCrp:sgsnCrpConf:sgsnProcessTable:1:\

sgsnProcessSmm 2

2. Enable the process.

dbset sgsn:sgsnCrp:sgsnCrpConf:sgsnProcessTable:\

<slot no>:sgsnProcess<process name> 1

Example 2.

The process name must start with a capital letter!

dbset sgsn:sgsnCrp:sgsnCrpConf:sgsnProcessTable:1:\

sgsnProcessSmm 1



25 (50)

Resetting processes

8 Fault monitoring and observation

The critical resources of the 3G SGSN system can be monitored via the Voyagerand CLI interfaces. Failure situations are also indicated with alarms. For moredetailed information see, Operating and Maintaining Nokia 3G SGSN, Nokia 3GSGSN Alarms and CLI Reference Guide for IPSO 3.9NET.



27 (50)

Fault monitoring and observation

9 Hardware faults

9.1 Recommended spare units

Reserve the following units as spares for each IP3400 system:

. 1 x CPCIs (one of each type used in the system)

. 1 x CRP

. 1 x fan tray

. 1 x GPLC

. 1 x harddisk

. 1 x power supply



29 (50)

Hardware faults

9.2 Faulty GPLC and CRP cards

Description

Figure 1. CRP card front panel details

In normal operation the POWER LED is lit. It is also normal that the FABRIC ledis lit or blinking. Also, blinking power LED indicates that the card is beingbooted, and a blinking fabric led indicates SW download via the fabric. Thefabric LED is also off on the CRP if there are no fabric connections to GPLCs.This is always the case for the standby CRP.

The conditions when GPLC or CRP LEDs are lit are shown in the followingtable.

FABRIC

ALERT

FAULT

POWER

LINK

10Base-T

100Base-Tx

ACTIVITY

RESET

FABRIC

ALERT

FAULT

POWER

LINK

10Base-T

100Base-Tx

ACTIVITY

RESET

S1 S2

CONSOLE(COM

1)

SLOT1

SLOT2

CRP status

Link LED

Ethernet Port

Activity LED

Reset Switch




LED Condition when lit

POWER The unit is in normal operation.

FABRIC Traffic indicator.

ALERT Indicates alert condition.

FAULT There is a fault condition.

If a switchover has occurred between two CRP cards one of the CRP cards couldbe faulty. In this case check the status indicated by the LEDs on the CRP frontpanel.

If no front panel LEDs are illuminated it could indicate a faulty CRP or GPLCcard.

If the FAULT LED is illuminated it could indicate a faulty CRP or GPLC card.

Symptoms

ALERT or FAULT LEDs are lit, or no LEDs are lit, on the front panel of the CRPor the GPLC card.

Recoveryprocedures

. Check the system status pages and alarm list in Voyager.

. Reset the faulty unit from the Voyager interface and check if the card hasrecovered from the fault situation.

. Reset the faulty unit using the Reset switch located on the front panel ofthe GPLC or CRP card and check if the unit has recovered from the faultsituation.

. Remove the faulty GPLC or CRP card from the system and insert it again.Always use the halt command to halt the operating system on the CRPbefore removing a CRP from the system. If you cannot halt the CRP forsome reason, press the reset button on the CRP before removing it. Waituntil it boots and check if it has recovered from the fault situation.

. Replace the faulty GPLC or CRP card with a spare card.

. Contact the Nokia Care organisation and report the problem. Report theversion information and serial number of the faulty card to Nokia.



31 (50)

Hardware faults

For information on the location of the card version information and serial numbersee Section Module identification.

9.3 Faulty CPCI interface cards

Description

This section describes what to do if a CPCI interface card is faulty.

Symptoms

One of the following alarms is sent: Unit Missing, Unit Mismatch, or InterfaceFailure.

Recoveryprocedures

. Check if the connectors on the CPCI interface card are damaged.

. Replace the faulty card with a spare CPCI interface card. It is possible tohotswap the faulty card and add a new card.

Before inserting CPCI interface cards into the GPLC card, inspect theGPLC connectors and the connectors on the CPCI interface card fordamage (such as slightly bent pins). In you observe any damage, do notinset the card.

After you partially insert a CPCI interface card into the slot, verify that it isproperly seated in the guide slots. If a card is not properly seated and youpush it all the way in, you can damage the CPCI interface card and theGPLC card. Do not force the card in.

. Contact Nokia Care organisation and report the problem. Report versioninformation and the serial number of the faulty card to Nokia.

For information on the location of the card version information and serial numbersee Section Module identification.

9.4 Faulty fan tray

Description

This section describes what to do if there is a hardware fault on the fan tray.




Symptoms

The system reports a faulty fan on the primary fan tray.

Recoveryprocedures

. Replace the whole fan tray with a spare unit.

Note

Do not operate the system for longer than one (1) minute under a full traffic loadwithout the primary fan tray installed.

. Contact Nokia Care organisation and report the problem. Report versioninformation and the serial number of the faulty fan tray to Nokia.

For information on the location of the version information and serial number seeSection Module identification.

9.5 Faulty power supply

Description

The power supply unit has three LEDs located on the front panel.

Figure 2. Power Supply LEDs



33 (50)

Hardware faults

The power supply LEDs are lit in the following conditions:

LED Condition when lit

AC OK: AC power OK The input power is within the specified operatingrange.

PS FLT: Power supply fault There is a detectable fault condition.

CL LED: Current limit The power supply output exceeds the specifiedoperating range (48 volts).

CT LED: Over temperature The power supply temperature exceeds thespecified operating range.

In normal operation the AC OK LED is lit.

Symptoms

The PS FLT LED is lit.

Recoveryprocedures

. Replace the faulty power supply unit with a spare unit. The IP3400-3system is able to operate with three power supplies during the serviceperiod.

. Contact Nokia Care organisation and report the problem. Report versioninformation and the serial number of the power supply to Nokia.

For information on the location of the version information and serial number seeSection Module identification.




10 Software faults

10.1 Checking system integrity

Purpose

Configuration errors are the most common source of problems in 3G SGSN. Atypical configuration error is using inconsistent parameters (and software builds)for linecards that have the same function in the 3G SGSN unit (for example,tunnelling units). 3G SGSN has a built-in feature that allows you to verify yoursystem configuration. This should be done as the first step when troubleshootingsoftware-related problems.

Steps

1. Verify the software package status.

a. In Voyager, enter the Configuration Integrity Tools dialogue byselecting the link Configuration Integrity Tools under the BasicServices category of the 3G SGSN configuration.

b. View the SGSN Packages Status table and verify that all bullets aregreen.

2. Verify configuration consistency.

a. In Voyager, enter the Configuration Integrity Tools dialogue byselecting the link Configuration Integrity Tools under the BasicServices category of the 3G SGSN configuration.

b. In the Consistency Check Status table, click the OK button to run asystem check.

c. View the Consistency Check Status table and verify that all bulletsare green.



35 (50)

Software faults

10.2 Charging fails

Description

There are three basic fault types concerning charging:

. CDRs are not created

. CDRs are not sent

. CDR fields are missing or incorrect

You main sources of information concerning these problems are the charging-related counters and alarms.

Symptoms

CDR queue gets full

This is indicated by alarms, so you need to follow up by first checking the activealarms.

CDRs are not created

CDR types are not being created as expected.

CDR fields are missing or incorrect

CDR field are empty (fixed format) or missing (TLV format), or they containunexpected values. This is usually indicated by a validation failure at the CG end,or errors in the Customer Care and Billing System (CCBS). The value 'abnormal'the Cause for Record Closing field also can indicate that fields are missing.

Recoveryprocedures

In most cases, charging problems arise due to configuration errors, either in the3G SGSN or in some part of the network. Charging can otherwise handle mostother error situations. If a problem does occur, you may need to investigate whathappened in the 3G SGSN or network several hours or days before the problemarose.




Isolating the source of a CDR queue overflow

Purpose

If the CDR queue of a cdrsender is getting too full, it usually means that themessages from the cdrsender process are getting blocked at some point betweenthe SMM or Tunnelling Unit and the target Charging Gateway (CG), or theacknowledgment messages from the CG are not getting back to the cdrsender.

If the connection to the Charging Gateway (CG) is lost, this also causes the queueto fill up. note that each GPLC can send this alarm, and that the alarm isgenerated per CG.

Summary

It is unlikely that the problem is caused by the cdrsender process itself, but by oneof the interfaces or elements between cdrsender and the CG. In most cases, analarm is raised that gives you details on what caused the problem.

When trying to locate the problem, it is best to work 'backwards', from the CGback to the cdrsender process. Check the traffic at each point to locate where themessages are being lost, then check the related network configuration.

Steps

1. Check active alarms.

Look for any charging-related alarms, in particular:

. 67016 NO CONNECTION TO CG

. 67015 CONNECTION LOST TO CDR SENDER

. 32333 CANNOT INITIALISE CHARGING

2. Check the GTP message log of the CG to see which messages are beingreceived from the 3G SGSN and which responses CG is sending back.

3. Check the traffic on any routers you are using between the 3G SGSNand CG to see if any message are being lost/blocked there.

4. Check the traffic on the Ethernet interface of the Forwarding Unit inquestion using sgndump.

5. Check the traffic on the VPP interface between the SMM orTunnelling Unit and Forwarding Unit in question using sgndump.



37 (50)

Software faults

If the acknowledgement messages are getting to the Forwarding Unit butnot to the SMM or Tunnelling Unit in question, then the problem is amissing static route. A static route must be configured between these unitsfor the response messages to get all the way back to the cdrsender process.

Further information

If the queue is full because the cdrsender process itself is not working, this isprobably a software bug that needs to be handled by Nokia. When informationNokia of such a case, you need to include the following information in yourreport:

. messages log file

. charging configuration parameter values

. alarm trap list

When dealing with connection problems to CG, keep in mind how the 3G SGSNselects CG addresses. For S-CDRs, the CG address is received from GGSN. Thereceived address is used if the address is in 3G SGSN list of allowed CGs(Charging Gateway configuration in Voyager). If the 3G SGSN checks theaddress against its own list and does not find it, the 3G SGSN selects an addressfrom the list based on configured priority levels.

All other CDR types are automatically sent to the CG address with the highestpriority according to the list in 3G SGSN.

Isolating the source of CDR creation failures

Steps

1. Check that the CDR type creation is enabled.

In Voyager you can find this information from the Basic ChargingConfiguration page.

The Tunnelling Unit creates S-CDRs, and the SMM Unit generates M-, S-SMO-, S-SMT-, LCS-MT, and LCS-MO-CDR types.

2. Check the current list of alarms.

Look for any charging-related alarms, in particular:

. 32335 CANNOT CREATE CDR

. 67012 CANNOT INIT SMM CHARGING

. 67013 CANNOT INIT TUN CHARGING




3. Monitor the charging counters.

There are CDR creation counters for each CDR type. Refresh the page inVoyager to see which counters are increasing. Any counter that does notincrease tells you exactly which CDR type is not being created.

Isolating the cause of corrupted CDRs

Summary

In most cases, if a CDR field is missing or has the wrong information, it usuallyconcerns information not generated by the 3G SGSN. So the configuration of thesource of the information should be checked first. The information usually is lostdue to either a failed inter-SGSN routing area update or a relocation procedurefails.

Steps

1. If the charging ID is missing from the CDR, check the GGSNconfiguration.

A missing charging ID usually indicates that the GGSN rejected the PDPcontext after the routing area update or relocation procedure.

2. If the MSISDN is missing from the CDR, check the HLRconfiguration.

A missing charging ID usually indicates that the HLR rejected the PDPcontext after the routing area update or relocation procedure.

10.3 SMM procedure fails

Description

Signalling sequences in Session and Mobility Management (SMM) proceduresare a potential source of problems in 3G SGSN due to the complex nature ofthese procedures in 3G networks. There are several points within each procedurethat can cause the entire sequence to fail. These troubleshooting instructions dealspecifically with the Attach and PDP Context Activation procedures.

Symptoms

SMM performance indicators recording failure instances increase rapidly.



39 (50)

Software faults

Recoveryprocedures

Interpreting the messages log file

Purpose

To locate the exact point of failure when an you have SMM problems, you needto carefully analyse the messages log file. Each SMM sequence generatescertain log entries that you can use as checkpoints, the objective being to findwhich checkpoints has an error message as the log entry.

Steps

1. Determine which SMM procedure failed.

Search the messages log file for the follow type of entry :

May 30 14:04:56 fe80::5501 [LOG_INFO] smm:

(1896:270284)

EXT_PROC_END: [MODULE]: [PROCEDURE NAME]: NOK:

cause=0x0000: [INFO] ([FILE]:[LINENRO])

The PROCEDURE NAME value indicates which procedure failed.

2. Analyse the messages log file.

See the examples below for details on which log entries to look for. Theentries correspond to certain checkpoints in the SMM procedure. Use theseto determine which phase (which interface) of the procedure failed.

3. Take corrective actions to fix the identified error.

The correct action of course depends on which phase of the SMMprocedure failed. However, the general guideline is to:

a. First check that the relevant parameters in 3G SGSN are configuredcorrectly.

b. If the 3G SGSN configuration is correct, check for configurationerrors elsewhere in the network. Also look for recent changes inyour network configuration that require 3G SGSN changes (thatwere not recognized or implemented at the time).

c. If you cannot locate any configuration problems, contact Nokia. Besure to provide all relevant error messages from the messages log.

Example 3. Attach sequence




The following figure illustrates the signalling sequence for the Attach procedurein 3G SGSN:

Figure 3. Attach sequence

UTRAN New SGSN Old SGSN

Attach message

Ue_state_Mm_Idl,Ue_state_Mm_con

Ue_state_Mm_det,Attach_proc,

After collision / OLC

Reuse cache hits, cache misses

Ue contexts in use

Identificationrequest

Identificationresponse

Identity request

Identity response

Identity Check procedureAuthentication

Auth. procedures, Security failuresIMEI Check

Common Id

Security ModeUpdate location

Insert subscriberdata

Insert subscriberdata ack

UpdateLocation Ack

Camel_GPRS_attach

Attach TDPs,Currently allocated GPRS sessions,

CAP dialogue failures,GPRS sessions continued afterCAP failure (def. handl. = cont)

Attach Accept /Reject

Attach failuresAttach Complete

HLR

17-20

EIR

12-13

6-7

2-3

MS

1

14

21

22

4-5

8-9

10-11

15-16

Identity Check procedure



41 (50)

Software faults

Each numbered step in the sequence produces an entry in the messages log file.The following are log entries that are produced at the corresponding checkpointin the Attach procedure.

1.May 30 14:00:16 fe80::5501 [LOG_INFO] smm: (1616:703218) EXT_MSG_IN: L3-

MM: Attach Request: trid=0x????????: buffer=08 01 02 E1 61 09 32 6B 08 29 44

50 01 10 00 80 50 42 F0 80 00 69 69 05 23 67 80 07 E0

(l3mm_msg_handler.cc:301) //191139

2.May 30 13:55:04 fe80::5501 [LOG_INFO] smm: (1304:199054) EXT_MSG_OUT:

GTP-C: Identification Request: teid=0x????????: seq_num=0x????:

ipv4_addr=10.20.82.131:2123: (gtpc_main_handler.cc:820) //146640a

3.May 30 13:55:04 fe80::5501 [LOG_INFO] smm: (1304:207240) EXT_MSG_IN: GTP-

C: Identification Response: teid=0x????????: seq_num=0x????:

ipv4_addr=10.20.82.131:2123: (gtpc_main_handler.cc:816) //146663

4.May 30 14:01:53 fe80::5501 [LOG_INFO] smm: (1713:734376) EXT_MSG_OUT: L3-

MM: Identity Request: trid=0x????????: buffer=08 15 01



MM: Identity Response: trid=0x????????: buffer=08 16 08 29 44

50 01 10 00 80 50 (l3mm_msg_handler.cc:487) //211255


MAP: sendAuthenticationInfo: trid=0x????????: muid=0x????????:

imsi=244051001000805f (mapu_proc_send_authentication_info.c:158)

7.May 30 14:01:53 fe80::5501 [LOG_INFO] smm: (1713:904369) EXT_MSG_IN: MAP:

sendAuthenticationInfoRes: trid=0x????????: muid=0x????????: error=

(mapu_proc_send_authentication_info.c:217) //211408


MM: Authentication And Ciphering Request: trid=0x????????: buffer=08 12 10

00 21 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 81 28 10 48 9A 4D 9B

FA 9D 24 17 00 01 48 9A 4D 9B FA 9D (l3mm_msg_handler.cc:131) //191268


MM: Authentication And Ciphering Response: trid=0x????????: buffer=08 13 00

22 11 23 31 47 23 09 43 09 31 17 90 25 00 01 F9 29 0C 51 63 71 8F 19 2B 39

4F 59 6B 79 0E (l3mm_msg_handler.cc:487) //191311


MM: Identity Request: trid=0x??????: buffer=08 15 03






MM: Identity Response: trid=0x????????: buffer=08 16 09 43 09 31 17 90 25 00

01 F9 (l3mm_msg_handler.cc:487) //150584


MAP: checkImei: trid=0x????????: muid=0x????????: imei=4901371095200109

(mapu_proc_check_imei.c:116) //191382


checkImeiRes: trid=0x????????: muid=0x????????: equip_status=0x00: error=

(mapu_proc_check_imei.c:145) //191408


RANAP: Common Id: trid=0x????????: connection_id=3c.ee (ranapu_sif.c:583)

//191430


RANAP: Security Mode Command: trid=0x????????: connection_id=3c.ee

(ranapu_sif.c:518) //191453

16.May 30 14:00:17 fe80::5501 [LOG_INFO] smm: (1617:056806) EXT_MSG_IN:

RANAP: Security Mode Complete: connection_id=3c.ee

(ranapu_successful_outcome_msg_handlers.c:112) //191466


MAP: updateGprsLocation: trid=0x????????: muid=0x????????:

imsi=244051001000805f (mapu_proc_update_gprs_location.c:143) //160114


insertSubscriberData: trid=0x????????: muid=0x????????:

imsi=244051001000805f (mapu_proc_insert_subscriber_data.c:71) //160139


MAP: insertSubscriberDataRes: trid=0x????????: muid=0x????????: error=

(mapu_proc_insert_subscriber_data.c:165) //160156


updateGprsLocationRes: trid=0x????????: muid=0x????????: error=

(mapu_proc_update_gprs_location.c:172) //160183


MM: Attach Accept: trid=0x????????: buffer=08 02 01 49 00 42 F0 80 00 69 69

19 19 DA 7F 18 05 F4 E5 3E 18 4E B8 (l3mm_msg_handler.cc:131) //191768



43 (50)

Software faults


MM: Attach Complete: trid=0x????????: buffer=08 03


Example 4. PDP Context Activation sequence

The following figure illustrates the signalling sequence for the PDP ContextActivation procedure in 3G SGSN:

Figure 4. PDP Context Activation sequence

Each numbered step in the sequence produces an entry in the messages log file.The following are log entries that are produced at the corresponding checkpointin the PDP Context Activation procedure.


SM: Activate PDP Context Request: mmid=a, ti=1, trid=0x000100d8

(l3sm_msg.cc:345) //160658

2. No log entry checkpoint.



GTP-C: Create PDP Context Request: teid=0x00000000: seq_num=0x0047:


MS RAN SGSN GGSN

Activate PDP Context

Create PDP Context

Create PDP Context

Update PDP Context

Update PDP Context

SetuRadio Access

Invoke

Activate PDP Context

C1

1

4

4

C2

6

8

8

9

5




May 30 13:56:14 fe80::5501 [LOG_INFO] smm: (1374:986746) EXT_MSG_IN: GTP

-C: Create PDP Context Response: teid=0x0a000050: seq_num=0x0047:



RANAP: Rab Assignment Request: trid=0x000100d8: connection_id=35.d2

(ranapu_sif.c:325) //161508

May 30 13:56:15 fe80::5501 [LOG_INFO] smm: (1375:006121) EXT_MSG_IN:

RANAP: RAB Assignment Response: connection_id=35.d2

(ranapu_outcome_msg_handlers.c:133) //161519




GTP-C: Update PDP Context Request: teid=0x0000ff51: seq_num=0x0065:

ipv4_addr=10.20.83.133:2123: (gtpc_main_handler.cc:820) //240997�

May 30 14:03:54 fe80::5501 [LOG_INFO] smm: (1834:916449) EXT_MSG_IN: GTP

-C: Update PDP Context Response: teid=0x0f000050: seq_num=0x0065:



SM: Activate PDP Context Accept: mmid=10, imsi=244051001000805F, ti=1,

BUFFER: 9A 42 05 0B 23 51 1F 91 96 40 48 46 00 00 00 01 27 1B 80 80 21 0A 03

00 00 0A 81 06 0A 10 3E E6 80 21 0A 04 00 00 0A 83 06 00 00 00 00 TRID:

100D8 (l3sm_msg.cc:1522) //161737



45 (50)

Software faults

11 Reporting faults to Nokia

After the system is up and running and the fault situation has been corrected withrecovery instructions, you should report the fault situation to Nokia by using theProblem Reporting Tool (refer to MPC Problem Reporting Guide).

The Problem Report (PR) helps analysing the fault situation and solving theproblem in the SGSN node. The PR should include a clear PR title, the faultdescription, and all the necessary file information. Provide the followinginformation to Nokia:

. A brief and precise problem description

. Information on how to reproduce the fault situation (for example step-by-step instructions)

. Impact to the user

- Severity of the problem for the user

- Likelihood of problem occurrence

. IPSO version used (obtain using the command uname -a)

. Hardware versions used. Obtain this information with the followingcommands:

ipsctl -a | grep hw:chassis:eeprom

ipsctl -a | grep hw:eeprom:serial_number

ipsctl -a | grep hw:eeprom:part_number

. Log files. To collect all the necessary files from the SGSN system fromboth CRPs, enter the following command:

ipsoinfo -c -a -l

The system saves the collected archive in the admin directory /var/admin. The file name is, for example, ipsoinfo-hostname-date.tar.gz.

Note



47 (50)

Reporting faults to Nokia

The ipsoinfo command requires a lot of processing memory. You should onlyuse it when you really need all system information. If you just need, for example,hardware information, use the ipsctl command.

Occasionally this information is not sufficient and you need to provide additionalinformation to Nokia. Additional information might include the network diagram,route information (enter the iclid, show route, quit command), tcpdump prints,interface information (enter the ifconfig �a command), configuration settings(enter the ipsctl -a command) etc. If such information is needed, Nokiarequests it separately with the Electra tool.




12 Glossary

ATM Asynchronous Transfer Mode

CLI Command Line Interface

CPU Central Processing Unit

CRP Central Routing Processor

CPCI Compact PCI

FCC Federal Communications Commission

FTP File Transfer Protocol

GPLC General Purpose Line Card

HTTP Hypertext Transfer Protocol

HTTPS Secure HTTP

IP Internet Protocol

IPSO Nokia Ipsilon Router Operating System

MMF Multi-mode Fibre

NEBS Network Equipment Building Standards

NIC Network Interface Card

NTP Network Time Protocol

PCI Peripheral Component Interconnect

SGSN Serving GPRS Support Node

SMF Single-mode fibre

SSH Secure Shell

SSL Secure Sockets Layer



49 (50)

Glossary

URL Uniform Resource Locator




SGSN Troubleshooting

Documents

Transcript of SGSN Troubleshooting