Onm Adm Administeringrnc50oms

Nokia Siemens Networks WCDMA RAN, rel. RU20, operating documentation, pre-release, issue 2

Administering RNC OMS

DN70339432

Issue 05 DRAFTApproval Date 2009/09/21

2 DN70339432Issue 05 DRAFT


Id:0900d80580625296

The information in this document is subject to change without notice and describes only the product defined in the introduction of this documentation. This documentation is intended for the use of Nokia Siemens Networks customers only for the purposes of the agreement under which the document is submitted, and no part of it may be used, reproduced, modified or transmitted in any form or means without the prior written permission of Nokia Siemens Networks. The documentation has been prepared to be used by professional and properly trained personnel, and the customer assumes full responsibility when using it. Nokia Siemens Networks welcomes customer comments as part of the process of continuous development and improvement of the documentation.

The information or statements given in this documentation concerning the suitability, capacity, or performance of the mentioned hardware or software products are given "as is" and all liability arising in connection with such hardware or software products shall be defined conclusively and finally in a separate agreement between Nokia Siemens Networks and the customer. However, Nokia Siemens Networks has made all reasonable efforts to ensure that the instructions contained in the document are adequate and free of material errors and omissions. Nokia Siemens Networks will, if deemed necessary by Nokia Siemens Networks, explain issues which may not be covered by the document.

Nokia Siemens Networks will correct errors in this documentation as soon as possible. IN NO EVENT WILL Nokia Siemens Networks BE LIABLE FOR ERRORS IN THIS DOCUMENTA-TION OR FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDI-RECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA,THAT MAY ARISE FROM THE USE OF THIS DOCUMENT OR THE INFORMATION IN IT.

This documentation and the product it describes are considered protected by copyrights and other intellectual property rights according to the applicable laws.

The wave logo is a trademark of Nokia Siemens Networks Oy. Nokia is a registered trademark of Nokia Corporation. Siemens is a registered trademark of Siemens AG.

Other product names mentioned in this document may be trademarks of their respective owners, and they are mentioned for identification purposes only.

Copyright © Nokia Siemens Networks 2009. All rights reserved

f Important Notice on Product Safety Elevated voltages are inevitably present at specific points in this electrical equipment. Some of the parts may also have elevated operating temperatures.

Non-observance of these conditions and the safety instructions can result in personal injury or in property damage.

Therefore, only trained and qualified personnel may install and maintain the system.

The system complies with the standard EN 60950 / IEC 60950. All equipment connected has to comply with the applicable safety standards.

The same text in German:

Wichtiger Hinweis zur Produktsicherheit

In elektrischen Anlagen stehen zwangsläufig bestimmte Teile der Geräte unter Span-nung. Einige Teile können auch eine hohe Betriebstemperatur aufweisen.

Eine Nichtbeachtung dieser Situation und der Warnungshinweise kann zu Körperverlet-zungen und Sachschäden führen.

Deshalb wird vorausgesetzt, dass nur geschultes und qualifiziertes Personal die Anlagen installiert und wartet.

Das System entspricht den Anforderungen der EN 60950 / IEC 60950. Angeschlossene Geräte müssen die zutreffenden Sicherheitsbestimmungen erfüllen.

DN70339432Issue 05 DRAFT

3


Id:0900d80580625296

Table of ContentsThis document has 169 pages.

Summary of changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1 Software management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.1 Software management in OMS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.1.1 Software subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.1.2 Software delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.1.3 Software set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.1.4 Software Version Viewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.2 Software management workflow in OMS. . . . . . . . . . . . . . . . . . . . . . . . 131.2.1 Upgrading incremental deliveries in OMS . . . . . . . . . . . . . . . . . . . . . . . 131.2.2 Activating an old software set in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . 151.2.3 Downgrading incremental deliveries in OMS. . . . . . . . . . . . . . . . . . . . . 161.3 Managing OMS software deliveries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191.3.1 Installing a single incremental software delivery in OMS . . . . . . . . . . . 191.3.2 Installing multiple incremental software deliveries in OMS . . . . . . . . . . 231.3.3 Querying software deliveries in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . 271.3.4 Checking current software deliveries in OMS. . . . . . . . . . . . . . . . . . . . 281.3.5 Verifying a software delivery in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . 291.3.6 Uninstalling an incremental software delivery in OMS . . . . . . . . . . . . . 301.4 Managing OMS software sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331.4.1 Creating a software set in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331.4.2 Listing software sets in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341.4.3 Activating a software set in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351.4.4 Querying software sets in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361.4.5 Checking the current software set in OMS . . . . . . . . . . . . . . . . . . . . . . 371.4.6 Removing a software set in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381.5 Managing LDAP of OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391.5.1 Upgrading LDAP data in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

2 Backup and restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412.1 Backup and restore in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412.2 Preparing for the backup of databases in OMS. . . . . . . . . . . . . . . . . . . 442.3 Making a full software backup in OMS . . . . . . . . . . . . . . . . . . . . . . . . . 452.4 Making a partial software backup in OMS . . . . . . . . . . . . . . . . . . . . . . 472.5 Making a custom software backup in OMS. . . . . . . . . . . . . . . . . . . . . . 492.6 Transferring the backup archive file from OMS to an external storage serv-

er . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512.7 Creating a self-contained USB rescue stick. . . . . . . . . . . . . . . . . . . . . . 522.8 Restoring the whole system in OMS using a backup server . . . . . . . . . 532.9 Restoring the whole system in OMS with self made full backup archive and

USB stick. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572.10 Restoring databases in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602.11 Restoring LDAP directory of OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622.12 Restoring a single file or directory in OMS . . . . . . . . . . . . . . . . . . . . . . 642.13 Backup of OMS fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66



Id:0900d80580625296

2.14 Restoring of database in OMS fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . 682.15 Restoring of LDAP directory of OMS fails . . . . . . . . . . . . . . . . . . . . . . . 702.16 Restoring of system image, single file, or directory in OMS fails . . . . . . 72

3 User management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 743.1 Management of user accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 743.2 User account storages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 773.3 Internal LDAP user management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 783.4 Command line interface for internal LDAP user management . . . . . . . . 793.4.1 Creating user accounts with CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 793.4.2 Modifying user accounts with CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 823.4.3 Deleting user accounts with CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 843.4.4 Creating user groups with CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 853.4.5 Deleting user groups with CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 863.4.6 Administering user-group mappings with CLI . . . . . . . . . . . . . . . . . . . . . 873.4.7 Administering group-permission mappings with CLI . . . . . . . . . . . . . . . . 893.4.8 Dumping service account data with CLI . . . . . . . . . . . . . . . . . . . . . . . . . 923.4.9 Generating a random password with CLI . . . . . . . . . . . . . . . . . . . . . . . . 943.4.10 Changing passwords for user accounts . . . . . . . . . . . . . . . . . . . . . . . . . 953.4.11 MMI mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 973.5 Credential service management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 983.5.1 Checking the password of a credential . . . . . . . . . . . . . . . . . . . . . . . . . . 983.5.2 Setting the password for a credential . . . . . . . . . . . . . . . . . . . . . . . . . . . 993.5.3 Changing the ownership of a credential . . . . . . . . . . . . . . . . . . . . . . . . 1013.5.4 Restoring a credential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1033.5.5 Deleting a credential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1043.6 Centralised user authentication and authorisation . . . . . . . . . . . . . . . . 1053.6.1 Centralised User Authentication and Authorisation . . . . . . . . . . . . . . . 1053.6.2 Centralised User Authentication and Authorisation replicator command line

interface tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1073.6.3 Enabling Centralised User Authentication and Authorisation . . . . . . . . 1103.6.4 Disabling Centralised User Authentication and Authorisation in OMS. 1113.6.5 Changing target external LDAP server . . . . . . . . . . . . . . . . . . . . . . . . . 112

4 Certificate management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1154.1 Certificate maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1154.2 Checking the expiration dates of OpenSSL certificates . . . . . . . . . . . . 1164.3 Updating X.509 certificates and private key . . . . . . . . . . . . . . . . . . . . . 1184.4 Checking that certificate and private key is a valid pair . . . . . . . . . . . . 120

5 Data management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1215.1 MySQL support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1215.2 Collecting MySQL log files in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . 1225.3 Monitoring MySQL mount points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1255.4 LDAP directory and parameter management . . . . . . . . . . . . . . . . . . . . 1295.4.1 Parameter management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1295.4.2 Management object model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1315.4.3 LDAP directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

6 Log management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136


5


Id:0900d80580625296

6.1 Syslog in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1366.2 Starter logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1386.3 Viewing syslog contents in OMS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1396.4 Enabling Starter logging in OMS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1416.5 Configuring log rotation in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1426.6 Enabling trace logs in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1446.7 OMS syslog is not working properly . . . . . . . . . . . . . . . . . . . . . . . . . . 145

7 Time management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1487.1 Time Management in OMS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1487.2 Configuring NTP services in OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . 1507.3 Testing NTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1527.4 Changing time zone settings in OMS. . . . . . . . . . . . . . . . . . . . . . . . . . 153

8 Fault management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

9 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

10 Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15710.1 Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15710.2 Scheduling backups in OMS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15910.3 Monitoring file system usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16110.4 Monitoring memory usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16510.5 Maintenance checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169



Id:0900d80580625296

List of FiguresFigure 1 Upgrading multiple incremental deliveries . . . . . . . . . . . . . . . . . . . . . . . 13Figure 2 Activating an old software set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Figure 3 Downgrading multiple incremental deliveries . . . . . . . . . . . . . . . . . . . . . 16Figure 4 Parameter management interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . 130Figure 5 MOM figure example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132Figure 6 Example of a Parameter Tool MOM view . . . . . . . . . . . . . . . . . . . . . . . 133Figure 7 Syslog master and syslog proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136Figure 8 Time management architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148


7


Id:0900d80580625296

List of TablesTable 1 Attributes for fsuseradd tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80Table 2 Attributes for fsusermod tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82Table 3 Attributes for fsuserdel tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84Table 4 Attributes for fsgroupadd tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85Table 5 Attributes for fsgroupdel tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86Table 6 Attributes for fsgpasswd tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87Table 7 Attributes for fspermadd tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89Table 8 Attributes for fspermdel tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90Table 9 Attributes for fsgroupperm -a command . . . . . . . . . . . . . . . . . . . . . . . . 90Table 10 Attributes for fsgroupperm -d command . . . . . . . . . . . . . . . . . . . . . . . . 91Table 11 fsdumpgroups tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92Table 12 fsdumpperm tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92Table 13 fsdumpusers tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93Table 14 The files created by the syslog.ng daemon. . . . . . . . . . . . . . . . . . . . . 137Table 15 Routine maintenance tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169



Id:0900d80580625296


9

Administering RNC OMS Summary of changes

Id:0900d805806217a7

Summary of changesChanges between document issues are cumulative. Therefore, the latest document issue contains all changes made to previous issues.

Note that the issue numbering system is changing. For more information, see Guide to WCDMA RAN operating documentation.

Changes between issue 05 and 4-2Sections Creating a self-contained USB rescue stick and Restoring the whole system with self made full backup archive and USB stick have been added.

Init level has been corrected in step 9 in section Restoring the whole system using the backup server.

Section Restoring the whole system using the backup server has been updated for the new hardware (the standalone RNC OMS).

Certificate paths have been updated in sections Updating X.509 certificates and private key and Checking that certificate and private key is a valid pair.

Changes between issue 4-2 and 4-1The following changes have been made to the document:

• Editorial changes • Chapter Credential service management has been added to the document. • Chapter Configuring NTP service in OMS has been updated with new procedure. • Step 13 and Further information section in chapter Installing a single incremental

software delivery in OMS have been updated. • Step 14 and Further information section in chapter Installing multiple incremental

software deliveries in OMS have been updated.

Changes between issue 4-1 and 4Editorial corrections have been made to the document.

New chapters Preparing for the backup of databases in OMS and Enabling trace logs in OMS have been added to the document.

Instructions in chapter Checking that certificate and privatekey is a valid pair have been updated.

Chapters Configuring log rotation in OMS and Monitoring MySQL mount points have been removed from the Maintenance section and moved under Log management and Data management sections.

A tip has been added to section Purpose in subchapter Configuring NTP service in OMS.

Changes between issue 4 and 3-3All commands in section Software management have been changed.

All login operations which are made by nokfsoperator have been changed to _nokfsoperator.

Chapter Software management in OMS has been updated with new commands. Infor-mation about accessing the Software Version Viewer has been added to section Software Version Viewer.



Id:0900d805806217a7

Summary of changes

A tip has been added before step 1 in Installing a single incremental software delivery in OMS.

SectionUninstalling multiple incremental software deliveries has been removed. All uninstalling instructions are described in Uninstalling an incremental software delivery in OMS.

A paragraph about backup failures and list of the databases that are not backed up have been added to section Backup types in Backup and restore in OMS.

Four new sections have been added to chapter Backup and restore in OMS. These sections are as follows: Administration of user accounts attributes, User home directo-ries, Credential service and Centralised user authentication and authorisation.

Sections Operating system and Internal LDAP server have been added to User account storages.

Chapter Remote user information management has been substituted by Centralised user authentication and authorisation.

A description of new options available with fsuseradd command has been added to the chapter Creating user accounts with CLI. Also two examples of creating user account with default and new attributes' values have been added to this chapter.

Section Modifying user accounts with CLI has been added to chapter Command line interface for internal LDAP user management.

Information about data bases has been removed from section My SQL support.

Step 1 has been inserted to the procedure of Collecting MySQL log files in OMS.

Expected and Unexpected outcome sections have been added to Configuring log rota-tation in OMS section.

Step 3 has been added to chapter Monitoring memory usage.


11

Administering RNC OMS Software management

Id:0900d8058056ab3a

1 Software management

1.1 Software management in OMSSoftware management (SWM) provides the function needed for supporting the installa-tion and online upgrades in the network element (NE). It is also used in commissioning.

The operator may have to install new software versions or modify the existing software due to bug corrections, security patches or new features. The software upgrade is done in the background and then activated, which means that the services are not disturbed.

The software version management utility is the Red Hat Package Manager (RPM) used in most Linux distributions to install software in the Staging Area (SA).

The SWM commands are implemented as shell scripts. There are three different types of commands:

• Software delivery management commands for managing the contents of the SA: • fsswcli --delivery --install

• fsswcli --delivery --uninstall • fsswcli --delivery --query

• fsswcli --delivery --current

• fsswcli --set --verify • Software set management commands for managing software sets in the system:

• fsswcli --set --make

• fsswcli --set --list

• fsswcli --set --query • fsswcli --set --current

• fsswcli --set --activate

• fsswcli --set --remove • LDAP upgrade commands

• fsswcli --ldap --upgrade for managing LDAP changes associated with software upgrades

• fsswcli --ldap --activate for activating the new LDAP data.

It is recommended that user of the software management commands and scripts has a basic knowledge of Linux and a good understanding of RPM (see the RPM man pages for detailed information).

1.1.1 Software subsystemA subsystem is the unit of software that can be installed and upgraded separately. It is a collection of software components and files, and the installation scripts operating in the OMS. Every software component in OMS belongs to a subsystem. A subsystem implements a part of the function of the system, for example, alarm management feature.

1.1.2 Software deliveryA software delivery contains the software items related to certain software shipment and the installation tools. A software delivery consists of one RPM package which contains one or more subsystem RPM packages.



Id:0900d8058056ab3a

Software management

There are two kinds of software deliveries: a base delivery and an incremental delivery.

Base deliveryA base delivery is a complete software build that is delivered to a NE for installation and it is used for the commissioning of a new system. A base delivery contains all packages of all the subsystems the NE is intended to contain.

A base delivery cannot be uninstalled.

Incremental deliveryAn incremental delivery is used for delivering corrections and function enhancements to an existing system. It contains the RPM packages of the subsystems that have changed from the base delivery, previously installed incremental deliveries, or new RPM pack-ages. An incremental delivery can be installed to an operational NE.

The first incremental delivery is built on top of the base delivery. The following incremen-tal deliveries are built on top of the previous incremental deliveries and they depend on the previously installed incremental deliveries.

To delete an incremental delivery, all the successor deliveries that are hierarchically on top of the current delivery must be also deleted.

1.1.3 Software setA software set is a full set of subsystems required for a certain software release. Before the system can take the software into use, you must create a software set from the deliv-eries that are installed in the SA. When you create a software set, you take a snapshot of the contents of the SA and copy it to a software set, name it, and take it into use. In other words, a software set is a named and stored snapshot of the contents of the RPM packages installed in the SA at a certain moment.

1.1.4 Software Version ViewerYou can check the OMS software versions with the web UI of Software Version Viewer.

Log into OMS web page (https://<OMS IP>/swversion.html) with Nemuadmin username. You can find Version Viever and Log Viewer under Element management link. Open Element management link and log with Nemuadmin username and nemuuser password.


13


Id:0900d80580550de8

1.2 Software management workflow in OMS

1.2.1 Upgrading incremental deliveries in OMS

PurposeTo upgrade multiple incremental deliveries and create one software set that includes all the deliveries to save disk space. This is preferred when you just want to take all the latest changes into use and do not need to create separate software sets for each deliv-ery.

SummaryRuntime upgrades are installed to the system as incremental deliveries. The procedures are divided into phases according to the software management commands needed. The commands must be executed in this particular order to achieve a successful software uprade.

The following figure illustrates the workflow when you are installing multiple incremental deliveries to one single software set.

Figure 1 Upgrading multiple incremental deliveries

Steps

1 Install incremental deliveries.Install all deliveries with the fsswcli --delivery --install command. When installing multiple incremental deliveries, list the names of the delivery packages as argument. For more information, see Installing a single incremental software delivery in OMS and Installing multiple incremental software deliveries in OMS

Install incremental deliveries

Upgrade LDAP data

Activate software set

Start

End

Create a new software set



Id:0900d80580550de8

2 Create a software setA new software set is created automatically after executing fsswcli --delivery --install command, however the software set is not created when fsswcli --delivery --install is executed with -N switch. Then a new software set can be created by executing the fsswcli --set --make command. For more instructions, see Creating a software set in OMS.

3 Upgrade LDAP data.Upgrade the LDAP data by executing the fsswcli --ldap --upgrade command. After executing this command the LDAP back end database for the software set exists, but it is not yet taken into use. For instructions, see Upgrading LDAP data in OMS.

4 Activate a software set.The new software set is activated with the fsswcli --set --activate command. Note that executing the command causes an automatic restart. During restarting, the new LDAP data is automatically activated.

After the restart is completed, the new software set is taken into use.

For instructions, see Activating a software set in OMS.

Example: To install three incremental deliveries named R_OMS1_4.29.release_oms.corr1-inc-1.rpm, R_OMS1_4.29.release_oms.corr2-inc-1.rpm, and R_OMS1_4.29.release_oms.corr3-inc-1.rpm, enter the following commands:

fsswcli --delivery --install R_OMS1_4.29.release_oms.corr1-inc-1.rpm \R_OMS1_4.29.release_oms.corr2-inc-1.rpm \R_OMS1_4.29.release_oms.corr3-inc-1.rpmfsswcli --set --make R_OMS1_4.29.release_oms.ALL-3fsswcli --ldap --upgrade R_OMS1_4.29.release_oms.ALL-3fsswcli --set --activate R_OMS1_4.29.release_oms.ALL-3

The character\ is used in this chapter to indicate that a command continues to the next line. Command lines ending with\ must not be entered as separate commands.


15


Id:0900d805802a926e

1.2.2 Activating an old software set in OMS

PurposeDowngrade a software set by activating an existing software set.

Note that the software downgrade is possible only if at least two software sets are created.

SummaryUse these instructions to use the previous software from a software set without removing the current software delivery from the system.

The following figure illustrates the workflow.

Figure 2 Activating an old software set

Steps

1 Activating a software set.The new software set in the is activated with the fsswcli --set --activate command. Note that executing the command causes an automatic restart. During restarting, the new LDAP data is automatically activated.



Example: The node uses the software set R_OMS1_4.29.release_oms.corr5. To downgrade the software to use the previous set named , enter the following command:

fsswcli --set --activate R_OMS1_4.29.debug_oms.corr5


Start

End



Id:0900d805803c64fe

1.2.3 Downgrading incremental deliveries in OMS

PurposeDowngrade an incremental delivery by uninstalling and removing some of the multiple incremental deliveries that were combined into one software set, making a new software set of the remaining incremental deliveries, and activating it.

SummaryUse these instructions to revert back to a previous delivery level and create a new software set if multiple deliveries are installed but only one software set is created. After you have uninstalled some of the multiple incremental deliveries and removed them, create a new software set from the incremental deliveries that still exist in the system and remove the old software set.

The following figure illustrates the workflow.

Figure 3 Downgrading multiple incremental deliveries

Steps

1 Uninstall incremental deliveries.Downgrade the software on the system from the current incremental delivery level to the previous incremental delivery level by unistalling multiple incremental deliveries. The latest incremental delivery can be uninstalled with the fsswcli --delivery --uninstall <delivery name> command and the number of incremental deliveries

Start

Uninstall incremental deliveries

End


Create software set

Remove software set

Upgrade LDAP data


17


Id:0900d805803c64fe

can be uninstalled with the fsswcli --delivery --uninstall –N <count> command. For instructions, see Uninstalling an incremental software deliveries in OMS.

2 Create a software set.Create a new software set from the remaining incremental deliveries with fsswcli --set --make <SW set name> command. For instructions, see Creating a software set in OMS.

3 Upgrade LDAP data.Upgrade the LDAP data by executing the fsswcli --ldap --upgrade command. After executing this command the LDAP back end database for the software set has been created, but it is not yet taken into use. For instructions, see Upgrading LDAP data in OMS.

4 Activate a software set.The new software set in the OMS is activated with the fsswcli --set --activate command. Note that executing the command causes an automatic restart. During restarting, the new LDAP data is automatically activated.



5 Remove a software set.Remove the old software set with fsswcli --set --remove <SW set name> command. For instructions, see Removing a software set in OMS.

Example: You have 10 incremental deliveries (R_OMS1_4.29.release_oms.corr1... R_OMS1_4.29.release_oms.corr10) installed into one software set R_OMS1_4.29.release_oms.corr10. To uninstall the latest incremental delivery, enter the following commands:

fsswcli --delivery --uninstall R_OMS1_4.29.release_oms.corr10fsswcli --set --make R_OMS1_4.29.release_oms.corr9fsswcli --ldap --upgrade R_OMS1_4.29.release_oms.corr9fsswcli --set --activate R_OMS1_4.29.release_oms.corr9fsswcli --set --remove R_OMS1_4.29.release_oms.corr10

Example: You have 10 incremental deliveries (R_OMS1_4.29.release_oms.corr1... R_OMS1_4.29.release_oms.corr10) installed into one software set R_OMS1_4.29.release_oms.corr10. To downgrade four levels to level 6, enter the following commands:

fsswcli --delivery --uninstall -N 4fsswcli --set --make R_OMS1_4.29.release_oms.corr6fsswcli --ldap --upgrade R_OMS1_4.29.release_oms.corr6



Id:0900d805803c64fe

fsswcli --set --activate R_OMS1_4.29.release_oms.corr6fsswcli --set --remove R_OMS1_4.29.release_oms.corr10


19


Id:0900d8058056c8cb

1.3 Managing OMS software deliveries

1.3.1 Installing a single incremental software delivery in OMS

PurposeTo install a single correction set on top of the OMS base package.

For instructions on how to remove a single incremental software delivery, see Uninstall-ing an incremental software delivery in OMS.

SummaryUse the fsswcli --delivery --install command to install all subsystem RPM packages in an incremental software delivery.

The syntax of the command is as follows:

fsswcli --delivery --install [-d | -N] [-f] [-l <log file>] [--no-rollback] <OMS_SWBUILD_delivery package>...

-h The -h prints the help for this command.

-d The -d does not import configuration files from the current set.

-N The -N does not create a software set.

-f The -f executes a forceful RPM installation, that is, installs the packages even if some of them are already installed on the system, overwriting the already installed packages. By default, the system does not install packages that are already installed.

-l <log file> The -l <log file> specifies the log file.

--no-rollback The --no-rollback omits automatic rollback in failure cases.

<OMS_SWBUILD_delivery package> The incremental delivery to be installed. Note that the full paths must be given. For example: R_OMS1_3.34.release_oms.corr1-inc-1.rpm

Steps

1 Download the incremental deliveries from NOLS.

a) Log in to NOLS at www.online.nokia.com. b) From the NOLS main page, select Software Delivery and find the relevant delivery.c) Download the relevant incremental correction delivery or deliveries

(<OMS_SWBUILD>.corr<x>-inc-1.rpm) to an FTP server or client computer.

2 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase.For more information, see Logging in to OMS in Installing and commissioningRNC OMS.

3 Create a download directory for the incremental deliveries.Run the following command:

mkdir /home/_nokfsoperator/inc_download_dir



Id:0900d8058056c8cb

4 Copy the incremental RPM package to the MCP18-B OMS.Copy the incremental RPM package (<OMS_SWBUILD>.corr<x>-inc-1.rpm) from the FTP server or client computer to the inc_download_dir directory on the MCP18-B OMS. Copying is done using scp in Linux or WinSCP in Windows.

5 Change to the root user permission by entering su - command.

6 Change to the download directory. Run the following command:

cd /home/_nokfsoperator/inc_download_dir

7 Install the increment RPM package.To install the increment run the following command::

fsswcli --delivery --install <OMS_SWBUILD>.release_oms.corr<x>-inc-1.rpm

For example:

fsswcli --delivery --install R_OMS1_3.34.release_oms.corr1-inc-1.rpm

8 Upgrade LDAP data.Run the following command:

fsswcli --ldap --upgrade <OMS_SWBUILD>.release_oms.corr<x>

☞ To get the name of the incremental correction set, press the TAB key.

For example:

fsswcli --delivery --upgrade R_OMS1_3.34.release_oms.corr1

Expected outcomeThe LDAP database for the software set is created.

9 Import LDIF to the LDAP database.If you are prompted to import the LDIF to the LDAP database, enter y (yes).

10 Activate the incremental correction set.To activate the incremental correction set, run the following command:

fsswcli --set --activate <OMS_SWBUILD>.release_oms.corr<x> [Node]

Example: fsswcli --set --activate R_OMS1_4.29.release_oms.corr2 CLA-0


21


Id:0900d8058056c8cb

Expected outcomeThe increment correction is activated and the OMS reboots. The new LDAP data is automatically activated.

11 Wait until OMS is rebooted.After reboot, the new incremental software set is taken into use in the OMS.

12 Verify that the incremental sets are installed correctly.Run the following command:

fsswcli --set --current

Expected outcomeThe fsswcli --set --current command lists the OMS base and incremental software sets that are currently installed and active.

☞ To list the packages and RPM files, use the fsswcli --delivery --current command.

Unexpected outcomeIf there are problems with the installation or increments are missing, re-install the incre-ment.

13 Copy the index.html if the correction updates the Application Launcher.If the incremental correction includes correction for Application Launcher (SS_AppLauncher), give following command:

cp -vf /opt/Nokia/SS_Modefarbe/www/index.html /var/opt/Nokia/www

Expected outcomeThe index.html is copied to OMS WWW-root.

Further information

g Number of sets is limited by the boot partition disk size. Set size can vary, but usually system can handle 6-10 sets. Amount of sets can be viewed with listsets command. When set limit is reached, making new set fails with following error message:

ERROR Not enough space on boot partition /dev/sda1

When encountering this error, do not try to clear /dev/sda1 manually, but remove unneeded sets by executing fsswcli --set --remove command. Then continue by uninstalling corrections and reinstalling them normally. Boot partition disk space can also be checked with the following commands:

#mkdir /root/mount#mount /dev/sda1 /root/mount#df -h /dev/sda1#umount /root/mount#rmdir /root/mount



Id:0900d8058056c8cb

Example: Example of the output:

# df -h /dev/sda1Filesystem Size Used Avail Use% Mounted on/dev/sda1 99M 46M 49M 49% /var/mnt/local/localimg/root/mount

Making new set requires at least 25Mb of free space.


23


Id:0900d805805685e7

1.3.2 Installing multiple incremental software deliveries in OMS

PurposeTo install multiple incremental software deliveries, that is, correction sets on top of the OMS base package.

For instructions on how to remove multiple incremental software deliveries, see Unin-stalling an incremental software delivery in OMS.

SummaryUse the fsswcli --delivery --install command to install all subsystem RPM packages in an incremental software delivery.


fsswcli --delivery --install [-d | -N] [-f] [-l <log file>] [--no-rollback] <OMS_SWBUILD_delivery package>...

-h The -h prints help for this command.

-d The -d does not import configuration files from the current set.

-N The -N does not create a software set.

-f The -f executes a forceful RPM installation, that is, installs the packages even if some of them are already installed on the system, overwriting the already installed packages. By default, the system does not install packages that are already installed.

-l <log file> The -l <log file> specifies the log file.

--no-rollback The --no-rollback omits automatic rollback in failure cases.

<OMS_SWBUILD_delivery package> The incremental delivery to be installed. Note that the full paths must be given. For example: R_OMS1_4.29.release_oms.corr1-inc-1.rpm

Steps

1 Download the incremental deliveries from NOLS.

a) Log in to NOLS at www.online.nokia.com. b) From the NOLS main page, select Software Delivery and find the relevant delivery.c) Download the relevant incremental correction deliveries

(<OMS_SWBUILD>.corr<x>-inc-1.rpm) to an FTP server or client computer.

2 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase.For more information, see Logging in to OMS in Installing and commissioningRNC OMS.

3 Create a download directory for the incremental deliveries.Run the following command:

mkdir /home/_nokfsoperator/inc_download_dir



Id:0900d805805685e7

4 Copy the incremental RPM packages to the MCP18-B OMS.Copy the incremental RPM packages (<OMS_SWBUILD>.corr<x>-inc-1.rpm) from the FTP server or client computer to the inc_download_dir directory on the MCP18-B OMS. Copying is done using scp in Linux or WinSCP in Windows.

5 Change to the root user permission by entering su - command.

6 Change to the download directory.Run the following command:

cd /home/_nokfsoperator/inc_download_dir

7 Install the increment RPM package(s).To install multiple increments run the following commands:

fsswcli --delivery --install -N <OMS_SWBUILD>.release_oms.corr<first_inc>-inc-1.rpm <OMS_SWBUILD>.release_oms.corr<second_inc>-inc-1.rpm

or

fsswcli --delivery --install -N <OMS_SWBUILD>.release_oms.corr<first_inc>-inc-1.rpm

fsswcli --delivery --install -N <OMS_SWBUILD>.release_oms.corr<second_inc>-inc-1.rpm

For example:

fsswcli --delivery --install -N R_OMS1_4.29.release_oms.corr1-inc-1.rpm R_OMS1_4.29.release_oms.corr2-inc-1.rpm

8 Create a software setRun the following commands

fsswcli --set --make <OMS_SWBUILD>.release_oms.corr<latest_inc>

This is the latest increment you have installed.

For example:

fsswcli --set --make R_OMS1_4.29.release_oms.corr2

☞ To list the sets, use the fsswcli --set --list command.

9 Upgrade LDAP data.Run the following command:

fsswcli --ldap --upgrade <OMS_SWBUILD>.release_oms.corr<x>

For example:

fsswcli --ldap --upgrade R_OMS1_4.29.release_oms.corr2

☞ To get the name of the incremental correction set, press the TAB key.


25


Id:0900d805805685e7

Expected outcomeThe LDAP database for the software set is created.

10 Import LDIF to the LDAP database.If you are prompted to import the LDIF to the LDAP database, enter y (yes).

11 Activate the incremental correction set.To activate the incremental correction set, run the following command:

fsswcli --set --activate <OMS_SWBUILD>.release_oms.corr<x> [Node]

Example: fsswcli --set --activate R_OMS1_4.29.release_oms.corr2 CLA-0

Expected outcomeThe increment correction is activated and the OMS reboots. The new LDAP data is automatically activated.

12 Wait until OMS is rebooted.After the reboot, the new incremental software set is taken into use in the OMS.

13 Verify that the incremental sets are installed correctly.Run the following command:


Expected outcomeThe fsswcli --set --current command lists the OMS base and incremental software sets that are currently installed and active.

☞ To list the packages and RPM files, use the fsswcli --delivery --current command.

To list the sets, use the fsswcli --set --list command

Unexpected outcomeIf there are problems with the installation or increments are missing, re-install the incre-ments.

14 Copy the index.html if the correction updates the Application Launcher.If the incremental correction includes correction for Application Launcher (SS_AppLauncher), give following command:

cp -vf /opt/Nokia/SS_Modefarbe/www/index.html /var/opt/Nokia/www

Expected outcomeThe index.html is copied to OMS WWW-root.



Id:0900d805805685e7

Further information

g Number of sets is limited by the boot partition disk size. Set size can vary, but usually system can handle 6-10 sets. Amount of sets can be viewed with listsets command. When set limit is reached, making new set fails with following error message:

ERROR Not enough space on boot partition /dev/sda1

When encountering this error, do not try to clear /dev/sda1 manually, but remove unneeded sets by executing fsswcli --set --remove command. Then continue by making new set normally. Boot partition diskspace can allso be checked with following commands:

#mkdir /root/mount#mount /dev/sda1 /root/mount#df -h /dev/sda1#umount /root/mount#rmdir /root/mount

Example: Example of the output:

# df -h /dev/sda1Filesystem Size Used Avail Use% Mounted on/dev/sda1 99M 46M 49M 49% /var/mnt/local/localimg/root/mount

Making new set requires at least 25Mb of free space.


27


Id:0900d80580424fee

1.3.3 Querying software deliveries in OMS

SummaryUse the fsswcli --delivery --query command to make RPM queries to the RPM database of the Staging Area.


fsswcli --delivery --query <RPM query options>

You have the following option:

• RPM query options: the options for the rpm -q command. See the RPM man pages for more information.

Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase, and then change the root user permission by entering su - command.

2 Query the software delivery.Enter:

fsswcli --delivery --query <RPM query options>

Further informationThe fsswcli --delivery --query command is an alias for the rpm -q command.



Id:0900d8058034d92b

1.3.4 Checking current software deliveries in OMS

SummaryCheck the paths of the delivery chain from the base delivery to the current delivery to find out what has been installed earlier. List the delivery names and the RPM packages of which it consists.


fsswcli --delivery --current [-h]


• -h: prints help for this command.

Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase, and then change to the root user permission by entering su - command.

2 Check the software delivery.Enter:

fsswcli --delivery --current

Expected outcomeThe path of delivery chains is listed starting from the base delivery and ending to the current delivery.


29


Id:0900d805804064cd

1.3.5 Verifying a software delivery in OMS

SummaryUse the fsswcli --delivery --verify command to verify that the delivery RPM package has been installed and that all the RPM packages the delivery package contains have also been installed. Also verify that all the RPM packages from the previous deliveries have been installed and that no other unprecedented RPM packages are installed.


fsswcli --delivery --verify { -h | <delivery name> }

<delivery name> is the name of the latest delivery. It is a mandatory argument.

In addition, you have the following option:


Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase, and then change to the root user permission by entering su - command.

2 Verify the delivery.Enter:

fsswcli --delivery --verify <delivery name>

Expected outcomeThe RPM packages are listed and the information in them is displayed in the format fsswcli --delivery --verify: ERROR in <SS name> <reason>. <SS name> refers to the name of the subsystem, and the <reason> can be as follows:

• size changed • mode changed • MD5 checksum changed • major/minor device number changed • symbolic link changed.

Example: Verifying a deliveryTo verify a delivery called R_OMS1_4.29.release_oms.corr1 enter: fsswcli --delivery --verify R_OMS1_4.29.release_oms.corr1.



Id:0900d80580498b7d

1.3.6 Uninstalling an incremental software delivery in OMS

PurposeTo uninstall an incremental software delivery that was incorrectly installed. Note that user can only remove an incremental software delivery that was installed after commis-sioning. Increments installed during commissioning cannot be removed with this proce-dure.

SummaryUse the fsswcli --delivery --uninstall command to uninstall the incremental deliveries and to downgrade the current incremental delivery to the previous level.


fsswcli --delivery --uninstall [-f] [-l <log file>] [--no-rollback] \ [-N <count> | <delivery name>]

<delivery name> is the name of the latest incremental delivery to be uninstalled. It is a mandatory argument. Note that with the <delivery name> argument only the latest incremental delivery can be uninstalled.

In addition, you have the following options:

• -h prints help for this command. • -f executes the forceful uninstallation. • -l <log file> specifies the log file where the logged events are directed. • --no-rollback omits automatic rollback in failure cases. • -N <count> the number of the incremental deliveries to be uninstalled.

Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase, and then change to the root user permission by entering the su - command.For more information, see Logging in to OMS in Installing and commissioning RNC OMS.

2 Check which incremental software set is currently active.Run the following command:


3 Activate the previous base or incremental software set.Run the following command:

fsswcli --set --activate <OMS_SWBUILD>

Orfsswcli --set --activate <OMS_SWBUILD>.release_oms.corr<x-1>

Example: To activate the R_OMS1_4.29.release_oms.corr1 set if the R_OMS1_4.29.release_oms.corr2 is the active set run the following command:


31


Id:0900d80580498b7d

fsswcli --set --activate R_OMS1_4.29.release_oms.corr1

Expected outcomeThe previous base or incremental software set is activated. OMS reboots.

4 Wait for OMS to reboot.

5 Check if the previous software set is now the active set. Check which set is active by running the following command:


Expected outcomeThe output shows the previous software set as the new active set.

6 Remove the incremental software set.Run the following command:

fsswcli --set --remove <OMS_SWBUILD>.release_oms.corr<x>

Example: To remove the R_OMS1_4.29.release_oms.corr2 set, run the following command:

fsswcli --set --remove R_OMS1_4.29.release_oms.corr2

7 Uninstall the latest increment.Run the following command for the latest RPM package:

fsswcli --delivery --uninstall <OMS_SWBUILD>.release_oms.corr<x>

OrYou can also remove a multiple increments by running the following command for each RPM package you want to uninstall:fsswcli --delivery --uninstall <OMS_SWBUILD>.release_oms.corr<x>

Example: To remove the R_OMS1_4.29.release_oms.corr4, R_OMS1_4.29.release_oms.corr3 and R_OMS1_4.29.release_oms.corr2 sets, run the following commands:

fsswcli --delivery --uninstall R_OMS1_4.29.release_oms.corr4



Expected outcomeThe increment is uninstalled. You can now try to re-install the increment again if needed.

☞ If you want to remove individual increment RPMs from the OMS, run the following command:

rm <OMS_SWBUILD>.release_oms.corr<x>-inc-1.rpm



Id:0900d80580498b7d

rm is a Linux delete file command. The example above has to be executed in direc-tory, where rpm package is located. Another way is to give rm command full path:

rm /home/_nokfsoperator/inc_download_dir/<OMS_SWBUILD>.release_oms.corr<x>-inc-1.rpm


33


Id:0900d8058055770b

1.4 Managing OMS software sets

1.4.1 Creating a software set in OMS

SummaryCreate a software set out of the Staging Area (SA) to the system image and distribute it to the local images with the fsswcli --set --make command.


fsswcli --set --make { -h | [-d] [-l <log file>] [-n] <SW set name> }

<SW set name> is the name of the software set to be created. It is a mandatory argu-ment.

The software set name must start with a letter of the alphabet followed by alphanumeric characters. The characters dot (.), hyphen (-), and underscore (_) are also allowed.


• -h: prints help for this command. • -l <log file>: specifies the log file where the logged events are directed. By

default the log file is/var/mnt/local/sysimg/flexiserver/opt/Nokia/var/swmgmt/SS_SoftwareManagement.log

Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase, and then change the root user permission by entering su - command.For more information, see Logging in to OMS.

2 Create the software set.Enter:

fsswcli --set --make [options] <SW set name>

Example: Creating a software setTo create a software set R_OMS1_4.29.release_oms.corr1 and distribute it to the local images, enter:

fsswcli --set --make R_OMS1_4.29.release_oms.corr1.



Id:0900d80580432a05

1.4.2 Listing software sets in OMS

SummaryUse the fsswcli --set --list command to list all the available software sets.


fsswcli --set --list [-h]



Steps


2 List the available software sets.Enter:

fsswcli --set --list

Example: Viewing help for the fsswcli --set --list commandEnter:

fsswcli --set --list -h


35


Id:0900d8058049c4cb

1.4.3 Activating a software set in OMS

SummaryUse the fsswcli --set --activate command to activate a software set in the node.


fsswcli --set --activate { -h | [-l <log file>] [-n] <SW set name> }

<SW set name> is the name of the software set to be activated. It is a mandatory argu-ment.



default the log file is:/var/mnt/local/sysimg/flexiserver/opt/Nokia/var/swmgmt/SS_SoftwareManagement.log

Note that executing the fsswcli --set --activate command causes the node to reboot automatically.

Steps


2 Activate the software set.Enter:

fsswcli --set --activate [options] <SW set name> [node]

Note that executing the fsswcli --set --activate command causes the node to reboot automatically.

Example: To activate the R_FSPR4CHA_1.18.release.corrections03 enter the following command:

fsswcli --set --activate R_FSPR4CHA_1.18.release.corrections03 CLA-0



Id:0900d8058037366f

1.4.4 Querying software sets in OMS

SummaryUse the fsswcli --set --query command to make RPM queries to the RPM database of the given software set.


fsswcli --set --query <SW set name> <RPM query options>

<SW set name> is the name of the software set whose RPM database will be used for the RPM queries. It is a mandatory argument.


• RPM query options: the options for the rpm -q command. See the RPM man pages for more information.

Steps


2 Query the software set.Enter:

fsswcli --set --query <SW set name> <RPM query options>

Further informationThe fsswcli --set --query command is an alias for the rpm -q command.


37


Id:0900d8058035cc95

1.4.5 Checking the current software set in OMS

SummaryUse the fsswcli --set --current command to check the current software set and list the currently active software sets.


fsswcli --set --current [-h]

You have the following options:


Steps


2 List the currently active software sets.Enter:




Id:0900d805804f2be8

1.4.6 Removing a software set in OMS

SummaryUse the fsswcli --set --remove command to remove a software set from the node if it is not used.


fsswcli --set --remove { -h | [-l <log file>] [-n] <SW set name> }

<SW set name> is the name of the software set to be removed. It is a mandatory argu-ment.




• -n lists the intended actions but does not commit them.

Steps


2 Remove the software set.Enter:

fsswcli --set --remove [options] <SW set name>

Expected outcomeThe software set is removed from the node.


39


Id:0900d805805558ee

1.5 Managing LDAP of OMS

1.5.1 Upgrading LDAP data in OMS

SummaryUse the fsswcli --ldap --upgrade command to create the LDAP back end database for the desired software set using the LDAP database of the current software set as the basis, that is, the current LDAP database is combined with the additions brought by the new incremental deliveries.

The syntax for the command is as follows:

fsswcli --ldap --upgrade { -h | [-c <deployment script>] [-i] \ [-I <IP script>] [-l <log file>] [-n] <SW set name> }

<SW set name> is the name of the proposed software set. It is a mandatory argument.


• -h: Prints help for this command. • -c <deployment script> : Regenerates the deployment data using

<deployment script>

• -i : Asks confirmation for each LDIF file before importing it. • -I <IP script> : Regenerates the IP configuration using <IP script>. • -l <log file>: specifies the log file where the logged events are directed. By


• -n : Lists the intended actions but does not commit them.

Steps


2 Upgrade the LDAP data.Enter:

fsswcli --ldap --upgrade [options] <SW set name>

Expected outcomeThe LDAP backend database is created for the desired software set but is not used by any running LDAP server.

Unexpected outcomeIf executing the fsswcli --ldap --upgrade command causes errors while import-ing the related ldif file and the LDAP will not be updated with the new schema, try to use the new version of upgradeldapdata command. Enter the following command:

/var/mnt/local/localimg/flexiserver/sets/<SW set name>/opt\/Nokia_BP/sbin/upgradeldapdata <SW set name>



Id:0900d805805558ee

If you receive the error message that says that the command is already run for this set, delete the setname directory and try again:

rm -r /var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/var/pmgmt/<SW set name>

The <SW set name> is the latest software set where you are upgrading to.

Example: Upgrading the LDAP dataEnter:

fsswcli --ldap --upgrade R_FSPR4CHA_1.18.release.corrections01

Example: Upgrading the LDAP data and regenerating the deployment data

Enter:fsswcli --ldap --upgrade -i -c create_hppro.sh R_FSPR4CHA_1.18.release.corrections01


41

Administering RNC OMS Backup and restore

Id:0900d80580621b36

2 Backup and restore

2.1 Backup and restore in OMSMaking regular backup copies of the software and databases of the OMS ensures that user has a functional copy of the software which can be used if there are problems with the software or hardware, for example, a logical disk crash or a configuration error.

User can make a full, a partial, or a custom backup of the software. All types of backups are made using the fsbackup shell script. When running the script, user has to be logged into the OMS as root.

User can restore the whole system or a part of the system with the fsrestore shell script.

Backup typesA backup can be a full, partial, or custom backup. A backup is modular, meaning that the backup procedure consists of executing backup modules. Each module can create the backup archive of a backup item, for example the database backup module creates a backup of the databases. Note that all items are not necessarily used in all deploy-ments.

Note that the backup should be taken when there are no major on-going activities in the network element. For example, the backup may occasionally fail if the LDAP Directory is manipulated at the time when the backup is taken.

• A full backup includes the following items: • system image, including variable data and configuration files • application file systems • databases • LDAP directory • system restoration tools

• A partial backup includes the same items as the full backup, except that from the system image only the configuration files and variable data are included.

• A custom backup allows user to choose the backup items to include in the backup.

During the backup procedure, the local image is not backed up, but regenerated from the system image when it is restored. A backup module synchronises the local variable data and configuration files from the local image to the system image before a backup is made.

The backup modules first create individual backup files of the application file systems, databases, LDAP directory, and system restoration tools. The individual backup files are stored in temporary subdirectories in the directory

/var/mnt/local/backup/SS_Backup/tar

Finally, the system image module makes the system image backup and combines it with the individual backup files into a single compressed tar file that is located in

/var/mnt/local/backup/SS_Backup

The backup scripts and the configuration files are located in the directory

/etc/opt/Nokia_BP/fsbackup



Id:0900d80580621b36

Backup and restore

Backup logsA backup operation creates entries into the system log, a cumulative backup log, and a backup-specific log.

The system log (syslog) contains entries of when the backup started and by whom, if it was interrupted and the status of the backup (succeeded or failed.) It is located in the directory /var/log.

The following are examples of syslog entries.

Example: The backup process started:

Mar 1 13:40:06 info CLA-0 logger: SYSLOG(fsbackup) \Starting backup procedure

Example: The user interrupted the backup process:

Mar 1 13:50:39 info CLA-0 logger: SIGINT(fsbackup:_trapSigInt) \ User interrupt EXITCODE 2

Example: The backup process failed:

Mar 1 13:50:39 info CLA-0 logger: SYSLOG(fsbackup:_trapSigExit) \Backup failed with ERRORS (FULL_R_FSPR4CHA_1.18-base-1_20061204_092831.log)

Example: The backup process succeeded:

Mar 1 13:50:39 info CLA-0 logger: SYSLOG(fsbackup:_trapSigExit) \ finished OK! (FULL_R_FSPR4CHA_1.18-base-1_20061204_092831.log)

The cumulative backup log, backup.log, contains log entries for all backups and restore operations. It is located in the directory

/var/mnt/local/backup/SS_Backup/log

In the cumulative backup log, a log entry for a backup can consist of several rows. The rows are numbered, and for each backup the numbering starts from 1.

The cumulative backup log entries contain the following information:

• the name of the backup log, including the start time of the backup • the end time of the backup • the status of the backup (succeeded or failed) • the command used to start the backup including the options • the user name of the user who started the backup • the size of the backup archive file • md5 checksum of the backup archive file.

Each backup also has a backup-specific log. It contains information on everything that occurred during the backup. If the backup failed, user can look for errors in the backup-specific log.


43

Administering RNC OMS Backup and restore

Id:0900d80580621b36

The name of the backup-specific log file includes the type of the backup (FULL, PARTIAL, or CUSTOM), the name of the delivery, and the backup execution date and time.

The backup-specific log file is stored in the backup directory /var/mnt/local/backup/SS_Backup/log.

Restore operation typesA partial restore may be necessary, for example, if the database(s) or LDAP directory is faulty. For partial restoring to be possible, the system must be running.

The shell script fsrestore is used for restore operations. The script offers the following restore possibilities:

• restoring an application file system • restoring all database backups • restoring one specified database backup • restoring LDAP directory backup • restoring a single file or a subdirectory

Note that all items are not necessarily used in all deployments

A full restore is required when the system cannot be repaired using a partial backup and user needs to restore the whole system from the backup archive. During a full restore, user restores all backup items: application file systems, databases, LDAP directory, and the system image.

A broken system image usually causes a boot loop. This may be because of a logical disk crash, a corrupted file system or an accidental deletion of files or directories on the system image. The system must be booted using an external USB boot device, for example a USB flash memory drive.

Backup scheduling and storingIt is recommended to always make a full backup before and after a software upgrade. It is recommended to make a partial backup every night. Backups can be scheduled using cron (a daemon used for executing scheduled commands). For instructions, see Sched-uling backups in OMS.

It is important to note that the backup archive, in both full and partial backup, takes up a large amount of disk space. If user makes the backup frequently, the disk will soon become full. To avoid this, user needs to delete old backup files in order to release disk space for new ones.

It is also recommended that user keeps a backup history for a certain period of time by always moving the existing backup file to an external server before executing a new backup. It is recommended to keep at least the full backup and the latest partial backup, but it is also recommended to keep a longer history of partial backups. For example, if a database becomes corrupted and this is not noticed immediately, the latest partial backup may also contain the corrupted data. In such a case, it is necessary to search through older partial (or full) backups to find the latest database backup that is not cor-rupted.

Obtaining additional information from backup and restore servicesUser can find the following options useful during troubleshooting:

• -v or --verbose displays on the screen the information that is written to the log. • --debug shows more information when user runs the program.



Id:0900d805804a01fc

2.2 Preparing for the backup of databases in OMSPurposeTo configure the OMS to prepare the backup of the database other than AidS.

SummaryDatabase AidS is the only dabase which is included to backup. Databases MySQL_DB_Alarm, MySQL_DB_CosNaming and MySQL_DB_PMData are not included to backup by default.

Steps

1 Open Application Launcher

• in the Start menu (Programs > Nokia Siemens Networks > Application Launcher Client 2 > Application Launcher Client) in Windowsor

• in the menu (Applications > Other > Application Launcher Client) in Linux.

2 Launch Parameter Tool

3 Enable the backup of a databse

• to activate the backup of a certain database edit the parameter fsClusterId=ClusterRoot, fsFragmentId=DB, fsdbName=DB_<DB Name>

• change the value of fsdbBackup: N > fsdbBackup: Y

For more information on using Parameter Tool see chapter Parameter Administration in Using Element Manager in RNC OMS.

Expected outcomeThe database is included to backups. The backup procedure can be started.

g Changing the value of fsdbBackup to Y causes that OMS backup partition is becoming full. Remember to enter the value back to N after needed backup is taken.


45


Id:0900d805802b574c

2.3 Making a full software backup in OMSPurposeMake a full backup of the whole system including the system image, variable data and configuration files, application file systems, databases, LDAP Directory, and the system restoration tools.

Before you startCheck that:

• the OMS is up and running in its normal working state. For instructions on how to check the state, see Checking and changing the state of a managed object in OMS.

• you have appropriate root privileges. • there is enough free disk space for the backup archive file. You can estimate the

amount of disk space needed from the previous backup archive files. The disk space you need during a full backup is at least two times the size of the old backup archive file.If necessary, free disk space by transferring backups to an external server and deleting unnecessary files. For instructions, see Transferring the backup archive file from OMS to an external storage server.

SummaryUse the fsbackup command to create a full backup of the system.


fsbackup -f [-q | -v]

The argument -f starts the full backup.


• -q: runs the command in the quiet mode, minimises output printing, and no progress indicator is shown.

• -v: displays on the screen the information that is written onto the log.

During the backup procedure, the local image is not backed up, but regenerated from the system when it is restored. The system synchronises the variable data and configu-ration files of all software sets from the local images to the system image before a backup is made.

First the backup modules create individual tar files of the application file systems, data-bases, LDAP Directory, and system restoration tools. The individual tar files are stored in temporary subdirectories in the directory


Finally, the system image module makes the system image backup and combines it with the individual tar files into a single archive that is located in


The name of the backup tar archive file includes the type of the backup (FULL), the name of the delivery directory, and the backup execution date and time, for example, FULL_R_OMS1_3.34-base-1_20061204_092831.tar.gz..



Id:0900d805802b574c

Steps


2 Start the full backup procedure.Enter the following command:

fsbackup -f

Expected outcomeA compressed backup archive is created in the subdirectory


Unexpected outcomeA backup archive file may exist but it is incomplete.

The system raises the alarm 70064 BACKUP ERROR.

In an error situation, information of the error is entered in the syslog and in the backup log.

The syslog entries can be seen from the syslog file located in /var/log. To check the error entries from syslog, enter:

tail -n 200 /var/log/syslog | grep -i backup

Also check the backup log for errors and warnings.

VerificationTo verify that the operation was successful:

• check that there are no backup error messages in the syslog file • check the existence of the backup archive by entering the following command:

ls /var/mnt/local/backup/SS_Backup


47


Id:0900d805804da32b

2.4 Making a partial software backup in OMSPurposeMake a partial backup to create a backup archive of the application file systems, data-bases, LDAP Directory, the system restoration tools, and a backup of the configuration files and variable data of the system image.


• the OMS is up and running in its normal working states. For instructions on how to check the state, see Checking and changing the state of a managed object in OMS.

• you have appropriate root privileges. • there is enough free disk space for the backup archive file. If necessary, free disk

space by transferring old backups to an external server and deleting unnecessary files. For instructions, see Transferring the backup archive file from OMS to an external storage server.

SummaryUse the fsbackup command to create a partial backup of the system.


fsbackup -p [-q | -v]

The argument -p starts the partial backup.



• -v: displays on the screen the information that is written to the log.

During the backup procedure, the local image is not backed up, but regenerated from the system when it is restored. The system synchronises the variable data and configu-ration files of the active software set to the system image before a backup is made.

First the backup modules first create individual tar files of the application file systems, databases, LDAP Directory, and system restoration tools.

The individual tar files are stored in temporary subdirectories in the directory


Finally, the system image module makes the system image backup and combines it with the individual tar files into a single tar file that is located in


The name of the tar file includes the type of the backup (PARTIAL), the name of the base delivery, and backup execution date and time, for example, PARTIAL_R_OMS1_3.34-base-1_20061204_092831.tar.gz.



Id:0900d805804da32b

Steps


2 Start the partial backup procedure.Enter the following command:

fsbackup -p

Expected outcomeThe backup archive is created in the subdirectory









• check that there are no backup error messages in the syslog file. • check the existence of the backup archive by entering the following command:



49


Id:0900d8058049131f

2.5 Making a custom software backup in OMSPurposeMake a custom software backup to create a backup archive of the backup items you have selected.


• OMS is up and running in its normal working state. For instructions on how to check the state, see Checking and changing the state of a managed object in OMS.

• you have appropriate root privileges • there is enough free disk space for the backup archive file. If necessary, free disk

space by transferring backups to an external server and deleting unnecessary files. For instructions, see Transferring the backup archive file from OMS to an external storage server

SummaryUse the fsbackup command to create a custom backup of the system.


fsbackup [-d [database name]] [-F [file system name]]

[-l] [-r] [-s | -S] [-q | -v]

You have the following options:

• -d [database name]: make a backup of the databases. • use the option -d alone to make a backup of all the databases. • use the option -d [database name] to make a backup of a specified data-

base. • -F [file system name].

• use the option -F alone to make a backup of all the application file systems. • use the option -F [file system name] to make a backup of a specified

application file system. • -l: make a backup of the LDAP Directory. • -r: make a backup of the restore tools. • -s: make a backup of the configuration files and variable data. During the backup

procedure the system synchronises variable data and the configuration files from the active software set from the local image to the system image.

• -S: make a backup of the whole system image. During the backup procedure the system synchronises variable data and the configuration files from all the software sets on the local image to the system image.



• -v: displays on the screen the information that is written onto the log.

During the backup procedure, the local image is not backed up, but regenerated from the system when it is restored. The system synchronises the local variable data and con-figuration files to the system image data before a backup is made.



Id:0900d8058049131f

The name of the backup tar archive file includes the type of the backup (CUSTOM), the name of the base delivery, and the backup execution date and time, for example,

CUSTOM_R_OMS1_4.29-base-1_20061204_092831.tar.gz

The backup archive is located in the subdirectory


Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase, and then change the root user permission by entering su - command.For more information, see Logging in to OMS in Installing and commissioning RNC OMS.

2 Start the custom backup procedure.Enter:

fsbackup [ options ]

Expected outcomeA backup tar file, consisting of the selected items, is created in the subdirectory









• check that there are no backup error messages in the syslog file • check the existence of the backup archive by entering the following command:



51


Id:0900d8058027ac38

2.6 Transferring the backup archive file from OMS to an external storage serverPurposeTransfer the backup archive file from the local hard disk to an external storage server as an additional safety precaution. In addition to that, transfer the backup archive file to free local hard disk space and keep a history of the different backup versions.


• backup tar archive file is available in the backup subdirectory • connection to the external server can be established.

SummaryBecause the backup is made locally, the backup archive file can be transferred to an external storage server using, for example, scp.

Steps

1 Transfer the backup file to the desired location.Enter the following command:

scp /var/mnt/local/backup/SS_Backup/<backup archive file> _nokfsoperator@<Ip address>:/home/_nokfsoperator

Expected outcomeThe backup archive file is copied to the desired storage server.

VerificationThe success of the file transfer can be verified by, for example md5 program.

The checksums can be computed and checked by the md5sum command.



Id:0900d8058061c449

2.7 Creating a self-contained USB rescue stick.PurposeCreate a bootable USB stick containing all necessary rescue tools and a backup archive.

Steps

1 Create a bootable USB stick.For instructions, see Creating a bootable USB stick in Installing and commisioning RNC OMS.

2 Perform a full software backup.For instructions, see Making a full software backup in OMS in Administering RNC OMS.

Expected outcomeA backup archive file FULL_R_OMS1_xxx_xxx.tar.gz is created in /var/mnt/local/backup/SS_Backup directory.

3 Generate MD5 checksum for the backup archive.cd /var/mnt/local/backup/SS_Backup

md5sum FULL_R_OMS1_xxx_xxx.tar.gz > FULL_R_OMS1_xxx_xxx.tar.gz.md5sum

4 Extract restoretools.tgz file from the backup archive.cd /var/mnt/local/backup/SS_Backuptar xvzf FULL_R_OMS1_xxx_xxx.tar.gz \ var/mnt/local/backup/SS_Backup/tar/bu_restoretools.sh/restoretools.tgz

5 Copy files to the USB stick.Connect the USB stick to an external computer.

Copy the following files to the /install directory on the USB stick:

– InstallOMS_USB.sh (to be downloaded from NOLS)– rescuetools.tgz (extracted in the previous step)– FULL_R_OMS1_xxx_xxx.tar.gz (the backup archive)– FULL_R_OMS1_xxx_xxx.tar.gz.md5sum (the MD5 checksum)

For instructions on copying files to an external computer, see Transferring files from OMS to client computer in Installing and commisioning RNC OMS.

g Make sure that the /install directory does not contain any other files.

Perform this procedure every time you make a backup. If no incremental software deliv-eris were installed since the last backup step 4 can be skipped and you can use the existing restoretools.tgz file.


53


Id:0900d8058060e78a

2.8 Restoring the whole system in OMS using a backup serverPurposeRestore the whole system if it cannot be restored from a partial backup.

Before you startExecute the following:

• locate from the backup server the latest full backup archive and the partial backup archives that have been taken after a full backup

• establish a physical connection to the backup server • connect an external monitor and keyboard to the OMS (O&M server)

SummaryA full restore is required when the system cannot be manually repaired from a partial backup. This may be due to a logical disk crash, a corrupted file system, or an accidental deletion of files and directories on the system image.

Broken system image usually causes the node to go into a boot loop. If the boot partition is broken, the boot procedure hangs already in the BIOS state. In this module it is assumed by default that the whole disk structure is broken, including the application file systems, partition structure, and boot block.

Steps

1 Boot up the OMS node.To boot up the OMS node, execute the following steps.

a) Connect the external USB boot device, for example an USB flash memory drive, into the OMS node's USB port. You can find the instructions how to setup the USB flash memory drive in the readme.txt -file that is included in the FlexiPlatform bootpacket archive delivered via Nokia Online Services.

b) Reset the OMS node by pressing the reset button in the front panel of the blade.c) Press DEL button of the external keyboard to enter BIOS settings.d) Change the boot order by moving the removable devices (USB stick) to highest on

the list that makes it the primary boot device. Move the hard drive to the second on the list to make it the secondary boot device. Make sure that the USB stick, for example Kingston DT Elite HS 2.0 – (USB), is the first selection within the secondary boot device.

e) Save the BIOS settings.f) Use the keyboard to choose the FP4_rescue_MCP18 for the integrated OMS or

FP4_rescue_HPProLiant for the standalone OMS from the list. Wait for the OMS node to boot from the external USB boot device.

g) Log into the preboot system as root user (no password needed).h) Configure the Ethernet interfaces of the OMS. Enter the following commands:

ifconfig eth# <ip address> netmask <netmask>route add default gw <gw address>where eth# is the Ethernet interface. The interface eth0 must be configured to the same network with the backup server. The interface eth1 must be configured to the node where the full restore is done.



Id:0900d8058060e78a

Example: ifconfig eth0 192.168.0.8 netmask 255.255.0.0ifconfig eth1 10.102.90.151 netmask 255.255.255.0route add default gw 10.102.90.1

Change the password of the root user account by entering the passwd command.

2 In the external backup server, extract the restore tools.

a) Extract the restore tools restoretools.tgz tar archive file from the backup tar archive file. Enter the following command:

tar xvzf <archive> var/mnt/local/backup/\SS_Backup/tar/bu_restoretools.sh/restoretools.tgz

The restore tools archive file contain the following items: • restore script fsrestore.sh • partition restore and mount script usb_prepare • boot partition population script updatepreboot.sh • unmount script usbumount.sh • local image creation script mkLocalImages.sh • post-restore script postRestore.sh

b) Transfer the restoretools.tgz file to the OMS with, for example, using scp.Enter the following command:

scp /var/mnt/local/backup/SS_Backup/tar/bu_restoretools.sh/restoretools.tgz \root@<ip address of the OMS>:/tmp

3 In the OMS, extract the restoretools.tgz archive file.

a) Change to the directory /tmp.Enter the following command:cd /tmp

b) Extract the restore tools.Enter the following command:tar xvzf restoretools.tgzThe directory structure containing all the tools for restoring the whole system is as follows:

/tmp/fsbackup/backup.d backup.c restore.d restore.c common.d tools

c) Change to the directory fsbackup/tools.Enter the following command:cd fsbackup/tools


55


Id:0900d8058060e78a

4 Prepare the devices for restore.Create the partitions devices for the system image, backup, and local image. Create the metadevices and file systems for system image, backup, and local images. Mount the metadevices for restore and synchronise the RAID devices.

Enter the following command:

./usb_prepare

5 Copy the backup archive files to the backup partition.Enter the following command:

scp <backup server>:<archive file> /var/mnt/local/backup/SS_Backup

After the transfer is completed, you can remove the physical connection to the backup server.

☞ You can use the md5sum command to verify the archive integrity on the network element.

6 Restore the system image and the LDAP directory.Enter the following command:

./fsrestore.sh -S -l <archive file>

Also check if you have newer partial and custom backups. If you do, restore the LDAP directory and/or system image from them.

• To restore the LDAP directory from a partial or custom backup, enter the following command:./fsrestore.sh -l <archive file>

• To restore system image from a partial or custom backup, enter the following command:./fsrestore.sh -s <archive file>

7 Create the local image for OMS node.Enter the following command:

./mkLocalImages.sh

8 Unmount and stop all metadevices.Enter the following command:

./usbumount.sh

9 Reboot the OMS node.

a) Enter the command sync.b) Reset the node by entering the command init 6.c) Press DEL button of the external keyboard to enter BIOS settings.



Id:0900d8058060e78a

d) Change the boot order back by moving the hard drive to highest on the list that makes it the primary boot device.

e) Save the BIOS settings.f) Wait for the OMS node to boot.

10 Unlock the node, the necessary recovery groups and recovery units.Enter the following commands:

cd /etc/opt/Nokia_BP/fsbackup/common.d

./postRestore.sh --stage1

11 Restore the databases and application file systems.Enter the following command:

fsrestore -d -F <archive file>

Also check if you have newer partial and custom backups. If you do, restore the data-bases and/or application file systems from them.

• To restore the databases from a partial or custom backup, enter the following command:fsrestore -d <archive file>

• To restore application file systems from a partial or custom backup, enter the follow-ing command:fsrestore -F <archive file>

12 Unlock the rest of the recovery groups and recovery units.Enter the following commands:

./postrestore.sh --stage2

Expected outcomeThe whole system is restored and is available to be used normally.

Unexpected outcomeFor instructions on how to proceed in error situations, refer to the following instructions:

• If the error concerns restoring databases, refer to Restoring of database in OMS fails.

• If the error concerns restoring LDAP directory, refer to Restoring LDAP directory of OMS fails.

• If the error concerns restoring the system image, a file, or a directory, refer to Restor-ing of system image, single file, or directory in OMS fails.


57


Id:0900d80580621b3f

2.9 Restoring the whole system in OMS with self made full backup archive and USB stick.PurposeTo recover the whole system with an bootable USB stick containing the rescue tools and self made full backup archive.

Before you startThe prerequisites for the OMS installation are as follows:

• Two hard drives are installed to the OMS. • The USB stick has been prepared according to the instructions given in the previous

sections. • The USB stick does not contain any files other than the USB boot part (syslinux

folder), one self made full backup (install folder). • Notice that on USB installation OMS BIOS must be in UTC/GMT time (United

Kingdoms time).

Note that log is written to USB stick /logs/Install--<timestamp>.log

Steps

1 Boot up the OMS.To boot up the OMS, execute the following in the order they are listed:

a) Connect the external USB boot stick into the OMS's USB port.b) Reset the OMS by pressing the reset button in the front panel of the blade.c) When prompted, press DEL button of the external keyboard to enter BIOS settings.d) During first boot of OMS in USB installation procedure, change HW clock (BIOS

clock) to UTC/GMT time (United Kingdoms time).e) Change the boot order by moving the "USB stick" (example JetFlash TS4GJF150-

(USB)) under Hard Drive to the top on the list so that it is the primary boot device.f) Save the BIOS settings (F10).g) To choose correct selection from the list, enter the following text:

fp4_mcp18 (for the integrated OMS)orfp4_hprm (for the standalone OMS)Wait for the OMS to boot from the external USB boot device.

h) Log into the preboot system as root user (no password needed).i) Verify time with date command after OMS first boot:

#date

#Wed Jan 30 13:36:31 EST 2008Time should be correct (UTC/GMT) and ignore the time zone information (here EST) in date printout.

2 In the OMS, mount the USB stick and copy the installation script.For the integrated OMS enter the following commands:



Id:0900d80580621b3f

# cd /tmp # mkdir -p /media/FPRESCUE # mount /dev/sdc1 /media/FPRESCUE # cp /media/FPRESCUE/install/InstallOMS_USB.sh /tmp/ # dos2unix InstallOMS_USB.sh # chmod 744 InstallOMS_USB.sh # cd /media/FPRESCUE/install/# dos2unix FULL_*.md5sum# cd /tmp

For the standalone OMS enter the following commands:

# cd /tmp # mkdir -p /media/FPRESCUE # mount /dev/sda1 /media/FPRESCUE # cp /media/FPRESCUE/install/InstallOMS_USB.sh /tmp/ # dos2unix InstallOMS_USB.sh # chmod 744 InstallOMS_USB.sh # cd /media/FPRESCUE/install/# dos2unix FULL_*.md5sum# cd /tmp

3 Start installation script.Start the installation script by running the following command:

# ./InstallOMS_USB.sh

After you have given the command, the following text appears on the screen:

This script will install OMS Running this script will take about 1 hour 20 minutes You should wait for text: Are you ready ? (Press ENTER) text Press ENTER to continue.

Now wait until the script finishes and asks to remove USB stick and boot OMS. Continue with next step.

☞ If the partition table configuration changes between installations (current vs. coming), the installation is likely to fail in partition creation and cause the following error message: "No such file /var/mnt/local/backup/SS_Backup...

If this happens, repeat step 3.

4 Remove the USB stick.Follow the instructions when the # ./InstallOMS_USB.sh has been completed:

Remove USB stick Rebooting, change BIOS boot order to boot from Hard Drive And save settings Are you ready? (Press ENTER)

Remove the USB stick first and then press ENTER to reboot the OMS.


59


Id:0900d80580621b3f

5 Reboot the OMS from the hard drive.To reboot the OMS from the hard drive, execute the following:

a) When prompted, press DEL button of the external keyboard to enter BIOS settings.b) Change the boot order back by moving the hard drive to highest on the list that

makes it the primary boot device. Check that the boot order is correct, hard drive is highest on the list and enabled (no exclamation mark in front of it). Make corrections if needed.

c) Save the BIOS settings.d) Wait for the OMS to boot.

6 Unlock the node, the necessary recovery groups and recovery units.Enter the following commands:

cd /etc/opt/Nokia_BP/fsbackup/common.d

./postRestore.sh --stage1

7 Restore the databases and application file systems.Enter the following command:

fsrestore -d -F <archive file>

Also check if you have newer partial and custom backups. If you do, restore the data-bases and/or application file systems from them.

• To restore the databases from a partial or custom backup, enter the following command:fsrestore -d <archive file>

• To restore application file systems from a partial or custom backup, enter the follow-ing command:fsrestore -F <archive file>

8 Unlock the rest of the recovery groups and recovery units.Enter the following commands:

./postrestore.sh --stage2

Expected outcomeThe whole system is restored and is available to be used normally.

Unexpected outcomeFor instructions on how to proceed in error situations, refer to the following instructions:

• If the error concerns restoring databases, refer to Restoring of database in OMS fails.

• If the error concerns restoring LDAP directory, refer to Restoring LDAP directory of OMS fails.

• If the error concerns restoring the system image, a file, or a directory, refer to Restor-ing of system image, single file, or directory in OMS fails.



Id:0900d805804bbd6d

2.10 Restoring databases in OMSPurposeRestore the required databases from the backup archive if the error situation concerns the databases only.


• you have appropriate root privileges. • the destination system directory is available. • the backup archive file to be restored is available.

Summary

Use the fsrestore command to restore a single database.


fsrestore -d [<database name>] [-v] <backup archive>

The argument -d starts the database restore operation.

You have the following arguments:

• <database name>: the name of the database you want to restore. • <backup archive>: the backup archive file (located in

/var/mnt/local/backup/SS_Backup).

In addition, you can use the option -v that displays also on the screen the information that is written to the log.

As a result the specified database backup is restored.

☞ It is also possible to restore all databases by omitting the name of the database. This is usually needed only during a full restore.

Steps


2 Lock the database recovery group.Enter the following command:

fshascli -X /<database RG>

!

Errors may occur if the state of managed objects, recovery groups, or recovery units, is locked or unlocked during restoration. Do not change the state of managed objects during restoration.


61


Id:0900d805804bbd6d

Example: To lock the Alarm database recovery group run the following command:

fshascli -X /AlarmDB

3 Restore the database from the backup archive file.Enter the following command:

fsrestore -d <database name> <backup archive>

Example: To restore the DB_Alarm from the backup archive file run the following command:

fsrestore -d DB_Alarm

PARTIAL_R_GOMS2_1.13.release_oms.corr4_20081027_145857.tar.gz

4 Unlock the database recovery group.Enter the following command:

fshascli -u /<database RG>

Expected outcomeThe specified databases are restored correctly to the destination location and are avail-able to be used normally. The database logs of a MySQL database are not deleted when a single MySQL database is restored.

Unexpected outcomeFor more instructions on how to proceed in an error situation, see Restoring of database in OMS fails.



Id:0900d8058045c606

2.11 Restoring LDAP directory of OMSPurposeRestore only an LDAP directory from the backup archive if the error situation concerns only the LDAP directory.


• you have the appropriate root privileges. • the backup archive file to be restored is available. • the LDAP server is up and running.

Summary

Use the fsrestore command to restore a LDAP directory.


fsrestore -l [-v] <backup archive>

The argument -l starts the LDAP directory restore operation.

You have the following argument:

• <backup archive>: the backup archive file(located in /var/mnt/backup/SS_Backup)


Steps


2 Restore the LDAP directory.Enter the following command:

fsrestore -l <backup archive>

Expected outcomeThe LDAP directory files are restored correctly to the destination location and are being used by running OpenLDAP server.

!



63


Id:0900d8058045c606

Unexpected outcomeFor instructions on how to proceed in an error situation, see Restoring of LDAP directory of OMS fails.

Example: Restoring LDAP directory from a full backupfsrestore -l FULL_R_OMS1_3.34-base-1_20061204_092831.tar.gz

Example: Restoring LDAP directory from a partial backupfsrestore -l PARTIAL_R_OMS1_3.34-base-1_20061204_092831.tar.gz

Example: Restoring LDAP directory from a custom backupfsrestore -l CUSTOM_R_OMS1_3.34-base-1_20061204_092831.tar.gz



Id:0900d805802914ef

2.12 Restoring a single file or directory in OMSPurposeIf an error situation concerns a directory or a file only, it is recommended that you only restore the required file or directory from the backup archive file.

When restoring a file or directory, check that it is safe to restore the file or directory to the system. It is not recommended to restore files or directories to the runtime environ-ment if you do not know exactly what you are doing.

Note that you cannot restore databases with this procedure.


• you have the appropriate root privileges. • the backup archive file to be restored is available.

Summary

Use the fsrestore command to restore a single file or directory.


fsrestore [-v] <backup archive> -s <file or directory name>

The argument -s starts the file or directory restore operation.

You have the following arguments:

• <file or directory name>: the name of the file or directory you want to restore. • <backup archive>: the backup archive file

(located in /var/mnt/backup/SS_Backup)


As a result the specified directory or file is restored into the destination location.

Steps


2 Lock the relevant recovery units.Check what recovery units (RU) you have to lock. You have to lock the RUs that use the file or directory you are restoring.


!



65


Id:0900d805802914ef

fshascli -l /<RU name>

3 Restore the file or directoryTo restore a directory or a file, enter the following command:

fsrestore <backup archive> -s <file or directory name>

4 Unlock the recovery units.To unlock the recovery units you locked in step 2, enter the following command to each RU:

fshascli -u /<RU or node name>

Expected outcomeThe specified directory or file is restored correctly to the destination location.

Unexpected outcomeAlarm

For instructions on how to proceed in an error situation, see Restoring of system image, single file or directory in OMS fails.

Example: Restoring the home directory of the user fsoperator from a full backup

fsrestore FULL_FLEXITEST_20060905_092831.tar.gz -s var/mnt/local/sysimg/\FlexiPlatform/home/fsoperator

Example: Restoring the directory /etc from a partial backupIn this example, you restore the directory /etc from the active software set on the OMS. The base delivery is named /R_OMS1_3.34-base-1 and the active software set is named R_OMS1_3.34-base-1.release.corrections10.

fsrestore PARTIAL_FLEXITEST_20061204_092831.tar.gz -s var/mnt/local/sysimg/\FlexiPlatform/sets/R_OMS1_3.34-base-1.release.corrections10/opt/\Nokia_BP/nodes/CLA-0/etc/



Id:0900d80580335476

2.13 Backup of OMS fails

DescriptionThe backup process may fail if, for example:

• the user does not have the required privileges to execute the backup. • the system is not fully functional. • there is not enough free disk space available for the backup archive.

The database backup process may also fail if, for example:

• there is an internal database system error. • there is a system timeout while making a backup of a database.

SymptomsWhen the backup process fails,

• the system creates the alarm 70064 BACKUP ERROR. • the backup process is either completed or interrupted. Even though the backup of

one backup item fails, the system continues making backups of the other backup items and creates a backup archive in the subdirec-tory/var/mnt/local/backup/SS_Backup

• the system displays an error message on the local output device and creates an entry in the syslog file.

Recovery proceduresRefer to the error message shown on the local output device and check the syslog file in /var/log to determine the cause for the error situation. Check the following points to solve the problem.

Recovering from a backup failure

Steps

1 Check the alarm and the log files.

a) Check the alarm with the appropriate alarm management tool.The alarm includes the name of the backup log.

b) Search for backup-related entries in the syslog.Enter the following command:grep -i backup /var/log/syslog

c) Search the backup log for strings ERROR or WARNING and check the status from the end of the log.

2 If the error message on the local output device is permission denied

Then

check that you have the required root privileges.

3 Check that there is enough free disk space for the backup.

• Check the amount of available disk space.Enter:

df -h /var/mnt/local/backup


67


Id:0900d80580335476

• If necessary, free disk space by transferring backup archive files from OMS to an external storage server and deleting unnecessary backup files.

4 If there is an internal database system error

Then

proceed as follows:

• check that the databases are up and running • refer to the database-specific documentation.

5 If the error message in the syslog is DBBackup: timeout while making a backup for database: <name of database>

Then

contact your local Nokia Siemens Networks representative.

6 If the above instructions solved the problem,

Then

refer to the instructions in Making a full software backup in OMS, Making a partial software backup in OMS or Making a custom software backup in OMS.Else




Id:0900d805802d7b44

2.14 Restoring of database in OMS fails

DescriptionRestoring of database may fail if, for example:

• the user does not have the required privileges to execute the restore command. • there is not enough free disk space available for the backup files.

Note that the restore operation takes more disk space than a backup archive file, because the backup archives are unzipped during the restore operation.

• the backup archive to be restored is faulty. • there is an internal database error.

SymptomsWhen restoring a database fails,

• the restore process is interrupted • the system displays an error message on the output device and creates an entry in

the syslog file.

Recovery proceduresRefer to the error message shown on the local output device and check the syslog file in /var/log directory to determine the cause for the error situation. Check the following points to solve the problem.

Recovering from a failure to restore database

Steps

1 Check the log files.

a) Search for restore-related entries in the syslog.Enter the following command:grep -i restore /var/log/syslog

b) Search the backup log for text strings ERROR or WARNING and check the status from the end of the log.

2 Execute the fsrestore command again with the options --debug and --verbose and examine the output.The options --debug and --verbose print to the screen the information that is also written to the log.

3 If the error message on the local output device is permission denied

Then

check that you have root privileges.

4 If the error message in the syslog is DBRestore: No such database

Then

check that you have entered the restore command correctly.

5 Check that there is enough free disk space available.You can check the amount of available disk space using the command


69


Id:0900d805802d7b44


If necessary, free disk space with the instructions in Transferring backup archive files from OMS to an external storage server and delete unnecessary backup files.

6 If the above instructions solved the problem,

Then

refer to the instructions in Restoring databases in OMSElse




Id:0900d8058049408c

2.15 Restoring of LDAP directory of OMS fails

DescriptionRestoring of LDAP directory may fail if, for example:

• the user does not have the required privileges to execute the restore command. • there is not enough free disk space available for the backup files.


• the backup archive to be restored is faulty. • the system is unable to write into destination files or replace existing files.

SymptomsWhen restoring the LDAP directory fails,


the syslog file.


Recovering from a failure to restore LDAP directory

Steps




2 Execute the fsrestore command again with the options --debug and --verbose and examine the output.The options --debug and --verbose display on the screen the information that is also written to the log.

3 Check that you have root privileges.

4 If the error message is invalid directory given as parameter

Then

check that the source and destination directories exist.

5 Check that there is enough free disk space available.You can check the amount of available disk space using the command



71


Id:0900d8058049408c

If necessary, free disk space by Transferring backup archive files from OMS to an external storage server and deleting unnecessary backup files.

6 Check that the LDAP server is up and running.Because OMS LDAP is running under RG /Directory you need to check if this Directory is enabled and active. Enter the following command:

fshascli -s /Directory

An example printout of the fshascli -s /Directory command:

Directory:administrative(UNLOCKED)operational(ENABLED)usage(ACTIVE)procedural()availability()unknown(FALSE)alarm()

Note that when you are restoring the whole system, you can skip this step as the LDAP server is not up and running.

Note that LDAP is a critical process of OMS and if it is not up and running OMS will reboot.

7 If the above instructions solved the problem

Then

refer to the instructions in Restoring LDAP directory in OMS.Else




Id:0900d805803e1d67

2.16 Restoring of system image, single file, or directory in OMS fails

DescriptionRestoring of the system image, a single file, or a directory may fail if, for example:

• the user does not have the required privileges to execute the restore command. • there is not enough free disk space available for restoring the backup.


• the backup archive to be restored is faulty.

SymptomsWhen restoring the system image, a single file, or a directory fails,


the syslog file


Recovering from a failure to restore system image, a file, or a directory

Steps




2 Execute the fsrestore command again with options --debug and --verbose and examine the output.The options --debug and --verbose print to the screen the information that is also written to the log.

3 If the error message states that there is no such file or directory

Then

check that the backup archive file is available in the directory /var/mnt/local/backup/SS_Backup.

4 If the error message is permission denied

Then

check that you have root privileges.

5 If the error message states that the disk is full


73


Id:0900d805803e1d67

Then

check that there is enough disk space available.You can check the amount of available disk space using the commands

df -h /var/mnt/local/backupdf -h /var/mnt/local/sysimg

If necessary, free disk space by transferring backup archive files from OMS to an external storage server and deleting unnecessary backup files.

6 If the above solved the problem,

Then

refer to the instructions in Restoring a single file or directory in OMS.Else




Id:0900d8058055b1a4

User management

3 User management

3.1 Management of user accountsThe term user security covers the management of user accounts, as well as the authen-tication and authorisation mechanisms used in the platform.

Administrators can connect to the services in one of the following ways:

• Graphical user interface (GUI) connections protected by SSL/TLS • SSH connections for command line interface (CLI) transactions or file transfer • Connections to the network management system (preferably) protected by IPSec.

The OMS offers centralised management of user accounts for all these types of connec-tions. With the exception of Linux system accounts and two fallback accounts _nokfsoperator and _nokfssysemoperatorfallback that are stored in the operating system, all user accounts (for humans) and service accounts (for process-to-process communication) are stored in the LDAP server. As a consequence, each user is assigned only a single username and password, no matter how the user is accessing the services or where the user account information is stored. Note also that shared user accounts are not permitted, since it is difficult to trace the actions if users are allowed to log in using shared accounts.

A high level of security is ensured by careful handling of root user access rights, by securing transmission lines (using SSL/TLS for GUI transactions and SSH for CLI trans-actions or file transfer), and by using virtual private network (VPN) technology based on IPSec wherever SSL/TLS or SSH solutions are not available. As an additional precau-tion, applications are designed in such a way that sensitive user credentials (such as passwords) do not appear as a part of the process listing (for example when using the Linux ps command). Most processes are started as non-root by using the subsystem-related service accounts. Processes are started as root only when needed.

All successful or failed login attempts are stored in log files, see Audit trail logging. Note also that the integrity of these log files is protected by file access permissions.

There exist a number of service accounts that are user accounts related to the services running on the platform. The processes are started with these service accounts that have permission only to the needed service(s). With the credential service feature sup-ported by the platform, it is also possible to give password protected permissions to other services, if needed. Note that the service accounts are created during commis-sioning and the account information, except the passwords, should not be changed.

Administration of user account attributesNumber of attributes are related to the user accounts. They define, for example, the requirements and expiration times for the password and the login information of the account. The attributes can be set for the user account when creating it by fsuseradd command, or later by fsusermod command. If no attributes are given as options for these commands the default values are used. The default values can be set in the login.defs file in the /etc directory and the useradd file in the /etc/default directory:

• /etc/login.defs file defines the following default values: • PASS_MAX_DAYS, PASS_MIN_DAYS, PASS_MIN_LEN and PASS_WARN_AGE:

Password controls.


75

Administering RNC OMS User management

Id:0900d8058055b1a4

• UID_MIN and UID_MAX: The range for automatic uid selection, for example, from 10 000 to 60 000.

• GID_MIN and GID_MAX: The range for automatic gid selection. Note that this option are not used by fsuseradd.

• USERDEL_CMD: The option for running a script or command when the account is removed. Note that this option is not used by fsuseradd

• CREATE_HOME: The option for create, or not to create, a home directory for users. If the value no is given the home directories are not created by default. If, by default, the home directories wanted to be created comment out the line with a hash (#) mark. Note that this option is not used by fsuseradd

• MAIL_DIR: The directory where the mail box of the user resides, for example /var/spool/mail. Note that this option are not used by fsuseradd.

• /etc/default/useradd file defines the following default values: • HOME: The location for the home directories, for example /home. The actual

home directory for the user account <user> will be concatenated to be the /home/<user>.

• INACTIVE: The number of days after a password expires until the account is permanently disabled. A value of 0 disables the account as soon as the password has expired. If the default value is not numeric or negative or the default value does not exist, then the default value is ignored. By default, in these cases, nothing is written to the local LDAP.

• EXPIRE: The user account expiration date. The user account will be disabled at 00:00:00 GMT on the date specified in the format YYYY-MM-DD (GMT) or in the number of days from UNIX EPOCH (1970-01-01 GMT). If the default value is not numeric or negative or the default value does not exist, then the default value is ignored. By default, in these cases, nothing is written to the local LDAP.

• SHELL: The default login shell, for example /bin/bash. • GROUP: The group ID attribute for the primary user group. Note that this option

are not used by fsuseradd. • SKEL: The start-up files for the login shell. The files will be copied to the user's

home directory when the account is created. For example, /etc/skel. Note that this option are not used by fsuseradd.

User home directoriesThe user accounts for human users can be created either with or without home directo-ries. The home directory is useful, for example, if the user wants to configure an auto-matic ssh login to the network element. By default, the location for the home directories in the operating system is /home/<user>. Physically the home directories are located in the local image of the node, that is the /var/mnt/local/localimg/flexiserver/home directory. Note that if the content of the home directory needs to be shared also to the other nodes in the cluster, the fsdistribute /usr/<home> command must be executed.

Credential serviceWith the credential service feature it is possible to enquire a password for a user account to access a certain service. The passwords are stored in the files in the /opt/Nokia_BP/etc/security/credentials/<service> directory. Each service has its own directory for credentials and each named credential has its own file named <credentialname.cred> containing the credential to a certain service. The



Id:0900d8058055b1a4

User management

read right to the file is given only to the user account and to the user group that are set as the owner of the credential. The write right to the file is given only to the user account that is set as the owner of the credential.

Example: The password for the service ldap and the user fsLDAPRoot is stored in the fsLDAPRoot.cred file in the /opt/Nokia_BP/etc/security/credentials/ldap directory. The password files for all user accounts that need access to the LDAP service are stored in the same directory. They can be checked by listing the content of the directory with ls command:

ls -l /opt/Nokia_BP/etc/security/credentials/ldap/

The printout is, for example, the following:

-rw-r----- 1 root _nokfssyscred_ldap_fssysldapdb \64 Dec 8 18:03 _nokfssysldapdb.cred-rw-r----- 1 root _nokfssyscred_ldap_fsysldapgui \64 Dec 8 18:04 _nokfssysldapguiauthorisation.cred-rw-r----- 1 root _nokfssyscred_ldap_fsysldapgui \64 Dec 8 18:04 _nokfssysldapguiauthorisation.cred-rw-r----- 1 root _nokfssyscred_ldap_fsldaproot \64 Dec 8 18:03 fsLDAPRoot.cred-rw------- 1 root root \64 Dec 8 18:03 fsLDAPRoot.cred.old

When the password is changed, the backup of the old password file is done by adding the extension .old to the end of the file name and the read/write rights are changed to the root user.

Centralised User Authentication and AuthorisationCentralised User Authentication and Authorisation (CUAA) is a concept specified and provided by NetAct so that permissions for management access to multiple network elements can be centrally managed within NetAct. Network elements are expected to use a centralised (external) LDAP server located within the network management system (NMS) domain. With CUAA feature, it is possible for users to log into the several network elements with the user accounts that are stored in the centralised LDAP server.


77


Id:0900d805803a68f4

3.2 User account storagesUser accounts can be stored to various storages according to their purpose. Due to the Pluggable Authentication Module (PAM) implemented in the platform, the user can log in with the user account stored in any of the supported storages.

Operating systemLinux system accounts related to the operating system (OS) are stored in the /etc/passwd file and they should be used for interactive sessions only in fallback sit-uations. In addition to system accounts, typically two fallback accounts are stored in the operating system. For external SSH sessions the fallback account _nokfsoperator and for Element Manager purposes the fallback account _nokfssysemoperatorfallback can be used. The system accounts must be always available.

Internal LDAP serverThe platform internal service-related user accounts are called service accounts. They are used, for example, for service starting purposes and they are stored into network element internal LDAP server. They are created by the system during commissioning and their settings (except passwords) must not be changed. Application-specific service accounts can also be stored into the network element internal LDAP server. Internal service accounts have high availability requirements. Note that all service accounts are not necessarily used in all deployments. For more information, see System accounts.

External LDAP serverIn addition to the network element internal LDAP server also an external LDAP server can be used, if the remote user information management (RUIM) feature is enabled.

If the centralised external LDAP server is located, for example, on the NetAct site, all user accounts of the Element Manager Client (EMC) applications can be stored into it. The content of the external LDAP database is replicated to the platform internal LDAP database, thus the user accounts in the two LDAP databases must have unique user identifiers (uid).

The database replication is done only from the external LDAP server to the internal LDAP server. In the internal LDAP server there is a cache fragment where the replicated data is stored. Within a certain time period, the content of the cache fragment is checked and, if necessary, updated from external LDAP server. If there exist user accounts that are not used for a long time, they will be removed from the cache fragment. In addition to scheduled replication, the replication is done for a certain user account, if such a user tries to log in, whose account information does not exist in the cache fragment. Note that the passwords and the user groups and permissions in the external LDAP server are not replicated. The replicated user accounts are added to the local user groups in the internal LDAP server. The addition is based on the original permissions that the user account have in the external LDAP server.



Id:0900d805802cf34a

3.3 Internal LDAP user managementUsers are authorised to establish sessions and to execute commands in the system depending on their authorisation. User accounts can further be divided into different user groups according to their operating tasks and expertise. It is possible for the same user account to be part of several user groups. Each user account or user group must have a user identifier (uid). Users must know their user identifiers and passwords to enter the system.

There are two alternative user interfaces to manage the user accounts and user groups stored into the network element internal LDAP server: command line interface (CLI) and graphical user interface (GUI). The GUI application for user management is Parameter Tool. The tasks listed below are possible for both user interfaces:

• creating new user accounts and user groups • assigning privileges to user accounts and user groups • modifying privileges of existing user accounts and user groups • deleting existing user accounts and user groups

PasswordsA password attached to the user account prevents users from accessing the system by using other than their own user ID. When logging in, the system asks for user name and password. After successful user identification, the user is entered to the system. The users can only do such operations, for which they have appropriate rights.

Users are allowed to change their own passwords by using the passwd command in the command line interface. Only a person with operating system (OS) administrator rights, that is, with root user account, is allowed to change other users' passwords. For more information, see Changing internal LDAP user account passwords.


79


Id:0900d8058054d72e

3.4 Command line interface for internal LDAP user manage-ment

3.4.1 Creating user accounts with CLI

PurposeCreate user accounts with command line interface to network element internal LDAP server. The accounts must be created in the system before the users can log in.

SummaryNew user accounts can be added to network element internal LDAP server with fsuseradd command.

For more information on the groups and permissions required for creating users, see Security in RNC OMS, Appendix.

Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase, and then change the root user permission by entering su - command.For more information, see Logging in to OMS in Installing and Commissioning RNC OMS.

2 Add a new user account.Add a new user account with fsuseradd <uid> command. The full syntax of the command with all options is the following.

fsuseradd [-c cn] [-u uidNumber] [-d homeDirectory] [-p userPassword] [-e expire_date] [-f inactive_time] [-s loginShell] [-G group[,...]] [-n mindays] [-x maxdays] [-w warndays] [--filter filter size] [--fast] [-v] [-h] [-t time_out] [-r retrial_count] -g gidNumber uid

The descriptions for the various attributes are presented in the table below. For more information, see the manual page of the command by executing the man fsuseradd command.

The mandatory attributes are the user name uid and the number of the primary user group gidNumber. It is recommended to use the user private group principle. That is, each primary user group contain only one user account.



Id:0900d8058054d72e

Example: Create a user account with default attribute valuesThe mandatory attributes are the user name uid and the number of the primary user group gidNumber. If the optional attributes are not given as options for the command the default values will be used. The default values are defined in the login.defs file in the /etc directory and the useradd file in the /etc/default directory.

-c Common name attribute in LDAP. Default value is empty.

-u User identifier number attribute in LDAP. It must be a unique number that is not reserved by any other user account. Values between 0 and 499 are reserved for Linux distribution. Values between 500 and 4999 are reserved for platform. Values between 5000 and 10000 are reserved for applications. Values between 10001 and 65000 should be used for end users. Default value is the first free number within allowed limits.

-d User’s Linux home directory. Default value is /home/<uid>.

-p The encrypted password attribute.

-e The user account expiration date.

-f The number of days after a password expires until the account is permanently disabled. A value 0 disables the account as soon as the password has expired.

-s Default login shell attribute. Default value is /bin/bash.

-G List of secondary groups. The groups must exist in the LDAP server. Default value is empty.

-n The number of days from the password change before which the password cannot be changed.

-x The number of days from the password change after which the password must be changed.

-w The number of days before which the user is notified when the password expires.

--filter The initial number of the uidNumbers in the LDAP search filter for the first free uidNumber search. This option is for optimising the speed of the search. The most optimal size is usually about the number of the users in the LDAP. The default value is 128. The value must be between 2 and 2048.

--fast The external LDAP server is not polled.

-v Verification from the user is requested before fsuseradd oper-ation is completed.

-h Prints help text.

-t Time in milliseconds to wait between retrials. The default value is 400 ms.

-r Number of times to retry the operation if the entry is not added to the LDAP. The default value is 30.

-g Primary group attribute. The groups must exist in the LDAP server. Attribute is mandatory.

uid User’s login name attribute. Attribute is mandatory.

Table 1 Attributes for fsuseradd tool


81


Id:0900d8058054d72e

To create a user account testuser and assign the user group 10001 as the primary group, enter the following command:

fsuseradd -g 10001 testuser

After a successful operation the following message will be printed on the screen:

Command executed successfully.

Example: Create a user account and define the attribute valuesCreate a user account testuser and define the following attributes:

• Assign the user group 10001 as the primary user group with -g option. • Set the home directory to /usr/home/testuser with -d option. • Set the login shell to /bin/bash with -s option. • Set the expiration date to 31.12.2008 with -e option. • Set the inactive time to 14 days with -f option. • Set the time when the password must be changed to 60 days with -x option. • Set the time to notify the user about the password change to 14 days with -w option.


fsuseradd -d /usr/home/testuser -s /bin/bash -e 2008-12-31 -f 14 \-x 60 -w 14 -g 10001 testuser

After a successful operation the following message will be printed on the screen:

Command executed successfully.



Id:0900d805803fe5d8

3.4.2 Modifying user accounts with CLI

PurposeAdd a new user to LDAP security fragment.

SummaryNew users can be added to LDAP security fragment with fsusermod command. It modifies the user account to LDAP security fragment, fsClus- terId=Cluster-Root,fsFragmentId=Security,ou=People, to reflect the changes that are specified on the command line.

Note that fsusermod is allowed only for root user.


Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator user and password assigned during installation phase, and then change the root user permission by entering su - command.For more information, see Logging in to OMS in Installing and Commissioning RNC OMS.

2 Add a new user to LDAP security fragment.Add a new user to LDAP security fragment with fsusermod <uid> command. The full syntax of the command with all options is the following.

[-c cn] [-u uidNumber] [-d homeDirectory] [-e expire_date] [-f inactive_time] [-s loginShell] [-G group[,...]] [-n mindays] [-x maxdays] [-w warndays] [-v] [-h] [-g gidNumber] uid

The various attributes are presented in the table below.

-c cn

The new common name attribute in LDAP. This field is used for the full name of the user, for example, "Jon Doe".

-u uidNumber

The new UID number attribute in LDAP. Must be a unique number not reserved by any other user account. Default is the first free number greater than UID_MIN and less than UID_MAX. Values between 0 and 499 are reserved for Linux dis-tribution. Values between 500 and 4999 are reserved for FlexiServer platform. Values between 5000 and 10000 are reserved for FlexiServer applications. Values between 10001 and 65000 should be used for end users.

Table 2 Attributes for fsusermod tool


83


Id:0900d805803fe5d8

-d homeDirectory

The new user's home directory attribute in LDAP.

-f inactive_days

The new value for the number of days after a password expires until the account is permanently disabled. A value of 0 disables the account as soon as the password has expired.

-e expire_date

The new user account expiration date. The user account will be disabled at 00:00:00 GMT on the date specified in the format YYYY-MM-DD (GMT) or in the number of days from UNIX EPOCH (1970-01-01 GMT).

-s loginShell

The new default login shell attribute in LDAP.

-G group[,...]

The new list of secondary groups in LDAP. Default is empty. The groups must exist in the system. If a group does not exist in LDAP, a warning is printed. The user is added only to groups in LDAP (no update to /etc/group). The mem-berships to old supplementary groups will be removed.

-n mindays

The new value for the number of days from the password change before which the password cannot be changed.

-x maxdays

The new value for the number of days from the password change after which the password must be changed.

-w warndays

The new value for the number of days how much before the user is notified when the password expires.

-v Verification from user is asked.

-h Help. Print short help.

-g gidNumber

The new primary group attribute in LDAP. Can be either number or name of the group. The group must exist in system. If the group does not exist in LDAP, a warning is printed. The user is added only to group in LDAP (no update to /etc/group).

uid User's login name attribute in LDAP. This attribute cannot be changed.

Table 2 Attributes for fsusermod tool (Cont.)



Id:0900d805804a0444

3.4.3 Deleting user accounts with CLI

PurposeDelete user accounts with a command line interface from the network element internal LDAP server.

SummaryThe user accounts can be deleted from the network element internal LDAP server with fsuserdel command.

Steps


2 Delete a user account.Delete a user account with fsuserdel <uid> command. The full syntax of the command with all options is the following:

fsuserdel [-v] [-h] uid


-v Verification from user is asked before fsuserdel operation is com-pleted.



Table 3 Attributes for fsuserdel tool


85


Id:0900d805804afd41

3.4.4 Creating user groups with CLI

PurposeCreate user groups with command line interface to network element internal LDAP server. User accounts that belong to the same group have the same access rights.

SummaryNew user groups can be added to network element internal LDAP server with fsgroupadd command.

Steps


2 Add a new user group.Add a new user group with fsgroupadd <group> command. The full syntax of the command with all the options is the following:

fsgroupadd [-g gidNumber] [-f filter size] [--fast] [-v] [-h] [-t time out] [-r retrial count] group


-g Group number attribute. It must be a unique number that is not reserved by any other group account. Values between 0 and 499 are reserved for Linux distribution. Values between 500 and 4999 are reserved for platform. Values between 5000 and 10000 are reserved for applications. Values between 10001 and 65000 should be used for end users. Default value is the first free number within allowed limits.

-f The number of the gidNumbers in the LDAP search filter for the first free gidNumber search. This option is for optimising the speed of the search. The most optimal size is usually about the number of the groups in the LDAP. The default value is 128. The value must be between 2 and 2048.

--fast The external LDAP server is not polled.

-v Verification from the user is requested before fsgroupadd operation is completed.


-t Time to wait between retrials in milliseconds. The default value is 400 ms.

-r Number of times to retry the operation if the entry is not added to the LDAP. The default value is 30.

group Name of the group. Attribute is mandatory.

Table 4 Attributes for fsgroupadd tool



Id:0900d805804f04b5

3.4.5 Deleting user groups with CLI

PurposeDelete user groups with command line interface from the network element internal LDAP server.

SummaryThe user groups can be deleted from the network element internal LDAP server with fspgroupdel command.

Steps


2 Check that no user account belongs to the group.

a) Find the gidNumber of the group. To do this, dump the group related data to the screen and find the gidNumber attribute of the group. Enter the following command:fsdumpgroups -g groupwhere group is the name of the group.

b) Dump the user account related data to the screen and check that the gidNumber attribute got in the previous step is not attached to any user account. That is, the group you are going to delete is not a primary group of any user.To dump the user account related data to the screen, enter the following command:fsdumpusers

3 If a gidNumber attribute is attached to any user account

Then

Change the gidNumber attribute of the user account to point to another group.

4 Delete a user group.Delete a user group with fsgroupdel <group> command. The full syntax of the command with all options is the following.

fsgroupdel [-v] [-h] group


-v Verification from the user is requested before fsgroupdel operation is completed.



Table 5 Attributes for fsgroupdel tool


87


Id:0900d805804708d3

3.4.6 Administering user-group mappings with CLI

PurposeAdd or remove user accounts to or from certain user groups with command line interface from the network element internal LDAP server. Note that when you remove a user account from a user group, the user account information is not deleted from the data-base.

SummaryUser-group mappings can be administered in the network element internal LDAP server with fsgpasswd command.


Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase, and then change to the root user permission by entering su - command.For more information, see Logging in to OMS in Installing and Commissioning RNC OMS.

2 If you want to add a new user account to the group

Then

Use fsgpasswd -a <uid> <group> command.Add a user account to the user group with fsgpasswd -a <uid> <group> command. The full syntax of the command with all options is the following:

fsgpasswd [-a uid group] [-v] [-h]


3 If You want to remove a user account from the group

Then

Use fsgpasswd –d <uid> <group> command.

a) Remove a user account from the user group with fsgpasswd –d <uid> <group> command. The full syntax of the command with all options is the following.fsgpasswd [-d uid group] [-v] [-h]



-v Verification from user is asked before fsgpasswd operation is completed.


Table 6 Attributes for fsgpasswd tool



Id:0900d805804708d3

b) Check that the removed group was not the user’s primary group.After removing a user account from a user group, check that the user account’s gidNumber attribute does not point to the group from where the account was removed. If it does, change the gidNumber attribute to point to another group.


89


Id:0900d8058026929c

3.4.7 Administering group-permission mappings with CLI

PurposeAdd or remove permission to or from certain user groups with a command line interface from the network element internal LDAP server.

SummaryThe permissions can be created, deleted and modified in network element internal LDAP server with fspermadd and fspermdel commands. The group-permission mappings can be administered with the fsgroupperm command.

Steps


2 If you want to add a new permission to the LDAP server

Then

Use fspermadd <permId> command.Add a new permission to the LDAP server and assign it to the certain user groups with fspermadd <permId> command. The full syntax of the command with all the options is the following:

fspermadd [-c cliCommand[,...]] [-o objectClass[,...]] \ [-g group[,...]] [-m corbaMethod[,...]] [-d description] [-v] [-h] permId


-c List of CLI commands that are allowed to execute by this permission. The full path names must be used, for example, /opt/Nokia_BP/sbin/fshascli --view. Default value is empty.

-o List of object classes that can be accessed by this permission. Default value is FSSecPermission. For CLI programs, FSSecPermission and FSSecCLIPermission are used. For CORBA permissions, FSSecPermission and FSSecCORBAMethodPermission are used. For java applications, FSSecPermission and FSSecJ2EEPermission are used.

-g List of groups to be assigned to this permission. Note that group existence is not checked. Default value is empty.

-m List of allowed CORBA methods that can be accessed by this permission. Wild-cards can be used to specify that all methods of an interface are allowed, for example, moduleName.interfaceName.*

-d Description of the permission.

-v Verification from the user is requested before fspermadd operation is com-pleted.

Table 7 Attributes for fspermadd tool



Id:0900d8058026929c

3 If you want to remove a permission from the LDAP server

Then

Use fspermdel <permId> command.Remove a permission from the LDAP server with fspermdel <permId> command. The full syntax of the command with all options is the following:

fspermdel [-v] [-h] permId


4 If you want to assign a new group to a permission

Then

Use fsgroupperm -a <group> <permission> command.Assign a new group to an existing permission in LDAP server with fsgroupperm -a <group> <permission> command. The full syntax of the command with all options is the following:

fsgroupperm -a group permId [-v] [-h]


If the group is mapped to the CLI permission, the changed information must be added to the /etc/sudoers file by executing the fsperm2sudo.sh command. Note that the command only outputs the information to the standard output and it must be written to the file by the user. In addition, the new file must be distributed to the permanent config-uration files and all nodes in the cluster by executing the fsdistribute /etc/sudoers command.


permId Name of the permission. For CLI permission, use only capital letters, for example, FSHASVIEW. Attribute is mandatory.

Table 7 Attributes for fspermadd tool (Cont.)

-v Verification from the user is requested before fspermdel operation is com-pleted.


permId Name of the permission. Attribute is mandatory.

Table 8 Attributes for fspermdel tool



-v Verification from user is requested before fsgroupperm operation is com-pleted.


Table 9 Attributes for fsgroupperm -a command


91


Id:0900d8058026929c

5 If you want to remove a group from a permission

Then

Use fsgroupperm -d <group> <permission> command.Remove a group from a permission in LDAP server with fsgroupperm -d <group> <permission> command. The full syntax of the command with all options is the follow-ing.

fsgroupperm -d group permId [-v] [-h]


If the group is mapped to the CLI permission, the changed information must be added to the /etc/sudoers file by executing the fsperm2sudo.sh command. Note that the command only outputs the information to the standard output and it must be written to the file by the user. In addition, the new file must be distributed to the permanent config-uration files and all nodes in the cluster by executing the fsdistribute /etc/sudoers command.



-v Verification from user is asked before fsgroupperm operation is completed.


Table 10 Attributes for fsgroupperm -d command



Id:0900d805803f3771

3.4.8 Dumping service account data with CLI

PurposeDump the group, permission or user account related data from network element internal LDAP server to the screen.

SummaryGroup related data can be dumped to the screen with fsdumpgroups command. Per-mission related data can be dumped to the screen with fsdumpperm command. User account related data can be dumped to the screen with fsdumpusers command.

Steps


2 Dump the group related data to the screen.Group related data can be dumped to the screen with fsdumpgroups command. The full syntax of the command with all options is the following:

fsdumpgroups [-h] [-G] [-g gid[,…]]


3 Dump the permission related data to the screen.Permission related data can be dumped to the screen with fsdumpperm command. The full syntax of the command with all options is the following.

fsdumpperm [-h] [-p permId[,…]]


4 Dump the user account related data to the screen.User account related data can be dumped to the screen with fsdumpusers command. The full syntax of the command with all options is the following.


-G Dumps also group data from /etc/group file.

-g Dumps the selected groups listed either according to the names or the numbers.

Table 11 fsdumpgroups tool


-p Dumps the selected permissions listed according to their names.

Table 12 fsdumpperm tool


93


Id:0900d805803f3771

fsdumpusers [-h] [-g] [-p] [-u uid[,...]]



-g Dumps also the secondary groups assigned to the user accounts.

-p Dumps also account data from /etc/passwd file.

-u Dumps the selected user accounts listed either according to the names or the numbers.

Table 13 fsdumpusers tool



Id:0900d805804bb785

3.4.9 Generating a random password with CLI

PurposeGenerate a random passwords with command line interface.

SummaryThe random passwords can be generated with fsmkpasswd tool.

Steps


2 Generate a random password.The random passwords can be generated with fsmkpasswd tool. The full syntax of the command with all options is the following.

fsmkpasswd [-h] [-l]

With the -l option the length of the password can be set. By default, the value is 64.


95


Id:0900d805803f202d

3.4.10 Changing passwords for user accounts

PurposeFor security reasons, the user account passwords must be changed occasionally.

Before you startPasswords are changed in the command line interface. You must be logged in as root user to change someone else's password. If the account is stored in LDAP you can change your own password logging in with your own user account. If the account is stored in the /etc you must be logged in as root to change the password.

SummaryNote that changing passwords is not supported in Application Launcher. Instead, please log into the network element with SSH and change the password with command passwd.

Steps

1 Log in as any user.

2 Change your own password.Enter:

passwd

Example: Enter:<password>Changing the password for the user Nemuadmin.Enter login(LDAP) password:

Enter here the current password.After this, enter new password.

3 Change someone else's password.

a) Switch to root user.Enter:su - root Enter the password of the root user.

b) Enter the user name for the user whose password you are going to change:passwd <username>The username is the same as the uid attribute in the Parameter Tool.

4 Change a password stored in the \etc.

a) Switch to root user.Enter:su - root Enter the password of the root user.



Id:0900d805803f202d

b) Enter the user name for the user whose password you are going to change:passwd <username>The username is the same as the uid attribute in the Parameter Tool.

c) Distribute the new password.After changing the password of the root account, or other user account that is stored in the /etc, the new password must be distributed to the permanent config-uration files and to all nodes in the cluster.Enter:fsdistribute /etc/shadow

Expected outcomeThe password is changed.

Unexpected outcomeThe system gives a notice if the password you entered is too weak. A strong password contains, for example, both upper and lower case letters with numbers.

VerificationLog in with the new password.


97


Id:0900d805802ef1fb

3.4.11 MMI mappingIf you want to give an MMI mapping use the command zmmimapping. This command adds/removes MMI mapping to/from users, and prints OMS MMI mapping information. Note that all users who have MMI mapping are printed.

MMI mapping is used by the MMI Window application to log OMS users into the OMU. For more information about MMI, see Chapter MMI window in Using Element Manager in RNC OMS.

Usage:

zmmimapping -a <OMS username> <mapped OMU username>zmmimapping -d <OMS username>zmmimapping -p

Options:

-a add mapping-d delete mapping-p print all mappings



Id:0900d8058055efbf

3.5 Credential service management

3.5.1 Checking the password of a credential

SummaryThe password of a credential can be checked with the fsgetcred command.

Steps

1 Log in as root user.

2 Check the password of the credential.To print out the credential password related to a certain service and user account, use the following command:

fsgetcred <service> <username>

If the user name is not given as an argument, the user's own user name is used.

Example: To check the credential password of the fsLDAPRoot user account for LDAP service, enter the following command:

fsgetcred ldap fsLDAPRoot

The password is printed on the screen:

cMFCupIsM6LnSPfIh2Q9eB1YOVAa9rQiwU05UO5DOHAsV7ilAWfHsBWxkzIeZdnt

Expected outcomeThe password is printed on the screen.

Unexpected outcomeIf the password for a credential does not exist or the user does not have a permission to read it, the following error code is printed on the screen:

PASSWORD NOT ACCESSIBLE!


99


Id:0900d805805698aa

3.5.2 Setting the password for a credential

SummaryThe password of a credential can be set with the fssetcred command.

Steps


2 Set the password for a credential.To set a credential password related to a certain service and user account, use the fol-lowing command:

fssetcred <service> <username> -c [OPTIONS]

A new /<service>/<username>.cred subdirectory and password file in the /opt/Nokia_BP/etc/security/credentials directory will be created, if they do not exist.

To specify the customised ownership of the credential for a certain user group, use either the option -g <groupname> or the option -k <gid number>.

To specify the customised ownership of the credential for a certain user account, use either the option -u <username> or the option -v <uid number>.

By default, the password for the credential is asked from the user when executing the command. If the -p option is used, the password is read from standard input (stdin) and if the option -f <filename> is used, the password is read from the text file specified.

Example: To set the password for the LDAP service for the user _nokfsFtpUser and to create a new credential, enter the following command:

fssetcred ldap _nokfsFtpUser -c

The following dialog is printed on the screen:

Please give the new password:

Please confirm the new password:

Password file CREATED for service ldap and user _nokfsFtpUser by root.User owner will be root (=default) and group owner will be root (=default)Distributing data to all nodes...

Example: To set the password for the LDAP service for the user _nokfsFtpUser, to set the group ownership of the credential file for the group _nokfssysnwi3adapter and to create a new credential, enter the following command:

fssetcred ldap _nokfsFtpUser -c -g _nokfssysnwi3adapter




Id:0900d805805698aa



Password file CREATED for service ldap and user _nokfsFtpUser by root.User owner will be root (=default) and group owner will be _nokfssysnwi3adapter.Distributing data to all nodes...

Example: To set the password for an existing credential for the LDAP service for the user _nokfsFtpUser, enter the following command:

fssetcred ldap _nokfsFtpUser




Password file MODIFIED for service ldap and user _nokfsFtpUser by root.User and group owner will not be changed.Distributing data to all nodes...

Expected outcomeThe password is set and the new credential file is created, if necessary. Depending on the options, for example the following output is printed on the screen:

Password file CREATED for service <service> and user <username> by root.User owner will be root (=default) and group owner will be root (=default)Distributing data to all nodes...

Unexpected outcomeIf the -c option is not used, when trying to create a new credential, the following error message is printed on the screen:

Password file MODIFICATION/CREATION failed for service <service> and user <username>

If the defined user group does not exist in the system, the following error message is printed on the screen:

Defined group <groupname> was not found from the system; exiting now!


101


Id:0900d80580552fd0

3.5.3 Changing the ownership of a credential

SummaryThe ownership of a credential can be changed with the fssetcred command.

Steps


2 Change the user ownership of the credential.To change the user ownership of the credential, use the following command:

fssetcred <service> <username> -u <username>|-v <uid number>

The user ownership of the <username>.cred password file in the /opt/Nokia_BP/etc/security/credentials/<service> directory will be changed.

Example: To change the user ownership of the credential for the LDAP service for the user _nokfsFtpUser to root, enter the following command:

fssetcred ldap _nokfsFtpUser -u root




Password file MODIFIED for service ldap and user _nokfsFtpUser by root.User owner will be root and group owner will not be changed.Distributing data to all nodes...

3 Change the group ownership of the credential.To change the group ownership of the credential, use the following command:

fssetcred <service> <username> -g <groupname>|-k <gid number>

The group ownership of the <username>.cred password file in the /opt/Nokia_BP/etc/security/credentials/<service> directory will be changed.

Example: To change the group ownership of the credential for the LDAP service for the user _nokfsFtpUser to _nokfssysnwi3adapter , enter the following command:

fssetcred ldap _nokfsFtpUser -g _nokfssysnwi3adapter




Id:0900d80580552fd0



Password file MODIFIED for service ldap and user _nokfsFtpUser by root.User owner will not be changed and group owner will be _nokfssysnwi3adapter.Distributing data to all nodes...

☞ Both user and group ownerships can be changed with the same command when using both -u and -g options in the command.

Expected outcomeThe ownership of the credential is changed. Depending on the options, for example the following output is printed on the screen:

Password file MODIFIED for service <service> and user <username> by root.User owner name will be <username> and group owner name <groupname>.Distributing data to all nodes...

Unexpected outcomeIf the defined user account does not exist in the system, the following error message is printed on the screen:

Defined user <username> was not found from the system; exiting now!

If the defined user group does not exist in the system, the following error message is printed on the screen:

Defined group <groupname> was not found from the system; exiting now!


103


Id:0900d8058056d056

3.5.4 Restoring a credential

SummaryThe previous password file of the credential can be restored with the fssetcred command.

Steps


2 Restore the password file of the credential.To restore the previous password file of a certain credential, use the following command:

fssetcred <service> <username> -r

The <username>.cred.old password file in the /opt/Nokia_BP/etc/security/credentials/<service> directory will be restored. The current password file <username>.cred will be changed to backup file.

Example: To restore the password file for the LDAP service for the user _nokfsFtpUser, enter the following command:

fssetcred ldap _nokfsFtpUser -r

The following output is printed on the screen:

Password file MODIFIED for service ldap and user _nokfsFtpUser by root.User and group owner will not be changed.Distributing data to all nodes...

Expected outcomeThe previous password file is restored. Depending on the options, for example the fol-lowing output is printed on the screen:

Password file MODIFIED for service <service> and user <username> by root.User and group owner will not be changed.Distributing data to all nodes...

Unexpected outcomeIf no backup password file for the credential exists in the system, the following error message is printed on the screen:

Credential Service password file /opt/Nokia_BP/etc/security/credentials/<service>/<username>.cred.old cannot be located



Id:0900d8058056e9f5

3.5.5 Deleting a credential

SummaryThe credential for a certain service for a certain user account can be deleted with the fssetcred command.

Steps


2 Delete the credential.To delete the credential for a certain service for a certain user account, use the following command:

fssetcred <service> <username> -d

The <username>.cred password file in the /opt/Nokia_BP/etc/security/credentials/<service> directory will be deleted and copied as the backup file named <username>.cred.old.

Example: To delete the credential for the LDAP service for the user _nokfsFtpUser, enter the following command:

fssetcred ldap _nokfsFtpUser -d

The following output is printed on the screen:Credential Service password file /opt/Nokia_BP/etc/security/credentials/ldap/_nokfsFtpUser.cred backed up successfully. Removing it with fsdistribute -d...Distributing data to all nodes...

Expected outcomeThe credential is deleted and the password file is copied as the backup file. The follow-ing output is printed on the screen:

Credential Service password file /opt/Nokia_BP/etc/security/credentials/<service>/<username>.cred backed up successfully. Removing it with fsdistribute -d...Distributing data to all nodes...

Unexpected outcomeIf no password file for the credential exists in the system, the following error message is printed on the screen:

Credential Service password file /opt/Nokia_BP/etc/security/credentials/<service>/<username>.cred.old cannot be located


105


Id:0900d8058056b82d

3.6 Centralised user authentication and authorisation

3.6.1 Centralised User Authentication and Authorisation☞ Note that NetAct integration has to be completed before enabling Centralised User

Authentication and Authorisation.

The Centralised User Authentication and Authorisation (CUAA) feature allows users to log in with user accounts that are stored in an external NetAct LDAP server, in addition to being able to log in with user accounts stored in either the passwd file in the /etc directory, or in the network element (NE) internal LDAP server. Centralised User Authentication and Authorisation functionality allows the utilisation of user information from a NetAct Centralised User Authentication and Authorisation remote centralised user management system, for user authentication and authorisation on external man-agement interfaces in the NE. Enabling and disabling Centralised User Authentication and Authorisation does not have any effect on how internally-defined users, stored in the passwd file in the /etc directory and in the internal LDAP server, are able to access the NE. Enabling and disabling Centralized User Authentication and Authorization can be done by CLI command fscontrolRUIM.

Centralised User Authentication and Authorisation allows logging in to the network element with a username defined in an external NetAct LDAP server, while permissions for remote users are set locally within the network element according to authorisation data in external LDAP server.

An important feature of Centralised user authentication and authorisation is external user replication or caching. According to Centralized User Authentication and Authori-sation architecture, external user accounts are replicated from the external NetAct LDAP server into the internal LDAP server fragment fsFragmentId=security-ruim-cache, fsClusterId=ClusterRoot, except for password and password expiration data. The main reason for this is to reduce dependency on the external LDAP server. By having the information locally cached, NE-internal operations, like the retrieval of groups for service accounts, will not be unnecessarily delayed if the external LDAP server is unavailable. Replicating Centralised User Authentication and Authorisa-tion user entries locally also allows adapting other user-specific entries, such as home directories or login shells as part of the replication.

When replicating user accounts, permissions are assigned to the local replica of a user account strictly according to permissions assigned to this account in the external LDAP server. Local permissions for a replicated account include operating system group mem-berships, login shell and home directory. Note that permissions must be explicitly assigned to an account in the external LDAP server. By default, a replicated account does not get for example a login shell or a home directory, and might be even not repli-cated at all if no valid permissions are assigned in the external LDAP server.

The NetAct user IDs 0 to 999 are reserved for NetAct internal services and they are ignored as Centralised User Authentication and Authorisation users by default, with the exception of the omc user account with ID 401 (required for compatibility reasons). The range for the valid NetAct user IDs are set with the ruim.replicator.uid_range configuration parameter in the replicator_properties.cfg configuration file located in the /opt/Nokia_BP/SS_AAA/etc directory. For example, it is possible to set the parameter as follows:

ruim.replicator.uid_range=401,1000-9999999



Id:0900d8058056b82d

In the internal LDAP server, the replicated user ID X is mapped to 226-X.

Authorisation data (groups and permissions) is presented uniformly for both internal and external users. Services in the network element that need to perform authorisation deci-sions for external users do that in the same way as for internal users.

Replication can be triggered by several events in the system:

• First login with a valid external username, or a login which happens after the local replica of an account in cache was cleaned up after cache expiry.

• Time-based replication (cache refreshing). • Replication forced by CLI command fsruimrepcli. • Execution of some system commands, for example the id command.

Replication is done only if the external user account passes validation checks, meaning it is created according to NE-internal policies, and does not conflict with internal user data.

If changes to the permissions associated to the external user account are detected during replicated account data checks while there are active sessions open with this account, the alarm 70269 is raised to notify that the permissions have changed. Note that changes in permissions typically do not have immediate effect on existing open ses-sions.


107


Id:0900d8058041801b

3.6.2 Centralised User Authentication and Authorisation replicator command line interface toolThe Centralised User Authentication and Authorisation replicator command line inter-face tool, fsruimrepcli, controls the Centralised User Authentication and Authorisa-tion replicator process which is responsible for replicating Centralised User Authentication and Authorisation user information from an external LDAP server to a internal LDAP server, noting that corresponding password information is not also repli-cated to the internal LDAP server. Replication is done only if the external user account passes validation checks, meaning it is created according to NE-internal policies, and does not conflict with internal user data.

The fsruimrepcli command can be executed in several modes to perform several different functions. Before the command can be used, though, two things must be assured:

1. The Centralised User Authentication and Authorisation feature is enabled (see Enabling Centralised User Authentication and Authorisation).

2. You are logged in as root user.

The following topics describe the modes for the fsruimrepcli tool. To see all options, enter man fsruimrepcli to view the manual pages for this command.

Using Centralised User Authentication and Authorisation in limited modeLimited mode can be enabled by executing:

fsruimrepcli --limitedmode

In this mode, user replication from the external LDAP server is stopped. This mode is useful when it is desired to use Centralised User Authentication and Authorisation still in the network element, but, at the same time, isolate the network element from NetAct Centralised User Authentication and Authorisation updates, if those updates might tem-porarily cause data inconsistency in the external LDAP server.

When in limited mode, only already-replicated external users can log in to the network element. Password verification is done against the external LDAP server, and all user authorisation data updates, with the exception of the fsruimrepcli --refreshcache command option, are denied.

To summarise, if fsruimrepcli --limitedmode is executed with no further argu-ments, then:

• Replication on the first login is disabled • Time-based, or scheduled, replication is disabled • Cache expiry (user invalidation) is enabled • fsruimrepcli --refreshusers is not allowed • fsruimrepcli --refreshcache is allowed

By adding the option --enablerefresh, time-based replication can be enabled. Adding the option --disableinvalidation turns the cache expiry function (user invalidation) off.

Finally, to exit from limited mode, back into the normal mode, execute:

fsruimrepcli --normalmode



Id:0900d8058041801b

Disabling external usersWhen Centralised User Authentication and Authorisation is enabled, it is possible, in the internal LDAP server, to disable certain selected external user accounts which were already replicated. To disable an external account means to remove group member-ships from all the supplementary groups. This functionality can be useful if it is needed to temporarily disable the external account from being operational within one network element, without affecting the centralised user management system. Removing all sup-plementary group memberships means taking away all the permissions associated with those groups.

Disabling can be accomplished to one, multiple or all user accounts by executing:

fsruimrepcli --disableusers --user <username>

or

fsruimrepcli --disableusers --user <username1> --user <username2> ...

or

fsruimrepcli --disableallusers

Note that the Centralised User Authentication and Authorisation replicator will reassign the disabled user with actual permissions when the next scheduled replication happens, when operating in normal mode (see Using Centralised User Authentication and Autho-risation in limited mode).

Forceful replicationReplication for certain external user accounts can be done forcefully by executing:

fsruimrepcli --refreshusers --user <username1> --user <username2> ...

This can be helpful if it is desired to immediately propagate the effect of some authori-sation changes made remotely to the network element. Centralised User Authentication and Authorisation replicator would do the same when the next scheduled replication happens. The --refreshusers command also works for those users which are not yet in the cache. Note that this command is not allowed in limited mode (see Using Cen-tralised User Authentication and Authorisation in limited mode).

To force cache refreshing for all user accounts already stored in the Centralised User Authentication and Authorisation cache fragment, execute:

fsruimrepcli --refreshcache

The above command is permitted in both limited and normal modes (see Using Centra-lised User Authentication and Authorisation in limited mode).

External user cleanupIt is possible to forcefully remove replicated user accounts from the internal LDAP server, together with their primary group and all supplementary group memberships. This can be done by executing the command:

fsruimrepcli --cleanupusers --user <username1> --user <username2> ...

This may be useful, for example, if a user account is removed from the external LDAP server and you want to make sure that this account is removed from the-internal LDAP server immediately. Note that the Centralised User Authentication and Authorisation replicator process also does cleanup of user accounts when operating in normal mode, but not in limited mode. To remove the home directory, use the optional argument --rmdir:


109


Id:0900d8058041801b

fsruimrepcli --cleanupusers --user <username> --rmdir

In order to remove all replicated user accounts from the LDAP server, the following command is used:

fsruimrepcli --cleanupallusers

For the above command, the home directory for the users can also be removed using the --rmdir option.

Invalidating user accountsCached user accounts have a timestamp attribute lastLoginTime which indicates the time of the last successful login for the user. Centralised User Authentication and Autho-risation replicator periodically goes through the replicated accounts and removes those which have not been used to log in for some period of time.

It is possible to override user invalidation actions of Centralised User Authentication and Authorisation replicator with the fsruimrepcli command. For example:

fsruimrepcli --invalidateusers --invalidatetime 10d --user <username>

The above command will verify if the cache has not been used for more than ten days, and will clean up the user account if it was not. The time value can be specified in seconds, minutes, hours, or days. The unit of time can be specified after the number. For example, 10 or 10s means 10 seconds, 10m indicates 10 minutes, 10h indicates 10 hours, and 10d indicates 10 days.



Id:0900d805803e10da

3.6.3 Enabling Centralised User Authentication and Authorisation

SummaryTo enable the Centralised User Authentication and Authorisation feature, use the fscontrolRUIM command with appropriate options. This command enables Centra-lised User Authentication and Authorisation by performing modifications on configura-tion files and on the internal LDAP server. Note that before the Centralised User Authentication and Authorisation is enabled, the related recovery groups /PAP and /RuimReplicator are running in dummy mode. That is, they are running, but they do not process any centralised user authentication and authorisation-related requests.

The script edits files related to pluggable authentication module (PAM), Centralised User Authentication and Authorisation fragment, network service switch (NSS), and NWI3 adapter fragment in the LDAP server. The files modified are:

• PAM: all files starting with fp in the /etc/pam.d directory. • NSS: fpnsswitch.conf file in the /etc directory. • NWI3 adapter: an LDIF file is added to the internal LDAP. This is optional an it is

generally used for the first time or when there is no LDIF present in the internal LDAP.

The values used for the editing of the above files are supplied in a configuration file. The default configuration file name is fscontrolRUIM.conf, located in the SS_AAA sub-system directory /opt/Nokia_BP/SS_AAA/etc. The LDIF file necessary for updating the NWI3 adapter mediator fragment in the internal LDAP server must be specified if the NWI3 adapter mediator fragment is to be configured. This file is typically provided by NetAct.

Note that, by default, the Centralised User Authentication and Authorisation feature is disabled, and it can be enabled after commissioning.

Steps


2 Execute the fscontrolRUIM command with appropriate options.

Example: Enable Centralised User Authentication and Authorisation with default configuration by running the command:

fscontrolRUIM --enable

For other options in enabling Centralised User Authentication and Authorisation, see the manual pages for fscontrolRUIM by entering man fscontrolRUIM.


111


Id:0900d805802a2a06

3.6.4 Disabling Centralised User Authentication and Authorisation in OMS

SummaryTo fully disable the Centralised User Authentication and Authorisation (CUAA) feature, run the command fscontrolRUIM with the --disable option.

Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator user, and then change the root user permission by entering su - command.For more information, see Logging in to OMS.

2 Execute the fscontrolRUIM command with the --disable option.To fully disable the Centralised User Authentication and Authorisation feature, enter the following command:

fscontrolRUIM --disable

This undoes the changes done to PAM and NSS configuration files, removes all repli-cated user accounts from internal LDAP server and restarts the /PAP and /RuimReplicator recovery groups in dummy mode. That is, they are running, but they do not process any Centralised User Authentication and Authorisation-related requests.

Disabling Centralised User Authentication and Authorisation does not have any effect on how internally-defined users, stored in the passwd file in the /etc directory and in the internal LDAP server, are able to access the network element.

For more information about the options, see the manual pages for fscontrolRUIM by entering man fscontrolRUIM.

Expected outcomeCentralised User Authentication and Authorisation is disabled.

Unexpected outcomeDisabling Centralised User Authentication and Authorisation will fail. The failure happens, if any of the replicated users is logged in when executing fscontrolRUIM --disable command.



Id:0900d8058043c400

3.6.5 Changing target external LDAP server

PurposeModify existing Centralised User Authentication and Authorisation configuration to change target external LDAP server.

Before you startThe Parameter tool can be used only with OMS user account (local account), not with Centralised User Authentication and Authorisation user account.

Check that:

• the Parameter Tool is running. • you have appropriate permission to modify the LDAP configuration.

SummaryThe target external LDAP server can be changed directly via attribute values in the internal LDAP server under fsFragmentId=mediator, fsFragmentId=NWI3, fsClusterId=ClusterRoot, however the active subfragment defining the attributes being used in centralised user authentication and authorisation must be determined first.

It is expected that NetAct may push new values for configuration attributes stored under the fsFragmentId=NWI3, fsClusterId=ClusterRoot fragment in internal LDAP server. All centralised user authentication and authorisation components that use these attributes are expected to take new values into use automatically. This is also true when attributes are updated manually.

Steps

1 Start the Parameter Tool.Start the Parameter Tool. For more information, see Starting Parameter Tool.

2 Browse to the Security fragment to find out which subfragment is active.The fragment fsFragmentId=mediator, fsFragmentId=NWI3, fsClusterId=ClusterRoot contains one or more fsnwi3N3CFId subfragments, one of which is supposed to be active. The active fragment defines the attributes being in use in centralised user authentication and authorisation. To find out which fsnwi3N3CFId is active (if there is more than one), please check attribute fsnwi3ActiveN3CF under fsFragmentId=security, fsFragmentId=NWI3, fsClusterId=ClusterRoot.

3 Browse to the active fsnwi3N3CFId subfragment.Navigate to the active fsnwi3N3CFId subfragment in fsFragmentId=mediator, fsFragmentId=NWI3, fsClusterId=ClusterRoot.

4 Modify the correct attribute values.To change target external LDAP server, replace the IP address of the external LDAP server with the new IP address in the attribute for the LDAP server. Note that the attri-bute names are descriptive and intuitive relative to their function. It is possible to change


113


Id:0900d8058043c400

the addresses for both the primary and secondary LDAP servers. Also, change the port number for the external LDAP server in the attribute for the LDAP server.

5 Restart /PAP and /RuimReplicator recovery groups.Log in as root user and restart the centralised user authentication and authorisation-related recovery groups with the following commands:

fshascli -r /PAPfshascli -r /RuimReplicator

Further informationFor the external LDAP server, there are several configuration items or attributes which should be set up correctly for centralised user authentication and authorisation to work, including configuration items for primary LDAP server, secondary LDAP server, initial registration account, and Network Element (NE) account. The NE account is the account used by the network element to authenticate itself on the external LDAP server.

Note that the initial registration username attribute, that is, fsnwi3initialRegistrationUsername can contain either the full distinguished name of the user in Centralised user authentication and authorisation ldap or just the plain username. In case only plain username is used then the full distinguished name is constructed from concatinating this plain username with the content of the fsnwi3PrimaryPeopleRootDN or fsnwi3SecondaryPeopleRootDN attribute, depending on which LDAP server is used. Below is an example about the valid user-names.

Example: Options for valid initial registration usernamesfsnwi3initialRegistrationUsername: uid=username,ou=People,ou=accounts,\ou=region-911080,ou=regions,ou=netact,dc=noklab,dc=net,dc=localdomain

fsnwi3PrimaryPeopleRootDN: ou=people,ou=accounts, ou=region-911080,ou=regions,\ou=netact,dc=noklab,dc=net, dc=localdomain

fsnwi3SecondaryPeopleRootDN: ou=people,ou=accounts, ou=region-911080,ou=regions,\ou=netact,dc=noklab,dc=net, dc=localdomain

or

fsnwi3initialRegistrationUsername: username

fsnwi3PrimaryPeopleRootDN: ou=people,ou=accounts, ou=region-911080,ou=regions,\ou=netact,dc=noklab,dc=net, dc=localdomain

fsnwi3SecondaryPeopleRootDN: ou=people,ou=accounts, ou=region-911080,ou=regions,\ou=netact,dc=noklab,dc=net, dc=localdomain

By default, primary LDAP server is used. If the primary LDAP server is not available, the secondary LDAP server is used. In this case, centralised user authentication and autho-risation components switch back to using the primary LDAP server only if the secondary LDAP server becomes unavailable.



Id:0900d8058043c400

By default, the network element (NE) account is used. If it is not available, or does not work, the initial registration account is used. The initial registration account is also used normally when integrating the network element with NetAct, when the real NE account has not yet been provided by NetAct.

Note that alarm 70268 is raised if there are problems with the external LDAP server.


115

Administering RNC OMS Certificate management

Id:0900d80580569fe9

4 Certificate management

4.1 Certificate maintenanceMore information from certificates see Security in RNC OMS document.



Id:0900d8058040f6a9

4.2 Checking the expiration dates of OpenSSL certificatesBefore you startCheck that you know the correct paths to the keystore files of the OpenSSL certificates.

SummaryThe expiration times of the OpenSSL certificates can be checked with the openssl command line tool.

Steps

1 Check the validity of the certificate with the openssl command line tool.To display the certificate information on the screen, enter the following command:

openssl x509 -in <path_to_keystore> -noout –text

Example: Checking the expiration date with the openssl command line tool.

openssl x509 -in cacert.pem -noout –text

The printout of the command is, for example, the following:

Certificate: Data: Version: 3 (0x2) Serial Number: 0 (0x0) Signature Algorithm: md5WithRSAEncryption Issuer: C=FI, ST=Helsinki, L= , O=Nokia, OU=Networks, CN=FS CA Validity Not Before: Apr 21 12:04:31 2006 GMT Not After : Apr 18 12:04:31 2008 GMT Subject: C=FI, ST=Helsinki, L= , O=Nokia, OU=Networks, CN=FS CA Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public Key: (1024 bit) Modulus (1024 bit): 00:9d:0c:13:98:df:fb:c2:19:bf:1a:43:0d:f3:ae: 8d:37:37:e3:94:1f:d4:d0:2a:8b:2b:3a:59:e3:b8: 91:4f:1b:2d:cc:8d:d5:f6:e5:10:c6:91:38:0e:95: 88:2b:be:1d:b0:ff:25:c8:59:64:45:2b:1f:95:09: ce:ad:fe:bd:6f:82:ef:de:21:37:b7:9a:d7:ff:74: 77:94:01:3d:6f:fb:c0:ae:b9:95:94:df:6c:8b:d0: 87:e0:52:86:b1:95:45:30:ae:29:38:19:23:e1:90: 8a:4c:ac:88:fc:89:d4:fb:fe:3c:41:91:9d:6e:ab: 29:02:99:47:00:31:f0:84:37 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: 26:30:12:A1:01:D9:B4:DE:4F:87:10:62:4E:DB:BE:2C:09:7D:9C:44 X509v3 Authority Key Identifier: keyid:26:30:12:A1:01:D9:B4:DE:4F:87:10:62:4E:DB:BE:2C:09:7D:9C:44 DirName:/C=FI/ST=Helsinki/L= /O=Nokia/OU=Networks/CN=FS CA


117


Id:0900d8058040f6a9

serial:00

X509v3 Basic Constraints: CA:TRUE Signature Algorithm: md5WithRSAEncryption 45:6b:04:5f:f0:0a:f2:c0:fc:1c:62:36:db:7e:c4:a1:90:88: 80:d4:61:3d:1f:bb:91:f6:cc:4e:88:37:6e:bd:94:f2:d3:81: 56:e3:4c:a9:2c:f1:24:ce:34:a0:5e:5f:85:db:a6:be:30:33: bb:6c:51:80:48:ee:79:e2:2e:43:5b:91:06:f6:37:57:02:f0: c1:69:cd:39:87:cd:77:bf:ae:60:bf:00:08:10:56:33:31:fe: 8d:62:42:b7:0d:58:bd:d0:e9:2c:30:ba:24:e6:66:ec:11:af: 09:c1:95:26:0b:b9:68:52:d9:01:c7:14:af:b1:8d:30:4b:b4: 29:c9



Id:0900d8058061c183

4.3 Updating X.509 certificates and private keyPurposeThese instructions describe how to update OMS certificates and private key pairs.

SummaryTo update an OpenSSL certificate, you need to request the updating from your trusted certification authority (CA) that has signed your previous keys. In the node, the certifi-cate is updated by replacing the expired certificate file with the valid one signed by your CA. Self-signed certificates can be updated by creating new self-signed certificates and replacing the old certificates with the valid ones.

Steps

1 If you have the trusted CA-signed keys

Then

Request the updating from CA.Send your existing certification request file to your CA, or create a new request file. For more information, contact your trusted CA.

Replace old certificates with the new ones. See Step 2 for information on correct paths for the ceritificate files.

2 If you have self-signed keys

Then

Old certificate files are replaced by new ones.Copy the updated certificate files and corresponding private keys to the correct locations in the node.

The correct location for the LDAP certificate and private key files are the /etc/certificates and /etc_ondisk/certificate directories.

The correct location for the HTTP certificate and private key files is the /opt/Nokia/SS_HTTPDPlat/etc directory.

The correct location for the CORBA certificate file and CORBA private key file is the /opt/Nokia/SS_Nwi3Adapter/etc directory.

Copy the files with the following command:

cp <filename> <target directory>

where <filename> is the name of the certificate file and <target directory> is the location where the appropriate certificate file must be installed.

3 Take new certificates into use.

1 LDAP certificateRestart the LDAP server:

fshascli -r /CLA-0/FSDirectoryServer/LDAPServer


119


Id:0900d8058061c183

2 CORBA certificateRestart the NWI3 Adapter:

fshascli -r /NWI3Adapter

3 HTTP certificateRemove /tmp/HTTPDPlat directory:

rm -rf /tmp/HTTPDPlat

Restart the HTTP server:

fshascli -r /HTTPDPlat



Id:0900d8058053abad

4.4 Checking that certificate and private key is a valid pairBefore you startCheck that you know the correct paths to the keystore files of the certificates.

SummaryThe validation of the certificates can be checked with the openssl command line tool.

Steps

1 Check the validity of the certificate and the private key with the openssl command line tool.Enter the following command:

openssl x509 -noout -modulus -in <path_to_cerstore> | openssl md5;openssl rsa -noout -modulus -in <path_to_keystore> | openssl md5

If printed rows are identical, certificate private key pair is valid.

Example: Checking validation with the openssl command line tool.

openssl x509 -noout -modulus -in /etc/certificates/ldapcert.pem | openssl md5;openssl rsa -noout -modulus -in /etc/certificates/ldap-private-key.pem | openssl md5

The printout of the command is, for example, the following:

7212522e39887c48b9099eae36c83a1c7212522e39887c48b9099eae36c83a1c


121

Administering RNC OMS Data management

Id:0900d80580566a3f

5 Data management

5.1 MySQL supportMySQL is a disk-based database system that is used for collecting and persistently storing large amounts of data. It can be used as a stand-alone data engine to manage local databases or with replication features to manage replicated databases. The MySQL database is used, for example, to store alarm system data and naming service data.

The MySQL database is a single set of tables, views, and triggers that are stored in same sub-directory under MySQL data directory on a local disk. It can also be cached into memory for faster access.

The database is maintained by MySQL server. One MySQL server instance can maintain several databases. The subsystems and applications using the MySQL services can be located either in the same node as MySQL itself or in a different node.

High availability services for MySQLThe MySQL server has a watchdog process called MySQLWD. The purpose of the watchdog process is to integrate the MySQL server and the disk database under the supervision of high availability services (HAS), ensuring the availability of the database in runtime environment. Note that HAS supervision does not cover replicated data-bases, that is, the MySQL watchdog does not check if the replication of a disk database is functioning or not.

MySQL database is stored on a mirrored disk. The MySQL server is either implemented as redundant in a cold active standby mode or without redundancy.

The database HAS for MySQL implements the following services:

• Supervises the MySQL server by establishing a connection to the database engine and using a heartbeat to monitor it.

• Monitors database fill ratio periodically, raising an alarm if it exceeds a certain threshold.

• Reports errors to the HAS or the alarm system.The HAS does not react to physical problems, such as hardware failures, until MySQL tries to access the broken component causing the operation to fail.

• Controls the MySQL server in graceful shutdown translating shutdown requests to corresponding administrative MySQL commands.

In the cold active standby configuration, the HAS unmounts the disk partition on the node holding the previous active recovery unit and mounts it on the node running the new active recovery unit in case of a switchover or a failure.

The operator can manually perform the graceful shutdown of MySQL server. The lock operation is also possible, but to be used only if graceful shutdown fails. For more infor-mation about shutdown operations, see the high availability services documentation.

MySQL databases implemented in OMSIn OMS, MySQL is implemented in the following two databases:

• Alarm System database • Corba Naming Service database

Both of these databases are implemented in a cold active standby mode.



Id:0900d805804f059b

5.2 Collecting MySQL log files in OMSPurposeCollect MySQL log files from the OMS node and send them to your Nokia Siemens Networks representative for analysing and troubleshooting purposes.


• you have root access rights to OMS.

Steps


2 Check where the active MySQL database is located.

a) Find out the recovery units for the recovery group by entering the following command:fshascli -v <RG>where RG is the recovery group that the MySQL database belongs to.Example: Example output of recovery group status

# fshascli -v /AlarmDBexecuted as root

# fshascli -v /AlarmDBexecuted as root

/AlarmDB:RecoveryGroup /AlarmDBRecoveryUnit /CLA-0/FSAlarmDBServerProcess /CLA-0/FSAlarmDBServer/ControlMySQLforAlarm command=(/opt/Nokia/SS_MySQL/script/ControlMySQL start DB_Alarm SS_AlProcessor ) status=(nonHA) startMethod=(requested) severity=(important)Process /CLA-0/FSAlarmDBServer/MySQLWDforAlarm command=(/opt/Nokia/SS_DBHAforMySQL/bin/MySQLWD DB_Alarm ) status=(fullHA) startMethod=(requested) severity=(important)RecoveryUnit /CLA-1/FSAlarmDBServerProcess /CLA-1/FSAlarmDBServer/ControMySQLforAlarm command=(/opt/Nokia/SS_MySQL/script/ControlMySQL start DB_Alarm SS_AlProcessor ) status=(nonHA) startMethod=(requested) severity=(important)Process /CLA-1/FSAlarmDBServer/MySQLWDforAlarm command=(/opt/Nokia/SS_DBHAforMySQL/bin/MySQLWD DB_Alarm ) status=(fullHA) startMethod=(requested) severity=(important)


123


Id:0900d805804f059b

This example indicates that the recovery units for the AlarmDB recovery group are running on nodes CLA-0 and CLA-1.

b) Find out on which node the MySQL database is active by entering the following com-mands:fshascli -s <RU_node1>fshascli -s <RU_node2>where RU_node1 and RU_node2 are the recovery units from the previous step.The commands display the recovery unit information for each node. Search for the node containing the line role(ACTIVE).Example: Example output of recovery unit information

# fshascli -s /CLA-0/FSAlarmDBServerexecuted as root/CLA-0/FSAlarmDBServer:administrative(UNLOCKED)operational(ENABLED)usage(ACTIVE)procedural()availability()unknown(FALSE)alarm()role(ACTIVE)

# fshascli -s /CLA-1/FSAlarmDBServerexecuted as root/CLA-1/FSAlarmDBServer:administrative(UNLOCKED)operational(ENABLED)usage(IDLE)procedural(NOTINITIALIZED)availability()unknown(FALSE)alarm()role(COLDSTANDBY)

This example indicates that the active instance of FSAlarmDBServer is running on the CLA-0 node.

3 Go to the directory where active database instance is located.Enter the following command:

cd /var/mnt/local/MySQL_<DB_name>

Example: cd /var/mnt/local/MySQL_DB_Alarm

4 Collect the log and configuration files.Collect the following files:

• mysql.err (MySQL server log file)



Id:0900d805804f059b

• my.cnf (MySQL configuration file) • odbc.ini (ODBC configuration file)

5 Send the files to your Nokia Siemens Networks representative.


125


Id:0900d805804c44b7

5.3 Monitoring MySQL mount pointsSummaryWith MySQL databases there are two alternative ways to configure the disk usage:

• by using preallocated table space file(s) • by using autoextended table space file(s)

In configurations using the preallocated size definition, the fill ratio of the disk resource is typically high (over 90%), but it stays almost stable. This is because the only growing file related to the database is the mysql.err message log file in the /var/mnt/local/MySQL_DB_<MySQL database name> directory.

In the configurations using the autoextended definition the maximum size for the file(s) is set. Thus, fill ratio may start from small percentage but grow in time. Note that the table space is never shrunk and the fill ratio can decrease only if logrotate archives the mysql.err file by packing it and creating a new one. If the configuration is correct, that is, the table space file size does not exceed disc resource size, the fill ratio should not exceed 95%.

The used configuration can be checked by finding the attribute value innodb_data_file_path in the my.cnf file in the /var/mnt/local/MySQL_DB_<MySQL database name> directory.

• If the value is of innodb_data_file_path = ibdata1:500M type, the preallo-cated table space configuration is used.

• If the value is of innodb_data_file_path = ibdata1:500M:autoextend:max:1G type, the autoextended table space con-figuration is used.

Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase , and then change the root user permission by entering su - command.For more information, see Logging in to OMS in Installing and commissioning RNC OMS.

2 Check the used configuration.With MySQL databases there are two alternative ways to configure the disk usage:

• by using preallocated table space file(s) • by using autoextended table space file(s)

The used configuration can be checked by finding the attribute value innodb_data_file_path in the my.cnf file in the /var/mnt/local/MySQL_DB_<MySQL database name> directory. Enter the fol-lowing command:

grep innodb_data_file_path /var/mnt/local/MySQL_DB_<MySQL database name>/my.cnf

If the value is of type innodb_data_file_path = ibdata1:500M the preallocated table space configuration is used. If the value is of type innodb_data_file_path = ibdata1:500M:autoextend:max:1G the autoextended table space configuration is used.



Id:0900d805804c44b7

3 If the preallocated table space configuration is used

Then

Check the stability of the fill ratio in the MySQL mount points.In the configurations using the preallocated size definition the fill ratio of the disk resource is typically high (over 90%) but it stays almost stable. This is because the only growing file related to the database is the mysql.err message log file in the /var/mnt/local/MySQL_DB_<MySQL database name> directory.

a) MySQL database mount point name is of type /var/mnt/local/MySQL_DB_<MySQL database name>. To check the fill ratio of the MySQL mount point, enter the following command:df –h /var/mnt/local/MySQL_DB_<MySQL database name>The command shows the fill ratio in megabytes of all MySQL database mount points.Example:

root@CLA-0:~# df -h /var/mnt/local/MySQL_DB_AlarmFilesystem Size Used Avail Use% Mounted on/dev/md9 2.9G 2.6G 193M 94% /var/mnt/local/MySQL_DB_Alarm

To check that the fill ratio has stayed stable, execute the command several times with short pauses between the executions and check if the used size has not grown.

b) If you find the fill ratio growing constantly, check that the general query log or slow query log options are not turned on in the my.cnf file in the /var/mnt/local/MySQL_DB_<MySQL database name> directory. They can be used temporarily, for example, for problem solving purposes and must be turned off after usage. The general query log status is defined with the attribute log in the my.cnf file. The slow query log status is defined with the attribute log_slow_queries in the my.cnf file.

4 If the autoextended table space configuration is used

Then

Check the fill ratio of the MySQL mount points.In the configurations using the autoextended definition, the maximum size for the file(s) is set. Thus, the fill ratio may start from a small percentage but grow in time. Note that the table space is never shrunk and the fill ratio can decrease only if the logrotate archives the mysql.err file by packing it and creating a new one. If the configuration is correct, that is, the table space file size does not exceed the disc resource size, the fill ratio should not exceed 95%. However, note that sizes of the files as well as number of the files depend on the database and accurate limits cannot be defined on general level.

a) MySQL database mount point name is of type /var/mnt/local/MySQL_DB_<MySQL database name>. To check the fill ratio of the MySQL mount point, enter the following command:df –h /var/mnt/local/MySQL_DB_<MySQL database name>The command shows the fill ratio in megabytes of all MySQL database mount points.


127


Id:0900d805804c44b7

Example: root@CLA-0:~# df -h /var/mnt/local/MySQL_DB_CosNamingFilesystem Size Used Avail Use% Mounted on/dev/md8 2.0G 205M 1.7G 11% /var/mnt/local/MySQL_DB_CosNaming

b) Detailed information on the disk space usage of the files in the mount point can be checked with the following command:ls -lah /var/mnt/local/MySQL_DB_<MySQL database name> Example:

root@CLA-0:~# ls -lah /var/mnt/local/MySQL_DB_CosNamingtotal 60Kdrwxr-xr-x 5 root root 4.0K Nov 29 10:27 .drwxr-xr-x 9 root root 0 Nov 29 10:26 ..drwx------ 5 root root 4.0K Nov 29 10:27 datadrwx------ 2 root root 4.0K Nov 29 10:26 ibdatadrwx------ 2 root root 16K Nov 29 09:18 lost+found-r-------- 1 root root 18K Nov 29 10:26 my.cnf-rw-rw---- 1 root root 1.3K Nov 29 10:27 mysql.errsrwxrwxrwx 1 root root 0 Nov 29 10:27 mysql.sock-rw-rw---- 1 root root 5 Nov 29 10:27 mysqld.pid-rw-r--r-- 1 root root 353 Nov 29 10:27 odbc.ini

c) Detailed information on the disk space occupied by the subdirectories in the mount point can be checked with the following command:du –h /var/mnt/local/MySQL_DB_<MySQL database name> | sort -grExample:

root@CLA-0:~# du -h /var/mnt/local/MySQL_DB_CosNaming | sort -gr170M /var/mnt/local/MySQL_DB_CosNaming165M /var/mnt/local/MySQL_DB_CosNaming/ibdata56K /var/mnt/local/MySQL_DB_CosNaming/data/publiccosnaming56K /var/mnt/local/MySQL_DB_CosNaming/data/privatecosnaming16K /var/mnt/local/MySQL_DB_CosNaming/lost+found5.5M /var/mnt/local/MySQL_DB_CosNaming/data5.4M /var/mnt/local/MySQL_DB_CosNaming/data/mysql

d) If you find that the MySQL mount point is reserving too much disk space, check that the general query log or slow query log options are not turned on in the my.cnf file in the /var/mnt/local/MySQL_DB_<MySQL database name> directory. They can be used temporarily, for example, for problem solving purposes, and must be turned off after usage. The general query log status is defined with the attribute log



Id:0900d805804c44b7

in the my.cnf file. The slow query log status is defined with the attribute log_slow_queries in the my.cnf file.

!

Deleting files from the mount points may damage the system and lead to a system outage. Make sure that you have knowledge in the file system and directory hierar-chy of a network element before deleting files.


129


Id:0900d8058055963b

5.4 LDAP directory and parameter management

5.4.1 Parameter managementAll configuration data affecting the behaviour of the system is stored in a centralised con-figuration directory managed by the parameter management (PMGMT) function.

The operator performs configuration tasks on the data stored in the parameter manage-ment. The parameter management maintains tree hierarchies of managed objects and their attributes according to management object models (MOMs). MOMs give common rules for the structure of the data stored in the directory. This helps to maintain the con-sistency of the directory. Instances of managed objects are also known as configuration data objects.

The most typical configuration task is modifying the attribute values of existing configu-ration data objects. The operator can also browse configuration data objects as well as add and delete configuration data objects and their attributes.

Parameter management offers many useful features that provide a more resilient man-agement system than traditional databases or configuration files:

• centralised and replicated data storage, which is achieved by running parameter management on two replica servers, one running in the same node as the primary, and the other one running on another node

• data entry verification for catching errors in new entries before they are propagated to the applications

• data validation by the target applications, which guarantees that erroneous values are not accepted

• protection against parallel modification of a parameter • possibility to revert back to a previous configuration if the changes cause undesired

results.

Parameter management componentsThe parameter management service is provided by a Lightweight Directory Access Protocol (LDAP) server that contains the configuration data. The server is based on the freely available OpenLDAP server, which has been enhanced by Nokia Siemens Networks to provide more robustness and traceability as well as a notification function-ality.

Communication to and from the server uses the standard LDAPv3 protocol. LDAP is used by the applications that store their configuration data in the directory. The operator can use the Parameter Tool for configuration data management. The LDAP connection between the Parameter Tool and the LDAP server in the managed network element is secure and encrypted (LDAP over Secure Sockets Layer Protocol (SSL)).

Parameter management uses the Parameter Management Event Service to notify appli-cations of configuration changes and for requesting data validation from applications.

All the data that is entered into the directory is ultimately stored on the hard disk.



Id:0900d8058055963b

Figure 4 Parameter management interfaces

The use of the Parameter Management Event Service is optional. The applications that do not use the service do not take updated configuration data into use before restart.

LDAPv3 LDAPv3

TCP socket interface

Parameter ManagementEvent Service

ParameterManagement(OpenLDAP

server)

PMGMT Store(OpenLDAP

DB)

ApplicationParameter Tool


131


Id:0900d80580408323

5.4.2 Management object modelParameter management uses a Lightweight Directory Access Protocol (LDAP) directory as a configuration repository to store object and attribute information related to the con-figuration of the system. To maintain the consistency of the directory, management object models (MOMs) are used.

Management object model refers to a modular collection of an object class and attribute definitions that enable the representation of configuration data in the configuration data repository (LDAP directory). The management object model is modular; it is divided into parts called fragments. Each fragment is formally defined by an LDAP schema.

The LDAP directory server and the Parameter Tool use the management object model at runtime in the format of LDAP schemas.

The structure of the management object model is modular to enable the addition of new manageable parts to the system in a controlled manner, even at runtime.

FragmentsA fragment in the management object model represents a functionality or a structure that has its own hierarchy of managed objects and attributes. Even though the fragments can be quite independent, they can also use the definitions of object classes and attributes from other fragments.

A fragment is a method for structuring information comparable to a subdirectory in a file system. Each fragment has its own substructure and LDAP schema.

A fragment may represent a cluster-wide functionality as well as a configuration specific to a logical network element type implemented in the cluster.

The network elements (NEs) can have their own LDAP schemas describing the struc-ture specific to the NE. The NEs also have their own fragments in the LDAP directory tree. The NE-specific fragments can utilise the definitions of attributes and object classes of the platform level.

Viewing management object modelsThe hierarchy of each configurable fragment is described in detail in an object model description. It contains a figure, which shows the containment hierarchy of object classes in the fragment, and a detailed description of each object class and its attributes.

The following figure is an example of a management object model figure:



Id:0900d80580408323

Figure 5 MOM figure example

The MOM figure of a fragment can be compared with the Parameter Tool view of the same fragment. This makes it easier to see the relationships between the object class and the attributes that are being configured.

1..n

FSClusterfrom fsCommon

In fsFragmentId=HW

FSHWEquipmentHolder

<<MUST>> fshwEquipmentHolderId

FSHWFanModule

<<MUST>> fshwFanModuleId

FSHWAlarmIOModule

<<MUST>> fshwAlarmIOModuleId

FSHWPowerModule

<<MUST>> fshwPowerModuleId

In FSHWModuleTypessubfragment

FSHWHolderType

<<MUST>> fshwHolderSpecificType

FSHWModuleType

<<MUST>> fshwModuleSpecificType

In FSHWHolderTypessubfragment

0..n

0..n

0..n

FSHWPlugInUnit

<<MUST>> fshwPlUId

FSHWConnector

<<MUST>> fshwConnectorId

0..n

1..n

0..n

In FSHWPIUTypessubfragment

FSHWPIUType

<<MUST>> fshwPIUSpecificType

FSHWBlock

<<MUST>> fshwBlockId

FSHWPIUConnector

<<MUST>> fshwConnectorId

0..n

1..n

1..n

0..n

0..n


133


Id:0900d80580408323

Figure 6 Example of a Parameter Tool MOM view



Id:0900d8058037beed

5.4.3 LDAP directoryThe Lightweight Directory Access Protocol (LDAP) is used for accessing information directories. LDAP enables access to configuration data objects not only for application components, but also for the management components used by the operator.

An extension of the standard LDAPv3 is used. The extension provides an ability to request validation of the configuration data, and to request the target system to take the modified configuration data into use at runtime. It also provides rollback and recovery features that can be useful for the operator if, for example, an update to the configuration data causes undesired results. The extension also prevents parallel modifications to the configuration data guaranteeing the consistency of the data.

For a detailed description of the standard LDAPv3 protocol, see RFC 2251 - Lightweight Directory Access Protocol (v3).

LDAP directory structureLDAP is used to create a generic data storage system to which the applications can adapt with minimum effort.

The LDAP directory contains a tree of entries (object instances). Each entry can contain several attributes. Each entry in LDAP can have one or more object classes. The object class is one attribute in the entry, and it can have multiple values.

An LDAP schema is a collection of attribute type definitions, object class definitions, and other information the LDAP server uses when determining the correctness of new or modified entries in LDAP.

An object class definition in LDAP is a template for a configuration data entity in the system. It lists the mandatory and optional attributes of the objects in that class. An object class definition does not contain actual attribute values, but only types of attri-butes, and a mention whether they are mandatory or optional. An entry in LDAP is the actual configuration data entity, which contains the actual values of attributes, specific to that entry.

A configuration data entity describes the attributes used by an application, a part of an application, or a feature in the runtime system. The attributes are configuration variables (parameters) whose values are used by the application at runtime, or when the applica-tion starts up.

Each entry also contains a naming attribute that unambiguously identifies the entry in the branch below its parent object. The distinguished name of the entry consists of the full name of its parent entry and a differentiating naming attribute. Therefore each dis-tinguished name is the complete tree path of the object.

LDAP Data Interchange FormatThe operator can import data to the directory or export data from the directory in a stan-dardised format known as the LDAP Data Interchange Format (LDIF). The operator can use either the Parameter Tool or the command line interface shell commands to perform the tasks.

The operator can also use LDIF to make manual backups of the directory and to populate database entries during a software installation and upgrade. LDIF is entirely text-based enabling the operator to create LDIF files even manually.

The LDIF format defines that an entry contains three types of information:

• a single line containing the distinguished name (DN) of the entry


135


Id:0900d8058037beed

• one or more object classes to define the type of an entry • one or more actual attributes

For more information on LDIF, see RFC 2849 - The LDAP Data Interchange Format (LDIF) - Technical Specification.



Id:0900d8058055ee6f

Log management

6 Log management

6.1 Syslog in OMSThe main method of general system logging in the OMS is the standard Linux syslog. It is implemented by the syslog-ng daemon running as a syslog proxy and a syslog master, that is, the syslog master and the syslog proxy are different instances of the same program. In a single node deployment, the syslog proxy and master are com-pletely redundant.

All services and applications running in OMS can write log entries to the syslog. The syslog entries include information about the service or component that caused the event (for example, backup or ssh).

Syslog stores information on the logged events in the syslog format. A syslog entry consists of two parts:

• The header contains the date and time when the logged event originated, the severity level of the message and the originating service's host name or IP address.

• The syslog message consists of the service name and the message itself.

Example: An example of a syslog entry related to the backup processMar 1 13:40:06 info CLA-0 logger: SYSLOG(fsbackup) Starting backup procedure

Syslog master and syslog proxyThe syslog-ng daemon runs as a syslog proxy and a syslog master. The syslog master runs in the Directory group and uses the OMS's redundant IP address. The syslog master writes to log files in the /var/log/master directory.

The syslog proxy receives log messages from processes and forwards them to the syslog master. Additionally, the syslog proxy writes to log files in the /var/log direc-tory. Syslog proxy uses the node IP address as a source IP address.

Figure 7 Syslog master and syslog proxy

By default, the syslog-ng daemon generates the following files:

Node

Application

/dev/log

/var/logProxy syslog-ng

Master syslog-ng /var/log/master


137

Administering RNC OMS Log management

Id:0900d8058055ee6f

The system also sends logs with severity set to emerg to all user consoles.

The system archives these log files by periodically saving their contents to another file, and recreating the active log file. The logrotate command automates this common practice. The command rotates one or more log files by copying the current contents to a numbered backup version, and then resetting the original file.

Syslog configurationSyslog is configured during commissioning. The configuration file for the syslog proxy is /etc/syslog-ng.conf and the configuration file for the syslog master is /opt/Nokia_BP/etc/syslog-ng.conf.

In the syslog configuration files, you can configure the facility, severity, and destination of the log message. In other words, you can select which log entries are written to syslog, which ones are directed to a separate file or to the user console, and which ones are discarded based on the importance and origin (service/component) of the log entry.

• Facility is what syslog names the various subsystems which generate messages. • Severity is the urgency and importance of a log message. • Destination is where syslog sends the log entry if the filtering rules match. The des-

tination can be, for example, a file, a device, a user, or a host name.

The syslog-ng is able to filter messages based on the contents of messages in addition to the severity/facility pair. Filters perform log routing inside the syslog-ng. A log message needs to match the defined conditions to pass on. Filters have also a uniquely identifying name, so you can refer to filters in your log statements.

You can also combine filters, destinations, and facilities to a log statement. If a log message coming from a listed source matches the filter, it is sent to all listed destina-tions. By default, all matching log statements are processed, therefore a single log message might be sent to the same destination several times, given that destination is listed on several log statements.

Log file Description

syslog Contains all logs with severity set between info and emerg except for the AUTH and AUTHPRIV facilities.

auth.log The authorisation log, which is one of the security logs. The system writes to it information on user actions and accesses to the system.

Contains logs of any severity with the AUTH and AUTHPRIV facilities.

Only the root user can view the auth.log file.

debug Belongs to troubleshooting logs.

Contains all logs with severity set to debug except for the AUTH and AUTHPRIV facilities.

Creating the debug is not enabled by default.

Table 14 The files created by the syslog.ng daemon.



Id:0900d805803d2893

6.2 Starter logsHigh availability services subsystem starts all the processes in recovery units using the Starter program.

By default the Starter does not print any output to a file. This behaviour can be changed by editing the startup script. This is useful when the startup of a process may contain additional important information about the problem under investigation.

The Starter startup script is located at /opt/Nokia_BP/SS_Sta/script/starter.sh.


139


Id:0900d805804de87a

6.3 Viewing syslog contents in OMSPurposeView the contents of the syslog file to help troubleshooting the system or verifying that the system or a certain part of it is working normally.


• you have basic knowledge of Linux operating systems and the platform software and hardware.

• you have root access rights.

Steps


2 View the contents of the syslog file.

• To view the master syslog file using the less command, enter the following command: less /var/log/master-syslog

• To view the proxy syslog file using the less command, enter the following command: less /var/log/syslog

OrSyslog is usually a large file that contains a lot of information, so viewing all the contents of syslog may not be feasible. Use the standard Linux commands to filter the necessary information from syslog.

Example: Searching the syslog for entries related to Diameter load balancerTo search the syslog for Diameter load balancer related log entries using the command grep, search for the text "LBSD" and direct the result to a file. In this example, the file name is output.txt.


grep -i lbsd /var/log/syslog >> output.txt

Example: Searching the last 200 lines in the local syslog for the string "backup"To search the last 200 lines in the syslog proxy for the text "backup" using the command tail, enter the following command:




Id:0900d805804de87a

3 View all the log contents from OMS.

• Log into OMS web page (https://<OMS IP>) with Nemuadmin username and nemuuser password.

• Go to Log Viewer > View Logs. • You can choose a logfile from the View Log File menu.


141


Id:0900d805804d3006

6.4 Enabling Starter logging in OMSSummaryHigh availability services subsystem starts all the processes in recovery units using the Starter program.

By default the Starter does not print any output to a file. This behaviour can be changed by editing the Starter’s startup script. This is useful when the startup of a process contain additional important information about the problem under investigation.

The Starter startup script is located at /opt/Nokia_BP/SS_Sta/script/starter.sh.

Steps


2 Change directory to the Starter subsystem directory.

a) Enter the following command:cd /opt/Nokia_BP/SS_Sta/script

b) Enter the following command for a diskless unit (directory):cd /opt/Nokia_BP_at_Sysimg/SS_Sta/script

c) Make a backup copy of the original script:cp starter.sh starter.sh.orig

d) Edit starter.sh:nano starter.shLocate the line:

./bin/starterReplace the line with:

HOSTNAME=`/bin/hostname` LOG_FILE=/var/log/${HOSTNAME}starter.out /rbin/mv $LOG_FILE $LOG_FILE.prev ./bin/Starter > $LOG_FILE 2>&1

And save the file.e) Reboot the node under investigation:

fshascli -r/<nodename>f) Recreate the problem.g) Send the log files (/logs/<hostname>starter.out) to your Nokia Siemens

Networks representative.h) Copy the backed up original script to starter.sh:

cd /opt/Nokia_BP/SS_Stacp starter.sh.orig starter.sh

i) Reboot the node:fshascli -r/<nodename>



Id:0900d805804fb278

6.5 Configuring log rotation in OMSPurposeConfigure the log file rotation interval, file size, and the amount of the rotated logs.

Before you start

g Editing the wrong data in the configuration files can make the log rotation nonfunc-tional, which can lead to excessive log file sizes. Do not edit the file unless you are certain you know all effects of the modifications.

Check that:

• you have basic knowledge of Linux operating systems and the platform software and hardware

• you have root access rights.

Summary

1. To configure the logrotate interval, edit the cron file to run logrotate on defined intervals.

2. To configure the logrotate file size and/or amount of rotated log files, do the follow-ing: edit the local-syslog.logrotate file in the /etc/logrotate.d directory on the OMS to define the amount of rotated log files and the size of the log file to be rotated for the node-specific syslog. • Edit the local-syslog.logrotate file in the /etc/logrotate.d directory

to define the amount of rotated log files and the size of the log file to be rotated for the syslog proxy.

• Edit the master-syslog.logrotate file in the /etc/logrotate.d direc-tory to define the amount of rotated log files and the size of the log file to be rotated for the syslog master.

Steps


2 Edit the cron file to set the desired interval for rotating log files.Enter the following command:

nano /etc/cron.d/logrotate

Example: Configuring the logrotate to cronIn the following example, the logrotate command runs every four hour 23 minutes past the hour.

23 */4 * * * root /usr/sbin/logrotate /etc/logrotate.conf

For more information on logrotate, see the logrotate manual pages (man logrotate).


143


Id:0900d805804fb278

3 Open the log rotation configuration file for editing.Go to the /etc/logrotate.d directory and open the file you want to edit:

To configure the amount of rotated log files and the size of the log file to be rotated for the syslog proxy, edit the local-syslog.logrotate file.

To configure the amount of rotated log files and the size of the log file to be rotated for the syslog master, edit the master-syslog.logrotate file.

4 Configure the amount of rotated log files and the log file size to be rotated.

g Editing the wrong data in the configuration files can make the log rotation nonfunc-tional, which could lead to excessive log file sizes. Do not edit the files unless you are certain you know all effects of the modifications.

In the logrotate configuration file, edit the following rows:

• To edit the amount of log files, enter the desired number on the rotate row. • To edit the log file size, enter the desired file size on the size row.

Example: Logrotate configuration file for the syslogIn the following example, logrotate keeps 10 files and rotates the log file when its size reaches 10 MB.

rotate 10 compress delaycompress missingok notifempty size 10M sharedscripts postrotate

Expected outcomeThe logrotation is configured without error messages.

Unexpected outcomeLogrotation may fail after the software upgrade if the logrotate configuration contained subdirectories in the node-local /var/log directories. Before executing the log rotation, make sure that all the nodes have the necessary subdirectories for the logs.

VerificationTo check that the changes made to the logrotate configuration file conform to the required format, enter the following command:

logrotate /etc/logrotate.conf

The command should not print any errors.



Id:0900d805804c7567

6.6 Enabling trace logs in OMSPurposeTo enable the writing of trace logs.

SummaryIn OMS trace logs are disabled by default.

Steps

1 Open Application Launcher.

• in the Start menu (Programs > Nokia Siemens Networks > Application Launcher Client 2 > Application Launcher Client) in Windowsor

• in the menu (Applications > Other > Application Launcher Client) in Linux.

2 Launch Parameter Tool.

3 Enable the trace log.Change the value dn:omsParameterId=dwFlags, omsFragmentId=Any, omsFragmentId=TraceConfig, omsFragmentId=System, fsFragmentId=OMS, fsClusterId=ClusterRoot from 0 to 3.

To disable the trace log change the value back to 0.

Expected outcomeTrace log writing is enabled. Directory where you can find the log is /var/log/oms.

g Trace logs consume disk space and CPU time. Disable trace logs when you do not need them.


145


Id:0900d805804a0d15

6.7 OMS syslog is not working properly

DescriptionIf the syslog in OMS is not working properly, it may be caused by one of the following reasons:

• There is not enough disk space in the OMS file system. • The logging processes are not running. • The syslog is not listening to the correct TCP port. • The syslog configuration is incorrect.

Symptoms • No entries are recorded in the OMS syslog.

Recovery procedures

Restarting syslog-ng

Steps

1 Send a SIGHUP to syslog-ng.As a first measure, try restarting the syslog-ng daemon using SIGHUP. Enter the following command:

killall -HUP syslog-ng

Checking the disk space

PurposeCheck that there is enough disk space in the OMS file systems where the syslog files are saved.

Steps


2 Display free disk space.To display the free disk space in the partitions, enter the following command:

df -h

3 Check the disk space on the relevant partitions.

a) To check the disk space for the proxy syslog file, find the free disk space for the local image partition:/var/mnt/local/localimg



Id:0900d805804a0d15

b) To check the disk space for the master syslog file, find the free disk space for the log partition:/var/mnt/local/logIf the partition cannot be found, find the free disk space for the system image partition:/var/mnt/local/sysimg

4 If the disk space is insufficient

Then

Delete unnecessary files and restart syslog-ng daemons.

a) Delete all unnecessary files from the partitions that do not have disk space left. b) To restart syslog-ng daemons, enter the following command:

killall -HUP syslog-ng

Checking that the processes are running

Steps

1 Display the running syslog processes.Enter the following command:

ps ax | grep syslog

In the OMS node, there should be two active syslog processes: syslog master and syslog proxy.

Example: The ps command printout for OMS node with two active syslog processes:

1613 ? Ss 4:51 /opt/Nokia_BP/SS_BPUtils/bin/syslog-ng -p \/var/run/syslog-ng.pid -f /etc/syslog-ng.conf2079 ? Ss 4:49 /opt/Nokia_BP/SS_BPUtils/bin/syslog-ng -F -p \/var/run/master-syslog-ng.pid -f /var/mnt/local/sysimg/flexiserver/opt/Nokia_BP/etc/syslog-ng.conf

2 If the syslog master process is not active

Then

Restart the process.Enter the following command:

fshascli -r /<nodename>/FSDirectoryServer/MasterSyslogDaemon

where <nodename> is the name of the OMS node.

3 If the syslog proxy is missing

Then

Restart syslog the proxy.To restart the syslog-ng proxy, enter the following command:

service syslog-ng restart

If the syslog-ng process does not restart and it is safe to restart the OMS node, restart the node by entering the following command:

fshascli -r <nodename>


147


Id:0900d805804a0d15

Checking that syslog is listening to the right TCP port

Steps

1 Check that the syslog-ng processes are listening to port 601.Enter the following command:

netstat -ltpn |grep syslog-ng

The printout should contain a line stating that the syslog-ng is listening to port 601.

Example: tcp 0 0.0.0.0:* 172.16.25.252:601 LISTEN syslog-ng

Checking that the configurations are correct

Steps

1 View the syslog configuration file and check that the configuration is correct.Check that the syslog configuration is correct, and make the corrections if required. For reference, see the backup copy of your configuration file or other known good example of a configuration file.



Id:0900d8058056e5be

Time management

7 Time management

7.1 Time Management in OMSPurposeTime management is required to guarantee data referentiality and functionality in the network. Time management in OMS uses the NTP, which is an Internet standard and version 3 is defined in RFC-1305.

NTP is a protocol built on top of UDP/IP that assures accurate local timekeeping with reference to radio, atomic or other clocks located on the Internet. This protocol is capable of synchronizing distributed clocks within milliseconds over long time periods.

SummaryOMS has built in automation for setting Daylight Saving Time on or off. Daylight Saving Time can be checked with zdump and grep.

In the figure, the NetAct acts as a NTP server from which the OMS obtains the time. BTS units obtain the time from OMS. The network elements also use an NTP server to obtain the time. In OMS time management software is installed, but it is only used for obtaining the time from NetAct.

The following figure gives an overview of Time Management in the network.

Figure 8 Time management architecture

Steps


OMS

NTP Server& Client

NTPclient

NTP client

NPM NTP server

intranet


149

Administering RNC OMS Time management

Id:0900d8058056e5be

2 Check the Daylight Saving Time settingsEnter the following command (where 2007 is the year):

zdump -v /etc/localtime | grep 2007

Example: Example outputExample output has the Daylight Saving Time settings for year 2007. Daylight saving settings are set in OMS timezone setting, which is set at commissioning or later.

# zdump -v /etc/localtime | grep 2007/etc/localtime Sun Mar 25 00:59:59 2007 UTC = Sun Mar 25 02:59:59 2007 EET isdst=0 gmtoff=7200/etc/localtime Sun Mar 25 01:00:00 2007 UTC = Sun Mar 25 04:00:00 2007 EEST isdst=1 gmtoff=10800/etc/localtime Sun Oct 28 00:59:59 2007 UTC = Sun Oct 28 03:59:59 2007 EEST isdst=1 gmtoff=10800/etc/localtime Sun Oct 28 01:00:00 2007 UTC = Sun Oct 28 03:00:00 2007 EET isdst=0 gmtoff=7200



Id:0900d80580621665

7.2 Configuring NTP services in OMSPurposeOMS NTP can be configured to external NTP. Restarting ClusterNTP service activates the new configuration.

☞ Note that NTP can also be reconfigured by executing zmodifyNetworkSettings script. However using zmodifyNetworkSettings requires also the following data beside the NTP IP address: OMS IP address, subnet mask, gateway address and DNS IP address. Also executing zmodifyNetworkSettings resets system in order to apply new settings.

Note that OMS can be configured for several NTP servers.

Before you startYou need root access to modify NTP settings and to restart NTP service.

Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase.For more information, see Logging in to OMS in Installing and commissioning RNC OMS.

2 Switch to root by executing su - command.

3 Check the current NTP server IP address.Run the following command:

ntpq –c pe

4 Modify the NTP server IP address. You can modify the OMS NTP IP configuration either by removing or adding the NTP servers. You can perform one of the following:

• Add the NTP server IP address:fsipnet service modify ClusterNTP addforwarder <ntp_ip_address>

• Remove the NTP server IP address:fsipnet service modify ClusterNTP delforwarder <ntp_ip_address>

• Remove whole ClusterNTP:fsipnet service delete ClusterNTP

• Add service:fsipnet service add ClusterNTP forwarder <ntp_ip_address> [forwarder <2nd_ntp_ip_address>] [forwarder <3rd_ntp_ip_address>]


151


Id:0900d80580621665

Example: To configure OMS to use NTP server which IP address is 10.0.0.1 run the following command:

fsipnet service modify ClusterNTP addforwarder 10.0.0.1

5 Update changes.fsipreconfigure commit

6 Update HAS parameters.fshascli -C

7 Restart the ClusterNTP service.Run the following command for restarting ClusterNTP service.

fshascli -r /ClusterNTP

After executing this command, the new configuration is in use.

8 Check that primary NTP server is configured correctly. Enter the following command:

ntpq -c pe



Id:0900d80580474c92

7.3 Testing NTPPurposeNTP operation can be tested with ntpq command. It opens console, which can query current NTP status. This can be used to test how OMS synchronises its time from NetAct.

Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase, and then change the root user permission by entering su - command.For more information, see Logging in to OMS in Installing and Commissioning RNC OMSLogging in to OMS.

2 Start ntpqEnter the following command:

ntpq

3 Query NTP severs with pe commandEnter the following command in ntpq:

pe

Example: Example outputExample output has the Daylight Saving Time settings for year 2007. Daylight saving settings are set in OMS timezone setting, which is set at commissioning or later.

ntpq> pe remote refid st t when poll reach delay offset jitter==============================================================================*10.8.122.67 LOCAL(0) 6 u 183 512 377 0.305 0.023 0.025 LOCAL(0) .INIT. 0 l 23 64 377 0.000 0.000 0.002

4 Exit ntpq with quit command


153


Id:0900d8058049cf8c

7.4 Changing time zone settings in OMSPurposeTo change the time zone setting in OMS for example after installation or an upgrade.

Before you startYou need root access to change the time zone setting in OMS.

SummaryUse the zchangetimezone script to change the time zone settings.

Steps

1 Run the script.Enter:

zchangetimezone

2 Enter the continent or ocean information when prompted.Enter the respective number of the continent or ocean that matches your location.

Note that option number 11 (Posix TZ format) is not supported and should not be used.

Example: # zchangetimezone Please identify a location so that time zone rules can be set correctly.Please select a continent or ocean. 1) Africa 2) Americas 3) Antarctica 4) Arctic Ocean 5) Asia 6) Atlantic Ocean 7) Australia 8) Europe 9) Indian Ocean10) Pacific Ocean11) none - I want to specify the time zone using the Posix TZ format.#? 8

3 Select country.Enter the respective number of the country that matches your location.

Example: Please select a country. 1) Aaland Islands 18) Greece 35) Norway 2) Albania 19) Guernsey 36) Poland 3) Andorra 20) Hungary 37) Portugal 4) Austria 21) Ireland 38) Romania 5) Belarus 22) Isle of Man 39) Russia 6) Belgium 23) Italy 40) San Marino 7) Bosnia & Herzegovina 24) Jersey 41) Serbia



Id:0900d8058049cf8c

8) Britain (UK) 25) Latvia 42) Slovakia 9) Bulgaria 26) Liechtenstein 43) Slovenia10) Croatia 27) Lithuania 44) Spain11) Czech Republic 28) Luxembourg 45) Sweden12) Denmark 29) Macedonia 46) Switzerland13) Estonia 30) Malta 47) Turkey14) Finland 31) Moldova 48) Ukraine15) France 32) Monaco 49) Vatican City16) Germany 33) Montenegro17) Gibraltar 34) Netherlands#? 14

4 Verify settings and restart.Verify time zone and proceed to restart if information is correct.

Example: The following information has been given:

Finland

Therefore TZ='Europe/Helsinki' will be used.Local time is now: Tue Oct 16 14:41:20 EEST 2007.Universal Time is now: Tue Oct 16 11:41:20 UTC 2007.Is the above information OK?1) Yes2) No#? 1

You can make this change permanent for yourself by appending the line TZ='Europe/Helsinki'; export TZto the file '.profile' in your home directory; then log out and log in again.

Here is that TZ value again, this time on standard output so that youcan use the /usr/bin/tzselect command in shell scripts:

Timezone: Europe/Helsinki set to OMSSystem restart needed...

/ is the cluster. The request will restart all the nodes in it.Are you sure you want to proceed? [y/n] y

Expected outcomeThe time zone settings are changed.


155

Administering RNC OMS Fault management

Id:0900d805804115f9

8 Fault managementFault management in RNC OMS consists of a set of functions to detect and correct fault situations in the system. The Fault Management GUI can be used for monitoring fault situations, alarm cancellation and alarm parameters change.

For more information, see Fault Management in RNC OMS.



Id:0900d805802bdc91

Security

9 SecurityFor more information, see document Security in RNC OMS.


157

Administering RNC OMS Maintenance

Id:0900d80580565bd6

10 Maintenance

10.1 MaintenanceMaintenance tasks cover general principles of use as well as daily, weekly and monthly activities to maintain the system health. Many routine tasks can be automated by con-figuring a cron daemon to execute scheduled commands.

File system healthIn general, in order to prevent data corruption on disks, the operator should not reset OMS or any hardware equipment or shut down the live system in an uncontrolled manner.

The file systems cannot be mounted manually in a live system. If the same file system is mounted accidentally twice, the file system gets corrupted.

All the platform (and most of the cluster) file systems run on disk partitions that are mirrored by either redundant array of independent disks (RAID) level 1 or distributed replicated block device (DRBD) in network-based solution. In case, such a file system must be mounted manually in a live system, ensure that you start the underlying lower-level device prior to mounting the file system in it. If you do not start the lower-level device, the mirroring is not done and the RAID or DRBD eventually becomes corrupted.

It is recommended that the file system disk usage is regularly monitored. For more infor-mation, see Monitoring file system usage.

Note that many reading errors on a disk may indicate that the physical disk is getting faulty and should be replaced with a new component. For further instructions, see hardware maintenance documentation.

Alarm monitoringDaily alarm monitoring is essential for system health maintenance. The alarm system indicates potential faults in the system as well as faults that require corrective actions. After an alarm is raised, the fault causing the alarm must be solved. The solution can be an automatic recovery or a manual corrective action. For a potential fault, the solution consists of preventive actions. For more information, see fault management documen-tation.

Scheduled tasks (cron)The daily, weekly or monthly routine maintenance tasks can be automated in the cluster using the cron daemon, which is a service that executes commands at specific dates and times. For a command that has to be executed repeatedly for certain purpose, create a specific crontab file.

Cron tasks can be used, for example, for scheduling regular backups.

Log rotation (syslog archiving)The traditional solution for managing Linux log files involves periodically saving their contents to another file, and recreating the active log file. Several old log files can be saved on the system and given names consisting of the original file name with a numeric extension: syslog.0, syslog.1 and so on. The logrotate command automates this common practice. The command rotates one or more log files by copying the current contents to a numbered backup version, and then resetting the original file.



Id:0900d80580565bd6

Maintenance

The logrotate feature controls that a log file size does not increase too much. It reg-ularly copies the existing log file into a backup file, stores it in the defined archive, and creates a new log file in specified intervals, for example, daily, weekly, or monthly. You can configure logrotate, for example, to keep a certain amount of backup files or compressing the old backup files.

The logrotate feature can also rotate logs based upon their size, saving and truncat-ing them only when they exceed a preset limit. For more information, see Configuring log rotation in OMS.

Gathering cluster dataYou can utilise the troubleshooting tools for gathering basic data of your network element when you suspect any particular malfunction.

The platform provides an environment camera script for collecting some basic data. The fsenvcam.sh script is located in the /opt/Nokia_BP/SS_SysReport/envcam directory.

It has no command line arguments. By default, the script prints the collected information into a standard output (usually a console or a terminal session). The output can also be easily redirected into a file for off-site storage or for forwarding to your Nokia Siemens Networks representative.

Command line utilities like fshascli, hwcli and top, among others, can also be used to gather information.

Monitor regularly also the syslog for possible CRITICAL or MAJOR faults (collected into active CLA in /var/log/master-syslog).


159


Id:0900d805804cd11f

10.2 Scheduling backups in OMSPurposeTo schedule the backup to take place automatically, for example, at night time.


• you have the necessary root privileges.

SummaryIt is recommended that you make a partial backup every night, and a full backup before and after a software upgrade.

Backup scheduling is done using a cron daemon that executes the scheduled com-mands. The crontab (cron table) file contains the schedule of the tasks to be run on specified times.

The crontab file includes the execution time, date, and information on the command to be executed. Each line in the crontab file has five fields for time and date in the fol-lowing order:

• minute (0-59) • hour (0-23) • day (1-31) • month (1-12) • day of week (0-7, or the name of the day, for example, Sun).

In the time and date fields it is possible to use the asterisk (*), which always stands for "first-last". Number ranges and lists are also allowed. (For more information, see the crontab manual pages.)

After you have added the date and time information, add information on the command to be executed.

Steps

1 Log in to the MCP18-B OMS remotely as the _nokfsoperator user and password assigned during installation phase, and then change the root user permission by entering su - command.For more information, see Logging in to OMS in Installing and commissioning RNC OMS.

2 Add the scheduled backup to the crontab file.

☞ If you change the active sotware set, check that there is a proper crontab file in the new active software set.

To add the minute, hour, day, month, day of the week, and the backup command to be executed to the crontab file, enter the following command:

vi /etc/crontab



Id:0900d805804cd11f

Example: Adding scheduled backups to cronBelow is an example of part of a crontab file which defines that a partial backup is taken every night from Monday to Saturday at 23:45 and a full backup on Sundays at 23:45 by the user root.

# Execute partial backup every night from monday to saturday 23:45:45 23 * * 1-6 root su - -c "/opt/Nokia_BP/bin/fsbackup --partial --quiet"

# Execute full backup every sunday night at 23:4545 23 * * Sun root su - -c "/opt/Nokia_BP/bin/fsbackup --full --quiet"

3 Restart the cron daemon.Enter the following command:

/etc/init.d/crond reload


161


Id:0900d805804ee205

10.3 Monitoring file system usagePurposeOperator should regularly monitor that the file systems do not become full. There is a risk that the system starts discarding important data when the file system becomes full. To warn about this condition, the system raises alarm 70158 FILE SYSTEM USAGE OVER LIMIT when any file system (on a disk partition or on logical volume) is more than 90% full. The alarm is cleared, when the full ratio decreases below 80%. Note that virtual and network file systems (NFS) are not monitored, thus the alarm is not raised, if these mount points get full.

Note that you can modify the osmon configuration file to raise the alarm for certain mount points either earlier or later than when the disks are 90% full.

You can also have to delete unnecessary files from the file systems if the file system starts reaching its 90% limit. Typically, there are unnecessary files in /tmp, /var/tmp, /var/log and in /var/crash directories.

Steps

1 Monitoring file system usage

Steps

a Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase, and then change the root user permission by entering su - command.For more information, see Logging in to OMS in Installing and commissioning RNC OMS.

b Monitor the file system fill ratio.Enter the following command:

df –h

The command shows all the partitions that have been taken into use. The command shows each directory fill ratio in kilo- (K), mega- (M), or gigabytes (G).

Example: An example printout of the df – h command:root@CLA-0:~# df -hFilesystem Size Used Avail Use% Mounted onnone 2.0G 208K 2.0G 1% /dev/dev/VG_62/localimg 4.9G 1.1G 3.6G 24% /var/mnt/local/localimgdirectory:/var/mnt/local/sysimg 29G 4.3G 24G 16% /var/mnt/remote/sysimgdirectory:/var/mnt/local/log 9.7G 144M 9.0G 2% /var/mnt/remote/log

!

Deleting files from the mount points may damage the system and lead to a system outage. Make sure that you know and understand the file system and directory hierarchy of a network element before deleting files.



Id:0900d805804ee205

/dev/drbd0 29G 4.3G 24G 16% /var/mnt/local/sysimg/dev/drbd2 9.7G 144M 9.0G 2% /var/mnt/local/log/dev/VG_62/backup 20G 77M 19G 1% /var/mnt/local/backup

c Check the content and size of each directory (mount point).Enter the following command:

ls -lah

Example: An example printout of the ls – lah command:root@CLA-0:~# ls -lah /var/mnt/remotetotal 4.0Kdrwxr-xr-x 4 root root 0 Nov 3 14:59 .drwxr-xr-x 5 root root 0 Nov 3 14:59 ..drwxr-xr-x 2 root root 0 Nov 3 14:59 logdrwxr-xr-x 4 root root 4.0K Sep 21 07:22 sysimg

To receive a short summary of the size of a directory you can also enter the following command:

du –h <path to directory> | sort -gr

Example: An example printout of the above command:root@CLA-0:~# du -h /var/lib/ |sort -gr368K /var/lib/300K /var/lib/pcp296K /var/lib/pcp/pmdas240K /var/lib/pcp/pmdas/linux52K /var/lib/pcp/pmdas/pmcd20K /var/lib/dhcp8.0K /var/lib/ntp8.0K /var/lib/alternatives4.0K /var/lib/slocate4.0K /var/lib/misc

The advantage of using this command option is that you can verify how much space the subdirectories occupy within a directory.

2 Configuring the master osmon.conf file to change alarm limit

SummaryYou can find the master configuration file osmon.conf in the /opt/Nokia_BP/etc directory. The currently used software set can be checked by executing the currentset command.

!

Deleting files from the mount points may damage the system and lead to a system outage. Make sure that you have knowledge about the file system and directory hierar-chy of a network element before deleting files.


163


Id:0900d805804ee205

Steps

a Log in to the MCP18-B OMS remotely as the _nokfsoperator using password assigned during installation phase, and then change the root user permission by entering su - command.For more information, see Logging in to OMS in Installing and commissioning RNC OMS.

b Edit the osmon.conf file.To modify the master configuration file in CLA node, open the file in text editor, for example nano, by entering the following command:

nano /opt/Nokia_BP/etc/osmon.conf

To modify the master configuration files in nodes other than CLA, open the file in a text editor, for example nano, by entering the following commands:

ssh directory nano /var/mnt/local/sysimg/flexiserver/sets/<software set in use>/opt/ Nokia_BP/etc/osmon.conf

c Modify parameter FILESYSTEM LIMITS for changing alarm limit for all mount points (directories).

FILESYSTEM LIMITS 70.0 80.0

The values above mean that an alarm is raised if any of the mount points is 80% full, and the alarm is cleared when this mount point is reduced to below 70% full.

Note that RAM and MySQL directories may be 90% full in the normal state. Thus, if the alarm limit values are decreased, the alarm will be raised for these mount points.

d Enable the new alarm limit values for the mount point.Enter the following command separately in all nodes where the configuration file has been modified:

killall –HUP osmon

3 Overriding the master configuration settings in a certain node

SummaryThe settings in the master configuration file can be overridden in a certain node with a node specific configuration file osmon_node.conf in the /etc directory. Currently the IP director nodes must use the node specific files.

Steps

a Log in to the MCP18-B OMS remotely as the _nokfsoperator user and password assigned during installation phase, and then change the root user permission by entering su - command.For more information, see Logging in to OMS in Installing and commissioning RNC OMS.



Id:0900d805804ee205

b Copy the master configuration file as a template for the node-specific file.Copy the master configuration file osmon.conf in the /opt/Nokia_BP/etc direc-tory as osmon_node.conf file in the /etc and /etc_ondisk directories. Enter the following commands:

cp /opt/Nokia_BP/etc/osmon.conf /etc/osmon_node.conf cp /opt/Nokia_BP/etc/osmon.conf /etc_ondisk/osmon_node.conf

c Edit the node-specific files.Open the file in text editor, for example nano, by entering the following command:

nano <path to directory>/osmon_node.conf

Note that you need to make the changes to the both osmon_node.conf files.




165


Id:0900d8058049f76e

10.4 Monitoring memory usagePurposeFor system maintenance, it is also recommended, that the operator monitors the memory consumption in the system regularly. When OMS has used all the memory it will eventually crash. To avoid this, operator should check OMS regularly.

Steps

1 Checking memory usage

Steps


b Check the memory usage on a holistic level as a snapshot.Enter the following command:

free

Example: An example printout:free total used free shared buffers cachedMem: 4154620 1645728 2508892 0 302012 455108-/+ buffers/cache: 888608 3266012Swap: 0 0 0

In the example above you can compare the kilobytes from columnfree with the kilo-bytes from columnused in the second row of the printout (starting with -/+ buffers/cache:). The values presented in this row also take the cached and buffered memories into account. If the value of free kilobytes is getting close to zero, it is a clear indication that the system is running out of memory for some reason. Note that also the amount of buffered and cached memories are counted as free memory.

c Check additional information from /proc/meminfo directory. To see the detailed information on memory usage, enter the following command:

cat /proc/meminfo

Example: An example printout:MemTotal: 4154620 kBMemFree: 2509084 kBBuffers: 302012 kBCached: 454848 kBSwapCached: 0 kBActive: 1092932 kBInactive: 172844 kB



Id:0900d8058049f76e

HighTotal: 3276800 kBHighFree: 2302592 kBLowTotal: 877820 kBLowFree: 206492 kBSwapTotal: 0 kBSwapFree: 0 kBDirty: 280 kBWriteback: 0 kBMapped: 551272 kBSlab: 355120 kBCommitLimit: 2077308 kBCommitted_AS: 3729984 kBPageTables: 5704 kBVmallocTotal: 106488 kBVmallocUsed: 2904 kBVmallocChunk: 103112 kBHugePages_Total: 0HugePages_Free: 0Hugepagesize: 2048 kB

Note that the amount of buffered and cached memories are counted as free memory. So, when counting the total amount of free memory, the MemFree, Buffers and Cached values must be taken into account. So, in the example above, the total amount of free memory is 3265944 kilobytes.

2 Checking memory usage on a process level

Steps


b Check the memory usage on a process level.Enter the following command:

top

Example: An example printout of summary area:top - 12:40:37 up 4 days, 21:40, 4 users, load average: 0.62, 0.80, 0.88Tasks: 158 total, 2 running, 156 sleeping, 0 stopped, 0 zombieCpu(s): 0.5% us, 1.1% sy, 0.0% ni, 96.4% id, 2.0% wa, 0.0% hi, 0.0% siMem: 4154620k total, 1645584k used, 2509036k free, 302012k buffersSwap: 0k total, 0k used, 0k free, 454848k cached[...]

Then press M to sort the processes based on their memory resident size.


167


Id:0900d8058049f76e

Example: An example printout of task area: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6573 root -21 0 63248 30m 7688 S 0 0.8 0:00.65 Starter 7049 root -21 -15 48320 16m 9.9m S 0 0.4 0:26.68 HASNodeAgent 7039 root 23 0 98348 12m 3852 S 0 0.3 1:13.46 slapd 7048 root -21 0 41032 11m 9044 S 0 0.3 0:00.36 ApplicationSupe 7840 root 22 0 73780 10m 4100 S 0 0.3 0:03.07 slapd13553 root 16 0 69348 8076 3076 S 0 0.2 0:00.34 hwmd 8041 ntp 16 0 27912 7852 6296 S 0 0.2 0:00.00 ntpd 7867 root 16 0 29096 4968 1892 S 0 0.1 0:05.97 syslog-ng 7888 root 16 0 35336 4212 3492 S 0 0.1 0:02.31 PmgmtEventServe 6449 root 16 0 7640 4092 464 S 0 0.1 0:01.26 dhcpd13229 root 16 0 51148 3952 3072 S 0 0.1 0:00.60 sshd [...]

The summary area there presents general information on the resource usage. For example, the number of processes running and the total amount of memory in use are presented in the summary area.

In the task area the resource usage is presented process by process. Descriptions of the columns are presented below.

• PID: The ID number of the task. • USER: The effective user name that is the owner of the task. • PR: The priority of the task • NI: So called nice value of the task. Zero in this field means that no priority will

be adjusted in determining the task's dispatchability. The negative value means higher priority, whereas the positive value means lower priority.

• VIRT: The amount of virtual memory used by the task. • RES: The amount of non-swapped physical memory used by the task • SHR: The amount of shared memory used by the task. • S: The status of the task. The value can be one of the following: D (uninterrupt-

ible sleep), R (running), S (sleeping), T (traced or stopped) or Z (zombie). • %CPU: The task's share of the elapsed processor time of the total processor

time. • %MEM: A task's currently used share of available physical memory. • TIME+: The total processor time the task has used since it was started. • COMMAND: The name of the task.

3 Configuring the master osmon.conf file to change the alarm limit

SummaryAll nodes have the master configuration file osmon.conf in the /opt/Nokia_BP/etc directory. In the CLA nodes, the settings can be modified by editing this file. In nodes other than CLA nodes, the settings can be modified by logging into the CLA node, where Directory service is running and editing the osmon.conf file in the /var/mnt/local/sysimg/flexiserver/sets/<software set in use>/opt/Nokia_BP/etc directory. The currently used software set can be checked by executing the currentset command.



Id:0900d8058049f76e

Note that the settings in the master configuration file can be overridden in certain nodes with a node-specific configuration file osmon_node.conf in the /etc directory. For more information, see Overriding the master configuration settings in a certain node.

Steps

a Log in to the OMS remotely as the _nokfsoperator user and password assigned during installation phase, and then change the root user permission by entering su - command.For more information, see Logging in to OMS in Installing and commissioning RNC OMS.

ssh <OMS IP>

b Edit the osmon.conf file.To modify the master configuration file in CLA node, open the file in a text editor, for example vi, by entering the following command:

vi /opt/Nokia_BP/etc/osmon.conf

To modify the master configuration files in other than CLA nodes, open the file in a text editor, for example vi, by entering the following commands:

ssh directoryvi /var/mnt/local/sysimg/flexiserver/sets/<software set in use>\/opt/Nokia_BP/etc/osmon.conf

c Modify parameter FILESYSTEM LIMITS for changing alarm limit for all mount points (directories).

MEM LIMITS 70.0 80.0

The values above mean that an alarm is raised if the memory usage ratio is 80%, and the alarm is cleared when the memory usage ratio decreases below 70%.




169


Id:0900d80580349b70

10.5 Maintenance checklistThe following table summarises the routine maintenance tasks of major importance to be carried out in a cluster on a regular basis.

Task When

Monitoring alarms Daily

Taking partial backups Daily (create a cron task)

Taking a full backup Before software upgrade

Updating certificate keys At specified time before the current keys become invalid

Monitoring file system usage Daily (or configure osmon.conf to raise alarm earlier)

Monitoring memory usage Weekly

Monitoring TimesTen mount points Daily (or configure osmon.conf to raise alarm earlier)

Monitoring TimesTen database fill ratio Daily (by monitoring alarms)

Monitoring MySQL mount points Daily (by monitoring alarms)

Checking for the latest embedded SW in HW

Monthly

Table 15 Routine maintenance tasks

Onm Adm Administeringrnc50oms

Documents

Transcript of Onm Adm Administeringrnc50oms