Uniformance® Troubleshooting and Optimization - … · Uniformance® Troubleshooting and...

Uniformance® Troubleshooting and Optimization

Larry M. Carr Senior Staff Engineer Engineering/Process Control Systems ExxonMobil Chemical Company

Uniformance® Troubleshooting and Optimization 2

Table of Contents

Introduction ................................................................................................................................................................................ 3

Background ................................................................................................................................................................................ 3

Technology Overview ................................................................................................................................................................ 3

Tools and Utilities ...................................................................................................................................................................... 4

Conclusion ............................................................................................................................................................................... 18

Table of Figures

ExxonMobil's Baytown refinery and petrochemical complex in Baytown, Texas ................................................................ 3 Figure 1 ....................................................................................................................................................................................... 5 Figure 2 ....................................................................................................................................................................................... 6 Figure 3 ....................................................................................................................................................................................... 6 Figure 4 ....................................................................................................................................................................................... 7 Figure 5 ....................................................................................................................................................................................... 7 Figure 6 ....................................................................................................................................................................................... 8 Figure 7 ....................................................................................................................................................................................... 8 Figure 8 ....................................................................................................................................................................................... 9 Figure 9 ..................................................................................................................................................................................... 10 Figure 10 ................................................................................................................................................................................... 10 Figure 11 ................................................................................................................................................................................... 11 Figure 12 ................................................................................................................................................................................... 11 Figure 13 ................................................................................................................................................................................... 12 Figure 14 ................................................................................................................................................................................... 12 Figure 15 ................................................................................................................................................................................... 13 Figure 16 ................................................................................................................................................................................... 13 Figure 17 ................................................................................................................................................................................... 14 Figure 18 ................................................................................................................................................................................... 14 Figure 19 ................................................................................................................................................................................... 15 Figure 20 ................................................................................................................................................................................... 15 Figure 21 ................................................................................................................................................................................... 16 Figure 22 ................................................................................................................................................................................... 16 Figure 23 ................................................................................................................................................................................... 16


Introduction

ExxonMobil Chemical Company, a division of Exxon Mobil Corporation, is one of the largest petrochemical companies in the world. ExxonMobil Chemical manufactures and markets various petrochemical products, including olefins, aromatics, synthetic rubber, polyethylene, polypropylene, plasticizers, synthetic lubricant basestocks and many others.

At approximately 60 ExxonMobil chemical plants and refineries worldwide, Honeywell’s Uniformance® PHD solution is the standard for plant information management. PHD’s collector/shadow architecture allows data to be gathered from disparate control systems and other data sources into a unified database for long-term storage.

ExxonMobil's Baytown refinery and petrochemical complex in Baytown, Texas, is home to the largest oil refinery in the U.S. with an

input capacity of 573,000 barrels per day. The chemical plant manufactures more than 7.2 billion pounds of products per year.

This white paper describes how to use various tools and utilities that may be unfamiliar to some Uniformance users. Proper application of these tools will enable optimal implementation of robust PHD capabilities.

Background

Today’s business environment is the same everywhere: “Do more with fewer people and less money, and find tools that allow you to be more efficient.” Industrial facilities must cope with increased global competition which, coupled with significant manpower reductions, limits the engineering resources needed to deploy and maintain advanced technologies for process automation and plant information management.

Like other process industry end-users, ExxonMobil seeks accurate and comprehensive information supporting the right operating decisions at its refineries and petrochemical facilities. Timely data on production processes, storage sites, terminals and other critical infrastructure empowers plant personnel to better align, plan, execute and improve operational strategies. This results in improved bottom-line business results.

Technology Overview

Honeywell’s Uniformance PHD is an advanced plant information management system designed to collect, store and replay historical and continuous process data. The PHD solution allows integration of existing control systems and applications across multiple plants and sites. Its seamless data interfaces, data collection fail-over and automatic history recovery ensure that reliable, long-term history is available for end-user applications.


Uniformance Process Studio (UPS) provides end-users with desktop tools for transforming data into knowledge. PHD calculated tags allow users to apply engineering and business knowledge to current and historical data, while built-in engineering unit conversions present data in a format that makes sense to the end-user. Process data may be combined with business data to provide a view of the entire operation.

PHD also provides the security and reliability to ensure data is available when it is needed. Robust data collection enables fail-over to a secondary collector upon loss of a primary collector. In addition, the Uniformance System Monitor ensures that support personnel are notified when abnormal events are detected, and can also automate corrective action if desired.

Tools and Utilities

The Uniformance solution includes a number of standard, but relatively unknown tools, which help optimize PHD performance. Utility programs written by Honeywell’s Uniformance development team in Calgary, Alberta, Canada, to troubleshoot issues for PHD users, as well as problems in its own development lab, evolved into a comprehensive, easy-to-use diagnostic package provided to customers at no additional charge.

The various tools and utilities available in this package include:

Diagnostic Utility: Like other users with a very large Uniformance installed base, ExxonMobil has configured all its PHD servers with standard procedures and scripts, as well as a common directory structure. One component of the standard installation adopted by ExxonMobil is the installation of the powerful Honeywell diagnostic utility. This utility saves a snapshot of all pertinent Uniformance and system configuration data into .UDG files that can be analyzed locally or sent to a central support organization or Honeywell Technical Assistance Center (TAC) for analysis when problems arise.

Once Uniformance is installed at a site, and its basic functionality has been verified, the diagnostic tool is run to establish a baseline for each PHD server. The output .UDG file is compared to the enterprise standard file established for a specific release/patch level. This makes it easy to determine whether the installation was done properly, and if all current patches have been applied.

At ExxonMobil, the Uniformance diagnostic utility is run as part of the normal PHD backup routine and includes the server name, date and time in the .UDG file names. Then, if PHD problems arise, the current configuration can be compared to the baseline .UDG file or the file created during a previous backup to determine if system or configuration changes were the root cause.

To compare .UDG files, enter the string from a CMD prompt as shown: <path>\HwlDiag.exe @compare /phdeng

When the screen in Figure 1 appears, click the “First File” button and select the first file you want to use in your comparison from the browse selection box. Do the same for the second file in the comparison. After clicking “Compare,” you will be prompted with a checklist of items for comparison.


Figure 1

You may select all tabs, or if you de-select “Select All,” the individual boxes can be selected and de-selected.

After making your desired selections, click “OK.” Once the program has completed, click the “Comparison Results” tab at the top (see Fig. 2).


Figure 2

In this example, the R210.1.4 standard UDC file was compared to the R210.1.5 standard. The program checks many items in the General category such as Peak Memory, Process Count, etc. It also verifies free space on the drives.

Some differences should be expected in many of the categories, but in this case, we were most concerned with differences in the version numbers and modified dates of the executable files (e.g., .exe and .dll). See Fig. 3.

Figure 3

Uniformance-related changes that have taken place in the registry will also be viewable on the report, but none are shown in the example above.

In the example in Figure 4, a particular ExxonMobil site was having problems with robust data collection. One of the active APP collectors had issues with “time outs,” which were causing it to fail-over to the standby. Then, the standby would detect that the active collector was available and return control to it. The log files showed a lot of flip-flopping between the active and standby servers.

Upon analysis of the .UDG file, an engineer checked the TCP Statistics from the Network tab and noticed that the re-transmission rate was about 50 percent. This compares with a normal re-transmission rate of < 1 percent. Once the Ethernet cable was replaced, the problem was corrected.


Figure 4

In a separate case, the Oracle directories were inserted at the front of the path environment variable during the Oracle installation. It was discovered that an old .dll in one of the Oracle directories had overridden a newer version of that .dll from the %SystemRoot% directory, which subsequently caused PHD to fail. This problem was corrected by moving the %SystemRoot% directory to the beginning of the path (see Fig. 5).

Figure 5

Another ExxonMobil facility reported it had a failure on an LCNP board on the active PHD APP collector, but it did not automatically fail-over to the standby collector. Figure 6 shows the Uniformance registry settings on the APP node. The failure was not detected because the DetectLxSCommFails key was missing from the registry. This entry was added and the problem was solved.


Figure 6

As indicated by the tabs across the top of the form, there are many different functions of the Honeywell diagnostics utility. One of the most useful is the Uniformance tab, which provides a snapshot of all Uniformance-related data on a given server. By expanding the browse tree on the left side of the screen, the user can view items such as files in the NTSite folder and their contents.

In Figure 7, PHD parameters are visible in the same way as if the user had gone to the PHD server console, brought up PHDMAN and entered “SHOW PARAM.”

Figure 7

With greater familiarity using the PHD diagnostic utility, plant personnel will find several useful functions that can assist in troubleshooting issues with PHD, as well as functions that can be used on all servers such as the Network Diagnostics, DCOM, OS Information, Environment, Services, etc.


To invoke other helpful Hwldiag functions, add one or more of these switches to the command line or shortcut, which invokes Hwldiag:

• /phdeng - set to Local PHD Admin view

• /f<filename> - saves directly to file

• /l<filename> - loads the specified filename when started

• /noui /f<filename> - does not create any GUI and saves directly to file

• /du - ignore collecting Uniformance info

• @tagdef - gathers all active tag definitions from PHD and writes the data to a CSV file

• @sectest - security validation utility, acts the same as PHDServer when under INTS mode

The first switch included on the PHD desktop shortcut, “/phdeng,” shows the local PHD admin view in the utility’s Graphical User Interface (GUI). By running the tool from a CMD window, the user can string the “slash” switches together (for example, use the “/noui /f<filename>” switch when running the program from the Windows Scheduler).

The “/du” switch allows diagnostics to be deployed on servers other than Uniformance.

In addition, the “@tagdef” switch is very helpful for exporting tag definitions to create a test system or for changing certain attributes in a large number of tags and re-importing them to PHD.

The “@sectest” function also can be employed when troubleshooting security issues to determine how the DBSecurity server handled the accounts in question.

RDI Telnet Interface: Uniformance PHD provides a Telnet interface to the RDI server, allowing users to monitor and report information about RDIs, and to the API server to identify connections and report statistics.

The RDI server interface is started from a CMD window by typing “Telnet localhost 4101” (4101 is the default port number used to access the RDI server). A prompt requesting the user name and password will then appear.

Users should be aware that a bug in the Telnet function in Windows Server 2003 echoes all information entered on the keyboard including passwords. (see Fig. 8).

Figure 8


From the RDIServer> prompt, enter “MON RDCHR” to show the current state of all RDIs as it pertains to history recovery.

The example shown in Figure 9 indicates which RDIs are in recovery, the current tag number being recovered and the % complete for each RDI. This display provides RDC information for all RDIs on a single screen, which is more user-friendly and efficient than monitoring RDC for each interface with the PHDMAN command: MON RDC <RDI_NAME>.

The information displayed is only valid for the server to which Telnet is connected.

The example in Figure 9 shows the history recovery status of the RDIs on the shadow server. Users will not see the statistics for any history recovery taking place between the primary and secondary collector nodes unless they initiate a telnet session to the rdiserver service on those nodes.

Figure 9

Ctl-C returns the user back to the rdiserver> prompt.

Entering “Help” at the RDIServer> prompt provides a list of available commands (see Fig. 10).

Figure 10

The Dump RDI command displays statistics on each RDI since the last time they were cleared. Statistics such as worst scan time and best scan time help in identifying RDI efficiency and can assist in load-balancing. Scan times can be optimized by moving some tags to different or new RDIs, or by changing the offsets in order to use the channels to the LCN more efficiently.


After changes are made, enter the CLEAR <rdi_name> (or CLEAR ALL) command to reset the statistics. The system should run for a period of time before dumping the statistics again to see if any gains in efficiency were obtained (see Fig. 11).

Figure 11

API Telnet Interface: The API server also has a telnet interface, and its logon is exactly the same as the RDI server except for the port number (the default port number is 3101 for the API server). As with the RDI server interface, entering “Help” at the APIServer> prompt will display all of the available functions as shown in Figure 12.

Figure 12

The display from the Session command in Figure 13 shows five concurrent API server connections. At one ExxonMobil site, the API server reached its 100-connection limit and Uniformance users began to encounter errors when they attempted to start a new API server process (such as PHDMAN or Modtag). Without knowing the root cause of the problem, the system engineer at the site had no choice but to reboot the server. However, once she was instructed on how to use the telnet tool, she was able to see all of the connections to the API server and identify the process that was allocating API connections and not releasing them back to the system. Also, through the telnet interface, as a temporary alternative to rebooting, she was able to kill the problem sessions and free up more API connections.


Figure 13

Additional information regarding a particular session can be obtained by entering SESS <session #>. This provides a wealth of valuable security token data that is also helpful for troubleshooting (see Fig. 14).

Figure 14

Parameter Settings: Some Uniformance users have found that various parameter and registry settings, as well as PHDMAN reporting functions, can have a significant impact on the performance and efficiency of PHD. This discussion is limited to parameters having the greatest impact on memory usage.

Until 64-Bit R300 systems become the norm, Honeywell customers are limited to 2 GB memory for PHD (or 3 GB if using the /3 GB switch in the boot.ini file, which can have other implications depending on the memory consumption of applications running on the box). As such, it is important to know the way that PHD allocates memory. The following formulas may not be 100 percent accurate, but will be very close when estimating memory consumption during tag collection.

PHD memory usage calculation:

Each value takes 11 bytes + Data Length of tag (11+DL) So for type F4 (11+4 =15), I4 (15), D8 (19), L8 (19) C20 (31) U20 (51)

Avgbytes = ((#F4*15)+(#I4*15)+(#D8*19)+(#L8*19)+(#C20*31)+(#U20*51)) / #tags


RAW memory = (Avgbytes * MAX_TAGS *3) DATA memory = (Avgbytes * values per min * MAX_TAGS * DEF_QMINUTE * 110%)

TOTAL memory = RAW memory + DATA memory

As the number of each type of tag is not known, nor is the times-per-minute each is scanned, an exact calculation of the Avgbyte number cannot be provided. However, an approximation is still possible:

237842432 /18/ 1.1 / 500000 = 24bytes per tag (18 represents 3 raw que + 15 data que)

Figure 15

Figure 16

As demonstrated here, MAX_TAGS and DEF_QMINUTE are both multipliers with a significant impact on memory. Thus, if a site is approaching its memory limit and needs to add more tags, the best approach is to tweak either or both of these variables.

In this example, as currently configured, the memory required for the tag definitions is about 232 mb. Changing the MAX_TAGS (shown here as tag capacity) from 500000 to 420000 would provide a 16 percent reduction to about 200 mb.

Next, changing DEF_QMINUTE from 15 to 10 (a 33.3 percent reduction) would result in only 133 MB of memory consumption—or a total reduction of 99 MB from the original allocation.

Every now and then, an ExxonMobil site reports that PHD won’t start. If the error “PRC Section initialization failed” appears in the log file, it is appropriate to ask whether parameters have been changed. The typical reply is “Yes, I changed MAX_TAGS from 250,000 to 1,000,000 because we were getting close to the 250K limit and I didn’t want to have to come back later and revise it again.”


PHDMAN Reporting: To determine the general health of a PHD system, enter the “REPORT SUMMARY” command from PHDMAN. This will provide a snapshot of all problems detected in the system. The first thing to check is the “Total queue overwrite tags.” In this case, the total is zero, which is the ideal; however, if the total is 36, for instance, the user could enter “REPORT OVER 36” and obtain a list of all 36 overwrite tags.

Queue overwrites occur because the specified queue length is too small for the incoming data, or because of a larger than expected volume of tag data. Queue overwrites are also logged to the PHD event log.

Queue overwrites should be investigated promptly because they can destroy good data before it is written to archive files. To correct queue overwrites, one or more of the following parameters may have to be adjusted:

• DEF_QMINUTE – Number of minutes the Data Queue holds data. Generally set at three times the store interval

• MIN_DQSIZE – Minimum size of Data Queue in CUR section

• MIN_RQSIZE - Minimum size of Raw Data Queue in CUR section

As a common practice, range errors should also be checked. In the following example (Fig. 17), there are 7,861 tags with range errors—more than the normal amount. But the tag values on this system were generated with the TEST RDI, and not from a live production system.

Figure 17

The “Report Range” command (Fig. 18) indicates tags with the highest number of range errors. The default number of tags to report is 10, but a higher number may be entered if additional tags need to be reported. Range errors should be corrected next because they generate zero confidence and will be repeated on the REP CONF report if not corrected.

Figure 18


As shown in Figure 19, the values are capped at the high extreme and assigned zero confidence. From looking at the value in the RAW QUEUE, it is obvious that the number is exceeding the high extreme by a significant amount. Obtaining the value from the wrong source tag may cause this situation. Or, it could be that the tag value represents a different engineering unit than was originally collected.

Figure 19

After the range errors have been fixed, entering CLEAR STAT will clear the statistics. Be sure to wait until the tags have gone through several collection cycles, and then run the REP CONF command again. Once all corrected tags are listed, the system should be in an acceptable working order (See Fig. 20).

Figure 20

In some situations, the PHD error log may indicate an error number but offer no explanation for the error. To get an explanation for the error, the user can enter PHDMAN /error=<error#> from a CMD window and see an output similar to the example shown in Figure 21. Here, it is clear what errors #5010 and #10068 actually mean. This command doesn’t work from within PHDMAN—only from the CMD prompt.


Figure 21

Tag Configuration Extract Utility: The Uniformance Tag Configuration Extract Utility identifies bad PHD tags by reading each active tag and providing an output file with the last collected value, date, time and status. Eliminating bad tags can provide a 30-90 percent reduction in the time it takes to cold start the PHD system.

The Uniformance Tag Configuration Extract Utility extracts tags directly from PHD (not from Oracle) in a number of different ways. For example, output files in text format with a .dat extension can be imported into Excel and then modified before they are exported to Oracle. The most beneficial feature of this utility is that it actually reads each processed tag and indicates the last collected date/time and status (see Fig. 22).

Figure 22

The output shown in Figure 23 is from the utility run with the /c switch, which pulls all of the collected tags into a .dat file that is imported into a spreadsheet.

Figure 23


Several ExxonMobil sites experiencing slow PHD cold starts ran the Tag Configuration Extract Utility and identified tags that did not contain data. These tags were disabled, resulting in a 30 to 90 percent reduction in startup time.

PHD Registry Settings: Several Uniformance PHD registry settings are also available, which help to optimize PHD and to prevent abnormal events:

• On PHD systems with a large number of users and processes connected to the API server, the 100-connection default setting may be insufficient. To increase the number of connections modify the following registry setting:

HKLM\System\CurrentControlSet\Services\APIServer\Parameters\MaxClientLinks

• Every time a bad value is retuned to PHD, it gets logged in the PHD server.log. In many cases, however, there are legitimate reasons for the tags to be bad, such as the temporary shutdown of a unit or piece of equipment. This can cause a lot of messages to be logged, subsequently filling up the log files or making them so large they are unusable. Setting this registry key will suppress the bad tag messages (Note: if the tag is constantly toggling between good and bad, the user will continue to get messages).

HKLM\System\CurrentControlSet\Services\RDIServer\Parameters\RDIList\ <rdi_name>\ PreventBadTagMessages (set to 1)

• By default, the OPC browse mode is turned off (set to 2). This value should be set to 1 if your application requires browsing:

HKLM\Software\Honeywell\Uniformance\OPCServer\Browsemode

• Some large sites may exceed the default maximum of 256 RDIs. Typically, this takes place when a shadow server collects data from multiple LCNs or other control systems. It can also occur in situations where multiple shadow servers with transparent data access are used, and the main server needs to know which system to go to for a particular RDI.

HKLM\Software\Honeywell|Uniformance\PHDServer\RDIHostTableSize (set to multiple of 128)

• One bothersome feature of newer APP nodes is the time it takes to reboot (often 10-15 minutes). By turning off “ClearPageFileatShutdown,” rebooting can be shortened to several minutes.

HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\ ClearPageFileatShutdown (set to 0)

• Some sites use Robust Data Collection on redundant pairs of PHD APP nodes. Occasionally, there are issues with the cable connected to the LCN or problems with the LCNP card. If the DetectLxSCommFails switch is set, data collection will automatically fail-over upon a loss of communication with the LCN. The following four registry keys control the response to LxS communication failures. In this example, the connection is queried for status on a 120-second interval. The system will try three times to connect. If it can’t connect after the third try, the failed server sends control over to the standby server. After a failure, there is a 10-minute wait before it is tested again.

HKLM\Software\Honeywell|Uniformance (all DWORD values) - DetectLxSCommFails 1 0-Disabled, 1-Enabled

- QueryStatSeconds 120 Scan Rate (sec.) - RetryCount 3 # times to recheck LxS fault before failure - RetryConnectTMO 600 Wait time after fault before next test (sec.)

• “Named Pipes” is the default connection under Windows when connecting to the domain controller for user authentication. If the user account information is too large, Named Pipes can fail to transfer information back to the application resulting in a “Failed to Wait for Pipe” error. To correct this error, it may be necessary to add two registry entries. The first additional entry is UseFirewallPackage, which specifies the SSPI authentication method. The user sets the value to NTLM (although Kerberos may also be specified). The second entry is UseFirewallSecurity, which can be set to 1 in order to force applications other than Legacy API to use SSPI instead of Named Pipes.


More Information For more information about Uniformance, visit our website at www.honeywell.com/ps or contact your Honeywell account manager. Automation & Control Solutions Process Solutions Honeywell 1860 W. Rose Garden Lane. Phoenix, AZ, 85027 www.honeywell.com/ps

SZ UseFirewallPackageNTLM Allows specification of SSPI auth. Method DW UseFirewallSecurity 10-Disabled, 1-Enabled

Conclusion

Thanks to the powerful but underutilized tools and utilities described in this paper, Honeywell customers can make the most of robust plant information management features offered by Uniformance PHD—and by doing so, significantly improve their operational and economic performance.

©2010 ExxonMobil. To the extent the user is entitled to disclose and distribute this document, the user may forward, distribute, and/or photocopy this copyrighted document only if unaltered and complete, including all of its headers, footers, disclaimers, and other information. You may not copy this document to a Web site. ExxonMobil does not guarantee the typical (or other) values. Analysis may be performed on representative samples and not the actual product shipped. The information in this document relates only to the named product or materials when not in combination with any other product or materials. We based the information on data believed to be reliable on the date compiled, but we do not represent, warrant, or otherwise guarantee, expressly or impliedly, the merchantability, fitness for a particular purpose, suitability, accuracy, reliability, or completeness of this information or the products, materials, or processes described. The user is solely responsible for all determinations regarding any use of material or product and any process in its territories of interest. We expressly disclaim liability for any loss, damage, or injury directly or indirectly suffered or incurred as a result of or related to anyone using or relying on any of the information in this document. There is no endorsement of any product or process, and we expressly disclaim any contrary implication. The terms, “we”, “our”, "ExxonMobil Chemical", or "ExxonMobil" are used for convenience, and may include any one or more of ExxonMobil Chemical Company, Exxon Mobil Corporation, or any affiliates they directly or indirectly steward. ExxonMobil, the ExxonMobil Logo, the Interlocking "X” Device, and all other product names used herein are trademarks of ExxonMobil unless indicated otherwise.

Uniformance® is a registered trademark of Honeywell International Inc.

WP-10-19-ENG December 2010 Printed in USA © 2010 Honeywell International Inc.

http://www.honeywell.com/ps�

http://www.honeywell.com/ps�

Uniformance® Troubleshooting and Optimization - … · Uniformance® Troubleshooting and...

Documents

Transcript of Uniformance® Troubleshooting and Optimization - … · Uniformance® Troubleshooting and...