The Why's and How's of Migrating to FoundationCarlton DoeSenior Systems EngineerNewCo Software and Solutions, Inc.

Who I Am10 + years of experience as DBA and engine admin mainly in retail environmentsCo-founded, presided over, and currently sit on the Board of Directors of the International Informix Users Group (IIUG)Written two Informix Press books:

What Im Going to Cover What is IDS and Foundation? Migration issues ONCONFIG parameters SQL statement cache Max Connect Fuzzy Checkpoints Smart large objects Dbspaces Simple/complex datatypes and casting User Defined Routines and Functions Collections DataBlades and the Blade Manager Java in the engine

What Is IDS and Foundation?Parallelism built-inParallel Data QuerySQL92 Entry LevelEnterprise ReplicationMany, many more...SQL 3 SupportDataBlade SupportDataBlade Developer KitUser Defined RoutinesUser Defined DatatypesUser Defined IndexingR-Tree IndexingExtended B-Tree SupportRow TypesCollections (sets, multiset, lists)InheritancePolymorphismPartitioning with new data typesSchema KnowledgeINFORMIX-Dynamic Server7.xIDS - UDO9.1x

What Is IDS and Foundation?INFORMIX-Dynamic Server7.xIDS - UDO9.1x+=

Then What Is Foundation?Informix Dynamic ServerWebDataBladeExcaliburTextDatabladeObjectTranslatorOfficeSpigotJavaVM

Which Version Is Loaded??(IDS)Odra: onstat -Informix Dynamic Server 2000 Version 9.21.UC2

(Foundation)Ebro: onstat -Informix Dynamic Server 2000 Version 9.21.UC2

(MSGPATH)Wed Oct 4 07:49:52 200007:49:52 Booting Language from module 07:49:52 Loading Module 07:49:57 Informix Dynamic Server 2000 Version 9.21.UC2 Software Serial Number AAB#J50070507:50:12 Informix Dynamic Server 2000 Initialized -- Shared Memory Initialized07:50:15 Physical Recovery Started07:50:25 Physical Recovery Completed: 0 pages restored

Which Version Is Loaded??Ebro: cd /usr/informix/9_21/extendEbro: ls -ldrwxr-xr-x 4 informix informix 1024 Jul 19 08:08 ETX.1.30.UC5drwxr-xr-x 4 informix informix 1024 Jul 19 08:07 TXT.1.10.UC5drwxr-xr-x 4 informix informix 1024 Jul 19 09:02 VTS.1.20.UC1drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxbuiltins.1.1drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxmngrdrwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxrltree.2.00drwxr-xr-x 4 informix informix 1024 Jul 19 07:49 krakatoadrwxr-xr-x 4 informix informix 1024 Jul 19 07:42 LLD.1.20.UC2drwxr-xr-x 4 informix informix 1024 Aug 22 13:36 WEB.4.10.UC1

Migration Issues

Migration Issues: Things to KnowYou can not directly migrate from < 7.x versions

Informix STRONGLY suggests that you first move to the latest available 7.x version then move to 9.x

While you may want to export/import your data, in-place upgrades have been quite successful -- just remember to wear your belt and suspenders

HP-UX 11.0 - RUN_AS_ROOT.server script is wrongLinks created for one library goes into /usr/lib instead of /usr/lib/pa20_64; if left alone, engine will not come up (library not found error). A defect has been entered

Migration Issues: Things to KnowWith IDS, CPU and user-defined VPs are run as user informix while other VPs are still run as root. This prevents UDRs from having root access

If you have auto-start scripts, you should modify them so they su to informix before starting the engine

Install engine after version 2.2 client products for finderr to work. The order is: tools network engine

Migration Issues: Things to KnowPrecision change in float / smallfloat to decimal conversion smallfloat 8 -> 9float 16 -> 17

Default behavior of create index is now to create detached indexes. 7.x attached indexes are supported by the upgrade process

Use the DEFAULT_ATTACH variable to temporarily mimic 7.x behavior, it will be discontinued however in a future release

Migration Issues: Things to KnowIf using DRDA Gateway version 7.31.UC1 with IDS 7.x, you must get the DRDA patch to avoid defect #132619

HADR has changed. DRAUTO is no longer supported so after a HADR failure condition is declared, the secondary server goes to read-only mode

You must manually convert the secondary server to standard mode onmode -d standard

No changes to DRINTERVAL, DRTIMEOUT, DRLOSTFOUND

Migration Issues: Things to Know9.2 ER and HPL -- may not support named types. This should be in place for 9.3

Test dbexport / dbimport to see if more complex named types work

Shut down HADR / ER before upgrading

Migration Issues: Things to KnowSystem catalog changes - columns added, moved, data types changed. New tables added and some dropped. Sysindexes and systables now a view rather than tablesLong identifiers!!The 8/18 rule is dead (the crowd cheers!!)user ids - 32 characters identifiers - 128 characters

table name, server name, database name, column name, index name, synonym name, constraint name, procedure name, dbspace name, blobspace name, optical cluster name, trigger name, type name, routine language name, access method name, operator class name, trace message class name, trace message name, procedure variable names

Migration Issues: Things to KnowEach user connection uses about 5% more shared memory. Watch your shared memory allocations - both instance and operating system

This can be offset by using MaxConnect

Migration Issues: New Reserved Words CACHE COSTFUNC ITEM SELCONST INNER JOIN LEFT LOCKS RETAIN

Migration Issues:New ONCONFIG Parameters ALLOW_NEWLINE BLOCKTIMEOUT DD_HASHMAX DD_HASHSIZE PC_HASHSIZE PC_POOLSIZE SBSPACENAME SYSSBSPACENAME STMT_CACHE STMT_CACHE_SIZE VPCLASS JDKVERSION JVPHOME JVPLOGFILE JVPPROPFILE JVPJAVAVM JVPJAVAHOME JVPJAVALIB JVPCLASSPATH JVMTHREAD

Migration Issues:New ONCONFIG ParametersVPCLASSEnables you to designate and create uniquely named, user-defined, or system classes of VPs. User-defined VPs have the same functional power as CPU VPsUser-defined VPs should be created to execute user- defined routines and/or DataBladesSome DataBlades (Verity, Excalibur Text) require their own VP as does the JVMUser-defined VPs can be added/dropped on the fly with the onmode -p class_name command

Migration Issues:New ONCONFIG ParametersVPCLASSSyntax tree:VPCLASS name,num=X,[max=X,aff=(X-Y),noyield,noage]

VP name is not case sensitive, options are order independent, no whitespace, 128 char max/entry Must declare multiple JVPs to execute Java UDRs in parallel (Unix) If using noyield, only declare 1 vp since the UDR will execute serially, causing others to queue for their turn

Migration Issues:New ONCONFIG ParametersUse the VPCLASS parameter to replace the CPU and AIO ONCONFIG parameters

VPCLASS cpu,num=3,max=10,noage

Comment out AFF_SPROC, AFF_NPROCS, NOAGE, NUMCPUVPS, SINGLECPUVP parameters

VPCLASS aio,num=5,max=10

Comment out NUMAIOVPS parameter

Migration Issues: How do I Migrate?1. Stop database processing and force a change to a new logical log onmode -sy; onmode -l; onmode -c; onmode -ky;

2. Back up the instance(s) and critical configuration files in $INFORMIXDIR

etc/ /aaodir/adtcfg $ONCONFIG (s)**/dbssodir/adtmasks ONCONFIG.std sm_versions sqlhosts** tctermcap termcap

Migration Issues: How do I Migrate?3. Restart the instance(s) and put them into single-user mode to ensure all open transactions are properly rolled back

oninit -sv

4. Verify data and index integrity with the oncheck utility(-cr, -ce -cc, -c [ I / D ] db_name)

5. Shut the instance(s) down again and create level 0 backup(s) and/or dbexports

6. Install the new IDS / Foundation software and change relevant scripts, global parameters, config files

My Recommended Directory Structure/usr/informix7_3

9_21

scripts

disks (also tapes)

ISA

logs

config_files

max_connkoetari -- symbolic links

odra -- symbolic links(current logs) /old_logsEnvironment Variables:

INFORMIXDIRONCONFIGINFORMIXSQLHOSTS

Migration Issues: How Do I Migrate?7. Change ALARMPROGRAM to [ null | no_log.sh ] if using On-Bar or LTAPEDEV to /dev/null

8. Install any DataBlades

9. Restart each instance into quiescent mode and monitor the MSG_PATH file. Sysmaster and reserved pages conversion is automatic (see $INFORMIXDIR/etc/dummyupds7x.sql). When completed, a notice is written to MSG_PATH

10. Execute an update statistics high command on the sysmaster database. Standard update stats command on all others

Migration Issues: How Do I Migrate?11. Verify data integrity with the oncheck command

12. Change LTAPEDEV / ALARMPROGRAM back to original value

13. Create a level 0 back-up

14. Let the users back in

If there are any problems -- RESTORE from tape

SQL Statement Cache

SQL Statement Cache: IntroductionAs implied, prepared and optimized SQL statements are stored in a section of the virtual portion of shared memory. Users executing identical DML SQL statements can pull the pre-parsed query plan and even query data structures from the cache

More complex queries (multiple columns, many filters in the WHERE clause) benefit most from caching

Turned off by default, the STMT_CACHE $ONCONFIG parameter and the SET STATEMENT CACHE SQL command control how the caching process works. Can be overridden with the onmode -e command

SQL Statement Cache: How It Works1. Incoming SQL statements are hashed to a value which is compared to other hash values in the SSC section of the virtual portion of shared memory

Caveats:statements only optimized at the PREPARE phasehost variable names/placeholders are not included in the hash algorithmwhite spaces and case of the statement is significantsession-specific parameters (OPTCOMPIND, OPT_GOAL) are significant and will affect hash value

SQL Statement Cache: How It Works2. If hash value matches an existing value, the cached copy is used. The existing copy is NOT re-optimized

3. If a match does not exist, the statement is evaluated for inclusion in the SSC

Qualifications for inclusion in the SSC:contains DML statements using built-in data types and operatorsno UDRs/UDFslocal database connections onlyno explicit temporary tables

SQL Statement Cache: How It WorksQualifications for inclusion in the SSC (cont):

select * statements can be fully expanded to include all columns in the listed tablesis not a statement generated by a stored procedureno imbedded subselects in the SELECT statement

SQL Statement Cache:Enabling and Sizingonmode -e {ENABLE|ON|OFF|FLUSH}

ENABLE - Allows statement caching to take place. This does not turn caching on, but allows it to be turned on

ON - Turns on statement caching. Caching must first be enabled through the STMT_CACHE parameter or onmode command

OFF - Turns off the SSC and immediately disables sharing. Cached statements that are in use remain in the SSC until their use count reaches zero

FLUSH - Active statements are marked and subsequently flushed when they are released. When the SSC is turned off, sharing is immediately disabled

SQL Statement Cache:Enabling and SizingSTMT_CACHE $ONCONFIG parameter:

0 - SQL Statement Cache disabled (default)1 - SSC enabled though sessions must explicitly call for its use2 - SSC enabled for all sessions unless explicitly denied on a session-by-session basis

Set the STMT_CACHE environment variable to 0 (off)or 1 (on)Execute a set statement cache [ on | off ]command

SQL Statement Cache:Enabling and SizingSTMT_CACHE_SIZE $ONCONFIG parameter: determines the size of the SSC in KBs, default = 524 kb

If SSC is full and all statements are active, if additional statements qualify for inclusion the SSC will expand to hold them. As a statements user counts drop to zero, the SSC will flush them and reduce back to STMT_CACHE_SIZE

SQL Statement Cache:Monitoring and Tuningonstat -g cac stmt displays size of the SSC the cached statements, whos currently executing a statement, and the number of times its been usedThe hash column is not the statements hash value, rather the hash bucket in which the statement residesTo tune, monitor the size of the SSC over time, then resize as appropriate

SQL Statement Cache:Monitoring and TuningUse the STMT_CACHE_DEBUG environment variable to trace what happens with any given SQL statement

export STMT_CACHE_DEBUG=debug_value:path_to_file

debug_value can be 1 (basic) or 2 (extended)

SQL Statement Cache:Monitoring and TuningSSC identity (OK:status=1): select * from yoyoSSC link (semantic error): select * from yoyoSSC identity (stmt not found): create table yoyo (col1 int) in mustang_2SSC qualify (invalid stmt type): create table yoyo (col1 int) in mustang_2SSC identity (stmt not found): insert into yoyo values (1)SSC qualify (OK): insert into yoyo values (1)SSC key insert (OK): insert into yoyo values (1)SSC full insert (OK): insert into yoyo values (1)SSC identity (stmt not found): insert into yoyo values (2)SSC qualify (OK): insert into yoyo values (2)SSC key insert (OK): insert into yoyo values (2)SSC full insert (OK): insert into yoyo values (2)SSC identity (OK:status=1): insert into yoyo values (2)SSC link (OK): insert into yoyo values (2)SSC identity (stmt not found): drop table yoyoSSC qualify (invalid stmt type): drop table yoyo

What Are FuzzyCheckpoints???

CheckpointsIDS introduced fuzzy operations and checkpoints to increase transactional throughput

Two types of checkpoints:

Full or sync Fuzzy

Steps of a Sync Checkpoint1. Engine blocks threads from entering critical sections of code2. The page cleaner thread flushes the physical log buffer to log on disk3. The page cleaner threads flush to disk all modified pages in the buffer pool (chunk write)4. The checkpoint thread writes a checkpoint record to the logical log buffer5. The logical log buffer is flushed to the current logical log file on disk6. The physical log on disk is logically emptied (current entries can be overwritten)7. The checkpoint thread updates the reserved pages with the checkpoint record information

What Causes Sync Checkpoints to Occur?Physical log becomes 75% fullonmode -c or -kyAdministrative actions (adding dbspaces, altering tables)A backup or restore operation using ontape or ON-BarEnd of fast recovery or full recoveryReuse of a logical log containing the oldest fuzzy operation not yet synced to diskLTXHWM reached with fuzzy transactionsThrough the onmonitor menuGood and bad:logically and physically data, interrupts user activity, expensive (I/O hit), must tune to reduce duration

Fuzzy CheckpointsRecord results of fuzzy operations in the logical log

Goal: reduce or eliminate checkpoint interruptions, eliminate the physical log (over 3 - 4 phases)

Cost: increased logical recovery time

Note: Fuzzy checkpoints are not used in HADR environments

Fuzzy CheckpointsThings to know:Inserts, updates, delete DML statements Currently supports built-in data types only (character, numeric values)Logged databasesNot old pages. Determined by page timestamp (4 byte, cycles 2 gb -> -2 gb -> 0)No BEFORE images are generated in the physical logDuring fuzzy checkpoints, buffered information on fuzzy operations is **not** flushed to diskAfter a fuzzy checkpoint, disks are not physically consistent

Steps of a Fuzzy Checkpoint1. Engine blocks threads from entering critical sections of code2. The page cleaner threads flush to disk all **non-fuzzy** modified pages in the buffer pool (chunk write) Note: MLRU queues still full 3. A Dirty Page Table (DPT) is constructed containing entries (buffer addresses) for fuzzy operations4. The DPT is flushed to the current logical log on disk 4A. The page cleaner thread flushes the physical and logical log buffers to disk5. The checkpoint thread updates the reserved pages with the checkpoint information including the Log Sequence Number (LSN).6. The physical log on disk is logically emptied (current entries can be overwritten)

Log Sequence Number (LSN)A value representing a position in the logical log (log # and log position)Stored in each dirty buffer heading containing fuzzy operation results along with a newly-stored timestampThe oldest LSN is tracked in shared memory because it contains the oldest fuzzy operation. Oldest LSN can not be further back than 1 logical log and 2 checkpointsLogs after the LSN can not be freed for re-use until a checkpoint has occurred

Net Net -- large logical logs are GOOD!!!

What Causes Fuzzy Checkpoints to Occur?Checkpoint interval has elapsedPhysical log becomes 75% fullonmode -c fuzzyReuse of a logical log containing the last checkpoint (either fuzzy or sync)

Recovery - Sync Checkpoint1. Physical log data used to return all disk pages to original synced state (physical restore)2. The most recent checkpoint (sync) record is located in the logical log files3. All subsequent logical log records are rolled forward4. Uncommitted transactions are rolled back

Recovery - Fuzzy Checkpoint1. Physical restore proceeds as normal however disk pages are not physically consistent since there are modified pages that were not logged in the physical log

2. Logical recovery begins in two phases:Phase A -- reapply DPT recordsPhase B -- post checkpoint records

Recovery - Fuzzy CheckpointPhase A - Logical Recovery:1. Locate DPT created before last fuzzy checkpoint2. Selectively apply fuzzy operations from the DPT/logical log from the oldest remaining logical log up to the most recent checkpoint

Conditional update based on page timestamp - if DPT record is newer than page timestamp, then the change is applied

Result: similar to end of physical restore when using sync checkpoints

Recovery - Fuzzy CheckpointPhase B - Logical Recovery:1. Beginning with last checkpoint, roll forward subsequent logical log records.2. Uncommitted transactions are rolled back

Result: similar to the logical restore when using sync checkpoints. Disks are physically and logically consistent to last recorded transaction

Fuzzy CheckpointsBuffer structures, logical log records, and reserved pages are automatically upgraded when migrating to 9.x

Fuzzy checkpoints enabled by default. Set the NOFUZZYCKPT environment variable to true/on/1 to disable them prior to initializing shared memory

Set the TRACEFUZZYCKPT environment variable to trace fuzzy checkpoints. It will affect instance performance but will log in the MSGLOG the number of buffers that remain dirty after a checkpoint is completed. It also lists the location of the old LSN

Large Objects, SBSpacesand Dynamic Lock Allocation

Large Objects and SBSpacesIDS has 4 types of LOs:Simple - BYTE & TEXTSmart - CLOB & BLOB

Large Objects and SBSpaces

Access to C/BLOBs is faster due to portion-level addressingDefault characteristics are easy to overrideWith sufficient resources, C/BLOBs are transactionally recoverable

Simple

Smart

Logging Method

logging depends on the table where the large-object is stored

logging can be turned on or off at the large-object level

Size

2 GB maximum

4 TB maximum

Locking Granularity

all or nothing

can lock portions or all of the object

Reads and writes

only the whole large-object can be read

all or a portion of the large-object can be read

Fragmentation

all large-objects for a table are stored in the same blobspace

ability to store the large-objects for one table in multiple sbspaces

Large Objects and SBSpacesSmart LOBs are stored in smart BLOBSpaces (sbspace) which contain meta-data as well as user dataUser DataSize and location ofthe metadata area isconfigurable atsbspace and/or chunk creation

Metadata is alwayslogged regardless ofdatabase or sbspacelogging mode

Large Objects and SBSpacesonspaces -S name -g pageunit -p pathname -o offset-s size [ -m pathname offset ] -Ms mdsize -Mo mdoffset-Df default list

-Mo mdoffset offset, in kbs, into the disk partition where metadata will be stored

-Ms mdsize the size, in kbs, of the metadata area for the initial chunk, remainder is user data space -- not recommended though possible. Each SLO needs ~500 bytes of metadata space

-Df default list default specifications for objects stored in the sbspace. Comma separated, in double quotes ( )

Large Objects and SBSpaces

Tag

Values

Default

Description

ACCESSTIME

ON or OFF

OFF

ON instance tracks access time of SLOs stored in the sbspace.

AVG_LO_SIZE

For Windows NT:

4 to 2^31

For UNIX:

2 to 2^31

8

Average size, in kbs, of SLOs stored in the sbspace. Used to calculate metadata area size if -Ms option is not specified.

Error 131 is returned if you run out of metadata area in the sbspace. To allocate additional chunks to the sbspace that consist of metadata area only, use the -Ms option.

BUFFERING

ON or OFF

ON

ON - instance uses resident portion buffer pool for SLO I/O operations.

OFF - instance uses light I/O buffers in the virtual portion (lightweight I/O operations) PREFERRED!!!!

LOCK_MODE

RANGE or BLOB

BLOB

RANGE - only a range of bytes in the SLO is locked.

BLOB - the entire SLO is locked.

Large Objects and SBSpaces

Create a level 0 backup after creating a sbspace

Tag

Values

Default

Description

LOGGING

ON or OFF

OFF

ON - instance logs changes to the SLOs in the user data. Create a level-0 backup of the sbspace after creating/changing

EXTENT_SIZE

4 to 2^31

16

Size, in kbs, of the first disk allocation for SLO storage when you create the table.

Let the system select the EXTENT_SIZE value.

MIN_EXT_SIZE

2 to 2^31

4

Minimum amount of space, in kbs, to allocate for each SLO.

NEXT_SIZE

4 to 2^31

16

Size, in kbs, of the next disk allocation for smart large objects when the initial extent becomes full.

Let the system select the NXT_SIZE value.

Large Objects and SBSpacesTo add a chunk, you can specify whether the chunk will contain only user data, metadata, or both:onspaces -a sbspace -p pathname -o offset -s size [-Ms mdsize -Mo mdoffset | -U ] [ -m pathname offset ]

To drop the sbspace:onspaces -d sbspacename -f (force)

Large Objects and SBSpacesSLOs can be distributed in round-robin form among sbspaces like regular dataCreates a 72 bytedescriptor handleas opposed to 56bytes with simpleLOscreate table products (prod_sku int, prod_desc clob, prod_photo blob) put prod_desc in (sbspace_1, sbspace_2), prod_photo in (sbspace_9, sbspace_10), fragment by expression (prod_sku < 1000 in db_1, prod_sku >= 1000 in db_2);

Large Objects and SBSpacesInformix recommends that you convert simple LOs to their smart counterparts as soon as feasible

You can, in many cases, use the alter table command to make the change(s):

alter table t_name modify col_name [ clob | blob ]put column_name in (sbspace(s));

Large Objects and SBSpacesMonitoring / tuningoncheck -cs -cS / -ps -pS to print/view metadata-ce / -pe has additional sbspace information

onstat -g smb s | c | h | e | tsmb s sbspace summary informationsmb c sbspace chunk informationsmb fdd LO file descriptor tablesmb lod LO header tablesmb t timing statistics for enabled server, not well documentedsmb e undocumented

onstat -k has additional columns to document SLO byte range locks

onstat -d has additional chunk-related columns for sbspaces

Large Objects and SBSpacesBacking up and restoring SLOs and sbspaces now operates like regular dbspaces -- at the page level. As a result, you can now have granular and truly incremental SLO/sbspace backups.

Large Objects and SBSpacesBackup: Metadata LO headers are read and a list made of pages to be backed upAll individual page timestamps are compared too since updates can occur without affecting the LO header timestampQualifying data is sent to the backup program through large buffer accesses

Restore:Sbspace header and metadata information is restoredArchive desriptor pages contain page addresses and extent sizes for the sbspace pages that follow. This is held in shared memory during the operationSbspace pages are restored to their correct places

Dynamic Lock AllocationIn IDS, the locking mechanism changes. LOCKS is now a starting allocation (resident portion)

If the lock table fills, additional locks are allocated (x 16) minimum allocation: smaller of current lock table or 100,00 (virtual portion)

To prevent lock table flooding, range locks will be upgraded to the whole SLO if a single transaction uses > 10% of lock table

Byte range locks are what lock portions of a SLO. They are sorted lists of rowids, partition #s, and byte address ranges

Dynamic Lock AllocationRange locks can be joined or split as needed if ranges to be locked adjoin, overlap, or become separate

Ranges locks are not stored in the lock hash table, rather an intent lock is placed for the LO header. Byte range locks are linked to associated hash intent lock then sorted by byte number

Byte ranges locks allocated if:< 50% of lock table is being used50%

MaxConnect

MaxConnectMax Connect is a middle-ware communications component that removes the overhead of managing user connections from the instance

Helps you get scale your user connections past the effective 2,000 user connection limit without any change or resource impact to the database server

It supports 7.3 and 9.2x+ Unix ports of the engine but accepts user connections from any platform

NO CHANGE to existing applications!!

MaxConnect PerformanceBaaN IVc World record:11,445 BRUs on HP N4000 (8CPU, 16GB) and IDS21% better than Oracle on identical HP systemBeat Oracles 64-CPU Sun E10000 result18,650 database connections using 98 transports !!

MaxConnect Architecture

MaxConnect ConfigurationA MaxConnect instance connects to only one DB server instanceConnections are managed through the SQLHOSTS file with a new connection protocol -- imcClients change hostname and portaddress to point to the MaxConnect instance, instance name remains the sameMaxConnect instance has an ADMIN entry, a client-facing entry, and a DB server-facing entryDB instance has a DBSERVERALIAS and a DBSERVERNAME entry

MaxConnect ConfigurationClient SQLHOSTS file:ebro_tcp ontlitcpmaxcon_11526

MaxConnect SQLHOSTS file:ebro_tcp ontlitcpmaxcon_11526ebro_db ontliimccronus1575ebro_adm onsoctcpmaxconn_11600

Server SQLHOSTS file:ebro onipcshmcronus ebro_ph (DBSERVERNAME)ebro_tcp ontliimccronus 1575 (DBSERVERALIAS)

MaxConnect ConfigurationNETTYPE entry changes:ipcshm,1,20,CPU# tlitcp,5,200,CPUtliimc,1,2000,CPUMaxConnect instance environment variables:IMCONFIG - location of MaxConnect config file -default is $INFORMIXDIR/etc/IMCconfigIMCSERVER - Client-side MaxConnect instance nameIMCADMIN - Admin-side MaxConnect instancename

MaxConnect ConfigurationMaxConnect config file contains:IMCLOG - location of MaxConnect message logIMCWORKERTHREADS - # of worker threads, range = 1 -> 64IMCWORKERDELAY - amt of time MaxConnect will wait and pool messages to transmit to DB server default = 0, range = 0 -> 100,000 msIMCTRANSPORTS - # of transport connections to DB server. Default = 2, range = 1 -> 64, ratio = 1/100 users

MaxConnect TuningUse onstat -g imc, the imcadmin utility, or the ISA to monitor and tune MaxConnect

Tune the NETTYPE sessions parameter by monitoring the q-exceed and alloc/max values. May also need to tweak the IFX_NETBUF_PVPOOL_SIZE variableTune IMCTRANSPORTS by monitoring blocked or partial writesTune IMCWORKERTHREADS by monitoring AvgQlen If > 5, increase number of threadsTune IMCWORKERDELAY by summing and comparing historgrams against AvgQlen

Complex and User-Defined Data Types

Complex and User-Defined Data Types

Built-in Data TypesNew Built-In:int8 (8 bytes)serial8 (8 bytes)Range is -9,223,372,036,854,775,807 to 9,223,372,036,854,775,807must add UC to serial8 to ensure uniquenessone serial8 and serial per table

boolean valid values: t, f, null. case sensitive

lvarchar (variable length character data type; 32k maximum)

Complex and User-Defined Data Types

Complex Data Types - RowRow DataTypesanalogous to C structure, come in two kindsNAMED - strongly typed, IDed by name, has inheritance, used to build columns and tablesUNNAMED - weakly typed, IDed by structure, no inheritance, used to build columnsCan contain built-in, collection, opaque, distinct, another row type data types Caveat: no serial or serial8

Complex Data Types - RowRow DataTypesNamed: create row type name_t (fname char(20), lname char(20));

create row type address_t (street_1 char(20), street_2 char(20), city char(20), state char(2), zip char(9));

create table student(student_id serial, name name_t, address address_t, company char(30));Unnamed:ROW (a int, b char (10))Note: is also equal to ROW(x int, y char(10))

create table part (part_id serial,cost decimal,part_dimensions row (length decimal,width decimal, height decimal, weight decimal));

Complex Data Types - RowUsing Named Row Types in SQL statements:

Insert statement:insert into student values (1234, row (John,Doe)::name_t, row ("1234 Main Street","", "Anytown","TX","75022")::address_t, "Informix Software")

Select statement:select * from student where name.lname matches "Doe;Result set:student_id 1234name ROW('John ','Doe ')address ROW('1234 Main Street ',' ,'Anytown ','TX','75022 ')company Informix Software

Complex Data Types - RowTo drop a named row type:

drop row type address_t restrict;

Complex Data Types - CollectionsGrouping of elements of the same datatype (char, int), max size = 32 KB

Used whenThe data is meaningless without the context of the other members in the collection (e.g., golf scores, to do list, set of names)Individual data elements are not likely to be directly queried by positionThe maximum number of data elements is less than 32Can be null

Complex Data Types - CollectionsThree kinds of collections:

Set - unordered, no duplicates allowed set {apple, orange, grapefruit, plum}

Multiset - unordered, duplicates allowed multiset {apple, orange, grapefruit, apple, plum, grapefruit}

List - ordered, duplicates allowed list {apple, orange, grapefruit, apple, plum, grapefruit}

Complex Data Types - Collectionscreate table class(class_id serial, class_name varchar(60), description lvarchar, prereqs set(char(20) not null));

Insert syntax is similar to named row types:

insert into class values (300, Performance and Tuning, Covers advanced information on tuning the Informix Dynamic Server, (SET{RDD,BSQL}));

Use the in keyword to query values in a collectionselect * from class where (ASQL) in prereqs;

define xyz char(20)define set_var set(char(20))select prereqs into set_var from class where class_id = 300

foreach del_set_cursor forselect * into xyz from table(set_var)if xyz matches RDD thendelete from table(set_var) where current of del_set_cursorend ifend foreach

Complex Data Types - CollectionsYou can not update one element in a collection, you must replace the whole collection:

update class set prereqs = (set{RDD,ASQL,BSQL}) where class_id = 300;

update class set prereqs = set_char where class_id = 300;

Complex and User-Defined Data Types

User-Defined Data Types - DistinctTwo user-defined data types (UDTs):Distinctdata type modeled on an existing data typehas a unique name to distinguish it from other similar typesinherits the internal structure from the source typemight have operations and casts defined over its source type

Opaquedata type that is unknown to the database serveryou must define the internal structure, functions, and operations

User-Defined Data Types - Distinctcreate distinct type dollar as decimal;create distinct type aus_dollar as decimal;

create table sales ( sku int, sales_date date, us_sales dollar, aus_sales aus_dollar);

insert into sales values (1234, today, 15.0::dollar,0::aus_dollar);insert into sales values (5678, today, 0::dollar, 75.0::aus_dollar);

select sku, (sum(us_sales) + sum(aus_sales)) from sales where sales_date = today group by 1;

error: 674 - routine (plus) can not be resolved

User-Defined Data Types - DistinctNeed to create some UDFs that handle the type and value conversion for you:create function usdlr_to_ausdlr(parm1 dollar) returning aus_dollar specific usd_to_ausd; return (parm1::decimal * 1.8)::aus_dollar;end function;

create function ausdlr_to_usdlr(parm1 aus_dollar) returning dollar specific ausd_to_usd; return (parm1::decimal / 1.8)::dollar;end function;

select sku, (sum(us_sales) + sum(ausdlr_to_usdlr(aus_sales))::dollar) from sales where sales_date = today group by 1;

User-Defined Data Types - OpaqueAn opaque data type stores a single value that cannot be divided into components by the engine.

Implemented as C or Java structures and manipulated by a set of routines written in C or Java

An opaque value is stored in its entirety by the engine without any interpretation of the contents or its structure

All access to an opaque type is through functions written by the user. You define the storage size of the data type and input and output routines

User-Defined Data Types - Opaque1. Create the C / Java data structure to represent the internal data structure2. Write the support functions in C / Java3. Register the opaque data type with the create opaque type statementcreate opaque type type_name ( internallength = length,alignment = num_bytes);length is in bytes, alignment = 1,2,4,8 bytes (default = 4)

create opaque type type_name ( internallength = variable,maxlen = length);length - default = 2 KB, max value = 32 kb

4. Register the support functions with the create function statement. If in Java, will be stored in sbspace(s)

User-Defined Data Types - Opaquecreate opaque type my_type(internallength=8, alignment=4);

create function support_in(lvarchar)returning my_type with (not variant);external name /funcs/my_type.solanguage Cend function;

create implict cast (lvarchar as my_type with support_in);5. Grant access to the opaque data type and support functions6. Write any user-defined functions needed to support the opaque data type - input, output, destroy, compare, aggregates, send, receive, etc.7. Provide any customized secondary-access methods for creating indexes

Casts and Casting

Casts and CastingCasts allow you to make comparisons between values of different data types or substitute a value of one data type for a value of another data type.

create function ausdlr_to_usdlr(parm1 aus_dollar) returning dollar specific ausd_to_usd; return (parm1::decimal / 1.8)::dollar;end function;

Engine provides a number of built-in casts (int to decimal, numeric to char, etc.) for most built-in datatypesMust create user-defined casts for user-defined types. Must be unique with respect to source and target data typesCan not create casts for collections, LOBs, or unnamed row types

Casts and CastingTwo kinds of user-defined casts:explicit. . . where price >= (cast aus_dollar as dollar) . . . .. . . return (parm1::decimal / 1.8)::dollar;create explicit cast (aus_dollar as dollar with aus_dlr_to_us_dlr);

implicitcreate implicit cast (aus_dollar as dollar);create implicit cast (aus_dollar as dollar with aus_dlr_to_us_dlr);

Automatically invoked when:one data type is passed to a user-defined routine whose parameters are of another data type.expressions are evaluated that need to operate on two similar data types.

Casts and Casting - Implicit Casts select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved.

create implicit cast (aus_dollar as dollar);select sum(us_sales) + sum(aus_sales) from sales;(expression) 120.00

drop cast (aus_dollar as dollar);create implicit cast (aus_dollar as dollar with ausdlr_to_usdlr);select sum(us_sales) + sum(aus_sales) from sales;(expression) 80.00

Casts and Casting - Explicit Casts select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved

create explicit cast (aus_dollar as dollar);select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved

select sum(us_sales) + sum(aus_sales)::dollar from sales;(expression) 120.00

drop cast (aus_dollar as dollar);create explicit cast (aus_dollar as dollar with ausdlr_to_usdlr);select sum(us_sales) + sum(aus_sales)::dollar from sales;(expression) 80.00

Casts and Casting Previous examples were straight casts. You can also create cast functions to cast types with dissimilar data structures

1. write the cast function in C / Java / SPLcreate function opaque_to_opaque (input_arg my_type_1)returns my_type_2return cast(cast(input_arg as lvarchar) as my_type_2);end function;

2. register the cast function with the create function command

3. register the cast with the create cast commandcreate explicit cast (my_type_1 as my_type_2 with opaque_to_opaque);

User-defined Routines

UDRs There are UDFs and UDPs

Can be internal or external in nature. Internal are written in SPL, external can be written in C or Java.

If written in Java, compile into a class then .jar file.When registered in the engine, .jar file will be brought into a sbspace then executed when neededby the JVM in the engine

If written in C, create a shared object library in a directory owned by informix with 755 permissions

UDRs UDRs should be concise and precise. Dont want engine wasting resources plowing though verbose code. The values returned should be relevant from a business perspective (e.g. proper domain)

Make sure you include error handling in the code

**ALWAYS** use a specific name, or alias, for every UDR to properly identify each one - used to drop, grant, or revoke permissions as well as to permit function overloading

UDRs Function overloading occurs when two or more functions have the same name but different signatures

signature = UDR name and parameter list

create function plus (in_1 dollar, in_2 aus_dollar) . .create function plus (in_1 aus_dollar, in_2 lira) . . .create function plus (in_1 lira, in_2 aus_dollar) . . .{Note: these probably should be formalized into user-defined aggregates}

When properly registered, instance will use the data types passed to determine which UDR should be used

UDRs create function routine_name (param_list){ returns | returning } typename[ specific specific_name ][ with (internal | handlesnulls | [ [not] variant] | class=vp_class) ]external name full_path_name_of_filelanguage language [ [not] variant ]end function;

in the $ONCONFIG file:VPCLASS fince_vp ,num=2 # vps for finance UDRs oronmode -p +2 fince_vp

USE UDVPs FOR UDRs!!!!

UDRs To drop UDRs, must either use full signature

drop function plus(lire, aus_dollar);

or the specific name

drop specific function lir_to_ausdlr;

UDRs Default behavior of UDRs is single-threaded, can be paralellized if conditions are met:

only in DSS environments (PDQPRIORITY > 1)parallelizable keyword is added to UDR creation statementexternal UDRs only (C / Java) that use PDQ thread-safe SAPI calls (see DataBlade manual for list of SAPI calls)no complex types in input parameters or return valuesmultiple fragments must be scanned

UDRs Not directly called (singleton execution)not an interator - called in loops to return multiple values similar to with resume in SPL

on some UDRs, you can add selectivity and cost modifiers that will affect optimizer activities and query plans used. Not really familiar with this

UDRs create function bigger(int,int) returns intspecific big_paralelwith(parallelizable,class=test_vps) external name/path/plludr.bld(bigger) language C;

alter function bigger (int,int) with (add parallelizable,add class=test_vps);

Can monitor the parallel execution of UDRs through set explain out command the ISA onstat -g mgm onstat -g ses sess_id onstat -g ath

DataBlades and the Blade Manager

DataBlades Datablades extend the functionality of the engine by adding new UDTs, casts, interfaces, UDRs, error messages, clientcode (where necessary) to manipulate the UDTs

Functionality available through SQL, SPL, API calls to external functions

Can be mixed and matched to support any application

While the code is loaded at an instance level, blade registration occurs at a database level. With registration information contained in the database system tables, if you drop the database, blades become unregistered

DataBlades Install the blade in $INFORMIXDIR/extend (tar, cpio, ?)

May have its own install script to brand binaries, use serial # / license key from engine install unless third-party requires unique key setup

Use the Blade Manager to manage blade registration graphical version available with NT and ISA command line only on Unix / Linux

Is case sensitive

DataBlades BladeManager supports the following commands:

list [database] - print list of registered DataBlade modulesset user [user]set server [server]show modules - print list of DataBlade modules on servershow databases - print list of databases on servershow servers - print list of serversshow client - print DataBlade modules with files on clientregister [module] [database]unregister [module] [database]add client [module] - install DataBlade module's client filesdel client [module] - remove DataBlade module's client filesinfo [module] - print information about a DataBlade moduledel logs - delete all the log filesshow log - prompt to display a logshow last log - display log from last actionhelp, ?bye, quit, exitset confirm on|off

DataBlades mustang>show databaseDatabases on server: agg1 dcs_demo dps_demo ireach kiosk media360 my_test_db visclass20

mustang>list media360DataBlade modules registered in database media360: lld.1.20.tc2 TXT.1.10.TC5 ifxrltree.2.00 ifxbuiltins.1.1 web.4.10.TC1 VTS.1.20.TC1 video.2.00.TC2

Introduction to Javain the Server

Java in the Server J/Foundation is the marriage of the Java and SQL languagesCan write internal or external Java routines that are accessed natively by the engineEJB RMI | Corba | IIOP Web Server

Java in the Server Informix provides a Type 4 JDBC driver

Type 1 JDBC-ODBC bridge provides JDBC access via most ODBC driversType 2 A native API and partly Java driver that converts JDBC calls into calls on the client API for Oracle, Sybase, Informix, DB2 etc.Type 3 A net protocol, all Java driver that translates JDBC calls into a DBMS independent net protocol which is then translated to a DBMS protocol by a serverType 4 A native protocol all-Java driver converts JDBC calls into the network protocol used by DBMSs directly. This allows a direct call from the client machine to the DBMS server

Java in the Server,How It Works Java UDRs execute on Java Virtual ProcessorsJava VPs have the same capability as CPU VPIt can process any SQL queryAvoids the inter-process communication overhead for executing Java UDRsThe Java Virtual Machine (JVM) is embedded directly into the VP codeJava runs threads in parallelDynamically load Java VPsIncrease JVPs to balance application load

Java in the Server,How It Works New ONCONFIG Parameters:

VPCLASS jvp,num=1JVPJAVAHOME /usr/javaJVPHOME /opt/informix/extend/krakatoaJVPPROPFILE /opt/informix/extend/krakatoa/.jvppropsJVPLOGFILE /opt/informix/jvp.logJDKVERSION 1.1

Questions???