Genic DataManager 6.7.0 Docker Configuration Guide 6.0/Install/GDM... · SECURITY_ DOMAIN SECURTIY_...

Docker Configuration

Release 6.7.0

Details are provided to set up Docker for:

l Genic DataManager

l GenicWorkFlow

l Genic DataLoader

l GDM - GCBRT Bridge

Pre-requisites

These tools need to be installed before configuring docker.

l Docker

l Docker Compose

Note: For installation details, refer to the Docker website: https://docs.docker.com/

Published: April 2020 / Edition: 2

Copyright © 2020 Enverus. All rights reserved.

https://docs.docker.com/

Docker Compose FileTheDocker Compose tool is used to define and runmulti-container Docker applications, using a YAML file toconfigure the application's services. You need to update the YAML file by specifying the environment variablesfor the docker images to be downloaded.

Important: When runningmulti-hosts, with GDMandGWF containers on two different hosts, separateinstances of the Docker configuration file must be used for each of the two servers to set up theconfiguration.

Details for these environment settings are specified in the YAML file:

l Image

l Volume

l Database

l Websocket

l Port

l Other optional environment values

Docker Compose File

Genic DataManager 6.7.0 | Docker Configuration Guide

Registering for a Docker ID

You need to create and register a Docker ID to access the Docker hub for downloading the container images.Your Docker ID becomes your user name to access the Docker hub.

TO REGISTER FOR A DOCKER ID

1. Launch the following Docker hub web page: https://hub.docker.com/.

2. ClickSign up for Docker Hub.3. Enter the following:

l Docker ID: Usernamewhich will be your Docker IDl Password: Passwordl Email: Valid email address

Once the user ID is created, contactEnverus Trading & RiskCustomer Support / PS / to add your ID tothe external users list. This will allow you to have read-only access to the docker images.

Docker Compose File


https://hub.docker.com/

Configuring DockerFollow these steps on the local machine where docker is installed and run.

Note: Details of the various environment parameters required for configuring docker are provided in theYAML file.

Important: The badge indicates the new addition for the current release.

TO CONFIGURE DOCKER

1. ContactEnverus Trading & RiskCustomer Support / PS / to get a sample docker project.

2. Create a docker folder under /opt/ as /opt/docker.3. Select and copy the sample docker YAML file from the project, e.g docker-compose.yml into

/opt/docker folder.4. Edit the docker-compose.yml file for the required docker configuration.5. Copy theWorkflow_Custom.jar to ./gdm-gwf-config/customResources.

./gdm-gwf-config/customResources is the location where the custom jars are placedmanually, onthe selected local drive.

Note: TheWorkflow_Custom.jar is just an example custom jar name. You can rename the jar file name,as preferred.

6. To download the image:

a. Login to the docker hub, by running the following command:

docker login

b. Enter the registered user name and password.

7. Run the following command:

docker-compose up -d

This commandwill download the image from the docker hub and start the container, bymapping thedetails specified in the docker-compose.yml file.

Note: Highlighted parameters in the YAML file are optional and should be edited by removing the # touncomment the line, as required.For boolean parameters, true / false valuesmust be entered in lower case.

Configuring Docker


8. For GDM:

The server setting and data setting values are exposed in the docker compose.yml file.a. With the SERVER_SETTING / DATA_SETTINGvalue set to true and the values defined in the

docker compose file, the database is now updated automatically with the values, upon server start-up.

Note: The Server_Setting values are in JSON array format.

b. Set the value as false if you want to restart the server without making any changes to the SERVER_SETTINGor DATA_ACCESS_SETTING.

9. For GDL:

Configuring the distribution datasources andmapping of datasources.transformUriis done using the docker-compose.yml file. You can provide inputs for these files in the docker-compose file.

Note: For more details onGDL, see theGDLReferenceGuide.

Example:

distribution.datasources=ARGUS_DLNGD,GDL,APX_EL_UK,EEX_ELEC_AUC_INT_FR,EEX_GS_SPOT

datasources.transformUri=ICE_GASOIL=transform://ICE_GASOIL_CUSTOMIZE_PROPERTIES_XSL

Multiple datasources and transformURIs can be entered as a comma separated list in thedistribution-config.xml file.

10. For GDM-GCBRT Bridge:

The required parameters for the GDM-GCBRT Bridge are specified in the docker-compose.yml file.

Example: gdm.jms.brokerurl=http://<machine-ip>:8080/[email protected]=******

Configuring Docker


https://www.datagenic.net/files/GDM 6.0/Section Guides/GDL 6 Section Guide.pdf

11. ForWorkflow:a. Once the image is downloaded, amessage is displayed on the server start up confirming the transfer

of custom libraries into ./gdm-gwf-config/customResources.

b. Once connected to the server, launch theGDMClient, go to theWorkflow node >WorkflowDeployment and deploy theWorkflow_custom.jar that was copied to ./gdm-gwf-config/customResources.

Note: You can copymultiple custom jar files into the ./gdm-gwf-config/customResourcesfolder.

Configuring Docker


#GDM Client Access Server

services:

gdm-cas:

image: <image path>

Location from where the docker image isdownloaded.

restart: unless-stopped

Restart is only done when the container exits.

privileged: true

Privilegedmode allows the container the sameaccess as the process running on the host.

volumes:

#Volumes stored in the host machine

Note: Do not make any changes to these entries.

./gdm-config/configuration: /02-docker-entrypoint/configuration

Location of GDM configuration folder.

./gdm-config/modules: /02-docker-entrypoint/modules

Location of GDMmodules.

./gdm-config/standalone: /02-docker-entrypoint/standalone

Location of GDM standalone folder.


./gdm-config/cli: /02-docker-entrypoint/cli

Location of CLI folder.

Note: The customization.cli file, located in theCLI folder, allows you to run docker in DEBUGmode.

Updating CLI file for Options Data

When a separate database is used for loading Options data, these details need to be included in the CLI file.

# Mark the commands below to be run as a batchbatch## Data source: optionsDSdata-source add \

--jta=false \--jndi-name=java:/datagenic.jdbc.optionsDS \--name=optionsDS \--enabled=true \--use-java-context=true \--connection-url=jdbc:oracle:thin:@${env.OPTION_HOST:localhost}:${env.OPTION_PORT:1521}:${env.OPTION_NAME:gdm}${env.OPTION_OPTIONS:} \--driver-name=oracle \--transaction-isolation=TRANSACTION_READ_COMMITTED \--min-pool-size=${env.DATABASE_MIN_POOL_SIZE:5} \--max-pool-size=${env.DATABASE_MAX_POOL_SIZE:100} \--pool-prefill=true \--pool-use-strict-min=false \--flush-strategy=FailingConnectionOnly \--user-name=${env.OPTION_USERNAME:gdm} \--password=${env.OPTION_PASSWORD:gdm} \--check-valid-connection-sql="select 1 from dual" \--exception-sorter-class-name=org.jboss.jca.adapters.jdbc.extensions.oracle.OracleExceptionSorter \


--stale-connection-checker-class-name=org.jboss.jca.adapters.jdbc.extensions.oracle.OracleStaleConnectionChecker \--valid-connection-checker-class-name=org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker \--blocking-timeout-wait-millis=5000 \--idle-timeout-minutes=1 \--prepared-statements-cache-size=100 \--share-prepared-statements=true

# Execute the batchrun-batch

./config/custom: /opt/jboss/wildfly/custom

Location of custom folder.

./license: /opt/jboss/wildfly/license

Location of license folder.

Note: Ensure that you enter the license folder path in the LIB_SETTING table underLICENSE/LICENSE_PATH

./gdm-logs: /opt/jboss/wildfly/standalone/log

Location of GDM log folder.

./gdm-data/executionLogs: /opt/jboss/wildfly/standalone/datagenic/executionLogs

Location of GDM execution logs folder.

./gdm-data/export: /opt/jboss/wildfly/standalone/datagenic/export

Location of data export files folder.


./gdm-data/gdmxMessage: /opt/jboss/wildfly/standalone/datagenic/gdmxMessage

Location where GDMX messages are collected.

./gdm-data/genicWorkflow: /opt/jboss/wildfly/standalone/datagenic/genicWorkflow

Location of GenicWorkflow folder.

./license_agreement: /opt/jboss/wildfly/standalone/datagenic/docs

Location of license agreement document.

ports:

http_PORT_NUMBER Enter the port numbers for connection:

["HOST PORT_NUMBER]:[CONTAINER PORT_NUMBER"]

e.g. "8080:8080"


Dynamic allocation of port numbers Enter the required port numbers for dynamicallocation.

For Windows:

1. Set the port number in theEnvironmental Variables > SystemVariables of your system.e.g. GDM_OUT_PORT=1111 , GDM_OUT_DEBUG_PORT=2222

2. Add these parameters to the docker-compose file before thedocker-compose up section.

3. Start the server, which will now startusing the exposed port numbers.

Note: You can run the command docker-compose config to check and confirmthe exposed port numbers.

For Linux:

1. Stop the server.

2. Run the following command in theconsole, to set the port number as anenvironment variable:export [env_var_name]=[env_value]e.g. (export GDM_PORT=83) , (exportDEBUG_PORT=8787)

3. Add these parameters in the docker-compose file.

4. Start the server, which will now startusing the exposed port numbers.

Note: You can run the command docker-compose config to check and confirmthe exposed port numbers.


environment:

AUTH_MODULE= Enter the authenticationmodule used:

rdbms

If Kerberos is used, enter the following:

AUTH_MODULE= Enter the authenticationmodule used as:

kerberos, rdbms

KERBEROS_PRINCIPAL= Enter themachine IP:[MACHINE_IP].domain.internal

KERBEROS_KEYTAB= /opt/jboss/wildfly/standalone/configuration/

gdm.machinename.keytab

Location where Keytab file is placed.

If Keycloak is used, enter details for the related parameters.

AUTH_MODULE= Enter the authenticationmodule used as:

keycloak, rdbms

ENTITLEMENTS= The entitlement is set to rdbms.

KEYCLOAK_REALM= Enter the Realm details, where users and any otherapplications will be created:

e.g. preprod

KEYCLOAK_SERVER_URL= Enter the keycloak server URL:

http://[SERVER_URL]/auth


KEYCLOAK_SSL= The keycloak SSL is set toAll.

KEYCLOAK_CLIENT_ID= Enter the client id for the resource server:

e.g. gdm-ab-resource-server

NODE_TYPE For GDM instance, enter the node type as gdm.

STORAGE_DB= Enter the storage database details:

mysql | sqlserver | oracle

DATABASE_HOST= Enter the database host machine IP:

[HOST_MACHINE_IP]

DATABASE_SERVER_NAME= Enter the GDM database server name:

[SERVER_NAME]

DATABASE_PORT= Enter the GDM database port:

[DATABASE_PORT]


DATABASE_NAME= Enter the GDM database name:

[DATABASE_NAME]

For Oracle database only:

You can use either the SID or service name to startthe database.

To create a connection to the database using a SID,you specify the SID.

e.g. DATABASE_NAME = ORCL

To start the container using the service name,specify the service name.

e.g. DATABASE_NAME =orcl.datagenic.internal

ORA_HOST_SEP=//

ORA_SERVICE_SEP=/

For Oracle database only:

This parameter is used tomake a connection usingthe service name, the values for these variables arefixed.

Important: If these variables are not provided,disabled or uncommented, then the connectionis made using SID.

DATABASE_USERNAME= Enter the GDM database username:

[USERNAME]

DATABASE_PASSWORD= Enter the GDM database password:

[PASSWORD]

DATABASE_OPTIONS= Note: MySQL database only.

Enter the UTC timezone details as:

&serverTimezone=UTC


DATA_CACHING= To enable data caching set this variable to true;otherwise set to false.

Database details for Options database

When a separate database is used for loading Options data, the following options database connection detailsneed to be included in the docker-compose file.

OPTION_HOST= Enter the Options database host machine IP:

[HOST_MACHINE_IP]

OPTION_SERVER_NAME= Enter the Options database server name:

[SERVER_NAME]

OPTION_PORT= Enter Options database port:

[DATABASE_PORT]

OPTION_NAME= Enter the Options database name:

[DATABASE_NAME]

OPTION_USERNAME= Enter the Options database username:

[USERNAME]

OPTION_PASSWORD= Enter the Options database password:

[PASSWORD]


DISABLE_HIBERNATE_FORMULAS= To enable reading of formula data, set this to true.

Note: If you set this to false, formula data isnot read.

When set to true, a new property GCB_PD_FORMULA is created automatically, which sendsall formula data to the GCB server.

Note: Once the formula data is sent to the GCBserver, the property value is rolled back tofalse, which will enable the server to send onlythe updated values.

Important: You cannot the delete the GCB_PD_FORMULA property from the dockercompose file.

GWF_SERVER_URL= Enter the GWF server URL:

http://[SERVER_URL]


JAVA_OPTS= Specify the required RAM for the JAVA OPTSenvironment variable:

Note: The port detail included in the parameter allowsyou to start the gdm-cas server using thepreferred port number.

As an example, port 8080 is specified here. Youcan choose to expose any preferred portnumber. The port number specified heremustbe the same as provided underPorts section.

e.g. -XX:MinRAMPercentage=60.0 -XX:MaxRAMPercentage=80.0Djava.net.preferIPv4Stack=true -Djboss.http.port=8080

Where:

-XX:MinRAMPercentage is theminimum JVMheap size, set as a percentage of the total memory.

-XX:MaxRAMPercentage is themaximum Javaheap size, set as a percentage of the total memory.

Important: The values for XX:MinRAMPercentage and -XX:MaxRAMPercentagemust be provided as adecimal value.

It is recommended that you do not enter thevalue for -XX:MaxRAMPercentage 100%.

Use the command $docker stats to viewthememory usage.

WEBSOCKET_SERVER URL= Enter the application server IP address and port.

ws://[IP address:port]


MAX_HEAP_PERCENTAGE= Enter a value, to check the database and heapmemory health.

e.g. MAX_HEAP_PERCENTAGE=70

Note: By default, theMAX_HEAP_PERCENTAGE to which the JAVA heap can grow is set to 90%.

#Distributed cache settings

l To enable distributed caching, when you havemultiple GDMCAS nodes in a single docker file.

SetDISTRIBUTED_CACHE to true and enter details for the related parameters.

DISTRIBUTED_CACHE= To enable distributed caching, set this variable totrue; otherwise set to false.

DISTRIBUTED_CACHE_BIND_ADDRESS= If distributed cache setting is set to true, enter thedistributed cache name:

e.g. gdm-cas

DISTRIBUTED_CACHE_EXT_ADDRESS= Enter the external address:

e.g. gdm-gwf

DISTRIBUTED_CACHE_HOSTS= If distributed cache setting is set to true, enter thedistributed cache host address:

e.g. [gdm-cas[7800], [gdm-gwf[7801]

Note: gdm-cas is the service (container) forGDM Client Access Server.

gdm-gwf is the service (container) forGWF Client WorkFlow instance.


l To enable distributed caching whenGDM is run inmulti-host environment.

GDM_HOST andGWF_HOST are the host names of themachine where gdm-cas and gdm-gwfservices (containers) are running, respectively.

DISTRIBUTED_CACHE_EXT_ADDRESS= To enable distributed caching for GDM onmulti hostenvironment, enter the external address:

e.g. GDM_HOST

This is themachine name or the IP address of themachine where GDM server is running.


e.g. [GDM_HOST[7800], [GWF_HOST[7801]

hostname= Enter the IP address of themachine where gdm-cascontainer is run.

e.g. [MACHINE_IP]

SERVER_SETTING

SERVER_SETTING= TRUE

Important: For server settings, the combination of Domain and Key details are unique. If a value is notdefined for a server setting key, then the result will be a null pointer exception. Hence, ensure a value isentered in this field.

Domain Key Value Description

CLUSTER CLUSTER_ENVIRONMENT

0 To identify the clusterenvironment.



DATA_DOMAIN DEFAULT_ACCESSOR

com.datagenic.data.accessor

.rdbms.TimeSeriesAccessor

The default accessorsetting.

GDMX_DOMAIN SUCCESS_PATH gdmMessage/success Location wheresuccessfully loadedGDMX files are stored.

GDMX_DOMAIN FAILURE_PATH gdmMessage/fail Location where failedGDMX files are stored.

LICENSE LICENSE_PATH /opt/jboss/wildfly/license Location ofgdmLicense.txt file.

LICENSEAGREEMENT

LICENSEAGREEMENT_DOMAIN_PATH

C://datagenic/docs Location of the licenseagreement document.

MATRIX MATRIXSIZE 250 Maximum number ofmatrix columns androws allowed.

NOTIFICATION TIMER 1800 To set the time (inseconds) to clear thenotification cache.

SECURITY_DOMAIN

SECURTIY_ADMINGROUP

Administrators The policy group namemapped to theadministrator.

SECURITY_DOMAIN

SECURITY_PERMISSIONS_CACHE_UPDATETIME

Security setting to loadthe user permissioncache on cluster nodes.



SECURITY_DOMAIN

SECURITY_PASSWORD_MINLEN

5 Security settingenforcing users to keepminimum passwordlength between 5 and30.

SMTP SMTP_PASS *** SMTP password.

SMTP SMTP_USER rsvp SMTP user name.

SMTP SMTP_FROM [email protected]

From SMTP emailaddress.

SMTP SMTP_PORT 25 SMTP server portnumber.

SMTP SMTP_HOST mail.domain.com SMTP server hostname.

SMTP SMTP_TLS 0 SMTP emailconfiguration to enableTransport Layer Security(TLS) protocol.

TEMPLATE_DOMAIN

DOCUMENT_PATH templateDocuments/ Location where filesadded to documentproperty of models arestored.

WORKFLOW_DOMAIN

EXECUTION_LOGS genicWorkfow/executionLogs

Location of executionlog files.



WORKFLOW_DOMAIN

STARTUP_DIR genicWorkflow/startUp/ The workflow serverreads this directorywhen theGDM server isstarted. Any<filename>,deployment.properties stored in thedirectory is read. Whendeployments areupdated, they areautomatically re-deployed.

WORKFLOW_DOMAIN

DEPLOYMENT_JARS

/opt/jboss/wildfly/standalone/datagenic/genicWorkflow

Location wheredeployment jars arestored.

WORKFLOW_DOMAIN

TMP_DIR genicWorkflow/tmpDir/ Location wheretransformation files suchas input files, csv filesare stored.

WORKFLOW_DOMAIN

JSP_PATH genicWorkflow/GWF/ Location of JSP formsused in workflow taskthat are saved anddisplayed in the GDMClient.

DATA_ACCESS_SETTING

Important: If a value is not defined for a data access setting key, then the result will be a null pointerexception. Hence, ensure a value is entered in this field.

DATA__ACCESS_SETTING= TRUE


Key Value

GCB_RT_SERVER_URL Enter the GCBRT server url:

http://localhost:8080/gcb-sa

GCB_RT_SERVER_USERNAME Enter the GCBRT server username:

e.g adminuser@domain

GCB_RT_SERVER_PASSWORD Enter the GCBRT server password:

#$123enDG

USE_GCB_RT_CREDENTIALS Set the GCBRT credentials to true.

MV_SERVER Enter theMarketView server url:

mvmts6gvsi.com

MV_USER Enter MV user name:

e.g. dg_sa

MV_PASSWORD Enter theMV password:

e.g S34!fghole

MV_ACCESSOR Enter theMV accessor details:

com.datagenic.data.accessor.marketView.MVTimeSeriesAccessor

MV_RETRIES (optional) Enter the server retries applicable for MV:

e.g. 3


command:

By default, the GDM Docker server is started in INFO mode.

Important: If you are updating the docker compose file and its version is prior to 6.7.0, ensure that thelatest customization.cli file is replaced in gdm-config/cli folder.

---debug

Mode in which to start the server

DEBUG_LEVEL= INFO (For INFOmode)

DEBUG_LEVEL= DEBUG (For DEBUGmode)


#GWF ClientWorkFlow instance

services:

gdm-gwf:

image: <image path>




privileged: true


volumes:



./gdm-gwf-config/configuration: /opt/jboss/wildfly/bin/configuration

Location of GWF configuration folder.

./gdm-gwf-config/custom: /opt/jboss/wildfly/custom

Location of GWF custom folder.

./gdm-gwf-config/cli: /02-docker-entrypoint/cli

Location of the cli folder.


./gdm-gwf-config/customResources:

Location where the custom jars are placedmanually, on the selected local drive.

/02-docker-entrypoint/lib/lib

Location to copy the custom jar to gdm-full-ear /libfolder.

./license: /opt/jboss/wildfly/license

Location of license folder.

Note: Ensure that you enter the license folder path in the LIB_SETTING table underLICENSE/LICENSE_PATH

./gdm-gwf-logs: /opt/jboss/wildfly/standalone/log

Location of GWF log folder.

./gdm-gwf-data/executionLogs: /opt/jboss/wildfly/standalone/datagenic/executionLogs

Location of GWF execution log folder.

./gdm-gwf-data/export: /opt/jboss/wildfly/standalone/datagenic/export

Location of data export files folder.

./gdm-gwf-data/gdmxMessage: /opt/jboss/wildfly/standalone/datagenic/gdmxMessageLocation where GDMX messages are collected.

./gdm-gwf-data/genicWorkflow: /opt/jboss/wildfly/standalone/datagenic/genicWorkflow

Location of GWF folder.


ports:



e.g. "8080:8080"

environment:

AUTH_MODULE= Enter the authenticationmodule used:

rdbms

NODE_TYPE= Enter the node type.

For GWF instance, enter gwf

STORAGE_DB= Enter the storage database details:

mysql | sqlserver | oracle

DATABASE_HOST= Enter the database host machine IP:

[HOST_MACHINE_IP]

DATABASE_SERVER_NAME= Enter the GDM database server name:

[SERVER_NAME]

DATABASE_PORT= Enter the GDM database port:

[DATABASE_PORT]


DATABASE_NAME= Enter the GDM database name:

[DATABASE_NAME]

Note: In case of Oracle database SID must beused instead of the database name.

DATABASE_USERNAME= Enter the GDM database username:

[USERNAME]

DATABASE_PASSWORD= Enter the GDM database password:

[PASSWORD]

DATABASE_OPTIONS= Note: MySQL database only.

Enter the UTC timezone details as:

&serverTimezone=UTC

DATA_CACHING= To enable data caching set this variable to true;otherwise set to false.

GDM_SERVER_URL= Enter the GDM server URL:

http://[SERVER_URL]


JAVA_OPTS= Specify the required RAM for the JAVA OPTS environmentvariable:

e.g. -XX:MinRAMPercentage=60.0 -XX:MaxRAMPercentage=80.0Djava.net.preferIPv4Stack=true

Where,

-XX:MinRAMPercentage is theminimum JVM heap size, setas a percentage of the total memory.

-XX:MaxRAMPercentage is themaximum Java heap size, setas a percentage of the total memory.

Important: The values for XX:MinRAMPercentage and -XX:MaxRAMPercentagemust be provided as a decimalvalue.

It is recommended that you do not enter the value for -XX:MaxRAMPercentage 100%.

Use command $docker stats, to view thememoryusage.

WEBSOCKET_SERVER URL= Enter the application server IP address and port.



#Distributed cache settings

l To enable distributed caching, when you havemultiple GWFCAS nodes in single docker file.

SetDISTRIBUTED_CACHE to true and enter details for the related parameters.

DISTRIBUTE_DCACHE= To enable distributed caching, set this variable totrue; otherwise set to false.

DISTRIBUTED_CACHE_BIND_ADDRESS= If distributed cache setting is set to true, enter thedistributed cache bind address:

e.g. gdm-gwf

DISTRIBUTED_CACHE_EXT_ADDRESS= Enter the external address:

e.g. gdm-gwf


e.g. gdm-cas[7800], gdm-gwf[7801]

Note: gdm-cas is the service (container) for GDMClient Access Server

gdm-gwf is the service (container) for GWFClientWorkflow Instance


l To enable distributed caching whenGWF is run inmulti-host environment.

GWF_HOST andGDM_HOST are the host names of themachineswhere the gdm-cas and gdm-gwfservices (containers) are running, respectively.

DISTRIBUTED_CACHE_BIND_ADDRESS=

If distributed cache setting is set to true, enter thedistributed cache bind address:

e.g. gdm-gwf

DISTRIBUTED_CACHE_EXT_ADDRESS= To enable distributed caching for GDM onmulti-hostenvironment, enter the address:

e.g. GWF_HOST

This is themachine name or the IP address of themachine where GWF server is running.

DISTRIBUTED_CACHE_HOSTS= If distributed cache setting (DISTRIBUTED_CACHE) is set to true, enter the distributed cachehost address:e.g. [GDM_HOST[7800], [GWF_HOST[7801]

command:

By default, the GWF docker server is started in INFO mode.

Important: If you are updating the docker compose file and its version is prior to 6.7.0, ensure that thelatest customization.cli file is replaced in gdm-config/cli folder.

---debug

Mode in which to start the server

DEBUG_LEVEL= INFO (For INFOmode)

DEBUG_LEVEL= DEBUG(For DEBUGmode)


networks:

gdmnet To connect a container to the network.

Default network is gdmnet.


#GDL Genic Data Loader instance

services:

genic-loader:

image: <image path>




privileged: true


environment:

l Specify the parameters for the appropriate database, e.g. SQL, Oracle or MySQL.

server.version The server version.


SQL

sql.gdmDS.driverClassName= The SQLGDMDS driver class name:com.microsoft.sqlserver.jdbc.SQLServerDriver

sql.gdmDS.url= In the SQLGDMDS URL, enter themachine IP andthe database name:

jdbc:sql://[MACHINE_IP]:3306/[DATABASE_NAME]?useSSL=false&useOldAliasMetadataBehavior=true&pinGlobalTxToPhysicalConnection=true

sql.gdmDS.username= Enter the username for the GDM DS SQL database:[USERNAME]

sql.gdmDS.password= Enter the GDM DS SQL database password:[PASSWORD]

sql.templateDS.driverClassName= The template DS driver class name:[com.sql.jdbc.Driver]

sql.templateDS.url= In the SQL template DS URL, enter themachine IPand the database name:

jdbc:sql://[MACHINE_IP]:3306/[DATABASE_NAME]?useSSL=false&useOldAliasMetadataBehavior=true&pinGlobalTxToPhysicalConnection=true

sql.templateDS.username= Enter the username for the SQL template DS:[USERNAME]

sql.templateDS.password= Enter the password for the SQL template DS:[PASSWORD]


Oracle

sql.gdmDS.driverClassName TheOracle driver class name:[oracle.jdbc.driver.OracleDriver]

sql.gdmDS.url Enter the Oracle GDMDS URLmachine IP and thedatabase name:jdbc:oracle:thin:@[MACHINE _ IP]:1521:[DATABASE_NAME]

sql.gdmDS.username Enter the username for the GDM DSORACLE database:[USERNAME]

sql.gdmDS.password Enter the GDM DS ORACLE database password:[PASSWORD]

sql.templateDS.driverClassName TheOracle template driver class name:[oracle.jdbc.driver.OracleDriver]

sql.templateDS.url Enter the ORACLE template DS URLmachine IPand the database name:jdbc:oracle:thin:@[MACHINE_IP]:1521:[DATABASE_NAME]

sql.templateDS.username Enter the ORACLE template DS username:[USERNAME]

sql.templateDS.password Enter the ORACLE template DS password:[PASSWORD]


Oracle for Options Data

The optionDS variables need to be updated when a separate database is used for loading Options data. Beloware the optionDS connection details to be included in the docker-compose file.

sql.optionDS.driverClassName= TheOptions DS driver class name.

[oracle.jdbc.driver.OracleDriver]

sql.optionDS.url= In the SQL options DS URL, enter themachine IPand the database name:

jdbc:oracle:thin:@[MACHINE_IP]:1521:[DATABASE_NAME]

sql.optionDS.username= Enter the username for the SQLoption DS:

[USERNAME]

sql.optionDS.password= Enter the password for the SQLoption DS:

[PASSWORD]


MySQL

sql.gdmDS.driverClassName TheMYSQLGDMDS driver class name.[com.mysql.jdbc.Driver]

sql.gdmDS.url TheMYSQLGDMDS URL:

jdbc:mysql://[MACHINE_IP]:3306/[DATABASE_NAME]?useSSL=false&useOldAliasMetadataBehavior=true&pinGlobalTxToPhysicalConnection=true

sql.gdmDS.username Enter the username for the GDM DSMYSQL database:[USERNAME]

sql.gdmDS.password Enter the GDM DS MYSQL database password:[PASSWORD]

sql.templateDS.driverClassName The template DS driver class name.[com.mysql.jdbc.Driver]

sql.templateDS.url TheMYSQL template DS URL:

jdbc:mysql://[MACHINE_IP]:3306/[DATABASE_NAME]?useSSL=false&useOldAliasMetadataBehavior=true&pinGlobalTxToPhysicalConnection=true

sql.templateDS.username Enter the username for theMYSQL template DS:[USERNAME]

sql.templateDS.password Enter the password for theMYSQL template DS:[PASSWORD]


Configuring GDL for FTP and SFTP

UsingGDL, you can download the files from FTP server in three ways:

l Using FTP

l Using SFTP

l Using SFTP with proxy

Using FTP

ftp.protocol= ftp

ftp.host= Enter the FTP host details:

e.g. ftp.domain.net

ftp.port= Enter the FTP port details:

The port number should be entered as 21.

ftp.user= Enter the FTP username:

e.g. dataload_Dev

ftp.password= Enter the FTP password:

e.g. 7HuhhKcLVNDkXmyV

ftp.inbox= This is the path from where GDL downloads the filefrom the FTP server.

e.g. /inbound

Note: The directory name should start with /andmust not end with ./


ftp.failed= This is the path on the FTP server where the failedfiles are stored.

e.g. /outbound/failed

Note: The directory name should start with /andmust not end with ./.

ftp.scheduler.cron= Enter a cron expression to schedule theFTP download:

e.g. 1+*+*+*+*+?

ftp.connection.info= passiveMode=true

Note: This is an optional parameter that can beentered when using the FTP protocol.

Using SFTP

ftp.protocol= sftp

ftp.port= Enter the SFTP port details

The port number should be entered as 22

ftp.host= Enter the server hostname or IP address:

e.g. ftp.domain.net

ftp.user Enter the user name

e.g.MACHINE_USERNAME

ftp.password= Enter the password.

e.gMACHINE_PASSWORD


ftp.inbox This is the path from which GDL downloads the filefrom the SFTP server.

e.g./inbound

Note: The directory name should start with /andmust not end with ./

ftp.failed This is the path on the SFTP server where the failedfiles are stored.

e.g. /outbound/failed

Note: The directory name should start with /andmust not end with ./.

ftp.scheduler.cron= Enter a cron expression to schedule:

e.g. 1+*+*+*+*+?

ftp.connection.info- passiveMode=true&knownHostsFile=/opt/gdl-standalone/config/.ssh/known_hosts

Note: This is amandatory parameter whenusing SFTP protocol.

Using SFTP with Proxy Server

Important: These parameters are only to be used where data is loaded from GDL using a proxy server. Otherwise,these parameters must be commented out.

ftp.protocol= sftp

sftp.port= Enter the SFTP port details.

The port number should be entered as 22.


ftp.connection.info= passiveMode=true&knownHostsFile=/opt/gdl-standalone/config/.ssh/known_hosts&proxy=#proxyHttp

Note: This is amandatory parameter.

ftp.proxy.host= Enter the proxy server hostname or IP address:

e.g.ftp.domain.net

ftp.proxy.port Enter the proxy server port:

e.g. 808

ftp.proxy.username= Enter the user name for the proxy machine:

e.g. PROXY_MACHINE_USERNAME

Note: If you are connecting to proxy serverwithout a username and password, thisparameter should be commented out.

ftp.proxy.password= Enter the password for the proxy machine:

e.g. PROXY_MACHINE_PASSWORD

Note: If you are connecting to proxy serverwithout a username and password, thisparameter should be commented out.

gdm.server.url= Enter the GDM server URL:

http://[MACHINE_IP]:8080/wildfly-services

gdm.server.user= Enter the GDM server user name:

[USERNAME]


gdm.server.password= Enter the GDM server password:

[PASSWORD]

websocket url= Enter the GDM application server IP address andport:


env_name= Enter the environment variable name:e. g: SUPERFAST_LOADER.

The environment namewill bemapped in the gdl-standalone-server.xml and gdl.properties files tomonitor the data loaded from GDL, using the GWSweb application.

log_type= Enter the log type as: DEBUG.

This will generate a detailed log report of the serveraction.

Note: The default log_type is INFO.

webmonitoring_url= Enter the GDM application server url:

http://server_url:port




Where,






email.from= Enter the from email address:

e.g. [email protected]

email.to= Enter the recipient's email address:

e.g. [email protected]

email.subject= Enter the subject of the email:

e.g. GDL Loader Failed Status

email.contentType= Enter the email content format:

e.g. text/plain;charset=UTF-8


enable_data_caching= To enable data caching, set this variable to true;otherwise set to false.

By default, this option is set to true.

Important: Ensure that the DATA_CACHINGparameter in GDM is also set to true.

enable_dataresource_caching= To enable caching of data resource set this variableto true; otherwise set to false.

By default, this option is set to false.

enable_liveupdate= To receive live updates, set this variable to true;otherwise set to false.


enable_notification To receive notifications from GDL, set this variableto true; otherwise set to false.

Important: With this option set to false,notifications are not triggered, even whenconfigured.


enable_qualitycheck= To receive quality checks from GDL, set thisvariable to true; otherwise set to false.

Important: With this option set to false, qualitychecks are not triggered, even whenconfigured.



enable_email= To receive email updates from GDL, set this variableto true; otherwise set to false.

Important: With this option set to false, noemails are received, even if all required emailsettings are configured.



When a file is processed, the following folders are created:

l failedDelivery: When a file fails to load, a failedDelivery folder is created. This folder will include a .csvfile containing the file name, client name, datasource, failed delivery date, Ftp path and error message.

l delivery: When a file is processed successfully, a folder called delivery is created. This folder will includea .csv file containing the filename, start time, end time and the actual time taken to load the file.

l .error: When the file processing has failed, a .error folder is created where all the files that have failed tobe processed are stored.

l .done: When a file is processed successfully, a .done folder is created where all the files that aresuccessfully processed are stored.

archiveLoadedFile= To archive files that are successfully processed intothe .done folder, set this variable to true.

Note: When set to false, loaded files aredeleted and they will not be available in the.done folder.

By default, this option is set to true.

uploadFailedFileToFtp= To upload the failed files to FTP, set this variable totrue. Once the files are uploaded to FTP, the filesfrom the local directory will be deleted.

Important: Once failed files are uploaded to thefailed folder in FTP, these files will not beprocessed. Manual action is required toprocess the files.

When set to false, failed files will not be uploaded toFTP and will remain in the local .error directory.


Important: An exceptionmessage is displayedif user with insufficient permission try to accessthe failed folder present in the system. Usershould be provided with enough permission tocreate it.


TZ= Enter the timezone details:

e.g. UTC.

distribution.datasources= Enter the datasources values; comma separated:

e.g. distribution.datasources=ARGUS_DLNGD,GDL,APX_EL_UK,EEX_ELEC_AUC_INT_FR,EEX_GS_SPOT

datasources.transformUri= Enter the transform values; comma separated.

e.g. datasources.transformURI=ICE_GASOIL=transform://ICE_GASOIL_CUSTOMIZE_PROPERTIES_XSL

Specific to the Linux environment, in addition todefining the datasource transformUri, youcan definemultiple inputs by extending the uris fromdatasources.transformUri1 todatasources.transformUrinth.

Note: The sequence should run from 1 to n.

e.g.

datasources.transformUri=APX_GTS=transform://CUSTOM_MAPPING

datasources.transformUri1=APX_GTS_CORR=transform://CUSTOM_MAPPING

datasources.transformUri2=APX_GTS_HIST=transform://CUSTOM_MAPPING

datasources.transformUri3=APX_NBP=transform://CUSTOM_MAPPING

datasources.transformUri4=APX_NBP_CORR=transform://CUSTOM_MAPPING

datasources.transformUri5=ARGUS_DAEE_HIST=transform://CUSTOM_MAPPING


ports:



e.g. "8080:8080"

command:

By default, the GDL Docker server is started in INFO mode.

l If required, change the command to start the server in a different mode.

volumes


./gdl-conf/ /opt/gdl-standalone/config:ZLocation of GDL config folder.

Add the datasource-config.xml file in ./gdl-confdirectory. This file will be provided as a part of therelease package.

Note: Prefix transformation is now added to theGDMX files loaded throughGDL. For moredetails, refer to the GDM 6.2.0 Release Notes.

./gdl-feedgdmx/ Location of feedgdmx folder.

/opt/gdl-standalone/feedgdmx:Z

./gdl-logs/ Location of GDL log folder.

/opt/gdl-standalone/log:Z


networks:




#GDM - GCBRT Bridge

services:

gdm-gcbrt-bridge:

image: <image path>




privileged: true


environment:

gdm.server.url= Enter the GDM server URL:

http://[MACHINE_IP]:8080/wildfly-services

gdm.server.user= Enter the GDM server user name:

[USERNAME]

gdm.server.password= Enter the GDM server password:

[PASSWORD]

gcb.amq.brokerurl= Enter the GCBRT ActiveMQ broker URL:

tcp://[MACHINE_IP]:[port number]

gcb.amq.user= Enter the GCBRT ActiveMQ username:

[USERNAME]


gcb.amq.password= Enter the GCBRT ActiveMQ password:

[PASSWORD]

gdm.jms.brokerurl= Enter the GDM JMS broker URL:

tcp://[MACHINE_IP]:[port number]

gdm.jms.user= Enter the GDM JMS username:

[USERNAME]

gdm.jms.password= Enter the GDM JMS password:[PASSWORD]

company= Enter the company name:

e.g. DATAGENIC

gcb.restUrl= Enter the GCBRT server REST URL:

http://[MACHINE_IP]:[PORT_NUMBER]/gcb-sa-rs

gcb.userName= Enter the GCB user name:

[USERNAME]

gcb.password= Enter the GCB password:

[PASSWORD]




Where,






log_type= Enter the log type as: DEBUG.

This will generate a detailed log report of the serveraction.

Note: The default log_type is INFO.

ports:



e.g. "8080:8080"

command:

By default, the GDM-GCBRT Docker server is started in INFO mode.

l If required, change the command to start the server in a different mode.


volumes:



./gcbrt-bridge-conf/: /opt/gdm-gcbrt-bridge/config:Z

Location of GCBRT configuration folder.

Note: If you are updating the docker composefile and its version is prior to 6.6.0, ensure thatthe latest gcb-bridge.xml file is replaced ingcbrt-bridge-conf folder.

./gcbrt-bridge-logs/: /opt/gdm-gcbrt-bridge/logs:Z

Location of GCBRT log folder.

networks:




Genic DataManager 6.7.0 Docker Configuration Guide 6.0/Install/GDM... · SECURITY_ DOMAIN SECURTIY_...

Documents

Transcript of Genic DataManager 6.7.0 Docker Configuration Guide 6.0/Install/GDM... · SECURITY_ DOMAIN SECURTIY_...