Visual Importer Professional user manual - ETL tools Professional User Manual.pdf · Visual...

Visual Importer ETL

Professional & Enterprise

User manual

This Documentation is no longer maintained

All documentation is available online here:

http://www.etl-tools.com/wiki/start

DB Software Laboratory 2008-2015

www.etl-tools.com

http://www.etl-tools.com/wiki/start

Visual Importer ETL Professional & Enterprise User Manual

Copyright 2015 DB Software Laboratory Page 2 of 305

Contents

Contents ....................................................................................................................................... 2

1. Introduction ............................................................................................................................. 8

2. Requirements ........................................................................................................................... 8

3. Key features ........................................................................................................................... 10

4. Visual Importer ETL Architecture ......................................................................................... 12

5. Demo Data ............................................................................................................................. 13

6. Options .................................................................................................................................. 15

6.1 Global Variables .............................................................................................................. 21

7. User Interface ........................................................................................................................ 22

7.1 System Menu ................................................................................................................... 22

7.2 Visual Importer ETL Graphical User Interface: Main window ....................................... 22

7.2.1 Quick Access Tabs ................................................................................................... 22

7.2.2 Objects tree toolbar ................................................................................................... 23

7.2.3 Objects tree description ............................................................................................ 23

7.2.4 Objects tree list ......................................................................................................... 24

7.2.5 Objects Groups ......................................................................................................... 26

8. Directories and Connections .................................................................................................. 29

8.1 Directories ....................................................................................................................... 29

8.2 Database Connections ..................................................................................................... 30

8.2.1 Oracle Connection .................................................................................................... 31

8.2.2 JDBC Connection ..................................................................................................... 32

8.2.3 Microsoft SQL Server Connection ........................................................................... 33

8.2.4 ODBC Connections .................................................................................................. 34

8.2.5 ODBC Connection Strings ....................................................................................... 36

8.2.6 MySQL Connections ................................................................................................ 37

8.2.7 PostgreSQL Connection ........................................................................................... 39

8.2.8 Interbase/Firebird Connection .................................................................................. 41

8.2.9 Ole DB Connection .................................................................................................. 43

8.2.10 SQLite Connection ................................................................................................. 44

8.2.11 Microsoft SQL Server CE Connection ................................................................... 45

8.2.12 BDE Connection ..................................................................................................... 46

8.3 POP3 Connection ............................................................................................................ 49

8.4 IMAP4 Connection .......................................................................................................... 50

8.5 SMTP Connection ........................................................................................................... 51

8.6 IP Socket connection ....................................................................................................... 52

8.7 FTP Connection ............................................................................................................... 53

8.8 Google Spread sheet connection...................................................................................... 56

8.9 SalesForce connection ..................................................................................................... 57

8.11 SMS Server connection ................................................................................................. 61

9. Importing Data ....................................................................................................................... 63

9.1 Mapping editor screen overview ..................................................................................... 65

9.1.1 Main tool bar ............................................................................................................ 66

9.1.2 Source tool bar .......................................................................................................... 66

9.1.3 Mapping panel .......................................................................................................... 67

9.1.4 SQL Statements ........................................................................................................ 68

9.1.5 Template tab ............................................................................................................. 69

9.1.6 Log File tab ............................................................................................................... 70



9.1.7 Rejected Records Tab ............................................................................................... 71

9.1.8 Import Process .......................................................................................................... 72

9.2 Data Mapping for a Flat File Data Source ....................................................................... 73

9.3 How to clear mapping ..................................................................................................... 79

9.4 Auto mapping .................................................................................................................. 80

9.5 Data Mapping for an ODBC Data Source ....................................................................... 83

9.7 Loading data from the pivot tables ................................................................................. 87

9.8 Performing Calculations .................................................................................................. 89

9.9 Filtering Records ............................................................................................................. 90

9.10 Working with Date fields .............................................................................................. 91

9.11 How to Update/Delete Records ..................................................................................... 92

9.12 Using Database Specific functions. ............................................................................... 94

9.13 Error Handling ............................................................................................................... 95

9.14. Date formats ................................................................................................................. 96

10. Data sources ......................................................................................................................... 97

10.1 Data source is a XML file .............................................................................................. 97

10.2 Data source is Excel file ................................................................................................ 99

10.3 Data source is Qvx file ................................................................................................ 101

10.4 Data source is Windows Event Log ............................................................................ 102

10.5 Data source is MS Access ........................................................................................... 103

10.6 Data source is a DBF File ............................................................................................ 108

10.7 Data source is a ODBC Connection ............................................................................ 109

10.8 Data source is a JDBC Connection ............................................................................. 111

10.9 Data source is an Ole DB Connection ......................................................................... 112

10.10 Data source is a MS SQL Server ............................................................................... 113

10.11 Data source is MS SQL Server CE ............................................................................ 114

10.12 Data source is an Oracle Database ............................................................................ 115

10.13 Data source is an MySql ............................................................................................ 116

10.14 Data source is an PostgreSQL Database .................................................................... 117

10.15 Data source is an Interbase or Firebird Database ...................................................... 118

10.16 Data source is SQLIte ................................................................................................ 119

10.17 Data source is Google Spread Sheets ........................................................................ 120

10.18 Data source is RSS feed ............................................................................................ 121

10.19 Data source is SalesForce .......................................................................................... 122

10.20 Data source is BrightPearl ......................................................................................... 123

10.21 Data source is BDE ................................................................................................... 126

11. Data targets ........................................................................................................................ 127

11.1 Target type is ODBC connection ................................................................................. 127

11.2 Target type is Ole DB .................................................................................................. 129

11.3 Target type is SQLIte ................................................................................................... 130

11.4 Target type is MS SQL Server ..................................................................................... 131

11.5 Target type is MS SQL Server CE .............................................................................. 133

11.6 Target type is Oracle .................................................................................................... 134

11.7 Target type is MySql ................................................................................................... 136

11.8 Target type is PostgreSQL ........................................................................................... 137

11.9 Target type is Interbase/Firebird .................................................................................. 138

11.10 Target type is BDE .................................................................................................... 139

11.11 Target type is a stored procedure ............................................................................... 140

12. SQL Scripts ....................................................................................................................... 141



12.1 SQL Scripts Tool Bar .................................................................................................. 143

12. Reports ............................................................................................................................... 144

12.1 Report Designer ........................................................................................................... 144

12.1.1 Report Title ........................................................................................................... 145

12.1.2 Page Header .......................................................................................................... 145

12.1.3 Group Header ....................................................................................................... 145

12.1.4 Code ...................................................................................................................... 145

12.1.5 Data....................................................................................................................... 145

12.1.6 Report tool bar ...................................................................................................... 146

12.1.7 Report Properties .................................................................................................. 146

12.1.8 Save Report to the Repository .............................................................................. 146

12.1.9 Report Connection ................................................................................................ 146

12.1.10 Show Only Connections for Current Project ...................................................... 146

12.1.11 Manage Versions ................................................................................................ 147

12.1.12 Add Versions ...................................................................................................... 147

12.1.13 Creating Basic Report ......................................................................................... 148

12.1.14 Report in Report Designer .................................................................................. 154

12.1.15 passing parameters into the Report ..................................................................... 156

13. Email Templates ................................................................................................................ 159

14. Packages ............................................................................................................................ 161

14.1 Package screen overview ............................................................................................. 163

14.2 Joining Actions ............................................................................................................ 164

14.3 Package Tool bar ......................................................................................................... 165

14.4 Debugging Package ..................................................................................................... 166

14.6 Working with filenames and directories ...................................................................... 172

14.7 Import Action .............................................................................................................. 173

14.8 SQL Data Check Action .............................................................................................. 174

14.9 SQL Script Action ....................................................................................................... 175

14.10 Package Action .......................................................................................................... 176

14.11 FTP Action ................................................................................................................ 177

14.12 HTTP Download Action............................................................................................ 180

14.13 Archive Action .......................................................................................................... 181

14.14 Receive Email POP3 Action ..................................................................................... 183

14.15 Receive Email IMAP4 Action ................................................................................... 186

14.17 External Application Action ...................................................................................... 191

14.18 Check file Action ....................................................................................................... 192

14.19 File Operation Action ................................................................................................ 193

14.20 Compare Files Action ................................................................................................ 195

14.21 Check Disk Space Action .......................................................................................... 196

14.22 Backup Repository Action ......................................................................................... 197

14.23 Pause Action .............................................................................................................. 198

14.24 Check date action ...................................................................................................... 199

14.25 Report Action ............................................................................................................ 200

14.26 Export Data Action .................................................................................................... 202

14.27 Show Message and Ask Question Action ................................................................. 206

14.28 Set Variable Action ................................................................................................... 208

14.29 Script Action .............................................................................................................. 210

14.30 Check Connection Action.......................................................................................... 213

14.31 PING Server Action.................................................................................................. 214



14.32 Case Action ............................................................................................................... 215

14.33 GPG4WIN Action ..................................................................................................... 217

14.34 HTTP Form Action.................................................................................................... 219

14.35 SMS Action ............................................................................................................... 221

14.36 Loop action ................................................................................................................ 223

15. Lookups ............................................................................................................................. 225

15.1 Tool bar ....................................................................................................................... 226

16. Scheduler ........................................................................................................................... 227

16.1 Tool bar ....................................................................................................................... 227

17. Execution Monitor ............................................................................................................. 232

17.1 Tool bar ....................................................................................................................... 234

17.2 Stopping execution ...................................................................................................... 234

18. SQL .................................................................................................................................... 235

18.1 SQL Toolbar ................................................................................................................ 235

18.2 Side toolbar .................................................................................................................. 236

19. Version Control ................................................................................................................. 237

20. Using Command Line interface ......................................................................................... 239

21. Scripting Language ............................................................................................................ 242

21.1 The Basic Structure ..................................................................................................... 242

21.2 Variables ...................................................................................................................... 243

21.3 Logical Operations ...................................................................................................... 248

21.4 Repeating sets of commands ....................................................................................... 252

21.5 Functions ..................................................................................................................... 255

21.6 Script Examples ........................................................................................................... 256

21.7 Passing variables between objects ............................................................................... 257

21.8 Supported Functions List ............................................................................................. 258

21.8.1 String Functions.................................................................................................... 258

21.8.2 Conversion Functions ........................................................................................... 264

21.8.3 File System Functions .......................................................................................... 266

21.8.4 Date and Time Functions ...................................................................................... 270

21.8.5 Numeric Functions ............................................................................................... 274

21.8.6 Miscellaneous Functions ...................................................................................... 276

21.8.7 Procedures ............................................................................................................ 278

21.8.8 Math functions ...................................................................................................... 279

22. Execution Agent ................................................................................................................ 284

Parallel Execution ................................................................................................................ 285

Checking Agent Status ........................................................................................................ 286

Agent Email Notifications ................................................................................................... 286

23. Repository database ........................................................................................................... 287

Transferring data between repositories ................................................................................ 287

Repository objects synchronization ..................................................................................... 288

Creating new repository ....................................................................................................... 290

ODBC Settings for PostgreSQL repository ......................................................................... 291

ODBC Settings for MySQL repository ............................................................................... 293

ODBC Settings for Interbase/Firebird repository ................................................................ 297

24. Management Console ........................................................................................................ 298

25. Support Procedure ............................................................................................................. 299

26. Dynamic file names ........................................................................................................... 300

27. Objects Library .................................................................................................................. 301



28. Registering Software ......................................................................................................... 302

29. License Agreement ............................................................................................................ 303



Copyright

Copyright 2008-2015 DB Software Laboratory Limited. All rights reserved.

No portion of this document may be reproduced, transmitted, transcribed, stored in

a retrieval system, or translated into any language, in any form or by any means, without prior written consent of DB Software Laboratory Limited.

Note to U.S. Government users: Documentation and programs related to restricted rights - use, duplication or

disclosure is subject to restrictions set forth in: GSA FMSS Schedule Contract No. GS00K92AFS2505-PS05.

License Information

You have access to Visual Importer ETL Professional and Enterprise software and documentation pursuant to the terms of a Software License Agreement granted by DB Software Laboratory Limited. As a user of this software and documentation, you are bound by the terms of the Software License Agreement. All rights, title, and interest to this software remain with DB Software Laboratory Limited.

Requests for copies of this publication and for technical information about DB Software Laboratory products should be made directly to DB Software Laboratory Limited.

Disclaimer All information in this manual is subject to periodic change and revision without

notice. While every effort has been made to ensure that this manual is accurate, DB Software Laboratory Limited excludes its liability for errors or inaccuracies (if any) contained herein.

Registered Marks

Any products or services mentioned or depicted in this document are identified by the trademarks or service marks of their respective companies or organisations.

Edition Information

This document refers to Visual Importer ETL Professional and Enterprise version 8.3.10.36



1. Introduction

Visual Importer is a business intelligence tool that provides an integrated solution for

designing and deploying data warehouses and data marts. It solves complex problem of

integration between different data sources and targets. It combines components of both ETL

and designs tools into one tool.

2. Requirements

Below is the list of Software that must be installed before installation of Visual Importer:

Software Version Notes

Microsoft Windows 98 or higher

MDAC 2.6 or higher Part of OS on W2K,

XP, Vista,Win7,Win8.

MS Access ODBC

driver

4.00.6364.00 or

higher

Part of OS on W2K,

XP, Vista.

Only to work with MS

Access Databases

MS Access 2007

ODBC driver

12.00.4518.1014 or

higher

Separate download Only to work with MS

Access 2007/2010

Databases

FoxPro ODBC

driver

6.1.8629.1 or

higher

Separate download Only to work with

DBF/FoxPro Files

SQL ODBC driver 2000.81.9041.40 Part of OS on W2K,

XP, Vista.

Only to work with MS

SQL Server 7/2008

Interbase client GDS32.DLL Only to work with

Interbase or Firebird

Databases

SQLite Sqlite3.dll Only to work with

SQLite databases

Oracle Client 7.3.4 or higher Provided by Oracle Only to work with

Oracle Databases

BDE Drivers Latest Version Only to work with

Paradox or other BDE

databases

Jet 4.0 Service pack 5 or

higher

Part of OS on W2K,

XP, Vista,Win7,Win8.

To connect to MS

Access repository

Separate Downloads:

FoxPro ODBC driver

http://msdn.microsoft.com/en-us/vfoxpro/bb190233.aspx

Office 2007 Data Access Components

http://www.microsoft.com/downloads/details.aspx?FamilyID=7554F536-8C28-4598-9B72-

EF94E038C891&displaylang=en

http://msdn.microsoft.com/en-us/vfoxpro/bb190233.aspxhttp://www.microsoft.com/downloads/details.aspx?FamilyID=7554F536-8C28-4598-9B72-EF94E038C891&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyID=7554F536-8C28-4598-9B72-EF94E038C891&displaylang=en



BDE

www.etl-tools.com/dmdocuments/BDE.zip

Working with Oracle:

Oracle client 8.1.7 to load data into/from Oracle

Or

Oracle client 9 to load data into/from Oracle

Or


Or


Note:

Depending on the Requirements you may or may not need to have all components installed

There is no need to install clients for MySql and PostgreSQL as they are integrated into the

software itself.

Repository Scripts

Repository Creation scripts are provided for following databases:

MS Access

Oracle

MS Sql Server

Interbase/Firebird

MySQL

PostgreSQL

Note:

Once installation is completed user may find all the scripts in

C:\Users\Public\Documents\DBSL\Repository Scripts\Repository

http://www.etl-tools.com/dmdocuments/BDE.zip



3. Key features

Data import

Data targets:

Oracle 7-11g database (using OCI API)

SQL server 7- 2008 (using BCP API)

ODBC source (using ODBC API)

Interbase/Firebird

MySQL

PostgreSQL

SQLite

Ole DB

BDE

SalesForce

Data sources:

Multiple Delimited or Fixed width Text files

Multiple Excel files + Multiple Excel Spreadsheets

Multiple MS Access Databases + Multiple Tables

Multiple DBF Files

QlikView QVX files

Multiple Tables

Oracle 7-11g database

MS SQL Server

ODBC source

Interbase/Firebird

MySQL

PostgreSQL

SQLite

Ole DB

XML

Windows Event log

Google Spreadsheets

SalesForce

BrightPearl

BDE

Repository:

MS Access

Oracle

MS Sql Server

Interbase

MySQL

PostgreSQL



This product features:

Great performance - thousands of records per second

Comprehensive logging

Rejected records file

Integrated Expressions builder

Filtering Data

Allows the user to perform calculation during the loading process such as splitting fields,

concatenations, data formatting, and loading cross/pivot tables.

Oracle

There are two ways of loading available:

Oracle direct path loading

Conventional path

MS SQL server

This software uses the same API as Microsoft DTS/SSIS service.

ODBC

Any ODBC compliant database.

Other features are:

SQL Scripts

Email Templates

Report Designer

Integrated Scheduler

Logging

Email Notifications

POP3 Email Receiver

IMAP4 Email Receiver

File Operations

Check Files

Check Disk Space

Compare Files

Full range of FTP operations

ZIP

Applications

SQL Data Check

Automatic Backups

Mass Data Export

Http Download

Ping Server

Check Database Connection



4. Visual Importer ETL Architecture



5. Demo Data

In order to provide a flavour of the types of data the Visual Importer ETL can handle, a number of different file types have been provided for demonstration purposes. A range of different types of data files have been provided, in addition to SQL scripts

to create database tables in Oracle, and MS SQL Server.

Text files

Examples provided are using text files in

C:\Users\Public\Documents\DBSL\Demo\Buffer and

C:\Users\Public\Documents\DBSL\Demo\Text Files

Access Databases

C:\Users\Public\Documents\DBSL\Demo\MS Access

DBF Files

C:\Users\Public\Documents\DBSL\Demo\DBF

Excel Files

C:\Users\Public\Documents\DBSL\Demo\Excel

Note:

If user wants to use other locations please amend Directories properties.

Demo tables

Use SQL Scripts provided to create demo tables for Oracle and MS SQL Server. Please adjust connection details before executing these scripts.

Most of the SQL Server Imports use DEMO database. User have to create demo tables within this database first before executing import scripts.

ODBC connections

During the setup process, the following ODBC DSN is created during the installation:

ODBC_FOXPRO

ODBC_ACCESS_SOURCE

ODBC_ACCESS_TARGET

ODBC_MS

ODBC_ORACLE

ODBC_EXCEL



These ODBC connections provide the basis for the Visual Importer ETL to perform its tasks and process different types of data, while ensuring that the automation of these processes is completely transparent.



6. Options

The Visual Importer ETL allows user to tweak key settings. Using this provision user can change the repository type, control the execution of packages and where the results will be logged, set interface actions such as the prevention of accidental deletion and set email options to report the outcome of the automation to individuals or groups of people.

To change Visual Importer ETL settings click System menu-> File-> Options.

The Dialog box will appear.



The Repository tab defines the repository connection. Once the logon parameters

have been verified via the Test Connection dialogue this information is saved so it is not necessary to provide this information again.

The Repository type can be:

MS Access

MS SQL Server

Oracle

Interbase (Firebird)

MySql

PostgreSQL

OleDB

Note:

Use Repository creation wizard to create new repository. Execution tab defines settings related to the logging and Packages execution.



Note:

To enable parallel execution select following options

Execute Actions in separate thread

Use Agent to execute Actions



The Interface tab defines key settings. These operate globally during the activities of

the Visual Importer ETL. For instance, in order to save time when creating packages to automate tasks, the editor can be set so that it automatically opens. In order to ensure that any changes made to scripts and packages upon exit, user can safeguard this by checking the Save all scripts and packages on exit box. Accidental deletion is prevented by requesting that the user is asked before deleting any objects. Another aspect of the interface actions is the updating of the log. This can be set so that once a package has completed its actions the log is updated with all the actions which took place during execution. Finally, if user wishes to automatically refresh table list when creating/opening transformation you simply check the Update Tables List to ensure that user this is accurate, and up to date.



The Email tab defines default Emails settings. As the Visual Importer ETL is designed to work automatically and in batch mode, it will be necessary to specify where emails will be sent from. Emails messages are used to convey when processes and activities have been completed. It is also possible to specify the default recipient. The default recipient can be set to a group email address, or it can be left blank.



The Notifications tab defines Email connection settings for Agent notifications. The server my run out of disc space or repository database can become corrupted. In case of repository connection failure agent will send two emails one after the failure another after the connection was restored. In most of the cases the agent will work as expected once reconnected however it is recommended to restart it.

Email Message Example:

Error Message: Reconnected!

Computer Name: COMPUTER

User Name: dbsl

Os Version: Windows 8.1 (Version 6.3, Build 9600, 64-bit Edition)

Number Of Threads: 11

Repository: MS Sql Server Repository

Type: MS Sql Server

Server: COMPUTER\SQLEXPRESS2008

Database: REPOSITORY

User: sa



6.1 Global Variables

Global Variables are used to replace Variable with Value, for example before SQL is executed

is be replaced with 1

EG

Select * from Customers where CustomerId =

Is changed to

Select * from Customers where CustomerId =1

Predifined variables:

OS user name



7. User Interface

Upon starting the Visual Importer ETL software, user will be presented with the main window. This is where the main tasks are controlled such as processing cycles, and other processing tasks taking place in the processor.

7.1 System Menu

The system menu allows user access to the essential setting and tools user will

need to use the processor.

7.2 Visual Importer ETL Graphical User Interface: Main window

System Menu

Objects tree

Status bar

Quick Access Tabs

Objects tree toolbar



7.2.1 Quick Access Tabs

The "Quick Access Tabs" gives the user easy access to various functions of

application: The titles of the tabs are self-explanatory, default tab is design.

7.2.2 Objects tree toolbar

1 2 3 4 5 6 7 8 9 10 11

1. Selected object properties

2. Refresh object tree

3. Create new object

4. Delete selected object

5. Cut

6. Copy

7. Paste

8. Find Object

9. Find Next Object

10. Expand selected tree node

11. Collapse selected tree node

7.2.3 Objects tree description

The "Objects" tree lists the main categories for the provision of objects. For instance,

the "Directories" category contains icons representing the location of files and related information belonging to an individual item. For example, by clicking on the "Excel Files" icon you will access the directory area where all the Excel files are stored for processing. You may also find icons representing individual database connection services. For

instance, you can see connections available for both MS SQL Server and Oracle. Other object types include the ODBC connections. You can see from the illustration

that the DEMO system has been configured, and as a result the ODBC connections

created are listed in the objects tree. The Visual Importer ETLprovides the Objects tree so that users have easy access to the main objects. The full list of objects is described below:



7.2.4 Objects tree list

Below is a list of possible objects with the short descriptions:

Projects Provides a way to group objects together

Directories Defines path to flat files, Excel files, MS Access databases or

DBF files

JDBC Connection Defines JDBC connection properties

SQL Server CE

Connection

Defines MS SQL Server CE connection properties

SQL Server Connection Defines MS SQL Server connection properties

ODBC Connection Defines ODBC connection properties

Oracle Connection Defines Oracle connection properties

Ole DB Defines Ole DB connection properties

MySQL Defines MySql connection properties

PostgreSQL Defines PostgreSQL connection properties

SQLite Defines SQLite connection properties

Interbase/Firebird Defines SQLite connection properties

FTP Connection Defines FTP Server connection properties

SMTP Connection Defines SMTP connection properties

POP3 Connection Defines POP3 server connection properties

IMAP4 Connection Defines IMAP4 server connection properties

SalesForce Connection Defines SalesForce connection properties

BrightPearl Connection Defines BrightPearl connection properties

Sugar CRM

Connection

Defines Sugar CRM connection properties

Google analytics

Connection

Defines Google analytics connection properties

Google Spreadsheet

Connection

Defines Google Spreadsheet connection properties

Youtube Connection Defines Youtube connection properties



Twitter Connection Defines Twitter connection properties

IP Socket Connection Defines IP Socket connection properties

Lookups and Lookups

Groups

Provides an easy access to user defined data entry screens

Imports and Imports

Groups

Defines mapping between Data source and Data target

Reports and Reports

Groups

Reports design

SQL Scripts and

Scripts Groups

Defines SQL statements to perform against target Databases

Email templates and

groups

Creates general email templates.

Packages and packages

groups

Combines complex Actions together like Ftp downloads File

operations, emails, Check files, SQL scripts and

Transformations.



7.2.5 Objects Groups

The Visual Importer ETL also has the ability to group objects together into groups. The

purpose of doing this is so that objects of a specific type can be viewed as a branch of the

object tree. Whenever a new object is added it will be placed under the appropriate branch for

its type.

Groups of Objects



To create a new object group select appropriate object type for example transformation and click new.

Depending on the object type, the appropriate dialogue will be presented. In this

example the Transformation Group has been selected. The dialogue in this case requests a description and comment about the object to be created. Some of the fields in a dialogue of this type will be mandatory and will therefore have to be entered. However, comments will be optional, and just provide the user with the ability to specify more detail about an object and what it does.



8. Directories and Connections

8.1 Directories

If a new directory is required, selecting New on the new directory icon brings up the

following dialogue:

The following steps are needed to allow creation of a directory, as follows:

In the Name Text Box type in a new name for the directory you are about to create

Fill in Directory path you wish to load data from

Click OK to close the directory Properties Window

Note:

The user may change connection or directory properties at any time by double clicking on it. It is also possible to use sub directory in existing directory

\Subdirectory. For example \tmp will expand to: C:\Program Files\DB Software Laboratory\Demo\Buffer\tmp



8.2 Database Connections

The database connection functionality provides a way of connecting to database

servers. Once a connection is set up, it is always available until deleted or amended. In order

to set up a new connection it will be necessary to have database credentials to hand. When setting up database connections, you should ensure that the account you

wish to use has access to all the available functionality you may wish to use. For instance, it is not much use, setting up a connection to an Oracle account to use certain stored procedures, if the underlying Oracle permissions are not set for this

facility. The account used in the Visual Importer ETL for the database should allow the user to perform underlying database actions, which may not be under control of the processor directly. To create a new connection select appropriate connection type and click New.

Once this step has been completed, you will be provided with an appropriate

dialogue which will request information about the new connection you wish to create, and the type of connection you are creating.



8.2.1 Oracle Connection

For an Oracle connection you will need to specify the TNS name required, and

provide the user name and password for the connection.

Note:

It is also possible to use Oracle instant client. (servername:1521/ORCL)

The procedure to set up an Oracle Connection is as follows:

In the Name Text Box type in a new name for the connection you are about to create

Select a TNS Name from the Server Drop Down List

Fill in Username/Password for the database you wish to connect to

a. If you are unsure of these parameters, please contact your Database

Administrator for the correct settings.

Click Test to ensure the details you have provided are correct

Click OK to close the connection properties window

Browse connection

tables and data



8.2.2 JDBC Connection

JDBC is a Java-based data access technology (Java Standard Edition platform) from Sun Microsystems, Inc.. It is an acronym as it is unofficially referred to as Java Database Connectivity, with DB being universally recognized as the abbreviation for database. This technology is an API for the Java programming language that defines how a client may access a database. It provides methods for querying and updating data in a database. JDBC is oriented towards relational databases. A JDBC-to-ODBC bridge enables connections to any ODBC-accessible data source in the JVM host environment. Source: Wikipedia.

Note:

SQL builder does not work with JDBC connections. List of JDBC drivers can be found

here http://developers.sun.com/product/jdbc/drivers

http://developers.sun.com/product/jdbc/drivers



8.2.3 Microsoft SQL Server Connection

The procedure for many database connections is very similar. In the case of MS SQL Server it is necessary to specify the server, database name and user name/password combination.

An SQL Server connection setup requires the following steps:


Select a Server Name from Server Drop Down List

Select a Database Name form the Drop Down List


a. If you are unsure of these parameters, please contact your Database

Administrator for the correct settings.



Note:

Leave user name and password blank to use Windows Authentication.

Double click to update

list of databases



8.2.4 ODBC Connections

ODBC history ODBC, standing for Open Data Base Connectivity, represents a standard database

access method working as SQL API software. It was developed back in 1992 with the sole purpose of facilitating the work with database management systems (DBMS). It was created by the Microsoft corporation and is since then included in every copy of Microsoft Windows. In 1995 the ODBC was included in the SQL Standard, which played a great role in making ODBC more popular. Today, there are versions for almost any operating system currently used.

How ODBC works ODBC was intended to enable developers' access to any data through any

application, regardless of the DBMS used for managing that data. ODBC boasts platform independence since it has been purposefully designed in a way that makes it distinct from database systems, programming languages and operating systems. Facilitating the data access from an application to a database management system

through ODBC is done through a specific mechanism. A common ODBC implementation contains one or more applications, a core ODBC 'Driver Manager' library, and one or more database drivers. The Driver Manager's role is to interpret the data queries coming from an application by using the DBMS-specific details contained in database drivers. The latter represent a middle layer inserted between an application and the DBMS in use. This way, the application's data queries are translated into commands that can easily be read by the DBMS. A basic requirement for an ODBC implementation to be run is that both the

application and the DBMS be ODBC-compliant. In other words, the application must be able to issue ODBC commands and the DBMS must be capable of responding to them. Thanks to its modular nature ODBC gives developers a great freedom in creating

the separate components of an ODBC implementation. Thus, a programmer can write applications that use standard features without needing to worry about the type of DBMS used for administering the database that the application tries to access. Likewise, the only thing the database driver programmers need to keep in mind during the development process is how to attach their database driver to the 'Driver Manager' library. The result is that currently there are hundreds of ODBC drivers created for a large variety of data sources. Thanks to the long period of existence and the fruitful efforts of its team of

developers, ODBC now offers access to a much wider range of data sources than any other database access method available today. It has turned into a universal data access standard, working with a great variety of operating systems and providing access to even non-relational data, including text and XML files.



The ODBC Connection properties dialogue allows the administrator to create appropriate connections to various databases. It is be necessary to create connections to databases, whenever they are to be processed.

In order to create an ODBC connection, the following steps should be followed:


Select a ODBC DSN Name from Server Drop Down List


a. You may use ODBC admin to create/modify ODBC data sources.



Build ODBC connection

string



8.2.5 ODBC Connection Strings

It also possible to use ODBC connection strings for both Reader and Writer connections. For example for MS SQL Server connection string is:

Driver={SQL Native Client};

Server=myServerAddress;Database=myDataBase;Uid=myUsername;Pwd=myPassword;

One of the major benefits of using connection strings that it makes it no longer necessary to

create ODBC Dsns manually on every single computer where Visual Importer ETL is

installed. It also gives a greater control over the connection parameters.

Note:

Leave user name and password blank and provide it within connection string

More information about connection strings can be found at:

http://www.connectionstrings.com

The simplest way to create ODBC connection string is to use ODBC Connection builder

dialog. Double click on ODBC driver name to create a connection string

http://www.connectionstrings.com/



8.2.6 MySQL Connections

About MySQL The MySQL database has become the world's most popular open source

database because of its consistent fast performance, high reliability and ease of use. It's used on every continent -- Yes, even Antarctica! -- by individual Web developers as well as many of the world's largest and fastest-growing organizations to save time and money powering their high-volume Web sites, business-critical systems and packaged software -- including industry leaders such as Yahoo!, Alcatel-Lucent, Google, Nokia, YouTube, and Zappos.com.

Creating MySQL connection

In order to create a MySQL connection, the following steps should be followed:


Type in the host name

Select a Database name from the drop down List

Select appropriate connection port default is 3306




Click Test Connection to ensure the details you have provided are correct




8.2.7 PostgreSQL Connection

About PostgreSQL PostgreSQL is a powerful, open source object-relational database system. It has

more than 15 years of active development and a proven architecture that has earned it a strong reputation for reliability, data integrity, and correctness. It runs on all major operating systems, including Linux, UNIX (AIX, BSD, HP-UX, SGI IRIX, Mac OS X, Solaris, Tru64), and Windows. It is fully ACID compliant, has full support for foreign keys, joins, views, triggers, and stored procedures (in multiple languages). It includes most SQL:2008 data types, including INTEGER, NUMERIC, BOOLEAN, CHAR, VARCHAR, DATE, INTERVAL, and TIMESTAMP. It also supports storage of binary large objects, including pictures, sounds, or video.

Creating PostgreSQL connection

In order to create a PostgreSQL connection, the following steps should be followed:




Type in the host name

Select a Database name from the drop down List

Select appropriate connection port default is 5432






8.2.8 Interbase/Firebird Connection

About Interbase InterBase is a relational database management system (RDBMS) currently

developed and marketed by Embarcadero Technologies. InterBase is distinguished from other DBMSs by its small footprint, close to zero administration requirements, and multi-generational architecture. InterBase runs on the Linux, Microsoft Windows, Mac OS X and Solaris operating systems.

About Firebird Firebird is a relational database offering many ANSI SQL standard features that

runs on Linux, Windows, and a variety of Unix platforms. Firebird offers excellent concurrency, high performance, and powerful language support for stored procedures and triggers. It has been used in production systems, under a variety of names, since 1981.

Creating Interbase/Firebird connection



In order to create an Interbase/Firebird connection, the following steps should be followed:


Type in the database name






8.2.9 Ole DB Connection

About Ole DB OLE DB (Object Linking and Embedding, Database, sometimes written as OLEDB

or OLE-DB) is an API designed by Microsoft for accessing different types of data stored in a uniform manner. It is a set of interfaces implemented using the Component Object Model (COM); it is otherwise unrelated to OLE. It was designed as a higher-level replacement for, and successor to, ODBC, extending its feature set to support a wider variety of non-relational databases, such as object databases and spreadsheets that do not necessarily implement SQL.

Creating Ole DB connection

In order to create an Ole DB connection, the following steps should be followed:


Type in the connection string or use Build Connection String button to create one.


Note:

Ole DB is one of the slowest ways of accessing the data and it uses a lot of memory. It is

not recommended for very large datasets.

Build Ole Db Connection string

connection string



8.2.10 SQLite Connection

SQLite is a software library that implements a self-contained, serverless, zero-

configuration, transactional SQL database engine. SQLite is the most widely deployed SQL

database engine in the world. The source code for SQLite is in the public domain.

To create a SQLite connection, follow these steps:

In the Name Text Box type in a new name for the SQLite connection you are about to

create

Select Database name




8.2.11 Microsoft SQL Server CE Connection

SQL Server Compact Edition contains the runtime bits that can be used to develop

applications and to deploy applications on desktop computers and on Tablet PCs. SQL Server

Compact Edition is a powerful yet lightweight relational database engine. SQL Server

Compact Edition supports familiar SQL syntax. Additionally, SQL Server Compact Edition

provides a development model and an API that are consistent with SQL Server. Therefore, you

can more easily develop desktop applications by using the RTW version of SQL Server

Compact Edition than by using SQL Server Mobile. SQL Server Compact Edition includes

new features such as |DataDirectory| substitution string support and ClickOnce support so that

deployment is easy.

To create a SQL Server Compact Edition connection, follow these steps:

In the Name Text Box type in a new name for the SQL Server Compact Edition connection

you are about to create

Select Database name




8.2.12 BDE Connection

Borland Database Engine (BDE) is the Windows-based core database engine and connectivity software behind Borland Delphi, C++Builder, IntraBuilder, Paradox for Windows, and Visual dBASE for Windows.

History Borlands Turbo Pascal included a "database" Toolbox, it was the beginning of the

Borland compiler add-ons that facilitated database connectivity. Then came the Paradox Engine for Windows PXENGWIN which could be compiled into a program to facilitate connectivity to Paradox tables. The first DLL-based connectivity engine was ODAPI (Open Database API). It

represented Borlands attempt to centralise connectivity in its suite of applications which included the brand-new Paradox for Windows 4 and Quattro. With version 4.5 / 5.0 of Paradox for Windows, this database engine was crystallised as IDAPI. The included set of database drivers enables consistent access to standard data

sources: Paradox, dBASE, FoxPro, Access, and text databases. You can add Microsoft ODBC drivers as needed to the built-in ODBC socket. Optionally, Borland's SQL Links product provides access to a range of database management systems (DBMS), including Informix, DB2, InterBase, Oracle, and Sybase. BDE is object-oriented in design. At runtime, application developers interact with

BDE by creating various BDE objects. These runtime objects are then used to manipulate database entities, such as tables and queries. BDE's application program interface (API) provides direct C and C++ optimized access to the database engine, as well as BDE's built-in drivers for dBASE, Paradox, FoxPro, Access, and text databases. The core database engine files consist of a set of DLLs that are fully re-entrant and

thread-safe. Included with BDE are a set of supplemental tools and examples with sample code. BDE system is configured using the BDE Administrator (BDEADMIN.EXE). Included with BDE is Borland's Local SQL, a subset of ANSI-92 SQL enhanced to

support Paradox and dBASE (standard) naming conventions for tables and fields (called "columns" in SQL). Local SQL lets you use SQL to query "local" standard database tables that do not reside on a database server as well as "remote" DBMS servers. Local SQL is also essential to make multi-table queries across both local standard tables and those on remote SQL servers. The older name for the BDE API is the "Integrated Database Application Program

Interface" or "IDAPI". Source: Wikipedia Recommended settings for BDE:



Note: Although it is possible to work with industrial data bases like Oracle or SQL

server using BDE it is not recommended. BDE is no longer supported by Embacadero.



To create a BDE connection, follow these steps:

In the Name Text Box type in a new name for the BDE connection you are about to create

Select appropriate driver

Fill in connection parameters


BDE Administrator



8.3 POP3 Connection

POP3 (Post Office Protocol 3) is the most recent version of a standard protocol for

receiving e-mail. POP3 is a client/server protocol in which e-mail is received and held for you by your Internet server. Periodically, you (or your client e-mail receiver) check your mail-box on the server and download any mail, probably using POP3. This standard protocol is built into most popular e-mail products, such as Eudora and Outlook Express. It's also built into the Netscape and Microsoft Internet Explorer browsers. POP3 is designed to delete mail on the server as soon as the user has downloaded

it. However, some implementations allow users or an administrator to specify that mail be saved for some period of time. POP can be thought of as a "store-and-forward" service.

To setup email access using POP as the protocol follow these steps:

In the Name Text Box type in a new name for the POP3 connection you are about to create

Fill in host name

Select TCP/IP port (default is 110)

Type in user name and password

Click OK to close the POP3 connection properties window



8.4 IMAP4 Connection

Internet message access protocol (IMAP) is one of the two most prevalent Internet

standard protocols for e-mail retrieval, the other being the Post Office Protocol (POP). Virtually all modern e-mail clients and mail servers support both protocols as a means of transferring e-mail messages from a server.

To setup email access using IMAP as the protocol follow these steps:

In the Name Text Box type in a new name for the IMAP4 connection you are about to

create

Fill in host name



Select folder

Click OK to close the IMAP4 connection properties window



8.5 SMTP Connection

SMTP stands for Simple Mail Transfer Protocol, a protocol for sending e-mail

messages between servers. Most e-mail systems that send mail over the Internet use SMTP to send messages from one server to another; the messages can then be retrieved with an e-mail client using either POP or IMAP. In addition, SMTP is generally used to send messages from a mail client to a mail server.

To create an SMTP connection, follow these steps:

In the Name Text Box type in a new name for the SMTP connection you are about to

create

Fill in host name



Click OK to close the SMTP connection properties window



8.6 IP Socket connection

To setup IP Socket Connection follow these steps:

In the Name Text Box type in a new name for the Google Spread sheet connection you are

about to create

Type in host name

Select relevant port number

Test the connection

Click OK to close the IP Socket connection properties window



8.7 FTP Connection

FTP stands for File Transfer Protocol, the protocol for exchanging files over the

Internet. FTP works in the same way as HTTP for transferring Web pages from a server to a user's browser and SMTP for transferring electronic mail across the Internet in that, like these technologies, FTP uses the Internet's TCP/IP protocols to enable data transfer. FTP is most commonly used to download a file from a server using the Internet or to

upload a file to a server (e.g., uploading a Web page file to a server).

Visual Importer ETL supports full range of FTP operations.

To create an FTP connection, follow these steps:

In the Name Text Box type in a new name for the FTP Connection you are about to create

Fill in the host name


Type the user name and password

Test connection

Click OK to close the FTP connection properties window



About FTP protocol FTP is a TCP based service exclusively. There is no UDP component to FTP. FTP

is an unusual service in that it utilizes two ports, a 'data' port and a 'command' port (also known as the control port). Traditionally these are port 21 for the command port and port 20 for the data port. The confusion begins however, when we find that depending on the mode, the data port is not always on port 20.

Active FTP In active mode FTP the client connects from a random unprivileged port (N > 1023)

to the FTP server's command port, port 21. Then, the client starts listening to port N+1 and sends the FTP command PORT N+1 to the FTP server. The server will then connect back to the client's specified data port from its local data port, which is port 20. From the server-side firewall's standpoint, to support active mode FTP the following

communication channels need to be opened:

FTP server's port 21 from anywhere (Client initiates connection)

FTP server's port 21 to ports > 1023 (Server responds to client's control port)

FTP server's port 20 to ports > 1023 (Server initiates data connection to client's data port)

FTP server's port 20 from ports > 1023 (Client sends ACKs to server's data port)

The main problem with active mode FTP actually falls on the client side. The FTP

client doesn't make the actual connection to the data port of the server--it simply tells the server what port it is listening on and the server connects back to the specified port on the client. From the client side firewall this appears to be an outside system initiating a connection to an internal client--something that is usually blocked.

Passive FTP In order to resolve the issue of the server initiating the connection to the client a

different method for FTP connections was developed. This was known as passive mode, or PASV, after the command used by the client to tell the server it is in passive mode. In passive mode FTP the client initiates both connections to the server, solving the

problem of firewalls filtering the incoming data port connection to the client from the server. When opening an FTP connection, the client opens two random unprivileged ports locally (N > 1023 and N+1). The first port contacts the server on port 21, but instead of then issuing a PORT command and allowing the server to connect back to its data port, the client will issue the PASV command. The result of this is that the server then opens a random unprivileged port (P > 1023) and sends the PORT P command back to the client. The client then initiates the connection from port N+1 to port P on the server to transfer data.



From the server-side firewall's standpoint, to support passive mode FTP the following communication channels need to be opened:

FTP server's port 21 from anywhere (Client initiates connection)

FTP server's port 21 to ports > 1023 (Server responds to client's control port)

FTP server's ports > 1023 from anywhere (Client initiates data connection to random port specified by server)

FTP server's ports > 1023 to remote ports > 1023 (Server sends ACKs (and data) to client's data port)

While passive mode FTP solves many of the problems from the client side, it opens

up a whole range of problems on the server side. The biggest issue is the need to allow any remote connection to high numbered ports on the server. Fortunately, many FTP daemons, including the popular WU-FTPD allow the administrator to specify a range of ports which the FTP server will use. See Appendix 1 for more information. A quick summary of the pros and cons of active vs. passive FTP is also in order: Active FTP is beneficial to the FTP server admin, but detrimental to the client side

admin. The FTP server attempts to make connections to random high ports on the client, which would almost certainly be blocked by a firewall on the client side. Passive FTP is beneficial to the client, but detrimental to the FTP server admin. The client will make both connections to the server, but one of them will be to a random high port, which would almost certainly be blocked by a firewall on the server side.



8.8 Google Spread sheet connection

To setup Google Spread sheet access follow these steps:

In the Name Text Box type in a new name for the Google Spread sheet connection you are

about to create


Test the connection

Click OK to close the Google Spread sheet connection properties window



8.9 SalesForce connection

To setup SalesForce Connection follow these steps:

In the Name Text Box type in a new name for the SalesForce connection you are about to

create

Type in Customer Key

Type in Customer Secret

Type in Security Token

Type in User name

Type in Password

Test the connection

Click OK to close the SalesForce connection properties window

Note:

1. To get Consumer Key and Consumer Secret login into SalesForce, click develop,

click Remote access authentication. To get security token click my personal

information, reset security token.

2. SalesForce support subset of SQL more information can be found here:

http://www.salesforce.com/us/developer/docs/api/index_Left.htm#CSHID=sforce_api_

calls_soql.htm|StartTopic=Content%2Fsforce_api_calls_soql.htm|SkinName=webhelp

http://www.salesforce.com/us/developer/docs/api/index_Left.htm%23CSHID=sforce_api_calls_soql.htm|StartTopic=Content%2Fsforce_api_calls_soql.htm|SkinName=webhelphttp://www.salesforce.com/us/developer/docs/api/index_Left.htm%23CSHID=sforce_api_calls_soql.htm|StartTopic=Content%2Fsforce_api_calls_soql.htm|SkinName=webhelp



Getting Consumer key and Consumer secret



Getting Security Token



8.10 BrightPearl connection

Brightpearl is a fully integrated, web-based business management software system. It allows you to run all your business processes through one piece of software, so you no longer have to suffer the pain of transferring data between multiple different business systems. All your business's information is visible in real-time across every department whether it is entered via your website, inventory control, accounts or CRM.

To setup BrightPearl Connection follow these steps:

In the Name Text Box type in a new name for the BrightPearl connection you are about to

create

Select datacentre

Type in email address

Type in account name

Type in password

Test the connection

Click OK to close the BrightPearl connection properties window

Note:

More information about datacentres:

http://www.brightpearl.com/developer/latest/concept/uri-syntax.html

http://www.brightpearl.com/developer/latest/concept/uri-syntax.html



8.11 SMS Server connection

Visual Importer ETL is capable of sending SMS messages. Following services are supported Twilio, BulkSMS, TelAPI and Essendex.

Twilio

Create an account at Twilio, http://www.twilio.com. For your account, you will get an Account

SID and an Auth Token. Set Key to the account SID and Secret to the auth. token. You will

have a telephone nr. associated with the Twilio account, set Name to this phone number

received.

BulkSMS

Create an account at BulkSMS, http://www.bulksms.com. Set Key to your BulkSMS account

name and Secret to your BulkSMS account password.

http://www.twilio.com./http://www.bulksms.com/



TelApi

Create an account at TelAPI, http://www.telapi.com. For your account, you will get an

Account SID and an Auth Token. Set Key to the account SID and Secret to the auth. token.

Esendex

Create an account at Esendex, http://www.esendex.co.uk. For your account, you will get an

account code. Set Key to your Esendex account name, set Secret to your account password and

set Name to your Esendex account ID.

http://www.telapi.com/http://www.esendex.co.uk/



9. Importing Data

In order to load data from the data source into the data target user must define data mapping

between target table and data source.

Possible data sources are:

Multiple Delimited or Fixed width Text files

Multiple Excel files + Multiple Excel Spread sheets

Multiple MS Access Databases + Multiple Tables

XML Files

QVX files

QVD files

Multiple DBF Files

Any ODBC compliant database

Any JDBC compliant database

Full support for OLE DB

Direct connection to Oracle, SQL Server, MySql, PostgreSQL, SQLite, SQL Server

Compact and Interbase/Firebird

Multiple Tables

Windows Event Log

RSS

Google Spread Sheets

SalesForce

BrightPearl

Import screen is designed to allow user to create, delete, modify, and test data mapping to the

target databases.

At the beginning the simplest way to create Import sctipt is to use import script wizard.

To create a new Import click System menu-> New-> Import.

Fill in Description edit box with the name of an Import you are about to create.

Follow wizard Instructions



9.1 Mapping editor screen overview

Double click on any demo Import.

Import editor will appear.

Main Tool bar

Source tool bar

Mapping Panel Target table fields list

Target Fields

Source fields

Source data

Source file/Table

Field No

Field Width

Starting Position



9.1.1 Main tool bar

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1. Data Target options

2. Loads Import Script From the file

3. Saves Import Script to the file

4. Saves as

5. Saves Import to the Repository

6. Refreshes fields list fro the database

7. Checks Import for mapping errors

8. Data preview

9. Allows user to clear field mapping

10. Hides mapping panel

11. Data Import

12. Manage Versions

13. Add Version

14. Revert to previous version

15. Make Import Script Read Only

9.1.2 Source tool bar

1 2 3 4 5 6

7 8

1. Data source options

2. Refreshes Source data

3. Adds new column

4. Deletes last column

5. Auto map the source fields to the target fields

6. Filter

7. Records to Show

8. Sources file name/ table name



9.1.3 Mapping panel

Mapping panel is designed to provide user with all information related to the mapping of one

particular field. There are two ways of mapping: direct and through calculations.

Alternately you may hide Mapping panel and use grid to perform mapping.

See the picture below:

Date format

Calculation

Default value

Mapping type Source Field

Errors Handling



9.1.4 SQL Statements

Visual Importer ETL provides functionality to execute SQL statements before and after data

import.

Note:

In order to Execute several SQL statements use

Visual Importer Professional user manual - ETL tools Professional User Manual.pdf · Visual...

Documents

Transcript of Visual Importer Professional user manual - ETL tools Professional User Manual.pdf · Visual...