File and FTP adatper

Working with Oracle File and FTP adapter is definitely like black magic! I have SOA 11.1.1.6.0. I need to use FTP adapter to copy a file from a remote server. Both FTP and File adapters are designed to read a file, not to copy a file. If you want to copy a file and not process it, you have to jump through the hoops.

I sifted through 3 versions  of "Oracle® Fusion Middleware, User's Guide for Technology Adapters, 11g Release" (11.1.1.6.0, 11.1.1.6.2 and 11.1.1.6.3). Part of the document is inaccurate to say the least.I even tried to look up the adapter java classes (Oracle_OSB1\lib\external\adapters\ftpAdapter.jar) looking for clues. In the end, I resorted to trial and error, and with the help of network sniffing to sort out the mess.

The main challenge is to follow the instruction in section "4.5.11.5 Moving a File from One Remote Directory to Another Remote Directory on the Same FTP Server". Here are the instructions with my comments:

1. Create an empty BPEL process 

I created a synchronous BPEL, selected all default.

2. Drag and drop FTP Adapter from the Component Palette to the External

References swim lane. The Adapter Configuration Wizard Welcome page is displayed.

3. Click Next. The Service Name page is displayed.

4. Enter a service name in the Service Name field.

5. Click Next. The Adapter Interface page is displayed.- 

Default "Define from operation and schema (specified later) is checked.

6. Click Next. The FTP Server Connection page is displayed.

7. Enter the JNDI name for the FTP server, and click Next. The Operation page is

displayed. – 

JDev default might be "eis/ftp/FtpAdapter", however, SOA server default may be eis/Ftp/FtpAdapter.Look closely, you may need to change the 2nd part from "ftp" to "Ftp" (upper case "F").

8. Select Synchronous Get File, enter FTPMove in the Operation Name field, and

then click Next. The File Directories page is displayed. – 

do not overlook that "FTPMove". This is part of the black magic, it has to be precisely spelled like that, or you will be doomed. Why can't there be a drop down selection of "FTPMove"? If you have to enter specific text, what's the point of using an IDE.

9. Enter a dummy physical path for the directory for incoming files, and then click Next. The File name page is displayed.

Note: The dummy directory is not used. You must manually change the directory in a later step.

This is misleading. You can put real path in here, and the adapter uses it if you don’t overwrite it dynamically in BPEL code. Same applies to the other parameters.

10. Enter a dummy file name, and then click Next. The File Name page is displayed.

11. Click Next. The Messages page is displayed.12. Select Native format translation is not required (Schema is opaque), and thenclick Next. The Finish page is displayed.13. Click Finish. The outbound Oracle File Adapter is now configured.14. Drag the small triangle in the BPEL process in the Components area to the dropzone that appears as a green triangle in FTPMove in the External References area.The BPEL component is connected to the Oracle FTP Adapter outbound service.

15. Click File, Save All.

16. Create an invoke activity for the FTPMove service that you just created.The next step is to modify the generated WSDL file for FTPMove service andconfigure it with the new interaction specification for the move operation.17. Open the FTPMove_ftp.jca file and modify the interaction-spec, as shownin the following example.You must configure the JCA file with the source and target directory and file details. You can either hardcode the source and target directory and file details in the JCA file or use header variables to populate them. In this example, header variables are used.

<adapter-config name="FTPMove" adapter="Ftp Adapter"

xmlns="http://platform.integration.oracle/blocks/adapter/fw/metadata">

<connection-factory location="eis/Ftp/FtpAdapter" adapterRef=""/>

<endpoint-interaction portType="FTPMove_ptt" operation="FTPMove">

<interaction-spec

className="oracle.tip.adapter.ftp.outbound.FTPIoInteractionSpec">

<property name="SourcePhysicalDirectory" value="foo1"/>

<property name="SourceFileName" value="bar1"/>

<property name="TargetPhysicalDirectory" value="foo2"/>

<property name="TargetFileName" value="bar2"/>

<property name="Type" value="MOVE"/>

</interaction-spec>

</endpoint-interaction>

</adapter-config>

18. Map the actual directory and file names to the source and target file parameters by performing the following procedure:

a. Create 4 string variables with appropriate names. You must populate these variables with the source and target directory details. The BPEL source view shows you this:

<variable name="sourceDirectory" type="xsd:string"/>

<variable name="sourceFileName" type="xsd:string"/>

<variable name="targetDirectory" type="xsd:string"/>

<variable name="targetFileName" type="xsd:string"/>

b. Create an assign activity to assign values to sourceDirectory,

sourceFileName, targetDirectory, and targetFileName variables.

The assign operation appears in the BPEL source view as in the following example:

 I use BPEL2.0, my assignment looks like:

   <assign name="Assign1">

      <copy>

        <from>'/test/input'</from>

        <to>$sourceDirectory</to>

      </copy>

      <copy>

        <from>'input.txt'</from>

        <to>$sourceFileName</to>

      </copy>

      <copy>

        <from>'/test/output</from>

        <to>$targetDirectory</to>

      </copy>

      <copy>

        <from>'output.txt'</from>

        <to>$targetFileName</to>

      </copy>

</assign>

c. Pass these parameters as headers to the invoke operation. The values in these

variables override the parameters in the JCA file.

The code snippet has apparent errors, here is my code in BPEL2.0 format:

    <invoke name="Invoke" partnerLink="mvFile" portType="ns1:FTPMove_ptt"

            operation="FTPMove" inputVariable="Invoke_FTPMove_InputVariable"

            outputVariable="Invoke_FTPMove_OutputVariable"

            bpelx:invokeAsDetail="no">

      <bpelx:toProperties>

        <bpelx:toProperty name="jca.file.SourceDirectory"

                          variable="sourceDirectory"/>

        <bpelx:toProperty name="jca.file.SourceFileName"

                          variable="sourceFileName"/>

        <bpelx:toProperty name="jca.file.TargetDirectory"

                          variable="targetDirectory"/>

        <bpelx:toProperty name="jca.file.TargetFileName"

                          variable="targetFileName"/>

      </bpelx:toProperties>

    </invoke>

A few notes here, do NOT use the UI and select the “properties”, these properties do not show up in the list, AND the properties show up in the list do NOT work  (for example, jca.ftp.FileName would look like a perfect choice, but it is misleading, it does not work!).

Additionally, there are “To” and “From” properties I can’t find any documents about these choices.  I finally figured out (network sniffing) that “to” appears to be what used to be called “input” properties. I would assume “from” would be the same as old “output” properties.

19. Finally, add an initial receive or pick activity.

You have completed moving or renaming a file from a remote directory to another remote directory on the same FTP server.

If you created a default BPEL, you may already have your receive activity. Run your test, that should take move “/test/input/input.txt” to “/test/output/output.txt” on the same FTP server.

If you want to move the file from remote “/test/input/input.txt” to local “/test/output/output.txt”, you need to add this property to the jac file:

<property name="TargetIsRemote" value="false"/>

Minimum File Age Oracle SOA File/FTP Adapter PropertyOracle SOA File or the FTP adapter comes equipped with several properties during design time that can be used to meet the project requirements. Minimum File Age is one such property though may not be used all the time during the design of the File/FTP adapter components in the composite but when used, it does play a significant role in the way the files are picked from the File/FTP server.

Important points to be noted regarding the Minimum File Age Property:

Minimum Age is applicable only in case of a File/FTP poller operation (Read or the GET)

Minimum Age value is always specified in units of seconds in the File or the FTP JCA file of your composite though you may set the property using a different time unit in the File/FTP configuration wizard.

The property more commonly used to process large files that takes longer time by the application to write to the File/FTP server location from where the File/FTP adapter polls it.

When Minimum File Age property is set, the adapter takes into account the file modification time in order to pick the file.

Significance of Minimum File Age

Let me illustrate it with an example - Consider a use case, where you have multiple files with varying size written onto the FTP server location by some application. The large files may take more time to have them completely written on the FTP server path. During the time such large files are being written to the file location, the file modification time keeps changing.The FTP poller of the composite polling for a file on this file location with minimum file age unset(or set to zero) the poller may pick the file that's corrupt or invalid.   To avoid this, minimum age need to be set to an appropriate value in the FTP Adapter, that way only those files that are completely written are picked by the adapter for further processing.

The Minimum File Age value can be specified in the same screen of the File/FTP Adapter configuration wizard screen where you mention the polling frequency (as shown above)

File/FTP Adapter Threading Model – Oracle SOA 11gThe Oracle File and FTP Adapters use the following threading models:

Default Threading Model

Modified Threading Model

Default Threading Model:In the default threading model, a poller is created for each inbound Oracle File or FTP Adapter endpoint. The poller enqueues file metadata into an in-memory queue, which is processed by a global pool of processor threads. We can set the thread count of global processor through the oracle.tip.adapter.file.numProcessorThreads property at the pc.properties file. Create pc.properties file with this property and copy it to server and then reference it in the WLS classpath in setDomainEnv.sh.

The following steps highlight the functioning of the default threading model:

The poller periodically looks for files in the input directory. The interval at which the poller looks for files is specified using the PollingFrequency parameter in the inbound JCA file.

For each new file that the poller detects in the configured inbound directory, the poller enqueues information such as file name, file directory, modified time, and file size into an internal in-memory queue.

A global pool of processor worker threads waits to process from the in-memory queue.

Processor worker threads pick up files from the internal queue, and perform the following actions:

Stream the file content to an appropriate translator (for example, a translator for reading text, binary, XML, or opaque data.)

Publishes the XML result from the translator to the SCA infrastructure.

Performs the required post processing, such as deletion or archival after the file is processed.

Modified Threading Model:We can modify the default threading behavior of Oracle File and FTP Adapters. Modifying the threading model results in a modified throttling behavior of Oracle File and FTP Adapters. The following sections describe the modified threading behavior of the Oracle File and FTP Adapters: Single Threaded Model

Partitioned Threaded Model

Single Threaded Model:The single threaded model is a modified threaded model that enables the poller to assume the role of a processor. The poller thread processes the files in the same thread. The global pool of processor threads is not used in this model. You can define the property for a single threaded model in the inbound JCA file as follows:<activation-spec className="oracle.tip.adapter.file.inbound.FileActivationSpec">   <property../>   <property name="SingleThreadModel" value="true"/>   <property../></activation-spec>Partitioned Threaded Model:The partitioned threaded model is a modified threaded model in which the in-memory queue is partitioned and each composite application receives its own in-memory queue. The Oracle File and FTP Adapters are enabled to create their own processor threads rather than depend on the global pool of processor worker threads for processing the enqueued files. You can define the property for a partitioned model in the inbound JCA file as follows:<activation-spec className="oracle.tip.adapter.file.inbound.FileActivationSpec">  <property../>

  <property name="ThreadCount" value="4"/>  <property../></activation-spec>In the preceding example for defining the property for a partitioned model: If the ThreadCount property is set to 0, then the threading behavior is like that of the single threaded model.

If the ThreadCount property is set to -1, then the global thread pool is used, as in the default threading model.

The maximum value for the ThreadCount property is 40.

How To Read a Single File in Chunks in SOA 11gThis is a feature of Oracle File and FTP Adapters that uses an invoke activity within awhile loop to process the target file. This feature enables you to process arbitrarilylarge files.

To achieve this follow the steps below.

1)Create File/FTP Adapter and select the operation type as "SynchRead"

2)Select the sample directory name and file name for incoming files

3)Select the message schema as a native

4)Click on Finish.

5)Now edit the .jca file as below

    <interaction-spec className="oracle.tip.adapter.file.outbound.ChunkedInteractionSpec">      <property name="DeleteFile" value="true"/>

      <property name="PhysicalDirectory" value="/temp"/>

      <property name="FileName" value="dummy.txt"/>

      <property name="ChunkSize" value="10"/>

    </interaction-spec>

6) Now create an invoke activity and create partner link to the created File Adapter.

Now your file Adapter will be ready to read in chunks.

Now we need to put this invoke in while loop and check for end of the file by using following properties.

By setting following properties in invoke activity

From Side         <bpelx:fromProperties>

                <bpelx:fromProperty name="jca.file.LineNumber" variable="<Variable>"/>(Returns Line Number)

                <bpelx:fromProperty name="jca.file.ColumnNumber" variable="<Variable>"/>(returnColumnNumber)

                <bpelx:fromProperty name="jca.file.IsEOF" variable="<Variable>"/>(returnIs End Of the File)

                <bpelx:fromProperty name="jca.file.IsMessageRejected" variable="<Variable>"/>(returnIs Message Rejected)

                <bpelx:fromProperty name="jca.file.RejectionReason" variable="<Variable>"/>(returnRejectionReason)

                <bpelx:fromProperty name="jca.file.NoDataFound" variable="<variable>"/>(returnNoDataFound)

          </bpelx:fromProperties>

Changing the Archive File Name in Inbound File/FTP Adapter

REQUIREMENT:

To customize the archive file name, remove the junk characters from the archived file name. The default format is as under. 

SOA 10g - filename_yyyymmdd_hh24mmssSOA 11g - filename_encryptedToken_yyyymmdd_hh24mmss (new format introduced with 11g)

The possible customization is given in the solution below.

SOLUTION:

SOA 10g: You have a JCA property for FIle/FTP adapter, UseLongArchiveFileName, which can be used to prefix the process name to the archive filename to make the formatfilename_processname_timestamp.

SOA 11g (11.1.1.3/+) UseLongArchiveFileName is not available in 11g. Apply Patch 10155914 and it will become available.    

SOA 11g (11.1.1.5/+) Apply Patch 13249896.A new property  UseDigest can be used now in addition to  UseLongArchiveFileName (11g format as filename_processname_digest_timestamp) with File/FTP adapter.By default UseDigest is set to true which leads to the digest (encrypted token) being sandwiched in the filename and timestamp as filename_digest_timestamp.Setting this property to false will change the format to filename_timetamp. To use this property edit the .jca file to include this property as given below.

File_Adapter.jca

1

2

3

...

<property name="UseDigest" value="false"/>

...

SOA 11g (11.1.1.7/+)UseDigest property will be available by default without the need to apply the above patch.

There is, however, no further customization possible (presently) in terms of adding instance id to the file name or specifying custom file name. Hope the same gets available in the 11.1.1.7(+) or 12C version.

Best practices – Oracle SOA 11G – File Adapter- Part1Sharing some tips on best practice of handling inbound files using file adapter in SOA Suite

11g PS3.

Oracle SOA 11g File Adapter, always pickup the file when it is created in the directory it is

configured to look out. This is file & directory are configured using the JCA property

“IncludeFiles” & “PhysicalDirectory” respectively.

That said, if the file being created in the directory is big enough or if the source system is

slow or any network latency, the adapter will pickup intermediate file and try to translate to

the target schema. This behavior will fault the BPEL process and result the inbound message

will be “rejected message” .

So how we solve this behavior?.. There are lot of things we should consider before proposing

the solution to this behavior including the attributes of the source system, location of the

source sytem, SOA WL server/domain topology, File Adapter configuration, file format, size,

file markers,target end system, target system location, integration pattern -sync/async/one-

way/ etc. etc

To make things simple, i will consider the input file is simple xml in this part. Here are out-of-

the-box option available with the SOA Suite 11g PS3.

Firstly, the approach is to set the “MinimumAge” property in the JCA Adapter configuration.

Oracle’s Definition:

MinimumAge – This parameter specifies the minimum age of files to be retrieved. This

enables a large file to be completely copied into the input directory before it is retrieved for

processing. The age is determined by the last modified time stamp

Now our issue is partially solved by setting this parameter, but what is the “ideal value” of

this parameter ? Let us consider we are setting this parameter to 300 (5 min).

At this time, 2 questions need to be answered

(1) Is it safe to assume the file will be created within 5 min in production ?

This might work well in unit/sys/uat environment, as it is less probably all the batch

interfaces will be tested simultaneously and thus server load is minimum.

Is this “5 min” works at the production environment too where we expect multiple batch

interfaces running parallely in a WL domain? If the content of the file is business critical like

shipment details, quote info, pricing details, asset maintenance – no business user afford to

leave these contents in the middle-ware layer as a rejected message.

(2) Is business afford to wait for 5 min, even if file got created in 1st minute itself ?

This is again customer’s decision on how the data is important for the enterprise.

Assuming the data is critical and the destination end system cannot afford to wait for 5 min,

the other approach can be proposed is creating a addition TriggerFile with actual data-file.

The interface should pickup data-file only when the source system creates the “TriggerFile”.

The below are some of the JCA property that you should set in file adaper JCA file –

TriggerFilePhysicalDirectory, TriggerFile.

Oracle’s Definition:

TriggerFilePhysicalDirectory- The directory path where the Oracle File or FTP Adapter looks

for the trigger files.

TriggerFile-The name of the trigger file that activates the inbound Oracle File or FTP Adapter

By this approach, we can always ensure the data-file will be pickedup not only when it is

complete but also as soon as it got created.

Ftp Adapter with BPEL ProcessThere are lots many blog available on Internet which show the way to use ftp or file adapter under oracle BPEL 11g. Here I am focusing on some of the important aspect which require in special conditions:

Change the file name on run time using jca.ftp.FileName properties.During process execution we can assign the file name using "Assign" activity.Like

 

We can create String variable which store the file name created on run time

Now on the Invoke activity before call the ftp adapter ,We can override the jca.ftp.FileName property with newly created file name variable"ftpFileNamePoUpdate".

Get the list of file on FTP server and if file exist then put file creation on wait :We will create the ftp adapter which will list all the file from FTP server location.Mention the ftp adapter JNDI name during the creation of ftp adapter

Select the operation type as "Listfiles" and mention the operation name as per the choice.

Mention the ftp location from where we can wants to query all the files names:

We can query all the files on ftp server location with * or we can query the specific files name like *.xml ,A*.txt etc...

Here we are calling the same created ftp adapter using invoke acticvity.

In below mention invoke activity we will create the Input andout Parameters for ftp adapter call as below:

and out put parameter:

The if condation will check here with "ftpfilename" and output comes as a result of ftp file list are equal or not.

If any of the file name which BPEL trying to create on the server and filename on the server are equal thehn the flow will goes to wait time as assign here is 8 hrs.

Error Handling in File Adapter SOA Suite 11g (uniqueMessageSeparator)I wanted to keep processing the record flat file even if there are records which are not adhering to the schema. So searched into Oracle® Fusion Middleware User's Guide for Technology Adapters 11g Release 1 (11.1.1) E10231-02, and found a very useful piece of information.

The Oracle File Adapter and Oracle FTP Adapter provide inbound error handling

capabilities, such as the uniqueMessageSeparator property.In the case of debatching (multiple messages in a single file), messages from the firstbad message to the end of the file are rejected. If each message has a unique separatorand that separator is not part of any data, then rejection can be more fine grained. Inthese cases, you can define a uniqueMessageSeparator property in the schemaelement of the native schema to have the value of this unique message separator. Thisproperty controls how the adapter translator works when parsing through multiplerecords in one file (debatching). This property enables recovery even when detectingbad messages inside a large batch file. When a bad record is detected, the adaptertranslator skips to the next unique message separator boundary and continues fromthere. If you do not set this property, then all records that follow the record with errorsare also rejected.The following schema file provides an example of using the uniqueMessageSeparator property:

Handling Large Payload Files

Problem:Want to create a BPEL process to read, transform and translate a large payload file (e.g. size 1GB)?

Thoughts:Reading and traversing large payload file was always a problem in SOA 10g. Supposedly Maximum limit for such operations are for less than 7MB files in 10g. Solution to such problems are answered in 11g where you can read and transform huge payload files.

Solution:Suppose you have a CSV file incoming from source directory which you want to transform to a Fixed length file. Keep the source and the destination schema (XSD) ready. Also prepare the Transformation file (XSL) and keep it handy.For accomplishing this, you will only be using one File Adapter which will take care of your read, transform and write - so all the three operations takes place under single I/O interaction.

Follow the steps below for making it possible:1) Drag and drop a file adapter in the external references swim lane. (For e.g. Give name as FileMove.)2) Select Synchronous File Read3) Give any dummy value for File physical path and File Name. It will be changed manually later.4) Select Native format translation is not required (Opaque Schema)5) Outbound File Adapter is now configured, click on Finish6) Open the relevant .jca file (in this case FileMove_file.jca)7) Ensure that className should be "oracle.tip.adapter.file.outbound.FileIoInteractionSpec"8) Add extra parameters as shown in below file:<adapter-config name="FileMove" adapter="File Adapter"xmlns="http://platform.integration.oracle/blocks/adapter/fw/metadata"><connection-factory location="eis/FileAdapter" adapterRef=""/><endpoint-interaction portType="FileMove_ptt" operation="FileMove"><interaction-specclassName="oracle.tip.adapter.file.outbound.FileIoInteractionSpec"><property name="SourcePhysicalDirectory" value="testDir1"/><property name="SourceFileName" value="test1"/><property name="SourceSchema" value="xsd/source-csv.xsd"/><property name="SourceSchemaRoot value="Root-Element"/><property name="SourceType" value="native"/><property name="TargetPhysicalDirectory" value="testDir2"/><property name="TargetFileName" value="test2"/><property name="TargetSchema" value="xsd/destination-fixedLength.xsd"/><property name="TargetSchemaRoot value="Root-Element"/><property name="TargetType" value="native"/><property name="Xsl value="xsl/SourceToDestination.xsl"/><property name="Type" value="MOVE"/></interaction-spec>

</endpoint-interaction></adapter-config>9) Save the file and deploy the process.