Carl Wang Indigo Books & Music - Sas Institute Group...What EG Automation can do Anything that you...
Transcript of Carl Wang Indigo Books & Music - Sas Institute Group...What EG Automation can do Anything that you...
Get Data - ETL Source
Data Warehouse
MS SQL
Teradata
Oracle
SAP ……
Connect via interface/adapter (require SAS/ACCESS module license)
Simple and straight forward
PC Files (txt, csv, excel etc)
Sometimes can be diverse and difficult to manipulate
Usually by feed and delivered periodically
Scenario External files: Files without such pattern that easy to manipulate, eg: CustomExtract-Data-ICO_D_1-1624521-100713.xlsx
CustomExtract-Data-ICO_D_1-1609742-100813.xlsx
No valid DBMS type for importing (SAS/ACCESS to PC File Formats is not licensed on the SAS Server *) EXCEL
XLS
* proc setinit; run;
Possible Approaches Automation via:
SAS Server
SAS Macro
X Command
Stored Process
EG
EG Custom Task
EG automation interface (OLE automation)
Real World SAS Server SAS Macro only has limited control outside of SAS
session (with FILENAME statement)
X command turned off (due to security concern?? del /f /q *.*)
Stored process is SAS program hosted in central server and has same limitation as above
EG Automation Require knowledge of computer programming language
But more powerful and flexible than other approaches
Bridge the two very powerful
worlds: the world of your
SAS session and the world
of your Windows desktop
Note: COM - Component
Object Model
What EG Automation can do Anything that you can do with a SAS program, you can do
with a (custom task). And, anything that you can do with a Windows application you can do with a (custom task). If you need access to data or resources that you cannot reach with a SAS program, a (custom task) can act as the bridge to bring this information into SAS.
from Custom Tasks for SAS® Enterprise Guide® Using Microsoft .NET by Chris Hemedinger
Replace the custom task with scripting and the statement still true
Well Known Practice Schedule EG Project (see below sample)Option ExplicitDim appCall dowork'shut down the appIf not (app Is Nothing) Thenapp.Quit
Set app = NothingEnd IfSub dowork()
On Error Resume Next'----
' Start up Enterprise Guide using the project name'----Dim prjNameDim prjObject
prjName = "C:\Users\cwang\Documents\EG Projects\Testschedule.egp" 'Project NameSet app = CreateObject("SASEGObjectModel.Application.4.2")If Checkerror("CreateObject") = True Then
Exit SubEnd If'-----
' open the project'-----Set prjObject = app.Open(prjName,"")
Interface Approach by VBScriptSet sasApp = CreateObject("SASEGObjectModel.Application.5.1") *
' Set to your metadata profile namesasApp.SetActiveProfile("sasserver") *
' Create a new ProjectSet sasProject = sasApp.New
' add a new code object to the ProjectSet sasProgram = sasProject.CodeCollection.Add
' set the results types, overriding Application defaultssasProgram.UseApplicationOptions = FalsesasProgram.GenListing = FalsesasProgram.GenSasReport = False
' Set the server and text for the codesasProgram.Server = "SASApp " *
* See next slide for the location where to grab those parameters
Files ManipulationSet objFSO = CreateObject("Scripting.FilesystemObject")
‘automatic retrive directory
workingDir = objFSO.GetParentFolderName(WScript.ScriptFullName)
‘obtain collections of the folder
Set myFolder = objFSO.GetFolder(workingDir)
‘obtain files under directory
Set fileCollection = myFolder.Files
Transform Excel files (Optional*)Set objExcel = CreateObject("Excel.Application")
objExcel.Visible = False
objExcel.DisplayAlerts= False
For Each allFile In fileCollection
ext = Right(allFile.Name,5)
If UCase(ext) = UCase(extension)
‘and allFile.DateLastModified>=CDate(Date)-1
‘if we want define days of files to be processed
then
revPath = Mid(allFile, 1, InStrRev(allFile, "\")) & "CSV\"
‘ revFileName = Mid(allFile,InStrRev(allFile,"-")+1,InstrRev(allFile,".")-InStrRev(allFile,"-")-1)
FileName = Left(allFile.Name,InStr(allFile.Name,".")-1)
'open excel
Set objWorkbook =
objExcel.Workbooks.Open(allFile)
SaveName = revPath & FileName & ".csv"
objWorkbook.SaveAs SaveName,6
objWorkbook.Close False
objExcel.Quit
txtSAS = txtSAS & "proc import file='" & SaveName & "' out=test.out" & revFileName & " replace; run;“
End If
Next
* If has SAS/ACCESS to PC File Formats installed
If Multi-Sheet (Optional)‘in case the excel file has multi-sheet:
i = objWorkbook.Worksheets.Count
For k = 1 to i
SaveName = "\\file path\" & objWorkbook.Worksheets(k).Name & ".csv"
owb.Sheets(k).Activate
owb.SaveAs SaveName,6
Next
Assemble the ScriptOption Explicit
Dim sasApp ' EG ApplicationDim objExcel ' ExcelDim objFSO ' Filesystem Navigation
Const Extension = ".xlsx"
Call Automate
'release object reference
Set sasApp= NothingSet objExcel = NothingSet objFso = Nothing
Sub Automate()
Dim myFolder,fileCollection,allFile,FileName,revFileName,revPath,SaveName,ext,workingDir
Dim txtSAS,readSAS,sasProgDim objWorkbookDim runDate
Dim sasProject ' Project objectDim sasProgram ' Code object
Set objFso = CreateObject("Scripting.FilesystemObject")workingDir = objFso.GetParentFolderName(WScript.ScriptFullName)
Set myFolder = objFso.GetFolder(workingDir)
Set fileColl ection= myFolder.Files
Set objExcel = CreateObject("Excel.Application")
objExcel.Visible = FalseobjExcel.DisplayAlerts= False
'SAS Portion
Set sasApp = CreateObject("SASEGObjectModel.Application.5.1")
sasApp.SetActiveProfile("sasserver")
Set sasProject = sasApp.New
Set sasProgram = sasProject.CodeCollection.Add
sasProgram.UseApplicationOptions = FalsesasProgram.GenListing = FalsesasProgram.GenSasReport = False
sasProgram.Server = "SASApp"
' Create the SAS program to runsasProgram.Text = "libname test '" & workingDir & "\DS';"
Cont.For Each allFile In fileCollection
ext = Right(allFile.Name,5)If UCase(ext) = UCase(extension and allFile.DateLastModified>=CDate(Date)-1
'open excelrevPath = Mid(allFile, 1, InStrRev(allFile, "\")) & "CSV\"
revFileName = Mid(allFile,InStrRev(allFile,"-")+1,InstrRev(allFile,".")-InStrRev(allFile,"-")-1)
FileName = Left(allFile.Name,InStr(allFile.Name,".")-1)
Set objWorkbook = objExcel.Workbooks.Open(allFile)SaveName = revPath & FileName & ".csv"objWorkbook.SaveAs SaveName,6objWorkbook.Close False
objExcel.Quit
txtSAS = txtSAS & "proc import file='" & SaveName & "' out=test.out" & revFileName & " replace; run;"
End IfNext
'execute stored sas code
Set readSAS = objFSO.OpenTextFile(workingdir&"\test.sas", 1)sasProg = readSAS.ReadAll
sasProgram.Text = sasProgram.Text & txtSAS & sasProg
'WScript.Echo sasProgram.Text
' Run the codesasProgram.Run
' Save the log file to LOCAL disk
runDate="ETL_" & Year(Date) & Month(Date) & Day(Date)
sasProgram.Log.SaveAs workingDir & "\DS\LOG\" & runDate & ".log"
Application.Quit
End Sub
Alternative Other Script Language
Power shell; Python; Perl etc.
Write own custom task C#
(ETLTask.dll) built by me and will be covered next slide
If has SAS/ACCESS to PC File Formats installed Using existing EG Add In: SysCommand.dll
http://support.sas.com/documentation/onlinedoc/guide/customtasks/samples/SystemCommand42.zip
Base SAS: FILENAME statement and SAS Macro
Choose the files’ extension - available from dropdown list: xlsx; xls; txt; csv or *.* for any flat files (default: txt)
More Options Choose the start line of the file, similar to the firstobs option of the data step
(default: 1)Whether or not transforms the excel files
(default: false) Specify the getnames option for the proc import
(default: yes) Select the folder where files reside or manually enter it if you are using UNC
path (default: C:\)
Whether or not provide the prefix for the output dataset name(default: false)
Specify the prefix if option 6 checked.(default: Out)
Specify how many days back files to be processed based on the modified date (default: 7 Days)
Powerful but not Easy Build your own
If you have reasonable computer programming knowledge, especially object-oriented language
If the job can be modularized and need to be performed regularly
Or Get the resource or ask help from SAS community, use the
pre-built
You can contact me at [email protected] to acquire a copy of the ETLTask.dll or the source code if you want compile by your own.
SAS Codedata dir;infile '\\file path\files.txt' truncover;input fn $100.;format dt date9.;dt=input(reverse(substr(left(reverse(fn)),6,6)),mmddyy8.);if intck('day', dt, today())>0; /*define days of files to be processed*/
run;
data _null_;set dir;call symput('filename',left(fn));run;%put &filename;
FILENAME statement with Macro%let path = \\file path;
data files_1; path = "&path";rc = filename("fref", path);if rc = 0 then did = dopen("fref"); else putlog "Error: cant open &path" ;
if did<=0 then putlog "Error: cant open &path" ;else do i = 1 to dnum(did);
filename = dread(did, i); fid = mopen(did, filename); if fid > 0 then do;
kb=finfo(fid,'File Size (bytes)')/1024; crdate=finfo(fid,'Create Time');moddate=finfo(fid,'Last Modified');if kb > 0 then output;
end;end;keep filename kb crdate moddate path;
run;
data files_2;set files_1; format created_on modified_on date9.;created_on=input(crdate,anydtdte22.);modified_on=input(moddate,anydtdte22.); keep filename path created_on modified_on;
run;
data _null_;set files_2;if intck('day', modified_on, today()) = 1 thencall symput('filename',filename);
run;
%put &filename;
Issues: 64 bit vs 32 bit EG version before 5.1 is 32bit only application
If running EG under 64bit Windows, CreateObject("SASEGObjectModel.Application.4.2") will prompt error
Solution: use C:\Windows\SysWOW64\cscript.exe instead
SysCommand.dll may not compatible with EG installed under 64bit Windows
Compiled working version, contact me if you need
Reference Creating Custom Add-In Tasks for SAS Enterprise
Guide (http://support.sas.com/documentation/onlinedoc/guide/customtasks/index.htm)
The SAS Dummy - A SAS® blog for the rest of us
by Chris Hemedinger
(http://blogs.sas.com/content/sasdummy/)
SAS Product Documentation
(http://support.sas.com/documentation)