NESDIS Scheduler Presentation Welkin LBL V0.4

8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4

1/24

Job Scheduling: History and Evolution

November 17, 2010Welkin Associates


2/24

Job Scheduler Terminology and Components

The Scheduler Study Overview

Brief History of Job Scheduling

Job Scheduler Functions

Job Scheduler Architectures

Evaluation of Job Schedulers and Vendors

Q/A

Agenda


3/24

TerminologyJob: An executable unit. Can be a systems command, a script, anapplication program, or a collection of executable units. Jobs can have oneor more steps, each containing an executable unit.Job-Dependencies: Logical events or conditions that must be satisfied inorder for jobs to be scheduled for execution. Predecessor and Successor.Job-Flow: A collection of two or more jobs related by job-dependencies.Schedule: The current active set of scheduled jobs and job-flows ready tobe released for execution and monitored by the Master Scheduler.

ComponentsMaster Scheduler: Central software component that defines, modifies,stores job and job-flow definitions, dependencies, security information,events, and schedules. Provides automatic submission of executables,monitors status of all jobs and job-flows, provides alerts, controls starting,stopping, restarting of jobs and job-flows. Creates Audit trail of all events.

Agent:A program that resides on each server that is controlled by theMaster. The agent monitors jobs on its server, communicates with theMaster about the status of jobs, and executes commands relayed by theMaster.GUI Console: Console display for Administrators and Operators.Relational Database: The central repository used by the Master to store all

job, job-flow, dependencies, security information, schedules and audit trails.

Job Scheduler Terminology and Components


4/24

Job Scheduler Terminology and Components (Cont.)

Agent

Master Scheduler

RDMS

Operator GUI

console

Administrator

Agent

Agent

Logs

Administrator

GUI console

Operator

Job-Flows

Job-Flows

Job-Flows


5/24

OPUS: Legacy Job Scheduler used for Satellite dataprocessing

Developed by Space Telescope Science Institute (STScI ) in 1994 for HSTVery fast and efficient pipeline processor. Powerful job-flow features andGUI console

Adopted and customized for NOAA as a Job Scheduler on AIX p-series formigration from z/OS Mainframe (2005)OPUS requires replacement by a COTS Scheduler:

STScI no longer supports OPUSOPUS GUI written in Java has compatibility issues with latest version ofJava

The Scheduler Study

Evaluated using two in-house Schedulers (NDE and STARS)Evaluated COTS Schedulers and vendors.

Scheduler Study Overview


6/24

In the beginning there were 80 column Hollerith cards

The term batch was derived from stacks of punched cards

The first true Job Schedulers were people and card readers

Therefore, originally Job Scheduling known as batch processing

1964 IBM OS/360 JCL presented first job dependency functions

Brief History of Job Scheduling


7/24

Early Mainframe era: Batch processing using card decks

Later Mainframe era: JCL on IBM Mainframes.

Jobs: Dependencies/Priorities/Queues/TOD-Calendar/Alerts/Reruns

Evolution of 3rdparty vendor Mainframe Job SchedulersIBM Tivoli Workload Scheduler (TWS)

CA 7 Mainframe Scheduler

BMC Control-M

ASG Zeke Scheduler

Unix systems: crons

Today: Job Scheduling has greatly matured, with a number of

software vendors that have developed and marketed their Job

Scheduler products for over 25 years

Brief History of Job Scheduling (Cont.)


8/24

Workload Automation Maturity Pyramid



9/24

Gartner: The future automation model will be IT Workload Automation Broker (ITWAB)



10/24

Scheduler Functions:

Create and edit jobs, job-flows, users, groups, schedules

Calendar scheduling

File-Watchers (Trigger file events)

Job parameters within job-flows available

Job Queues, Classes, Priority

Virtual resources to control jobs and files

High Availability / Failover capabilityReliable and fast Scheduler database with open schema

Scheduler commands to start, stop, suspend, resume, and restart jobs

Alerts generated for job errors and sent to Operators (GUI console, email, cell)

Reporting tools

GUI console:

Windows, Web-based, Unix/Linux

Flexible and configurable for Administrators and Operators, easy to use

Graphical display of jobs and job-flows. Click and drag highly preferable

Detailed display of job-flow and job status, dependencies, errors/alerts

Job Scheduler Functions


11/24

Architecture: Master-Agent vs. Peer-to-Peer

Job Scheduler Architectures


12/24

Architecture: MasterAgent

Job Scheduler Architectures (Cont.)


13/24

EMA Criteria and Methodology for Evaluating Job Schedulers

Research/Advisory Firms Job Scheduler Evaluations


14/24

EMA Radar Report for Job Schedulers 01/2010

Research/Advisory Firms Job Scheduler Evaluations (Cont.)


15/24

Forrester List of Evaluated Job Schedulers

November 2009 Market Overview: Workload Automation, Q3 2009

Workload Automation Vendors



16/24

Gartner Job Scheduler Magic Quadrant Report 04/2009

The Gartner Magic Quadrant is copyrighted

(04/28/2009) by Gartner, Inc., and is reused with

permission. The Magic Quadrant is a graphicalrepresentation of a marketplace at and for a specific

time period. It depicts Gartnersanalysis of how certain

vendors measure against criteria for that marketplace,

as defined by Gartner. Gartner does not endorse any

vendor, product or service depicted in the Magic

Quadrant, and does not advise technology users to

select only those vendors placed in the Leaders

quadrant. The Magic Quadrant is intended solely as a

research tool, and is not meant to be a specific guide

to action. Gartner disclaims all warranties, express or

implied, with respect to this research, including any

warranties of merchantability or fitness for a particular

purpose.

Source: Gartner, Inc.

Magic Quadrant for Job Scheduling

28 April 2009, Report Id Number: G00166953

Milind Govekar, Gartner Analyst



17/24

Q/A


18/24

Additional Slides for reference

18


19/24

Workload Automation Example

19


20/24

Future Growth of WLA and Job Scheduling

Forrester: Forecast for Growth of WLA and Job Scheduling


21/24


22/24


Vendor Scheduler Product NameOpen Source,

Freeware,COTS

Corporate Practical Solutions Bqplus COTS

Cronless.com cronless Open Source

David G. Walker php JobScheduler Free Ware

Exact JobBOSS COTS

Exspans Automan COTS

Flux Corp. Flux COTS

Grand Logic JobServer COTS

Help/Systems Robot/SCHEDULE Enterprise COTS

Honico Batchman COTS

Hux Software HQ Scheduler COTSIBM

Tivoli Workload Automation SuiteCOTS

IndependIT BICsuite Scheduler Open Source

ISE EnterpriseSCHEDULE COTS

JOX IT JOX COTS

LXI LXI Scheduler COTS


23/24



Freeware,COTS

MJTNet Macro Scheduler COTS

MVP Systems JAMS COTS

Neteject.com VisualCron COTS

Network Automation Automate7 COTS

Nguyen Anthuan SetCronJob Free Ware

OAR Team OAR Open Source

Open Systems Mgmt Cosbatch COTS

OpsWise Automation Center COTS

ORSYP Dollar Universe COTS

OW2 Consortium ProActive Parallel Suite Open Source

Platform Computing Load Sharing Facility (LSF) COTS

PTC Software PTC Scheduler COTS

Redwood Software Cronacle COTS

ROC Software Maestro Scheduler COTS

Runtime Design

AutomationNetwork Computer (NC) COTS

SMA OpCon/xps COTS

SoftTree Tech 24x7 Scheduler COTS


24/24



Freeware,COTS

Sourceforge Jcrontab Open Source

Stonebranch Indesca COTS

Sun Microsystems N1 Grid Engine COTS

Sypsoft Job2Do COTSTenth Generation Job Scheduler for Windows COTS

Terracotta Quartz Enterprise Job Scheduler Open Source

The Coens Group Jobstream Open Source

UC4 Workload Automation Suite COTS

Univa UD Grid MP COTS

University of Warwick TITAN Grid Scheduler Open Source

University of Wisconsin Condor Open Source

Vinzant Global ECS (GECS) COTS

Xi Software GNUbatch Open Source

XuiS Software EnterpriseSchedule COTS

NESDIS Scheduler Presentation Welkin LBL V0.4

Documents

Transcript of NESDIS Scheduler Presentation Welkin LBL V0.4