06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) •...

13
Integra(ng Globus into the NCAR Research Data Archive User Services Thomas Cram Doug Schuster Steven Worley NCAR, Boulder, CO

Transcript of 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) •...

Page 1: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

Integra(ng  Globus  into  the  NCAR  Research  Data  Archive  User  Services  

Thomas  Cram  Doug  Schuster  Steven  Worley  

NCAR,  Boulder,  CO  

Page 2: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

What  is  the  RDA?  

•  600+  datasets  for  climate  and  weather  research  

•  Free  and  open  access  •  Worldwide  usage  •  Various  data  access  pathways  – HTTP  (wget,  cURL,  etc.)  – OPeNDAP,  WCS,  WMS  – Web  services  (CLI,  API)  – Direct  analysis  on  HPC  systems  (NCAR  users)  

Page 3: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

RDA  Usage  

2014:  •  17+  PB  virtual  processing  •  Web  downloads:  7300  users,  750  TB  served  •  Custom  orders:  4000  users,  380  TB  served,  45,000  orders  

Page 4: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

That’s  a  lot  of  data.  How  can  we  improve  download  

efficiency  and  reliability?  

Page 5: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

Globus  @  RDA:  How  we  do  it  

•  Single  shared  endpoint  – Data  copied  to  subdirectories  under  endpoint  source  path  

– ACLs  managed  programa(cally  via  Globus  CLI  – Allow  read  permission  to  subdirectories  under  the  shared  endpoint  

Page 6: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

RDA  example  user  interface  

acl-add rda#data_request --perm=r --path=/download.auto/CRAM80279/ [email protected]

Page 7: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

RDA  Alternate  Iden(ty  login  

Page 8: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

RDA  Alternate  Iden(ty  login  

Page 9: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

Collec(ng  usage  metrics  

Globus  Endpoint  Manager  API •  Method = GET •  URL = https://transfer.api.globusonline.org/v0.10/endpoint_manager/task_list

•  Headers = {‘Authorization’:’Globus-Goauthtoken $GOTOKEN’}

•  Filters = {‘filter_endpoint’ : ‘rda#data_request’}

Page 10: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

What’s  next  /  improvements  

•  Globus  access  for  general  file  lists  (not  just  custom  orders)  – Files  are  hard-­‐linked  to  enable  data  shares  – Globus  groups  to  manage  access  to  restricted  datasets  

•  Migra(on  to  “data  cart”  func(onality  •  Map  Globus  user  ID  to  RDA  user  in  API  metrics  – Easier  linkage  to  individual  user  ac(vity  

Page 11: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

Upcoming  

•  GlobusWorld  2015  poster  —  this  evening    

•  ESIP  Summer  Mee(ng  •  14-­‐17  July  2015,  Pacific  Grove,  CA  •  Globus  breakout  session:  Speakers  wanted  •  hOp://esipfed.org/  

Page 12: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

Contact  us  

rda.ucar.edu  [email protected]  

Page 13: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)

Collec(ng  usage  metrics  (cont.)  

Metrics  stored  in  RDA  DB  – Globus  user  ID  – Transfer  volume  – Timestamps  – Source  &  des(na(on  endpoints  – #  files  transferred/skipped  


Module 3 RDA Basics Using the RDA Toolkit

Module 3 RDA Basics Using the RDA Toolkit

Appetizers & Side Orders Soups & Chili Famous lb. Burgers 1/2 · 1/18/2019  · Appetizers & Side Orders Chili Fries 5.95 Cheese 6.25 Mrs. T’s Homemade Soups Served fresh daily

Appetizers & Side Orders Soups & Chili Famous lb. Burgers 1/2 · 1/18/2019  · Appetizers & Side Orders Chili Fries 5.95 Cheese 6.25 Mrs. T’s Homemade Soups Served fresh daily

COACHELLA REDEVELOPMENT AGENCY · Coachella Redevelopment Agency Asset Transfer Review-4- Findings and Orders of the Controller The Coachella Redevelopment Agency (RDA) transferred

COACHELLA REDEVELOPMENT AGENCY · Coachella Redevelopment Agency Asset Transfer Review-4- Findings and Orders of the Controller The Coachella Redevelopment Agency (RDA) transferred

RDA 3R - NEOS · 2018-08-03 · Principals for RDA, 2015) RDA Toolkit, RIMMF, RDA Registry and RDA Vocabulary Server all draw on RDA Vocabularies published in RDF. Based on LRM implementation

RDA 3R - NEOS · 2018-08-03 · Principals for RDA, 2015) RDA Toolkit, RIMMF, RDA Registry and RDA Vocabulary Server all draw on RDA Vocabularies published in RDF. Based on LRM implementation

Module 2 RDA Basics Recognizing and Using RDA Records

Module 2 RDA Basics Recognizing and Using RDA Records

Second of two restraining orders served to James Lemus

Second of two restraining orders served to James Lemus

Quality of Service - tra.gov.om · Percentage of orders for access lines in the Served

Quality of Service - tra.gov.om · Percentage of orders for access lines in the Served

WPROWADZENIE WSB... · WPROWADZENIE. Zagadnienia • Geneza RDA • RDA i standardy bibliograficzne – AACR2, ISBD, ICP, FRBR • Podstawy RDA • RDA w Europie • RDA w Polsce.

WPROWADZENIE WSB... · WPROWADZENIE. Zagadnienia • Geneza RDA • RDA i standardy bibliograficzne – AACR2, ISBD, ICP, FRBR • Podstawy RDA • RDA w Europie • RDA w Polsce.

RDA, FRBR & MARC RDA Cataloguing Seminars September 2012.

RDA, FRBR & MARC RDA Cataloguing Seminars September 2012.

Bright Ideas. Brilliant Results.™ - halo.com · Brilliant Results.TM • Clients served - 45,000 • Account Executives - 500 • Custom Orders Processed - 300,000 • On time orders

Bright Ideas. Brilliant Results.™ - halo.com · Brilliant Results.TM • Clients served - 45,000 • Account Executives - 500 • Custom Orders Processed - 300,000 • On time orders

RDA Element Sets e RDA Value Vocabularies

RDA Element Sets e RDA Value Vocabularies

Representative Expressions - rda-rsc.org...Representative Expressions Kathy Glennan for Kate James –RDA Examples Editor kjam@rdatoolkit.org RDA Preconference: RDA Toolkit Redesign

Representative Expressions - rda-rsc.org...Representative Expressions Kathy Glennan for Kate James –RDA Examples Editor [email protected] RDA Preconference: RDA Toolkit Redesign

RDA & Serials. RDA Toolkit CONSER RDA Cataloging Checklist for Textual Serials (DRAFT) CONSER RDA Core Elements Where’s that Tool? CONSER RDA Cataloging.

RDA & Serials. RDA Toolkit CONSER RDA Cataloging Checklist for Textual Serials (DRAFT) CONSER RDA Core Elements Where’s that Tool? CONSER RDA Cataloging.

Pan-Canadian RDA Training Module: Cataloguing with RDA.

Pan-Canadian RDA Training Module: Cataloguing with RDA.

Introdução ao RDA - Módulo 2: Estrutura do RDA

Introdução ao RDA - Módulo 2: Estrutura do RDA

RDA 2019 CO.pdf · RDA future RDA 2019 Chris Oliver IFLA satellite Chair, RDA Board August 21, 2019. A quick glance at: ... in a MARC environment. ... RDA 2015: alignment with all

RDA 2019 CO.pdf · RDA future RDA 2019 Chris Oliver IFLA satellite Chair, RDA Board August 21, 2019. A quick glance at: ... in a MARC environment. ... RDA 2015: alignment with all

RDA and MARC 21 Development - rda-rsc.org and MARC 21... MARC RDA Working Group (Charge) • Evaluate the scope and impact of the extensions to RDA arising from the RDA 2019 revision

RDA and MARC 21 Development - rda-rsc.org and MARC 21... MARC RDA Working Group (Charge) • Evaluate the scope and impact of the extensions to RDA arising from the RDA 2019 revision

RDA$elements · RSC/RDA/Element-analysis-table--12April-2016--Table-of-RDA-elements-in-RDA-instruction-order.-Page-2- Element’ Aggregating’ or’Super’ Element’

RDA$elements · RSC/RDA/Element-analysis-table--12April-2016--Table-of-RDA-elements-in-RDA-instruction-order.-Page-2- Element’ Aggregating’ or’Super’ Element’

ELLIOT’S BISTRO EVENING MENU Side Orders £2.75 Served … · 2019-04-26 · ELLIOT’S BISTRO EVENING MENU Served Monday to Sunday 5.00pm – 9.00 pm Wherever possible our Chef’s

ELLIOT’S BISTRO EVENING MENU Side Orders £2.75 Served … · 2019-04-26 · ELLIOT’S BISTRO EVENING MENU Served Monday to Sunday 5.00pm – 9.00 pm Wherever possible our Chef’s

Manual de Procedimientos · Formato RDA-03 Formato RDA-001 Formato RDA-004 Formato RDA-03 Formato RDA-001 Formato RDA-004 A 9. Revisa que los documentos reúnan los requisitos fiscales

Manual de Procedimientos · Formato RDA-03 Formato RDA-001 Formato RDA-004 Formato RDA-03 Formato RDA-001 Formato RDA-004 A 9. Revisa que los documentos reúnan los requisitos fiscales