Batch changes inVoyageran overview of the possibilities, processes and tools

GUGM 2014Adam Kubik, Clayton State UniversitySusan Wynne, Georgia State University

Selected Gary Strawn programs

Caveat emptor

● The Strawn tools are very powerful and do lots of things● Require a working ODBC connection to your database ● Operate directly on records in your database -- There is

no undo button!● Some tools are well documented, some not so well● We don’t pretend to have tested, used or understand

every possible option and setting● All of these tools can be run in preview or test mode.

Use it and start small

VgerSelect

● An alternative or complement to Access Reports

● Can output results as either:o Tab-delimited files of selected elements, oro Records in MARC or XML format (to work with in

MarcEdit, etc.)

Creating tab-delimited output file

● Limit to sound recording format based on record type (LDR/06)

● Export full MARC records

● Or choose fields, subfields and/or fixed field byte positions to export

● Here the bib ID number, 007, 008, 047 and 048 are chosen

Output can be filtered in Excel

Filtering with VgerSelect

● You can also ask VgerSelect to filter its own output, but...o Jobs might run for a whileo Fewer options than Excel

● If you want to play with the data in multiple ways it is faster to output all the data once and use Excel to manipulate it

Bib Delete

Bib Delete Reports

● Problems--any records the program could not delete (usually will be suppressed)

● List of records modified and what was done (suppressed, unsuppressed, deleted)

● Report--total number of records read● List of OCLC numbers of deleted records, in

case you need to delete holdings in WorldCat

Recent uses of Bib Delete at GSU

● Suppressed most print gov docs records for a weeding projecto Needed Location Changer for another part of this

project● Deleted obsolete Duke ebrary records due to change of

platform ● Deleted a group of ETD records that had been re-

harvested with better mapping & new OCLC control nos.

Recent uses of Bib Delete at Clayton State

● Suppressed Films on Demand titles from OPAC using vendor supplied sets of MARC records representing titles that had been removed from the collection

● Removed sets of locally bulk imported records when something went awry with the load (e.g., MFHD records not created as expected)

● For these jobs, we just used the set of bib records as the input file and the option to match the 001 in the vendor set against the 035 in Voyager

Location Changer

● Like Bib Delete, can (un)suppress or delete MFHDs, items and bibs

● Can also automate multiple changes to other data elements in MFHD and item records

● Can work on records one-by-one or in batches

Selecting records to change

● Select records to change by using criteria in the records:○ MFHD or item location (perm or temp)○ bib record type○ item status○ call number range

● Or, input a text file of barcodes, item IDs, MFHD IDs, or bib IDs

Selecting records to change

Use record criteria

Use an input file

Use in a weeding project at Clayton State

● Specify records with an input file of scanned barcodes generated by student assistant as items are boxed

● Run job to change to permanent location (in MFHD) to withdrawn, suppress holdings, and reset the item status to withdrawn

File of barcodes

Change locationChange item status

Use of Location Changer in weeding project

● If the last item attached to a MFHD is suppressed, the MFHD is also suppressed

● If the last MFHD attached to a bib is suppressed, the bib is also suppressed

● Location changer also generates a report of OCLC numbers from suppressed bib records so that holdings can be batch deleted from OCLC as well

Use of Location Changer in weeding project

● Since Location Changer can’t adjust MFHD holdings statements (866) -- or create new MFHDs -- some cleanup is still required

● Titles with complex holdings will need to be adjusted manually if volumes on a MFHD were only partially withdrawn

Recent uses of Location Changer at GSU

● Flipped a subset of the obsolete Rare location to a new Treasures location

● Flipped the rest of the Rare items to general Special Collections location

● Applied a temp location to a group of items being digitized

● Suppressed print MFHDs only for gov docs bibs with multiple locations/formats attached

URL Changer

● Change URL stems● Change or delete 856 $x, $y, $z and/or $3● Works on URLs in both bibs and MFHDs!

Recent uses of URL Changer at GSU

● Changing proxy prefix● Changing URL stem when the name of our

institutional repository changed

Recent uses of URL Changer at Clayton State?

● Adding proxy prefixes and other URL changes made using a different method: validation rules in Cataloger’s Toolkit

● URL Changer is less flexible but much more user friendly

Strawn’s RDA Conversion tool

● Use to make batch changes to pre-RDA access points in bib (or authority) recordso Changes are the same as those made during the

“Phase 2” changes to the LC authority file● Can also generate reports on other access

points that may require review

Changes that can be made to access points

● Expanding abbreviations (“arr.” “acc.” “unacc.” “Dept.”)● Expanding or deleting “O.T.” and “N.T.”, as required● Replacing “violoncello” with “cello”● Replacing “Koran” with “Qur’an”● Changing “Selections” in $a or $t to “Works. $k Selections”● Shifting $k “Selections” before $l and $f● Adding or removing parentheses for certain instances of $c in personal

name access points● Expanding abbreviations of months● Replacing “fl.” and “ca.” with “active” and “approximately”● Replacing “b.” or “d.” with open hyphenated dates● Expanding dates (e.g., changing “1090 or 91” to “1090 or 1091”)

Changes that can be made to access points

● The RDA Conversion tool allows some “options” for access points…

● But you will probably want to follow standard practice

Didn’t want to change dates in subject access points.

Reports generated for review

● Access points that might not be RDA compliant, or that need human review to appropriately correcto Musical ensemble terms in $mo “Polyglot” or “&” in $lo “Libretto” or “Text” in $so Treatieso Conferenceso Personal name access point $c terms that might be improper

● Access points flagged due to possible typos, or errors in subfield coding

Record Reloader

● An alternative to bulk import with webadmin for overlaying existing Voyager records

● Can use VgerSelect to output a MARC file, make changes, and then overlay with Record Reloader

Record Reloader main screen

Considerations for options & settings in Strawn programs

● Do you need to limit to one or more owning libraries?o Select the owning library/ies you’re interested in

● When making changes to the database, do you want the changes to be reflected in the Universal Catalog?o Choose the appropriate “happening”/ “cataloging”

location

Selected options in URL Changer

Global Data Change

GSU’s experience with Global Data Change

● GSU and GSU Law have run a total of 3 successful data change jobs in 2013o change 049 GLLM to GLLOo add 007 fields to DVD records lacking themo delete an obsolete 590 note from selected microform

records● GDC is not currently working for us!

o “record export failed”--error logso have tickets in with USG and Ex Libris

Basic steps for GDC

● Create a record set● Create one or more data change rule sets● Create a data change rule set group with one or

more rule setso keep it simple!

● Preview● Run data change job

o everything works until we run jobs--they fail

My Add DVD 007 rule set

GDC preview of a “replace string with string” example

Cool things about GDC

● Save record sets, rule sets, rule set groups for reuse● Multiple ways to create a record set:

o From a list of record IDs previously identifiedo From a search of the same indexes as in the

cataloging cliento Scan an existing record set or the entire database

for desired criteria (set up scan rules & run scan jobs within GDC)

● Preview

Not so cool things about GDC

● IT DOESN’T WORK (for us *right now*)● Creating rules is not intuitive● Takes some trial and error (this is why you preview)● Mysterious “validation errors”

o Missing 005?o Corrupted or obsolete data in LDR/008?o These errors don’t cause the job to fail--these records are

just skipped in a successful job

Validation rules in Cataloger’s Toolkit

Validation rules in Cataloger’s Toolkit

● Validation rules are extremely versatile at making batch changes to bibs and MFHDs

● Like GDC, validation rules will make changes directly to your database

● Like GDC, this method can be unintuitive...

Using custom Validation Rules

● Rules are contained in plain text files that can be edited in a text editor

● Rules have their own peculiar syntax● Cataloger’s toolkit comes with a default set

of canned validation rules● You can add new rules to the file or create a

supplemental file

Sample Validation Rules

● A rule to add the specified 007 to every bib record that lacks an 007:#4=BDFMPSU 007! F <37:007,,cr_cn_||||||||>

● A rule to perform a find and replace in every MFHD 856 $u:

#4=H 856/u F <43:856/u,http://xxxxxx/,http://zzzzzz/>

Defining the records to be changed

● Rules are normally run on sets of records specified by bib ID or MFHD ID in a text file

● These files can be generated with some other tool, such as VgerSelect

Examples of simple Toolkit validation changes

● Batch changes to e-resource vendor recordso Adding and/or fixing 006-008 fields in bibso Adding and/or fixing GMDso Adding genre headingso Adding and/or fixing 007-008 fields in MFHDso Standardizing SMD in 300 $a and deleting 300 $c

● Added proxy prefixes to our NetLibrary URLs

More complex projects using Toolkit validation rules

● Copied classification portion of call number from bib to MFHD for our ebooks

● Added local call numbers in the form “Streaming Video (Title)” to our streaming video bibs and MFHDs

● Adding genre headings to our feature films based on data compiled from IMDB by a student assistant

IMDB feature film genre heading project

● Used VgerSelect to extract bib IDs and all 655 fields from our VHS and DVD records

● Used Excel to filter by “Feature films” heading (1246 resulting bibs)

● Student assistant searched for titles in IMDB and added genre information to spreadsheet

IMDB feature film genre heading project

IMDB feature film genre heading project

● Used Excel to filter for particular genres● Then filtered out records that already had

the corresponding genre heading● Ran a validation rule on the remaining bib

IDs to add the corresponding heading, such as:

#3=BDFMPSU 245 F <37:655,_7,|aWar_films.|2lcgft>

MarcEdit by Terry Reese

Possible uses of MarcEdit● Edit a file received from a vendor or other source

before batchloading● Use the Connexion Bib File Reader plug-in to work

with a group of WorldCat records● Use the WorldCat APIs to search, update holdings,

create or replace records, and more● Extract a MARC file from Voyager with VgerSelect,

make changes, and overlay

Recent uses of MarcEdit at GSU● Clean up ETD records harvested from the IR into

WorldCat before loading into Voyagero I have 15+ MarcEdit steps, but can pre-set up and save most of them

in Task Lists!

● Add 9XX field(s) to most vendor batchloads to identify different e-resource sets, e.g.,o SFX MARCito YBPeBookApproval

ETDs & MarcEdit

● Create a Connexion local save file of harvested records since last Voyager load

● Use Connexion Bib File Reader plug-in to bring into MarcEdit

● Basic cleanup, including:o add 007o delete unwanted fields generated during harvestingo add a 970o RDA Helper to add 33Xo create local headings for the discipline using Swap Field

=LDR 00000cam 22000003u 4500=001 868162197=005 20140226122043.0=008 140214s2012 xx om 000 0 und d=040 \\$aGSU$cGSU$dGSU=042 \\$adc=049 \\$aGSUU=100 1\$aBrien, Spencer T.=245 10$aThree Essays on the Formation and Finance of Local Governments$h[electronic resource].=260 \\$bScholarWorks @ Georgia State University$c2012-01-06T08:00:00Z=500 \\$aapplication/pdf=502 \\$aThesis / Dissertation ETD=653 \\$aproperty tax=653 \\$acontract cities=653 \\$alocal government=653 \\$atax relief=653 \\$agovernment outsourcing=653 \\$atax exemption=655 \4$atext=786 08$nPublic Management and Policy Dissertations=856 40$uhttp://scholarworks.gsu.edu/pmap_diss/37$3Item Resolution URL$xThis 856 field was generated using the

WorldCat Digital Collection Gateway$yView online$iPut this Resolution URL in a web browser to view this item.=856 40$uhttp://scholarworks.gsu.edu/cgi/viewcontent.cgi?article=1036&context=pmap_diss=029 0\$aGSU$boai:scholarworks.gsu.edu:pmap_diss-1036$chttp://scholarworks.gsu.edu/do/oai/

publication:pmap_diss$tDGCNT

Harvested ETD record “before” MarcEdit

=LDR 00000cam 22000003i 4500=001 868162197=005 20140226122043.0=007 cr=008 140214s2012 gau om 000 0 eng d=035 \\$a(OCoLC)ocn868162197=040 \\$aGSU$cGSU$dGSU$beng=042 \\$adc=049 \\$aGSU1=100 1\$aBrien, Spencer T.=245 10$aThree Essays on the Formation and Finance of Local Governments.=260 \\$aAtlanta, Ga. :$bGeorgia State University,$c2012.=300 \\$a1 online resource.=336 \\$atext$btxt$2rdacontent=337 \\$acomputer$bc$2rdamedia=338 \\$aonline resource$bcr$2rdacarrier=500 \\$aapplication/pdf=502 \\$aThesis / Dissertation ETD=653 \\$aproperty tax=653 \\$acontract cities=653 \\$alocal government=653 \\$atax relief=653 \\$agovernment outsourcing=653 \\$atax exemption=690 \\$aDissertations $xPublic Management and Policy.=786 08$nPublic Management and Policy Dissertations=856 40$uhttp://scholarworks.gsu.edu/pmap_diss/37=029 0\$aGSU$boai:scholarworks.gsu.edu:pmap_diss-1036$chttp://scholarworks.gsu.edu/do/oai/ publication:pmap_diss$tDGCNT=970 \\$aETDBulkMarch2014

Added 33X fields with RDA Helper (requires correct 007)

Created a 690 from the 786 using Swap Field & Edit Subfield

Created 035 from 001 using Swap Field & Edit Subfield

Added 260 $a, edited $b with Edit Subfield, stripped extra characters from $c with a regex in Find/Replace

Added basic 007 using Add/Delete Field

Deleted unwanted 856 subfields with Edit Subfield

Added 970 with Add/Delete Field (so I can identify records if things go wrong!)

Harvested ETD record “after” MarcEdit

Don’t fear the RegEx

● Ask the MarcEdit list for help (or search the archives)

● My First RegEx!o I had a bunch of dates in 260 $ c like this:

$c2013-05-01T07:00:00Z $c2011-05-07

o Total length varied, but I just wanted to keep the first four characters

My First RegEx!

MarcEdit recommendations● Ask for help on the MarcEdit listserv● Save your original file before making changes● Save frequently after *successful* edits● There is an Undo button, but it will only undo your

last change● MarcEdit is great when you need to apply multiple

changes to a group of records● Just. Try. It. (Trust me!)

An alternative to GDC

● Output a MARC file from Voyager with VgerSelect

● Make changes with MarcEdit● Overlay the records with Record Reloader● (VgerSelect & Record Reloader require

ODBC drivers)

Putting it all togethera Special Collections project at GSU

Splitting the Rare location

● Special Collections staff separated the *really* rare items from the existing Rare location for a new Treasures location

● Most items in Rare were integrated into the existing general Special Collections (Spec.) location

● ~1000 records to Treasures● ~1200 records to Spec.

Lots of reports on the Rare location● VgerSelect or Access Reports to generate bib IDs for all records with a

Rare MFHD attached● VgerSelect to retrieve the existing 590 notes and more● Access Reports to find *all*the attached MFHDs

o Some had existing Spec. copies or multiple Rare, requiring some extra attention to 590 notes

o I created a new table of the bib IDs & ran query to retrieve MFHDs● Also needed MFHD ids & suppression status

o need Access Reports to get suppression status

Location Changer to update MFHDs & items

VgerSelect + MarcEdit + Record Reloader to update bibs

● Output a MARC file with VgerSelect using the appropriate list of bib IDs

● MarcEdito Edit Subfield to change location codes in 049o For Rare to Spec lists, Add/Delete Field to add a

standard 590: Special Collections copy: Rare Book Collection.

● Record Reloader to overlay bibs

Bibliographic maintenance projects

● Use VgerSelect and Excel to locate invalid, obsolete, contradictory or missing data, such as:o “N/A” (or other invalid codes/typos) in 008/Lango Sound recordings, videorecordings, maps, microforms lacking 007so records with a 240 $l and no 041 (or an 041 0X)o incompatible 008/DtSt, 008/Date, 260/264 $c combinations

● List of possible maintenance projects at: http://www.carli.illinois.edu/products-services/i-share/cat/Cat-maintpriority

Bibliographic maintenance projects

● RDA conversion of descriptive fieldso Gary’s RDA conversion programo MarcEdit RDA Helper

● Upcoming changes to music subject headingso Musical forms will be moved from LCSH to LCGFTo LCMPT terms in 382?

● Changes needed to facilitate data migration?

Strawn’s RDA Conversion tool

● Can be used to make RDA style batch changes to data other than access points, as wello Generate 336, 337 and 338 fields from data in record -- or -- force

336, 337 and 338 to certain values for a given set of recordso Expand certain abbreviations in certain descriptive fields (see Toolkit

documentation)o Remove GMDso Reformat a 502 into RDA style

● Tool reports when selected data cannot be convertedo For example, if data such as 007 and SMD are missing, ambiguous or

contradictory

MarcEdit RDA Helper

● Similar capabilities to Gary’s RDA Conversion tool● Will also generate the new 34X fields● Option to generate GMDs on RDA records if

desired● GSU uses to add 33X to ETDs

o limited testing on other fileso explore using on selected or all vendor batchloads?

RDA Helper default options

Recommendations

● Start with easy programs, like Bib Delete or Location Changer

● Test on small groups of 5-10 records● Get comfy with reports! (Access Reports and/or

VgerSelect)o Generate lists of record numbers to work witho Check your results

Recommendations

● *Always* preview/test● Give yourself time to experiment and learn● Ask questions…

o MarcEdit-L o Voyager-L for Strawn programs

Coming attractions in Voyager 9

● Some Location Changer functionality incorporated into Pick and Scan

● Options to delete, suppress or unsuppress added to GDC● Option to export record sets and files added to GDC● See Voyager product update from ELUNA and

Voyager 9 release notes (requires login to Ex Libris Documentation Center)

Links● Gary Strawn’s programs

o His excellent documentation is also available hereo RDA Conversion program

● Voyager-L

ELUNA archives (login required)o good source of presentations on GDC and Strawn programs

MarcEdit download● MARCEDIT-L discussion list● Terry Reese’s YouTube videos on MarcEdit● Fear No Longer Regular Expressions

Contact

Adam Kubikadamkubik@clayton.edu

Susan Wynneswynne@gsu.edu https://www.facebook.com/susan.wynne.94https://gsu.academia.edu/SusanWynne