Where has all the AV content gone? How do we preserve it for the future? Jim Lindner Media Matters.

Where has all the AV Where has all the AV content gone? How do we content gone? How do we preserve it for the future?preserve it for the future?

Jim LindnerJim Lindner

Media MattersMedia Matters

Our Audio Visual HeritageOur Audio Visual Heritage

How we will be understood by history will How we will be understood by history will in part be based on what materials they in part be based on what materials they have to study.have to study.

We are in the early years of a technology We are in the early years of a technology evolution that will certainly take hundreds evolution that will certainly take hundreds of years, maybe thousands.of years, maybe thousands.

What is at risk are the documents of our What is at risk are the documents of our time.time.

What are these Documents?What are these Documents?

Feature, Documentary, Independent FilmFeature, Documentary, Independent Film

Network, Independent, Art TelevisionNetwork, Independent, Art Television

Industrial / Educational Film and VideoIndustrial / Educational Film and Video

Audio from Network to Soundtrack to Field Audio from Network to Soundtrack to Field Recordings and Oral HistoriesRecordings and Oral Histories

Community RecordingsCommunity Recordings

Home RecordingsHome Recordings

Where has the content gone?Where has the content gone?

We Have Lost Much More then most We Have Lost Much More then most people suspectpeople suspect– All but a small fraction of the films of the silent All but a small fraction of the films of the silent

film era are lost foreverfilm era are lost forever– Essentially all of television “pre history” is lostEssentially all of television “pre history” is lost

E.G. Broadcasts of J.L. BairdE.G. Broadcasts of J.L. BairdTV Before Tape – All lost except what was a kine TV Before Tape – All lost except what was a kine or news film and has survivedor news film and has survivedTV After Tape – A huge amount has been lost TV After Tape – A huge amount has been lost

– 1958 – 1978 3000 news shows remain versus 11,000 1958 – 1978 3000 news shows remain versus 11,000 DaysDays

Where has all the content gone?Where has all the content gone?

– Landfill – space, cost, not “needed” anymoreLandfill – space, cost, not “needed” anymore– Erasure – intentionally to save money and Erasure – intentionally to save money and

unintentionallyunintentionally– Into brown cardboard boxes after the edit Into brown cardboard boxes after the edit

sessions, unlabeled, sitting in warehouses or sessions, unlabeled, sitting in warehouses or worseworse

– Closets, Attics, BasementsClosets, Attics, Basements

Where has all the content gone?Where has all the content gone?

Some of what is left has found its way to an Some of what is left has found its way to an archive.archive.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

How Much of it survives?How Much of it survives?

No one really knows - but there are No one really knows - but there are estimates…..estimates…..

CBC – 300,000 tapesBBC – 600,000 hoursMTV – 1.2 Million hoursUNESCO estimate –200 million hours of culturally important archives at risk

What do the Records in What do the Records in Archives look like?Archives look like?

For the most part our A/V heritage is :For the most part our A/V heritage is :A mixture of analog and digital formats A mixture of analog and digital formats including…including…

A Vast Array of Media Types A Vast Array of Media Types and Formatsand Formats

– Film and Film Elements Film and Film Elements Nitrate – Acetates – Polyester - Mixtures Nitrate – Acetates – Polyester - Mixtures (Magnetic Full Coat)(Magnetic Full Coat)

– AudioAudio Disc, Cylinder, Vinyl, Wire (steel), Polyester Disc, Cylinder, Vinyl, Wire (steel), Polyester

(PET), Acetate, CD-R(PET), Acetate, CD-R

– Video – Polyester (PET) Video – Polyester (PET)

Media is EXTREMELY Media is EXTREMELY FRAGILEFRAGILE

– Nitrate Highly CombustibleNitrate Highly Combustible– Acetates – Vinegar SyndromeAcetates – Vinegar Syndrome– Binder issues – Sticky ShedBinder issues – Sticky Shed– Laminates – DelaminateLaminates – Delaminate

Media is Subject to DisastersMedia is Subject to Disasters

Natural…..Natural…..– Chemical DeteriorationChemical Deterioration– Physical/Mechanical DamagePhysical/Mechanical Damage– Natural DisastersNatural Disasters

… … and Man Madeand Man Made

WarWarTheftTheftLossLossBad Economic TimesBad Economic TimesFormat ObsolescenceFormat ObsolescenceTechnological EvolutionTechnological Evolution

““Digital” Media is subject to Digital” Media is subject to many of the same perilsmany of the same perils

Format Type may be irrelevant in some Format Type may be irrelevant in some situationssituations

Storage as a file adds many levels of Storage as a file adds many levels of complexity that did not exist with complexity that did not exist with

Analog MediaAnalog Media

File Format CompatibilityFile Format Compatibility

Application Level CompatibilityApplication Level Compatibility

Operating System CompatibilityOperating System Compatibility

Firmware / BIOS Level CompatibilityFirmware / BIOS Level Compatibility

Hardware / Controller Level CompatibilityHardware / Controller Level Compatibility

How?How?

QuickTime™ and aTIFF (LZW) decompressor




What about the Content in the What about the Content in the File? File?

Content Management Content Management – Few standards across collections – no union Few standards across collections – no union

catalogcatalogLet me see all the films that have Ford carsLet me see all the films that have Ford cars

– Description of Sounds and Pictures with Description of Sounds and Pictures with wordswords

– Primitive Searching toolsPrimitive Searching tools

No real LONG term strategy:No real LONG term strategy:

Best Practice Environmental ControlBest Practice Environmental Control– Short term postponement of the Short term postponement of the

inevitableinevitable

No single media type lasts foreverNo single media type lasts foreverNo single format / technology lasts No single format / technology lasts foreverforeverNo single location lasts forever.No single location lasts forever.

How long do we store the stuff How long do we store the stuff anyhow – how long is forever?anyhow – how long is forever?Manufacturers think 90 days – well OK – Manufacturers think 90 days – well OK – 10 Years10 YearsWe have paper documents for hundreds of We have paper documents for hundreds of yearsyearsWe have clay tablets for thousands of We have clay tablets for thousands of yearsyearsWe have cave paintings for many We have cave paintings for many thousands of yearsthousands of years

Can we realistically reformat Can we realistically reformat forever?forever?

Consider the resources required to Consider the resources required to reformat every 25 years, physical space reformat every 25 years, physical space for each object, cost for media for each object, cost for media Consider the environmental cost to Consider the environmental cost to produce all that mediaproduce all that mediaIs it desirable even if you did?Is it desirable even if you did?

What we are doing is What we are doing is not working very well!not working very well!

We need to rethink and We need to rethink and try some new things…try some new things…

Can the past help us Can the past help us with the future?with the future?

The First Television was The First Television was Mechanical!Mechanical!

Mechanical television existed Mechanical television existed for quite some time – some for quite some time – some systems scanning horizontally systems scanning horizontally others vertically. John Logie others vertically. John Logie Baird generally is given credit Baird generally is given credit for the first working for the first working mechanical television system mechanical television system – as well as developing the – as well as developing the first way of RECORDING his first way of RECORDING his television signals.television signals.



This is what it looked likeThis is what it looked like

This is what it took to record TV…This is what it took to record TV…

Magnetic RecordingMagnetic Recording


In The Beginning:In The Beginning:– Initial market was for for TIME SHIFTING – Initial market was for for TIME SHIFTING –

Not for Editing or anything elseNot for Editing or anything else– No thought given to other markets or No thought given to other markets or

applicationsapplications– It did not take very long before other uses It did not take very long before other uses

were found for video recording and the were found for video recording and the manufacturers responded with new productmanufacturers responded with new product


Different Markets / Different NeedsDifferent Markets / Different Needs– Broadcast – High Quality, High Cost, Low Broadcast – High Quality, High Cost, Low

Quantity of Machines Sold, FlexibilityQuantity of Machines Sold, Flexibility– Industrial / Educational – Medium Quality, Industrial / Educational – Medium Quality,

Medium Cost, Higher Volume of Machines Medium Cost, Higher Volume of Machines Sold, SimplicitySold, Simplicity

These Markets were later segmented These Markets were later segmented Further and new markets like consumers Further and new markets like consumers came into beingcame into being


BroadcastBroadcast

Videotape RecordingVideotape Recording

MilitaryMilitary

Videotape RecordingVideotape Recording

Industrial / Educational Industrial / Educational

Maybe we can predict that…Maybe we can predict that…

Change and new formats have always Change and new formats have always been a part of our AV Heritage and we been a part of our AV Heritage and we need to think in those terms.need to think in those terms.

Innovation will continue - and perhaps the Innovation will continue - and perhaps the best we can do is not mess it up for the best we can do is not mess it up for the next generation of innovationnext generation of innovation

It will continue to get smaller, better and It will continue to get smaller, better and cheaper…. andcheaper…. and

There WILL be more of it!There WILL be more of it!



Here is the good news - Here is the good news - it is getting cheaper it is getting cheaper

and easier to store it!and easier to store it!

Economic Viability of storage as Economic Viability of storage as FILES versus VideotapeFILES versus Videotape

In Canberra – Today June 6In Canberra – Today June 6

Using Sony Stock Using Sony Stock

1 Digital Betacam Tape costs $27.08, 1 Digital Betacam Tape costs $27.08, Records 1 hour, Obsolete Format ?Records 1 hour, Obsolete Format ?

1 LTO3 Datatape costs $60.521 LTO3 Datatape costs $60.52QuickTime™ and a

TIFF (LZW) decompressorare needed to see this picture.



Economic Viability of storage as Economic Viability of storage as FILES versus VideotapeFILES versus Videotape

Using MPEG2 50 Mbit or MJPEG 2000Using MPEG2 50 Mbit or MJPEG 2000

1 hour of content requires 25 – 30 1 hour of content requires 25 – 30 Gigabytes using MJPEG2000 (24 using Gigabytes using MJPEG2000 (24 using MPEG2)MPEG2)

1 LTO3 tape stores 400 Gigabytes1 LTO3 tape stores 400 Gigabytes– 400 / 25 = 16400 / 25 = 16

1 LTO3 tape stores 16 Hours1 LTO3 tape stores 16 Hours

Cost to store 1 hour of contentCost to store 1 hour of content

LTO = $60.52 / 16 LTO = $60.52 / 16 Cost = $3.78 Per Hour Cost = $3.78 Per Hour

Digital Betacam = $27.08Digital Betacam = $27.08

Videotape is 7.16 Videotape is 7.16 TIMESTIMES more more expensive!!!!! – storing the exact same expensive!!!!! – storing the exact same quality.quality.

How Much Will Storage Cost, How Much Will Storage Cost, What about small collections?What about small collections?350 Hour Collection350 Hour Collection

350 x 30 Gigabytes each hour =350 x 30 Gigabytes each hour =

10,500 Gigabytes (10.5 Terabytes)10,500 Gigabytes (10.5 Terabytes)

Media Cost @ $.15 per Gigabyte = $1575Media Cost @ $.15 per Gigabyte = $1575

27 LTO Tapes (versus 700 Umatic)27 LTO Tapes (versus 700 Umatic)

2333 DVD’s….2333 DVD’s….

But what about the decks!?But what about the decks!?

LTO3 $3600 (Tandberg Internal)LTO3 $3600 (Tandberg Internal)

DVW 2000 $40,500DVW 2000 $40,500

UVW 1800 $9,00 (Used)UVW 1800 $9,00 (Used)

How is the Migration How is the Migration Performed?Performed?

VideoVideo

VideotapeAudioTime Code

Meta Data

Manual MigrationTedious and error prone

Requires constant attentionDifficult to maintain quality control

SlowTeam of 6 experts (3 per shift – 2 shifts per day) can migrate 5000 hrs/yr

ExpensiveRequires highly skilled laborNo economies of scale

No Metadata No history of past actions or condition

Automated Migration

Higher QualityVerification of tape condition and processConsistent quality control

Faster Multiple streams at a time (up to 6)Runs 24/7 (up to 140 hours per day)

Less Expensive10 to 40% of the cost of manual migration

What about What about CompressionCompression

??

Compression is NOT ALWAYS Compression is NOT ALWAYS BAD for archival applicationsBAD for archival applicationsDepends on the type of compressionDepends on the type of compression

Depends on the applicationDepends on the application

Depends on the implementationDepends on the implementation

Depends on the user (using things for Depends on the user (using things for what they were designed to do)what they were designed to do)

Compression is NOT ALWAYS Compression is NOT ALWAYS BAD for archival applicationsBAD for archival applicationsMust define termsMust define terms– Preservation and/or Access?Preservation and/or Access?

Must understand underlying issuesMust understand underlying issues– Technical ConsiderationsTechnical Considerations– Marketing ConsiderationsMarketing Considerations– Practical ConsiderationsPractical Considerations– Political Considerations (NTSC vs. PAL)Political Considerations (NTSC vs. PAL)

Many Different Standards - and Many Different Standards - and it IS confusing…it IS confusing…

MPEG - Moving Picture Experts GroupMPEG - Moving Picture Experts GroupSeveral major categories use entirely different approachesSeveral major categories use entirely different approaches

MPEG 1 Samples 4:2:0 at 1.2 Mbits/sec at 352x240, 30 FIELDS / MPEG 1 Samples 4:2:0 at 1.2 Mbits/sec at 352x240, 30 FIELDS / SecondSecond

MPEG 2 Samples 4:2:0 at 1.2 to 15 Mbits/sec at 704x496, 60 Fields MPEG 2 Samples 4:2:0 at 1.2 to 15 Mbits/sec at 704x496, 60 Fields / Second/ Second

MPEG 2 Professional Profile @ Main Level Samples 4:2:2 at about MPEG 2 Professional Profile @ Main Level Samples 4:2:2 at about 50-60Mbits/sec (variable) at 704x496, 60 fields/sec50-60Mbits/sec (variable) at 704x496, 60 fields/sec

MPEG 4MPEG 4MPEG 7 …. MetaDataMPEG 7 …. MetaData

Compression for A/V MaterialsCompression for A/V Materials

In general terminology there are two In general terminology there are two classes of Compression – LossLESS, and classes of Compression – LossLESS, and LossyLossy

Most Common formats in AV are LossyMost Common formats in AV are Lossy– MPEG2, MPEG4, H.264, Windows MediaMPEG2, MPEG4, H.264, Windows Media

LossLESS formats offer Uncompressed LossLESS formats offer Uncompressed Quality, with storage savings of about 3:1Quality, with storage savings of about 3:1– MJPEG 2000MJPEG 2000

Determination of Quality cannot Determination of Quality cannot be “It looks fine to me”be “It looks fine to me”

Do No HarmDo No Harm

Area of Rapid Technology DevelopmentArea of Rapid Technology Development

Real possibility of new class of “Orphan Real possibility of new class of “Orphan Video” by encoding type – VideocubeVideo” by encoding type – Videocube

What will Search Engines use in the What will Search Engines use in the future?future?

We Are Now in a Transition Phase We Are Now in a Transition Phase – and We Will Be for Some Time– and We Will Be for Some Time

• • Digital asset management is very new and it Digital asset management is very new and it will take some time for the technology and will take some time for the technology and standards to mature.standards to mature.• • Migration of analog media to digital systems is Migration of analog media to digital systems is still very unclear – compression/formats/bitratesstill very unclear – compression/formats/bitrates• • Current production process still has Current production process still has “conventional” media used at times“conventional” media used at times• • Storage is still too expensive, networks are still Storage is still too expensive, networks are still too slow and unreliable, systems are still largely too slow and unreliable, systems are still largely incompatible.incompatible.

What Happens When This…..What Happens When This…..

Turns Into ThisTurns Into This

What will the role of the future What will the role of the future A/V Archivist be?A/V Archivist be?

• • Archivist as Information “miner”Archivist as Information “miner”

• • Archivist as architect of information Archivist as architect of information standards and systemsstandards and systems

• • Archivist as Information ManagerArchivist as Information Manager

Where has all the AV content gone? How do we preserve it for the future? Jim Lindner Media Matters.

Documents

Transcript of Where has all the AV content gone? How do we preserve it for the future? Jim Lindner Media Matters.