1 The VQEG HDTV Test Plan A presentation by Vivaik Balasubrawmanian (Intel) Stephen Wolf (NTIA/ITS)

11

The VQEG HDTV Test The VQEG HDTV Test PlanPlan

A presentation byA presentation by

Vivaik Balasubrawmanian Vivaik Balasubrawmanian (Intel)(Intel)

Stephen Wolf (NTIA/ITS)Stephen Wolf (NTIA/ITS)

22

OutlineOutline

Introduction: Setting the stageIntroduction: Setting the stageHDTV Source Scenes and HRCsHDTV Source Scenes and HRCsTest methodologiesTest methodologiesKey challengesKey challengesProposed VQEG approachProposed VQEG approachFinal thoughtsFinal thoughts

33

HD is no longer the future. It is HD is no longer the future. It is the present!the present!

““NBC will upgrade the rest of the network's top-NBC will upgrade the rest of the network's top-rated late-night lineup into HDTV in 2005”rated late-night lineup into HDTV in 2005”

““The Fox network is better prepared then ever for The Fox network is better prepared then ever for its hi-def broadcast of Super Bowl XXXIX.” its hi-def broadcast of Super Bowl XXXIX.”

““The swanky future of hotels is a big-screen HDTV The swanky future of hotels is a big-screen HDTV set.”set.”

““2005 would be the year of high definition video 2005 would be the year of high definition video and TV”and TV”

““Shipments of High Definition televisions and set-Shipments of High Definition televisions and set-top boxes are gaining momentum”top boxes are gaining momentum”

““HDTV content is increasing rapidly”HDTV content is increasing rapidly”

44

HD studio workflows are HD studio workflows are mainstream mainstream

““Adobe, HP, Intel, Microsoft, and Dell have joined Adobe, HP, Intel, Microsoft, and Dell have joined together to deliver a line of open, scalable, together to deliver a line of open, scalable, certified desktop HDV and HD solutions to deliver certified desktop HDV and HD solutions to deliver optimal performance at a price point that is optimal performance at a price point that is unbeatable today.” unbeatable today.”

““JVC's ProHD initiative represents an economical JVC's ProHD initiative represents an economical solution to getting the stations into the HD solution to getting the stations into the HD production business. The company has products production business. The company has products to help with this transition on two fronts.” to help with this transition on two fronts.”

55

Content acquisition is Content acquisition is affordableaffordable

Panasonic: The AG-HVX200 DVCPRO HD P2 represents the Panasonic: The AG-HVX200 DVCPRO HD P2 represents the company's first HD solid-state handheld camcorder. The AG-company's first HD solid-state handheld camcorder. The AG-HVX200 has two card slots and records in 1080/60i, 30p and HVX200 has two card slots and records in 1080/60i, 30p and 24p; 720/60p, 30p and 24p; and DVCPRO50, DVCPRO or DV. 24p; 720/60p, 30p and 24p; and DVCPRO50, DVCPRO or DV. It also records 480i and 480p on Mini-DV tape from a built-in It also records 480i and 480p on Mini-DV tape from a built-in drive. AG-HVX200 will come available in the fall for about drive. AG-HVX200 will come available in the fall for about $5,995. $5,995.

Alec Shapiro, senior vice president of Sony Electronics' Alec Shapiro, senior vice president of Sony Electronics' Broadcast and Production Systems Division, touted the Broadcast and Production Systems Division, touted the acceptance Sony's HDC-1000 1080P camera and the XDCAM acceptance Sony's HDC-1000 1080P camera and the XDCAM tapeless acquisition system have found in the industry. tapeless acquisition system have found in the industry.

In the high-end HD production arena, Shapiro noted the In the high-end HD production arena, Shapiro noted the Sony's HDCAM format is used by virtually 100 percent of the Sony's HDCAM format is used by virtually 100 percent of the

network sitcoms produced in the United States.network sitcoms produced in the United States.

66

Viewer adoption is Viewer adoption is acceleratingaccelerating

Maxent (tm), announced the availability of its value-priced, high-Maxent (tm), announced the availability of its value-priced, high-performance 26-inch WXGA LCD Display through Costco Wholesale performance 26-inch WXGA LCD Display through Costco Wholesale stores nationwide at a retail price of $1,199. stores nationwide at a retail price of $1,199.

HDTVs keep getting more affordable. Olevia's new 32-inch LCD HDTVs keep getting more affordable. Olevia's new 32-inch LCD HDTV has a lot of the right ingredients for a good low-cost HDTV.HDTV has a lot of the right ingredients for a good low-cost HDTV.

10m homes around the world were watching HDTV programming on 10m homes around the world were watching HDTV programming on a high-definition TV set, according to research from market analysts a high-definition TV set, according to research from market analysts In-Stat. In-Stat.

By the end of 2005, the worldwide total of these HDTV service By the end of 2005, the worldwide total of these HDTV service households is projected to reach 15.5m. By 2009, HDTV households households is projected to reach 15.5m. By 2009, HDTV households worldwide are forecast to reach 52m. worldwide are forecast to reach 52m.

HDTV services are currently widely available in five countries: HDTV services are currently widely available in five countries: Australia, Canada, Japan, the United States, and South Korea. Australia, Canada, Japan, the United States, and South Korea.

Nearly half of all consumers plan to make their next television Nearly half of all consumers plan to make their next television purchase a high-definition (HDTV) television set, according to the purchase a high-definition (HDTV) television set, according to the CEA. CEA.

77

Technology building blocks are Technology building blocks are in placein place

British Sky Broadcasting (BSkyB) has selected Thomson as British Sky Broadcasting (BSkyB) has selected Thomson as its primary HD decoder set top supplier as it prepares to roll its primary HD decoder set top supplier as it prepares to roll out a new digital HDTV service in 2006. The company will out a new digital HDTV service in 2006. The company will use H.264-based advanced video coding for its HD service. use H.264-based advanced video coding for its HD service.

Royal Philips Electronics today unveiled a High-Definition Royal Philips Electronics today unveiled a High-Definition Multimedia Interface™ (HDMI™ ) receiver integrated circuit Multimedia Interface™ (HDMI™ ) receiver integrated circuit (IC), the TDA9975A, to meet consumer demands for (IC), the TDA9975A, to meet consumer demands for superior image quality in next-generation TVs. superior image quality in next-generation TVs.

The single cable digital multimedia solution has arrived. The single cable digital multimedia solution has arrived. High-Definition video and audio capability quietly rolled into High-Definition video and audio capability quietly rolled into the NZ home entertainment market in the lead-up to the NZ home entertainment market in the lead-up to Christmas, with the release of a number of DVD players, Christmas, with the release of a number of DVD players, projectors and TV sets from various manufacturers projectors and TV sets from various manufacturers featuring the new High-Definition Multimedia Interface featuring the new High-Definition Multimedia Interface (HDMI(HDMITMTM) connectivity standard.) connectivity standard.

88

Differentiation based on quality Differentiation based on quality is the next stepis the next step

OnHD.TV has generated international attention for our list OnHD.TV has generated international attention for our list of the 10 celebrities who look better or worse in crystal-of the 10 celebrities who look better or worse in crystal-clear High-Definition TV. The list has been featured in such clear High-Definition TV. The list has been featured in such publications as publications as The Chicago TribuneThe Chicago Tribune, , The National EnquirerThe National Enquirer, , The New YorkThe New York PostPost and TV shows such as and TV shows such as Inside EditionInside Edition and and The Fox Report With Shepard SmithThe Fox Report With Shepard Smith. .

Driving much of the company's growth is the worldwide Driving much of the company's growth is the worldwide move to HD production, and Thomson has products for the move to HD production, and Thomson has products for the acquisition, editing and distribution of both SD and HD acquisition, editing and distribution of both SD and HD content. The top of the company's line in terms of image content. The top of the company's line in terms of image quality is its Viper FilmStream digital cinema camera.quality is its Viper FilmStream digital cinema camera.

99

Hence the need for standardized Hence the need for standardized metricsmetrics

To verify and make credible claimsTo verify and make credible claimsTo objectively validate the promised To objectively validate the promised

media experiencemedia experience

1010

OutlineOutline

Introduction: Setting the stageIntroduction: Setting the stageHDTV Source Scenes and HRCsHDTV Source Scenes and HRCsTest methodologiesTest methodologiesKey challengesKey challengesProposed VQEG approachProposed VQEG approachFinal thoughtsFinal thoughts

1111

HD landscape todayHD landscape today

BroadcastHead-end

Set-top Box

DTH Satellite

Cable Distribution

IP TV

1212

Content acquisitionContent acquisitionQuality: Consumer ($6K), Professional Quality: Consumer ($6K), Professional

($50K), Studio ($110K) cameras in use.($50K), Studio ($110K) cameras in use.Storage: HDV, HDCAM, DVCPRO-HD Storage: HDV, HDCAM, DVCPRO-HD

decks are prevalent.decks are prevalent.Frame Rate: 24p, 29.97i, 30i, 59.94p, Frame Rate: 24p, 29.97i, 30i, 59.94p,

60p.60p.Frame Size: 1920 x 1080 (1080p), 1920 Frame Size: 1920 x 1080 (1080p), 1920

x 540 (1080i), 1280 x 720 (720p).x 540 (1080i), 1280 x 720 (720p).

1313

Source Scenes - CopyrightSource Scenes - Copyright

Copyright IssuesCopyright Issues Public DomainPublic Domain Shot by VQEG Members (put into public Shot by VQEG Members (put into public

domain or approved for unlimited research domain or approved for unlimited research use)use)

Purchased specifically for testing (e.g., Purchased specifically for testing (e.g., VClips)VClips)

Proposal: All scenes must have non-Proposal: All scenes must have non-expiring use clause for VQEG testing expiring use clause for VQEG testing and be obtainable by any interested and be obtainable by any interested partyparty

1414

Source Scenes - QualitySource Scenes - Quality

Formats?Formats?Camera/recording quality?Camera/recording quality?Uncompressed?Uncompressed?Mildly compressed?Mildly compressed?Professional Consumer (e.g., Sony Professional Consumer (e.g., Sony

HVR-Z1U)?HVR-Z1U)?Allowing some percentage of Allowing some percentage of

lower grade material will greatly lower grade material will greatly increase available sourcesincrease available sources

1515

Source Scenes - Source Scenes - CharacteristicsCharacteristics

Motion?Motion?Detail?Detail?Color?Color?Size of source scene pool?Size of source scene pool?Percent unknown to proponents?Percent unknown to proponents?

1616

HRC CharacteristicsHRC Characteristics

Bit-rates?Bit-rates?Coder/decoders?Coder/decoders?Error conditions?Error conditions?Pre/post processing?Pre/post processing?Percent unknown to proponents?Percent unknown to proponents?

1717

CompressionCompression

Technologies: MPEG4-AVC, MPEG2, Technologies: MPEG4-AVC, MPEG2, VC-1, DV compression.VC-1, DV compression.

Bit rate: 8 Mbps – 25 MbpsBit rate: 8 Mbps – 25 MbpsRate Control: Variable and constant.Rate Control: Variable and constant.Rate Conversion: 3:2 pull down.Rate Conversion: 3:2 pull down.

1818

TransmissionTransmission

Technologies: Satellite, Cable, IPTechnologies: Satellite, Cable, IPPacket loss: 0-25%Packet loss: 0-25%

1919

Decode and display Decode and display processingprocessing

3:2 pull down3:2 pull down Image enhancementImage enhancementDe-interlacingDe-interlacingScalingScaling

2020

Display ConditionsDisplay Conditions

Should we test many types of video Should we test many types of video displays or should we use a high displays or should we use a high quality “transparent” video display?quality “transparent” video display?

What combinations of display types What combinations of display types and video formats should be tested?and video formats should be tested?

2121

Display technologiesDisplay technologies

Broadcast CRTBroadcast CRTRear projectionRear projectionLCDLCDPlasmaPlasmaProjectorsProjectors

2222

HRC vectors at a glanceHRC vectors at a glance

Content acquisition:Content acquisition: Capture qualityCapture quality Storage formatStorage format Frame rateFrame rate Frame sizeFrame size

Compression:Compression: Coding technologyCoding technology Bit rateBit rate Rate controlRate control Rate conversionRate conversion

Transmission:Transmission: TechnologyTechnology Packet lossPacket loss

Decode and display Decode and display Processing:Processing: 3:2 pull down3:2 pull down Image enhancementImage enhancement De-interlacingDe-interlacing ScalingScaling

Display:Display: Broadcast CRTBroadcast CRT Rear ProjectionRear Projection LCDLCD PlasmaPlasma ProjectorsProjectors

2323

OutlineOutline

Introduction: Setting the stageIntroduction: Setting the stageHDTV HRCsHDTV HRCsTest methodologiesTest methodologiesKey challengesKey challengesProposed VQEG approachProposed VQEG approachFinal thoughtsFinal thoughts

2424

Test methodsTest methods

Subjective testingSubjective testingSequence processingSequence processing

2525

Subjective MethodologySubjective Methodology

What subjective testing methodology What subjective testing methodology should we use?should we use?Double/single stimulus?Double/single stimulus?Explicit/hidden reference?Explicit/hidden reference?Continuous/discrete assessment?Continuous/discrete assessment?

2626

Subjective testingSubjective testing

Double Stimulus Impairment ScaleDouble Stimulus Impairment Scale 4 times picture height for all HD formats4 times picture height for all HD formats Different display technologiesDifferent display technologies 24 subjects per test with:24 subjects per test with:

20/20 visual acuity with or without eye glasses20/20 visual acuity with or without eye glasses Normal color visionNormal color vision Familiarity with the languageFamiliarity with the language

No audioNo audio Subjects screened per BT.500Subjects screened per BT.500 Subjects rejected if failed votes >1 in one Subjects rejected if failed votes >1 in one

session.session.

2727

Sequence processingSequence processing

1080p, 1080i and 720p formats.1080p, 1080i and 720p formats.No up-sampling or down-sampling.No up-sampling or down-sampling.

2828

OutlineOutline


2929

Key ChallengesKey Challenges

HD ContentHD ContentReliable HD playback for subjective Reliable HD playback for subjective

testingtesting Investments required for HD testingInvestments required for HD testingTools for HRC error injectionTools for HRC error injectionReference display processing Reference display processing

algorithmsalgorithmsSelection of back channel bandwidth for Selection of back channel bandwidth for

HDHD

3030

OutlineOutline


3131

Step 1: Subjective testingStep 1: Subjective testing

Select sequences and run a subjective Select sequences and run a subjective test to get MOS results.test to get MOS results.

Include as many display technologies Include as many display technologies as possible.as possible.

Make sure a variety of HRCs are used Make sure a variety of HRCs are used for the test sequences.for the test sequences.

Compile results from multiple labs to Compile results from multiple labs to get a consolidated score for the get a consolidated score for the sequences.sequences.

3232

Step 2: Evaluation of existing Step 2: Evaluation of existing modelsmodels

Run through existing (already Run through existing (already submitted) models through the same submitted) models through the same sequences.sequences.

Follow procedure and rules for Follow procedure and rules for calibration in the test plan.calibration in the test plan.

Include full reference, reduced Include full reference, reduced reference and no reference models in reference and no reference models in the test.the test.

3333

Step 3: Results analysis and next Step 3: Results analysis and next stepssteps

Analyze objective results to get a Analyze objective results to get a correlation to the subjective results.correlation to the subjective results.

Follow evaluation metrics in the test plan.Follow evaluation metrics in the test plan. Decide on next steps based on the results.Decide on next steps based on the results.

If correlations are high, we may have a model If correlations are high, we may have a model for HD TV.for HD TV.

If correlations are low, proponents will have a If correlations are low, proponents will have a test set to tweak their models.test set to tweak their models.

3434

Lab readiness pollLab readiness poll

Of the labs represented here:Of the labs represented here:How many of them have HD playback How many of them have HD playback

capability?capability?Of those who don’t have it now, how Of those who don’t have it now, how

many of them are planning to have this many of them are planning to have this capability in the next 3 - 6 months.capability in the next 3 - 6 months.

How many labs are willing to run HD How many labs are willing to run HD subjective tests in the next 3 months?subjective tests in the next 3 months?

3535

OutlineOutline


3636

Final thoughtsFinal thoughtsHD TV metrics are in urgent need.HD TV metrics are in urgent need.We should test existing models for We should test existing models for

their correlations before seeking new their correlations before seeking new ones.ones.

HD testing is happening now in HD testing is happening now in organizations. A timely standardized organizations. A timely standardized metric will have the greatest impact.metric will have the greatest impact.

VQEG should complete testing the VQEG should complete testing the existing models in the next 6 months.existing models in the next 6 months.

3737

Next stepsNext steps

Off line meetings to coordinate HD Off line meetings to coordinate HD testing activities.testing activities.Date for next meeting?Date for next meeting?

Action items captured so far:Action items captured so far:

3838

Calibration of model inputsCalibration of model inputs

Peak video level +/- 10%Peak video level +/- 10% Block level +/- 10%Block level +/- 10% Horizontal shift +/- 1 pixelHorizontal shift +/- 1 pixel Vertical shift +/- 1 lineVertical shift +/- 1 line Horizontal cropping < 30 pixelsHorizontal cropping < 30 pixels Vertical cropping < 20 linesVertical cropping < 20 lines No horizontal or vertical re-scalingNo horizontal or vertical re-scaling Temporal alignment +/- 1 frame excluding Temporal alignment +/- 1 frame excluding

dropped/repeated framesdropped/repeated frames No visible chroma differential timingNo visible chroma differential timing No visible picture jitterNo visible picture jitter

3939

Model evaluation metricsModel evaluation metrics

Attributes:Attributes:Prediction accuracyPrediction accuracyPrediction monotonicityPrediction monotonicityPrediction consistencyPrediction consistency

Metrics:Metrics:Root mean square errorRoot mean square errorPearson linear correlationPearson linear correlationOutlier ratioOutlier ratio

4040

Results analysisResults analysis

ILG is responsibleILG is responsible

1 The VQEG HDTV Test Plan A presentation by Vivaik Balasubrawmanian (Intel) Stephen Wolf (NTIA/ITS)

Documents

Transcript of 1 The VQEG HDTV Test Plan A presentation by Vivaik Balasubrawmanian (Intel) Stephen Wolf (NTIA/ITS)