Algebraic Structures for Distributed Source Codingdineshk/thesis/ITA_09_thesis_overview.pdf ·...

Algebraic Structures for Distributed Source Coding

Dinesh Krithivasan and S. Sandeep Pradhan

University of Michigan

ITA 2009

D. Krithivasan & S.S. Pradhan (U of M) Algebraic Structures for DSC ITA 2009 1 / 21

Introduction

Random coding - common in information theory

Introduction

Structured codes - explored mainly for complexity reasons

Introduction

Structured codes have performance unattainable by random codes

Introduction

Can achieve expurgated bound on the BSC

Introduction

Some works on first order performance gains offered by linear codes

Introduction

This thesis: Unified framework for using structured codes

Introduction

Attains/Exceeds known performance limits for many problems

Introduction

Attains/Exceeds known performance limits for many problems

Suggests practical code constructions

Thesis Overview

Very general framework - includes many famous problems

Thesis Overview

Describe avenues for rate gains

Thesis Overview

Role of structured codes in obtaining these rate gains

Thesis Overview

Gaussian sources - Lattice codes

Thesis Overview

Discrete sources - Group codes

Thesis Overview

Discrete sources - Group codes

Applications to sensor networks, video coding etc.

A Distributed Source Coding Problem

f1(·)

f2(·)

Encoder 1

Encoder 2

Decoder

Ed(X, Y, Z) ≤ D

Set of encoders observe different components of a vector source

Central decoder receives quantized observations from the encoders

Decoder interested in minimizing a joint distortion criterion

Joint Distortion Criterion

Distortion criterion depends on all the sources and decoder

reconstructions

Special cases of this framework

Slepian-Wolf problem

Wyner-Ziv problem

Ahlswede-Korner-Wyner problem

Berger-Yeung problem

Berger-Tung problem

Korner-Marton binary XOR problem

reconstructions

Special cases of this framework

Slepian-Wolf problem

Wyner-Ziv problem

Ahlswede-Korner-Wyner problem

Berger-Yeung problem

Berger-Tung problem

Korner-Marton binary XOR problem

Best known rate region - Berger-Tung based

Motivation for our coding scheme

Suppose decoder receives auxiliary random variables U ,V

Encoders don’t communicate with each other: V −Y −X −U

Decoder reconstructs G(U ,V )

G(U ,V ) , arg minG :Z=G(U ,V )d (X ,Y , Z )

Decoder not interested in (U ,V ). Only in G(U ,V )

Encode such that decoder can only reconstruct what it needs

An Illustrative Example

X ,Y - 3 bit correlated binary sources, dH (X ,Y )≤ 1

Lossless reconstruction of Z = X ⊕2 Y ∈ {000,001,010,100}

Berger-Tung based coding scheme:

Reconstruct sources X ,Y . Compute Z = X ⊕2 Y

Sum rate: H(X ,Y ) = 5 bits

Can we do better?

An Illustrative Example contd.

A linear coding scheme:

Z1 ⊕ Z2

Z1 ⊕ Z3

X1X2X3

Y1Y2Y3

X1 ⊕ X2

X1 ⊕ X3

Y1 ⊕ Y2

Y1 ⊕ Y3

Z1 ⊕ Z2

Z1 ⊕ Z3

X1X2X3

Y1Y2Y3

X1 ⊕ X2

X1 ⊕ X3

Y1 ⊕ Y2

Y1 ⊕ Y3

Sum rate: 2+2 = 4 bits = 2H (Z )

Z1 ⊕ Z2

Z1 ⊕ Z3

X1X2X3

Y1Y2Y3

X1 ⊕ X2

X1 ⊕ X3

Y1 ⊕ Y2

Y1 ⊕ Y3

Significant features:

Identical binning at both encoders.

Z1 ⊕ Z2

Z1 ⊕ Z3

X1X2X3

Y1Y2Y3

X1 ⊕ X2

X1 ⊕ X3

Y1 ⊕ Y2

Y1 ⊕ Y3

Identical binning at both encoders. Binning performed by Linear codes

Z1 ⊕ Z2

Z1 ⊕ Z3

X1X2X3

Y1Y2Y3

X1 ⊕ X2

X1 ⊕ X3

Y1 ⊕ Y2

Y1 ⊕ Y3

Identical binning at both encoders. Binning performed by Linear codes

Lossless reconstruction. Entire space is binned

Jointly Gaussian sources and linear function reconstruction

Source (X1, X2) is bivariate Gaussian

X1, X2 ∼N (0,1), E(X1 X2)= ρ > 0

Decoder interested in Z = X1 −c X2 with mean square distortion, c > 0

X1, X2 ∼N (0,1), E(X1 X2)= ρ > 0

Decoder interested in Z = X1 −c X2 with mean square distortion, c > 0

Berger-Tung based coding scheme

Encoders: Quantize X1 to W1, X2 to W2. Transmit W1,W2

Decoder: Reconstruct Z = E(Z |W1,W2)

Optimal for c < 0 with Gaussian test channels

Nested Lattice codes

Lattice codes: Equivalent to linear codes for continuous sources

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−1.5

−0.5

Lattice code: points in Rn closed under addition

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−1.5

−0.5

Need to do quantization first - nested lattice codes

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−1.5

−0.5

Quantizer - nearest neighbour

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−1.5

−0.5

Binning - coset leader of the

quantized fine lattice point in

the coarse lattice

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−1.5

−0.5

the coarse lattice

Fine code - Quantizes the source

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−1.5

−0.5

the coarse lattice

Fine code - Quantizes the source

Coarse code - bins fine code−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−1.5

−0.5

Rate region using nested lattice codes

Nested lattice codes Λ2 ⊂Λ11,Λ2 ⊂Λ12

Λ11,Λ12 - good lattice source codes

Λ2 - good lattice channel code

Achievable tuples satisfy 2−2R1 +2−2R2 ≤

For certain sources and distortions: better than Berger-Tung based

scheme

For certain sources and distortions: better than Berger-Tung based

scheme

Gains based on alignment of [1, −c] with eigenvectors of source

covariance matrixD. Krithivasan & S.S. Pradhan (U of M) Algebraic Structures for DSC ITA 2009 11 / 21

Lattice coding vs Berger-Tung based coding

0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10

Distortion D

Comparison between the two coding schemes

Berger−Tung Sum RateLattice Sum Rate

ρ = 0.95c = 1

Discrete sources and arbitrary distortions

Arbitrary discrete sources and arbitrary distortions

Fix test channels: PX Y UV = PX Y PU |X PV |Y

Decoder interested only in G(U ,V )

Function G(U ,V ) equivalent to addition in some abelian group G

Abelian groups decomposable into primary cyclic groups Zpr

g ∈G ⇔ g = (g1, . . . , gk ), gi ∈Zpiei

Suffices to build codes over Zpr

g ∈G ⇔ g = (g1, . . . , gk ), gi ∈Zpiei

Suffices to build codes over Zpr - nested group codes

g ∈G ⇔ g = (g1, . . . , gk ), gi ∈Zpiei

Suffices to build codes over Zpr - nested group codes

Extension to arbitrary groups through sequential coding

Nested Group codes

Group code over Znpr : C <Z

C = ker(φ) for some homomorphism φ : Znpr →Z

(C1,C2) nested if C2 ⊂C1

Nested Group codes

We need:

C1 <Znpr : “good” source code

C2 <Znpr : “good” channel code

Nested Group codes

We need:

Can find un ∈C1 jointly typical with source xn

Nested Group codes

We need:

Can find un ∈C1 jointly typical with source xn

Can distinguish between typical channel noise sequences

Group source codes: Example

Consider Z4 = {0,1,2,3}

One non-trivial subgroup 2Z4 = {0,2}

Consider Z4 = {0,1,2,3}

Good group source code for (X ,U ,PXU )

Consider Z4 = {0,1,2,3}

Assume U =Z4

Consider Z4 = {0,1,2,3}

Assume U =Z4

Exists for large n if 1n

log |C1| ≥ log 4−min{H (U |X ),2|H (U |X )−1|+}

Consider Z4 = {0,1,2,3}

Assume U =Z4

Optimal random code’s size: H (U )−H (U |X ) = I (X ;U )

Consider Z4 = {0,1,2,3}

Assume U =Z4

Optimal random code’s size: H (U )−H (U |X ) = I (X ;U )

Penalty for imposing group structure

Group Channel codes: Example

Consider Z4 = {0,1,2,3}

Good group channel code for (Z ,S ,PZ S )

Consider Z4 = {0,1,2,3}

Assume Z =Z4

Consider Z4 = {0,1,2,3}

Assume Z =Z4

Exists for large n if1n log |C2| ≤ log 4−max{H (Z |S),2(H (Z |S)−H ([Z ]1|S))}

Consider Z4 = {0,1,2,3}

Assume Z =Z4

Optimal random code’s size: log 4−H (Z |S)

Consider Z4 = {0,1,2,3}

Assume Z =Z4

Optimal random code’s size: log 4−H (Z |S)

Penalty for presence of non-trivial subgroups

Coding Strategy

Nested group codes C2 <C11,C12

1n log |C11| ≥ log 4−

min{H (U |X ),2|H (U |X )−1|+}

log |C12| ≥ log 4−

min{H (V |Y ),2|H (V |Y )−1|+}

1n log |C2| ≤ log 4−

max{H (Z ),2(H (Z )−H ([Z ]1))}

Achievable Rates

The set of tuples (R1,R2,D) that satisfy

R1 ≥ max{H (Z ),2(H (Z )−H ([Z ]1))}−min{H (U |X ),2|H (U |X )−1|+}

R2 ≥ max{H (Z ),2(H (Z )−H ([Z ]1))}−min{H (V |Y ),2|H (V |Y )−1|+}

D ≥ Ed (X ,Y ,G(U ,V ))

are achievable.

Achievable Rates

The set of tuples (R1,R2,D) that satisfy

R1 ≥ max{H (Z ),2(H (Z )−H ([Z ]1))}−min{H (U |X ),2|H (U |X )−1|+}

R2 ≥ max{H (Z ),2(H (Z )−H ([Z ]1))}−min{H (V |Y ),2|H (V |Y )−1|+}

D ≥ Ed (X ,Y ,G(U ,V ))

are achievable.

More general rate region possible

Group codes vs Berger-Tung based coding

0.02 0.04 0.06 0.08 0.1 0.12 0.14

Distortion D

Comparison of the two lower convex envelopes

2 Berger−Tung based coding schemeGroup code based coding scheme

Reconstruction of XOR with Hamming distortion

Group codes vs Berger-Tung based coding

0.02 0.04 0.06 0.08 0.1 0.12 0.14

Distortion D

Comparison of the two lower convex envelopes

2 Berger−Tung based coding schemeGroup code based coding scheme

Reconstruction of XOR with Hamming distortion

Implies Berger-Tung bound not tight for more than 2 users

Other results

Existence of “good” nested lattices - arbitrary finite levels of nesting

Other results

Lossless and lossy compression using abelian group codes - achievable

Other results

Nested linear codes achieve Shannon rate-distortion bound for

arbitrary discrete sources and additive distortions

Other results

Nested linear codes recover known rate regions of

Other results

Berger-Tung problem

Other results

Berger-Tung problem

Wyner-Ziv problem, Wyner-Ahlswede-Korner problem

Other results

Berger-Tung problem

Yeung-Berger problem

Other results

Berger-Tung problem

Yeung-Berger problem

Slepian-Wolf problem, Korner-Marton problem

Future research directions

Extension of lattice coding schemes to arbitrary continuous sources

Construction of practical group codes using LDPC/LDGM codes with

iterative decoding

Use of structured codes in multi-terminal channel coding problems

iterative decoding

Connections between information theory (typicality) and abstract

algebra (subgroups)

iterative decoding

Connections between information theory (typicality) and abstract

algebra (subgroups)

Codes over non-abelian groups

Algebraic Structures for Distributed Source Codingdineshk/thesis/ITA_09_thesis_overview.pdf ·...

Documents

Transcript of Algebraic Structures for Distributed Source Codingdineshk/thesis/ITA_09_thesis_overview.pdf ·...

Algebraic Topology and Distributed Computingcs.brown.edu/courses/cs2951s/Ch01.pdfIntroduction . Distributed Computing though Combinatorial Topology 1 . Companion slides for . Distributed

C3.4 ALGEBRAIC GEOMETRY - University of Oxfordpeople.maths.ox.ac.uk/ritter/algebraic-geometry/... · Shafarevich, Basic Algebraic Geometry. Harris, Algebraic Geometry, A First Course.

CURRICULUM VITAE - Indian Institute of Technology Madraskamala/cv.pdf · CURRICULUM VITAE Name: Dr. Kamala Krithivasan ... Studies in Distributed Probabilistic Automaton and Simple

Algebraic K-theory, algebraic cycles and arithmetic geometry

Algebraic Homogeneous Spaces of Algebraic Groups

INTRODUCTION TO ALGEBRAIC TOPOLOGYmsc.tsinghua.edu.cn/sli/Algebraic-Topology.pdf · This is course note for Algebraic Topology in Spring 2018 at Tsinghua university. ... Algebraic

Algebraic Topology and Distributed Computingcs.brown.edu/courses/cs2951s/Ch09.pdfSimplicial Complex . Distributed Computing Through 5 Combinatorial Topology . A simplicial complex

Algebraic Topology and Distributed Computingcs.brown.edu/courses/csci2951-s/Ch02.pdfDistributed Computing through Combinatorial Topology 22 . ... Distributed Computing through Combinatorial

BEAMS_Unit 3 Algebraic Expressions and Algebraic Formulae

N-ALGEBRAIC STRUCTURES AND S-N-ALGEBRAIC STRUCTURES

Algebraic Fractions Algebraic Fractions Simplify Rational Expressions Multiply Rational Algebraic Expressions Add Rational Algebraic Expressions Solve.

Lattices for Distributed Source Coding - Reconstruction of a Linear function of Jointly Gaussian Sources -D. Krithivasan and S. Sandeep Pradhan - University.

Stability of algebraic varieties and algebraic geometryStability of algebraic varieties and algebraic geometry ... complex manifold, and particularly on a complex projective manifold.

An Algebraic Oak - Algebraic Poem

Algebraic Transformation Groups and Algebraic Varieties: Proceedings of the conference Interesting Algebraic Varieties Arising in Algebraic Transformation Group Theory held at the

Party Animal: Rolling in the Deep Arjun Sinha, Rajeev Krithivasan, Will Sergeant

Classification of Algebraic Varieties · Classification of Algebraic Varieties Algebraic Geometry Conference on Classification of Algebraic Varieties May 22-30, 1992 ... Departamento

Dinesh Krithivasan and S. Sandeep Pradhan arXiv:0808.2659v1 …web.eecs.umich.edu/~pradhanv/paper/ittrans08_1.pdf · 2008. 8. 21. · Dinesh Krithivasan and S. Sandeep Pradhan∗,

Algebraic Quantum Mechanics, Algebraic Spinors and Hilbert Space.

GENERAL ALGEBRAIC CREEP ANALYSIS OF ......27-28 August 2015, Singapore _____ † ECSD Srl, Milan, Italy 1 GENERAL ALGEBRAIC CREEP ANALYSIS OF CONCRETE MEMBERS WITH POINT OR DISTRIBUTED