Spot & DataContainer

Daniel (me)Henryk & Thomas & Siavash

Agenda• The way we work with data and metadata NOW and how we

want to improve it• Make codes shorter and easier to read

– Basic Clean-up using DataContainer– Two new operator constructor syntaxes

• Avoid/detect programming mistake -> more robust code• Some Things to Keep in Mind

– Children operator dimension– Operation History– Dynamic Operator

What is Metadata?

Image Source: USGS

--------------> Data

1. Unit on the axis2. How far apart are the sample3. The label on the axis4. <etc>

--------------> Metadata

How We Treat Metadata

We have to change the metadata ourselves.

Data and metadata are separated.

What if we load different data and the unit is different

Why keeping track of metadata ourselves is error prone?

How We Treat MetadataWhy keeping track of metadata ourselves is error prone?

How do we know what actually happened in the function?

How We Treat MetadataWhy keeping track of metadata ourselves is error prone?

Data is structured differently than we thought:What if the data we get actually has space-domain on 1st dimension and time domain on 2nd dimension?

How We Treat MetadataWhy keeping track of metadata ourselves is error prone?

Keep track of metadata is a total pain for higher dimension data. Remember there are other metadata too, not just unit.

What if We Want to Keep Track of Metadata Automatically

Operator(s) Data

MetaData

Operator (not us) should modify BOTH the data and metadata at the same time.

Originally

We

What if We Want to Keep Track of Metadata Automatically

Operator(s)

Data

MetaData

DataContainer

What we want

So what does a DataContainer look like?

Store Data and MetaData in one Object

Same Vector Operation as MATLAB Array

x = randn(3,5);

>> x1 = vec(x);>> x2 = x’;>> x3 = norm(x);>> x4 = 2*x + 3*x – 4*x>> % and more

MATLAB array

C = iCon(randn(3,5));

>> C1 = vec(C);>> C2 = C’;>> C3 = norm(C);>> C4 = 2*C + 3*C – 4*C>> % and more

DataContainer

DataContainer : Works like MATLAB array

Like RSF at File Level

Data(variable: m)

Meta Data(variable: model)

Saving to File

Spot Operator Apply Changes to DataContainer’s Metadata

What DataContainer Enables to Do

What if we load a different set of data and the unit is different?

What DataContainer Enables to Do

Keep track of metadata is a total pain

What DataContainer Enables to DoOriginally,We don’t know if the data is structured the way we expected.We don’t know if the function is doing things properly.

We can put something to check the metadata.Here we check unit as an example.

Agenda• The way we work with data and metadata NOW and how we

want to improve it• Make codes shorter and easier to read

– Basic Clean-up using DataContainer– Two new operator constructor syntaxes

• Avoid/detect programming mistake -> more robust code• Some Things to Keep in Mind

– Children operator dimension– Operation History– Dynamic Operator

Example code – L1 RecoveryHow to use DataContainer in Your Code

Example code

The data

metadata

Let’s Clean up the Code!

Clean-up Stage 0: Put data and metadata into DataContainer

Clean-up Stage 0: Put data and metadata into DataContainer

Clean-up Stage 0: Put data and metadata into DataContainer

New Operator Constructor Syntaxes

A = opDFT(model.n(1))

A = opDFT(size(C,1))Version 0

A = opDFT({C,1})Version 1

A = opDFTVersion 2

Clean-up Stage 1: Using new Syntax to Clean Up and Give more info to the spot operator

Clean-up Stage 1: Using new Syntax to Clean Up and Give more info to the spot operator

New Operator Constructor Syntaxes

A = opDFT(model.n(1))

A = opDFT(size(C,1))Version 0

A = opDFT({C,1})Version 1

A = opDFTVersion 2

Clean-up Stage 2: Using Dynamic Operator

Dynamic Operator

Simple Dynamic Operator Example : opDFT

Clean-up Stage 2: Using Dynamic Operator

Simple Dynamic Operator Example : opDFT

opDFT( )

DataContainer

Dimension: 3 X 1opDFT( ) *3

Clean-up Stage 2: Using Dynamic Operator

Clean-up Stage 2: Using Dynamic Operator

Agenda• The way we work with data and metadata NOW and how we

want to improve it• Make codes shorter and easier to read

– Basic Clean-up using DataContainer– Two new operator constructor syntaxes

• Avoid/detect programming mistake -> more robust code• Some Things to Keep in Mind

– Children operator dimension– Operation History– Dynamic Operator

Example - Catching ErrorHaving spot operate on MATLAB array

Correct

Example - Catching ErrorHaving spot operate on MATLAB array

Wrong!!

Example - Catching ErrorHaving spot operate on DataContainer

Agenda• The way we work with data and metadata NOW and how we

want to improve it• Make codes shorter and easier to read

– Basic Clean-up using DataContainer– Two new operator constructor syntaxes

• Avoid/detect programming mistake -> more robust code• Some Things to Keep in Mind

– Children operator dimension– Operation History– Dynamic Operator

Keep in Mind - Operator

opKron( , ) Size : dim1 X dim2 X dim3

(DataContainer)

For operating onDataContainer:

opKron(m x n) (n x 1)XFor operating on

MATLAB Array:

(MATLAB Array)

Keep in Mind - Operator

opTaper = opKron(opDirac(size(C,3)),opDirac(size(C,2)),opLinMute(size(C,1),0.1*size(C,1),0.2*size(C,1)));

opTaper = opKron(opDirac({C,2:3}),opLinMute({C,1},0.1,0.2)); ({<Container>, <selected dimensions>})

To Properly Define the Operator:

opTaper = opKron(opDirac,opDirac, opLinMute({C,1},0.1,0.2)); Dynamic operator

Keep in Mind - DataContainer

In our example, we must initialize the container as 3-dimensional THEN vectorize it.

Keep in Mind – DataContainerKeeps Track of Metadata History

res2 should have the same metadataas container C.

History Stack

A.ID Mode = 1 Delta, Origin, Unit,…, etc

A.ID Mode = 1 Delta, Origin, Unit,…, etc

res2 = A’ * A * C

How operating on DataContainer helps recover information that we usually can’t recover

A.ID Mode = 2

res1 = A * C

Keep in Mind - DataContainer

• When doing addition or subtraction between 2 DataContainers, we keep the history of the one on the left.

Container 1

C1_history

Container 2

C2_history+ =

Result Container

C1_history

Keep in Mind – Dynamic Operator

• A dynamic operator is activated in the scope it is used.

Keep in Mind – Dynamic Operator

• A dynamic operator is activated in the scope it is used.

Advantage?Disadvantage?

Keep in Mind – Dynamic Operator

• Some dynamic operators cannot activate in adjoint mode.– They require that they had been used previously

in the forward mode.

Some dynamic operators require that they had been used previously in the forward mode.

A = opCurvelet(23,29)A = opCurvelet(?)

A’ * A * C Example

History Stack

A.ID Mode = 1 Delta, Origin, Unit, info for A,…, etc

A.ID Mode = 1 Delta, Origin, Unit, info for A,…, etc

A.ID Mode = 2

C = iCon(randn(23,29))

A * C

A’ * A * C

Keep in Mind – Dynamic Operator

• The operator will have different ID when created at different time.

A.ID Mode = 1 …

Conclusion

Why consider using DataContainer?• Works just like MATLAB array.

– Not too difficult to change– Added feature

• Organize your data and metadata• Can be saved to file – no more editing your script

– Apply changes to your metadata• Shorten your code => easier to read• Dynamic operator

– Automatically initialize operator for you• Use the same script for different size data => easily scale to larger data sets

• Catch error that MATLAB array can’t normally catch

• The new spot operators are backward compatible

That’s it!

Questions