Octane Reference Manualoctane.sourceforge.net/releases/octane-1.01.20.pdf · 2004. 5. 20. · •...

Octane Reference Manualv1.01.20

Generated by Doxygen 1.3.3

Thu May 20 05:47:44 2004

Contents

1 The Open Compression Toolkit for C++ 1

1.1 Contents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Octane Module Index 3

2.1 Octane Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3 Octane Hierarchical Index 5

3.1 Octane Class Hierarchy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4 Octane Compound Index 7

4.1 Octane Compound List. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

5 Octane File Index 9

5.1 Octane File List. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

6 Octane Page Index 11

6.1 Octane Related Pages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

7 Octane Module Documentation 13

7.1 bitio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13

7.2 Coders. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

7.3 Compressors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

7.4 Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

7.5 Core . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

7.6 Modelers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

7.7 Parsers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20

8 Octane Class Documentation 21

8.1 arraybitreader Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

8.2 arraybitwriter Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

8.3 BitParser Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

ii CONTENTS

8.4 bitreader Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

8.5 bitwriter Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

8.6 CompressorManager Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

8.7 CompressorManagerSingletonInsurer Class Reference. . . . . . . . . . . . . . . . . . . 58

8.8 file bitreader Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

8.9 file bitwriter Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

8.10 HuffmanCoder Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

8.11 HuffNode Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

8.12 HuffNodeLeaf Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

8.13 HuffNodeMiddle Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

8.14 mtfllCompressor Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

8.15 null bitreader Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

8.16 null bitwriter Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

8.17 OctaneClass Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

8.18 OctaneCoder Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

8.19 OctaneCompressor Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

8.20 OctaneCompressorStatistical Class Reference. . . . . . . . . . . . . . . . . . . . . . . .108

8.21 OctaneModeler Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .117

8.22 OctaneModelerWeightVectored Class Reference. . . . . . . . . . . . . . . . . . . . . .122

8.23 OctaneParser Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .126

8.24 OctaneScopeTimer Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . .131

8.25 OctaneTester Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133

8.26 OctaneTimer Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .155

8.27 PQWeightGreater< T > Class Template Reference. . . . . . . . . . . . . . . . . . . . .157

8.28 SampleCoder Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .158

8.29 SampleCompressor Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . .162

8.30 SampleModeler Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .169

8.31 SampleParser Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .172

8.32 SampleStatCompressor Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . .178

8.33 streambitreader Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .180

8.34 streambitwriter Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182

8.35 stringbitreader Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .184

8.36 stringbitwriter Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .187

8.37 stringparser Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .189

8.38 SubStrHuffCompressor Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . .191

8.39 SubstringParser Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .193

Generated on Thu May 20 05:48:10 2004 for Octane by Doxygen

CONTENTS iii

8.40 SubstringSymbol Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .203

8.41 SymbolWeightVector Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . .205

8.42 TSubstringSymbolStringGreater< T > Class Template Reference. . . . . . . . . . . . . 210

8.43 TSubstringSymbolWeightGreater< T > Class Template Reference. . . . . . . . . . . . . 211

8.44 vectorbitreader Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .212

8.45 vectorbitwriter Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .215

8.46 ZeroOrderModeler Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . .217

8.47 zleCompressor Class Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .219

9 Octane File Documentation 223

9.1 bitio/bitio.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .223

9.2 bitio/bitreader.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .224

9.3 bitio/bitreaders.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .225

9.4 bitio/bitwriter.cpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .226

9.5 bitio/bitwriter.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .227

9.6 bitio/bitwriters.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .228

9.7 bitio/file bitreader.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . .229

9.8 bitio/file bitwriter.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . .230

9.9 coders/coder.cpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .231

9.10 coders/coder.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .232

9.11 coders/coderhuffman/huffmancoder.cpp File Reference. . . . . . . . . . . . . . . . . . . 233

9.12 coders/coderhuffman/huffmancoder.hpp File Reference. . . . . . . . . . . . . . . . . . 234

9.13 coders/coderhuffman/huffmannodes.cpp File Reference. . . . . . . . . . . . . . . . . . 235

9.14 coders/coderhuffman/huffmannodes.hpp File Reference. . . . . . . . . . . . . . . . . . 236

9.15 coders/codersample/samplecoder.cpp File Reference. . . . . . . . . . . . . . . . . . . . 238

9.16 coders/codersample/samplecoder.hpp File Reference. . . . . . . . . . . . . . . . . . . . 239

9.17 compressors/compressor.cpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . .240

9.18 compressors/compressor.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . .241

9.19 compressors/compressormtfll/mtfll.cpp File Reference. . . . . . . . . . . . . . . . . . . 242

9.20 compressors/compressormtfll/mtfll.hpp File Reference. . . . . . . . . . . . . . . . . . . 243

9.21 compressors/compressorsample/samplecompressor.cpp File Reference. . . . . . . . . . 244

9.22 compressors/compressorsample/samplecompressor.hpp File Reference. . . . . . . . . . 245

9.23 compressors/compressorsample/samplestatcompressor.cpp File Reference. . . . . . . . . 246

9.24 compressors/compressorsample/samplestatcompressor.hpp File Reference. . . . . . . . 247

9.25 compressors/compressorstatistical.cpp File Reference. . . . . . . . . . . . . . . . . . . 248

9.26 compressors/compressorstatistical.hpp File Reference. . . . . . . . . . . . . . . . . . . 249

9.27 compressors/compressorsubstrhuff/substrhuff.cpp File Reference. . . . . . . . . . . . . 250


iv CONTENTS

9.28 compressors/compressorsubstrhuff/substrhuff.hpp File Reference. . . . . . . . . . . . . 251

9.29 compressors/compressorzle/zle.cpp File Reference. . . . . . . . . . . . . . . . . . . . .252

9.30 compressors/compressorzle/zle.hpp File Reference. . . . . . . . . . . . . . . . . . . . .253

9.31 core/compressormanager.cpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . .254

9.32 core/compressormanager.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . .255

9.33 core/octane.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .256

9.34 core/octaneclass.cpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . .257

9.35 core/octaneclass.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . .258

9.36 modelers/modeler.cpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . .259

9.37 modelers/modeler.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . .260

9.38 modelers/modelersample/samplemodeler.cpp File Reference. . . . . . . . . . . . . . . . 261

9.39 modelers/modelersample/samplemodeler.hpp File Reference. . . . . . . . . . . . . . . . 262

9.40 modelers/modelerweightvectored.cpp File Reference. . . . . . . . . . . . . . . . . . . . 263

9.41 modelers/modelerweightvectored.hpp File Reference. . . . . . . . . . . . . . . . . . . . 264

9.42 modelers/modelerzeroorder/zeroordermodeler.cpp File Reference. . . . . . . . . . . . . 265

9.43 modelers/modelerzeroorder/zeroordermodeler.hpp File Reference. . . . . . . . . . . . 266

9.44 modelers/symbolweightvector.cpp File Reference. . . . . . . . . . . . . . . . . . . . . .267

9.45 modelers/symbolweightvector.hpp File Reference. . . . . . . . . . . . . . . . . . . . . .268

9.46 octanetester/octanetester.cpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . .269

9.47 octanetester/octanetester.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . .270

9.48 octanetester/standalonetestermain.cpp File Reference. . . . . . . . . . . . . . . . . . . . 271

9.49 parsers/parser.cpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .272

9.50 parsers/parser.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .273

9.51 parsers/parserbitlevel/bitparser.cpp File Reference. . . . . . . . . . . . . . . . . . . . .274

9.52 parsers/parserbitlevel/bitparser.hpp File Reference. . . . . . . . . . . . . . . . . . . . .275

9.53 parsers/parsersample/sampleparser.cpp File Reference. . . . . . . . . . . . . . . . . . . 276

9.54 parsers/parsersample/sampleparser.hpp File Reference. . . . . . . . . . . . . . . . . . . 277

9.55 parsers/parsersubstring/substringparser.cpp File Reference. . . . . . . . . . . . . . . . . 278

9.56 parsers/parsersubstring/substringparser.hpp File Reference. . . . . . . . . . . . . . . . . 279

9.57 utilityclasses/stringparser/stringparser.hpp File Reference. . . . . . . . . . . . . . . . . . 281

9.58 utilityclasses/timer/scopetimer.hpp File Reference. . . . . . . . . . . . . . . . . . . . .282

9.59 utilityclasses/timer/timer.hpp File Reference. . . . . . . . . . . . . . . . . . . . . . . . .283

10 Octane Page Documentation 285

10.1 Introduction to Octane. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .285

10.2 A Simple Walkthrough with the Octane Tester Console. . . . . . . . . . . . . . . . . . . 288

10.3 The Octane Compressor API - How to Write Your Own Octane Compressor Class. . . . . 293


CONTENTS v

10.4 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .301

10.5 Future Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .302

10.6 Octane License. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .304

10.7 References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .305

10.8 Todo List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .306

10.9 Bug List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .308


Chapter 1

The Open Compression Toolkit for C++

The Open Compression Toolkit is a set of modular C++ classes and utilities for implementing and testingcompression algorithms.

• Simple interface and skeleton code for creating new compression algorithms.

• Complete testing framework for validating and comparing new algorithms.

• Support for algorithms that use external dictionaries/headers.

• Utility classes and sample code for bitio, frequency counting, etc.

1.1 Contents

• Introduction- a short introduction to the Octane toolkit.

• A Simple Walkthrough- the Octane Tester application.

• The Compressor API- how to write your own Octane compressor.

• History - changes between releases.

• Future Work- our ideas for the future development.

• Octane License- the license under which Octane is available.

2 The Open Compression Toolkit for C++


Chapter 2

Octane Module Index

2.1 Octane Modules

Here is a list of all modules:

bitio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13Coders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15Compressors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17Core . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18Modelers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19Parsers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20

4 Octane Module Index


Chapter 3

Octane Hierarchical Index

3.1 Octane Class Hierarchy

This inheritance list is sorted roughly, but not completely, alphabetically:

bitreader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32arraybitreader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21file bitreader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .59null bitreader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79streambitreader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .180string bitreader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .184vectorbitreader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .212

bitwriter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44arraybitwriter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24file bitwriter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62null bitwriter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81streambitwriter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182string bitwriter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .187vectorbitwriter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .215

CompressorManagerSingletonInsurer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58HuffNode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70

HuffNode Leaf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72HuffNode Middle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

OctaneClass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82CompressorManager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52OctaneCoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89

HuffmanCoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64SampleCoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .158

OctaneCompressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94mtfllCompressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76OctaneCompressorStatistical . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .108

SampleStatCompressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .178SubStrHuffCompressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .191

SampleCompressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .162zleCompressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .219

OctaneModeler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .117OctaneModelerWeightVectored . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .122

6 Octane Hierarchical Index

SampleModeler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .169ZeroOrderModeler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .217

OctaneParser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .126BitParser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26SampleParser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .172SubstringParser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .193

OctaneTester . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133SymbolWeightVector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .205

OctaneScopeTimer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .131OctaneTimer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .155PQWeightGreater< T > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .157stringparser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .189SubstringSymbol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .203TSubstringSymbolStringGreater< T > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .210TSubstringSymbolWeightGreater< T > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .211


Chapter 4

Octane Compound Index

4.1 Octane Compound List

Here are the classes, structs, unions and interfaces with brief descriptions:

arraybitreader(Specialization for C pointer to char ) . . . . . . . . . . . . . . . . . . . . . . .21arraybitwriter (Specialization for C pointer to char ) . . . . . . . . . . . . . . . . . . . . . . .24BitParser(Sample parser class is just character based, uses one symbol for each ascii character

plus one for end-of-stream ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26bitreader(Base class for bit input wrappers ) . . . . . . . . . . . . . . . . . . . . . . . . . . . .32bitwriter (Base class for bit output wrappers ) . . . . . . . . . . . . . . . . . . . . . . . . . . .44CompressorManager(The CompressorManager classes holds a collection of Compressor derived

classes and is used for runtime selection of which algorithm to test ) . . . . . . . . . .52CompressorManagerSingletonInsurer(This is a sneaky little class which allows us to create

self-registering compressors ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58file bitreader(Specialization for files ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .59file bitwriter (Specialization for files ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62HuffmanCoder(During compression, the job of the Coder class is use probability information

from the modeler to figure out the optimal bitcodes to use when writing symbols intothe output stream ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64

HuffNode(The base HuffNode class is used to represent both leaf and internal HuffNodes ) . . .70HuffNode Leaf (The leaves of the huffman tree contains actual symbol information ) . . . . . .72HuffNode Middle (A derived huffman node for a node in the middle of the tree ) . . . . . . . .74mtfllCompressor(Experimental move to front encoding using a linked list instead of tracing the

array ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76null bitreader(Specialization without data ) . . . . . . . . . . . . . . . . . . . . . . . . . . . .79null bitwriter (Specialization without data ) . . . . . . . . . . . . . . . . . . . . . . . . . . . .81OctaneClass (Base class from which most octane classes derive (including

parser,coder,modeler,etc) ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82OctaneCoder(During compression, the job of the Coder class is use probability information

from the modeler to figure out the optimal bitcodes to use when writing symbols intothe output stream ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89

OctaneCompressor(The Base Compressor class implements our generic interface to all com-pressors, and includes wrappers to measure compression/decompression times anddiskspace ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94

OctaneCompressorStatistical(The Statistical compressor class is a base class from which to de-rive specific statistical compressors that make use of the parser/modeler/coder frame-work ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .108

8 Octane Compound Index

OctaneModeler(The Modeler class is for statistical compression methods ) . . . . . . . . . . .117OctaneModelerWeightVectored(This is a derived modeler class which has some default func-

tions for modelers which work with a vector of weights as probabilities (very common)) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .122

OctaneParser(The Base Parser class is responsible for dividing the input stream into a series ofnumerical symbols, and for converting symbol numbers to their symbol texts duringdecompression ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .126

OctaneScopeTimer(Simple timer, measuring time from construction to destruction ) . . . . . .131OctaneTester(Functions for testing compressors and providing a commandline and interactive

interface to octane functions ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133OctaneTimer(Simple class for timing code (low resolution) ) . . . . . . . . . . . . . . . . . . .155PQWeightGreater< T > (We need to make a special Greater<> classes used during sorting,

since our collection stl elements are built from pointers, and the default sort will bedone on pointer addresses instead of weights if we don’t ) . . . . . . . . . . . . . . . .157

SampleCoder(The Sample Coder is useful only as an example of how coders work ) . . . . . .158SampleCompressor(This Sample Statistical Compressor is a minimal example of how to

coordinate the 3 components of the statistical compressor framework, a sampleparser,modeler,and coder ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .162

SampleModeler(This is a sample modeler which simply ignores all ’training’ information andassigns equal probability to each symbol in the parser (sometimes called a -1 ordermodel) ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .169

SampleParser(Sample parser class is just character based, uses one symbol for each ascii char-acter plus one for end-of-stream ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .172

SampleStatCompressor(This Sample Statistical Compressor is a minimal example of howto coordinate the 3 components of the statistical compressor framework, a sampleparser,modeler,and coder ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .178

streambitreader(Specialization for streams ) . . . . . . . . . . . . . . . . . . . . . . . . . . .180streambitwriter (Specialization for streams ) . . . . . . . . . . . . . . . . . . . . . . . . . . .182string bitreader(Specialization for string ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .184string bitwriter (Specialization for string ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .187stringparser(Simple string parser ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .189SubStrHuffCompressor(The SubStrHuff Compressor is a basic Statistical compressor, that in-

stantiates a Zero-order modeler, with a huffman coder, and a SubString parser ) . . . .191SubstringParser(Flexible parser, which uses arbitrarily long substrings of text as symbols ) . . .193SubstringSymbol(The SubString Symbol class represents the primitive symbols used by the

SubString parser ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .203SymbolWeightVector(Helper class which holds a vector of probabilities, typically associated

with a modeler ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .205TSubstringSymbolStringGreater< T > (Greater operator for comparing strings of symbol point-

ers alphabetically ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .210TSubstringSymbolWeightGreater< T > (Greater operator for comparing weights of symbol

pointers ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .211vectorbitreader(Specialization for vector<char> ) . . . . . . . . . . . . . . . . . . . . . . . .212vectorbitwriter (Specialization for vector<char> ) . . . . . . . . . . . . . . . . . . . . . . . .215ZeroOrderModeler(A simple zero-order modeler, that builds its probability model by counting

frequencies in a training file ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .217zleCompressor(This is a zero-length-encoder, it performs run-length-encoding for 0 bytes only )219


Chapter 5

Octane File Index

5.1 Octane File List

Here is a list of all documented files with brief descriptions:

bitio/bitio.hpp(Bit-wise I/O wrapper classes ) . . . . . . . . . . . . . . . . . . . . . . . . . . .223bitio/bitreader.hpp(Base class for bit input wrappers ) . . . . . . . . . . . . . . . . . . . . . . .224bitio/bitreaders.hpp(Bitreader specializations ) . . . . . . . . . . . . . . . . . . . . . . . . . .225bitio/bitwriter.cpp(Base class for bit output wrappers ) . . . . . . . . . . . . . . . . . . . . . .226bitio/bitwriter.hpp(Base class for bit output wrappers ) . . . . . . . . . . . . . . . . . . . . . .227bitio/bitwriters.hpp(Bitwriter specializations ) . . . . . . . . . . . . . . . . . . . . . . . . . . .228bitio/file bitreader.hpp(Bitreader specialization for files ) . . . . . . . . . . . . . . . . . . . . .229bitio/file bitwriter.hpp(Bitwriter specialization for files ) . . . . . . . . . . . . . . . . . . . . .230coders/coder.cpp(Base Coder class ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .231coders/coder.hpp(Base Coder class ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .232coders/coderhuffman/huffmancoder.cpp(Simple Huffman Coder class ) . . . . . . . . . . . . .233coders/coderhuffman/huffmancoder.hpp(Simple Huffman Coder class ) . . . . . . . . . . . . .234coders/coderhuffman/huffmannodes.cpp(Huffman node classes ) . . . . . . . . . . . . . . . .235coders/coderhuffman/huffmannodes.hpp(Huffman node classes ) . . . . . . . . . . . . . . . .236coders/codersample/samplecoder.cpp(Sample Coder useful as a skeleton for writing new coders )238coders/codersample/samplecoder.hpp(Sample Coder useful as a skeleton for writing new coders )239compressors/compressor.cpp(Compression/Decompression classes ) . . . . . . . . . . . . . . .240compressors/compressor.hpp(Compression/Decompression classes ) . . . . . . . . . . . . . . .241compressors/compressorstatistical.cpp(Skeleton for statistical compressors ) . . . . . . . . . .248compressors/compressorstatistical.hpp(Skeleton for statistical compressors ) . . . . . . . . . .249compressors/compressormtfll/mtfll.cpp(Move to front encoding ) . . . . . . . . . . . . . . . .242compressors/compressormtfll/mtfll.hpp(Move-to-front encoding ) . . . . . . . . . . . . . . . .243compressors/compressorsample/samplecompressor.cpp(Sample compressor ) . . . . . . . . . .244compressors/compressorsample/samplecompressor.hpp(Sample compressor ) . . . . . . . . .245compressors/compressorsample/samplestatcompressor.cpp(Sample statistical compressor ) . .246compressors/compressorsample/samplestatcompressor.hpp(Sample statistical compressor ) . .247compressors/compressorsubstrhuff/substrhuff.cpp(Compressor that uses the SubString Parser,

a 0-Order Model and a Huffman Coder ) . . . . . . . . . . . . . . . . . . . . . . . . .250compressors/compressorsubstrhuff/substrhuff.hpp(Compressor that uses the SubString Parser,

a 0-Order Model and a Huffman Coder ) . . . . . . . . . . . . . . . . . . . . . . . . .251compressors/compressorzle/zle.cpp(Zero length encoding ) . . . . . . . . . . . . . . . . . . .252compressors/compressorzle/zle.hpp(Zero length encoding ) . . . . . . . . . . . . . . . . . . .253core/compressormanager.cpp(Compression/Decompression manager classes ) . . . . . . . . . .254

10 Octane File Index

core/compressormanager.hpp(Compression/Decompression manager classes ) . . . . . . . . . .255core/octane.hpp(Version information and main help page comments ) . . . . . . . . . . . . . .256core/octaneclass.cpp(Root class for most derived Octane classes ) . . . . . . . . . . . . . . . .257core/octaneclass.hpp(Root class for most derived Octane classes ) . . . . . . . . . . . . . . . .258modelers/modeler.cpp(Base Modeler class ) . . . . . . . . . . . . . . . . . . . . . . . . . . . .259modelers/modeler.hpp(Base Modeler class ) . . . . . . . . . . . . . . . . . . . . . . . . . . . .260modelers/modelerweightvectored.cpp(Derived modeler which uses a simple vector of probabil-

ities ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .263modelers/modelerweightvectored.hpp(Derived modeler which uses a simple vector of probabil-

ities ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .264modelers/symbolweightvector.cpp(Helper classes for working with probability/weight vectors )267modelers/symbolweightvector.hpp(Helper classes for working with probability/weight vectors )268modelers/modelersample/samplemodeler.cpp(A sample modeler which basically assigns fixed

equal probabilities to every symbol ) . . . . . . . . . . . . . . . . . . . . . . . . . . .261modelers/modelersample/samplemodeler.hpp(A sample modeler which basically assigns fixed

equal probabilities to every symbol ) . . . . . . . . . . . . . . . . . . . . . . . . . . .262modelers/modelerzeroorder/zeroordermodeler.cpp(A simple zero-order modeler, which just

uses summary frequencies of symbols ) . . . . . . . . . . . . . . . . . . . . . . . . .265modelers/modelerzeroorder/zeroordermodeler.hpp(A simple zero-order modeler, which just

uses summary frequencies of symbols ) . . . . . . . . . . . . . . . . . . . . . . . . .266octanetester/octanetester.cpp(Classes for testing octane functions and compressors ) . . . . . . .269octanetester/octanetester.hpp(Classes for testing octane functions and compressors ) . . . . . .270octanetester/standalonetestermain.cpp(Standalone tester file with intmain(), useful for testing

compressors through commandline or interactively ) . . . . . . . . . . . . . . . . . . .271parsers/parser.cpp(The base parser class which defines a generic API for statistical parsers ) . .272parsers/parser.hpp(The base parser class which defines a generic API for statistical parsers ) . .273parsers/parserbitlevel/bitparser.cpp(A bit-level parser that can be told how many bits per sym-

bol, and returns numbers converted bit chunks ) . . . . . . . . . . . . . . . . . . . . .274parsers/parserbitlevel/bitparser.hpp(A bit-level parser that can be told how many bits per sym-

bol, and returns numbers converted bit chunks ) . . . . . . . . . . . . . . . . . . . . .275parsers/parsersample/sampleparser.cpp(A simple sample parser for use in statistical compres-

sion ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .276parsers/parsersample/sampleparser.hpp(A simple sample parser for use in statistical compres-

sion ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .277parsers/parsersubstring/substringparser.cpp(A flexible parser which can creates a symbol dic-

tionary of substrings ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .278parsers/parsersubstring/substringparser.hpp(A flexible parser which can creates a symbol dic-

tionary of substrings ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .279utilityclasses/stringparser/stringparser.hpp(Simple string parser ) . . . . . . . . . . . . . . . . .281utilityclasses/timer/scopetimer.hpp(Simple timer, measuring time from construction to destruc-

tion ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .282utilityclasses/timer/timer.hpp(Simple class for timing code (low resolution) ) . . . . . . . . . .283


Chapter 6

Octane Page Index

6.1 Octane Related Pages

Here is a list of all related documentation pages:

Introduction to Octane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .285A Simple Walkthrough with the Octane Tester Console . . . . . . . . . . . . . . . . . . . . . .288The Octane Compressor API - How to Write Your Own Octane Compressor Class . . . . . . . .293History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .301Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .302Octane License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .304References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .305Todo List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .306Bug List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .308

12 Octane Page Index


Chapter 7

Octane Module Documentation

7.1 bitio

7.1.1 Detailed Description

The need to access data as a stream of bits arises in a number of applications, with data compression as themost common.

Since there is little support for this in C/C++, most applications implement their own functionality, resultingin a plethora of choices and characteristics.

The bitio wrapper classes are my addition to this confusion. They represent the choices I found mostappropriate for the uses I have. I am releasing them in the hope that others will find them useful too.

The main design choice was to implement a hierarchy of classes, so that the main functionality is imple-mented by, and available through, the base classes. This makes it very easy to implement derived classesfor specific uses, while facilitating code independent of the type of the underlying data.

Another choice was to implement mainly through lightweight wrappers, which do not manage the under-lying data, but rather add functionality when needed.

Author:Joergen Ibsen / Jibz

Date:2003.08.02

Compounds

• classbitreader

Base class for bit input wrappers.

• classnull bitreader

Specialization without data.

• classstreambitreader

Specialization for streams.

14 Octane Module Documentation

• classvectorbitreader

Specialization for vector<char>.

• classstring bitreader

Specialization for string.

• classarraybitreader

Specialization for C pointer to char.

• classbitwriter

Base class for bit output wrappers.

• classnull bitwriter


• classstreambitwriter


• classvectorbitwriter


• classstring bitwriter


• classarraybitwriter


• classfile bitreader

Specialization for files.

• classfile bitwriter



7.2 Coders 15

7.2 Coders


During compression, the job of the Coder class is use probability information from the modeler to figureout the optimal bitcodes to use when writing symbols into the output stream.

During decompression, the coder is responsible for reading bitcodes from the input and providing a streamof symbol numbers to the compressor.

Compounds

• classOctaneCoder

During compression, the job of the Coder class is use probability information from the modeler to figure outthe optimal bitcodes to use when writing symbols into the output stream.

• classHuffmanCoder


• classSampleCoder

The Sample Coder is useful only as an example of how coders work.



7.3 Compressors


The Base Compressor class implements our generic interface to all compressors, and includes wrappers tomeasure compression/decompression times and diskspace.

Compounds

• classOctaneCompressor


• classmtfllCompressor

Experimental move to front encoding using a linked list instead of tracing the array.

• classSampleCompressor

This Sample Statistical Compressor is a minimal example of how to coordinate the 3 components of thestatistical compressor framework, a sample parser,modeler,and coder.

• classSampleStatCompressor


• classOctaneCompressorStatistical

The Statistical compressor class is a base class from which to derive specific statistical compressors thatmake use of the parser/modeler/coder framework.

• classSubStrHuffCompressor

The SubStrHuff Compressor is a basic Statistical compressor, that instantiates a Zero-order modeler, with ahuffman coder, and a SubString parser.

• classzleCompressor

This is a zero-length-encoder, it performs run-length-encoding for 0 bytes only.


7.4 Utility 17

7.4 Utility


Utility and helper classes, including timing.

Compounds

• classstringparser

Simple string parser.

• classOctaneScopeTimer

Simple timer, measuring time from construction to destruction.

• classOctaneTimer

Simple class for timing code (low resolution).



7.5 Core


Core components.

Compounds

• classCompressorManager

The CompressorManager classes holds a collection of Compressor derived classes and is used for runtimeselection of which algorithm to test.

• classOctaneClass

Base class from which most octane classes derive (including parser,coder,modeler,etc).


7.6 Modelers 19

7.6 Modelers


The Modeler class is for statistical compression methods.

It’s job is to continually maintain a probability distribution over the next symbol to be parsed. This proba-bility information can be used by a coder (like an arithmetic or huffman coder) to optimally code the parsedsymbol.

Compounds

• classOctaneModeler


• classSampleModeler

This is a sample modeler which simply ignores all ’training’ information and assigns equal probability toeach symbol in the parser (sometimes called a -1 order model).

• classOctaneModelerWeightVectored

This is a derived modeler class which has some default functions for modelers which work with a vector ofweights as probabilities (very common).

• classZeroOrderModeler

A simple zero-order modeler, that builds its probability model by counting frequencies in a training file.



7.7 Parsers


The Base Parser class is responsible for dividing the input stream into a series of numerical symbols, andfor converting symbol numbers to their symbol texts during decompression.

Parsers are commonly just character-based, returning the ascii character for each symbol parsed, and there-fore require no training. But some parsers can be more sophisticated, scanning input streams and buildinga list of all words or common words, etc.

Compounds

• classOctaneParser


• classBitParser

Sample parser class is just character based, uses one symbol for each ascii character plus one for end-of-stream.

• classSampleParser


• classSubstringParser

The SubstringParser class is a flexible parser, which uses arbitrarily long substrings of text as symbols.


Chapter 8

Octane Class Documentation

8.1 array bitreader Class Reference

#include <bitreaders.hpp >

Inheritance diagram for arraybitreader::

array_bitreader

bitreader



Definition at line 122 of file bitreaders.hpp.

Public Member Functions

• arraybitreader(const char∗ s, sizet size in bits)

Constructor with optional size specification.

• bool supportsseek() const

Check if seek functions are supported.

Protected Member Functions

• charreadnext byte()

Defines the method of reading a single byte of input.

• void setposition(size t pos)

Defines the method of setting the position for the next read.

22 Octane Class Documentation

Protected Attributes

• const char∗ s• size t index

8.1.2 Constructor & Destructor Documentation

8.1.2.1 array bitreader::array bitreader (const char∗ s, sizet sizein bits) [inline]


Parameters:s - the data to use.

sizein bits - the number of bits available.


129 : bitreader(size_in_bits), s(_s), index(0) { ; }

8.1.3 Member Function Documentation

8.1.3.1 bool arraybitreader::supports seek () const [inline, virtual]


Returns:true if seek functions are supported.

Implementsbitreader.


133 { return true; }

8.1.3.2 char array bitreader::read next byte () [inline, protected, virtual]


Derived classes must implement this function.

Returns:the next byte (8 bits) from the input.



136 { return s[index++]; }


8.1 array bitreader Class Reference 23

8.1.3.3 void array bitreader::set position (sizet pos) [inline, protected, virtual]


Derived classes should implement this function if possible.

Parameters:pos - the position for the next read.

Reimplemented frombitreader.


137 { index = pos; }

The documentation for this class was generated from the following file:

• bitio/bitreaders.hpp



8.2 array bitwriter Class Reference

#include <bitwriters.hpp >

Inheritance diagram for arraybitwriter::

array_bitwriter

bitwriter



Definition at line 89 of file bitwriters.hpp.


• arraybitwriter (char∗ s)

Constructor.


• void write next byte(char c)

Defines the method of writing a single byte of output.


• char∗ s


8.2.2.1 array bitwriter::array bitwriter (char ∗ s) [inline]

Constructor.

Parameters:s - the pointer to use.


94 : s(_s) { ; }


8.2 array bitwriter Class Reference 25


8.2.3.1 void array bitwriter::write next byte (char c) [inline, protected, virtual]


Derived classes should implement this function.

Parameters:c - the 8 bits to write to the output.

Implementsbitwriter.


99 { *s++ = c; }


• bitio/bitwriters.hpp



8.3 BitParser Class Reference

#include <bitparser.hpp >

Inheritance diagram for BitParser::

BitParser

OctaneParser

OctaneClass



Because it uses a fixed symbol set it is ready to parse without any training.

Definition at line 41 of file bitparser.hpp.


• BitParser()

constructor

• ∼BitParser()

destructor

• virtual boolCreateSymbolSetUsingStream(bitreader&from)

Process (train on) an input stream to update/create a symbol set from it.

• virtual boolIsReadyToParse()

are we ready to parse? i.e. has symbol set been built.

• virtual boolRewindAnyBufferedInput(bitreader&from)

Let go of any buffered stream - this is an odd function that can be called be compressor if it wants to handoff the input stream to a new parser or otherwise access the input bitstream from after last symbol parsed.

• virtual voidSynchronizeStateForNewStream()

Synchronize state for a new stream - this will always be called before beginning a new parsing stream, andshould be used to reset the parse into any initial predictable state.

• virtual int GetSymbolCount()

Get a count of the number of symbols stored in the parser.

• virtual boolParseNextSymbolFromInput(bitreader&from, int &symbolnum)

Parse the next symbol from the input and return its #.


8.3 BitParser Class Reference 27

• virtual boolWriteSymbolText(bitwriter &to, int symbolnum, bool &isendofstreamsymbol)

Parse the next symbol from the input, and set symbolnum to the symbol id#,.

• virtual stringLookupSymbolText(int symbolnum)

Helper function to return the text string of a symbol number.

• virtual voidShowDebuggingInfo()

show any debugging info on request (used by various derived classes) [optional]

• virtual unsigned intGetMemoryUsed()

Report actual current memory usage (in bytes).

• virtual unsigned intGetDiskspaceUsed(bool fortempdecompressiononly)

Report disk space (in bytes) that would be used when saving state to file (in bytes).

• virtual std::stringGetParametersInformation()

Reports information about any parameters available.

• virtual voidSetDefaultParameters()

Reset any parameters to their default values.

• virtual boolSetParameter(const std::string &parametername, const std::string &parametervalue)

This function is called to set a variable value.

• virtual boolSaveState(bitwriter &to, bool fortempdecompressiononly)

Save state of object to output stream (if appropriate).

• virtual boolLoadState(bitreader&from)

Load state of object from input stream (if appropriate).


• int Parameterbitlength

The number of bits per chunk.

• bool senteos

We need to track whether we sent an EOS yet.


8.3.2.1 virtual bool BitParser::CreateSymbolSetUsingStream (bitreader & from) [inline,virtual]


Returns:true on success



Reimplemented fromOctaneParser.


60 {return true;};

8.3.2.2 virtual bool BitParser::RewindAnyBufferedInput (bitreader & from) [inline,virtual]


it is necessary because some parsers can read-ahead in input buffer, and so must rewind the bitstream.



62 {return true;};

8.3.2.3 bool BitParser::ParseNextSymbolFromInput (bitreader & from, int & symbolnum)[virtual]


Note:The end of stream situation must to be handled specially:When a parser encounters the end of a stream, it∗MUST∗ return a symbol signifying an end-of-streamsymbol.This end of stream symbol must be a unique symbol from the symbol set.

Returns:false∗after∗ the end of stream symbol is returned on prior call.

ImplementsOctaneParser.

Definition at line 32 of file bitparser.cpp.

References bitreader::empty(), bitreader::get(), and senteos.

33 {34 // grab an input stream symbol and set its INDEX (in symbol vector) for symbolnum35 // return false after EOS36 unsigned char c;3738 // are we at end of from input - this is the only non-intuitive step39 // the issue is that only the parser knows about end-of-stream symbols, and when it hits and end of input40 // it basically needs to reply TWICE, first with an end-of-stream, and then with a reply saying ’no more symbols’41 if (from.empty())42 {43 // no more symbols left - BUT the question now is, do we return an EOS symbol, or false for no symbols left44 if (senteos)45 {46 // we already sent an EOS so from now on any requests for a symbol returns false saying no more symbols available47 return false;48 }49 else



50 {51 // we are going to drop down to return the EOS signal, but we set flag so we don’t do it again52 senteos=true;53 }54 // end of stream symbol number55 symbolnum=SampleParser_EndOfStreamSYMBOL;56 }57 else58 {59 // grab an unsigned character from input and assign the symbol # to its ascii number60 from.get(c);61 symbolnum=(int)c;62 }6364 // return true, saying we read a symbol65 return true;66 }

8.3.2.4 bool BitParser::WriteSymbolText (bitwriter & to, int symbolnum, bool &isendofstreamsymbol) [virtual]


Returns:false∗after∗ end of stream (i.e. first response at end of stream should be the end-of-stream symbol).



References bitwriter::put().

70 {71 // write the symbol indexed by symbolnum72 // sets isendofostreamsymbol to true or false depending on if the symbol written is the EOS symbol73 // return true on success7475 if (symbolnum==SampleParser_EndOfStreamSYMBOL)76 {77 // this is end of stream symbol, so do nothing but set EOS flag78 isendofstreamsymbol=true;79 }80 else81 {82 // not EOS, so write it and set EOS flag false83 to.put((unsigned char)symbolnum);84 isendofstreamsymbol=false;85 }8687 // return success88 return true;89 }

8.3.2.5 string BitParser::LookupSymbolText (int symbolnum) [virtual]


Returns:a string with the text of the symbol



Note:this should be the empty string to signify end of stream symbol.



93 {94 // return the string text corresponding to symbol ’symbolnum’95 if (symbolnum==SampleParser_EndOfStreamSYMBOL)96 return "";97 char cstr[2];98 cstr[0]=(unsigned char)symbolnum;99 cstr[1]=’\0’;100 return string(cstr);101 }

8.3.2.6 bool BitParser::SetParameter (const std::string &parametername, const std::string &parametervalue) [inline, virtual]


Note:this can be called with a variable not owned by this class, in which case the function should returnfalse.

Returns:true if the variable was found (whether it was set properly or not).

Reimplemented fromOctaneClass.


81 {return false;};

8.3.2.7 bool BitParser::SaveState (bitwriter & to, bool fortempdecompressiononly) [inline,virtual]


Returns:true on success (or if no information needs to be saved).



82 {return true;};



8.3.2.8 bool BitParser::LoadState (bitreader & from) [inline, virtual]


Returns:true on success (or if no information needs to be loaded).



83 {return true;};

8.3.3 Member Data Documentation

8.3.3.1 boolBitParser::senteos [protected]


Todothere should be a way to do this automatically perhaps with a wrapper, rather than insisting all derivedparsers deal with returning an EOS symbol.


Referenced by ParseNextSymbolFromInput(), and SynchronizeStateForNewStream().

The documentation for this class was generated from the following files:

• parsers/parserbitlevel/bitparser.hpp• parsers/parserbitlevel/bitparser.cpp



8.4 bitreader Class Reference

#include <bitreader.hpp >

Inheritance diagram for bitreader::

bitreader

array_bitreader file_bitreader null_bitreader stream_bitreader string_bitreader vector_bitreader



The bitreader class is a wrapper which provides bit input functionality to anything supporting input of 8-bit chars (bytes). It attempts to be as clean and transparent as possible, while maintaining a fairly decentefficiency.

The base class contains functions for handling the input by holding bits in a temporary store (a byte),reading more when the store is empty.

It is used by creating a sub-class and implementing thereadnext byte()andsupportsseek()member func-tions.

The constructor of a derived class should initializelength(either directly or through the base constructor).The destructor of the derived class should callfinalize().

Definition at line 47 of file bitreader.hpp.


• bitreader(size t size in bits=0)

Constructor.

• virtual∼bitreader()

Destructor.

• virtual boolerror() const

Check for error.

• void align byte()

Align bitreader on a byte boundary.

Size Functions

• boolempty() constCheck if bitreader is empty.

• size t bits left () constReturn the number of bits left.


8.4 bitreader Class Reference 33

• size t bytesleft () constReturn the number of bytes left.

• size t get bit length() constReturn the total number of bits available.

• size t get byte length() constReturn the total number of bytes available.

• size t tell bit () constReturn the current bit position within the bit stream.

• size t tell byte() constReturn the current byte position within the bit stream.

Seek Functions

• virtual boolsupportsseek() const=0Check if seek functions are supported.

• size t seekbit (size t pos)Seek to a specified bit position within the bit stream.

• size t seekbyte(size t pos)Seek to a specified byte position within the bit stream.

• size t seekbit (int pos, std::ios::seekdir rpos)Seek to a specified relative bit position within the bit stream.

• size t seekbyte(int pos, std::ios::seekdir rpos)Seek to a specified relative byte position within the bit stream.

Input Functions

• boolget bit ()Get and return the next bit.

• unsigned charget byte()Get and return the next byte (8-bit unsigned value).

• unsigned shortget word ()Get and return the next word (16-bit unsigned value).

• unsigned longget dword()Get and return the next dword (32-bit unsigned value).

• floatget float ()Get and return the next float.

• doubleget double()Get and return the next double.

• std::stringget string()



Read and return a string.

• void read(char∗buffer, sizet len)Copy bytes from the bitreader to an array.

Polymorphic Input Function Wrappers

• void get (bool &v)• void get (char &v)• void get (unsigned char &v)• void get (short &v)• void get (unsigned short &v)• void get (int &v)• void get (unsigned int &v)• void get (long &v)• void get (unsigned long &v)• void get (float &v)• void get (double &v)• void get (std::string &s)


• void finalize()

Perform cleanup.

• virtual charreadnext byte()=0


• virtual voidsetposition(size t pos)


• void reposition()

Helper function for reinitializing current and mask after a seek operation.


• size t offs

Current bit position within the bit stream.

• size t length

Total number of bits available.


8.4.2.1 bitreader::bitreader (sizet sizein bits= 0) [inline]

Constructor.



Parameters:sizein bits - the number of bits available.


References length, and offs.

53 : offs(0), length(size_in_bits), current(0), mask(0) { ; }

8.4.2.2 virtual bitreader::∼bitreader () [inline, virtual]

Destructor.

Derived classes should implement a destructor which calls thefinalize()member function.


57 { ; }


8.4.3.1 virtual bool bitreader::error () const [inline, virtual]

Check for error.

Returns:true if an error occurred.

Reimplemented infile bitreader.


62 { return false; }

8.4.3.2 bool bitreader::empty () const [inline]

Check if bitreader is empty.

Returns:true if bitreader is empty, otherwisefalse .



Referenced by SampleCoder::DecodeSymbolFromInput(), HuffmanCoder::DecodeSymbolFrom-Input(), zleCompressor::DoProtectedCompress(), SampleCompressor::DoProtectedCompress(), mtfll-Compressor::DoProtectedCompress(), SampleCompressor::DoProtectedCreateSymbolsAndModelsUsing-Stream(), zleCompressor::DoProtectedDecompress(), SampleCompressor::DoProtectedDecompress(),mtfllCompressor::DoProtectedDecompress(), SampleParser::ParseNextSymbolFromInput(), and Bit-Parser::ParseNextSymbolFromInput().

73 { return offs >= length; }



8.4.3.3 sizet bitreader::bits left () const [inline]

Return the number of bits left.

Returns:the number of bits left in the bitreader.



Referenced by bytesleft().

76 { return length - offs; }

8.4.3.4 sizet bitreader::bytes left () const [inline]

Return the number of bytes left.

Returns:the number of bytes left in the bitreader.


References bitsleft().

Referenced by OctaneTester::CheckFilesAreIdentical().

79 { return bits_left() / 8; }

8.4.3.5 sizet bitreader::get bit length () const [inline]

Return the total number of bits available.

Returns:the number of bits available.


References length.

Referenced by getbyte length().

83 { return length; }

8.4.3.6 sizet bitreader::get byte length () const [inline]

Return the total number of bytes available.

Returns:the number of bytes available.


References getbit length().

87 { return get_bit_length() / 8; }



8.4.3.7 sizet bitreader::tell bit () const [inline]

Return the current bit position within the bit stream.

Returns:the current bit position.


References offs.

Referenced by OctaneCompressorStatistical::DoProtectedCreateSymbolsAndModelsUsingStream(),SubstringParser::RewindAnyBufferedInput(), and tellbyte().

91 { return offs; }

8.4.3.8 sizet bitreader::tell byte () const [inline]

Return the current byte position within the bit stream.

Returns:the current byte position.


References tellbit().

Referenced by OctaneCompressor::Compress(), OctaneCompressor::Decompress(), Octane-Compressor::LoadState(), and OctaneTester::RunCommandDecompress().

95 { return tell_bit() / 8; }

8.4.3.9 virtual bool bitreader::supports seek () const [pure virtual]



Implemented innull bitreader, streambitreader, vectorbitreader, string bitreader, arraybitreader, andfile bitreader.

8.4.3.10 sizet bitreader::seek bit (size t pos) [inline]

Seek to a specified bit position within the bit stream.

Parameters:pos - the bit position to seek to.

Returns:the new position.




References offs, and reposition().

Referenced by OctaneCompressorStatistical::DoProtectedCreateSymbolsAndModelsUsingStream(),SubstringParser::RewindAnyBufferedInput(), seekbit(), and seekbyte().

110 { offs = pos; reposition(); return offs; }

8.4.3.11 sizet bitreader::seek byte (sizet pos) [inline]

Seek to a specified byte position within the bit stream.

Parameters:pos - the byte position to seek to.



References seekbit().

Referenced by zleCompressor::DoProtectedCompress().

115 { return seek_bit(8 * pos); }

8.4.3.12 sizet bitreader::seek bit (int pos, std::ios::seekdir rpos) [inline]

Seek to a specified relative bit position within the bit stream.

Parameters:pos - the relative bit position to seek to.

rpos - the position to seek relative to,

• std::ios::beg - beginning of stream• std::ios::cur - current position• std::ios::end - end of stream



References length, offs, and seekbit().

125 {126 switch (rpos)127 {128 case std::ios::beg:129 return seek_bit(pos);130 break;131 case std::ios::cur:132 return seek_bit(offs + pos);



133 break;134 case std::ios::end:135 return seek_bit(length - pos);136 break;137 }138 return 0;139 }

8.4.3.13 sizet bitreader::seek byte (int pos, std::ios::seekdir rpos) [inline]

Seek to a specified relative byte position within the bit stream.

Parameters:pos - the relative byte position to seek to.

rpos - the position to seek relative to,

• std::ios::beg - beginning of stream• std::ios::cur - current position• std::ios::end - end of stream



References seekbit().

149 {150 return seek_bit(8 * pos, rpos);151 }

8.4.3.14 bool bitreader::getbit () [inline]

Get and return the next bit.

Returns:the next bit (true for 1, false for 0).


References offs, and readnext byte().

Referenced by HuffmanCoder::DecodeSymbolFromInput().

162 {163 ++offs;164165 if (!mask)166 {167 current = read_next_byte();168 mask = 0x80;169 }170171 const bool bit = (bool)(current & mask);172 mask >>= 1;173174 return bit;175 }



8.4.3.15 unsigned char bitreader::getbyte () [inline]

Get and return the next byte (8-bit unsigned value).

Returns:the next 8 bits.


References offs, and readnext byte().

Referenced by OctaneTester::CheckFilesAreIdentical(), zleCompressor::DoProtectedCompress(),SampleCompressor::DoProtectedCompress(), mtfllCompressor::DoProtectedCompress(), Sample-Compressor::DoProtectedCreateSymbolsAndModelsUsingStream(), zleCompressor::DoProtected-Decompress(), SampleCompressor::DoProtectedDecompress(), mtfllCompressor::DoProtected-Decompress(), getdword(), getstring(), getword(), OctaneCompressor::LoadGUID(), Substring-Parser::ParseNextSymbolFromInput(), and read().

180 {181 offs += 8;182183 if (!mask) return read_next_byte();184185 size_t bit_offs = offs & 0x0007;186 unsigned char tmp = current << (unsigned char)bit_offs;187 current = read_next_byte();188189 return tmp | (current >> (8 - (unsigned char)bit_offs));190 }

8.4.3.16 unsigned short bitreader::getword () [inline]

Get and return the next word (16-bit unsigned value).



References getbyte().

Referenced by getstring().

195 {196 unsigned short v = get_byte();197 v = (v << 8) | get_byte();198199 return v;200 }

8.4.3.17 unsigned long bitreader::getdword () [inline]

Get and return the next dword (32-bit unsigned value).






Referenced by getstring().

205 {206 unsigned long v = get_byte();207 v = (v << 8) | get_byte();208 v = (v << 8) | get_byte();209 v = (v << 8) | get_byte();210211 return v;212 }

8.4.3.18 float bitreader::getfloat () [inline]

Get and return the next float.

Returns:the next float.

Warning:Experimental, not portable.

TodoExperimental, not portable.


References read().

219 {220 float v;221 read((char*)&v, sizeof(float));222 return v;223 }

8.4.3.19 double bitreader::getdouble () [inline]

Get and return the next double.

Returns:the next double.




References read().

230 {231 double v;232 read((char*)&v, sizeof(double));233 return v;234 }



8.4.3.20 std::string bitreader::get string () [inline]

Read and return a string.

Returns:the string.


References getbyte(), getdword(), and getword().

239 {240 size_t len = get_word();241242 if (len == 0) len = get_dword();243244 std::string s;245246 s.reserve(len);247 for ( ; len; --len) s += get_byte();248249 return s;250 }

8.4.3.21 void bitreader::read (char∗ buffer, sizet len) [inline]

Copy bytes from the bitreader to an array.

Parameters:buffer - pointer to where the bytes should be stored.

len - the number of bytes to copy.



Referenced by getdouble(), and getfloat().

256 {257 for ( ; len; --len) *buffer++ = get_byte();258 }

8.4.3.22 void bitreader::finalize () [inline, protected]

Perform cleanup.

Derived classes should call this method from their destructor.


References alignbyte().

290 { align_byte(); }



8.4.3.23 virtual char bitreader::read next byte () [protected, pure virtual]




Implemented innull bitreader, streambitreader, vectorbitreader, string bitreader, arraybitreader, andfile bitreader.

Referenced by getbit(), get byte(), and reposition().

8.4.3.24 virtual void bitreader::set position (sizet pos) [inline, protected, virtual]




Reimplemented invectorbitreader, string bitreader, arraybitreader, andfile bitreader.


Referenced by reposition().

300 { ; }


• bitio/bitreader.hpp



8.5 bitwriter Class Reference

#include <bitwriter.hpp >

Inheritance diagram for bitwriter::

bitwriter

array_bitwriter file_bitwriter null_bitwriter stream_bitwriter string_bitwriter vector_bitwriter



The bitwriter class is a wrapper which provides bit output functionality to anything supporting output of8-bit chars (bytes). It attempts to be as clean and transparent as possible, while maintaining a fairly decentefficiency.

The base class contains functions for handling the output by collecting bits in a temporary store (a byte),writing them out when the store is full.

It is used by creating a sub-class and implementing thewrite next byte()member function.

The destructor of the derived class should callfinalize().

Definition at line 40 of file bitwriter.hpp.


• bitwriter ()

Constructor.

• virtual∼bitwriter ()

Destructor.

• virtual boolerror() const

Check for error.

• void align byte()

Align bitwriter on a byte boundary.

Size Functions

• size t bits done() constReturn the number of bits written.

• size t bytesdone() constReturn the number of bytes written.

• size t tell bit () constReturn the current bit position within the bit stream.


8.5 bitwriter Class Reference 45

• size t tell byte() constReturn the current byte position within the bit stream.

Output Functions

• void put bit (bool v)Write one bit to the output.

• void put byte(unsigned char v)Write one byte (8-bit unsigned value).

• void put word (unsigned short v)Write one word (16-bit unsigned value).

• void put dword(unsigned long v)Write one dword (32-bit unsigned value).

• void put float (float v)Write one float.

• void put double(double v)Write one double.

• void put string(std::string &s)Write a string.

• void write (const char∗buffer, sizet length)Copy bytes from an array to the bitwriter.

Polymorphic Output Function Wrappers

• void put (bool v)• void put (char v)• void put (unsigned char v)• void put (short v)• void put (unsigned short v)• void put (int v)• void put (unsigned int v)• void put (long v)• void put (unsigned long v)• void put (float v)• void put (double v)• void put (std::string &s)


• void finalize()

Perform cleanup.

• virtual voidwrite next byte(char c)=0





• size t num

Number of bytes written.


8.5.2.1 virtual bitwriter:: ∼bitwriter () [inline, virtual]

Destructor.

Derived classes should implement a destructor which calls thefinalize()member function.


48 { ; }


8.5.3.1 virtual bool bitwriter::error () const [inline, virtual]

Check for error.

Returns:true if an error occurred.

Reimplemented infile bitwriter.



8.5.3.2 sizet bitwriter::bits done () const [inline]

Return the number of bits written.

Returns:the number of bits written by the bitwriter.


References num.

Referenced by tellbit().

72 { return 8*num + 8 - log2_tab[mask]; }



8.5.3.3 sizet bitwriter::bytes done () const [inline]

Return the number of bytes written.

Returns:the number of bytes written by the bitwriter.


References num.

Referenced by tellbyte().

75 { return num; }

8.5.3.4 sizet bitwriter::tell bit () const [inline]

Return the current bit position within the bit stream.

Returns:the current bit position.


References bitsdone().

79 { return bits_done(); }

8.5.3.5 sizet bitwriter::tell byte () const [inline]

Return the current byte position within the bit stream.

Returns:the current byte position.


References bytesdone().

Referenced by OctaneCompressor::Compress(), OctaneCompressor::Decompress(), and Octane-Compressor::Save().

83 { return bytes_done(); }

8.5.3.6 void bitwriter::put bit (bool v) [inline]

Write one bit to the output.

Parameters:v - bit value to write (true for 1, false for 0).




References num, and writenext byte().

Referenced by HuffNodeLeaf::AddBitsToBitWriter().

94 {95 if (v) current |= mask;96 if (!(mask >>= 1))97 {98 ++num; write_next_byte(current);99 current = 0;100 mask = 0x80;101 }102 }

8.5.3.7 void bitwriter::put byte (unsigned charv) [inline]

Write one byte (8-bit unsigned value).

Parameters:v - byte value to write.


References num, and writenext byte().

Referenced by zleCompressor::DoProtectedCompress(), SampleCompressor::DoProtectedCompress(),mtfllCompressor::DoProtectedCompress(), zleCompressor::DoProtectedDecompress(), Sample-Compressor::DoProtectedDecompress(), mtfllCompressor::DoProtectedDecompress(), putdword(),put word(), OctaneCompressor::SaveCompressionHeader(), OctaneCompressor::SaveGUID(), andwrite().

107 {108 ++num;109110 if (mask == 0x80) { write_next_byte(v); return; }111112 unsigned char ml = log2_tab[mask];113 current |= v >> (8 - ml);114 write_next_byte(current);115 current = v << ml;116 }

8.5.3.8 void bitwriter::put word (unsigned shortv) [inline]

Write one word (16-bit unsigned value).

Parameters:v - word value to write.


References putbyte().

Referenced by putstring().



121 {122 put_byte((v >> 8) & 0x00ff);123 put_byte((v ) & 0x00ff);124 }

8.5.3.9 void bitwriter::put dword (unsigned longv) [inline]

Write one dword (32-bit unsigned value).

Parameters:v - dword value to write.



Referenced by putstring().

129 {130 put_byte((unsigned char) ((v >> 24) & 0x00ff));131 put_byte((unsigned char) ((v >> 16) & 0x00ff));132 put_byte((unsigned char) ((v >> 8 ) & 0x00ff));133 put_byte((unsigned char) ((v ) & 0x00ff));134 }

8.5.3.10 void bitwriter::put float (float v) [inline]

Write one float.

Parameters:v - float value to write.




References write().

140 { write((char*)&v, sizeof(float)); }

8.5.3.11 void bitwriter::put double (doublev) [inline]

Write one double.

Parameters:v - double value to write.






References write().

146 { write((char*)&v, sizeof(double)); }

8.5.3.12 void bitwriter::put string (std::string & s) [inline]

Write a string.

Parameters:s - the string to write.


References putdword(), putword(), and write().

151 {152 size_t len = s.length();153154 if ((len > 0x0000ffff) || (len == 0))155 {156 put_word(0);157 put_dword(len);158 } else {159 put_word(len);160 }161162 write(s.c_str(), len);163 }

8.5.3.13 void bitwriter::write (const char ∗ buffer, sizet length) [inline]

Copy bytes from an array to the bitwriter.

Parameters:buffer - pointer to where the bytes should be read.

length - the number of bytes to copy.



Referenced by putdouble(), putfloat(), putstring(), and SubstringParser::WriteSymbolText().

169 {170 for ( ; length; --length) put_byte(*buffer++);171 }



8.5.3.14 void bitwriter::finalize () [inline, protected]

Perform cleanup.

Derived classes should call this method from their destructor.


References alignbyte().

203 { align_byte(); }

8.5.3.15 virtual void bitwriter::write next byte (char c) [protected, pure virtual]




Implemented innull bitwriter, streambitwriter, vectorbitwriter, string bitwriter, arraybitwriter, andfile -bitwriter.

Referenced by alignbyte(), putbit(), and putbyte().


• bitio/bitwriter.hpp• bitio/bitwriter.cpp



8.6 CompressorManager Class Reference

#include <compressormanager.hpp >

Inheritance diagram for CompressorManager::

CompressorManager

OctaneClass



Definition at line 38 of file compressormanager.hpp.


• CompressorManager()

Constructor.

• ∼CompressorManager()

Destructor.

• int GetBaseCompressorCount()

accessor for instantiated compressor vector size

• int GetInstantiatedCompressorCount()

accessor for instantiated compressor vector size

• OctaneCompressor∗ GetBaseCompressorp(int index)

accessor for instantiated compressor vector

• OctaneCompressor∗ GetInstantiatedCompressorp(int index)

accessor for instantiated compressor vector

• OctaneCompressor∗ FindBaseCompressorClass(const std::string &compressorname)

Find a base compressor for subsequent operations.

• OctaneCompressor∗ FindInstantiatedCompressorpFromStringId(std::string idstring)

Find the pointer to the instantiated compressor referred to by idstring idstring can be[$name|$num|G$guid].

• OctaneCompressor∗ FindInstantiatedCompressorpFromGuid(unsigned short int guidindex)

Find the pointer to the instantiated compressor referred to by a guid.

• void RegisterBaseCompressor(OctaneCompressor∗compressorp)


8.6 CompressorManager Class Reference 53

Register a new compressor.

• void UnRegisterBaseCompressor(OctaneCompressor∗compressorp)

Unregister a new compressor.

• void Initialize ()

Initialize the compressdecompress manager before using it.

• void DeInitialize()

Deinitialize the compressdecompress manager when done with it.

• OctaneCompressor∗ CreateCompressorFromSavedStream(bitreader&from)

Create a new compressor based on file info and load state.

• boolAddCompressor(OctaneCompressor∗compressorp)

Add an instantiated compressor to our instantiated list.

• boolDeleteCompressor(OctaneCompressor∗compressorp)

Delete an instantiated compressor to our instantiated list.


• std::vector< OctaneCompressor∗ > compressorvectorbasetypes

Vector holding the registered base compressor types.

• std::vector< OctaneCompressor∗ > compressorvectorinstantiated

Vector holding the registered compressors.


8.6.2.1 OctaneCompressor∗ CompressorManager::FindBaseCompressorClass (const std::string& compressorname)

Find a base compressor for subsequent operations.

Returns:true on success.

Definition at line 96 of file compressormanager.cpp.

References compressorvectorbasetypes.

Referenced by CreateCompressorFromSavedStream(), and OctaneTester::RunCommandCreate-Compressor().

97 {98 // remove a compressor from our list of registered components.99 // you don’t have to call this on exit of the program, only if you are for some reason100 // removing compressors during the execution of the program.101 vector<OctaneCompressor*>::iterator index;



102 for (index=compressorvector_basetypes.begin();index!=compressorvector_basetypes.end();++index)103 {104 if ((*index)->GetClassName()==compressorname)105 return (*index);106 }107 return NULL;108 }

8.6.2.2 OctaneCompressor∗ CompressorManager::FindInstantiatedCompressorpFromStringId(std::string idstring)

Find the pointer to the instantiated compressor referred to by idstring idstring can be[$name|$num|G$guid].

Returns:NULL if not found


References compressorvectorinstantiated.

Referenced by OctaneTester::RunCommandDeleteCompressor(), and OctaneTester::SelectInstantiated-Compressor().

194 {195 // look up a compressor based on different strings196 if (idstring=="")197 return NULL;198199 // there are 3 ways to specify an instantiated compressor200 OctaneCompressor* compressorp=NULL;201 int indexnum;202 indexnum = atoi(idstring.c_str());203204 if (indexnum>0 || idstring=="0")205 {206 // user gave us an index207 if (indexnum<0 || (unsigned int)indexnum>=compressorvector_instantiated.size())208 return NULL;209 compressorp=compressorvector_instantiated[indexnum];210 return compressorp;211 }212213 // ok its either a full name or a guid214 // first we try to look it up by name215 vector<OctaneCompressor*>::iterator index;216 for (index=compressorvector_instantiated.begin();index!=compressorvector_instantiated.end();++index)217 {218 if ((*index)->GetInstantiatedName()==idstring)219 return (*index);220 }221222 // we didn’t find it, so try to check its guid223 if (idstring[0]!=’g’ && idstring[0]!=’G’)224 return NULL;225 idstring=idstring.substr(1,idstring.length()-1);226 indexnum=atoi(idstring.c_str());227 if (indexnum>0 || idstring=="0")228 {229 for (index=compressorvector_instantiated.begin();index!=compressorvector_instantiated.end();++index)230 {231 if ((*index)->GetGuid()==indexnum)



232 return (*index);233 }234 }235 return NULL;236 }

8.6.2.3 OctaneCompressor∗ CompressorManager::FindInstantiatedCompressorpFromGuid(unsigned short intguidindex)

Find the pointer to the instantiated compressor referred to by a guid.

Returns:NULL if not found



Referenced by OctaneTester::RunCommandDecompress().

240 {241 // Find the pointer to the instantiated compressor referred to by a guid242 // return NULL if not found243 vector<OctaneCompressor*>::iterator index;244 for (index=compressorvector_instantiated.begin();index!=compressorvector_instantiated.end();++index)245 {246 if ((*index)->GetGuid()==guidindex)247 return (*index);248 }249 return NULL;250 }

8.6.2.4 OctaneCompressor∗ CompressorManager::CreateCompressorFromSavedStream(bitreader & from)

Create a new compressor based on file info and load state.

Returns:NULL on failure


References FindBaseCompressorClass(), bitreader::get(), OctaneCompressor::LoadState(), Octane-Compressor::MakeCompressorInstantiation(), and OctaneCompressor::SetInfo().

Referenced by OctaneTester::RunCommandDecompress(), and OctaneTester::RunCommandLoad-Compressor().

116 {117 // in the saved stream we expect to find a ’header’ which tells us the 1)compressorclass, 2)compressorname, and 3)compressorguid118 // then we create a compressor of that type, and tell it to read its state info from the remainder of stream119120 // read standard compressor header (see OctaneCompressor::Save() for where this is written)121 string compressorclassname;122 string compressorname;123 short int compressorguid;124 from.get(compressorclassname);



125 from.get(compressorname);126 from.get(compressorguid);127128 // select base compressor class type129 OctaneCompressor *compressorp = FindBaseCompressorClass(compressorclassname);130 if (compressorp==NULL)131 {132 // couldn’t find base class133 cout << "ERROR: compressor in file is based on compressor class ’"<<compressorclassname<<"’ which is not known."<<endl;134 return NULL;135 }136137 // ask base class to make us a derived class138 compressorp=compressorp->MakeCompressorInstantiation();139 if (compressorp==NULL)140 {141 // couldn’t find base class142 cout << "ERROR: failed to instantiate a derived compressor from base class ’"<<compressorclassname<<"’."<<endl;143 return NULL;144 }145146 // now set details (compressor name and guid) from file147 compressorp->SetInfo(compressorname,compressorguid);148149 // now load the details from the file150 bool bretv=compressorp->LoadState(from);151 if (!bretv)152 {153 delete compressorp;154 compressorp=NULL;155 }156157 // return pointer to newly created compressor158 return compressorp;159 }

8.6.2.5 bool CompressorManager::AddCompressor (OctaneCompressor∗ compressorp)

Add an instantiated compressor to our instantiated list.




Referenced by OctaneTester::RunCommandCreateCompressor(), and OctaneTester::RunCommand-LoadCompressor().

164 {165 // just add a compressor to the instantiated vector166 // show some debug info?167 // cout << "$DEBUG Adding instantiated compressor ’" << compressorp->get_compressor_name() << "’."<<endl;168 compressorvector_instantiated.push_back(compressorp);169 return true;170 }

8.6.2.6 bool CompressorManager::DeleteCompressor (OctaneCompressor∗ compressorp)

Delete an instantiated compressor to our instantiated list.






Referenced by OctaneTester::RunCommandDeleteCompressor().

175 {176 // just add a compressor to the instantiated vector177 vector<OctaneCompressor*>::iterator index;178 for (index=compressorvector_instantiated.begin();index!=compressorvector_instantiated.end();++index)179 {180 if (*index==compressorp)181 {182 // found it, so remove it183 compressorvector_instantiated.erase(index);184 return true;185 }186 }187 return false;188 }


• core/compressormanager.hpp• core/compressormanager.cpp



8.7 CompressorManagerSingletonInsurer Class Reference

#include <compressormanager.hpp >


This is a sneaky little class which allows us to create self-registering compressors.

The point of CompressorManagerSingletonInsurer is simply to insure that a singletonCompressor-Manageris initialized prior to any derived Compressor instantiations. It is instantiated by all objects prioron their construction.



• CompressorManagerSingletonInsurer(OctaneCompressor∗compressorp)

the constructor creates our global singleton if appropriate, and then registers the passed compressor.


8.7.2.1 CompressorManagerSingletonInsurer::CompressorManagerSingletonInsurer(OctaneCompressor∗ compressorp) [inline]

the constructor creates our global singleton if appropriate, and then registers the passed compressor.

So called should just create and destroy a CompressorManagerSingletonInsurer in their constructor (usinga local variable is fine)


References globalcompressormanagerp, and CompressorManager::RegisterBaseCompressor().

118 {if (global_compressormanagerp==NULL) global_compressormanagerp= new CompressorManager();global_compressormanagerp->RegisterBaseCompressor(compressorp);};


• core/compressormanager.hpp


8.8 file bitreader Class Reference 59

8.8 file bitreader Class Reference

#include <file bitreader.hpp >

Inheritance diagram for filebitreader::

file_bitreader

bitreader



This class opens a file and reads from it. Unlike thestreambitreaderit manages the file that it operates on,and thus hides the details of file access.

It adds the member functionerror(), which checks if there was an error opening the file in the constructor.

Definition at line 33 of file filebitreader.hpp.


• file bitreader(const char∗name)

Constructor.

• file bitreader(const char∗name, sizet size in bits)

Constructor with size specification.

• boolerror() const

Check the state of the file.









• std::ifstreams




8.8.2.1 filebitreader::file bitreader (const char∗ name) [inline]

Constructor.

Sets the number of bits available to the size of the file.

Parameters:name - the name of the file to open.


References bitreader::length.

40 : s(name, std::ios::binary)41 {42 if (s)43 {44 length = 8 * s.seekg(0, std::ios::end).tellg();45 s.seekg(0);46 }47 }

8.8.2.2 filebitreader::file bitreader (const char∗ name, sizet sizein bits) [inline]


Parameters:name - the name of the file to open.



52 : bitreader(size_in_bits), s(name, std::ios::binary) { ; }


8.8.3.1 bool filebitreader::error () const [inline, virtual]


Returns:true if a file error has occurred.



Referenced by OctaneTester::CheckFilesAreIdentical(), OctaneTester::RunCommandCompress(),OctaneTester::RunCommandDecompress(), OctaneTester::RunCommandLoadCompressor(), andOctaneTester::RunCommandMakeState().

58 { return !s; }


8.8 file bitreader Class Reference 61

8.8.3.2 bool filebitreader::supports seek () const [inline, virtual]





60 { return true; }

8.8.3.3 char filebitreader::read next byte () [inline, protected, virtual]






63 { char c; s.get(c); return c; }

8.8.3.4 void filebitreader::set position (sizet pos) [inline, protected, virtual]






64 { s.seekg((std::streamoff)pos); }


• bitio/file bitreader.hpp



8.9 file bitwriter Class Reference

#include <file bitwriter.hpp >

Inheritance diagram for filebitwriter::

file_bitwriter

bitwriter



This class creates a file and writes to it. Unlike thestreambitwriter it manages the file that it operates on,and thus hides the details of file access.

It adds the member functionerror(), which checks if there was an error creating the file in the constructor.

Definition at line 33 of file filebitwriter.hpp.


• file bitwriter (const char∗name)

Constructor.

• boolerror() const






• std::ofstreams


8.9.2.1 filebitwriter::file bitwriter (const char ∗ name) [inline]

Constructor.

Parameters:name - the name of the file to create.


8.9 file bitwriter Class Reference 63


38 : s(name, std::ios::binary) { ; }


8.9.3.1 bool filebitwriter::error () const [inline, virtual]


Returns:true if a file error has occurred.

Reimplemented frombitwriter.


Referenced by OctaneTester::RunCommandCompress(), OctaneTester::RunCommandDecompress(),and OctaneTester::RunCommandSaveCompressor().

44 { return !s; }

8.9.3.2 void filebitwriter::write next byte (char c) [inline, protected, virtual]






47 { s.put(c); }


• bitio/file bitwriter.hpp



8.10 HuffmanCoder Class Reference

#include <huffmancoder.hpp >

Inheritance diagram for HuffmanCoder::

HuffmanCoder

OctaneCoder

OctaneClass




The Huffman Coding algorithm was invented by David A. Huffman,http://www.wikipedia.org/wiki/Huffman coding

The idea of using an STL priority queue to build the Huffman tree is from Mark Nelson,http://dogma.net/markn/articles/pq stl/priority.htm.

Definition at line 49 of file huffmancoder.hpp.


• HuffmanCoder()

constructor

• virtual∼HuffmanCoder()

destructor

• virtual std::stringGetName()

provide a unique name for the coder, used in some cases to automatically register the object with a manager

• virtual std::stringGetDescription()

optionally provide a longer (maybe 20-60 characters) description

• virtual std::stringGetHelpInformation()

optionally provide more information about the object on request for help






http://www.wikipedia.org/wiki/Huffman_coding

http://dogma.net/markn/articles/pq_stl/priority.htm.

8.10 HuffmanCoder Class Reference 65

• virtual boolWriteSymbolBits(int symbolnum,bitwriter &bw)

write bits for symbol number

• virtual boolDecodeSymbolFromInput(int &symbolnum,bitreader&br)

Decode the next symbol from the input.

• virtual voidResetState()• virtual voidSynchronizeStateForNewStream()

Synchronize state for a new stream - this∗MUST∗ be called before beginning a new parsing stream.

• virtual boolIsReadyToCode()

Are we ready to actually code and decode?

• virtual voidReceiveNotificationModelChangeAllSymbolWeights(OctaneModeler∗modelerp)

Notify coder that all probabilities are being updated.

• virtual void ReceiveNotificationModelChangeSingleSymbolWeight(OctaneModeler∗modelerp,int symbolnum)

notify that a single symbol probability has changed (by default just calls AllSymbolWeights change above)


• boolBuildUpdateHuffmanTreeAndBitcodes(OctaneModeler∗modelerp)

Using the current probability vector from the modeler, build a huffman tree and compute bitcodes for eachsymbol.

• void TraverseHuffmanTreeBuildBitcodes()

Traverse built tree and assign bitcodes.

• void FreeSymbols()

Free any current tree and symbols in preparation for rebuilding.

• void ShowBitStrings()

Show bitcodes for each symbol - useful for debugging.


• THuffmanNodePriorityQueuesymbolsetpq

Stl priority queue for Huffman tree, built from ”huffman nodes”.

• vector< HuffNode Leaf∗ > symbolvector

vector of pointers to symbols, for fast lookup when it comes time to writing symbols by symbol#

• vector< HuffNode Leaf∗ >::iteratorsymbolvectorpos

vector iterator




8.10.2.1 virtual std::string HuffmanCoder::GetName () [inline, virtual]


Returns:a short∗unique∗ name

Reimplemented fromOctaneCoder.


69 {return "HuffmanCoder";}

8.10.2.2 virtual std::string HuffmanCoder::GetDescription () [inline, virtual]


Returns:a one line description



70 {return "Huffman Coder";}

8.10.2.3 virtual std::string HuffmanCoder::GetHelpInformation () [inline, virtual]


Returns:a long string (can be multiplenewlines)



71 { return ""; }

8.10.2.4 bool HuffmanCoder::WriteSymbolBits (int symbolnum, bitwriter & bw) [virtual]





ImplementsOctaneCoder.

Definition at line 54 of file huffmancoder.cpp.

References symbolvector.

55 {56 // write bits for symbol number - this is easy because we’ve precomputed the bitvectors for every symbol57 assert(symbolnum<(int)(symbolvector.size()));58 return symbolvector[symbolnum]->AddBitsToBitWriter(bw);59 }

8.10.2.5 bool HuffmanCoder::DecodeSymbolFromInput (int &symbolnum, bitreader & br)[virtual]


Returns:true on success, false when there are no more symbols to read.


Definition at line 61 of file huffmancoder.cpp.

References bitreader::empty(), bitreader::getbit(), HuffNode::getchild0(), HuffNode::getchild1(), Huff-Node::isleaf(), and symbolsetpq.

62 {63 // decode a symbol from the input, and set symbolnum to symbol number64 // return false on no symbols left in stream65 HuffNode* nodep;66 bool bitvalue;6768 // now walk through the input bits69 nodep = symbolsetpq.top();70 while (!br.empty())71 {72 // grab a bit73 bitvalue = br.get_bit();7475 // traverse down tree76 if (bitvalue==false)77 nodep=nodep->get_child0();78 else79 nodep=nodep->get_child1();80 if (nodep==NULL)81 {82 cerr << "ERROR: Internal error traversing huffman tree in huffmancoder.cpp, in HuffmanCoder::DecodeSymbolFromInput()." << endl;83 break;84 }8586 // if its leaf we got our symbol87 if (nodep->isleaf())88 {89 // we hit a leaf, so this is our decoded symbol - return its number90 symbolnum=((HuffNode_Leaf*)nodep)->get_symbolid();91 return true;92 }93 else94 {95 // just a middle node, so keep traversing96 }



97 }9899 // no symbols to read100 return false;101 }

8.10.2.6 virtual bool HuffmanCoder::IsReadyToCode () [inline, virtual]





References symbolvector.

90 {if (symbolvector.size()>0) return true; else return false;};

8.10.2.7 virtual void HuffmanCoder::ReceiveNotificationModelChangeAllSymbolWeights(OctaneModeler∗ modelerp) [inline, virtual]


We can be informed by a model when the underlying probabilities are changing this part of the API existsso that we can flexibly and efficiently handle different kinds of situations where the coder needs to besynchronized with the model probabilities. in some cases, a coder may ignore these messages and simplyrebuild its internal datastructure on each coding event in other cases, it will want to incrementally updateas model changes. Two kinds of notifications are supported, depending on how the model updates itselfduring processing, whether it updates one symbol per iteration, or modifies all symbols at once.

See also:OctaneCoder::ReceiveNotificationModelChangeSingleSymbolWeight



References BuildUpdateHuffmanTreeAndBitcodes().

95 {BuildUpdateHuffmanTreeAndBitcodes(modelerp);};

8.10.2.8 virtual void HuffmanCoder::ReceiveNotificationModelChangeSingleSymbolWeight(OctaneModeler∗ modelerp, int symbolnum) [inline, virtual]


Note:the SingleSymbolWeight() call will NOT be made when all weights change; see ReceiveNotification-ModelChangeAllSymbolWeights instead.



See also:OctaneCoder::ReceiveNotificationModelChangeAllSymbolWeights



References BuildUpdateHuffmanTreeAndBitcodes().

96 {BuildUpdateHuffmanTreeAndBitcodes(modelerp);};


8.10.3.1 THuffmanNodePriorityQueueHuffmanCoder::symbolsetpq [protected]

Stl priority queue for Huffman tree, built from ”huffman nodes”.

See also:huffmannodes.hpp


Referenced by BuildUpdateHuffmanTreeAndBitcodes(), DecodeSymbolFromInput(), FreeSymbols(),GetMemoryUsed(), ShowBitStrings(), and TraverseHuffmanTreeBuildBitcodes().


• coders/coderhuffman/huffmancoder.hpp• coders/coderhuffman/huffmancoder.cpp



8.11 HuffNode Class Reference

#include <huffmannodes.hpp >

Inheritance diagram for HuffNode::

HuffNode

HuffNode_Leaf HuffNode_Middle


The base HuffNode class is used to represent both leaf and internal HuffNodes.

leaf HuffNodes have 0s in the child pointers, and their value member corresponds to the character theyencode. internal HuffNodes don’t have anything meaningful in their value member, but their child pointerspoint to other HuffNodes.

The huffman node class is used by theHuffmanCoder.

Definition at line 110 of file huffmannodes.hpp.


• HuffNode()

constructor

• virtual∼HuffNode()

destructor

• booloperator> (constHuffNode∗&a) const

The comparison operator used to order the priority queue.

• virtual void traversebuildbitcodes(std::bitset< DEF HUFFMANPQ MAXFUNCBITS > &cur-rent bitset, int bitsetlength)

recursive traversal of huffman tree for building bitcodes

• virtual void recursivefreechildren()

freeing memory occupied by this node and any children

• THuffmanNodeWeightget weight() const

get weight of node

• void incrementweight(int increment)

increment node weight

• void setweight(THuffmanNodeWeight val)

set node weight


8.11 HuffNode Class Reference 71

• virtual unsigned intget memoryused() const

get memory used by the node and its children

• virtual HuffNode∗ get child0 ()

get pointer to left child


get pointer to right child

• virtual boolisleaf()

is this a leaf node (this is an RTTI type function)


• THuffmanNodeWeightweight

the weight of a node is just the frequency count of the node if it is a leaf, or the∗sum∗ of frequencies for allchildren if it is not


• coders/coderhuffman/huffmannodes.hpp• coders/coderhuffman/huffmannodes.cpp



8.12 HuffNode Leaf Class Reference


Inheritance diagram for HuffNodeLeaf::

HuffNode_Leaf

HuffNode


The leaves of the huffman tree contains actual symbol information.

Each node holds a bitvector representing its bitcode for output/input, which is calculated by traversing thefinal huffman tree. We store it explicitly for fast access during coding. A node also contains the symbolid# that it corresponds to in the parser.



• HuffNode Leaf (int insymbolid, THuffmanNodeWeight inweight)

constructor

• virtual∼HuffNode Leaf ()

destructor

• void recursivefreechildren()




• int get symbolid()

get the symbol id# corresponding to this symbol

• void set symbolid(int val)

set the symbol id#

• unsigned charget bitcodelength()

get length of bitcode

• std::stringget bitcodestring()

get a string representation of the bitcode (useful for debugging)



8.12 HuffNode Leaf Class Reference 73

space used by the node

• virtual boolisleaf()


• virtual boolAddBitsToBitWriter(bitwriter &bw)

add the bits from this leaf symbol to the output bitwriter, will depend on derived class


• unsigned charcodebitsetlength• std::bitset< DEF HUFFMANPQ MAXIMUM CODEBITDEPTH> codebitset

the bits used to code this symbol in the huffman tree

• int symbolid

the parser symbol id# that this node refers to


8.12.2.1 virtual unsigned int HuffNodeLeaf::get memoryused () const [inline, virtual]

space used by the node

Tododo we really need to add sizeof(codebitset) as we are doing?

Reimplemented fromHuffNode.


References codebitset.

212 {return (unsigned int)(sizeof(this)+sizeof(code_bitset));}





8.13 HuffNode Middle Class Reference


Inheritance diagram for HuffNodeMiddle::

HuffNode_Middle

HuffNode


A derived huffman node for a node in the middle of the tree.

It holds pointers to other children nodes.



• HuffNode Middle (HuffNode∗c0,HuffNode∗c1)

constructor

• virtual∼HuffNode Middle ()

destructor

• void recursivefreechildren()





get memory used by the node and its children


get pointer to left child


get pointer to right child

• bool isleaf()



8.13 HuffNode Middle Class Reference 75


• HuffNode∗ child0

pointer to left child

• HuffNode∗ child1

pointer to right child


8.13.2.1 virtual HuffNode Middle:: ∼HuffNode Middle () [inline, virtual]

destructor

Note:this does not free children


162 { ; }





8.14 mtfllCompressor Class Reference

#include <mtfll.hpp >

Inheritance diagram for mtfllCompressor::

mtfllCompressor

OctaneCompressor

OctaneClass



Move to front moves the recently encoded byte to the front, so when it appears the next time in the inputstream it will be encoded with 00. If there comes another byte to the front the previous byte will be movedback by 1 position, so it will be encoded as 01.

Definition at line 25 of file mtfll.hpp.


• mtfllCompressor (bool registerme=false)

• virtual std::stringGetClassName()

Get the assigned name of an instantiated compressor.



• virtual OctaneCompressor∗ MakeCompressorInstantiation()

Instantiate a compressor from this class (like a factory).


• virtual boolDoProtectedCompress(bitreader&s, bitwriter &d)

This is the member function which does the actual compression, and this is the function that should besubclasses by derived classes.

• virtual boolDoProtectedDecompress(bitreader&s, bitwriter &d)

This is the member function which does the actual decompression, and this is the function that should besubclasses by derived classes.


8.14 mtfllCompressor Class Reference 77


8.14.2.1 virtual std::string mtfllCompressor::GetDescription () [inline, virtual]



Reimplemented fromOctaneCompressor.

Definition at line 32 of file mtfll.hpp.

32 {return "Move To Front using linked list";}

8.14.2.2 bool mtfllCompressor::DoProtectedCompress (bitreader & s, bitwriter & d)[protected, virtual]


It is wrapped by the public API callCompress()which performs some measurements on compression speedand streamsize.

Note:this function should never be called by an outside class - the API interface should be called instead.


ImplementsOctaneCompressor.

Definition at line 25 of file mtfll.cpp.

References bitreader::empty(), bitreader::getbyte(), and bitwriter::putbyte().

26 {27 unsigned int array[256], pointers[256];2829 for (unsigned int i = 0; i < 256; i++) array[i] = pointers[i] = i;3031 while (!s.empty())32 {33 unsigned char b = s.get_byte();34 d.put_byte((unsigned char)pointers[b]);3536 if (pointers[b])37 {38 for (unsigned char c = (unsigned char)pointers[b]; c; c--)39 {40 array[c] = array[c-1];41 pointers[array[c]]++;42 }43 array[0] = b;44 pointers[b] = 0;45 }46 }47 return true;48 }



8.14.2.3 bool mtfllCompressor::DoProtectedDecompress (bitreader & s, bitwriter & d)[protected, virtual]


It is wrapped by the public API callDecompress()which performs some measurements on decompressionspeed and streamsize.




Definition at line 50 of file mtfll.cpp.


51 {52 unsigned int array[256], pointers[256];5354 for (unsigned int i = 0; i < 256; i++) array[i] = pointers[i] = i;5556 while (!s.empty())57 {58 unsigned char b = (unsigned char)array[s.get_byte()];59 d.put_byte(b);6061 if (pointers[b])62 {63 for (unsigned char c = (unsigned char)pointers[b]; c; c--)64 {65 array[c] = array[c-1];66 pointers[array[c]]++;67 }68 array[0] = b;69 pointers[b] = 0;70 }71 }72 return true;73 }


• compressors/compressormtfll/mtfll.hpp• compressors/compressormtfll/mtfll.cpp


8.15 null bitreader Class Reference 79

8.15 null bitreader Class Reference


Inheritance diagram for nullbitreader::

null_bitreader

bitreader



Always returns zero.



• null bitreader(size t size in bits=0)








8.15.2.1 nullbitreader::null bitreader (size t sizein bits= 0) [inline]


Parameters:sizein bits - the number of bits available.


34 : bitreader(size_in_bits) { ; }




8.15.3.1 bool nullbitreader::supports seek () const [inline, virtual]






8.15.3.2 char nullbitreader::read next byte () [inline, protected, virtual]






41 { return 0; }




8.16 null bitwriter Class Reference 81

8.16 null bitwriter Class Reference


Inheritance diagram for nullbitwriter::

null_bitwriter

bitwriter





• null bitwriter ()

Constructor.





8.16.2.1 void nullbitwriter::write next byte (char c) [inline, protected, virtual]






37 { ; }





8.17 OctaneClass Class Reference

#include <octaneclass.hpp >

Inheritance diagram for OctaneClass::

OctaneClass

CompressorManager OctaneCoder OctaneCompressor OctaneModeler OctaneParser OctaneTester SymbolWeightVector

HuffmanCoder

SampleCoder

mtfllCompressor

OctaneCompressor_Statistical

SampleCompressor

zleCompressor

OctaneModeler_WeightVectored BitParser

SampleParser

SubstringParser



Definition at line 32 of file octaneclass.hpp.


• OctaneClass()

constructor

• virtual∼OctaneClass()

destructor


















8.17 OctaneClass Class Reference 83







Static Public Member Functions

• void NiceifyHighAsciiString(std::string &str)

static helper to convert a raw binary string into a string which is displayable on screen

• std::stringPrefixNonEmptyString(const std::string headerstr, const std::string str)

prefix a string (can contain newlines) with a header, IFF the string is non-empty


• virtual boolDoProtectedSaveState(bitwriter &to, bool fortempdecompressiononly)

Save the state of the object - this is the virtual function that derived classes should override.

• virtual boolDoProtectedLoadState(bitreader&from)

Load the state of the object - this is the virtual function that derived classes should override.

Static Protected Member Functions

• boolParseParameter(const std::string &parametervalue, unsigned int &param)

get value of an unsinged int parameter

• boolParseParameter(const std::string &parametervalue, int &param)

get value of an int parameter

• boolParseParameter(const std::string &parametervalue, unsigned char &param)

get value of an an unsigned char parameter

• boolParseParameter(const std::string &parametervalue, char &param)

get value of a char parameter

• boolParseParameter(const std::string &parametervalue, bool &param)

get value of a bool parameter




8.17.2.1 virtual std::string OctaneClass::GetName () [inline, virtual]



Reimplemented inOctaneCoder, HuffmanCoder, SampleCoder, OctaneCompressor, OctaneModeler,SampleModeler, OctaneModelerWeightVectored, ZeroOrderModeler, SymbolWeightVector, andOctane-Parser.


46 {return "OctaneClass";}

8.17.2.2 virtual std::string OctaneClass::GetDescription () [inline, virtual]



Reimplemented inOctaneCoder, HuffmanCoder, SampleCoder, OctaneCompressor, mtfllCompressor,SampleCompressor, SampleStatCompressor, OctaneCompressorStatistical, SubStrHuffCompressor, zle-Compressor, OctaneModeler, SampleModeler, OctaneModelerWeightVectored, ZeroOrderModeler,SymbolWeightVector, andOctaneParser.


49 {return "Base Octane Class";}

8.17.2.3 virtual std::string OctaneClass::GetHelpInformation () [inline, virtual]



Reimplemented inOctaneCoder, HuffmanCoder, SampleCoder, OctaneCompressor, OctaneCompressor-Statistical, OctaneModeler, SymbolWeightVector, andOctaneParser.


52 { return ""; }



8.17.2.4 virtual bool OctaneClass::SetParameter (const std::string &parametername, conststd::string & parametervalue) [inline, virtual]




Reimplemented inSampleCompressor, OctaneCompressorStatistical, BitParser, SampleParser, andSubstringParser.


Referenced by OctaneTester::ParseCommand(), and OctaneCompressorStatistical::SetParameter().

70 {return false;};

8.17.2.5 virtual bool OctaneClass::SaveState (bitwriter & to, bool fortempdecompressiononly)[inline, virtual]



Reimplemented inOctaneModelerWeightVectored, SymbolWeightVector, BitParser, SampleParser, andSubstringParser.


References DoProtectedSaveState().

Referenced by OctaneCompressorStatistical::DoProtectedSaveState().

73 {return DoProtectedSaveState(to,fortempdecompressiononly);};

8.17.2.6 virtual bool OctaneClass::LoadState (bitreader & from) [inline, virtual]



Reimplemented inOctaneCompressor, OctaneModelerWeightVectored, SymbolWeightVector, BitParser,SampleParser, andSubstringParser.


References DoProtectedLoadState().

Referenced by OctaneCompressorStatistical::DoProtectedLoadState().

76 {return DoProtectedLoadState(from);};



8.17.2.7 bool OctaneClass::ParseParameter (const std::string &parametervalue, unsigned int &param) [inline, static, protected]

get value of an unsinged int parameter



Referenced by SubstringParser::SetParameter(), and SampleCompressor::SetParameter().

92 {param=(unsigned int)(atoi(parametervalue.c_str()));return true;}

8.17.2.8 bool OctaneClass::ParseParameter (const std::string &parametervalue, int & param)[inline, static, protected]

get value of an int parameter



95 {param=(int)(atoi(parametervalue.c_str()));return true;}

8.17.2.9 bool OctaneClass::ParseParameter (const std::string &parametervalue, unsigned char &param) [inline, static, protected]

get value of an an unsigned char parameter



98 {param=(unsigned char)(atoi(parametervalue.c_str()));return true;}

8.17.2.10 bool OctaneClass::ParseParameter (const std::string &parametervalue, char & param)[inline, static, protected]

get value of a char parameter



101 {param=(char)(atoi(parametervalue.c_str()));return true;}



8.17.2.11 bool OctaneClass::ParseParameter (const std::string &parametervalue, bool & param)[static, protected]

get value of a bool parameter


Definition at line 20 of file octaneclass.cpp.

21 {22 // convert and store a bool parameter value23 // ATTN: this is case sensitive and should be changed to case-insensitive24 if (parametervalue == "true")25 param = true;26 else if (parametervalue == "false")27 param = false;28 else if (atoi(parametervalue.c_str()) == 0)29 param = false;30 else if (atoi(parametervalue.c_str()) == 1)31 param = true;32 else33 return false;34 return true;35 }

8.17.2.12 virtual bool OctaneClass::DoProtectedSaveState (bitwriter & to, boolfortempdecompressiononly) [inline, protected, virtual]

Save the state of the object - this is the virtual function that derived classes should override.

It is wrapped by the non-virtual functionSaveState()above which can perform some timing and size mea-surements.

Reimplemented inOctaneCompressor, SampleCompressor, andOctaneCompressorStatistical.


Referenced by SaveState().

112 {return true;};

8.17.2.13 virtual bool OctaneClass::DoProtectedLoadState (bitreader & from) [inline,protected, virtual]

Load the state of the object - this is the virtual function that derived classes should override.

It is wrapped by the non-virtual functionLoadState()above which can perform some timing and sizemeasurements.

Reimplemented inOctaneCompressor, SampleCompressor, andOctaneCompressorStatistical.


Referenced by LoadState().

115 {return true;};




• core/octaneclass.hpp• core/octaneclass.cpp


8.18 OctaneCoder Class Reference 89

8.18 OctaneCoder Class Reference

#include <coder.hpp >

Inheritance diagram for OctaneCoder::

OctaneCoder

OctaneClass

HuffmanCoder SampleCoder




Definition at line 50 of file coder.hpp.


• OctaneCoder()

constructor

• virtual∼OctaneCoder()

destructor







• virtual boolWriteSymbolBits(int symbolnum,bitwriter &bw)=0


• virtual boolDecodeSymbolFromInput(int &symbolnum,bitreader&br)=0


• virtual voidResetState()• virtual voidSynchronizeStateForNewStream()

Synchronize state for a new stream - this∗MUST∗ be called before beginning a new parsing stream.



• virtual boolPrepareForCoding(OctaneModeler∗modelerp)

Prepare for coding mode; returns true on success.





• virtual void ReceiveNotificationModelChangeSingleSymbolWeight(OctaneModeler∗modelerp,int symbolnum)


• void AlignOutputStreamToByteBoundry(bitwriter &bw)

Align output stream - called after writing an End-of-stream symbol This throws away any spare bits in theoutput which are not aligned on a byte boundary.

• void AlignInputStreamToByteBoundry(bitreader&br)

Align input stream - called after reading an End-of-stream symbol This throws away any spare bits in theoutput which are not aligned on a byte boundary.


8.18.2.1 virtual std::string OctaneCoder::GetName () [inline, virtual]




Reimplemented inHuffmanCoder, andSampleCoder.


60 {return "OctaneCoder";}

8.18.2.2 virtual std::string OctaneCoder::GetDescription () [inline, virtual]






61 {return "Base Coder";}



8.18.2.3 virtual std::string OctaneCoder::GetHelpInformation () [inline, virtual]






Referenced by OctaneCompressorStatistical::GetHelpInformation().

62 { return ""; }

8.18.2.4 virtual bool OctaneCoder::WriteSymbolBits (intsymbolnum, bitwriter & bw) [purevirtual]



Implemented inHuffmanCoder, andSampleCoder.

Referenced by OctaneCompressorStatistical::DoProtectedCompress().

8.18.2.5 virtual bool OctaneCoder::DecodeSymbolFromInput (int &symbolnum, bitreader & br)[pure virtual]



Implemented inHuffmanCoder, andSampleCoder.

Referenced by OctaneCompressorStatistical::DoProtectedDecompress().

8.18.2.6 virtual bool OctaneCoder::PrepareForCoding (OctaneModeler∗ modelerp) [inline,virtual]

Prepare for coding mode; returns true on success.



Referenced by OctaneCompressorStatistical::PrepareForCompression().

82 {return true;};



8.18.2.7 virtual bool OctaneCoder::IsReadyToCode () [inline, virtual]





Referenced by OctaneCompressorStatistical::IsReadyToCompress().

85 {return true;};

8.18.2.8 virtual void OctaneCoder::ReceiveNotificationModelChangeAllSymbolWeights(OctaneModeler∗ modelerp) [inline, virtual]






Referenced by OctaneCompressorStatistical::DoProtectedCreateSymbolsAndModelsUsingStream(),OctaneCompressorStatistical::DoProtectedLoadState(), ReceiveNotificationModelChangeSingle-SymbolWeight(), and OctaneCompressorStatistical::SetupAnyDefaultParser().

101 {;};

8.18.2.9 virtual void OctaneCoder::ReceiveNotificationModelChangeSingleSymbolWeight(OctaneModeler∗ modelerp, int symbolnum) [inline, virtual]


Note:the SingleSymbolWeight() call will NOT be made when all weights change; see ReceiveNotification-ModelChangeAllSymbolWeights instead.

See also:OctaneCoder::ReceiveNotificationModelChangeAllSymbolWeights



Reimplemented inHuffmanCoder.


References ReceiveNotificationModelChangeAllSymbolWeights().

105 {ReceiveNotification_ModelChange_AllSymbolWeights(modelerp);};

8.18.2.10 void OctaneCoder::AlignOutputStreamToByteBoundry (bitwriter & bw) [inline]

Align output stream - called after writing an End-of-stream symbol This throws away any spare bits in theoutput which are not aligned on a byte boundary.

This is required when we hit and end of stream symbol, to avoid treating leftover bits as meaningfulsymbols.



References bitwriter::alignbyte().

Referenced by OctaneCompressorStatistical::DoProtectedCompress().

113 {bw.align_byte();};

8.18.2.11 void OctaneCoder::AlignInputStreamToByteBoundry (bitreader & br) [inline]

Align input stream - called after reading an End-of-stream symbol This throws away any spare bits in theoutput which are not aligned on a byte boundary.

This is required when we hit and end of stream symbol, to avoid treating leftover bits as meaningfulsymbols.



References bitreader::alignbyte().


118 {br.align_byte();};


• coders/coder.hpp



8.19 OctaneCompressor Class Reference

#include <compressor.hpp >

Inheritance diagram for OctaneCompressor::

OctaneCompressor

OctaneClass

mtfllCompressor OctaneCompressor_Statistical SampleCompressor zleCompressor

SampleStatCompressor SubStrHuffCompressor



Definition at line 45 of file compressor.hpp.


• OctaneCompressor(bool registerme=false)

constructor

• virtual∼OctaneCompressor()

destructor







• virtual unsigned short intGetCompressorStateFileGUID()

Return a universally unique unsigned-short-int id# for a predefined dictionary id file.

• virtual OctaneCompressor∗ MakeCompressorInstantiation()=0




• std::stringGetInstantiatedName()

Get the user-set instantiated object name for this compressor.


8.19 OctaneCompressor Class Reference 95

• unsigned short intGetGuid()

Get guid.

• void SetInstantiatedName(std::string val)

Set the user-set instantiated object name for this compressor.

• void SetInfo(std::string compressorname, unsigned short int compressorguid)

Set details (compressor name and guid) from file.

• virtual voidResetState()

Reset the state of the compressor to its initial state.


Synchronize state for a new stream - this will be called each time before beginning a new parsing stream.

• virtual boolPrepareForCompression()

Prepare for parsing mode, this will be called parsing begins.

• virtual voidSetupAnyDefaultParser()

Some compressors contains parsers which begin ready to parse, without any ’training’.

• virtual boolIsReadyToCompress()

is the compressor ready to compress? Normally this is false until a dictionary is built or loaded.

• bool Compress(bitreader&from, bitwriter &to, bool needtowriteheader, bool savedecompressstate-info)

The Compress function is an API wrapper around a protected function which does the actual compression,in order to perform some timing/stats gathering functions.

• boolDecompress(bitreader&from, bitwriter &to, bool needtoreadheader)

The Decompress function is an API wrapper around a protected function which does the actual compression,in order to perform some timing/stats gathering functions.

• boolCreateSymbolsAndModelsUsingStream(bitreader&from)

Wrappers around the actual calls to DoProtected.., in order to perform some timing and space measurement.

• boolSave(bitwriter &to, bool fortempdecompressiononly)


• boolLoad(bitreader&from)

Normally the compressor manager loads the header of a saved file and dynamically instantiates a compres-sor, but we provide this function for manually calling load on an already instantiated compressor.

• boolLoadState(bitreader&from)


• void ResetStatistics()

Reset the cumulative statistics tracking.

• unsigned intGetStatisticsCumulativeCompressionBytesRead()



cumulative stats, can be reset with ResetCumulativeStatistics()

• unsigned intGetStatisticsCumulativeDecompressionBytesRead()


• unsigned intGetStatisticsCumulativeCompressionBytesWritten()


• unsigned intGetStatisticsCumulativeDecompressionBytesWritten()


• floatGetStatisticsCumulativeCompressionBitrate()

cumulative stats, can be reset with ResetCumulativeStatistics(); bit rate is in bits per bytes.

• floatGetStatisticsCumulativeDecompressionBitrate()

cumulative stats, can be reset with ResetCumulativeStatistics(); bit rate is in bits per bytes.

• floatGetStatisticsCumulativeCompressionRatio()

cumulative stats, can be reset with ResetCumulativeStatistics(); bit ratio is decompressed-size/compressedsize.

• floatGetStatisticsCumulativeDecompressionRatio()

cumulative stats, can be reset with ResetCumulativeStatistics(); bit ratio is decompressed-size/compressedsize.

• floatGetStatisticsCumulativeCompressionExecutionTime()

cumulative stats, can be reset with ResetCumulativeStatistics(); time is in seconds.

• floatGetStatisticsCumulativeDecompressionExecutionTime()

cumulative stats, can be reset with ResetCumulativeStatistics(); time is in seconds.

• floatGetStatisticsCumulativeExecutionTime()

cumulative stats, can be reset with ResetCumulativeStatistics(); this returns the sum of compres-sion+decompression times; time is in seconds.

• int GetStatisticsCumulativeErrorCount()

cumulative stats, can be reset with ResetCumulativeStatistics(); number of errors since last reset.

• unsigned intGetStatisticsLastOperationCompressionBytesRead()

returns the statistics for the last operation.

• unsigned intGetStatisticsLastOperationDecompressionBytesRead()


• unsigned intGetStatisticsLastOperationCompressionBytesWritten()


• unsigned intGetStatisticsLastOperationDecompressionBytesWritten()




• unsigned intGetStatisticsLastOperationStateBytesRead()

returns the statistics for the last operation on state data.

• unsigned intGetStatisticsLastOperationStateBytesWritten()

returns the statistics for the last operation on state data.

• floatGetStatisticsLastOperationCompressionBitrate()

returns the statistics for the last operation; bit rate is in bits per bytes.

• floatGetStatisticsLastOperationDecompressionBitrate()

returns the statistics for the last operation; bit rate is in bits per bytes.

• floatGetStatisticsLastOperationCompressionRatio()

returns the statistics for the last operation; bit ratio is decompressedsize/compressedsize.

• floatGetStatisticsLastOperationDecompressionRatio()

returns the statistics for the last operation; bit ratio is decompressedsize/compressedsize.

• floatGetStatisticsLastOperationCompressionExecutionTime()

returns the statistics for the last operation; time is in seconds.

• floatGetStatisticsLastOperationDecompressionExecutionTime()

returns the statistics for the last operation; time is in seconds.

• floatGetStatisticsLastOperationExecutionTime()

returns the statistics for the last operation; this returns the sum of compression+decompression times; timeis in seconds.

• floatGetStatisticsLastOperationStateExecutionTime()

returns the statistics for the last state operation; time is in seconds.

• int GetStatisticsLastOperationErrorCount()

returns number of errors during the last operation.

• void Add LastOperationCompressionBytesRead(int val)

kludge fixup to manually adjust counts (useful for when we read a header through a separate call)

• void Add LastOperationCompressionBytesWritten(int val)


• void Add LastOperationDecompressionBytesRead(int val)


• void Add LastOperationDecompressionBytesWritten(int val)


• boolSaveCompressionHeader(bitwriter &to, bool savedecompressstateinfo)

Save header info for a compression stream (either supershort 1 byte, or long state info).

• boolLoadCompressionHeader(bitreader&from)



Load header info for a compression stream (either supershort 1 byte, or long state info).

• boolSaveGUID(bitwriter &to)

Save GUID field.

Static Public Member Functions

• int LoadGUID(bitreader&from)

Load GUID field.


• virtual boolDoProtectedCompress(bitreader&from, bitwriter &to)=0


• virtual boolDoProtectedDecompress(bitreader&from, bitwriter &to)=0


• virtual boolDoProtectedCreateSymbolsAndModelsUsingStream(bitreader&from)

This is the member function which does the actual training on an input in order to generateparsers/modelers/coders, and this is the function that should be subclasses by derived classes.


This is the member function which does the actual saving of state, and this is the function that should besubclasses by derived classes.


This is the member function which does the actual loading of state, and this is the function that should besubclasses by derived classes.


• std::stringinstantiatedname

name of instantiated instance of this class

• unsigned short intguid

unique short id for a compressor


8.19.2.1 virtual std::string OctaneCompressor::GetName () [inline, virtual]







References GetInstantiatedName().

55 {return GetInstantiatedName();};

8.19.2.2 virtual std::string OctaneCompressor::GetDescription () [inline, virtual]




Reimplemented inmtfllCompressor, SampleCompressor, SampleStatCompressor, OctaneCompressor-Statistical, SubStrHuffCompressor, andzleCompressor.


Referenced by OctaneTester::ListCompressors(), and OctaneTester::ShowHelp().

56 {return "Base Compressor";};

8.19.2.3 virtual std::string OctaneCompressor::GetHelpInformation () [inline, virtual]




Reimplemented inOctaneCompressorStatistical.


57 {return "";};

8.19.2.4 virtual unsigned short int OctaneCompressor::GetCompressorStateFileGUID ()[inline, virtual]

Return a universally unique unsigned-short-int id# for a predefined dictionary id file.

This number is used to very compactly identify a preset compressor+statefile(dictionary) for use whentransmission bandwidth is at a premium, such as in an online chat program with very short transmission(<<80 characters per transmission).



Returns:0 when this compressor+configuration has no globally-registered GUID.


References guid.

Referenced by OctaneTester::ListCompressors().

62 {return guid;};

8.19.2.5 virtual bool OctaneCompressor::PrepareForCompression ()[inline, virtual]



Todois this a duplication of functionality inResetState()andSynchronizeStateForNewStream()?



90 {return true;};

8.19.2.6 virtual void OctaneCompressor::SetupAnyDefaultParser () [inline, virtual]

Some compressors contains parsers which begin ready to parse, without any ’training’.

Here is where such a parser should be initialized.



93 {;};

8.19.2.7 virtual bool OctaneCompressor::IsReadyToCompress ()[inline, virtual]



Reimplemented inSampleCompressor, andOctaneCompressorStatistical.


Referenced by OctaneTester::CheckCompressorReady().

96 {return true;};



8.19.2.8 bool OctaneCompressor::Decompress (bitreader & from, bitwriter & to, boolneedtoreadheader)

The Decompress function is an API wrapper around a protected function which does the actual compres-sion, in order to perform some timing/stats gathering functions.

Todowhat should we do on aLoadCompressionHeader()error?

Definition at line 77 of file compressor.cpp.

References bitwriter::alignbyte(), DoProtectedDecompress(), LoadCompressionHeader(), Octane-Timer::start(), OctaneTimer::stop(), bitwriter::tellbyte(), and bitreader::tellbyte().

Referenced by OctaneTester::RunCommandDecompress(), and OctaneTester::RunCommandTest-String().

78 {79 // just wrap the calls to derived classes80 bool bretv;81 OctaneTimer octanetimer;82 size_t fromstart,tostart;8384 // do decompression and measure it85 octanetimer.start();86 fromstart=from.tell_byte();87 tostart=to.tell_byte();8889 if (needtoreadheader)90 {91 // this decompress could be called before the header is read, or after it.92 // if the header is already read by caller, we wont try to re-read it here.93 // if it hasn’t been read yet, we can read it here, and it MUST match our current compressor.94 // load header - this is either a supershort GUID only (usually only 1 byte), or the complete stateinfo for the compressor95 bretv=LoadCompressionHeader(from);96 }9798 // do decompression99 bretv=DoProtectedDecompress(from,to);100101 // align output bitwriter (this must be performed when we finish)102 to.align_byte();103104 // finish measurements105 LastOperation_Decompression_ExecutionTime=octanetimer.stop();106 Cumulative_Decompression_ExecutionTime+=LastOperation_Decompression_ExecutionTime;107 LastOperation_Decompression_BytesRead=from.tell_byte()-fromstart;108 Cumulative_Decompression_BytesRead+=LastOperation_Decompression_BytesRead;109 LastOperation_Decompression_BytesWritten=to.tell_byte()-tostart;110 Cumulative_Decompression_BytesWritten+=LastOperation_Decompression_BytesWritten;111112 // update error counts113 if (!bretv)114 {115 ++Cumulative_ErrorCount;116 ++LastOperation_ErrorCount;117 }118119 // return success120 return bretv;121 }



8.19.2.9 bool OctaneCompressor::Save (bitwriter & to, bool fortempdecompressiononly)


This function also saves HEADER info which identifies the compressor class, which is read byCompressorManager


References DoProtectedSaveState(), GetClassName(), GetInstantiatedName(), guid, bitwriter::put(),OctaneTimer::start(), OctaneTimer::stop(), and bitwriter::tellbyte().

Referenced by OctaneTester::RunCommandSaveCompressor(), and SaveCompressionHeader().

152 {153 // wrap call to save state and measure it154 bool bretv;155 size_t tostart;156 OctaneTimer octanetimer;157158 // do state save and measure it159 octanetimer.start();160 tostart=to.tell_byte();161162 // save compressor header (see OctaneCompressor::Save() for where this is written)163 string tempstring=GetClassName();164 to.put(tempstring);165 tempstring=GetInstantiatedName();166 to.put(tempstring);167 to.put(guid);168169 // now call derived protected method to save state170 bretv=DoProtectedSaveState(to,fortempdecompressiononly);171172 // measure result173 LastOperation_State_BytesWritten=to.tell_byte()-tostart;174 LastOperation_State_ExecutionTime=octanetimer.stop();175176 // update error counts177 if (!bretv)178 {179 ++Cumulative_ErrorCount;180 ++LastOperation_ErrorCount;181 }182183 // return success184 return bretv;185 }

8.19.2.10 bool OctaneCompressor::Load (bitreader & from)

Normally the compressor manager loads the header of a saved file and dynamically instantiates a compres-sor, but we provide this function for manually calling load on an already instantiated compressor.

It reads the header, verifies it is for proper class, and then callsLoadState().


References bitreader::get(), GetClassName(), instantiatedname, and LoadState().

Referenced by LoadCompressionHeader().

189 {



190 // Normally the compressor manager loads the header of a saved file and dynamically instantiates a compressor,191 // but we provide this function for manually calling load on an already instantiated compressor.192 // It reads the header, verifies it is for proper class, and then calls LoadState().193 bool bretv;194195 // read standard compressor header (see OctaneCompressor::Save() for where this is written)196 string compressorclassname;197 string compressorname;198 short int compressorguid;199 from.get(compressorclassname);200 from.get(compressorname);201 from.get(compressorguid);202203 if (compressorclassname!=GetClassName())204 {205 cout << "Error: class name from file does not match compressor class name being read; load aborted."<<endl;206 return false;207 }208209 // set name but not guid210 if (instantiatedname=="")211 instantiatedname=compressorname;212213 // now load state214 bretv=LoadState(from);215 return bretv;216 }

8.19.2.11 virtual bool OctaneCompressor::DoProtectedCompress (bitreader & from, bitwriter &to) [protected, pure virtual]





Implemented inmtfllCompressor, SampleCompressor, OctaneCompressorStatistical, andzleCompressor.

Referenced by Compress().

8.19.2.12 virtual bool OctaneCompressor::DoProtectedDecompress (bitreader & from, bitwriter& to) [protected, pure virtual]







Implemented inmtfllCompressor, SampleCompressor, OctaneCompressorStatistical, andzleCompressor.

Referenced by Decompress().

8.19.2.13 virtual bool OctaneCompressor::DoProtectedCreateSymbolsAndModelsUsingStream(bitreader & from) [inline, protected, virtual]


It is wrapped by the public API callCreateSymbolsAndModelsUsingStream()which performs some mea-surements on speed and streamsize.





Referenced by CreateSymbolsAndModelsUsingStream().

142 {return true;};

8.19.2.14 virtual bool OctaneCompressor::DoProtectedSaveState (bitwriter & to, boolfortempdecompressiononly) [inline, protected, virtual]


It is wrapped by the public API callSaveState()which performs some measurements on speed and stream-size.






Referenced by Save().

148 {return true;};



8.19.2.15 virtual bool OctaneCompressor::DoProtectedLoadState (bitreader & from) [inline,protected, virtual]


It is wrapped by the public API callLoadState()which performs some measurements on speed and stream-size.






Referenced by LoadState().

154 {return true;};

8.19.2.16 int OctaneCompressor::GetStatisticsLastOperation ErrorCount () [inline]

returns number of errors during the last operation.

Todoi don’t think we currently reset this, so it may behave like cumulative error until we fix it.


217 {return LastOperation_ErrorCount;};

8.19.2.17 bool OctaneCompressor::SaveCompressionHeader (bitwriter & to, boolsavedecompressstateinfo)

Save header info for a compression stream (either supershort 1 byte, or long state info).

Note:this is kind of ugly, maybe we can find a better solution.


References bitwriter::putbyte(), Save(), and SaveGUID().

Referenced by Compress().

284 {285 if (savedecompressstateinfo)286 {287 // save the state information for the compressor prior to the data



288 // first write a special byte signifying that full state info follows289 to.put_byte((unsigned char)254);290 Save(to,true);291 }292 else293 {294 // save only the GUID of the compressor295 // if the GUID is valid, it should let receiver know which decompressor to use, or otherwise issue a warning296 SaveGUID(to);297 }298299 // return success300 return true;301 }

8.19.2.18 bool OctaneCompressor::SaveGUID (bitwriter & to)

Save GUID field.

Todo• we will add code here so that we use only 1 byte for most guid, and 2 bytes for others, but for

now we always use 1 byte for GUID


References guid, and bitwriter::putbyte().

Referenced by SaveCompressionHeader().

337 {338 // save a short guid339 to.put_byte((unsigned char)guid);340 // return success341 return true;342 }

8.19.2.19 int OctaneCompressor::LoadGUID (bitreader & from) [static]

Load GUID field.

Returns:the guid or -1 if its the NOGUID constant


References bitreader::getbyte().

Referenced by LoadCompressionHeader(), and OctaneTester::RunCommandDecompress().

347 {348 // load a short guid349 unsigned char c;350351 // read first character352 c=from.get_byte();353354 // special NONGUID character355 if (c==254)



356 return -1;357358 return c;359 }


• compressors/compressor.hpp• compressors/compressor.cpp



8.20 OctaneCompressorStatistical Class Reference

#include <compressor statistical.hpp >

Inheritance diagram for OctaneCompressorStatistical::


OctaneCompressor

OctaneClass

SampleStatCompressor SubStrHuffCompressor



Definition at line 39 of file compressorstatistical.hpp.


• OctaneCompressorStatistical(bool registerme=false)

constructor

• virtual∼OctaneCompressorStatistical()

destructor














8.20 OctaneCompressorStatistical Class Reference 109





• virtual boolSetParameter(const std::string &parametername, const std::string &parametervalue)• virtual voidResetState()

Reset the state of the compressor to its initial state.


Synchronize state for a new stream - this will be called each time before beginning a new parsing stream.

• virtual boolPrepareForCompression()


• virtual voidSetupAnyDefaultParser()• virtual boolIsReadyToCompress()



• virtual boolDoProtectedCompress(bitreader&from, bitwriter &to)


• virtual boolDoProtectedDecompress(bitreader&from, bitwriter &to)


• virtual boolDoProtectedCreateSymbolsAndModelsUsingStream(bitreader&from)• virtual boolDoProtectedSaveState(bitwriter &to, bool fortempdecompressiononly)




• OctaneParser∗ parserp

Pointer to the derived (and dynamically allocated) statistical parser.

• OctaneModeler∗ modelerp

Pointer to the derived (and dynamically allocated) statistical modeler.

• OctaneCoder∗ coderp

Pointer to the derived (and dynamically allocated) statistical coder.




8.20.2.1 virtual std::string OctaneCompressorStatistical::GetDescription () [inline,virtual]




Reimplemented inSampleStatCompressor, andSubStrHuffCompressor.

Definition at line 58 of file compressorstatistical.hpp.

58 {return "Statistical Compressor Base Class";}

8.20.2.2 string OctaneCompressorStatistical::GetHelpInformation () [virtual]




Definition at line 69 of file compressorstatistical.cpp.

References coderp, OctaneCoder::GetHelpInformation(), OctaneModeler::GetHelpInformation(), Octane-Parser::GetHelpInformation(), modelerp, parserp, and OctaneClass::PrefixNonEmptyString().

70 {71 // show help info from our parser,model,coder72 string helpinfo;7374 // generic help information for children75 helpinfo += PrefixNonEmptyString("Parser Help:\n-------------------------\n",parserp->GetHelpInformation());76 helpinfo += PrefixNonEmptyString("Modeler Help:\n-------------------------\n",modelerp->GetHelpInformation());77 helpinfo += PrefixNonEmptyString("Coder Help:\n-------------------------\n",coderp->GetHelpInformation());7879 // return amalgamated string80 return helpinfo;81 }

8.20.2.3 bool OctaneCompressorStatistical::SetParameter (const std::string &parametername,const std::string & parametervalue) [virtual]

Note:this is a little weird; there are two ways we could do this, we could use a prefix to indicate whetherthey want to set a parameter in parser/modeler/coder, or we could simply ask each in turn if they ownthe variable





References coderp, modelerp, parserp, and OctaneClass::SetParameter().

132 {135 bool bretv;136137 bretv=parserp->SetParameter(parametername,parametervalue);138 if (!bretv)139 bretv=modelerp->SetParameter(parametername,parametervalue);140 if (!bretv)141 bretv=coderp->SetParameter(parametername,parametervalue);142 return bretv;143 }

8.20.2.4 bool OctaneCompressorStatistical::PrepareForCompression () [virtual]



Todois this a duplication of functionality inResetState()andSynchronizeStateForNewStream()?



References coderp, modelerp, parserp, OctaneCoder::PrepareForCoding(), OctaneModeler::PrepareFor-Modeling(), and OctaneParser::PrepareForParsing().

234 {235 // prepare for parsing mode, must be called after zero or more Updating calls, and before parsing begins; returns true on success236 bool bretv;237238 // first prepare the parser (this should ensure we have a valid symbol set)239 bretv=parserp->PrepareForParsing();240 // now prepare the model (give it parser info in case it wants it)241 if (bretv)242 modelerp->PrepareForModeling(parserp);243 // now build coder based on modeler244 if (bretv)245 coderp->PrepareForCoding(modelerp);246247 // return success248 return bretv;249 }

8.20.2.5 void OctaneCompressorStatistical::SetupAnyDefaultParser () [virtual]

now tell the coder that all the weights on symbols have changed





References coderp, OctaneModeler::CreateModelUsingParser(), OctaneModeler::IsReadyToModel(),OctaneParser::IsReadyToParse(), modelerp, parserp, and OctaneCoder::ReceiveNotificationModel-ChangeAllSymbolWeights().

Referenced by SampleStatCompressor::SampleStatCompressor(), and SubStrHuffCompressor::SubStr-HuffCompressor().

188 {189 // if a parser starts out ready to parse, we might need to build a model/coder from it190 if (parserp->IsReadyToParse())191 {192 modelerp->CreateModelUsingParser(parserp);193 if (modelerp->IsReadyToModel())194 {196 coderp->ReceiveNotification_ModelChange_AllSymbolWeights(modelerp);197 }198 }199 }

8.20.2.6 bool OctaneCompressorStatistical::IsReadyToCompress () [virtual]





References coderp, OctaneCoder::IsReadyToCode(), OctaneModeler::IsReadyToModel(), Octane-Parser::IsReadyToParse(), modelerp, and parserp.

252 {253 // is the compressor ready to compress? Normally this is false until a dictionary is built or loaded.254 bool bretv1,bretv2,bretv3;255256 if (!(bretv1=parserp->IsReadyToParse()))257 cout << "the parser is not ready."<<endl;258 if (!(bretv2=modelerp->IsReadyToModel()))259 cout << "the modeler is not ready."<<endl;260 if (!(bretv3=coderp->IsReadyToCode()))261 cout << "the coder is not ready."<<endl;262 return (bretv1 && bretv2 && bretv3);263 }

8.20.2.7 bool OctaneCompressorStatistical::DoProtectedCompress (bitreader & from, bitwriter& to) [protected, virtual]









References OctaneCoder::AlignOutputStreamToByteBoundry(), coderp, modelerp, OctaneParser::Parse-NextSymbolFromInput(), parserp, SynchronizeStateForNewStream(), OctaneModeler::UpdateModel-AfterReceivingSymbol(), and OctaneCoder::WriteSymbolBits().

281 {282 // compress all of from and send to to283 // ATTN: the only thing slightly troubling about this otherwise elegant routine, is the284 // fact that there is no way for the components to change the symbol set during compression285 bool bretv=true;286 int symbolnumber;287288 // we *must* inform our components that we are about to compress or decompress a new stream289 SynchronizeStateForNewStream();290291 // loop parsing a symbol from parser until we hit end (note the last symbol will be an End-Of-Stream)292 while (parserp->ParseNextSymbolFromInput(from,symbolnumber))293 {294 // debug show some info?295 // cout << "$READ SYMBOL "<<symbolnumber<<": ’"<<parserp->LookupSymbolText(symbolnumber)<<"’."<<endl;296 // code the symbol given current model and codes297 if (!(bretv=coderp->WriteSymbolBits(symbolnumber,to)))298 break;299 // now update the model based on this symbol (pass coderp so it can inform it if probabilities change).300 // note that for four zero-order model this doesn’t do anything, but we call it for completeness.301 if (!(bretv=modelerp->UpdateModelAfterReceivingSymbol(symbolnumber,coderp)))302 break;303 }304305 // when we finish coding a stream we want to align to byte boundary in the output306 // this will happen automatically if we close the output stream now, but we do it manually in case we want to pack multiple307 // input streams into the same output.308 coderp->AlignOutputStreamToByteBoundry(to);309310 // return success311 return bretv;312 }

8.20.2.8 bool OctaneCompressorStatistical::DoProtectedDecompress (bitreader & from, bitwriter& to) [protected, virtual]









References OctaneCoder::AlignInputStreamToByteBoundry(), coderp, OctaneCoder::DecodeSymbol-FromInput(), modelerp, parserp, SynchronizeStateForNewStream(), OctaneModeler::UpdateModelAfter-ReceivingSymbol(), and OctaneParser::WriteSymbolText().

315 {316 // decompress all of from and send to to317 // ATTN: the only thing slightly troubling about this otherwise elegant routine, is the318 // fact that there is no way for the components to change the symbol set during compression319 bool bretv=true;320 int symbolnumber;321 bool isendofstreamsymbol;322323 // we *must* inform our components that we are about to compress or decompress a new stream324 SynchronizeStateForNewStream();325326 // loop decoding a symbol from coder until we hit end327 while (coderp->DecodeSymbolFromInput(symbolnumber,from))328 {329 // debug show some info?330 // cout << "$DECODED SYMBOL "<<symbolnumber<<": ’"<<parserp->LookupSymbolText(symbolnumber)<<"’."<<endl;331 // deParse the symbol given current model and symbol set332 if (!(bretv=parserp->WriteSymbolText(to,symbolnumber,isendofstreamsymbol)))333 break;334 // did we just parse an EOS(end-of-stream) symbol? if so we need to align input to byte boundary (normally its end of file)335 if (isendofstreamsymbol)336 {337 // when we finish coding a stream we want to align to byte boundary in the output338 // this will happen automatically if we close the output stream now, but we do it manually in case we want to pack multiple339 // input streams into the same output. note that this we will almost always be leaving this loop after this symbol.340 coderp->AlignInputStreamToByteBoundry(from);341 // cout << "$DEBUG aligning input."<<endl;342 }343 // now update the model based on this symbol (pass coderp so it can inform it if probabilities change).344 // note that for four zero-order model this doesn’t do anything, but we call it for completeness.345 if (!(bretv=modelerp->UpdateModelAfterReceivingSymbol(symbolnumber,coderp)))346 break;347 }348349 // return success350 return bretv;351 }

8.20.2.9 bool OctaneCompressorStatistical::DoProtectedCreateSymbolsAndModelsUsingStream(bitreader & from) [protected, virtual]

now tell the coder that all the weights on symbols have changed



References coderp, OctaneModeler::CreateModelUsingStream(), OctaneParser::CreateSymbolSetUsing-Stream(), modelerp, parserp, OctaneCoder::ReceiveNotificationModelChangeAllSymbolWeights(), bi-treader::seekbit(), and bitreader::tellbit().

203 {204 // optionally (pre)process an input stream to update/create a symbol set from it; returns true on success205 bool bretv;206 // build parser symbol set using stream207 // HOWEVER this can be a bit tricky, because we have 2 structures (parser and modeler) which may want to process from stream208 // so we need to rewind it after the parser gets done with it.



209210 // record starting position of from stream (probably 0)211 size_t fromstartpos=from.tell_bit();212213 // give it to parser214 bretv = parserp->CreateSymbolSetUsingStream(from);215 // now build the model using the symbols created by the parser, and again on the stream from216 if (bretv)217 {218 // reset from stream to start219 from.seek_bit(fromstartpos);220 // now have model process it221 bretv = modelerp->CreateModelUsingStream(parserp,from);222 if (bretv)223 {225 coderp->ReceiveNotification_ModelChange_AllSymbolWeights(modelerp);226 }227 }228229 // return success230 return bretv;231 }

8.20.2.10 bool OctaneCompressorStatistical::DoProtectedSaveState (bitwriter & to, boolfortempdecompressiononly) [protected, virtual]







References coderp, modelerp, parserp, and OctaneClass::SaveState().

146 {147 // save state; returns true on success148 bool bretv;149 bretv=parserp->SaveState(to,fortempdecompressiononly);150 if (bretv)151 bretv=modelerp->SaveState(to,fortempdecompressiononly);152 if (bretv)153 bretv=coderp->SaveState(to,fortempdecompressiononly);154 return bretv;155 }

8.20.2.11 bool OctaneCompressorStatistical::DoProtectedLoadState (bitreader & from)[protected, virtual]

now tell the coder that the modeler (the weights on symbols) have changed





References coderp, OctaneClass::LoadState(), modelerp, parserp, and OctaneCoder::ReceiveNotification-ModelChangeAllSymbolWeights().

158 {159 // save state; returns true on success160 bool bretv;161 bretv=parserp->LoadState(from);162 if (bretv)163 bretv=modelerp->LoadState(from);164 if (bretv)165 {166 bretv=coderp->LoadState(from);168 coderp->ReceiveNotification_ModelChange_AllSymbolWeights(modelerp);169 }170 return bretv;171 }


• compressors/compressorstatistical.hpp• compressors/compressorstatistical.cpp


8.21 OctaneModeler Class Reference 117

8.21 OctaneModeler Class Reference

#include <modeler.hpp >

Inheritance diagram for OctaneModeler::

OctaneModeler

OctaneClass

OctaneModeler_WeightVectored

SampleModeler ZeroOrderModeler



It’s job is to continually maintain a probability distribution over the next symbol to be parsed. This proba-bility information can be used by a coder (like an arithmetic or huffman coder) to optimally code the parsedsymbol.

Definition at line 51 of file modeler.hpp.


• OctaneModeler()

constructor

• virtual∼OctaneModeler()

destructor







• virtual voidResetState()• virtual boolIsReadyToModel()

Are we ready to work? this is usually only true after the model has been trained.

• virtual voidSynchronizeStateForNewStream()• virtual boolCreateModelUsingStream(OctaneParser∗parserp,bitreader&from)

Build a model using a stream and a parser.



• virtual boolCreateModelUsingParser(OctaneParser∗parserp)

Build a model just using parser.

• virtual boolPrepareForModeling(OctaneParser∗parserp)

Prepare for modeling mode given a trained parser.

• virtual boolUpdateModelAfterReceivingSymbol(int symbolnumber,OctaneCoder∗coderp)

Update our model after each symbol is received during compression and decompression.


Return the number of symbols in the modeler (should be same as number of symbols in the parser); calledby coder.

• virtual SymbolWeightVector∗ GetWeightVectorp()

Return a pointer to the current weight vector for symbols (used by coder) [optional] This is provided as ashortcut function to avoid having to make numerous calls toGetWeightVectorItem()below.

• virtual TSymbolWeightVectorWeightGetWeightVectorItem(int symbolnum)

Return a specific weight index value; (used by coder).


8.21.2.1 virtual std::string OctaneModeler::GetName () [inline, virtual]




Reimplemented inSampleModeler, OctaneModelerWeightVectored, andZeroOrderModeler.


61 {return "OctaneModeler";}

8.21.2.2 virtual std::string OctaneModeler::GetDescription () [inline, virtual]




Reimplemented inSampleModeler, OctaneModelerWeightVectored, andZeroOrderModeler.


62 {return "Base Modeler";}



8.21.2.3 virtual std::string OctaneModeler::GetHelpInformation () [inline, virtual]






63 {return "";}

8.21.2.4 virtual bool OctaneModeler::IsReadyToModel () [inline, virtual]


Returns:true if the modeler is ready to begin predicting probabilities.

Reimplemented inOctaneModelerWeightVectored.


Referenced by OctaneCompressorStatistical::IsReadyToCompress(), and OctaneCompressor-Statistical::SetupAnyDefaultParser().

71 {return true;};

8.21.2.5 virtual bool OctaneModeler::CreateModelUsingStream (OctaneParser∗ parserp,bitreader & from) [inline, virtual]



Reimplemented inSampleModeler, andZeroOrderModeler.


Referenced by OctaneCompressorStatistical::DoProtectedCreateSymbolsAndModelsUsingStream().

75 {return true;};



8.21.2.6 virtual bool OctaneModeler::CreateModelUsingParser (OctaneParser∗ parserp)[inline, virtual]


This is called on startup, before any possible training, when a default parser exists.

Note:that no input (training) stream is provider; some modelers may ignore this.


Reimplemented inSampleModeler.


Referenced by OctaneCompressorStatistical::SetupAnyDefaultParser().

80 {return false;};

8.21.2.7 virtual bool OctaneModeler::PrepareForModeling (OctaneParser∗ parserp) [inline,virtual]

Prepare for modeling mode given a trained parser.




83 {return true;};

8.21.2.8 virtual bool OctaneModeler::UpdateModelAfterReceivingSymbol (intsymbolnumber,OctaneCoder∗ coderp) [inline, virtual]

Update our model after each symbol is received during compression and decompression.

Modelers track symbols during parsing and update the probability distribution predictions for upcomingsymbols. This function should notify the coder of changes to probabilities.

See also:OctaneCoder::ReceiveNotificationModelChangeAllSymbolWeightsOctaneCoder::ReceiveNotificationModelChangeSingleSymbolWeight



Referenced by OctaneCompressorStatistical::DoProtectedCompress(), and OctaneCompressor-Statistical::DoProtectedDecompress().

90 {return true;};



8.21.2.9 virtual SymbolWeightVector∗ OctaneModeler::GetWeightVectorp () [inline,virtual]


Not all modelers will maintain probabilities in the form of aSymbolWeightVector, and a modeler can returnNULL in order to force the coder to get probabilities for each symbol individually.

See also:OctaneModeler::GetWeightVectorItem

Returns:a pointer to a localSymbolWeightVectoror NULL if it is not available.

Reimplemented inOctaneModelerWeightVectored.


103 {return NULL;};


• modelers/modeler.hpp



8.22 OctaneModelerWeightVectored Class Reference

#include <modeler weightvectored.hpp >

Inheritance diagram for OctaneModelerWeightVectored::


OctaneModeler

OctaneClass

SampleModeler ZeroOrderModeler


This is a derived modeler class which has some default functions for modelers which work with a vectorof weights as probabilities (very common).

Definition at line 33 of file modelerweightvectored.hpp.


• OctaneModelerWeightVectored()

constructor

• virtual∼OctaneModelerWeightVectored()

destructor














8.22 OctaneModelerWeightVectored Class Reference 123



• virtual boolIsReadyToModel()


• virtual voidResetState()• virtual int GetSymbolCount()

Return the number of symbols in the modeler (should be same as number of symbols in the parser); calledby coder.

• virtual SymbolWeightVector∗ GetWeightVectorp()


• virtual TSymbolWeightVectorWeightGetWeightVectorItem(int symbolnum)

Return a specific weight index value; (used by coder).


• SymbolWeightVectorweightvector

vector based data structure for computing and storing frequencies


8.22.2.1 virtual std::string OctaneModeler WeightVectored::GetName () [inline,virtual]



Reimplemented fromOctaneModeler.



46 {return "OctaneModelerWeightVectored";}

8.22.2.2 virtual std::string OctaneModeler WeightVectored::GetDescription () [inline,virtual]








47 {return "OctaneModeler using a WeightVector";}

8.22.2.3 bool OctaneModelerWeightVectored::SaveState (bitwriter & to, boolfortempdecompressiononly) [virtual]




Definition at line 34 of file modelerweightvectored.cpp.

References SymbolWeightVector::SaveState(), and weightvector.

35 {36 // save state37 // return true on success38 return weightvector.SaveState(to,fortempdecompressiononly);39 }

8.22.2.4 bool OctaneModelerWeightVectored::LoadState (bitreader & from) [virtual]




Definition at line 41 of file modelerweightvectored.cpp.

References SymbolWeightVector::LoadState(), and weightvector.

42 {43 // load state44 // return true on success45 return weightvector.LoadState(from);46 }

8.22.2.5 virtual bool OctaneModelerWeightVectored::IsReadyToModel () [inline,virtual]


Returns:true if the modeler is ready to begin predicting probabilities.


8.22 OctaneModelerWeightVectored Class Reference 125



References SymbolWeightVector::GetSymbolCount(), and weightvector.

61 {if (weightvector.GetSymbolCount()>0) return true; else return false;};

8.22.2.6 virtualSymbolWeightVector∗ OctaneModeler WeightVectored::GetWeightVectorp ()[inline, virtual]


Not all modelers will maintain probabilities in the form of aSymbolWeightVector, and a modeler can returnNULL in order to force the coder to get probabilities for each symbol individually.

See also:OctaneModeler::GetWeightVectorItem

Returns:a pointer to a localSymbolWeightVectoror NULL if it is not available.



References weightvector.

68 {return &weightvector;};


• modelers/modelerweightvectored.hpp• modelers/modelerweightvectored.cpp



8.23 OctaneParser Class Reference

#include <parser.hpp >

Inheritance diagram for OctaneParser::

OctaneParser

OctaneClass

BitParser SampleParser SubstringParser



Parsers are commonly just character-based, returning the ascii character for each symbol parsed, and there-fore require no training. But some parsers can be more sophisticated, scanning input streams and buildinga list of all words or common words, etc.

Definition at line 48 of file parser.hpp.


• OctaneParser()

constructor

• virtual∼OctaneParser()

destructor







• virtual boolResetState()• virtual boolCreateSymbolSetUsingStream(bitreader&from)


• virtual boolPrepareForParsing()

Prepare for parsing mode; must be called after a CreateSymbol call, and before parsing begins.



8.23 OctaneParser Class Reference 127






• virtual int GetSymbolCount()=0


• virtual boolParseNextSymbolFromInput(bitreader&from, int &symbolnum)=0


• virtual boolWriteSymbolText(bitwriter &to, int symbolnum, bool &isendofstreamsymbol)=0


• virtual stringLookupSymbolText(int symbolnum)=0


Static Protected Member Functions

• bool IsNonWordCharacter(unsigned char c)

Static helper function to decide whether c is a character found in words, or should be treated as a non-wordcharacter.


8.23.2.1 virtual std::string OctaneParser::GetName () [inline, virtual]





58 {return "OctaneParser";}

8.23.2.2 virtual std::string OctaneParser::GetDescription () [inline, virtual]







59 {return "Base Parser Class";}

8.23.2.3 virtual std::string OctaneParser::GetHelpInformation () [inline, virtual]






60 { return ""; }

8.23.2.4 virtual bool OctaneParser::CreateSymbolSetUsingStream (bitreader & from)[inline, virtual]



Reimplemented inBitParser, SampleParser, andSubstringParser.


Referenced by OctaneCompressorStatistical::DoProtectedCreateSymbolsAndModelsUsingStream().

68 {return true;};

8.23.2.5 virtual bool OctaneParser::PrepareForParsing () [inline, virtual]

Prepare for parsing mode; must be called after a CreateSymbol call, and before parsing begins.




71 {return true;};


8.23 OctaneParser Class Reference 129

8.23.2.6 virtual bool OctaneParser::RewindAnyBufferedInput (bitreader & from) [inline,virtual]



Reimplemented inBitParser, SampleParser, andSubstringParser.


77 {return true;};

8.23.2.7 virtual bool OctaneParser::ParseNextSymbolFromInput (bitreader & from, int &symbolnum) [pure virtual]




Implemented inBitParser, SampleParser, andSubstringParser.

Referenced by SymbolWeightVector::CountSymbolFrequencies(), and OctaneCompressorStatistical::Do-ProtectedCompress().

8.23.2.8 virtual bool OctaneParser::WriteSymbolText (bitwriter & to, int symbolnum, bool &isendofstreamsymbol) [pure virtual]





8.23.2.9 virtual string OctaneParser::LookupSymbolText (intsymbolnum) [pure virtual]







8.23.2.10 bool OctaneParser::IsNonWordCharacter (unsigned charc) [static,protected]

Static helper function to decide whether c is a character found in words, or should be treated as a non-wordcharacter.

used by some parsers to differentiate between word characters and separators.

Definition at line 17 of file parser.cpp.

18 {19 // return true if c is a non-word character, ie a delimiter between words20 // ATTN: this is a pretty inefficient function; we could use a static lookup table if we wanted to do this fast21 if (c==39)22 {23 // apostrophe24 return false;25 }26 else if (c<48)27 {28 // punctuation and nonprintables29 return true;30 }31 else if (c<58)32 {33 // digits34 return false;35 }36 else if (c<65)37 {38 // punctuation39 return true;40 }41 else if (c<91)42 {43 // uppercase44 return false;45 }46 else if (c<97)47 {48 // punctuation49 return true;50 }51 else if (c<123)52 {53 // lowercase54 return false;55 }56 else if (c<154)57 {58 // punctuation and nonprintables59 return true;60 }61 else62 {63 // ATTN: we’ve got some non-english characters here which mind form characters in other languages64 // for now we treat as word separators65 return true;66 }67 }


• parsers/parser.hpp• parsers/parser.cpp


8.24 OctaneScopeTimer Class Reference 131

8.24 OctaneScopeTimer Class Reference

#include <scope timer.hpp >



Definition at line 27 of file scopetimer.hpp.


• OctaneScopeTimer(const std::stringpre=”< scopetimer >”, const std::stringpost=”seconds\n”,std::ostream &to=std::cout)

Constructor.

• floatget elapsed()

Get the elapsed time since construction.

• ∼OctaneScopeTimer()

Destructor.


8.24.2.1 OctaneScopeTimer::OctaneScopeTimer (const std::stringpre= ”<scopetimer> ”, conststd::string post= ” seconds\n”, std::ostream & to = std::cout) [inline]

Constructor.

Parameters:pre - string to display before time.

post - string to display after time.

to - stream for output.


37 : pre(_pre), post(_post), to(_to)38 { ; }

8.24.2.2 OctaneScopeTimer::∼OctaneScopeTimer() [inline]

Destructor.

Outputs the elapsed time along with the pre- and post messages.


References OctaneTimer::stop().

46 { to << pre << t.stop() << post; }




8.24.3.1 float OctaneScopeTimer::getelapsed () [inline]

Get the elapsed time since construction.

Returns:elapsed time.


References OctaneTimer::stop().

42 { return t.stop(); }


• utilityclasses/timer/scopetimer.hpp


8.25 OctaneTester Class Reference 133

8.25 OctaneTester Class Reference

#include <octanetester.hpp >

Inheritance diagram for OctaneTester::

OctaneTester

OctaneClass


The OctaneTester class contains functions for testing compressors and providing a commandline and inter-active interface to octane functions.

Definition at line 32 of file octanetester.hpp.


• OctaneTester()

Constructor.

• ∼OctaneTester()

Destructor.

• void ShowHelp()

Show user help about how to use interactive commands within testloop; this is not to be customized forderived classes.

• void TestLoop()

Built in test loop that just asks user for input and compresses and decompresses in debug mode and lets userload and save files.

• boolParseCommand(const std::string &inputstr)

Parse an interactive command or a commandline statement.

• boolExecuteCommandsFromStream(std::ifstream &from, bool quiet)

Run a stream (input file) full of commands.

• boolExecuteCommandsFromFile(const std::string &filename, bool quiet)

Run a file full of commands.


• boolGrabLineFromConsole(std::string &line)

Grab a line from cin and return true on success.



• void ListCompressors(bool briefmode)

List all registered compressors.

• OctaneCompressor∗ SelectInstantiatedCompressor(const std::string &compressorname)

Select a compressor for subsequent operations; sets current compressorp in the process.

• boolCheckCompressorSelected(OctaneCompressor∗compressorp)

Helper function during parsing, to verify that the current compressor is ready to compress and decompressshow warning and return false if not.

• boolCheckCompressorReady(OctaneCompressor∗compressorp)


• std::stringMakeAssumedOutputFilename(const std::string &origname, const std::string &origext,const std::string &newext)

helper function to smartly add an extension to a file name, based on current compressor

• boolRunCommandTestString(std::string inputstring)

Run the TestString command, to compress and then decompress a string and check for errors.

• boolRunCommandTestFile(std::string filename)

Run the TestFile command, to compress and then decompress a file and check for errors.

• bool RunCommandCreateCompressor(std::string compressorclassname, std::string compressor-name, unsigned short guidnum)

Run the CreateCompressor command, to create a new instantiated compressor.

• boolRunCommandLoadCompressor(std::string filename)

Run the LoadCompressor command, to recall saved state space.

• boolRunCommandSaveCompressor(std::string filename)

Run the SaveCompressor command, to recall saved state space.

• boolRunCommandDeleteCompressor(std::string compressoridstring)

Run the DeleteCompressor command, to delete an instantiated compressor.

• boolRunCommandMakeState(std::string infilename)

Run the MakeState command, to create state info (i.e.

• bool RunCommandCompress(std::string infilename, std::string outfilename, bool savedecom-pressstateinfo)

Run the Compress command, to compress a file and write result to another file.

• bool RunCommandDecompress(std::string infilename, std::string outfilename, bool savedecom-pressstateinfo)

Run the Decompress command, to decompress a file and write result to another file.

• boolDeleteAFile(std::string filename)

Helper function, delete a file if it exists, or do nothing if not.



• boolCheckFilesAreIdentical(std::string filename1, std::string filename2)

Check two files and see if they are identical.


• OctaneCompressor∗ currentcompressorp

Currently selected Compressor class for interactive testing.


8.25.2.1 bool OctaneTester::ParseCommand (const std::string &inputstr)

Parse an interactive command or a commandline statement.

Returns:true on successful parse of command

Definition at line 117 of file octanetester.cpp.

References CheckCompressorReady(), CheckCompressorSelected(), currentcompressorp, Execute-CommandsFromFile(), stringparser::gettoken(), OctaneCompressor::GetInstantiatedName(), Octane-Class::GetParametersInformation(), OctaneCompressor::GetStatisticsLastOperationStateExecution-Time(), ListCompressors(), MakeAssumedOutputFilename(), RunCommandCompress(), Run-CommandCreateCompressor(), RunCommandDecompress(), RunCommandDeleteCompressor(),RunCommandLoadCompressor(), RunCommandMakeState(), RunCommandSaveCompressor(),RunCommandTestFile(), RunCommandTestString(), SelectInstantiatedCompressor(), Octane-Compressor::SetInfo(), OctaneClass::SetParameter(), OctaneClass::ShowDebuggingInfo(), and Show-Help().

Referenced by ExecuteCommandsFromStream(), and TestLoop().

118 {119 // process a single command, called from commandline or TestLoop above120 bool bretv=false;121122 // return if no command provided123 if (inputstr.empty())124 return false;125126 // create parser for string127 stringparser parser(inputstr);128 // grab main command keyword129 string command(parser.get_token());130131 // NOW PARSE COMMAND132133 if (command == "version")134 {135 // version info136 cout << OCTANE_VERSIONSTRING << endl;137 return true;138 }139 else if (command == "help")140 {



141 // show help information142 ShowHelp();143 return true;144 }145 else if (command == "debug")146 {147 // tell compressor to display any debug info it can148 if (!CheckCompressorSelected(currentcompressorp))149 return false;150 currentcompressorp->ShowDebuggingInfo();151 return true;152 }153 else if (command == "compressors")154 {155 // list registered compressors156 ListCompressors(false);157 return true;158 }159 else if (command == "parameters")160 {161 // list available parameters162 if (!CheckCompressorSelected(currentcompressorp))163 return false;164 cout << currentcompressorp->GetParametersInformation();165 return true;166 }167 else if (command == "select")168 {169 // select a registered compressor170 string param(parser.get_token());171 if (param == "")172 {173 cout << "you must specify the instantiated compressor to select."<<endl;174 return false;175 }176 if (SelectInstantiatedCompressor(param)==NULL)177 {178 cout << "error: compressor not found; type ’compressors’ for a list of registered compressors."<<endl;179 return false;180 }181 cout << "compressor "<< currentcompressorp->GetInstantiatedName() << " selected."<<endl;182 return true;183 }184 else if (command == "run")185 {186 // run octane commands from a file187 string param(parser.get_token());188 if (param == "")189 {190 cout << "you must specify the filename to run."<<endl;191 return false;192 }193 return ExecuteCommandsFromFile(param,false);194 }195 else if (command == "set")196 {197 // set a parameter198 if (!CheckCompressorSelected(currentcompressorp))199 return false;200 string param1(parser.get_token());201 if (param1 == "")202 {203 cout << "you must specify the parameter to set."<<endl;204 return false;205 }206 string param2(parser.get_token());207 if (param2 == "")



208 {209 cout << "you must specify a value for the parameter."<<endl;210 return false;211 }212 bretv = currentcompressorp->SetParameter(param1, param2);213 if (bretv)214 cout << "ok."<<endl;215 else216 cout << "error setting parameter or parameter not known."<<endl;217 return bretv;218 }219220 else if (command == "load")221 {222 // load previously saved compressor223 // note that this operation creates a new compressor, so we don’t need to insist that an instantiated compressor224 string param(parser.get_token());225 if (param == "")226 {227 cout << "you must specify the filename to load the compressor from."<<endl;228 return false;229 }230 // run the command231 return RunCommand_LoadCompressor(param);232 }233 else if (command == "save")234 {235 // load previously saved state236 if (!CheckCompressorSelected(currentcompressorp))237 return false;238 string param(parser.get_token());239 if (param == "")240 {241 cout << "you must specify the filename to save the compressor to."<<endl;242 return false;243 }244 // run the command245 return RunCommand_SaveCompressor(param);246 }247 else if (command == "create")248 {249 // create a new compressor based on a compressor class250 string param(parser.get_token());251 if (param == "")252 {253 cout << "you must specify the base class of the compressor to build."<<endl;254 return false;255 }256257 // optional name258 string param2(parser.get_token());259260 // optional guid261 string param3(parser.get_token());262 short int guid=0;263 if (param3!="")264 guid=atoi(param3.c_str());265266 // run the command267 bretv=RunCommand_CreateCompressor(param,param2,guid);268 if (bretv)269 cout << "compressor "<< currentcompressorp->GetInstantiatedName() << " created and selected."<<endl;270 return bretv;271 }272273 else if (command == "delete")274 {



275 // load previously saved state276 string param(parser.get_token());277 if (param == "")278 {279 cout << "you must specify the instantiated compressor to delete."<<endl;280 return false;281 }282 // run the command283 bool bretv=RunCommand_DeleteCompressor(param);284 if (bretv)285 {286 cout << "compressor deleted."<<endl;287 return true;288 }289 }290291 else if (command == "setinfo")292 {293 // set name and guid for a compressor294 if (!CheckCompressorSelected(currentcompressorp))295 return false;296 string param(parser.get_token());297 if (param == "")298 {299 cout << "you must specify a name for the compressor."<<endl;300 return false;301 }302 string param2(parser.get_token());303 // set the values304 unsigned short int guidnum=0;305 if (param2!="")306 guidnum=atoi(param2.c_str());307 currentcompressorp->SetInfo(param,guidnum);308 cout << "compresser info changed."<<endl;309 return true;310 }311312 else if (command == "makestate")313 {314 // make state information (i.e. build dictionary etc) using a file as training input315 if (!CheckCompressorSelected(currentcompressorp))316 return false;317 string param1(parser.get_token());318 if (param1 == "")319 {320 cout << "you must specify the filename to use for input."<<endl;321 return false;322 }323 // run command324 bool bretv=RunCommand_MakeState(param1);325 cout << " execution time: "<< currentcompressorp->GetStatistics_LastOperation_State_ExecutionTime()<<" seconds."<<endl;326 return bretv;327 }328329 else if (command == "compress")330 {331 // compress a file and write compressor info to front332 if (!CheckCompressorReady(currentcompressorp))333 return false;334 string param1(parser.get_token());335 if (param1 == "")336 {337 cout << "you must specify the filename to use."<<endl;338 return false;339 }340 string param2(parser.get_token());341 if (param2 == "")



342 {343 // assume an output filename344 param2 = MakeAssumedOutputFilename(param1, "decompressed", "compressed");345 }346 // run command (when savedecompresstateinfo==true it means write compressor state before raw data)347 bool savedecompressstateinfo=true;348 return RunCommand_Compress(param1,param2, savedecompressstateinfo);349 }350 else if (command == "decompress")351 {352 // decompress a file and expect compressor state info at front353 // we don’t need an existing compressor for this decompress, since we expect full compressor state at head354 // if (!CheckCompressorReady(currentcompressorp))355 // return false;356 string param1(parser.get_token());357 if (param1 == "")358 {359 cout << "you must specify the filename to use."<<endl;360 return false;361 }362 string param2(parser.get_token());363 if (param2 == "")364 {365 // assume an output filename366 param2 = MakeAssumedOutputFilename(param1, "compressed", "decompressed");367 }368 // run command (when savedecompresstateinfo==false it means only write compressed data)369 bool savedecompressstateinfo=true;370 return RunCommand_Decompress(param1,param2,savedecompressstateinfo);371 }372 else if (command == "compressr")373 {374 // compress a raw file without state info375 if (!CheckCompressorReady(currentcompressorp))376 return false;377 string param1(parser.get_token());378 if (param1 == "")379 {380 cout << "you must specify the filename to use."<<endl;381 return false;382 }383 string param2(parser.get_token());384 if (param2 == "")385 {386 // assume an output filename387 param2 = MakeAssumedOutputFilename(param1, "decompressed", "compressed");388 }389 // run command (when savedecompresstateinfo==false it means only write compressed data)390 bool savedecompressstateinfo=false;391 return RunCommand_Compress(param1,param2, savedecompressstateinfo);392 }393 else if (command == "decompressr")394 {395 // decompress a raw file, without compressor info396 if (!CheckCompressorReady(currentcompressorp))397 return false;398 string param1(parser.get_token());399 if (param1 == "")400 {401 cout << "you must specify the filename to use."<<endl;402 return false;403 }404 string param2(parser.get_token());405 if (param2 == "")406 {407 // assume an output filename408 param2 = MakeAssumedOutputFilename(param1, "compressed", "decompressed");



409 }410 // run command (when savedecompresstateinfo==false it means only write compressed data)411 bool savedecompressstateinfo=false;412 return RunCommand_Decompress(param1,param2,savedecompressstateinfo);413 }414415 else if (command == "testfile")416 {417 // compress and decompress a text string as a test418 if (!CheckCompressorReady(currentcompressorp))419 return false;420 string param1(parser.get_token());421 if (param1 == "")422 {423 cout << "you must specify the file to test compress."<<endl;424 return false;425 }426427 // run command428 return RunCommand_TestFile(param1);429 }430 else if (command == "test")431 {432 // compress and decompress a text string as a test433 if (!CheckCompressorReady(currentcompressorp))434 return false;435 if (inputstr.length() <= 5)436 {437 cout << "error: you need to specify text to compress."<<endl;438 return false;439 }440 // get text to compress441 string inputstring(inputstr.substr(5));442 // run command443 return RunCommand_TestString(inputstring);444 }445446 // if we get here it means the command was not understood447 cout << "command not understood(" << inputstr << "); type ’help’ for a list of known commands."<<endl;448 return false;449 }

8.25.2.2 bool OctaneTester::ExecuteCommandsFromStream (std::ifstream &from, bool quiet)

Run a stream (input file) full of commands.

Returns:true on no errors encountered during processing.


References ParseCommand(), and OctaneTimer::stop().

Referenced by ExecuteCommandsFromFile().

452 {453 // run a stream full of commands and time them454 // return true on success455 bool bretv=true;456457 // record start time458 OctaneTimer octanetimer;459



460 // parse stream461 string line;462 int linenum=1;463 while (getline(from, line))464 {465 // skip comments and blank lines466 string::size_type i = line.find_first_of("\015\012");467 if (i != string::npos)468 line.erase(i);469 if (line.length() < 3)470 continue;471 if (line[0] == ’/’ && line[1] == ’/’)472 continue;473 // show debugging info474 if (!quiet)475 cout << "]" << line << ""<<endl;476 // run it and stop on exit477 if (!(bretv=ParseCommand(line)))478 break;479 ++linenum;480 }481482 if (!quiet)483 {484 if (bretv)485 cout << "Script finished successfully after "<<octanetimer.stop()<<" seconds."<<endl;486 else487 cout << "Script aborted with error on line "<<linenum<< " after "<<octanetimer.stop()<<" seconds."<<endl;488 }489490 // return success491 return bretv;492 }

8.25.2.3 bool OctaneTester::ExecuteCommandsFromFile (const std::string &filename, bool quiet)

Run a file full of commands.

Returns:true on no errors encountered during processing.


References ExecuteCommandsFromStream().

Referenced by ParseCommand().

496 {497 // open a file input stream and run it498 // return true on success499 ifstream from(filename.c_str());500 if (!from)501 {502 cout << "error: unable to open file ’" << filename << "’."<<endl;503 return false;504 }505 return ExecuteCommandsFromStream(from,quiet);506 }

8.25.2.4 bool OctaneTester::GrabLineFromConsole (std::string &line) [protected]

Grab a line from cin and return true on success.



Note:this is necessary because of bugs in vc 6


Referenced by TestLoop().

513 {514 // read a line of input from console and return true if a line was read or false if error ctrl-break type abort515 // we had to write this because of a known bug in vc6 getline() which requires double CRs on getline input. good work bill gates.516 // return true on successful read of a line (can be blank)517518 // clear line519 line="";520521 // loop reading characters until CR522 unsigned char c;523 for (;;)524 {525 c = cin.get();526 if (c==’\n’)527 break;528 line+=c;529 }530 return true;531 }

8.25.2.5 OctaneCompressor∗ OctaneTester::SelectInstantiatedCompressor (const std::string &compressorname) [protected]

Select a compressor for subsequent operations; sets current compressorp in the process.



References currentcompressorp, CompressorManager::FindInstantiatedCompressorpFromStringId(), andglobal compressormanagerp.


600 {601 // lookup compressor602 OctaneCompressor* compressorp=global_compressormanagerp->FindInstantiatedCompressorpFromStringId(compressorname);603 // set current compressor604 if (compressorp!=NULL)605 currentcompressorp=compressorp;606 return compressorp;607 }

8.25.2.6 bool OctaneTester::CheckCompressorSelected (OctaneCompressor∗ compressorp)[protected]




Returns:true if compressor is valid and ready


Referenced by CheckCompressorReady(), and ParseCommand().

610 {611 // check if the compressor is nonnull and is ready612 // show warning and return false if not.613 if (compressorp==NULL)614 {615 cout << "No instantiated compressor currently selected (run ’create’ or ’load’)."<<endl;616 return false;617 }618 // one is selected619 return true;620 }

8.25.2.7 bool OctaneTester::CheckCompressorReady (OctaneCompressor∗ compressorp)[protected]


Returns:true if compressor is valid and ready


References CheckCompressorSelected(), currentcompressorp, OctaneCompressor::GetInstantiated-Name(), and OctaneCompressor::IsReadyToCompress().


623 {624 // check if the compressor is nonnull and is ready625 // show warning and return false if not.626 if (!CheckCompressorSelected(compressorp))627 return false;628 if (!compressorp->IsReadyToCompress())629 {630 cout << "compressor "<< currentcompressorp->GetInstantiatedName() << " is not ready for compression; do you need to make or load state data first?"<<endl;631 return false;632 }633 // it is ready634 return true;635 }

8.25.2.8 string OctaneTester::MakeAssumedOutputFilename (const std::string &origname, conststd::string & origext, const std::string & newext) [protected]

helper function to smartly add an extension to a file name, based on current compressor

Returns:a new filename with extensions added to it if appropriate




References currentcompressorp, and OctaneCompressor::GetInstantiatedName().


638 {639 // helper function to smartly add an extension to a file name, based on current compressor640 // and removing old extension if appropriate.641642 if (currentcompressorp==NULL)643 {644 string origfullext(string(".") + origext);645 string::size_type pos = origname.rfind(origfullext);646 if (pos == string::npos)647 return origname + "." + newext;648 return string(origname, 0, pos) + newext;649 }650651 string newfullext(string(".") + currentcompressorp->GetInstantiatedName() + "." + newext);652653 if (origext == "")654 return origname + newfullext;655656 string origfullext(string(".") + currentcompressorp->GetInstantiatedName() + "." + origext);657658 string::size_type pos = origname.rfind(origfullext);659660 if (pos == string::npos)661 return origname + "." + newext;662663 return string(origname, 0, pos) + newext;664 }

8.25.2.9 bool OctaneTester::RunCommandTestString (std::string inputstring) [protected]

Run the TestString command, to compress and then decompress a string and check for errors.



References bitwriter::alignbyte(), OctaneCompressor::Compress(), currentcompressorp, Octane-Compressor::Decompress(), OctaneCompressor::GetInstantiatedName(), and OctaneClass::NiceifyHigh-AsciiString().


671 {672 bool bretv;673 // create bitreader and bitwriter, they will be deleted on exit674 string compressedstring;675 string_bitreader from(inputstring);676 string_bitwriter to(compressedstring);677678 // do compression of text (third argument==false means don’t write header info)679 bretv = currentcompressorp->Compress(from, to, false, false);680 if (!bretv)681 {682 cout << "Error compressing."<<endl;683 return false;



684 }685686 // align output bitwriter (this must be performed when we finish)687 to.align_byte();688689 // show some info about compression690 cout << "Testing " << currentcompressorp->GetInstantiatedName() << ".."<<endl;691 cout << " input length: " << setw(3)<< (int)(inputstring.length()) << "\n";692 cout << " output length: " << setw(3)<< (int)(compressedstring.length()) << endl;693694 // show result in nice format695 string nicestring(compressedstring);696 NiceifyHighAsciiString(nicestring);697 cout << " compressed: " << nicestring << endl;698699 // decompress it back to original to check it700 // create bitreader and bitwriter, they will be deleted on exit701 string resultstring;702 string_bitreader from2(compressedstring);703 string_bitwriter to2(resultstring);704 // do decompression (third argument==false means don’t expect header info)705 bretv = currentcompressorp->Decompress(from2, to2, false);706 if (!bretv)707 {708 cout << "Error decompressing."<<endl;709 return false;710 }711712 // show result713 cout << " original: "<< inputstring << "\n";714 cout << " decompressed: "<< resultstring << endl;715 if (inputstring != resultstring)716 {717 cout << "ERROR: decompression does not match original text."<<endl;718 return false;719 }720721 // we are done with success722 return true;723 }

8.25.2.10 bool OctaneTester::RunCommandTestFile (std::string filename) [protected]

Run the TestFile command, to compress and then decompress a file and check for errors.



References CheckFilesAreIdentical(), DeleteAFile(), RunCommandCompress(), and RunCommand-Decompress().


727 {728 bool bretv,bretv2;729 std::string tempoutfilename_compressed="testfiletemp_compressed.out";730 std::string tempoutfilename_decompressed="testfiletemp_decompressed.out";731732 // compress file with no state info733 cout << "Compressing file without state: ";



734 bretv=RunCommand_Compress(filename,tempoutfilename_compressed,false);735 if (bretv)736 {737 cout << "Decompressing compressed file: ";738 bretv=RunCommand_Decompress(tempoutfilename_compressed,tempoutfilename_decompressed,false);739 if (bretv)740 {741 // now see if they are identical742 cout << "--> Comparing decompressed file with original: ";743 if (CheckFilesAreIdentical(tempoutfilename_decompressed,filename))744 cout << "FILES MATCH."<<endl;745 else746 cout << "FILES DO *NOT* MATCH."<<endl;747 }748 }749 cout << endl;750751 // again with state752 cout << "Compressing file with state: ";753 bretv2=RunCommand_Compress(filename,tempoutfilename_compressed,true);754 if (bretv2)755 {756 cout << "Decompressing compressed file: ";757 bretv2=RunCommand_Decompress(tempoutfilename_compressed,tempoutfilename_decompressed,true);758 if (bretv2)759 {760 // now see if they are identical761 cout << "--> Comparing decompressed file with original: ";762 if (CheckFilesAreIdentical(tempoutfilename_decompressed,filename))763 cout << "FILES MATCH."<<endl;764 else765 cout << "FILES DO *NOT* MATCH."<<endl;766 }767 }768 cout << endl;769770 // now delete temporary files771 DeleteAFile(tempoutfilename_compressed);772 DeleteAFile(tempoutfilename_decompressed);773774 // return success775 return (bretv&&bretv2);776 }

8.25.2.11 bool OctaneTester::RunCommandCreateCompressor (std::stringcompressorclassname,std::string compressorname, unsigned shortguidnum) [protected]

Run the CreateCompressor command, to create a new instantiated compressor.



References CompressorManager::AddCompressor(), currentcompressorp, CompressorManager::Find-BaseCompressorClass(), globalcompressormanagerp, OctaneCompressor::MakeCompressor-Instantiation(), and OctaneCompressor::SetInfo().


783 {784 // create a new compressor based on a base class



785786 // select base compressor class type787 OctaneCompressor *compressorp = global_compressormanagerp->FindBaseCompressorClass(compressorclassname);788 if (compressorp==NULL)789 {790 // couldn’t find base class791 cout << "ERROR: could not find compressor base class ’"<<compressorclassname<<"’."<<endl;792 return false;793 }794795 // ask base class to make us a derived class796 compressorp=compressorp->MakeCompressorInstantiation();797 if (compressorp==NULL)798 {799 // couldn’t find base class800 cout << "ERROR: failed to instantiate a derived compressor from base class ’"<<compressorclassname<<"’."<<endl;801 return false;802 }803804 // set details for it805 compressorp->SetInfo(compressorname,guidnum);806807 // now add it808 bool bretv=global_compressormanagerp->AddCompressor(compressorp);809 if (!bretv)810 {811 // failed to add it812 delete compressorp;813 return false;814 }815 else816 {817 // set this new compressor to be our current compressor?818 if (currentcompressorp==NULL || true)819 currentcompressorp=compressorp;820 }821822 return true;823 }

8.25.2.12 bool OctaneTester::RunCommandLoadCompressor (std::string filename)[protected]

Run the LoadCompressor command, to recall saved state space.



References CompressorManager::AddCompressor(), CompressorManager::CreateCompressorFrom-SavedStream(), currentcompressorp, filebitreader::error(), OctaneCompressor::GetStatisticsLast-OperationStateBytesRead(), OctaneCompressor::GetStatisticsLastOperationStateExecutionTime(),and globalcompressormanagerp.


827 {828 // create bitreader, they will be deleted on exit829 file_bitreader from(filename.c_str());830 if (from.error())831 {



832 cout << "Error opening file " << filename << " for input. Loadstate failed."<<endl;833 return false;834 }835 // load the state836 OctaneCompressor * compressorp= global_compressormanagerp->CreateCompressorFromSavedStream(from);837 if (compressorp==NULL)838 {839 cout <<"Failed to load compressor."<<endl;840 return false;841 }842843 // add compressor844 bool bretv=global_compressormanagerp->AddCompressor(compressorp);845 if (bretv==false)846 {847 cout <<"Failed to add compressor to compressor manager."<<endl;848 delete compressorp;849 return false;850 }851 else852 {853 // set this new compressor to be our current compressor?854 if (currentcompressorp==NULL || true)855 currentcompressorp=compressorp;856 }857858 // show results859 cout << "Loaded compressor from file ’" << filename << "’.\n";860 cout << " state file size: "<< currentcompressorp->GetStatistics_LastOperation_State_BytesRead()<<" bytes.\n";861 cout << " execution time: "<< currentcompressorp->GetStatistics_LastOperation_State_ExecutionTime()<<" seconds."<<endl;862 // show new compressor info864865 // return success866 return true;867 }

8.25.2.13 bool OctaneTester::RunCommandSaveCompressor (std::stringfilename)[protected]

Run the SaveCompressor command, to recall saved state space.



References currentcompressorp, filebitwriter::error(), OctaneCompressor::GetStatisticsLastOperation-StateBytesWritten(), and OctaneCompressor::Save().


871 {872 // create bitreader, they will be deleted on exit873 file_bitwriter to(filename.c_str());874 if (to.error())875 {876 cout << "Error opening file " << filename << " for input. SaveCompressor failed."<<endl;877 return false;878 }879 // save the state (second parameter==false means save complete state, not just state required for decompression)880 bool bretv = currentcompressorp->Save(to,false);881 if (!bretv)



882 {883 cout <<"Failed to save state data."<<endl;884 return false;885 }886 // show result887 cout << "Saved compressor file ’" << filename << "’.\n";888 cout << " state file size: "<< currentcompressorp->GetStatistics_LastOperation_State_BytesWritten()<<" bytes.\n";889890 // return success891 return true;892 }

8.25.2.14 bool OctaneTester::RunCommandDeleteCompressor (std::stringcompressoridstring)[protected]

Run the DeleteCompressor command, to delete an instantiated compressor.



References currentcompressorp, CompressorManager::DeleteCompressor(), CompressorManager::Find-InstantiatedCompressorpFromStringId(), CompressorManager::GetInstantiatedCompressorCount(),CompressorManager::GetInstantiatedCompressorp(), and globalcompressormanagerp.


896 {897 // delete a compressor898899 // look up the compressor900 OctaneCompressor *compressorp = global_compressormanagerp->FindInstantiatedCompressorpFromStringId(compressoridstring);901 if (compressorp==NULL)902 {903 cout <<"Couldn’t locate compressor ’"<<compressoridstring<<"’."<<endl;904 return false;905 }906907 // delete it908 global_compressormanagerp->DeleteCompressor(compressorp);909910 // if this was our current one, then we need to change it911 if (currentcompressorp==compressorp)912 {913 if (global_compressormanagerp->GetInstantiatedCompressorCount()>0)914 currentcompressorp=global_compressormanagerp->GetInstantiatedCompressorp(0);915 else916 currentcompressorp=NULL;917 }918919 // return success920 return true;921 }

8.25.2.15 bool OctaneTester::RunCommandMakeState (std::string infilename) [protected]

Run the MakeState command, to create state info (i.e.

train on file).





References OctaneCompressor::CreateSymbolsAndModelsUsingStream(), currentcompressorp, file-bitreader::error(), and OctaneCompressor::GetInstantiatedName().


925 {926 // create bitreader and bitwriter, they will be deleted on exit927 file_bitreader from(infilename.c_str());928 if (from.error())929 {930 cout << "Error opening file "<<infilename<< " for input. Makestate failed."<<endl;931 return false;932 }933934 // make state data935 cout << "Computing state data using "<<currentcompressorp->GetInstantiatedName() << ":"<<endl;936 bool bretv = currentcompressorp->CreateSymbolsAndModelsUsingStream(from);937 if (!bretv)938 {939 cout << "Failed to generate state data."<<endl;940 return false;941 }942 return true;943 }

8.25.2.16 bool OctaneTester::RunCommandCompress (std::string infilename, std::stringoutfilename, bool savedecompressstateinfo) [protected]

Run the Compress command, to compress a file and write result to another file.



References OctaneCompressor::Compress(), currentcompressorp, filebitwriter::error(), file-bitreader::error(), OctaneClass::GetDiskspaceUsed(), OctaneCompressor::GetInstantiatedName(), Octane-Compressor::GetStatisticsLastOperationCompressionBitrate(), OctaneCompressor::GetStatisticsLast-OperationCompressionBytesRead(), OctaneCompressor::GetStatisticsLastOperationCompression-BytesWritten(), OctaneCompressor::GetStatisticsLastOperationCompressionExecutionTime(), andOctaneCompressor::GetStatisticsLastOperationCompressionRatio().

Referenced by ParseCommand(), and RunCommandTestFile().

947 {948 // create bitreader and bitwriter, they will be deleted on exit949 file_bitreader from(infilename.c_str());950 if (from.error())951 {952 cout << "Error opening file " << infilename << " for input. Compression failed."<<endl;953 return false;954 }955 file_bitwriter to(outfilename.c_str());956 if (to.error())957 {



958 cout << "Error opening file " << outfilename << " for output. Compression failed."<<endl;959 return false;960 }961962 // do the compression963 cout << "Compressing with " << currentcompressorp->GetInstantiatedName() << ":"<<endl;964 bool bretv = currentcompressorp->Compress(from, to, savedecompressstateinfo, savedecompressstateinfo);965 if (!bretv)966 {967 cout << "Compression failed."<<endl;968 return false;969 }970971 // show output972 cout << " original size: " << currentcompressorp->GetStatistics_LastOperation_Compression_BytesRead() << " bytes (" << infilename << ")\n";973 cout << " compressed size: " << currentcompressorp->GetStatistics_LastOperation_Compression_BytesWritten() << " bytes (" << outfilename << ")\n";974 if (savedecompressstateinfo)975 cout << " est. cost of header: " << currentcompressorp->GetDiskspaceUsed(true)<< " bytes.\n";976 float executiontime=currentcompressorp->GetStatistics_LastOperation_Compression_ExecutionTime();977 float bytespersec=(currentcompressorp->GetStatistics_LastOperation_Compression_BytesRead()/executiontime)/1024.0;978 cout << " execution time: " << executiontime << " seconds ("<<bytespersec<<" kb/s).\n";979 cout << " compression ratio: " << currentcompressorp->GetStatistics_LastOperation_Compression_Ratio()*100.0 << "% (" << currentcompressorp->GetStatistics_LastOperation_Compression_Bitrate() << " bits per byte)."<<endl;980981 // return success982 return true;983 }

8.25.2.17 bool OctaneTester::RunCommandDecompress (std::stringinfilename, std::stringoutfilename, bool savedecompressstateinfo) [protected]

Run the Decompress command, to decompress a file and write result to another file.



References OctaneCompressor::AddLastOperationDecompressionBytesRead(),CompressorManager::CreateCompressorFromSavedStream(), currentcompressorp, Octane-Compressor::Decompress(), filebitwriter::error(), filebitreader::error(), CompressorManager::Find-InstantiatedCompressorpFromGuid(), OctaneCompressor::GetInstantiatedName(), Octane-Compressor::GetStatisticsLastOperationDecompressionBitrate(), OctaneCompressor::GetStatistics-LastOperationDecompressionBytesRead(), OctaneCompressor::GetStatisticsLastOperation-DecompressionBytesWritten(), OctaneCompressor::GetStatisticsLastOperationDecompression-ExecutionTime(), OctaneCompressor::GetStatisticsLastOperationDecompressionRatio(), global-compressormanagerp, OctaneCompressor::LoadGUID(), and bitreader::tellbyte().

Referenced by ParseCommand(), and RunCommandTestFile().

987 {988 // create bitreader and bitwriter, they will be deleted on exit989 OctaneCompressor *compressorp;990 int headersize=0;991992 file_bitreader from(infilename.c_str());993 if (from.error())994 {995 cout << "Error opening file " << infilename << " for input. Decompression failed."<<endl;996 return false;997 }



998 file_bitwriter to(outfilename.c_str());999 if (to.error())1000 {1001 cout << "Error opening file " << outfilename << " for output. Decompression failed."<<endl;1002 return false;1003 }10041005 if (savedecompressstateinfo)1006 {1007 // first read guid info - and if found try to create a compressor from known compressors1008 int retvguid=OctaneCompressor::LoadGUID(from);1009 if (retvguid==-1)1010 {1011 // compressor details are in from stream1012 compressorp=global_compressormanagerp->CreateCompressorFromSavedStream(from);1013 if (compressorp==NULL)1014 {1015 cout << "Error creating compressor from the input stream."<<endl;1016 return false;1017 }1018 headersize=from.tell_byte();1019 // cout << "Recreated temporary compressor from input stream."<<endl;1020 }1021 else1022 {1023 // only the guid is in the from stream, now we have to look it up1024 compressorp=global_compressormanagerp->FindInstantiatedCompressorpFromGuid(retvguid);1025 if (compressorp==NULL)1026 {1027 cout << "Error, could not find an instantiated compressor which matches input stream expectation(GUID="<<retvguid<<")."<<endl;1028 return false;1029 }1030 cout << "Using instantiated compressor with guid "<<retvguid<<"."<<endl;1031 }1032 }1033 else1034 {1035 // use current compressor1036 compressorp=currentcompressorp;1037 }10381039 // do the decompression1040 cout << "Decompressing with " << compressorp->GetInstantiatedName() << ":"<<endl;1041 // we pass the last argument false because we have already loaded any compressor state data1042 bool bretv = compressorp->Decompress(from, to, false);1043 if (!bretv)1044 cout << "Decompression failed."<<endl;1045 else1046 {1047 // kludge to fixup lastoperation size to account for header size1048 compressorp->Add_LastOperation_Decompression_BytesRead(headersize);1049 // show result1050 cout <<" original size: " << compressorp->GetStatistics_LastOperation_Decompression_BytesRead() << " (" << infilename << ")\n";1051 cout << " decompressed size: " << compressorp->GetStatistics_LastOperation_Decompression_BytesWritten() << " (" << outfilename << ")\n";1053 float executiontime=compressorp->GetStatistics_LastOperation_Decompression_ExecutionTime();1054 float bytespersec=(currentcompressorp->GetStatistics_LastOperation_Decompression_BytesWritten()/executiontime)/1024.0;1055 cout << " execution time: " << executiontime << " seconds ("<<bytespersec<<" kb/s).\n";1056 cout << " compression ratio: " << compressorp->GetStatistics_LastOperation_Decompression_Ratio()*100.0 << "% (" << compressorp->GetStatistics_LastOperation_Decompression_Bitrate() << " bits per byte)."<<endl;1057 }10581059 if (savedecompressstateinfo)1060 {1061 // kill the temporary compressor1062 delete compressorp;1063 }10641065 // return success



1066 return bretv;1067 }

8.25.2.18 bool OctaneTester::DeleteAFile (std::stringfilename) [protected]

Helper function, delete a file if it exists, or do nothing if not.

Returns:true on deletion of file or no file

Todothis is not implemented yet


Referenced by RunCommandTestFile().

1074 {1075 // helper function, delete a file if it exists, or do nothing if not10761077 remove(filename.c_str());10781079 // return true on deletion of file or no file1080 return true;1081 }

8.25.2.19 bool OctaneTester::CheckFilesAreIdentical (std::stringfilename1, std::string filename2)[protected]

Check two files and see if they are identical.

Returns:true if identical

Tododo we need to do this more efficiently


References bitreader::bytesleft(), file bitreader::error(), and bitreader::getbyte().

Referenced by RunCommandTestFile().

1084 {1085 // test files for equality10861087 // inefficient way but works10881089 file_bitreader from1(filename1.c_str());1090 file_bitreader from2(filename2.c_str());1091 if (from1.error() || from2.error())1092 {1093 // error1094 return false;1095 }1096



1097 if (from1.bytes_left()!=from2.bytes_left())1098 {1099 // dont match1100 return false;1101 }11021103 int len=from1.bytes_left();1104 unsigned char c1,c2;1105 while (len>0)1106 {1107 c1=from1.get_byte();1108 c2=from2.get_byte();1109 if (c1!=c2)1110 return false;1111 --len;1112 }11131114 // return true on success1115 return true;1116 }


• octanetester/octanetester.hpp• octanetester/octanetester.cpp


8.26 OctaneTimer Class Reference 155

8.26 OctaneTimer Class Reference

#include <timer.hpp >



The timer class uses the clock() function from the C standard library.

Definition at line 32 of file timer.hpp.


• OctaneTimer()

Constructor. Sets the begin time to the current time.

• void start()

Reset the begin time to the current time.

• float stop()

Compute the elapsed time between the begin time and the current time.

• floatget elapsed() const

Get the elapsed time between the laststart() and the lateststop().


8.26.2.1 float OctaneTimer::stop () [inline]

Compute the elapsed time between the begin time and the current time.



Referenced by OctaneCompressor::Compress(), OctaneCompressor::CreateSymbolsAndModelsUsing-Stream(), OctaneCompressor::Decompress(), OctaneTester::ExecuteCommandsFromStream(), Octane-ScopeTimer::getelapsed(), OctaneCompressor::LoadState(), OctaneCompressor::Save(), and Octane-ScopeTimer::∼OctaneScopeTimer().

45 {46 clock_t end = clock();47 elapsed = (float)(end - begin)/(float)CLOCKS_PER_SEC;48 return elapsed;49 }



8.26.2.2 float OctaneTimer::getelapsed () const [inline]

Get the elapsed time between the laststart()and the lateststop().



54 { return elapsed; }


• utilityclasses/timer/timer.hpp


8.27 PQWeightGreater< T > Class Template Reference 157

8.27 PQWeightGreater< T > Class Template Reference



template<class T> class PQWeightGreater< T >

We need to make a special Greater<> classes used during sorting, since our collection stl elements arebuilt from pointers, and the default sort will be done on pointer addresses instead of weights if we don’t.



• booloperator() (const T &pLeft, const T &pRight) const


• coders/coderhuffman/huffmannodes.hpp



8.28 SampleCoder Class Reference

#include <samplecoder.hpp >

Inheritance diagram for SampleCoder::

SampleCoder

OctaneCoder

OctaneClass



It simply outputs the symbol number as the binary contents of a c int type (4 byes), ignoring any informationabout modeler probabilities.

Definition at line 32 of file samplecoder.hpp.


• SampleCoder()

constructor

• virtual∼SampleCoder()

destructor











• virtual boolWriteSymbolBits(int symbolnum,bitwriter &bw)


• virtual boolDecodeSymbolFromInput(int &symbolnum,bitreader&br)


8.28 SampleCoder Class Reference 159







8.28.2.1 virtual std::string SampleCoder::GetName () [inline, virtual]





42 {return "SampleCoder";}

8.28.2.2 virtual std::string SampleCoder::GetDescription () [inline, virtual]





43 {return "Sample Coder";}

8.28.2.3 virtual std::string SampleCoder::GetHelpInformation () [inline, virtual]





44 { return ""; }



8.28.2.4 bool SampleCoder::WriteSymbolBits (intsymbolnum, bitwriter & bw) [virtual]




Definition at line 19 of file samplecoder.cpp.


20 {21 // write compressed output for symbol specified22 // return true on success2324 // for our sample coder, we will just write the raw binary data in an int25 bw.put(symbolnum);2627 // return success28 return true;29 }

8.28.2.5 bool SampleCoder::DecodeSymbolFromInput (int &symbolnum, bitreader & br)[virtual]




Definition at line 31 of file samplecoder.cpp.

References bitreader::empty(), and bitreader::get().

32 {33 // decode a symbol from the input34 // return false on no symbols left in stream (or error)3536 // input is empty yet?37 if (br.empty())38 return false;3940 // for our sample coder, we just read the binary data for an int41 br.get(symbolnum);4243 // return success44 return true;45 }

8.28.2.6 virtual bool SampleCoder::IsReadyToCode () [inline, virtual]



8.28 SampleCoder Class Reference 161




61 {return true;};

8.28.2.7 virtual void SampleCoder::ReceiveNotificationModelChangeAllSymbolWeights(OctaneModeler∗ modelerp) [inline, virtual]






66 {;};


• coders/codersample/samplecoder.hpp• coders/codersample/samplecoder.cpp



8.29 SampleCompressor Class Reference

#include <samplecompressor.hpp >

Inheritance diagram for SampleCompressor::

SampleCompressor

OctaneCompressor

OctaneClass



Definition at line 34 of file samplecompressor.hpp.


• SampleCompressor(bool registerme=false)

constructor

• ∼SampleCompressor()

destructor
















8.29 SampleCompressor Class Reference 163





• virtual boolIsReadyToCompress()



• virtual boolDoProtectedCompress(bitreader&from, bitwriter &to)


• virtual boolDoProtectedDecompress(bitreader&from, bitwriter &to)


• virtual boolDoProtectedCreateSymbolsAndModelsUsingStream(bitreader&from)







• unsigned charParameterxorbyte

a byte which the sample compressor xors with incoming or outgoing bytes


8.29.2.1 virtual std::string SampleCompressor::GetDescription () [inline, virtual]






Definition at line 48 of file samplecompressor.hpp.

48 {return "Sample Compressor";}

8.29.2.2 bool SampleCompressor::SetParameter (const std::string &parametername, conststd::string & parametervalue) [virtual]





Definition at line 87 of file samplecompressor.cpp.

References Parameterxorbyte, and OctaneClass::ParseParameter().

88 {89 // Generic function for user-interactive modification of parameters90 // Return true if we know this variable and use it91 bool bretv=false;9293 // we have 1 variable we use, called xorbyte94 if (parametername=="xorbyte")95 bretv=ParseParameter(parametervalue,Parameter_xorbyte);9697 // return success98 return bretv;99 }

8.29.2.3 bool SampleCompressor::IsReadyToCompress ()[virtual]





106 {107 // Are we ready to compress?108 // This sample compressor is always ready (needs no training)109 return true;110 }



8.29.2.4 bool SampleCompressor::DoProtectedCompress (bitreader & from, bitwriter & to)[protected, virtual]







References bitreader::empty(), bitreader::getbyte(), Parameterxorbyte, and bitwriter::putbyte().

118 {119 // Here is the main compression routine120 // Our sample compressor doesn’t actually compress, instead it just xors121122 // loop through input and ’compress’ (xor) it123 unsigned char c;124 while (!from.empty())125 {126 // grab a character from input127 c=from.get_byte();128 // xor it with our parameter129 c= c ˆ Parameter_xorbyte;130 // send it to output131 to.put_byte(c);132 }133134 // return success135 return true;136 }

8.29.2.5 bool SampleCompressor::DoProtectedDecompress (bitreader & from, bitwriter & to)[protected, virtual]







References bitreader::empty(), bitreader::getbyte(), Parameterxorbyte, and bitwriter::putbyte().



140 {141 // Here is the main decompression routine142 // Our sample compressor doesn’t actually decompress, instead it just xors143144 // loop through input and ’compress’ (xor) it145 unsigned char c;146 while (!from.empty())147 {148 // grab a character from input149 c=from.get_byte();150 // xor it with our parameter151 c= Parameter_xorbyte ˆ c;152 // send it to output153 to.put_byte(c);154 }155156 // return success157 return true;158 }

8.29.2.6 bool SampleCompressor::DoProtectedCreateSymbolsAndModelsUsingStream (bitreader& from) [protected, virtual]


It is wrapped by the public API callCreateSymbolsAndModelsUsingStream()which performs some mea-surements on speed and streamsize.





References bitreader::empty(), and bitreader::getbyte().

166 {167 // OPTIONAL: this is called to ’train’ a compressor, to let a compressor gather statistics from an input stream.168 // Our sample compressor does not need to implement this, but we include it here just for fun so you see what it looks like.169 // return false only on an error.170171 // let’s read the training input stream, just to show you how its done.172 unsigned char c;173 while (!from.empty())174 {175 // grab a character, but we dont do anything with it.176 c=from.get_byte();177 }178179 // return success180 return true;181 }



8.29.2.7 bool SampleCompressor::DoProtectedSaveState (bitwriter & to, boolfortempdecompressiononly) [protected, virtual]







References Parameterxorbyte, and bitwriter::put().

184 {185 // OPTIONAL: save any compressor internal state (like variables or learned statistics).186 // Many compressors have no interenal state to save, but this sample compressor has a variable187 // which should be saved.188 // The flag fortempdecompressiononly will be set if the state is being saved for decompression *only*189 // so that if there is some information is only needed for compression, we dont have to save it if we dont need it.190191 // just tell the bitwriter to save the variables192 to.put(Parameter_xorbyte);193194 // return true on success195 return true;196 }

8.29.2.8 bool SampleCompressor::DoProtectedLoadState (bitreader & from) [protected,virtual]


It is wrapped by the public API callLoadState()which performs some measurements on speed and stream-size.





References bitreader::get(), and Parameterxorbyte.

199 {200 // OPTIONAL: load any compressor internal state (like variables or learned statistics).



201 // Many compressors have no interenal state to save, but this sample compressor has a variable202 // which should be saved.203204 // just tell the bitwriter to save the variables205 from.get(Parameter_xorbyte);206207 // return true on success208 return true;209 }


• compressors/compressorsample/samplecompressor.hpp• compressors/compressorsample/samplecompressor.cpp


8.30 SampleModeler Class Reference 169

8.30 SampleModeler Class Reference

#include <samplemodeler.hpp >

Inheritance diagram for SampleModeler::

SampleModeler


OctaneModeler

OctaneClass



Definition at line 28 of file samplemodeler.hpp.


• SampleModeler()

constructor

• virtual∼SampleModeler()

destructor





• virtual boolCreateModelUsingStream(OctaneParser∗parserp,bitreader&from)


• virtual boolCreateModelUsingParser(OctaneParser∗parserp)



8.30.2.1 virtual std::string SampleModeler::GetName () [inline, virtual]





Reimplemented fromOctaneModelerWeightVectored.


38 {return "SampleModeler";}

8.30.2.2 virtual std::string SampleModeler::GetDescription () [inline, virtual]





39 {return "Fixed Equal-Probability Modeler";}

8.30.2.3 virtual bool SampleModeler::CreateModelUsingStream (OctaneParser∗ parserp,bitreader & from) [inline, virtual]





References CreateModelUsingParser().

44 {return CreateModelUsingParser(parserp);};

8.30.2.4 bool SampleModeler::CreateModelUsingParser (OctaneParser∗ parserp) [virtual]


This is called on startup, before any possible training, when a default parser exists.

Note:that no input (training) stream is provider; some modelers may ignore this.



8.30 SampleModeler Class Reference 171


Definition at line 16 of file samplemodeler.cpp.

References OctaneParser::GetSymbolCount(), SymbolWeightVector::NormalizeToProbability-Distribution(), SymbolWeightVector::SetSymbolCount(), SymbolWeightVector::SetSymbolWeight(),and OctaneModelerWeightVectored::weightvector.

Referenced by CreateModelUsingStream().

17 {18 // initialize vector with proper number of symbols as specified by the parser19 int symbolcount=parserp->GetSymbolCount();20 weightvector.SetSymbolCount(symbolcount);2122 // set them all equal23 for (int index=0;index<symbolcount;++index)24 weightvector.SetSymbolWeight(index,1);2526 // normalize to probability distribution27 weightvector.NormalizeToProbabilityDistribution();2829 // success30 return true;31 }


• modelers/modelersample/samplemodeler.hpp• modelers/modelersample/samplemodeler.cpp



8.31 SampleParser Class Reference

#include <sampleparser.hpp >

Inheritance diagram for SampleParser::

SampleParser

OctaneParser

OctaneClass



Because it uses a fixed symbol set it is ready to parse without any training.

Definition at line 45 of file sampleparser.hpp.


• SampleParser()

constructor

• ∼SampleParser()

destructor














8.31 SampleParser Class Reference 173






















• bool senteos



8.31.2.1 virtual bool SampleParser::CreateSymbolSetUsingStream (bitreader & from)[inline, virtual]







61 {return true;};

8.31.2.2 virtual bool SampleParser::RewindAnyBufferedInput (bitreader & from) [inline,virtual]





63 {return true;};

8.31.2.3 bool SampleParser::ParseNextSymbolFromInput (bitreader & from, int & symbolnum)[virtual]





Definition at line 28 of file sampleparser.cpp.

References bitreader::empty(), bitreader::get(), and senteos.

29 {30 // grab an input stream symbol and set its INDEX (in symbol vector) for symbolnum31 // return false after EOS32 unsigned char c;3334 // are we at end of from input - this is the only non-intuitive step35 // the issue is that only the parser knows about end-of-stream symbols, and when it hits and end of input36 // it basically needs to reply TWICE, first with an end-of-stream, and then with a reply saying ’no more symbols’37 if (from.empty())38 {39 // no more symbols left - BUT the question now is, do we return an EOS symbol, or false for no symbols left40 if (senteos)41 {42 // we already sent an EOS so from now on any requests for a symbol returns false saying no more symbols available43 return false;44 }45 else46 {47 // we are going to drop down to return the EOS signal, but we set flag so we don’t do it again48 senteos=true;49 }



50 // end of stream symbol number51 symbolnum=SampleParser_EndOfStreamSYMBOL;52 }53 else54 {55 // grab an unsigned character from input and assign the symbol # to its ascii number56 from.get(c);57 symbolnum=(int)c;58 }5960 // return true, saying we read a symbol61 return true;62 }

8.31.2.4 bool SampleParser::WriteSymbolText (bitwriter & to, int symbolnum, bool &isendofstreamsymbol) [virtual]






66 {67 // write the symbol indexed by symbolnum68 // sets isendofostreamsymbol to true or false depending on if the symbol written is the EOS symbol69 // return true on success7071 if (symbolnum==SampleParser_EndOfStreamSYMBOL)72 {73 // this is end of stream symbol, so do nothing but set EOS flag74 isendofstreamsymbol=true;75 }76 else77 {78 // not EOS, so write it and set EOS flag false79 to.put((unsigned char)symbolnum);80 isendofstreamsymbol=false;81 }8283 // return success84 return true;85 }

8.31.2.5 string SampleParser::LookupSymbolText (intsymbolnum) [virtual]








89 {90 // return the string text corresponding to symbol ’symbolnum’91 if (symbolnum==SampleParser_EndOfStreamSYMBOL)92 return "";93 char cstr[2];94 cstr[0]=(unsigned char)symbolnum;95 cstr[1]=’\0’;96 return string(cstr);97 }

8.31.2.6 virtual bool SampleParser::SetParameter (const std::string &parametername, conststd::string & parametervalue) [inline, virtual]






82 {return false;};

8.31.2.7 virtual bool SampleParser::SaveState (bitwriter & to, bool fortempdecompressiononly)[inline, virtual]





83 {return true;};

8.31.2.8 virtual bool SampleParser::LoadState (bitreader & from) [inline, virtual]







84 {return true;};


8.31.3.1 boolSampleParser::senteos [protected]


Todothere should be a way to do this automatically perhaps with a wrapper, rather than insisting all derivedparsers deal with returning an EOS symbol.


Referenced by ParseNextSymbolFromInput(), and SynchronizeStateForNewStream().


• parsers/parsersample/sampleparser.hpp• parsers/parsersample/sampleparser.cpp



8.32 SampleStatCompressor Class Reference

#include <samplestatcompressor.hpp >

Inheritance diagram for SampleStatCompressor::

SampleStatCompressor


OctaneCompressor

OctaneClass



Definition at line 35 of file samplestatcompressor.hpp.


• SampleStatCompressor(bool registerme=false)

constructor

• ∼SampleStatCompressor()

destructor








8.32.2.1 virtual std::string SampleStatCompressor::GetDescription () [inline, virtual]




8.32 SampleStatCompressor Class Reference 179

Reimplemented fromOctaneCompressorStatistical.

Definition at line 46 of file samplestatcompressor.hpp.

46 {return "Sample Statistical Compressor";}


• compressors/compressorsample/samplestatcompressor.hpp• compressors/compressorsample/samplestatcompressor.cpp



8.33 streambitreader Class Reference


Inheritance diagram for streambitreader::

stream_bitreader

bitreader





• streambitreader(std::istream &s, sizet size in bits=0)








• std::istream &s


8.33.2.1 streambitreader::stream bitreader (std::istream & s, sizet sizein bits= 0) [inline]


Parameters:s - the stream to use.



52 : bitreader(size_in_bits), s(_s) { ; }


8.33 streambitreader Class Reference 181


8.33.3.1 bool streambitreader::supports seek () const [inline, virtual]






8.33.3.2 char streambitreader::read next byte () [inline, protected, virtual]






59 { char c; s.get(c); return c; }





8.34 streambitwriter Class Reference


Inheritance diagram for streambitwriter::

stream_bitwriter

bitwriter





• streambitwriter (std::ostream &s)

Constructor.





• std::ostream &s


8.34.2.1 streambitwriter::stream bitwriter (std::ostream & s) [inline]

Constructor.

Parameters:s - the stream to use.


46 : s(_s) { ; }


8.34 streambitwriter Class Reference 183


8.34.3.1 void streambitwriter::write next byte (char c) [inline, protected, virtual]






51 { s.put(c); }





8.35 string bitreader Class Reference


Inheritance diagram for stringbitreader::

string_bitreader

bitreader





• string bitreader(const std::string &s)

Constructor.

• string bitreader(const std::string &s, sizet size in bits)










• const std::string &s• size t index


8.35.2.1 stringbitreader::string bitreader (const std::string & s) [inline]

Constructor.


8.35 string bitreader Class Reference 185

Sets the number of bits available to the size of the string.

Parameters:s - the string to use.


100 : bitreader(8 * _s.size()), s(_s), index(0)101 { ; }

8.35.2.2 stringbitreader::string bitreader (const std::string & s, sizet sizein bits) [inline]







8.35.3.1 bool stringbitreader::supports seek () const [inline, virtual]





110 { return true; }

8.35.3.2 char stringbitreader::read next byte () [inline, protected, virtual]









8.35.3.3 void stringbitreader::set position (sizet pos) [inline, protected, virtual]






115 { index = pos; }




8.36 string bitwriter Class Reference 187

8.36 string bitwriter Class Reference


Inheritance diagram for stringbitwriter::

string_bitwriter

bitwriter





• string bitwriter (std::string &s)

Constructor.





• std::string &s


8.36.2.1 stringbitwriter::string bitwriter (std::string & s) [inline]

Constructor.



78 : s(_s) { ; }




8.36.3.1 void stringbitwriter::write next byte (char c) [inline, protected, virtual]






83 { s += c; }




8.37 stringparser Class Reference 189

8.37 stringparser Class Reference

#include <stringparser.hpp >



Breaks up a string into space-delimited tokens.

The class operates on a copy of the string, to avoid any synchronization issues. This should not be aproblem for performance if the string implementation is reference counted.

Definition at line 33 of file stringparser.hpp.


• stringparser(const std::string &s)

Constructor.

• std::stringget token()

Get next token from string.


8.37.2.1 stringparser::stringparser (const std::string & s) [inline]

Constructor.

Parameters:s - the string to parse.


38 : s(_s), idx(0) { ; }


8.37.3.1 std::string stringparser::gettoken () [inline]

Get next token from string.

Returns:the next token.

Todoshould be able to handle quoted tokens.


Referenced by OctaneTester::ParseCommand().



44 {45 if (idx == std::string::npos) return "";4647 std::string::size_type p = s.find_first_not_of(’ ’, idx);4849 if (p == std::string::npos) return "";5051 idx = s.find_first_of(’ ’, p);5253 if (idx == std::string::npos) return s.substr(p);5455 return s.substr(p, idx - p);56 }


• utilityclasses/stringparser/stringparser.hpp


8.38 SubStrHuffCompressor Class Reference 191

8.38 SubStrHuffCompressor Class Reference

#include <substrhuff.hpp >

Inheritance diagram for SubStrHuffCompressor::

SubStrHuffCompressor


OctaneCompressor

OctaneClass


The SubStrHuff Compressor is a basic Statistical compressor, that instantiates a Zero-order modeler, witha huffman coder, and a SubString parser.

Definition at line 34 of file substrhuff.hpp.


• SubStrHuffCompressor(bool registerme=false)

constructor

• ∼SubStrHuffCompressor()

destructor








8.38.2.1 virtual std::string SubStrHuffCompressor::GetDescription () [inline, virtual]





Reimplemented fromOctaneCompressorStatistical.

Definition at line 45 of file substrhuff.hpp.

45 {return "SubString Huffman Compressor";}


• compressors/compressorsubstrhuff/substrhuff.hpp• compressors/compressorsubstrhuff/substrhuff.cpp


8.39 SubstringParser Class Reference 193

8.39 SubstringParser Class Reference

#include <substringparser.hpp >

Inheritance diagram for SubstringParser::

SubstringParser

OctaneParser

OctaneClass



It can be configured to build a symbol set using only whole words, or with substrings that are not affectedby word boundary (max symbol length can be set). Various input parsing strategies can be used, fromgreedy (fastest) to longest first, to near optimal (slowest). Various heuristics for pruning the symbol set canbe applied, which can be used to get the symbol set down to a desired size.

Definition at line 154 of file substringparser.hpp.


• SubstringParser()

Constructor.

• ∼SubstringParser()

Destructor.








Synchronize state for a new stream - this is called every time a new stream is parsed, to allow parser to getinto ’start’ state,.



























• int GetSetSymbolCount()

• int LookupSymbolNumberFromText (string wordstring)

Protected Types

• typedef std::set< SubstringSymbol∗, TSubstringSymbolStringGreater< SubstringSymbol∗ > >TSymbolSetTYPE

Typedef for stl set of symbols (used for fast alphabetical lookup).

• typedef std::vector< SubstringSymbol∗ > TSymbolVectorTYPE

Typedef for stl vector of symbols (used for fast lookup by id#).




• TSymbolSetTYPEsymbolset

STL set of symbols (used for fast alphabetical lookup).

• TSymbolVectorTYPEsymbolvector

STL vector of symbols (used for fast lookup by id#).

• TSymbolSetTYPE::iteratorsymbolsetpos

STL set iterator - this is a non-local variable to eliminate cost of building it on stack on every local function.

• TSymbolVectorTYPE::iteratorsymbolvectorpos

Stl vector iterator - this is a non-local variable to eliminate cost of building it on stack on every localfunction.

• charinputbufferstr[SubStrHuffBiggestSubStringLen]

Current window buffer for parsing, filled as we read input characters.

• int inputbufferlen

Length of input window buffer.

• bool senteos

The parser needs to return an end-of-symbol character before finishing reading the stream, so we need totrack whether we sent an EOS yet.

• unsigned intParameterMaxSymbolsDuringBuild

The dictionary will never be allowed to get bigger than this size (smaller will also increase building speed)when this size is exceeded, we will eliminate all symbols with frequencies of smaller than some small #.

• unsigned intParameterMaxSymbolsFinal

After the entire training file is parsed, we will repeat pruning steps to reduce it to a∗maximum∗ of this manyentries the actual symbols kept may be smaller depending on other pruning parameters.

• int ParameterMaxSubstringSize

How many characters is largest (sub)string stored in dictionary.

• boolParameterOnlyCodeWholeWords

Only store characters and whole words?

• boolParameterSpanWordBoundaries

Store Substrings that span words (i.e. they contain multiple words).

• boolParameter CountCRsAsEOTs• unsigned intParameterPruneMinimumWeight

Minimum count for a word or Substring in order to even consider leaving it in the dictionary.

• int ParameterParseMode

We can either use a greedy search to find longest Substring starting with first char or we can use smarterbut slower heuristics.

• boolParameterUseSmartLookup



We can either use a smart search using stl findlower bound() to find longest Substring or we can searchstarting at biggest string backwards.

• boolParameterPruneReCalculations

How many iterations should we recompress to test heuristic symbolset cost, if any this tests the currentlybuilt symbol set to ’reencode’ the input stream and calculates actual usage counts and then reprunes.

• int symbolcount

Keeps track of how many symbols have we created for fast replies.

• int endofstreamsymbolnum

Keeps track of end of stream symbol number for fast replies.

• bitreader∗ currentbitreaderp

Stores last bitreader , so that we can rewind when we want to reparse and check our heuristic symbolsetcosts.

• size t start bitreaderposition

Stores last bitreader starting position before parsing, so that we can rewind when we want to reparse andcheck our heuristic symbolset costs.


8.39.2.1 bool SubstringParser::CreateSymbolSetUsingStream (bitreader & from) [virtual]




Definition at line 281 of file substringparser.cpp.

References currentbitreaderp, ParameterMaxSubstringSize, ParameterMaxSymbolsDuringBuild,start bitreaderposition, and symbolset.

282 {283 // scan a file and add its "tokens" into our symbol set284 string valuestring;285 string slidingwindowstring;286 unsigned char c;287 bool slidingwindowfull=false;288 string eotstring=string("");289290 // clear any existing symbols291 FreeData_Symbols();292 // create primitive characters293 AddPrimitiveCharacterSubstringSymbols();294295 // save the current bitreader and bitreader position for reparsing during pruning296 current_bitreaderp = &from;297 start_bitreaderposition = from.tell_bit();298299 // now parse the file



300 while (!from.empty())301 {302 // grab a character303 c=from.get_byte();304305 // add the character to our window306 slidingwindowstring+=c;307308 // trim lefthand side if it is too big309 if (!slidingwindowfull)310 {311 // we keep a sliding window which is slightly bigger than the maxSubstring, to account for delimiter boundaries312 if ((int)(slidingwindowstring.length())>Parameter_MaxSubstringSize+5)313 {314 // set the flag saying from now on, we are full315 slidingwindowfull=true;316 }317 }318 if (slidingwindowfull)319 {320 // trim leftmost character321 slidingwindowstring.erase(0,1);322 }323324 // now add the appropriate Substrings of slidingwindowstring to our dictionary SymbolSet325 AddSubstringsFromSlidingWindowString(slidingwindowstring);326327 // when encoding single lines, each line must end with an end-of-stream (EOS) symbol328 // but if we train on a file, which is large and has only one EOS at end of file329 // then the freq. of EOS will be very low (1), so here we can say to count a pretend EOS330 // whenever we see a CR331 if (c==13 && Parameter_CountCRsAsEOTs)332 {333 // increment frequency for EOS334 UpdateValueStringInSymbolSet(eotstring,1);335 }336337 if (symbolset.size()>Parameter_MaxSymbols_DuringBuild)338 {339 // periodically we may have to prune away lowest frequency symbols in order to keep dictionary manageable340 // how many should we remove - we don’t want to remove just one since that would result in many repetitive calls here341 // so we just pick some fraction to remove342 PruneSymbolSet_ReduceSizeTo(((unsigned int)(symbolset.size())/5)+1);343 }344 }345346 // now final pruning347 PruneSymbolSet();348349 // and now build the vector for fast random access350 BuildSymbolVector();351352 // return success353 return true;354 }

8.39.2.2 bool SubstringParser::RewindAnyBufferedInput (bitreader & from) [virtual]







References inputbufferlen, bitreader::seekbit(), and bitreader::tellbit().

259 {260 // let go of any buffered stream - this is an odd function that can be called be compressor if it wants to hand261 // off the input stream to a new parser or otherwise access the input bitstream from after last symbol parsed.262 // it is necessary because some parsers can read-ahead in input buffer, and so must rewind the bitstream.263 // we have to implement this since we do a read-ahead. note this wouldn’t be called in normal circumstances,264 // just if compressors wants us to stop prematurely and hand off the job to someone else.265 // return true on success266267 // the number of characters we have read ahead in buffer is inputbufferlen268 from.seek_bit(from.tell_bit()-(inputbufferlen*8));269 return true;270 }

8.39.2.3 bool SubstringParser::ParseNextSymbolFromInput (bitreader & from, int & symbolnum)[virtual]






References currentbitreaderp, bitreader::getbyte(), SubstringSymbol::getsymbolvectorpos(), input-bufferlen, inputbufferstr, ParameterMaxSubstringSize, and senteos.

363 {364 // grab an input stream symbol and set its INDEX (in symbol vector) for symbolnum365 // return false after EOS366 unsigned char c;367 int maxSubstringlen=Parameter_MaxSubstringSize;368 SubstringSymbol *symbolp;369 // note that inputbufferlen are preserved across calls, and inputbufferstr[] is preserved state info370371 // try to fill up the inputqueue to Parameter_MaxSubstringSize size, or as much as we got372 current_bitreaderp=&from;373 while (inputbufferlen<maxSubstringlen && !from.empty())374 {375 // add the character376 c=from.get_byte();377 inputbufferstr[inputbufferlen]=c;378 // increment character positions379 ++inputbufferlen;380 }381382 if (inputbufferlen==0)383 {384 // no more symbols left - BUT the question now is, do we return an EOS symbol, or false for no symbols left



385 if (senteos)386 {387 // we already sent an EOS so from now on any requests for a symbol returns false saying no more symbols available388 return false;389 }390 else391 {392 // we are going to drop down to return the EOS signal, but we set flag so we don’t do it again393 senteos=true;394 }395 }396397 // ok now we have a block of up to Parameter_MaxSubstringSize bytes from the left of the inputstr398 // now find the longest leftmost (prefix) Substring in our dictionary and encode it, and shift inputqueuestr to the left with remaining bytes399 // and return the new inputqueuestr with remaining bytes, and return length of remaining bytes.400 symbolp=FindNextSymbolToEncode(inputbufferstr,inputbufferlen);401 // symbolp=FindNextSubstringSymbolpFromInputQueueStr(inputbufferstr,inputbufferlen);402403 symbolnum=symbolp->get_symbolvectorpos();404 inputbufferlen=SwallowSymbolFromInputQueueStr(symbolp,inputbufferstr,inputbufferlen,false);405 return true;406 }

8.39.2.4 bool SubstringParser::WriteSymbolText (bitwriter & to, int symbolnum, bool &isendofstreamsymbol) [virtual]





References endofstreamsymbolnum, SubstringSymbol::getvaluep(), symbolvector, and bitwriter::write().

410 {411 // write the symbol indexed by symbolnum412 // sets isendofostreamsymbol to true or false depending on if the symbol written is the EOS symbol413 // return true on success414415 SubstringSymbol *symbolp=symbolvector[symbolnum];416 int valuelength=(int)((symbolp->get_valuep())->length());417 to.write((symbolp->get_valuep())->c_str(),valuelength);418419 // set EOS flag420 isendofstreamsymbol=(symbolnum==endofstreamsymbolnum);421422 // return success423 return true;424 }

8.39.2.5 virtual string SubstringParser::LookupSymbolText (int symbolnum) [inline,virtual]








References GetDiskspaceUsed(), GetMemoryUsed(), GetParametersInformation(), LoadState(), Save-State(), SetDefaultParameters(), SetParameter(), ShowDebuggingInfo(), and symbolset.

235 {assert(symbolnum<(int)(symbolvector.size()));return symbolvector[symbolnum]->get_value();};

8.39.2.6 bool SubstringParser::SetParameter (const std::string &parametername, conststd::string & parametervalue) [virtual]






References ParameterMaxSubstringSize, ParameterMaxSymbolsDuringBuild, ParameterMax-SymbolsFinal, ParameterOnlyCodeWholeWords, ParameterParseMode, ParameterPruneMinimum-Weight, ParameterPruneReCalculations, ParameterSpanWordBoundaries, ParameterUseSmartLookup,and OctaneClass::ParseParameter().

Referenced by LookupSymbolText().

179 {180 // generic function for user-interactive modification of parameters181 // (see SetDefaultDictionaryBuildingParameters implementation for descriptions)182 // return true if we know this variable and use it183 bool bretv=false;184 if (parametername=="maxbuildsymbols")185 bretv=ParseParameter(parametervalue,Parameter_MaxSymbols_DuringBuild);186 else if (parametername=="maxsymbols")187 bretv=ParseParameter(parametervalue,Parameter_MaxSymbols_Final);188 else if (parametername=="maxSubstring")189 bretv=ParseParameter(parametervalue,Parameter_MaxSubstringSize);190 else if (parametername=="onlywords")191 bretv=ParseParameter(parametervalue,Parameter_OnlyCodeWholeWords);192 else if (parametername=="spanwords")193 bretv=ParseParameter(parametervalue,Parameter_SpanWordBoundaries);194 else if (parametername=="crends")195 bretv=ParseParameter(parametervalue,Parameter_CountCRsAsEOTs);196 else if (parametername=="minweight")197 bretv=ParseParameter(parametervalue,Parameter_PruneMinimumWeight);198 else if (parametername=="smartlookup")199 bretv=ParseParameter(parametervalue,Parameter_UseSmartLookup);200 else if (parametername=="parsemode")201 bretv=ParseParameter(parametervalue,Parameter_ParseMode);202 else if (parametername=="recalculations")203 bretv=ParseParameter(parametervalue,Parameter_PruneReCalculations);204 return bretv;205 }



8.39.2.7 bool SubstringParser::SaveState (bitwriter & to, bool fortempdecompressiononly)[virtual]






56 {57 // save state58 // return true on success59 bool bretv;60 bretv=SaveParameters(to);61 if (bretv)62 bretv=SaveSymbols(to);63 return bretv;64 }

8.39.2.8 bool SubstringParser::LoadState (bitreader & from) [virtual]






67 {68 // load state69 // return true on success70 bool bretv;71 bretv=LoadParameters(from);72 if (bretv)73 bretv=LoadSymbols(from);74 return bretv;75 }


8.39.3.1 boolSubstringParser::senteos [protected]

The parser needs to return an end-of-symbol character before finishing reading the stream, so we need totrack whether we sent an EOS yet.

Todothis is an odd responsibility for the parser to have to deal with, so we should probably find a way toeliminate this.




Referenced by ParseNextSymbolFromInput(), SubstringParser(), and SynchronizeStateForNewStream().


• parsers/parsersubstring/substringparser.hpp• parsers/parsersubstring/substringparser.cpp


8.40 SubstringSymbol Class Reference 203

8.40 SubstringSymbol Class Reference



The SubString Symbol class represents the primitive symbols used by the SubString parser.

Each symbol can hold an arbitrary string of character, and is annotated with a ’weight’ which stores thefrequency of that substring during the construction of the symbol table. Symbols also contain a numericalindex which allows each symbol to report its symbol id (position in the symbol vector), which is howparsers interact with other components of statistical coding (by symbol id #).



• SubstringSymbol(const std::string &invalue, TSubStrParserWeight inweight=0)• booloperator> (constSubstringSymbol∗&a) const

The comparison operator used to order the priority queue.

• TSubStrParserWeightget weight() const

Returns the current weight (frequency) of the symbol.

• void incrementweight(int increment)

Increment weight/frequency of symbol.

• void setweight(TSubStrParserWeight val)

Set weight of symbol.

• std::string∗ get valuep()

Get pointer to string value of symbol.

• std::string &get value()

Get string value of symbol (i.e. the text of the symbol).

• int get valuelen()

Get length of symbol string.

• void set value(const std::string &invalue)

Set value of symbol string.

• int get symbolvectorpos()

Return the symbol id # (the position of the symbol in the symbol vector).

• void set symbolvectorpos(int inpos)

Set the symbol id #.

• booldontprune()

Returns true if this is a ”primitive” protected symbol and should not be pruned.



• unsigned intget memoryused() const

Returns the actual memory used by this symbol (uses string.capacity).

• doubleget cost()

The parser may evaluate the quality of its symbol set by estimating the ”cost” of the symbolset even inthe absence of any specified coding and compression strategy; it calculates the cost of each symbol as theinverse of frequency, lower is better.


• TSubStrParserWeightweight

Frequency of the symbol, as calculated during a ’training’ phase.

• std::stringvalue

Value represented by the symbol (i.e. the character, word, or Substring).

• int symbolvectorpos

Each symbol keeps track of its position in symbol vector.


8.40.2.1 bool SubstringSymbol::dontprune () [inline]

Returns true if this is a ”primitive” protected symbol and should not be pruned.

used for things like insisting that the ascii characters are never pruned from symbol set even though theyare not used during training.


References value.

106 {if (value.length()<=1) return true; else return false;}


• parsers/parsersubstring/substringparser.hpp


8.41 SymbolWeightVector Class Reference 205

8.41 SymbolWeightVector Class Reference

#include <symbolweightvector.hpp >

Inheritance diagram for SymbolWeightVector::

SymbolWeightVector

OctaneClass


The SymbolWeightVector class is a helper class which holds a vector of probabilities, typically associatedwith a modeler.

It provides basic functionality to save and load state and display debugging information, and report memoryusage.

Definition at line 61 of file symbolweightvector.hpp.


• SymbolWeightVector()

constructor

• virtual∼SymbolWeightVector()

destructor



















• void ResetWeights()

Reset all probabilities.

• void NormalizeToProbabilityDistribution()

Normalize vector values into a probability distribution.

• void SetSymbolCount(int newsize)

Set the number of symbols we should store weights for; can be called at any time to shrink or grow size.

• int GetSymbolCount()

Get count of symbols.

• TSymbolWeightVectorWeightGetSymbolWeight(int index)

Get the weight (probability or freq count) of a symbol.

• void SetSymbolWeight(int index, TSymbolWeightVectorWeight weight)

Set the weight (probability or freq count) of a symbol.

• void IncremementSymbolWeight(int index)

Increment the weight of a given entry.

• boolCountSymbolFrequencies(OctaneParser∗parserp,bitreader&from)

Higher level function for automatic setting of frequencies with help of parser.

• void EnforceZeroCountFloor(TSymbolWeightVectorWeight symbolfloorval)

It’s common that we may have symbols for characters we dont see for completeness and it’s common towant to force these to some minimal value (like 1); some coders (like huffman), will break without this.


• int symbolcount

size of vector (# of symbol probabilities)

• vector< TSymbolWeightVectorWeight> weightvector

vector of probabilities


8.41.2.1 virtual std::string SymbolWeightVector::GetName () [inline, virtual]







76 {return "SymbolWeightVector";}

8.41.2.2 virtual std::string SymbolWeightVector::GetDescription () [inline, virtual]





77 {return "Symbol Weight Vector";}

8.41.2.3 virtual std::string SymbolWeightVector::GetHelpInformation () [inline,virtual]





78 {return "";}

8.41.2.4 bool SymbolWeightVector::SaveState (bitwriter & to, bool fortempdecompressiononly)[virtual]




Definition at line 58 of file symbolweightvector.cpp.

References bitwriter::put(), symbolcount, and weightvector.

Referenced by OctaneModelerWeightVectored::SaveState().



59 {60 // save state61 // return true on success62 to.put(symbolcount);63 for (int count=0;count<symbolcount;++count)64 to.put(weightvector[count]);65 return true;66 }

8.41.2.5 bool SymbolWeightVector::LoadState (bitreader & from) [virtual]





References bitreader::get(), SetSymbolCount(), symbolcount, and weightvector.

Referenced by OctaneModelerWeightVectored::LoadState().

69 {70 // load state71 from.get(symbolcount);72 // force size of vector73 SetSymbolCount(symbolcount);74 // read weights75 for (int count=0;count<symbolcount;++count)76 from.get(weightvector[count]);77 // return true on success78 return true;79 }

8.41.2.6 bool SymbolWeightVector::CountSymbolFrequencies (OctaneParser∗ parserp, bitreader& from)

Higher level function for automatic setting of frequencies with help of parser.

This will actually parse the input using the parser, and count frequency of each symbol, and then normalizeit into a probability distribution.



References OctaneParser::GetSymbolCount(), IncremementSymbolWeight(), OctaneParser::ParseNext-SymbolFromInput(), ResetWeights(), and SetSymbolCount().

Referenced by ZeroOrderModeler::CreateModelUsingStream().

117 {118 // higher level function for automatic setting of frequencies with help of parser; returns true on success119 bool bretv=true;120 int symbolnumber;



121122 // set our weight vector to size of symbolcount123 SetSymbolCount(parserp->GetSymbolCount());124 // reset probabilities125 ResetWeights();126127 // now parse input and count frequencies, OOP parsing a symbol from parser until we hit end128 while (parserp->ParseNextSymbolFromInput(from,symbolnumber))129 IncremementSymbolWeight(symbolnumber);130131 // return success132 return bretv;133134 }


• modelers/symbolweightvector.hpp• modelers/symbolweightvector.cpp



8.42 TSubstringSymbolStringGreater< T > Class Template Refer-ence



template<class T> class TSubstringSymbolStringGreater< T >

Greater operator for comparing strings of symbol pointers alphabetically.








8.43 TSubstringSymbolWeightGreater< T > Class Template Reference 211

8.43 TSubstringSymbolWeightGreater< T > Class Template Refer-ence



template<class T> class TSubstringSymbolWeightGreater< T >

Greater operator for comparing weights of symbol pointers.









8.44 vectorbitreader Class Reference


Inheritance diagram for vectorbitreader::

vector_bitreader

bitreader





• vectorbitreader(const std::vector< char> & s)

Constructor.

• vectorbitreader(const std::vector< char> & s, sizet size in bits)










• const std::vector< char> & s• size t index


8.44.2.1 vectorbitreader::vector bitreader (const std::vector< char > & s) [inline]

Constructor.


8.44 vectorbitreader Class Reference 213

Sets the number of bits available to the size of the vector.

Parameters:s - the vector to use.


72 : bitreader(8 * _s.size()), s(_s), index(0) { ; }

8.44.2.2 vectorbitreader::vector bitreader (const std::vector< char > & s, sizet sizein bits)[inline]







8.44.3.1 bool vectorbitreader::supports seek () const [inline, virtual]





81 { return true; }

8.44.3.2 char vectorbitreader::read next byte () [inline, protected, virtual]









8.44.3.3 void vectorbitreader::set position (sizet pos) [inline, protected, virtual]






86 { index = pos; }




8.45 vectorbitwriter Class Reference 215

8.45 vectorbitwriter Class Reference


Inheritance diagram for vectorbitwriter::

vector_bitwriter

bitwriter





• vectorbitwriter (std::vector< char> & s)

Constructor.





• std::vector< char> & s


8.45.2.1 vectorbitwriter::vector bitwriter (std::vector < char > & s) [inline]

Constructor.



62 : s(_s) { ; }




8.45.3.1 void vectorbitwriter::write next byte (char c) [inline, protected, virtual]






67 { s.push_back(c); }




8.46 ZeroOrderModeler Class Reference 217

8.46 ZeroOrderModeler Class Reference

#include <zeroorder modeler.hpp >

Inheritance diagram for ZeroOrderModeler::

ZeroOrderModeler


OctaneModeler

OctaneClass



Definition at line 28 of file zeroordermodeler.hpp.


• ZeroOrderModeler()

constructor

• virtual∼ZeroOrderModeler()

destructor





• virtual boolCreateModelUsingStream(OctaneParser∗parserp,bitreader&from)



8.46.2.1 virtual std::string ZeroOrderModeler::GetName () [inline, virtual]







38 {return "ZeroOrderModeler";}

8.46.2.2 virtual std::string ZeroOrderModeler::GetDescription () [inline, virtual]





39 {return "Zero-order Modeler";}

8.46.2.3 bool ZeroOrderModeler::CreateModelUsingStream (OctaneParser∗ parserp, bitreader& from) [virtual]




Definition at line 19 of file zeroordermodeler.cpp.

References SymbolWeightVector::CountSymbolFrequencies(), SymbolWeightVector::EnforceZero-CountFloor(), SymbolWeightVector::NormalizeToProbabilityDistribution(), and OctaneModelerWeight-Vectored::weightvector.

20 {21 // build a model of frequencies22 bool bretv;2324 // calculate symbol frequencies, by asking the weightvector to do our dirty work in conjunction with the parser25 bretv=weightvector.CountSymbolFrequencies(parserp,from);2627 // make sure no symbol has freq count == 0 which would break huffman coder and others28 weightvector.EnforceZeroCountFloor(1);2930 // normalize to probability distribution31 weightvector.NormalizeToProbabilityDistribution();3233 // success34 return bretv;35 }


• modelers/modelerzeroorder/zeroordermodeler.hpp• modelers/modelerzeroorder/zeroordermodeler.cpp


8.47 zleCompressor Class Reference 219

8.47 zleCompressor Class Reference

#include <zle.hpp >

Inheritance diagram for zleCompressor::

zleCompressor

OctaneCompressor

OctaneClass



Any contiguous string of 0 bytes up to 255 characters long will be replaced by a 0 byte followed by countersignifying the length of the string of 0s.

Definition at line 22 of file zle.hpp.


• zleCompressor(bool registerme=false)








• virtual boolDoProtectedCompress(bitreader&s, bitwriter &d)


• virtual boolDoProtectedDecompress(bitreader&s, bitwriter &d)





8.47.2.1 virtual std::string zleCompressor::GetDescription () [inline, virtual]




Definition at line 29 of file zle.hpp.

29 {return "Zero length encoding";}

8.47.2.2 bool zleCompressor::DoProtectedCompress (bitreader & s, bitwriter & d)[protected, virtual]






Definition at line 25 of file zle.cpp.

References bitreader::empty(), bitreader::getbyte(), bitwriter::putbyte(), and bitreader::seekbyte().

26 {27 while (!s.empty())28 {29 unsigned char b = s.get_byte();30 unsigned char c;31 d.put_byte(b);3233 if (!b)34 {35 for (c= 0; c < 255; c++)36 {37 if (s.empty()) break;38 if (s.get_byte())39 {40 s.seek_byte(-1, ios::cur);41 break;42 }43 }44 d.put_byte(c);45 }46 }47 return true;48 }


8.47 zleCompressor Class Reference 221

8.47.2.3 bool zleCompressor::DoProtectedDecompress (bitreader & s, bitwriter & d)[protected, virtual]






Definition at line 50 of file zle.cpp.


51 {52 while (!s.empty())53 {54 unsigned char b = s.get_byte();55 d.put_byte(b);5657 if (!b)58 {59 for (unsigned char c = s.get_byte(); c; c--) d.put_byte(0);60 }61 }62 return true;63 }


• compressors/compressorzle/zle.hpp• compressors/compressorzle/zle.cpp


Chapter 9

Octane File Documentation

9.1 bitio/bitio.hpp File Reference


Bit-wise I/O wrapper classes.

Author:Jibz

Date:2003.08.02

This is the main include file. You should include this instead of the specific implementation files.

Definition in filebitio.hpp.

#include "bitreader.hpp"

#include "bitwriter.hpp"

#include "bitreaders.hpp"

#include "bitwriters.hpp"

#include "file bitreader.hpp"

#include "file bitwriter.hpp"

224 Octane File Documentation

9.2 bitio/bitreader.hpp File Reference



Author:Jibz

Date:2003.08.02

Definition in filebitreader.hpp.

#include <iostream >

#include <string >

#include <cstddef >

#include <climits >

Compounds

• classbitreader



9.3 bitio/bitreaders.hpp File Reference 225

9.3 bitio/bitreaders.hpp File Reference


bitreader specializations.

Author:Jibz

Date:2003.08.02

Definition in filebitreaders.hpp.


#include <vector >

#include <string >


Compounds

• classnull bitreader


• classstreambitreader


• classvectorbitreader


• classstring bitreader


• classarraybitreader




9.4 bitio/bitwriter.cpp File Reference



Author:Jibz

Date:2003.08.02

Definition in filebitwriter.cpp.



9.5 bitio/bitwriter.hpp File Reference 227

9.5 bitio/bitwriter.hpp File Reference



Author:Jibz

Date:2003.08.02

Definition in filebitwriter.hpp.

#include <string >

#include <cstddef >

#include <climits >

Compounds

• classbitwriter




9.6 bitio/bitwriters.hpp File Reference


bitwriter specializations.

Author:Jibz

Date:2003.08.02

Definition in filebitwriters.hpp.


#include <vector >

#include <string >


Compounds

• classnull bitwriter


• classstreambitwriter


• classvectorbitwriter


• classstring bitwriter


• classarraybitwriter



9.7 bitio/file bitreader.hpp File Reference 229

9.7 bitio/file bitreader.hpp File Reference


bitreader specialization for files.

Author:Jibz

Date:2003.08.02

Definition in filefile bitreader.hpp.

#include <fstream >


Compounds

• classfile bitreader




9.8 bitio/file bitwriter.hpp File Reference


bitwriter specialization for files.

Author:Jibz

Date:2003.08.02

Definition in filefile bitwriter.hpp.

#include <fstream >


Compounds

• classfile bitwriter



9.9 coders/coder.cpp File Reference 231

9.9 coders/coder.cpp File Reference


Base Coder class.

Remarks:This is the base class from which all coders are derived.

Author:mouser

Date:2003.07.29

Definition in filecoder.cpp.

#include "coder.hpp"



9.10 coders/coder.hpp File Reference


Base Coder class.

Remarks:This is the base class from which all coders are derived.

Author:mouser

Date:2003.07.29

Definition in filecoder.hpp.

#include "../bitio/bitio.hpp"

#include "../core/octaneclass.hpp"

#include "../modelers/modeler.hpp"

Compounds

• classOctaneCoder



9.11 coders/coderhuffman/huffmancoder.cpp File Reference 233

9.11 coders/coderhuffman/huffmancoder.cpp File Reference


Simple Huffman Coder class.

Remarks:This Coder performs standard, simple huffman coding.

Author:mouser

Date:2003.07.29

Definition in filehuffmancoder.cpp.

#include "huffmancoder.hpp"

#include <assert.h >



9.12 coders/coderhuffman/huffmancoder.hpp File Reference


Simple Huffman Coder class.

Remarks:This Coder performs standard, simple Huffman coding.

TodoThere seems to be a bug in this code which is causing very large bitcodes for rare symbols - they shouldnot get this large afaik. it may have to do with a flaw in the implementation, or be due to the fact thatwe are artifically forcing the tree to keep some primitive ascii characters which actually get frequencycounts of 0.

Author:mouser

Date:2003.07.29

Definition in filehuffmancoder.hpp.

#include "huffmannodes.hpp"

#include "../coder.hpp"

#include "../../modelers/modeler.hpp"

Compounds

• classHuffmanCoder



9.13 coders/coderhuffman/huffmannodes.cpp File Reference 235

9.13 coders/coderhuffman/huffmannodes.cpp File Reference


Huffman node classes.

Remarks:The HuffNode Leaf is the base class for all symbols, and is (despite the elaborate interface), a verylightweight data structure, containing only a symbol id and a numeric weight.

Author:mouser

Date:2003.07.29

Definition in filehuffmannodes.cpp.

#include "huffmannodes.hpp"

Namespaces

• namespacestd



9.14 coders/coderhuffman/huffmannodes.hpp File Reference


Huffman node classes.

Remarks:The HuffNode Leaf is the base class for all symbols, and is (despite the elaborate interface), a verylightweight data structure, containing only a symbol id and a numeric weight.

The techinique to use a priority queue to generate the huffman tree is described in [nelson].

Author:mouser

Date:2003.07.29

Definition in filehuffmannodes.hpp.

#include "../../bitio/bitio.hpp"

#include <string >


#include <iomanip >

#include <vector >

#include <queue >

#include <set >

#include <bitset >

Compounds

• classHuffNode

The base HuffNode class is used to represent both leaf and internal HuffNodes.

• classHuffNode Middle

A derived huffman node for a node in the middle of the tree.

• classHuffNode Leaf

The leaves of the huffman tree contains actual symbol information.

• classPQWeightGreater


Defines

• #defineOCTANE USEJIBZHUFFMANPQGREATER


9.14 coders/coderhuffman/huffmannodes.hpp File Reference 237

• #defineTHuffmanNodeWeightfloat

Precision for representing a huffman weight.

• #defineTHuffmanNodeWeightMAX FLOATMAX

Largest value a weight can hold.

• #defineDEF HUFFMANPQ MAXIMUM CODEBITDEPTH160

Maximum depth that will be seen in the priority queue (huffman tree).

• #defineDEF HUFFMANPQ MAXFUNCBITS 200

simple define which should represent more bits than we could ever see in a symbol (depth of tree) note thatthis size does not effect the per-symbol memory used, its just a one time thing, so we can make it as large aswe want.

Typedefs

• typedef std::priorityqueue< HuffNode ∗, std::vector< HuffNode ∗ >, PQWeightGreater< Huff-Node∗ > > THuffmanNodePriorityQueue

9.14.2 Define Documentation

9.14.2.1 #define DEFHUFFMANPQ MAXIMUM CODEBITDEPTH 160

Maximum depth that will be seen in the priority queue (huffman tree).

Which is equivalent to the maximum bitlength for any code. Setting this is a little tricky because it directlyaffects the memory used by every dictionary entry. an entry needs to reserver a number of bits equal tothe maximum depth of any node in the huffman tree. so this value should be a multiple of 8, and can beestimated from the number of dictionary entries that will be in the final huffman tree. The code will detectif it is not large enough and tell you during building of the codebook. A reasonable value for a dictionaryof several thousand symbols would be something like 104.


Referenced by HuffNodeLeaf::AddBitsToBitWriter(), HuffNodeLeaf::getbitcodestring(), HuffNode-Leaf::traversebuildbitcodes(), and HuffNodeMiddle::traversebuildbitcodes().



9.15 coders/codersample/samplecoder.cpp File Reference


Sample Coder useful as a skeleton for writing new coders.

Note:This coder ignores any statistics that the modeler may provide

Author:mouser

Date:2003.07.29

Definition in filesamplecoder.cpp.

#include "samplecoder.hpp"


9.16 coders/codersample/samplecoder.hpp File Reference 239

9.16 coders/codersample/samplecoder.hpp File Reference


Sample Coder useful as a skeleton for writing new coders.

Note:This coder ignores any statistics that the modeler may provide

Author:mouser

Date:2003.07.29

Definition in filesamplecoder.hpp.

#include "../coder.hpp"

#include "../../modelers/modeler.hpp"

Compounds

• classSampleCoder




9.17 compressors/compressor.cpp File Reference


Compression/Decompression classes.

Remarks:This is the base class from which all compressors are derived.

Author:mouser

Date:2003.07.29

Definition in filecompressor.cpp.

#include "compressor.hpp"

#include "../utilityclasses/timer/timer.hpp"


9.18 compressors/compressor.hpp File Reference 241

9.18 compressors/compressor.hpp File Reference


Compression/Decompression classes.

Remarks:This is the base class from which all compressors are derived.

Author:mouser

Date:2003.07.29

Definition in filecompressor.hpp.


#include "../core/compressormanager.hpp"


#include <string >

Compounds

• classOctaneCompressor




9.19 compressors/compressormtfll/mtfll.cpp File Reference


Move to front encoding.

Remarks:Experimental move to front encoding using a linked list

Author:loco

Date:2003.08.14

Definition in filemtfll.cpp.

#include "mtfll.hpp"

#include <iomanip >

Variables

• mtfllCompressorGloballyInstantiated mtfll (true)


9.20 compressors/compressormtfll/mtfll.hpp File Reference 243

9.20 compressors/compressormtfll/mtfll.hpp File Reference


Move-to-front encoding.

Remarks:Experimental move to front encoding using a linked list

Author:loco

Date:2003.08.14

Definition in filemtfll.hpp.

#include "../compressor.hpp"

Compounds

• classmtfllCompressor




9.21 compressors/compressorsample/samplecompressor.cpp FileReference


Sample compressor.

Remarks:This is a silly sample compressor which does nothing useful.

Note:if you are interested in making a statistical style compressor, then you should see the samplestatcom-pressor.

Author:mouser

Date:2003.08.12

Definition in filesamplecompressor.cpp.

#include "samplecompressor.hpp"

#include <iomanip >

Variables

• SampleCompressorGloballyInstantiated SampleCompressor(true)


9.22 compressors/compressorsample/samplecompressor.hpp File Reference 245

9.22 compressors/compressorsample/samplecompressor.hpp FileReference


Sample compressor.

Remarks:This is a silly sample compressor which does nothing useful.

Note:if you are interested in making a statistical style compressor, then you should see the samplestatcom-pressor.

Author:mouser

Date:2003.08.12

Definition in filesamplecompressor.hpp.


Compounds

• classSampleCompressor




9.23 compressors/compressorsample/samplestatcompressor.cppFile Reference


Sample statistical compressor.

Remarks:This sample compressor is based on statistical compressor, and just creates a 3-component compressorout of sample parser/model/coder.

Author:mouser

Date:2003.07.29

Definition in filesamplestatcompressor.cpp.

#include "samplestatcompressor.hpp"

Variables

• SampleStatCompressorGloballyInstantiated SampleStatCompressor(true)


9.24 compressors/compressorsample/samplestatcompressor.hpp File Reference 247

9.24 compressors/compressorsample/samplestatcompressor.hppFile Reference


Sample statistical compressor.

Remarks:This sample compressor is based on statistical compressor, and just creates a 3-component compressorout of sample parser/model/coder.

Author:mouser

Date:2003.07.29

Definition in filesamplestatcompressor.hpp.

#include "../compressor statistical.hpp"

#include "../../coders/coder sample/samplecoder.hpp"

#include "../../parsers/parser sample/sampleparser.hpp"

#include "../../modelers/modeler sample/samplemodeler.hpp"

Compounds

• classSampleStatCompressor




9.25 compressors/compressorstatistical.cpp File Reference


Skeleton for statistical compressors.

Remarks:This is a base class available for use in building statistical coders which use a parser+model+codercomponent architecture. You can’t use this ”as-is”, you need to derive from it and fill in the constructorat a minimum, to create the appropriate model/parser/coder.

Author:mouser

Date:2003.07.29

Definition in filecompressorstatistical.cpp.

#include "compressor statistical.hpp"


9.26 compressors/compressorstatistical.hpp File Reference 249

9.26 compressors/compressorstatistical.hpp File Reference


Skeleton for statistical compressors.

Remarks:This is a base class available for use in building statistical coders which use a parser+model+codercomponent architecture. You can’t use this ”as-is”, you need to derive from it and fill in the constructorat a minimum, to create the appropriate model/parser/coder.

Author:mouser

Date:2003.07.29

Definition in filecompressorstatistical.hpp.

#include "../modelers/modeler.hpp"

#include "../parsers/parser.hpp"

#include "../coders/coder.hpp"


#include "compressor.hpp"


Compounds

• classOctaneCompressorStatistical




9.27 compressors/compressorsubstrhuff/substrhuff.cpp File Refer-ence


Compressor that uses the SubString Parser, a 0-Order Model and a Huffman Coder.

Remarks:This compressor basically does nothing but coordinate these three components.

Author:mouser

Date:2003.07.29

Definition in filesubstrhuff.cpp.

#include "substrhuff.hpp"

Variables

• SubStrHuffCompressorGloballyInstantiated SubStrHuffCompressor (true)


9.28 compressors/compressorsubstrhuff/substrhuff.hpp File Reference 251

9.28 compressors/compressorsubstrhuff/substrhuff.hpp File Refer-ence


Compressor that uses the SubString Parser, a 0-Order Model and a Huffman Coder.

Remarks:This compressor basically does nothing but coordinate these three components.

Author:mouser

Date:2003.07.29

Definition in filesubstrhuff.hpp.

#include "../compressor statistical.hpp"

#include "../../coders/coder huffman/huffmancoder.hpp"

#include "../../parsers/parser substring/substringparser.hpp"

#include "../../modelers/modeler zeroorder/zeroorder modeler.hpp"

Compounds

• classSubStrHuffCompressor

The SubStrHuff Compressor is a basic Statistical compressor, that instantiates a Zero-order modeler, with ahuffman coder, and a SubString parser.



9.29 compressors/compressorzle/zle.cpp File Reference


Zero length encoding.

Remarks:Loco’s first attempt to make a compressor for octane

Author:loco

Date:2003.08.13

Definition in filezle.cpp.

#include "zle.hpp"

#include <iomanip >

Variables

• zleCompressorGloballyInstantiated zle (true)


9.30 compressors/compressorzle/zle.hpp File Reference 253

9.30 compressors/compressorzle/zle.hpp File Reference


Zero length encoding.

Remarks:Loco’s first attempt to make a compressor for octane

Author:loco

Date:2003.08.13

Definition in filezle.hpp.


Compounds

• classzleCompressor




9.31 core/compressormanager.cpp File Reference


Compression/Decompression manager classes.

Remarks:These classes holds a collection of Compressor derived classes and is used for runtime selection ofwhich algorithm to test.

Author:mouserJibz

Date:2003.07.29

Definition in filecompressormanager.cpp.

#include "compressormanager.hpp"


#include <iomanip >

Variables

• CompressorManager∗ global compressormanagerp

global single instance of theCompressorManager; all compressors will register with this global instance.


9.32 core/compressormanager.hpp File Reference 255

9.32 core/compressormanager.hpp File Reference


Compression/Decompression manager classes.

Remarks:These classes holds a collection of Compressor derived classes and is used for runtime selection ofwhich algorithm to test.

Author:mouserJibz

Date:2003.07.29

Definition in filecompressormanager.hpp.

#include "../compressors/compressor.hpp"

#include <string >

#include <vector >


Compounds

• classCompressorManager


• classCompressorManagerSingletonInsurer

This is a sneaky little class which allows us to create self-registering compressors.

Variables

• CompressorManager∗ global compressormanagerp

global single instance of theCompressorManager; all compressors will register with this global instance.



9.33 core/octane.hpp File Reference


Version information and main help page comments.

Remarks:This file contains the text for the doxygen generated help page, and the version information for thecurrent Octane build.

Author:Jibzmouser

Date:2003.08.12

Definition in fileoctane.hpp.

Defines

• #define OCTANE VERSIONSTRING ”OCTANE v1.01.20 - may 19, 2004 -http://octane.sourceforge.net”


9.34 core/octaneclass.cpp File Reference 257

9.34 core/octaneclass.cpp File Reference


Root class for most derived Octane classes.

Remarks:It defines some generic methods available to all octane derived parsers, models, coders, etc.

Author:mouser

Date:2003.07.29

Definition in fileoctaneclass.cpp.

#include "octaneclass.hpp"



9.35 core/octaneclass.hpp File Reference


Root class for most derived Octane classes.

Remarks:It defines some generic methods available to all octane derived parsers, models, coders, etc.

Author:mouser

Date:2003.07.29

Definition in fileoctaneclass.hpp.


#include <string >

#include <stdlib.h >

Compounds

• classOctaneClass



9.36 modelers/modeler.cpp File Reference 259

9.36 modelers/modeler.cpp File Reference


Base Modeler class.

Remarks:This is the base class from which all modelers are derived.

Author:mouser

Date:2003.07.29

Definition in filemodeler.cpp.

#include "modeler.hpp"



9.37 modelers/modeler.hpp File Reference


Base Modeler class.

Remarks:This is the base class from which all modelers are derived.

Author:mouser

Date:2003.07.29

Definition in filemodeler.hpp.

#include "symbolweightvector.hpp"



#include "../coders/coder.hpp"


Compounds

• classOctaneModeler



9.38 modelers/modelersample/samplemodeler.cpp File Reference 261

9.38 modelers/modelersample/samplemodeler.cpp File Reference


A sample modeler which basically assigns fixed equal probabilities to every symbol.

Author:mouser

Date:2003.07.29

Definition in filesamplemodeler.cpp.

#include "samplemodeler.hpp"



9.39 modelers/modelersample/samplemodeler.hpp File Reference


A sample modeler which basically assigns fixed equal probabilities to every symbol.

Author:mouser

Date:2003.07.29

Definition in filesamplemodeler.hpp.

#include "../modeler weightvectored.hpp"

Compounds

• classSampleModeler



9.40 modelers/modelerweightvectored.cpp File Reference 263

9.40 modelers/modelerweightvectored.cpp File Reference


Derived modeler which uses a simple vector of probabilities.

Remarks:This is a derived (but still abstract) modeler class which can serve as the foundation for common simplestatistic modelers which use a single vector of weights.

Author:mouser

Date:2003.07.29

Definition in filemodelerweightvectored.cpp.

#include "modeler weightvectored.hpp"



9.41 modelers/modelerweightvectored.hpp File Reference


Derived modeler which uses a simple vector of probabilities.

Remarks:This is a derived (but still abstract) modeler class which can serve as the foundation for common simplestatistic modelers which use a single vector of weights.

Author:mouser

Date:2003.07.29

Definition in filemodelerweightvectored.hpp.

#include "modeler.hpp"

Compounds

• classOctaneModelerWeightVectored

This is a derived modeler class which has some default functions for modelers which work with a vector ofweights as probabilities (very common).


9.42 modelers/modelerzeroorder/zeroorder modeler.cpp File Reference 265

9.42 modelers/modelerzeroorder/zeroorder modeler.cpp File Ref-erence


A simple zero-order modeler, which just uses summary frequencies of symbols.

Author:mouser

Date:2003.07.29

Definition in filezeroordermodeler.cpp.

#include "zeroorder modeler.hpp"



9.43 modelers/modelerzeroorder/zeroorder modeler.hpp File Ref-erence


A simple zero-order modeler, which just uses summary frequencies of symbols.

Author:mouser

Date:2003.07.29

Definition in filezeroordermodeler.hpp.

#include "../modeler weightvectored.hpp"

Compounds

• classZeroOrderModeler



9.44 modelers/symbolweightvector.cpp File Reference 267

9.44 modelers/symbolweightvector.cpp File Reference


Helper classes for working with probability/weight vectors.

Remarks:

• These classes are basically just wrappers around vectors of numbers which represent the probabilityor occurrence count of symbols.

• A typical use for one of these classes is as the vector of symbol probabilities which is updated by themodeller and given to the coder in order to perform arithmetic coding.

• Alternatively it could be used as a vector of frequencies to be given to a huffman coder.

• This class implements the standard octane API, so can save/load state and report on memory usage.

Author:mouser

Date:2003.07.29

Definition in filesymbolweightvector.cpp.

#include "symbolweightvector.hpp"



9.45 modelers/symbolweightvector.hpp File Reference


Helper classes for working with probability/weight vectors.

Remarks:

• These classes are basically just wrappers around vectors of numbers which represent the probabilityor occurrence count of symbols.

• A typical use for one of these classes is as the vector of symbol probabilities which is updated by themodeller and given to the coder in order to perform arithmetic coding.

• Alternatively it could be used as a vector of frequencies to be given to a huffman coder.

• This class implements the standard octane API, so can save/load state and report on memory usage.

Author:mouser

Date:2003.07.29

TodoI considered making a base class and then derived template classes for integer and floating point weightvectors. however I’m not sure the memory advantage for integers (the rare case) would offset the speedhit to be paid for virtual functions, and the inconvenience of dealing with different types for differentcoders/modelers.

Definition in filesymbolweightvector.hpp.





Compounds

• classSymbolWeightVector

The SymbolWeightVector class is a helper class which holds a vector of probabilities, typically associatedwith a modeler.

Defines

• #defineTSymbolWeightVectorWeightfloat

what kind of precision should we use for the weights (freqs/probabilities)? we could use a typedef here, butthat complicates things when we want to write/read with streams


9.46 octanetester/octanetester.cpp File Reference 269

9.46 octanetester/octanetester.cpp File Reference


Classes for testing octane functions and compressors.

Remarks:TheOctaneTesterclass contains functions for testing compressors and providing a commandline andinteractive interface to octane functions.

Author:mouserJibz

Date:2003.07.29

Version:2004.05.19 mouser [+] added code to report kb/s compression and decompression speeds when testingfiles in theOctaneTester.

Definition in fileoctanetester.cpp.

#include "octanetester.hpp"

#include "../utilityclasses/stringparser/stringparser.hpp"

#include "../utilityclasses/timer/timer.hpp"


#include <iomanip >

#include <fstream >

#include <cstdio >



9.47 octanetester/octanetester.hpp File Reference


Classes for testing octane functions and compressors.

Remarks:TheOctaneTesterclass contains functions for testing compressors and providing a commandline andinteractive interface to octane functions.

Author:mouserJibz

Date:2003.07.29

Definition in fileoctanetester.hpp.

#include "../core/octane.hpp"


#include <string >

Compounds

• classOctaneTester

The OctaneTester class contains functions for testing compressors and providing a commandline and inter-active interface to octane functions.


9.48 octanetester/standalonetestermain.cpp File Reference 271

9.48 octanetester/standalonetestermain.cpp File Reference


Standalone tester file with intmain(), useful for testing compressors through commandline or interactively.

Remarks:The main procedure initializes a global OctaneCompressorManager object, and either sends it com-mandline arguments as commands, or initiates an interactive loop of executing commands from user.

Author:mouser

Date:2003.07.29

Definition in filestandalonetestermain.cpp.

#include "octanetester.hpp"

Defines

• #defineOCTANE DEFAULTINIFILENAME ”octane.ini”

Functions

• int main (int argc, char∗∗argv)



9.49 parsers/parser.cpp File Reference


The base parser class which defines a generic API for statistical parsers.

Author:mouser

Date:2003.07.29

Definition in fileparser.cpp.

#include "parser.hpp"


9.50 parsers/parser.hpp File Reference 273

9.50 parsers/parser.hpp File Reference


The base parser class which defines a generic API for statistical parsers.

Author:mouser

Date:2003.07.29

Definition in fileparser.hpp.


Compounds

• classOctaneParser




9.51 parsers/parserbitlevel/bitparser.cpp File Reference


A bit-level parser that can be told how many bits per symbol, and returns numbers converted bit chunks.

Remarks:This is a sample parser that can be used as the skeleton for building new parsers. It is a simplecharacter parser - that is, it builder 256 symbols, one for each character plus one for end-of-stream,and just returns the ascii character # it reads.

Author:mouser

Date:2004.01.25

Todo: this is not finished yet

Definition in filebitparser.cpp.

#include "bitparser.hpp"

#include <string >


9.52 parsers/parserbitlevel/bitparser.hpp File Reference 275

9.52 parsers/parserbitlevel/bitparser.hpp File Reference


A bit-level parser that can be told how many bits per symbol, and returns numbers converted bit chunks.

Remarks:the basic idea is you can set bitlength from 1 to 99, and it will parse the input in chunks of bitlengthbytes, returning a symbol # corresponding to the index of the bitstring in the list of possible bit patternsof bitlength.

Author:mouser

Date:2004.01.25

Todo: this is not finished yet

Definition in filebitparser.hpp.

#include "../parser.hpp"


Compounds

• classBitParser




9.53 parsers/parsersample/sampleparser.cpp File Reference


A simple sample parser for use in statistical compression.


Author:mouser

Date:2003.07.29

Definition in filesampleparser.cpp.

#include "sampleparser.hpp"

#include <string >


9.54 parsers/parsersample/sampleparser.hpp File Reference 277

9.54 parsers/parsersample/sampleparser.hpp File Reference


A simple sample parser for use in statistical compression.


Author:mouser

Date:2003.07.29

Definition in filesampleparser.hpp.



Compounds

• classSampleParser


Defines

• #defineSampleParserEndOfStreamSYMBOL 255



9.55 parsers/parsersubstring/substringparser.cpp File Reference


A flexible parser which can creates a symbol dictionary of substrings.

Remarks:This is a general purpose class which is designed to be used with any compressor that wants to parsefiles and count occurrences of certain characters/words/Substrings. Parameters can be set to regulatehow symbols are extracted. Several utility functions are provided so that this class can by default domuch of the work in parsing files and pruning dictionaries.

Author:mouser

Date:2003.07.29

Version:2004.05.19 mouser [-] fixed bug in smartlookup mode of SubStringParser that could cause crashingwhen incoming stream matched last symbol in set.

Definition in filesubstringparser.cpp.

#include "substringparser.hpp"


#include <iomanip >

#include <string >


9.56 parsers/parsersubstring/substringparser.hpp File Reference 279

9.56 parsers/parsersubstring/substringparser.hpp File Reference


A flexible parser which can creates a symbol dictionary of substrings.

Remarks:This is a general purpose class which is designed to be used with any compressor that wants to parsefiles and count occurrences of certain characters/words/Substrings. Parameters can be set to regulatehow symbols are extracted. Several utility functions are provided so that this class can by default domuch of the work in parsing files and pruning dictionaries.

Author:mouser

Date:2003.07.29

Definition in filesubstringparser.hpp.



#include <vector >

#include <set >


Compounds

• classSubstringSymbol

The SubString Symbol class represents the primitive symbols used by the SubString parser.

• classTSubstringSymbolWeightGreater

Greater operator for comparing weights of symbol pointers.

• classTSubstringSymbolStringGreater

Greater operator for comparing strings of symbol pointers alphabetically.

• classSubstringParser


Defines

• #defineDictionary ParseModeGREEDY 0• #defineDictionary ParseModeLFF 1• #defineDictionary ParseModeDEEP 2• #defineTSubStrParserWeight MAX UINTMAX• #defineSubStrHuff BiggestSubStringLen100



Typedefs

• typedef unsigned intTSubStrParserWeight


9.57 utilityclasses/stringparser/stringparser.hpp File Reference 281

9.57 utilityclasses/stringparser/stringparser.hpp File Reference



Author:Jibz

Date:2003.08.01

Definition in filestringparser.hpp.

#include <string >

Compounds

• classstringparser




9.58 utilityclasses/timer/scopetimer.hpp File Reference



Author:Jibz

Date:2003.07.30

Definition in filescopetimer.hpp.


#include <string >

#include "timer.hpp"

Compounds

• classOctaneScopeTimer



9.59 utilityclasses/timer/timer.hpp File Reference 283

9.59 utilityclasses/timer/timer.hpp File Reference



Author:Jibz

Date:2003.07.30

Definition in file timer.hpp.

#include <ctime >

Compounds

• classOctaneTimer



Chapter 10

Octane Page Documentation

10.1 Introduction to Octane

10.1.1 What is Octane?

The Open Compression Toolkit is a modern object-oriented framework for developing and testing com-pression algorithms.

While we have tried to make the code as efficient as possible, the primary focus of Octane has been on de-veloping a well-organized and flexible system for experimenting with a variety of compression algorithms.

10.1.2 Not Just for File Compression

Most existing compression algorithms are designed specifically for file compression, where a large file (orgroup of files) is compressed and the result is a ”stand-alone” compressed file. The underlying assumptionin such operations is that sufficient statistics/redundancy can be gathered from the file itself.

However, there are compression tasks with different requirements than file compression. For example,consider the problem of compressing text messages in an on-line chat program. In this case, individualmessages don’t contain enough redundancy to yield effective compression. Instead, one may want to gatherstatistics ”off-line”, on some large corpus (for example on a collection of common English text), and thenuse the ”pre-computed” statistics to compress short text messages. Difficult issues remain, such as dealingwith messages that may deviate from the pre-computed statistical model (for example what happens if youbuild a model on English text and the users begin to chat in Chinese?).

While the Octane classes are well suited for file compression, a major focus of the development of Octanehas been on supporting alternate, less-common compression tasks such as the on-line chat example above,which can be difficult to implement with existing off-the-shelf compression algorithms.

10.1.3 The Modern Statistical Approach

Most compression algorithms can be classified as either heuristic dictionary approaches, or statistical ap-proaches.

In dictionary-based compression algorithms, such as the popular Lempel-Ziv derivatives [bell], a dictionaryof phrases or strings is built during compression, and references to these dictionary entries can be used inplace of the original text. Redundancy in the files leads to higher compression.

286 Octane Page Documentation

In statistical compression, the goal is to explicitly model the probabilities of symbols(characters) duringa parsing of the file, and use these predicted probabilities to minimise the bits needed to code observedevents.

Statistical compression is based on the principles of information theory [shannon], which describe theminimal bits needed to transmit a message based on the ”surprise” that the message represents; a perfectlypredictable sequence of symbols requires no bits to code, while describing which outcome occurred outof a set of equally likely possibilities requires the most bits. In statistical compression, the better theprediction of symbols, the less bits are needed to encode the message (note that in statistical compression,the predictions do not have to be perfectly accurate in order to yield significant compression).

The modern statistical approach to compression [witten] describes the compression job as being performedby two semi-independent components.

The first component is theModel, whose job it is to model (or predict) the probability distribution ofsymbols over time. There are a wide range of algorithms that can be used to model symbol probabilities,from simple static frequency counting, to complex Markovian models, to complex heuristic predictionarchitectures.

The second component is theCoder. It is the job of the coder to use the probability distributions fromthe Model to efficiently generate output codes for the compressed file (and to decode these bits duringdecompression in order to decompress files). In a given context, symbols with high likelyhood of appearingare coded with fewer bits than symbols that are unlikely to occur.

The most well known algorithm for implementing a coder is the Huffman coder [apiki], which is sometimesdescribed as ”optimal” in the case where each symbol must be assigned an integral number of bits.

More recently [moffat], efficient implementations of an algorithm known as Arithmetic Coding have be-come more common. While more complicated than Huffman coding, Arithmetic coding is a theoreticallyoptimal coder, and so is becoming the algorithm of choice in statistical compression.

Octane contains a complete set of classes to support the modern statistical approach to compression, pro-viding semi-independent components for Modeling and Coding. We have extended this idea of separatingcomponents by introducing a third component, called aParser. The job of the parser (a better but lessobvious term to hose outside of compiler design might be lexer) is to ’tokenize’ the input file(stream) intoa set of symbols for processing by the Modeler and Coder. In the simplest case, a parser would simplyread characters from a file and assign them a symbol number based on their ascii character code. In morecomplex cases, a parser might build a symbol set of words or common sub-strings.

Part of the appeal of modern statistical compression comes from the separation of these components, sothat replacement Parsers, Modelers, Coders can be interchanged in a plug-and-play fashion. That is, oncewe have a good modeler and coder, we should be able to use the components with a character based parser,and also with a word based parser, etc.

10.1.4 Organization of The Octane Toolkit Classes

The Octane Tookit consists of a hierarchy of compression related classes and some auxiliary supportclasses. Although Octane is designed to allow a great deal of flexibility in how compression algorithms arewritten, a common interface between different components is the bitio io classes.

Rather than the the traditional C language file reading/writing operations seen in many compression algo-rithms, all Octane functions which perform input and output do so by operating onbitio objects. The bitioclasses provide an extremely flexible and transparent interface to working with files, memory, or strings.They provide the standard character-level reading and writing, as well as the bit-level input and outputwhich is needed for compression routines, and additional convenience routines for writing other primitivedata types. In addition to the simplification of code that comes with using the bitio classes, they insure thata compression function will work unchanged on files, direct memory, or C++ strings.


10.1 Introduction to Octane 287

Octane uses Object-Oriented Programming (OOP) principles throughout its design. To write a new com-pression algorithm, you create a new object class derived from theOctaneCompressorclass. All Com-pressors are therefore guaranteed to have a standard interface that they expose to the rest of the system.The base Compressor class also automatically perform some bookkeeping operations behind the scenes,such as timing compression/decompression algorithms and calculating compression ratios, and providingan interface for exposing certain parameters for user adjustment.

At a higher level of organization, Octane uses the Factory Pattern [gamma], to manage the collection ofimplemented compressors, and dynamically create compressor’s when it needs to. In this way, Octaneis not simply a collection of independent compressors that are instantiated by a target application. Com-pressor classes include code that automatically registers them with aCompressorManagerobject, whichmanages a list of all known compressors, and is able to dynamically instantiate compressors on demand.TheCompressorManagerprovides a higher-level interface to a collection of compressors.

While the Octane Toolkit can support any kind of compression algorithm, it includes a a substantial set ofclasses specifically designed for modern statistical compression. These classes include the baseOctane-Modeler, OctaneCoder, andOctaneParserclasses, as well as sample derivations of each of these classeswhich may serve as the basis for writing new compressors.

Octane also includes substantial code for helping you to test and evaluate new compressors. AnOctane-Testerclass utility shows how to work with the OctaneManager to dynamically create and use compressors,and parameterize them. It can be used to interact with new compressors, compress/decompress files andstrings, get timing information, etc. It can be run in a non-interactive command-line mode, or in an inter-active console mode.



10.2 A Simple Walkthrough with the Octane Tester Console

10.2.1 Starting the Octane Tester Console

An important part of the Octane toolkit is a standalone utility to help you interact with compression al-gorithms, and anOctaneTesterclass to support this utility. Using theOctaneTesterand theCompressor-Managerclass, the testing utility automatically gains access to new compressor added to the library, andthe Octane API provides a standard interface for Compressors to expose parameters that can be queriedand set through the tester utility.

Here we provide a simple walkthrough to show you how to use the interactive tester utility using a com-pressor (SubStrHuffCompressor) that is part of the core Octane Toolkit.

Begin by runningoctane.exe , which will launch a console program:

OCTANE v1.01.15 - august 8, 2003 - http://octane.sourceforge.netCommands are case-sensitive (to use as commandline arguments separate with ’;’)

Compression and Decompression’compress $inputfile [$outputfile]’ to compress a file+compressor.’decompress $inputfile [$outputfile]’ to decompress a file+compressor.’compressr $inputfile [$outputfile]’ to compress a file; save only raw data.’decompressr $inputfile [$outputfile]’ to decompress a file of raw data.’test $text’ to test compress and decompress an arbitrary line of $text.’testfile $inputfile’ to test compress and decompress a file.

Compressor Creation, Configuration, and File Management’compressors’ to get a list of compressors and compressor classes.’create $compressorclass’ to instantiate a compressor class and select it.’setinfo $name [$guid]’ to set name and GUID for instantiated compressor.’select [$name|$num|G$guid]’ to select an instantiated compressor.’makestate $inputfile’ to create state data using $inputfile.’save $outputfile’ to save current compressor (set GUID with setinfo).’load $inputfile’ to load/instantiate a previously saved compressor.’delete [$name|$num|G$guid]’ to delete an instantiated compressor.

Miscellaneous Commands’debug’ to show debugging info for current compressor/dictionary.’parameters’ to see a list of available runtime parameters.’set $parametername $value’ to set a runtime parameter to some value.’run $commandfile’ to run octane commands from a file.’help’ to show these instructions.’quit’ to exit this program.

Registered Compressor Types: StatSample SubStrHuffInstantiated Compressors: (none)>

When the Octane tester utility launches it displays a brief list of commands and then lists the ’RegisteredCompressor Types’ (in our case there are 2 compressors registered), as well as any ’Instantiated Compres-sors’ (we start with no compressors instantiated). You can redisplay this help at any time by typing the’help’ command (then hit enter).

It’s important to understand how the tester utility, as well as the underlyingCompressorManagerclass,differentiates between Registered Compressor Types and Instantiated Compressors. The Registered Typesreflect different Compressor classes written by developers in C++ using the Octane Toolkit. InstantiatedCompressors are compressors that are built from (or instantiated from) these Registered Types. You couldinstantiate many compressors based on the same Registered Compressor Type. A registered compressorcan be named, parameterized, and trained differently. Instantiated compressors can be saved to and re-stored from a file. All compression and decompression is done withinstantiated compressors, which arethemselves built from Registered Compressor types.


http://octane.sourceforge.net

10.2 A Simple Walkthrough with the Octane Tester Console 289

Ok, so before we can compress anything, we need to instantiate a compressor to use. If we had previouslycreated and saved one, we could load it now, but let’s create and train a new one.

There are two Registered Compressor Types for us to choose from. The StatSample type is not really usefulfor compression (it’s a sample compressor made to show the simplest skeleton needed to make your owncompressor), so we will create an instance of the SubStrHuff compressor, which is a statistical compressorwhich calculates static frequencies of the most popular substrings found in a file, and uses huffman codingto assign bitcodes to each substring symbol. To create the compressor type (the test utility is case sensitive):create SubStrHuff and hit enter:

>create SubStrHuffcompressor SubStrHuff.unnamed created and selected.

We can see confirmation that we’ve created a new compressor, and for a little more information on theRegistered Compressor Types, usecompressors:

>compressorsRegistered Compressor Types---------------------------

StatSample Sample Statistical CompressorSubStrHuff SubString Huffman Compressor

Instantiated Compressors------------------------

*0 SubStrHuff.unnamed SubString Huffman Compressor (NOGUID)

We now have a new instantiated compressor, SubStrHuff.unnamed, with index 0 and no GUID; the asteriskindicates that this is the currently selected compressor.

We will not use for compressor names or GUIDs in this walkthrough. Their purpose is to provide a globallyunique # associated with a compressor in order to specify compressor indentities in very short messages.

Let’s get a list of the parameters supported by the SubStrHuff compressor, as well as their current values,by typingparameters:

>parametersmaxbuildsymbols 200000 parsing dictionary pruned to not exceed this sizemaxsymbols 2000 final dictionary pruned to reach this sizemaxSubstring 12 max size of Substrings in dictionaryonlywords 0 only add whole word symbols? (bool)spanwords 0 encode symbols that span words? (bool)crends 1 count CRs as end-of-stream symbols? (bool)minweight 10 minimum threshold weight for final pruningsmartlookup 1 when true, use lower bound stl dictionary searchparsemode 0 parse heuristic (0=greedy 1=lff 2=deep&slow)recalculations 0 recompression test&prune iterations

Each compressor type can have its own custom parameters. Let’s change one of the parameters, maxSub-String, to the value 10, by typingset maxSubstring 10:

>set maxSubstring 10ok.

Now let’s ”train” this compressor. Some compressors start out ready to perform compression, while othersexpect to have their state built (this is sometimes thought of as ’priming’ or ’training’). For example the



traditional file compression algorithms that people are most familiar with, such as the zip algorithm, don’tneed any pre-training, but the SubStrHuff algorithm is designed to first be trained on some large corpusof text, and then subsequently be used to compress files using the statistical information generated duringtraining.

We will train the SubStrHuff compressor using the book1.txt file, an 800k text file from the Calgary Corpus(which by default is already in the bin directory of Octane along with the octane.exe utility), by typingmakestate book1.txt:

>makestate book1.txtComputing state data using SubStrHuff.unnamed:Total symbols in dictionary before pruning: 94273After Stage 1 Pruning (removed scores below 10), symbols: 14332After Stage 2 Pruning (removed redundant prefixes), symbols: 10803After Stage 3 Pruning (killed worst scores to reach target), symbols: 1991

execution time: 4.637 seconds.

The SubStrHuffCompressordisplay some information about the training process, and theCompressor-Managerprovides us with some timing information.

Now that the compressor is trained we can begin to use it to compress files and text.

Let’s begin by testing a short text string:

>test Hello my friend, how are you feeling today?Testing SubStrHuff.unnamed..

input length: 43output length: 22

compressed: ...original: Hello my friend, how are you feeling today?

decompressed: Hello my friend, how are you feeling today?

The test command uses the currently selected instantiated compressor to compress and then decompress thestring specified, and check to make sure that the decompression yielded the original string. The originaland compressed size are listed. Note that the compressed string is a binary string that may not displayproperly - it is used only as a visual aid.

Now let’s try to compress a file. There are actually two ways to compress a file from the Octane testerutility. The first way is to produce a standalone file which contains both the raw compressed data, as wellas the complete state information for the compressor used to generate it. This produces a bigger file, butone that can be decompressed without any other information (like a standard zip file). The other way is toproduce an output file with only the raw compressed data. This yields a smaller data file, but to decompressthe file, the compressor must first be recreated. Which method you use will depend on your application,and whether there is a benefit to pre-exchanging compressor state information. For compressors which donot require a separate training stage, there is no difference between these two methods.

Let’s begin by compressing book1.txt itself, as a stand-alone file, by typingcompress book1.txt:

>compress book1.txtCompressing with SubStrHuff.unnamed:

original size: 785393 bytes (book1.txt)compressed size: 355178 bytes (book1.txt.compressed)

est. cost of header: 19922 bytes.execution time: 0.691 seconds.

compression ratio: 45.223% (3.61784 bits per byte).

After compression we are shown a summary of the compression details, as well as an estimate of theamount of space used by the compressor state information (header cost).


10.2 A Simple Walkthrough with the Octane Tester Console 291

We can decompress the created file using the decompress command:

>decompress book1.txt.compressedRecreated temporary compressor from input stream.Decompressing with SubStrHuff.unnamed:

original size: 331216 (book1.txt.compressed)decompressed size: 785393 (book1.txt.compressed.decompressed)

execution time: 0.22 seconds.compression ratio: 42.172% (3.37376 bits per byte).

Note that the tester says ”Recreated temporary compressor from input stream.” That is because when wecompressed the file we saved the entire state of the compressor with the compressed data, so a temporarycompressor was actually recreated to perform the decompression and then destroyed on completion.

Let’s save this compressor so we can use it another day, by typingsave mysubstr.oct:

>save mysubstr.octSaved compressor file ’mysubstr.oct’.

state file size: 23961 bytes.

Quit the tester utility by typing ’quit’.

And now let’s restart the tester utility and make sure the saving and loading of compressors actually works.

Type load mysubstr.octto load our previously trained and saved compressor:

>load mysubstr.octLoaded compressor from file ’mysubstr.oct’.

state file size: 23927 bytes.execution time: 0 seconds.

We can list the parameters of the compressor to verify that it has remembered all the parameters we set lasttime:

>parametersmaxbuildsymbols 200000 parsing dictionary pruned to not exceed this sizemaxsymbols 2000 final dictionary pruned to reach this sizemaxSubstring 10 max size of Substrings in dictionaryonlywords 0 only add whole word symbols? (bool)spanwords 0 encode symbols that span words? (bool)crends 1 count CRs as end-of-stream symbols? (bool)minweight 10 minimum threshold weight for final pruningsmartlookup 1 when true, use lower bound stl dictionary searchparsemode 0 parse heuristic (0=greedy 1=lff 2=deep&slow)recalculations 0 recompression test&prune iterations

Ok, now let’s retest the string we tried before to see if it has remembered its training:

>test Hello my friend, how are you feeling today?Testing SubStrHuff.unnamed..

input length: 43output length: 22

compressed: ...original: Hello my friend, how are you feeling today?

decompressed: Hello my friend, how are you feeling today?



Good, same results. Lastly, lets make a raw compressed file from book2.txt, another file from the Calgarycorpus:

>compressr book2.txtCompressing with SubStrHuff.unnamed:

original size: 54083 bytes (book2.txt)compressed size: 22875 bytes (book2.txt.compressed)


Notice that this time we used the ’compressr’ command instead of the ’compress’ command, so that thecompressor state information is∗not∗ saved inside the output file, resulting in less wasted space.

To decompress the raw compressed file we use the decompressr command:

>decompressr book2.txt.compressedDecompressing with SubStrHuff.unnamed:

original size: 22875 (book2.txt.compressed)decompressed size: 54083 (book2.txt.compressed.decompressed)


That’s the end of the walkthrough for the Octane tester utility.


10.3 The Octane Compressor API - How to Write Your Own Octane Compressor Class 293

10.3 The Octane Compressor API - How to Write Your Own OctaneCompressor Class

10.3.1 Writing Compressors for Octane

The main design goal of the Octane Open Compression Toolkit was to make it easy for programmers todevelop and test well-behaved compression algorithms, without having to worry about many of the lowlevel issues such as bitwise io and file operations. Octane provides an object-oriented framework whichguarantees a standard interface to every compressor, which means that any compression algorithm youwrite becomes transparently available to all applications that uses the Octane Toolkit.

When you write a new compression-decompression algorithm, you do so by implemented a class derivedfrom the baseOctaneCompressorclass. A detailed list of the member functions which make up the OctaneCompressor API (program interface) is listed in the documentation for theOctaneCompressorclass. Thissection is an attempt to explain the details of this API and describe the process by which you would goabout writing your own custom Compressor class.

In addition to this document, we have provided a complete set of files for implementing a bare-minimumstatistical compressor, which you can use as the skeleton for creating new compressors (seeSample-Compressor, SampleModeler, SampleCoder, SampleParser).

10.3.2 The Octane API - Basic Runtime Type Information

Almost all classes in Octane derive from the baseOctaneClassclass. This class defines some basic virtualfunctions which are designed to provide some RTTI (run-time type information) to other classes. Yourcompressor class should at the minimum implement the GetDescription() member function, which returnsa short (best if less than 60 character) description of the compressor class. Optionally you might provide amore elaborate description of your compressor class by implementing the GetHelpInformation() function.

10.3.3 The Octane API - State Information

TheOctaneClassinterface also defines functions for reporting memory/disk usage and debugging informa-tion. Its important that your compressor accurately report its memory usage, via GetMemoryUsed(), andthe disk space required to store the state of your compression, via GetDiskspaceUsed(), so that accurateinformation is displayed when reporting performance of the compressor. You can optionally implementthe ShowDebuggingInfo() function in order to display some internal details of the current compressor onrequest (for example showing the currently built dictionary).

In order to provide a uniform interface for working with compressor parameters, theOctaneClassdefines aset of functions for setting and querying parameters which you define. By exposing compressor parameterswith this interface you provide an easy way to test various settings either through command-line argumentsor through the interactive tester console. Three functions make up the parameter interface: GetParameters-Information() which should list all parameters, their descriptions, and their current values, SetDefault-Parameters() which should reset the parameters to their default values, and SetParameter() which sets aparameter to a given value. If your compressor has no adjustable parameters, you do not need to implementthese functions.

If your compressor has any user modifiable parameters and/or contains any state information (i.e. if itbuilds state information by processing a training file), then you need to implement the LoadState() andSaveState() functions, which are responsible for saving and loading state information to a bitstream. Thebitio classes provide convenient functions for saving and loading various data types, and do not require youto worry about the actual source or destination (which is typically a file).



10.3.4 The OctaneCompressor API - Auto Registration

If you examine the sample statistical compressor class (samplestatcompressor.cpp), you will see an unusualsnippet of code in the sample compressor Constructor:

// ...// REGISTER WITH COMPRESSOR MANAGER// this use a trick to make sure the global manager is ready for registration

// and at the same time automatically register this instance with the global manager// note that the insurer is just temporary, and deletes when we exit the procedureif (registerme)

CompressorManager SingletonInsurer managerinsurance(this);// ...

This code is designed to automatically register the compressor with a globalCompressorManagerobject.This mechanism insures that the global compressor manager is informed about all the different compressortypes no matter where they are defined, without having to add explicit code to register each new compressorin some central source code file.

The other piece of code that is required to make this scheme work is to add a global instance of thecompressor:

// Create a global instance in order to register it automatically with the global managerSampleStatCompressor GloballyInstantiated SampleStatCompressor(true);

This global instance of the derived compressor also serves another important purpose, it creates a compres-sor object which acts as a ”Factory” for dynamically creating new instances of the compressor at runtime[gamma].

To support the dynamic creation of new derived compressors (A Simple Walkthrough with the OctaneTester Console”create command”), your derived compressor classmust implement the virtual Make-CompressorInstantiation() function, which should create a new instance of your compressor. In this way,theCompressorManagerand tester utility can create new instances of your compressor when needed.

10.3.5 The OctaneCompressor API - Runtime Type Information

The OctaneCompressorclass provides some additional runtime type information classes which your de-rived compressor needs to support. The GetClassName() needs to be implemented by you and should returnthe name of your compressor class. You donot need to implement the GetName() and GetInstantiated-Name(), as they are automatically handled by the baseOctaneCompressorclass.

10.3.6 The OctaneCompressor API - Miscelaneous Functions

TheOctaneCompressorclass defines a variety of virtual functions which you can override in your derivedcompressor class if appropriate. We’ve tried to document these functions well enough in the header file(compressor.hpp) to make the functions self-explanatory. Many of these functions have suitable defaultimplementations so that you do not need to reimplement them in your derived class.

// ...// Reset the state of the compressor to its initial statevirtual void ResetState() {; };// Synchronize state for a new stream - this will be called each time before beginning a new parsing streamvirtual void SynchronizeStateForNewStream() {; };



// Prepare for parsing mode, this will be called parsing begins.// return true on successvirtual bool PrepareForCompression() {return true; };// Some compressors contains parsers which begin ready to parse, without any ’training’.// Here is where such a parser should be initialized.virtual void SetupAnyDefaultParser() {; };// is the compressor ready to compress? Normally this is false until a dictionary is built or loaded.// return true on successvirtual bool IsReadyToCompress() {return true; };// ...

10.3.7 The OctaneCompressor API - Main Compression/Decompression Func-tions

There are two main functions that you must implement in order to support compression and decompressionin your derived compressor class:

// This is the member function which does the actual compression, and this is the function that should be subclasses by derived classes.// It is wrapped by the public API call Compress() which performs some measurements on compression speed and streamsize.// note this function should never be called by an outside class - the API interface should be called instead.// return true on successvirtual bool DoProtectedCompress(bitreader &from, bitwriter &to)=0;// This is the member function which does the actual decompression, and this is the function that should be subclasses by derived classes.// It is wrapped by the public API call Decompress() which performs some measurements on decompression speed and streamsize.// note this function should never be called by an outside class - the API interface should be called instead.// return true on successvirtual bool DoProtectedDecompress(bitreader &from, bitwriter &to)=0;

All compression and decompression is to and frombitio classes, which provide a very flexible interface forreading and writing, and allow your compress and decompress routines to operate transparently on files,strings, and raw memory. The reason that these functions are named the way they are (beginning with”DoProtected...”) is that they are not meant to be called directly in order to compress or decompress data.Instead, external objects will invoke the base Compress() and Decompress() functions exposed by the baseOctaneCompressorclass, which wrap these protected functions and perform some additional operations.

10.3.8 The OctaneCompressor API - State Information

For all compressors that maintain state information (i.e. are trainable), you will need to provide implemen-tations of some virtual functions for saving, loading, and creating state:

// ...// This is the member function which does the actual training on an input in order to generate parsers/modelers/coders,// and this is the function that should be subclasses by derived classes.// It is wrapped by the public API call CreateSymbolsAndModelsUsingStream() which performs some measurements on speed and streamsize.// note this function should never be called by an outside class - the API interface should be called instead.// return true on successvirtual bool DoProtectedCreateSymbolsAndModelsUsingStream(bitreader &from) {return true; };// This is the member function which does the actual saving of state,// and this is the function that should be subclasses by derived classes.// It is wrapped by the public API call SaveState() which performs some measurements on speed and streamsize.// note this function should never be called by an outside class - the API interface should be called instead.// return true on successvirtual bool DoProtectedSaveState(bitwriter &to,bool fortempdecompressiononly) {return true; };// This is the member function which does the actual loading of state,



// and this is the function that should be subclasses by derived classes.// It is wrapped by the public API call LoadState() which performs some measurements on speed and streamsize.// note this function should never be called by an outside class - the API interface should be called instead.// return true on successvirtual bool DoProtectedLoadState(bitreader &from) {return true; };// ...

Again you’ll notice that these functions begin with ”DoProtected...” to signify that they are wrapped bypublicly exposed functions. The DoProtectedCreateSymbolsAndModelsUsingStream() function is the oneyou will override in order to train your compressor based on an input file. The fortempdecompressiononlyargument in the DoProtectedSaveState() function will be set to true when the compressor state is beingsaved in the header of a self-contained compressed file; the purpose of this flag is to allow your save statefunction to save only the minimal information needed to decompress a file - it can skip saving parametersand other information that is not needed for decompression.

10.3.9 Statistical Compressors - Overview

In the previous sections we have reviewed the basic Compressor API that is the core of all compressors.Octane also contains some higher level compressor classes to make it even easier for you to write certainkinds of compressors. The remainder of this section will describe how to build a new statistical compressorby writing your own parsers, modelers, and coders.

The modern (statistical) approach to compression algorithms treats the compressor as made up of twocomponents, a Model(er) and a Coder. The statistical compressor framework provided in Octane extendsthis idea by introducing a third component, a Parser, which is responsible for tokenizing the input/outputstream into discrete symbols. See theIntroduction to Octanefor a more detailed description of thesecomponents and statistical compression.

10.3.10 Statistical Compressors - The OctaneCompressorStatistical Class

To write a new statistical compressor, you will begin by deriving your compressor from theOctane-CompressorStatisticalclass. The API of this class is identical to the API of theOctaneCompressorclassdescribed earlier, except that the default implementations of the state management and compression de-compression functions do not have to be changed by you at all.

This is because these functions delegate most work to the three components: anOctaneParser, anOctane-Modeler, and anOctaneCoder.

For your derived compressor you will generally only need to implement the constructor, the GetClass-Name() function, and the GetDescription() function.

The real work will be done in implementing your derived Parser, Modeler, and Coder.

Before we describe these components, we note that to interface them with the compressor, you simplycreate and assign them in the constructor of your derived statistical compressor. For example, from thesample statistical compressor:

SampleStatCompressor::SampleStatCompressor(bool registerme) :OctaneCompressor Statistical(registerme){// constructor: create the children objectsparserp=new SampleParser();modelerp=new SampleModeler();coderp=new SampleCoder();// REGISTER WITH COMPRESSOR MANAGER



// this use a trick to make sure the global manager is ready for registration// and at the same time automatically register this instance with the global manager// note that the insurer is just temporary, and deletes when we exit the procedureif (registerme)

CompressorManager SingletonInsurer managerinsurance(this);// now default setup of componentsSetupAnyDefaultParser();}

OctaneCompressor Statistical::˜OctaneCompressor Statistical(){// destructor: delete children objectsif (coderp!=NULL)delete coderp;if (modelerp!=NULL)delete modelerp;if (parserp!=NULL)delete parserp;}

It’s important to note that statistical compressors are designed so that the various components can be inter-changed in a plug and play fashion. This means that often when you write a new compressor you may onlybe writing one new component and combining that with other existing parsers, modelers, or coders.

10.3.11 Statistical Compressors - The OctaneParser Class

TheOctaneParserclass defines a standard API with some default functions that you can override in yourderived class. The main functions you may want to override are described below:

Some parsers may begin with a fixed symbol set (such as the ascii character), while others will build symbolsets by parsing a training file. You also need to be able to reset the parser to a default state when parsingnew streams:

// Process (train on) an input stream to update/create a symbol set from it.// return true on successvirtual bool CreateSymbolSetUsingStream(bitreader &from) {return true; };// Synchronize state for a new stream - this will always be called before beginning a new parsing stream, and// should be used to reset the parse into any initial predictable state.virtual void SynchronizeStateForNewStream() {; };

In order to use the parser, other components of the statistical compressor will need to be able to access theparser symbols:

// Get a count of the number of symbols stored in the parser.virtual int GetSymbolCount()=0;// Helper function to return the text string of a symbol number.// return a string with the text of the symbol// note this should be the empty string to signify end of stream symbol.virtual string LookupSymbolText(int symbolnum)=0;

The main functions that do the work of the parser are those that read symbols from the input stream andwrite symbols to the output stream:

/// Parse the next symbol from the input and return its #.



/// note The end of stream situation must to be handled specially:/// note When a parser encounters the end of a stream, it *MUST* return a symbol signifying an end-of-stream symbol.// note This end of stream symbol must be a unique symbol from the symbol set.// return false *after* the end of stream symbol is returned on prior call.virtual bool ParseNextSymbolFromInput(bitreader &from,int &symbolnum)=0;// Parse the next symbol from the input, and set symbolnum to the symbol id#,// return false *after* end of stream (i.e. first response at end of stream should be the end-of-stream symbol).virtual bool WriteSymbolText(bitwriter &to, int symbolnum,bool &isendofstreamsymbol)=0;

Notice that the parser interacts with other components in the statistical compressors by referring to symbolnumbers. This is an important part of keeping the components modular. Only the Parser component knowswhat a symbol represents, whether it is a single character or a long phrase. The rest of the components justwork on the basis of symbol #s.

10.3.12 Statistical Compressors - The OctaneModeler Class

The job of theOctaneModeleris to operate between the parser and the coder, producing continuouslyupdated probability distributions over the symbolset created by the parser.

All of the components that make up a statistical compressor are responsible for keeping their own indepen-dent data structures. This design decision can lead to some duplication in data structures, but it providesmaximum independence for these different components.

Like theOctaneParser, theOctaneModelerhas functions for saving state, loading state, and creating state(training) from an input stream. Since training a modeler is so coupled to the symbolset built by theparser, the state building functions of the modeler both take a pointer to a parser. There are two alternativefunctions which may be called, depending on whether the parser begins initialized or not:

// Build a model using a stream and a parser.// return true on success.virtual bool CreateModelUsingStream( OctaneParser *parserp, bitreader &from) {return true; };// Build a model just using parser.// This is called on startup, before any possible training, when a default parser exists.// note that no input (training) stream is provider; some modelers may ignore this.// return true on successvirtual bool CreateModelUsingParser(OctaneParser *parserp) {return false; };// repare for modeling mode given a trained parser.// return true on successvirtual bool PrepareForModeling(OctaneParser *parserp) {return true; };

The main work function that you must implement in your modeler is the function that is called as eachsymbol is parsed for compression, or decoded during decompression. The job of this function is to recom-pute the probability distribution after each symbol. This function is called by the statistical compressorduring compression and decompression, and it passes a pointer to a Coder:

// Update our model after each symbol is received during compression and decompression.// Modelers track symbols during parsing and update the probability distribution predictions// for upcoming symbols. This function should notify the coder of changes to probabilities.// return true on successvirtual bool UpdateModelAfterReceivingSymbol(int symbolnumber, OctaneCoder *coderp) {return true; };

The reason that the compressor passes the coder pointer to the modeler when symbol probabilities areupdated, is because it is the job of the modeler to inform the coder when symbol probabilities change. Thissomewhat unusual arrangement is used so that if the modeler decidesnot to change symbol probabilities



(either because the model is static, or because the current context does not change), then the coder shouldnot have to rearrange its state, which can often be quite expensive.

A sample zero-order modeler (ZeroOrderModeler) is provided that counts static symbol frequencies givenan arbitrary parser.

To inform the Coder that symbol probabilities have changed, the Modeler makes a call to one of twonotification member functions exposed by thecoder, and described in detail below: ReceiveNotification-ModelChangeAllSymbolWeights(), and ReceiveNotificationModelChangeSingleSymbolWeight().

The coder in turn will want to make calls to the modeler to get values for the symbol weights (probabil-ities). Two functions are provided, one very general which your modelermust support, which makes noassumptions about the underlying data structure used by the modeler to store symbol weights, and anotheroptional accessor function for providing a more efficient interface to a symbol weight vector, for modelerswhich use a standard weightvector class which is provided with Octane (SymbolWeightVector):

// Return the number of symbols in the modeler (should be same as number of symbols in the parser); called by coder.virtual int GetSymbolCount() {return 0; };// Return a pointer to the current weight vector for symbols (used by coder) [optional]// This is provided as a shortcut function to avoid having to make numerous calls to GetWeightVectorItem() below.// Not all modelers will maintain probabilities in the form of a SymbolWeightVector , and a modeler can return NULL in order to// force the coder to get probabilities for each symbol individually.// return a pointer to a local SymbolWeightVector or NULL if it is not available.virtual SymbolWeightVector* GetWeightVectorp() {return NULL; };// Return a specific weight index value; (used by coder).virtual TSymbolWeightVectorWeight GetWeightVectorItem(int symbolnum) {return 0; };

A derived modeler which explicitly uses this weightvector is provided to make creating custom modelerseven easier (OctaneModelerWeightVectored).

10.3.13 Statistical Compressors - The OctaneCoder Class

The job of the coder component during compression is to assign bitcodes to each symbol number. Duringdecompression the job of the coder is to read bits from the input stream and translate them into symbolnumbers for translation by the parser.

The main work of the coder is performed in two functions which you need to override for your coder:

// Write bits for symbol number// return true on successvirtual bool WriteSymbolBits(int symbolnum, bitwriter &bw)=0;// Decode the next symbol from the input// return true on success, false when there are no more symbols to read.virtual bool DecodeSymbolFromInput(int &symbolnum, bitreader &br)=0;

Note that as with the Modeler, the coder cares only about symbol numbers, and knows nothing about whateach symbol number represents, whether it is a single character, a word, or an arbitrarily large chunk ofbytes.

Your coder must implement functions to receive notifications from the modeler for when symbol weights(probabilities) change:

// Notify coder that all probabilities are being updated.// We can be informed by a model when the underlying probabilities are changing// this part of the API exists so that we can flexibly and efficiently// handle different kinds of situations where the coder needs to be synchronized



// with the model probabilities. in some cases, a coder may ignore these// messages and simply rebuild its internal datastructure on each coding event// in other cases, it will want to incrementally update as model changes.// Two kinds of notifications are supported, depending on how the model updates// itself during processing, whether it updates one symbol per iteration, or// modifies all symbols at once.virtual void ReceiveNotification ModelChange AllSymbolWeights(OctaneModeler *modelerp) {; };// notify that a single symbol probability has changed (by default just calls AllSymbolWeights change above)// note the SingleSymbolWeight() call will NOT be made when all weights changevirtual void ReceiveNotification ModelChange SingleSymbolWeight(OctaneModeler *modelerp,int symbolnum);

The pointer to the calling modeler is provided in these calls, so that the coder can in turn call the modeler toget the actual values for symbol weights, using the followingmodelermember functions described earlier:GetSymbolCount(), GetWeightVectorp(), and GetWeightVectorItem().

A working Huffman Coder (HuffmanCoder) is provided with the Octane classes to show a real workingcoder.


10.4 History 301

10.4 History

10.4.1 Version 1.01.20 (2004-05-19)

[-] fixed bug in smartlookup mode of SubStringParser that could cause crashing when incoming streammatched last symbol in set. (mouser, 2004-05-19).

[+] added code to report kb/s compression and decompression speeds when testing files in theOctaneTester.(mouser, 2004-05-19).

10.4.2 Version 1.01.19 (2003-09-09)

[-] fixed bug in huffman coder leading to long bitcodes for unseen ascii characters (zeroordermodeler.cppupdated to prevent 0 freq counts). (mouser+jibz, 2003-09-09).

10.4.3 Version 1.01.18 (2003-08-27)

[-] eliminated use of one last itoa call inhuffmannodes.cpp(replaced with sprintf). (mouser+jibz, 2003-08-27).

[-] fixed spelling error inhuffmannodes.cpp(bistset -> bitset). (mouser+jibz, 2003-08-27).

[-] removed unused variable (inputsize) inhuffmancoder.cpp. (mouser, 2003-08-27).

10.4.4 Version 1.01.17 (2003-08-14)

[+] added zle encoder (zero-length-encoder, which is run-length encoding for 0 bytes). (loco, 2003-08-13).

[+] added mtfll encoder (move-to-front with linked lists). (loco, 2003-08-14).

10.4.5 Version 1.01.16 (2003-08-12)

[-] fixed spelling error in octane.ini. (mouser, 2003-08-12, found by loco).

[-] fixed grammar error in OctaneTester.cpp. (mouser, 2003-08-12, found by loco).

[+] added a new sample compressor to show how to make non-statistical compressors. (mouser, 2003-08-12).

[+] testfile command now deletes temporary files. (mouser, 2003-08-12).

10.4.6 Version 1.01.15 (2003-08-11)

This is the initial release of Octane.



10.5 Future Work

The Octane Toolkit was started in July 2003, and is still in the early stages of development. Many featuresremain to be written, and we are hoping that other developers will help to make Octane a useful resource.

What follows is a list of some of the things we would like to see added to Octane. This list does notrepresent a commitment on our part to code these features, but rather a wishlist and public request for helpin adding these features.

• Add custom training class, for custom segmentation of training files

• An Efficient Arithmetic Coder - Arithmetic coding is considered the most space-efficient codingalgorithm (though see [witten] for a discussion of why they ultimately chose to use a Huffman coderfor their system). See [moffat] for implementation details of a good arithmetic coder. See also:http://www.cs.mu.oz.au/ ∼alistair/arith coder/.

• Class Framework for Dictionary Compressors - Much like we have provided a set of integratedclasses for statistical compression, we would like to have a set of classes to support Lempel-Ziv typedictionary compressor, and make writing new dictionary compressors easy.

• More Compressor Implementations - Octane is currently a framework lacking a real col-lection of implemented compressors. We would like to see it used to implement someof the more popular compression algorithms (ppm, dmc, bwt, lz∗, mtf, etc.). See also:http://www.datacompression.info/.

• Experimental Compressors - We would like to see more experimental compressors, such as neuralnetwork statistical compressors, for use with external dictionary type compression tasks.

• Cryptographic Extensions - We would like to see cryptographic functionality added to Octane. Cur-rently we believe the best approach to this is to in fact generalize the concept of compressor so thatwe work with Transformer classes, which can involve compression, encryption, etc. We would thenallow for transformers to be ”chained” together, so that compressors and encryptors could be chainedtogether to produce a pipeline of transformations.

• More Testing/Reporting Tools - The testing utility of Octane needs to be extended to provide addi-tional ways to do batch comparison of compression algorithms on known corpora. The output ofsuch routines should be standardized for easy comparison to published results.

• Custom Memory Use Tracking - It is often desirable when testing compression algorithms to eval-uate memory usage during compression and decompression. Octane currently implements a simplememory use reporting function which is not very accurate. A custom memory manager is neededwhich can track and report memory usage exactly.

• Compresor Switching Code - Octane was built to support applications like chat programs whichmight switch dynamically between a set of compressors/dictionaries during operation in order totrack the most efficient compressor over time. A supervisor class which manages such switching andbalances efficiency vs. computational effort is needed.

• Higher Level Package Archiver - We would like to see the Octane Classes used to build a multi-file archiver, using Compressor Switching to choose the best compressor for each file (or in factswitching within a file). See also:http://datacompression.info/ArchiveFormats/.

• Platform (endian) compatibility tests and final fixes; almost all of the bitio code is completely plat-form neutral, and can be used to compress on one platform (endianness) and decompress on another.The only area that needs to be fixed is the writing of floating point values, which is not used in normalcompression but could be used by compressor’s saving their state.


http://www.cs.mu.oz.au/~alistair/arith_coder/.

http://www.datacompression.info/.

http://datacompression.info/ArchiveFormats/.

10.5 Future Work 303

We need help implementing these features! Join us on the efnet irc network in #octane.



10.6 Octane License

Octane - The Open Compression Toolkit for C++

Copyright ©2003 by Octane Project Group

All Rights Reserved.

http://octane.sourceforge.net/

This software is provided ’as-is’, without any express or implied warranty. In no event will the authors beheld liable for any damages arising from the use of this software.

Permission is granted to anyone to use this software for any purpose, including commercial applications,and to alter it and redistribute it freely, subject to the following restrictions:

1. The origin of this software must not be misrepresented; you must not claim that you wrote the originalsoftware. If you use this software in a product, an acknowledgment in the product documentationwould be appreciated but is not required.

2. Altered source versions must be plainly marked as such, and must not be misrepresented as beingthe original software.

3. This notice may not be removed or altered from any source distribution.


http://octane.sourceforge.net/

10.7 References 305

10.7 References

Apiki, Steve.,

Lossless data compression: two algorithms - Huffman coding and LZW coding - are at the root of mostcompression.

Byte, 1991 Mar; 16, n3309(7).

Bell, Timothy C, Cleary, John G, and Witten, I. H.,

Text compression.

Prentice Hall Advanced Reference Series. Englewood Cliffs, N.J: Prentice Hall, 1990.

Beveridge, J.,

Self-Registering Objects in C++.

Dr. Dobb’s Journal 288: 38-41, 1998.

Gamma, E., Helm, R., Johnson, R., and Vlissides, J.,

Design Patterns: Elements of Reusable Object-Oriented Software.

Reading, MA: Addison-Wesley, 1995.

Moffat, A., Neal, R. M., and Witten, I. H. ,

Arithmetic Coding Revisited.

ACM Transactions on Information Systems, 16(3): 256-94, 1998.

Nelson, Mark.,

Priority Queues and the STL.

Dr. Dobb’s Journal 1996(1).

Shannon, C.,

The mathematical theory of communications.

Bell System Technical Journal, Vol. 27, pp. 379-423 and 623-565, 1948.

Witten, Ian H. Author, Moffat, Alistair Author, Bell, Timothy Author, and Fox, Ed. ,

Managing Gigabytes: Compressing and Indexing Documents and Images.

Morgan Kaufmann, May 1999.



10.8 Todo List

Member BitParser::senteos there should be a way to do this automatically perhaps with a wrapper,rather than insisting all derived parsers deal with returning an EOS symbol.

Member bitreader::get float() Experimental, not portable.

Member bitreader::get double() Experimental, not portable.

Member bitwriter::put float(float v) Experimental, not portable.

Member bitwriter::put double(double v) Experimental, not portable.

Member HuffNode Leaf::get memoryused() const do we really need to add sizeof(codebitset) as weare doing?

Member OctaneCompressor::PrepareForCompression() is this a duplication of functionality in Reset-State() and SynchronizeStateForNewStream()?

Member OctaneCompressor::Decompress(bitreader &from, bitwriter &to, bool needtoreadheader)what should we do on a LoadCompressionHeader() error?

Member OctaneCompressor::GetStatisticsLastOperation ErrorCount () i don’t think we currentlyreset this, so it may behave like cumulative error until we fix it.

Member OctaneCompressor::SaveGUID(bitwriter &to) - we will add code here so that we use only 1byte for most guid, and 2 bytes for others, but for now we always use 1 byte for GUID

Member OctaneCompressor::LoadGUID(bitreader &from) - we will add code so that we use only 1byte for most guid, and 2 bytes for others, but for now we always use 1 byte for GUID

Member OctaneTester::RunCommandLoadCompressor(std::string filename) show compressor idinfo

Member OctaneTester::DeleteAFile(std::string filename) this is not implemented yet

Member OctaneTester::CheckFilesAreIdentical(std::string filename1, std::string filename2) do weneed to do this more efficiently


10.8 Todo List 307

Member SampleParser::senteosthere should be a way to do this automatically perhaps with a wrapper,rather than insisting all derived parsers deal with returning an EOS symbol.

Member stringparser::get token() should be able to handle quoted tokens.

Member SubstringParser::senteos this is an odd responsibility for the parser to have to deal with, so weshould probably find a way to eliminate this.

File huffmancoder.hpp There seems to be a bug in this code which is causing very large bitcodes for raresymbols - they should not get this large afaik. it may have to do with a flaw in the implementation,or be due to the fact that we are artifically forcing the tree to keep some primitive ascii characterswhich actually get frequency counts of 0.

File symbolweightvector.hpp I considered making a base class and then derived template classes forinteger and floating point weight vectors. however I’m not sure the memory advantage for integers(the rare case) would offset the speed hit to be paid for virtual functions, and the inconvenience ofdealing with different types for different coders/modelers.

File bitparser.cpp : this is not finished yet

File bitparser.hpp : this is not finished yet



10.9 Bug List

Member OctaneTester::RunCommandDecompress(std::string infilename, std::string outfilename, bool savedecompressstateinfo)- should bytespersec use BytesRead or BytesWritten? using BytesWritten during decompressionmakes comparing times easier so we use that.


Index

∼HuffNode MiddleHuffNode Middle, 75

∼OctaneScopeTimerOctaneScopeTimer,131

∼bitreaderbitreader,35

∼bitwriterbitwriter, 46

AddCompressorCompressorManager,56

AlignInputStreamToByteBoundryOctaneCoder,93

AlignOutputStreamToByteBoundryOctaneCoder,93

arraybitreader,21arraybitreader,22readnext byte,22setposition,22supportsseek,22

arraybitwriter, 24arraybitwriter, 24write next byte,25

bitio, 13bitio/bitio.hpp,223bitio/bitreader.hpp,224bitio/bitreaders.hpp,225bitio/bitwriter.cpp,226bitio/bitwriter.hpp,227bitio/bitwriters.hpp,228bitio/file bitreader.hpp,229bitio/file bitwriter.hpp,230BitParser,26BitParser

CreateSymbolSetUsingStream,27LoadState,30LookupSymbolText,29ParseNextSymbolFromInput,28RewindAnyBufferedInput,28SaveState,30senteos,31SetParameter,30WriteSymbolText,29

bitreader,32

∼bitreader,35bitreader,34bits left, 35bytesleft, 36empty,35error,35finalize,42get bit, 39get bit length,36get byte,39get byte length,36get double,41get dword,40get float,41get string,41get word,40read,42readnext byte,42seekbit, 37, 38seekbyte,38, 39setposition,43supportsseek,37tell bit, 36tell byte,37

bits donebitwriter, 46

bits leftbitreader,35

bitwriter, 44∼bitwriter, 46bits done,46bytesdone,46error,46finalize,50put bit, 47put byte,48put double,49put dword,49put float,49put string,50put word,48tell bit, 47tell byte,47write, 50write next byte,51

310 INDEX

bytesdonebitwriter, 46

bytesleftbitreader,36

CheckCompressorReadyOctaneTester,143

CheckCompressorSelectedOctaneTester,142

CheckFilesAreIdenticalOctaneTester,153

Coders,15coders/coder.cpp,231coders/coder.hpp,232coders/coderhuffman/huffmancoder.cpp,233coders/coderhuffman/huffmancoder.hpp,234coders/coderhuffman/huffmannodes.cpp,235coders/coderhuffman/huffmannodes.hpp,236coders/codersample/samplecoder.cpp,238coders/codersample/samplecoder.hpp,239CompressorManager,52CompressorManager

AddCompressor,56CreateCompressorFromSavedStream,55DeleteCompressor,56FindBaseCompressorClass,53FindInstantiatedCompressorpFromGuid,55FindInstantiatedCompressorpFrom-

StringId,54CompressorManagerSingletonInsurer,58

CompressorManagerSingletonInsurer,58CompressorManagerSingletonInsurer

CompressorManagerSingletonInsurer,58Compressors,16compressors/compressor.cpp,240compressors/compressor.hpp,241compressors/compressormtfll/mtfll.cpp, 242compressors/compressormtfll/mtfll.hpp, 243compressors/compressor-

sample/samplecompressor.cpp,244compressors/compressor-

sample/samplecompressor.hpp,245

compressors/compressor-sample/samplestatcompressor.cpp,246

compressors/compressor-sample/samplestatcompressor.hpp,247

compressors/compressorstatistical.cpp,248compressors/compressorstatistical.hpp,249compressors/compressor-

substrhuff/substrhuff.cpp,250

compressors/compressor-substrhuff/substrhuff.hpp,251

compressors/compressorzle/zle.cpp,252compressors/compressorzle/zle.hpp,253Core,18core/compressormanager.cpp,254core/compressormanager.hpp,255core/octane.hpp,256core/octaneclass.cpp,257core/octaneclass.hpp,258CountSymbolFrequencies

SymbolWeightVector,208CreateCompressorFromSavedStream

CompressorManager,55CreateModelUsingParser

OctaneModeler,119SampleModeler,170

CreateModelUsingStreamOctaneModeler,119SampleModeler,170ZeroOrderModeler,218

CreateSymbolSetUsingStreamBitParser,27OctaneParser,128SampleParser,173SubstringParser,196

DecodeSymbolFromInputHuffmanCoder,67OctaneCoder,91SampleCoder,160

DecompressOctaneCompressor,100

DEF HUFFMANPQ MAXIMUM -CODEBITDEPTH

huffmannodes.hpp,237DeleteAFile

OctaneTester,153DeleteCompressor

CompressorManager,56dontprune

SubstringSymbol,204DoProtectedCompress

mtfllCompressor,77OctaneCompressor,103OctaneCompressorStatistical,112SampleCompressor,164zleCompressor,220

DoProtectedCreateSymbolsAndModelsUsingStreamOctaneCompressor,104OctaneCompressorStatistical,114SampleCompressor,166

DoProtectedDecompressmtfllCompressor,77


INDEX 311

OctaneCompressor,103OctaneCompressorStatistical,113SampleCompressor,165zleCompressor,220

DoProtectedLoadStateOctaneClass,87OctaneCompressor,104OctaneCompressorStatistical,115SampleCompressor,167

DoProtectedSaveStateOctaneClass,87OctaneCompressor,104OctaneCompressorStatistical,115SampleCompressor,166

emptybitreader,35

errorbitreader,35bitwriter, 46file bitreader,60file bitwriter, 63

ExecuteCommandsFromFileOctaneTester,141

ExecuteCommandsFromStreamOctaneTester,140

file bitreader,59error,60file bitreader,60readnext byte,61setposition,61supportsseek,60

file bitwriter, 62error,63file bitwriter, 62write next byte,63

finalizebitreader,42bitwriter, 50

FindBaseCompressorClassCompressorManager,53

FindInstantiatedCompressorpFromGuidCompressorManager,55

FindInstantiatedCompressorpFromStringIdCompressorManager,54

get bitbitreader,39

get bit lengthbitreader,36

get bytebitreader,39

get byte length

bitreader,36get double

bitreader,41get dword

bitreader,40get elapsed

OctaneScopeTimer,132OctaneTimer,155

get floatbitreader,41

get memoryusedHuffNode Leaf,73

get stringbitreader,41

get tokenstringparser,189

get wordbitreader,40

GetCompressorStateFileGUIDOctaneCompressor,99

GetDescriptionHuffmanCoder,66mtfllCompressor,77OctaneClass,84OctaneCoder,90OctaneCompressor,99OctaneCompressorStatistical,110OctaneModeler,118OctaneModelerWeightVectored,123OctaneParser,127SampleCoder,159SampleCompressor,163SampleModeler,170SampleStatCompressor,178SubStrHuffCompressor,191SymbolWeightVector,207ZeroOrderModeler,218zleCompressor,220

GetHelpInformationHuffmanCoder,66OctaneClass,84OctaneCoder,90OctaneCompressor,99OctaneCompressorStatistical,110OctaneModeler,118OctaneParser,128SampleCoder,159SymbolWeightVector,207

GetNameHuffmanCoder,66OctaneClass,84OctaneCoder,90OctaneCompressor,98OctaneModeler,118


312 INDEX

OctaneModelerWeightVectored,123OctaneParser,127SampleCoder,159SampleModeler,169SymbolWeightVector,206ZeroOrderModeler,217

GetStatisticsLastOperationErrorCountOctaneCompressor,105

GetWeightVectorpOctaneModeler,120OctaneModelerWeightVectored,125

GrabLineFromConsoleOctaneTester,141

HuffmanCoder,64HuffmanCoder

DecodeSymbolFromInput,67GetDescription,66GetHelpInformation,66GetName,66IsReadyToCode,68ReceiveNotificationModelChange-

AllSymbolWeights,68ReceiveNotificationModelChange-

SingleSymbolWeight,68symbolsetpq,69WriteSymbolBits,66

huffmannodes.hppDEF HUFFMANPQ MAXIMUM -

CODEBITDEPTH,237HuffNode,70HuffNode Leaf,72HuffNode Leaf

get memoryused,73HuffNode Middle, 74HuffNode Middle

∼HuffNode Middle, 75

IsNonWordCharacterOctaneParser,129

IsReadyToCodeHuffmanCoder,68OctaneCoder,91SampleCoder,160

IsReadyToCompressOctaneCompressor,100OctaneCompressorStatistical,112SampleCompressor,164

IsReadyToModelOctaneModeler,119OctaneModelerWeightVectored,124

LoadOctaneCompressor,102

LoadGUIDOctaneCompressor,106

LoadStateBitParser,30OctaneClass,85OctaneModelerWeightVectored,124SampleParser,176SubstringParser,201SymbolWeightVector,208

LookupSymbolTextBitParser,29OctaneParser,129SampleParser,175SubstringParser,199

MakeAssumedOutputFilenameOctaneTester,143

Modelers,19modelers/modeler.cpp,259modelers/modeler.hpp,260modelers/modelersample/samplemodeler.cpp,

261modelers/modelersample/samplemodeler.hpp,

262modelers/modelerweightvectored.cpp,263modelers/modelerweightvectored.hpp,264modelers/modelerzeroorder/zeroorder-

modeler.cpp,265modelers/modelerzeroorder/zeroorder-

modeler.hpp,266modelers/symbolweightvector.cpp,267modelers/symbolweightvector.hpp,268mtfllCompressor,76mtfllCompressor

DoProtectedCompress,77DoProtectedDecompress,77GetDescription,77

null bitreader,79null bitreader,79readnext byte,80supportsseek,80

null bitwriter, 81write next byte,81

OctaneClass,82OctaneClass

DoProtectedLoadState,87DoProtectedSaveState,87GetDescription,84GetHelpInformation,84GetName,84LoadState,85ParseParameter,85, 86


INDEX 313

SaveState,85SetParameter,84

OctaneCoder,89OctaneCoder

AlignInputStreamToByteBoundry,93AlignOutputStreamToByteBoundry,93DecodeSymbolFromInput,91GetDescription,90GetHelpInformation,90GetName,90IsReadyToCode,91PrepareForCoding,91ReceiveNotificationModelChange-

AllSymbolWeights,92ReceiveNotificationModelChange-

SingleSymbolWeight,92WriteSymbolBits,91

OctaneCompressor,94OctaneCompressor

Decompress,100DoProtectedCompress,103DoProtectedCreateSymbolsAndModelsUs-

ingStream,104DoProtectedDecompress,103DoProtectedLoadState,104DoProtectedSaveState,104GetCompressorStateFileGUID,99GetDescription,99GetHelpInformation,99GetName,98GetStatisticsLastOperationErrorCount,

105IsReadyToCompress,100Load,102LoadGUID,106PrepareForCompression,100Save,101SaveCompressionHeader,105SaveGUID,106SetupAnyDefaultParser,100

OctaneCompressorStatistical,108OctaneCompressorStatistical

DoProtectedCompress,112DoProtectedCreateSymbolsAndModelsUs-

ingStream,114DoProtectedDecompress,113DoProtectedLoadState,115DoProtectedSaveState,115GetDescription,110GetHelpInformation,110IsReadyToCompress,112PrepareForCompression,111SetParameter,110SetupAnyDefaultParser,111

OctaneModeler,117OctaneModeler

CreateModelUsingParser,119CreateModelUsingStream,119GetDescription,118GetHelpInformation,118GetName,118GetWeightVectorp,120IsReadyToModel,119PrepareForModeling,120UpdateModelAfterReceivingSymbol,120

OctaneModelerWeightVectored,122OctaneModelerWeightVectored

GetDescription,123GetName,123GetWeightVectorp,125IsReadyToModel,124LoadState,124SaveState,124

OctaneParser,126OctaneParser

CreateSymbolSetUsingStream,128GetDescription,127GetHelpInformation,128GetName,127IsNonWordCharacter,129LookupSymbolText,129ParseNextSymbolFromInput,129PrepareForParsing,128RewindAnyBufferedInput,128WriteSymbolText,129

OctaneScopeTimer,131OctaneScopeTimer,131

OctaneScopeTimer∼OctaneScopeTimer,131get elapsed,132OctaneScopeTimer,131

OctaneTester,133OctaneTester

CheckCompressorReady,143CheckCompressorSelected,142CheckFilesAreIdentical,153DeleteAFile,153ExecuteCommandsFromFile,141ExecuteCommandsFromStream,140GrabLineFromConsole,141MakeAssumedOutputFilename,143ParseCommand,135RunCommandCompress,150RunCommandCreateCompressor,146RunCommandDecompress,151RunCommandDeleteCompressor,149RunCommandLoadCompressor,147RunCommandMakeState,149


314 INDEX

RunCommandSaveCompressor,148RunCommandTestFile,145RunCommandTestString,144SelectInstantiatedCompressor,142

octanetester/octanetester.cpp,269octanetester/octanetester.hpp,270octanetester/standalonetestermain.cpp,271OctaneTimer,155OctaneTimer

get elapsed,155stop,155

ParseCommandOctaneTester,135

ParseNextSymbolFromInputBitParser,28OctaneParser,129SampleParser,174SubstringParser,198

ParseParameterOctaneClass,85, 86

Parsers,20parsers/parser.cpp,272parsers/parser.hpp,273parsers/parserbitlevel/bitparser.cpp,274parsers/parserbitlevel/bitparser.hpp,275parsers/parsersample/sampleparser.cpp,276parsers/parsersample/sampleparser.hpp,277parsers/parsersubstring/substringparser.cpp,

278parsers/parsersubstring/substringparser.hpp,

279PQWeightGreater,157PrepareForCoding

OctaneCoder,91PrepareForCompression

OctaneCompressor,100OctaneCompressorStatistical,111

PrepareForModelingOctaneModeler,120

PrepareForParsingOctaneParser,128

put bitbitwriter, 47

put bytebitwriter, 48

put doublebitwriter, 49

put dwordbitwriter, 49

put floatbitwriter, 49

put stringbitwriter, 50

put wordbitwriter, 48

readbitreader,42

readnext bytearraybitreader,22bitreader,42file bitreader,61null bitreader,80streambitreader,181string bitreader,185vectorbitreader,213

ReceiveNotificationModelChange-AllSymbolWeights

HuffmanCoder,68OctaneCoder,92SampleCoder,161

ReceiveNotificationModelChange-SingleSymbolWeight

HuffmanCoder,68OctaneCoder,92

RewindAnyBufferedInputBitParser,28OctaneParser,128SampleParser,174SubstringParser,197

RunCommandCompressOctaneTester,150

RunCommandCreateCompressorOctaneTester,146

RunCommandDecompressOctaneTester,151

RunCommandDeleteCompressorOctaneTester,149

RunCommandLoadCompressorOctaneTester,147

RunCommandMakeStateOctaneTester,149

RunCommandSaveCompressorOctaneTester,148

RunCommandTestFileOctaneTester,145

RunCommandTestStringOctaneTester,144

SampleCoder,158SampleCoder

DecodeSymbolFromInput,160GetDescription,159GetHelpInformation,159GetName,159IsReadyToCode,160


INDEX 315

ReceiveNotificationModelChange-AllSymbolWeights,161

WriteSymbolBits,159SampleCompressor,162SampleCompressor

DoProtectedCompress,164DoProtectedCreateSymbolsAndModelsUs-

ingStream,166DoProtectedDecompress,165DoProtectedLoadState,167DoProtectedSaveState,166GetDescription,163IsReadyToCompress,164SetParameter,164

SampleModeler,169SampleModeler

CreateModelUsingParser,170CreateModelUsingStream,170GetDescription,170GetName,169

SampleParser,172SampleParser


SampleStatCompressor,178SampleStatCompressor

GetDescription,178Save

OctaneCompressor,101SaveCompressionHeader

OctaneCompressor,105SaveGUID

OctaneCompressor,106SaveState

BitParser,30OctaneClass,85OctaneModelerWeightVectored,124SampleParser,176SubstringParser,200SymbolWeightVector,207

seekbitbitreader,37, 38

seekbytebitreader,38, 39

SelectInstantiatedCompressorOctaneTester,142

senteos

BitParser,31SampleParser,177SubstringParser,201

setpositionarraybitreader,22bitreader,43file bitreader,61string bitreader,185vectorbitreader,213

SetParameterBitParser,30OctaneClass,84OctaneCompressorStatistical,110SampleCompressor,164SampleParser,176SubstringParser,200

SetupAnyDefaultParserOctaneCompressor,100OctaneCompressorStatistical,111

stopOctaneTimer,155

streambitreader,180readnext byte,181streambitreader,180supportsseek,181

streambitwriter, 182streambitwriter, 182write next byte,183

string bitreader,184readnext byte,185setposition,185string bitreader,184, 185supportsseek,185

string bitwriter, 187string bitwriter, 187write next byte,188

stringparser,189get token,189stringparser,189

SubStrHuffCompressor,191SubStrHuffCompressor

GetDescription,191SubstringParser,193SubstringParser


SubstringSymbol,203


316 INDEX

SubstringSymboldontprune,204

supportsseekarraybitreader,22bitreader,37file bitreader,60null bitreader,80streambitreader,181string bitreader,185vectorbitreader,213

symbolsetpqHuffmanCoder,69

SymbolWeightVector,205SymbolWeightVector

CountSymbolFrequencies,208GetDescription,207GetHelpInformation,207GetName,206LoadState,208SaveState,207

tell bitbitreader,36bitwriter, 47

tell bytebitreader,37bitwriter, 47

TSubstringSymbolStringGreater,210TSubstringSymbolWeightGreater,211

UpdateModelAfterReceivingSymbolOctaneModeler,120

Utility, 17utilityclasses/stringparser/stringparser.hpp,281utilityclasses/timer/scopetimer.hpp,282utilityclasses/timer/timer.hpp,283

vectorbitreader,212readnext byte,213setposition,213supportsseek,213vectorbitreader,212, 213

vectorbitwriter, 215vectorbitwriter, 215write next byte,216

writebitwriter, 50

write next bytearraybitwriter, 25bitwriter, 51file bitwriter, 63null bitwriter, 81streambitwriter, 183

string bitwriter, 188vectorbitwriter, 216

WriteSymbolBitsHuffmanCoder,66OctaneCoder,91SampleCoder,159

WriteSymbolTextBitParser,29OctaneParser,129SampleParser,175SubstringParser,199

ZeroOrderModeler,217ZeroOrderModeler

CreateModelUsingStream,218GetDescription,218GetName,217

zleCompressor,219zleCompressor

DoProtectedCompress,220DoProtectedDecompress,220GetDescription,220


Octane Reference Manualoctane.sourceforge.net/releases/octane-1.01.20.pdf · 2004. 5. 20. · •...

Documents

Transcript of Octane Reference Manualoctane.sourceforge.net/releases/octane-1.01.20.pdf · 2004. 5. 20. · •...