Accelerating forensic and incident response workflow: the case for a new standard in forensic...

Accelerating your forensic & incident response workflow: the case for a new standard in forensic imaging

Dr. Bradley SchatzDirector, Schatz Forensic

v1.2 - HTCIA Conference 2016© Schatz Forensic 2016

© 2016 Schatz Forensic

The volume problem increases the latency between evidence identification and useful

findings

Identify Acquire Analyse Reporting

Latency


Pick one of the belowYou can’t have both

Latency

Com

plet

enes

sPhysical Acquisition

Triage

You preserve everything but

analysis will have to wait

Near immediate results at the expense of

potentially missing evidence

Live forensics


How can we reduce latency?While maximising completeness

Latency

Com

plet

enes


Triage

IncreaseI/O

throughput?

Live analysis while we acquire?

Dynamic partial acquisition?

Live forensics


Current forensic image formats are now a bottleneck

• Deflate compression is inefficient• Linear hashing does not scale to multi-core• Copying blocks of zero filled sectors is a waste of

time• Linear images prevent efficient out of order

acquisition


The Advanced Forensic Format v4 (AFF4) image format is the solution

• Scalable to GB/s IO & multi-core• Enable forensically reproducible partial non-

linear images (reproducible triage)• Scientifically peer reviewed (ref Daubert)• Unencumbered, open specification• Open source implementations

What’s stopping me increasing I/O throughput?Background


Forensic Imaging v1.0: RawLinear bitstream copy + linear bitstream hash

$ dd if=/dev/hda bs=4k conv=sync,noerror | tee C1.D1.raw | md5sum > C1.D1.md5.txt


Forensic Imaging v1.0: Raw

MD5

Source Hard Drive

ACMECo.C1.D1.raw

ACMECo.C1.D1.raw.txt

# Linear Bitstream Hash


What affects throughput in acquisition?

Target Storage Interconnect Hash Filesystem Interconnect Evidence

storage


I/O throughput in Acquisition is a systems problem


storage

Target Storage Sustained Read

1TB Seagate 3.5” 7200rpm SATA 100 MB/s

Current generation 3.5” 7200rpm SATA 200 MB/s

Intel 730 SSD 550 MB/s

Macbook Pro 1TB ~1 GB/s

RAID 15000rpm SAS > 1 GB/s

Samsung 850 NVMe 1.5 – 2.5 GB/s




storage

Algorithm Average Throughput MB/s

SHA1 619.23MD5 745.65Blake2b 601.87




storage

Interconnect Gb/s Actual Gb/s

Max MB/s Max GB/m

PCIe / NVMe / Thunderbolt > 1000 > 60SATA3 / SAS 6G 6 4.8 600 36USB3 5 4 500 30Gigabit Ethernet 1 ~100USB2 .48 .38 48 2.9




storage

Interconnect Gb/s Actual Gb/s

Max MB/s Max GB/m

PCIe / NVMe / Thunderbolt > 1000 > 60SATA3 / SAS 6 4.8 600 36USB3 5 4 500 30Gigabit Ethernet 1 ~100USB2 .48 .38 48 2.9

Can we practically achieve this?


Not all bridges are made equalManuf. Source Dest Form factor Year

purchasedMB/s

Orico USB3 SATA 3.5” slide dock 2014 219

Orico USB3 SATA 2.5” enclosure 2016 247

Orico USB3 SATA 3.5” dual dock 2016 402

Kanex Thunderbolt eSATA Cable 2015 213

Nexstar USB3 SATA 3.5” dock 2014 189

Nexstar USB3 eSATA Cable 2016 249

Probox USB3 SATA Bridge 2016 416*

Samsung T3 USB3 integrated SSD 2016 400

Testing tool: BlackMagicDesign Disk Speed Test Destination disk: Samsung 850 Pro SSD* Fails under heavy load


Take Away #1Faster destination IO is important, but beware choice of bridge

• Not an issue if imaging to spinning disk <200MB/s

• Raw SATA & SAS IO fastest (duplicators)• SSD/RAID speed levels require decent bridges• Thunderbolt and UASP promising – more testing

needed


Example: Forensic Duplicator1TB Seagate Target


storage

SHA1600MB/s

SATA3Spinning Disk93.6MB/s

SAS 6G600MB/s

SATA3Spinning Disk200MB/s

Acquisition 1TB @ 93.6MB/s = 2h 58mVerification 1TB @ 200MB/s = 1h 23mTOTAL = 4h 21m

SAS 6G600MB/s


Bare Metal (LiveCD) Ancient Workstation Acquisition


storage

SHA1600MB/s


USB245MB/s


Acquisition 1TB @ 45MB/s = 6h 10mVerification 1TB @ 45MB/s = 6h 10mTOTAL = 12h 20m


Bare Metal (LiveCD) Ancient Workstation Acquisition


storage

SHA1600MB/s


USB245MB/s



After copy, verify image on device with

faster interconnect


Take Away #2Plan your acquisitions to maximise throughput

• Relocate image for verification• Add a USB3 expresscard / PCIe card• Pull disks from slower machines and go bare

metal (live CD) on faster ones• Use GigE (100Mb/s) instead of USB2


Bare Metal (LiveCD) Example:Caveat User Space Filesystem


storage

SHA1600MB/s


USB3500MB/s



NTFS-3g100MB/s

SAS 6G600MB/s


Is your forensic liveCD slowing you down?

$ ntfs-3g /dev/sdd1 /mnt

$ time sh –c “time dd if=/dev/zero of=/mnt/zeros bs=512k count=20k ; sync”

104s 102MB/s

$ mount –t ntfs-3g –o max_read=131072,big_writes /dev/sdd1/ /mnt

$ time sh –c “time dd if=/dev/zero of=/mnt/zeros bs=512k count=20k ; sync”

33s 318 MB/s

Destination: Samsung T3 USB3 SSD


Take Away #3NTFS may be a convenient destination filesystem, but is it costing

time?

• Use a kernel based FS implementation• -or- • Tune the filesystem if it is a user space variant


Forensic Imaging v2.0: EWFOriginal design

Source Hard Drive

MD5

Deflate

ACMECo.C1.D1.e01

Source Hard Drive

# Linear BitStream Hash

Linear Compressed Block Stream


The deflate algorithm is a significant bottleneck

Target Storage Interconnect Hash Compress Filesystem Interconnect Evidence

storage

Data Deflate MB/s Inflate MB/s

High entropy 40.4 439

Low entropy 259 IO bound

*Single core of quad core i7-4770 3.4Ghz measured with gzip


FTK Imager EWF Acquisition1TB Seagate 75% full, 4 core i5-750


storage

SHA1600MB/s


SATA3600MB/s


Acquisition 1TB @ 67.8MB/s = 4h 06mVerification 1TB @ 106MB/s = 2h 36mTOTAL = 6h 42m

Deflate67.8 MB/s


Forensic Imaging v2.1: Threaded EWFGuymager (2008), X-Ways, recent ewfacquire

MD5

Deflate DeflateDeflate

Source Hard Drive

ACMECo.C1.D1.e01



Lacklustre throughput reports (2013)

• Practitioner reports– Low 100’s MB/s [Zimmerman 2013]

• Research publications– FastDD <= 110 MB/s [Bertasi & Zago 2013]

• Our experience– Low powered CPU’s give low throughtput


Threaded EWF Acquisition240GB Intel 730 SSD 50% full, Core 2 Duo (Lenovo X200 circa 2009)


storage

SHA1SATA3Intel 730 SSD~500MB/s

USB3500MB/s

SATA3Samsung840 EVOSSD~500MB/s

Acquisition 240GB @ 91MB/s = 40m 21s

Deflate45 MB/s per core

SATA2300MB/s

Our approach to increasing I/O throughput


Scale to 8-core i7 & uncontended IO?Threaded EWF is CPU bound


storage

SHA1600MB/s

SATA3Intel 720 SSD~500MB/s

SATA3600MB/s

SATA3Samsung850 EVO Pro~500MB/s

Acquisition 240GB @ 255MB/s = 14m 35sVerification 240GB @ 350MB/s = 10m 37sTOTAL = 25m 12s

Deflate31.9MB/s/core

*8 core i7-5820k @ 3.20 GHz


How about using a faster compression algorithm?

Target Storage Interconnect Hash Compress Interconnect Evidence

storage

Compression Algorithm Throughput MB/s/core*

Deflate (ZIP, gzip) 31.9Snappy (Google BigTable) 1,400LZO (ZFS) 1,540


Forensic Imaging v4.0: AFF4 (2009)

• ZIP64 based container• Storage virtualization

• Open source implementation & specification


AFF4: Storage Virtualisation

ACMECo.S1.RAID0.af4

ACMECo.S1.D1.af4 # Linear Bitstream Hash

ACMECo.S1.D2.af4


Compressed Block Storage Stream

Virtual Storage Stream (Map)



ACMECo.S1.RAID0.af4


ACMECo.S1.D2.af4




Storage virtualisation



ACMECo.S1.RAID0.af4


ACMECo.S1.D2.af4




Inter –container referencing


Linear bitstream hashing isn’t parallelizable.Max. rate ~600 MB/s on current gen. CPU’s


storage

Algorithm Throughput MB/s

SHA1 619.23MD5 745.65Blake2b 601.87


Our solution: Block based hashing.

Hash

Compress CompressCompress

Source Hard Drive

Hash Hash

Block Hashes

# Block Hashes Hash

Test standard compositionStored block size –v- LBA address

Windows 8.1 10.2G

Govdocs1 (1-75,1-40) 59.8G

/dev/random 38.4G

Empty space (zeros)

Block based hashing beats linear stream hashing with low powered multicore CPU’s

Dual core i5-3337U 1.8GHz

Sparse dataMax CPU hash

throughput

Sparse dataRead I/O limited


Block hashing shifts the bottleneck from from CPU to I/O


storage

SHA1600 MB/s/core

SATA3Intel 730 SSD500MB/s

4xSATA32.4GB/s

RAID04x SATA32TB800MB/s

SnappyAvg1.5GB/s/core

*8 core i7-5820k @ 3.20 GHz

Acquisition application Linear Acquisition Verification

X-Ways Forensics 14:35255 MB/s (15.3 GB/min)

10:37350 MB/s (21.0 GB/min)

Wirespeed (linear) 7:23500 MB/s (30.3 GB/min)

4:12888 MB/s (53.33 GB/min)

How can we take advantage of these speeds?


Block hashing shifts the bottleneck from from CPU to I/O


storage

SHA1600 MB/s/core


4xSATA32.4GB/s



*8 core i7-5820k @ 3.20 GHz

Acquisition application Linear Acquisition Verification

X-Ways Forensics 14:35255 MB/s (15.3 GB/min)

10:37350 MB/s (21.0 GB/min)

Wirespeed (linear) 7:23500 MB/s (30.3 GB/min)

4:12888 MB/s (53.33 GB/min)

Realistic?More likely USB3

or 1GbE


Idea: can we aggregate output I/O?Use 2x USB3 drives?


storage

SHA1600 MB/s/core


2xUSB31GB/s

2x SATA32TB400MB/s


*8 core i7-5820k @ 3.20 GHz


AFF4 Striping

ACMECo.S1.D1.2.af4

ACMECo.S1.D1.1.af4


Disk 1

Disk 2

Source blocks striped over multiple containers on multiple output disks


AFF4 Striping

ACMECo.S1.D1.2.af4

ACMECo.S1.D1.1.af4


Disk 1

Disk 2

A copy of the map is stored in each container.

Test standard compositionStored block size –v- LBA address

Windows 8.1 10.2G

Govdocs1 (1-75,1-40) 59.8G

/dev/random 38.4G

Empty space (zeros)

Multiple output channels increases throughputEspecially for uncompressible data

High entropy data


Multi-destination throughput is even higher for current generation drives1TB NVMe (Core i7-4578U, 2 Cores)Macbook Pro A1502 (Evimetry 2.1.0)

Acquisition technique Acquire + Verify

Evimetry Wirespeed 0:52:04

Xways + WinFE 2:48:00

Macquisition EWF 7:08:38


Multi-destination throughput is even higher for current generation drives 1TB NVMe (Core i7-4578U, 2 Cores)

Macbook Pro A1502 (Evimetry 2.2.0a)






Multi-destination throughput is even higher for current generation drives 512GB Samsung 850 NVMe w/ 4 core i5

(Evimetry 2.2.0a)





How can we analyse while we acquire?


How can we reduce latency?While maximising completeness

Latency

Com

plet

enes


Triage

IncreaseI/O

throughput?

Live analysis while we acquire?

Dynamic partial acquisition?

Live forensics


Idea: Start with a non-linear partial image and add from there

Entire disk

All allocated

Interactive analysis artifacts

High value files

Volume & FS Metadata, Memory

Analysis


Acquire and access in parallel? dd + iSCSI access to target

MD5

Source Hard Drive

ACMECo.C1.D1.raw



iSCSIRemote analysis tools



MD5

Source Hard Drive

ACMECo.C1.D1.raw




Access is contended.Poor interactive

performance (lag )



MD5

Source Hard Drive

ACMECo.C1.D1.raw




Early termination may not have a

complete filesystem


Raw Image : Non-linear acquisition via sparse raw file, driven by live analysis?

Source Hard Drive

ACMECo.C1.D1.raw



iSCSI How do you generate a hash over a non-linear image?

* X-Ways does similar, only not remote



• Non-linear acquisition• Hash based imaging

(deduplication)


Partial, non-linear, block based hashing

Hash

Compress CompressCompress

ACMECo.C1.D1.af4

Volume Metadata

Filesystem Metadata

Sparse Data

File Content

Unknown

Hash Hash

Block Hashes

Compressed Block Stream

# Block Hashes Hash

Virtual Block Stream (Map)

Source Hard Drive



• Partial acquisition – Represent what we didn’t

acquire vs. what we couldn’t acquire

• Block based hashing


Partial, non-linear, block based hashing

ACMECo.C1.D1.af4ACMECo.C1.D1.af4

Block Hashes


##


Linear Block Hash

MapHash

Block Hashes Hash

##

##


Evimetry & AFF4 Non-linear, partial physical acquisition driven by live analysis

Source Hard DriveSource Hard Drive

ACMECo.C1.D1.af4ACMECo.C1.D1.af4

Block Hashes


## Block Hashes Hash


I/O Planning &

Scheduling

Acquisition

Virtual Disk

File categories

Blocks


Partial acquisition brings reproducibility and elasticity to IR and triage

Target Storage Interconnect Hash Compress Network Evidence

storage

SHA1600 MB/s/core

SATA3Spinning disk200MB/s

1GbE100MB/s



*8 core i7-5820k @ 3.20 GHz

Partial IR acquisition 21.9GiB @ 102MiB/s = 3m 39s

Volume metadata, filesystem metadata, 16G pagefile, Registries, Logs, Link files, Jump lists, WMI CIM Repo, Prefetch, USN Journal, $Logfile, Scheduler artefacts

How can I work with AFF4 images?


Why adopt this?My toolset doesn't support AFF4.

• Wait for support from vendors?– In progress

• Convert AFF4 to EWF on fast workstation– Can be done in roughly the same time it takes to

simply copy (only compress low entropy blocks)

• Emulate Raw image in the filesystem


Virtual FS Emulation of AFF4 containers as emulated raw images


Emulated Raw is faster than native EWF.

X-Ways processing task X-Ways Native EWF X-Ways w/ Evimetry FS Bridge

Verify 0:42:00 0:08:00FS Data Recovery 0:03:35 0:03:20Hashing & header validation

1:59:03 1:05:25

Carving unallocated 0:41:00 0:44:00Total 3:25:43 2:02:09

Image: 1TB Macbook Pro i7, processed on 8 core i7 w/ RAID

How does this affect workflow?


Native EWF Acquisition vs AFF4Native EWF Processing vs AFF4 FS Bridge



Single Threaded EWF?



Multi Threaded EWF



AFF4



AFF4: Copies in half the time due to

striped acquisition over 2 x 200 MB/s

spinning disks.

EWF: I/O bound on single 200MB/s disk



AFF4: Verification completes in 8m. I/O

bound by RAID.

EWF: CPU bound



AFF4: Filesystem search in around ½

time.

EWF: CPU bound?



AFF4 & EWF around the same throughput.

Will the courts accept the AFF4 format?


Courts accept expert evidenceIs it reliable?

• Is the expert reliable?

• Is the underlying theory reliable?– Reliable by way of the application of Scientific methods (eg.

Daubert)– 4 scientifically peer reviewed papers, unrefuted

• Are the methods implementing the theory reliable?– Tool testing (as always, the expert’s ultimate responsibility)

AdoptionWho is using AFF4?


AFF4 is used in the following


Near Future• Evimetry Community Edition

– Free creation, conversion & consumption of AFF4 images (Windows)

• AFF4 Standardisation Effort (AFF4 Working Group)– Bradley Schatz (Evimetry), Michael Cohen (Google) chairing– Open source implementation and specification in progress– Blackbag recently joined

• Sleuthkit/Autopsy– Support planned

• Open Source Digital Forensic Conference 2016– AFF4 status update

More information


More informationImplementations• https://evimetry.com/ • https://github.com/google/aff4• http://www.rekall-forensic.com/docs/Tools/• https://github.com/google/grr

Ongoing specification and papers• http://www.aff4.org/ • http://dfrws.org/2009/proceedings/p57-cohen.pdf• http://dfrws.org/2010/proceedings/2010-314.pdf• http://dfrws.org/2015/proceedings/DFRWS2015-16.pdf

https://evimetry.com/

https://github.com/google/aff4

http://www.rekall-forensic.com/docs/Tools/

https://github.com/google/grr

https://github.com/google/grr

http://www.aff4.org/



http://dfrws.org/2009/proceedings/p57-cohen.pdf



http://dfrws.org/2010/proceedings/2010-314.pdf

http://dfrws.org/2010/proceedings/2010-314.pdf

http://dfrws.org/2015/proceedings/DFRWS2015-16.pdf

http://dfrws.org/2015/proceedings/DFRWS2015-16.pdf

Conclusion


Conclusion

• Optimising forensic workflow is a systems problem

• Existing forensic formats are a bottleneck for todays systems

• Existing forensic image formats are generally incompatible with triage and reproducible live analysis

• The Advanced Forensic Format 4 solves the above

Contact

Hard disk head by amckgillFootprints by kimba

Dr Bradley Schatzhttps://evimetry.com/[email protected]

http://www.flickr.com/photos/amagill/89623319/

http://www.flickr.com/photos/kimba/4314602/

https://evimetry.com/

mailto:[email protected]

Accelerating forensic and incident response workflow: the case for a new standard in forensic...

Data & Analytics

Transcript of Accelerating forensic and incident response workflow: the case for a new standard in forensic...