ChangeLog

TODO
* change ltree to read directory entries on demand
* calculate the MD5 while incremental ltree read, use line reader instead?
* ewfinfo -H complete bodyfile support

* info_handle: move dfxml functions to seperate file
* clean up libewf_chunk_group_correct_v1

* ewfexport test with different bytes per sector between input and ouptput ?

* ensure a multitude of sector size is written

* handle corruption scenarios
** missing single segment file
** corrupted single segment file

* rename attribute to extended attribute?

* move overlapping/generic header and hash value functions into value table
** libewf_hash_values_get_identifier_size
** libewf_hash_values_get_identifier
** move value table functions into libfvalue?

* move use of libfvalue_table_get_value_by_identifier out of header_values and hash_values
* refactor libewf_hash_sections_set_digest_from_hash_values move into hash_values?

* change single files tree to B-tree
* access control entry / permission group add get_owner and get_group functions
* file entry add 0 if not set support for values
* file entry get permission group from single files

* ewftools ignore name of root - check ewfmount / ewfverify

* be able to handle large chunk tables for version 2
  - move chunk list access into chunk group
  - throttle maximum chunk group size
* pattern fill chunk support (set chunk table entry, test)
* complete: libewf_chunk_table_fill_v2

* move write resume functionality into ewfacquire ?
* add support for write resume of images with ENCASE1/SMART table sections ?
* check behavior of ewfacquire resume and specified offset

* pass zero byte empty block compressed data cache as a pointer
  do not copy data to compressed chunk cache

* allow to set the number of values in the queue, thread pools and list

* memory usage of acquisition of Ex01 file? - chunk group size?
* memory usage of acquisition of Ex01 file? - -x versus non-x
* speed of verification of Ex01 file? - debug outout related?

* clean up TODO refactor notes
* look into issues with high speed variations between chunk and non-chunk modes
* process status change the code to have to determine decimal point only once

* add multi threaded tool support
  - ewfmount - fuse multi thread support ?
* throttle the size of the sections list?
* ewfacquire allow to set the the maximum number of table entries?

Threat safety
* libewf_metadata
 - libewf_handle_copy_header_values (write lock)
 - libewf_handle_copy_media_values
 - use on_error (continue from libewf_handle_get_header_value_identifier_size)
 - write lock libewf_handle_get_ ... _hash_values
 - refactor libewf_handle_set_format
 - refactor libewf_handle_set_md5_hash
 - refactor libewf_handle_set_sha1_hash

Multi-threading:
* make process by chunks the default

Tests:
* extend ewf_test_handle_get_values with
  - session informatation
  - tracks
  - acquiry_errors
  - checksum_errors? (needs read)
  - libewf_handle_set_md5_hash
  - libewf_handle_set_sha1_hash
  - header value date and time formats?
* clean up ewf_test_read_chunk.c to match ewf_test_read.c
* write resume from volume/disk section - basically re-image
* ewfacquire:
  - add non-unattended tests
  - add option sets for single/multi threaded, chunks
* ewfacquire_optical:
  - add non-unattended tests
* ewfexport:
  - add non-unattended tests
  - add option sets for single/multi threaded, chunks
  - export of split RAW
  - add support for different input and output chunk sizes?
* ewfexport_logical:
  - add non-unattended tests
* ewfinfo:
  - add DFXML output tests
*ewfverify
  - add option sets for single/multi threaded, chunks

Error tolerance:
* add setting to be less strict
  - non matching data section (set identifier)
  - expand number of chunks

Roadmap:
* lvf support
  - add support for permissions
  - thread aware support (check)
* error tollerance
  - allow for missing segment file(s)
  - allow for missing section(s)
  - allow for missing chunk(s). Can the chunks be shifted to match up with the
  last segment file.
* bindings
  - Python (partial)
  - .Net (partial)
  - Java/JNI
* improve device handle (libsmdev)
  - audio track support - libsmdev
* libbfio API extensions (partial)
* low and high level API
  - to allow low level access to sections etc. (for recovery)
  - allow access to segment table
  - allow access to offset/chunk table
  - ewfdebug
* internationalisation/localization (i18n/l10n)
  - Unicode support (added)
  - locale support (partial)
  - multi language string support (gettext)
    need WINAPI version of gettext without libiconv dependency
* library/tools enhancements
  - getoptlong
* ewfx enhancements
  - xsignature section to sign xhash and/or hash

Code clean up:
* clean up configure.ac and acinclude.m4
  - refactor dfxml check
* ewfinfo
  - move DFXML code to separate code files, what about separate library ?
  - implement a statefull XML writer e.g. open/close_section, open/close_value
* check TODO in code, mainly regarding improvements

BFIO support:
* improve open file io pool (set basename ?)

Further analysis of the EWF format:
* password hash
* volume/disk section (EWF-E01)
   - do EnCase 5 or earlier versions support 64-bit number of sectors?
   - unknowns
* hash section (EWF-E01)
  - 16 byte unknown
* values in EnCase5/6 header2 (+/-)
* values in ltree strings (EWF-L01 LVF)
* values in session section (+/-)
* number of sectors in volume/data section has changed to a 64-bit value
  but what about error2/sessions ? (sessions can be 32-bit)
* does newer linen6 has sha1 support?
* LVF: single file entry:
  - are flags similar to file attribute flags ?
  - parse first entries differently ? for file hierarchy
* FTK Imager: support for "Use AD Encryption"
  - requires support for ADCRYPT file format

Features
* add bzip2 support and tests
* implement low memory usage option
  - trade off IO versus memory use
* disk full check
  - how to detect this condition (cross-platform) ?
  - solution (A) at creation time of the segment file:
    linux/unix posix_fallocate(), does not provide the desired result
    windows SetFileInformationByHandle()
  - solution (B) on write (backtrack last chunk)
    implemented close segment file on LIBERROR_OUTPUT_ERROR_INSUFFICIENT_SPACE 
    although if linux write fails (write_count != write_size) before ENOSPC is set
    there is still room left on the device, but cannot be claimed by libewf
* what about header values parsing with preservation of timestamps?
  - change localtime to gmtime in libewf_date_time_values_copy_to_string ? (NO!)
  - force set header value to require string formatted in the date format ?

Recovery and analysis:
* imaging: detect disk full and ask for new segment file location?
  - have libewf allow to set the segment file path during write (done)
  - have separate function to preallocate segment file?
  - like alternative segment file path in EnCase
    however EnCase seems to fully test if the segment file upfront by
    allocating the necessary disk space
    how to detect the destination has no available space ?
* recovering corrupt EWF files (missing segment file, corrupt sections, etc.)
  - can the values in data be used to correct a corrupted volume ?
* check into resume write error for 32-bit compile (different zlib sizes)
* detect consistent number of chunks in table sections

Error handling/tollerance
* what about a function that translates error into an equivalent errno?
* have wchar error messages for WINAPI?
* improve offset table correction
  - if not correctable skip chunk while reading and mark as crc error
    or try to determine offset/size of chunk somehow
* handle corruption in both table and table2 sections
  - non-overlapping corruption
  - overlapping corruption
* handle missing sections
* handle missing segment files
* handle corrupt segment files

Additional (integrity) checks:
* read: check if successive headers match first header
* read: add check for next and done segment
  - really the last segment in the file
  - offset pointing at themselves
  - correct size

Possible performance improvements:
* libewf_device_info_parse do this directly from the UTF16 stream ?
* libewf_case_data_parse do this directly from the UTF16 stream ?
* tune read and write buffer sizes
  - read/write block size seems to have little effect for dd on average
  - buffered bfio ?
* reduce alloc, realloc and free (this reduces breaks) use strace to analyze
  - pass values as parameters or use of handle
  - what allocation causes the break in strace of ewfacquire with compression libewf? zlib?
* optimize string functions
  - pass length/size as much as possible
* do not write section start on data update
  - performance improvement is insignificant
* read certain section data on demand ?

Python bindings (pyewf)
* add WINAPI wide character support for open() and glob()
* add more functions
  - sessions
  - acquiry errors
  - read errors

.Net bindings (ewf.net)
* handle 
  - add more functions
* file entry
  - add more functions
* mono support ?

API changes
* low level API
  - segment file access
  - section access
  - chunk table access
  - add function to retrieve EWFX xml header/hash directly
* chunk table access
  - rewrite read/write chunk functions, pass chunk index
* add wide character functions definitions by default?
  - not necessary on most platforms
* libbfio API extension
  - add means to obtain libbfio handle of media data in ewf
  - add means to set narrow string codepage
  - export libewf_glob_file_io_handle
    refactor glob functions to use libbfio handle input and return libbfio pool?
    think about libewf_glob_file_io_handle and abstration of file io handle get/set name
* LVF support
  - add LVF write support: libewf_add/remove_file
  - add support for single file entry with "no content"
* file structure API extension (idea)

Debug
* print offsets of segment files

Tools:
* if one option available just print and continue
* if set by command lien just print and continue
* port multi threaded ewfacquirestream
* add multi threading support
* allow to set date and time
* locale support
  - print output convert to system string
  - reduce string conversions, use as much as possible the system string type in the tools
* when writing check getrlimit( RLIMIT_FSIZE )
* check char/wchar in/output
* add getoptlong support
* make compilation resilient for missing crypto library
  - compile on system without libcrypto
  - remove the cryptographic options if crypto library is missing
* have failed on open return a more sensible error (extract from error)
* process status
  - improve and remove "This could take a while." ?
* libsystem path implement GetVolumeInformation
* ewfacquire:
  - print status info without confirmation in unattended mode
  - restrict error granularity and amount per sectors to > encase5 and > linen5 format ?
    no, although not set in EWF file, the granualarity is still used in handling read errors
  - warn if values exceed maximum segment count
  - adjust error granularity if error granularity < process buffer size
    should not matter for libsmdev
  - warn if media does not fit using chunk size or APIv1
    a (32-bit) chunks size of ( 64 * 512 ) will allow up to 128 TiB
  - for EnCase 6 format request serial number and model if not set ?
  - add support to manually provide sessions for optical images ?
  - add support for unattended write resume ?
  - improve output when encountering a read error
  - add support to set device label header value
  - allow for multi stage imaging, where user suspends and resumes imaging process?
  - add multi threading support
* ewfacquirestream
  - implement stream handle
  - add multi threading support
* ewfdebug
  - create a tool to analyse a single segment file
  - implement low and high level api separation
* ewfexport:
  - if EWF file is empty still write an emtpy file
  - add support for multi threaded logical export
  - EnCase7 L01 ignore root entry like ewfmount
  - support for volume letter in -t argument
  - add support for different output chunk mode
  - implement more ewfacquire acquiry parameters?
  - handle different input and ouput chunk sizes in low level functions build
  - allow different input and output header codepages?
  - handle ewf/raw like in smioexport
  - creates write only raw export file with msvscpp compilation
  - limit output formats for input types (lvf vs ewf)
  - add information and media values to raw output
  - improve -f files error tollerability
  - add multi threading support
  - add multi threading support for logical images
* ewfmount
  - Windows finish Dokan implementation
  - Windows add ImDisk implementation ?
  - MacOS native API support ?
  - add multi threading support
  - add multi threading support for logical images
* ewfverify:
  - allow for multi stage verification, where user suspends and resumes?
  - for multi threading measure: number of items queued, number of items
  requeued, number of threads
  - add multi threading support for logical images
  - add support for multi threaded logical verify

Unsorted:
* user header and header2 time difference to calculate the timezone

* fix last_chunk_data_offset calculation for table2
  in combination with libewf_chunk_group_correct_v1

  - sections list on write: cache section descriptors instead of free-ing them?

  - segment files cache on write set to 1 ?

  - create a single chunk groups cache ? instead per segment file ?

  - check if chunks groups list is filled in on demand scenario

  - only use backup range (table2) during initial phase?
    - nope also necessary during reread

  - use mapped offset to determine first chunk index (+/-)
    - what about base offset ?
    - make mapped offsets absolute ?

  - handle: replace chunk_index calcuations by current_chunk_index tracking ?
  - libewf_filename_create: change to accept ( segment_number - 1 )
  - libewf_write_io_handle_initialize_resume: re-map mapped offset ranges ?

  - write_io_handle->number_of_table_entries still needed ?

  - add write support to the various fdata lists ?

    and refactor it to have a generic solution if range list is not set fall back

* pyewf
  - change header and hash values into sequence objects

* add function to retrieve:
  - compression values

* add support for set identifier change ?
* check alignment of:
  section_data_read CHS cylinders, SMART logs start sector

* complete ewfrecover
  - max segment size
  - try to read the chunks by calculating the checksums and see where they align

* corruption scenarios
  - libewf_segment_file_read deal with corrupted segment file header ?
  - add strict chunk data flag handling
  - document how libewf deals with corruption scenarios
  - valid table and table2 with different information
    (for now use table if number of entries is not 0)
  - tainted table and table2
  - tainted table or table2 (use the one that is not tainted)
* handle DEFLATE uncompressed block data with copy of uncompressed data size of 0?

* add stream write support for EWF-S01 use fill space solution in table section

* fix double _FILE_OFFSET_BITS=64

* fix output: number of entries: 19175 exceeds maximum: 16375

* add light-weight function to determine byte frequency ? to check if compression is viable ?
* work on deduplication support

* allow zlib to be replaced by alternative back-end

* libewf_filename_create split off wide variant ?

* check if compression level none is passed for EWF-S01
  - validate SMART forced compression of uncompressed chunks

* Ex01/Lx01 format
  - encryption support (read/write)
  - analytical data
    - parse (read)
  - case data (write)
    - compression method support
    - write blocker type support
  - device information
    - check TODO (read/write)

* Ex01/Lx01 test images
  - memory image -> winen and encase ?
  - resumed ?
  - encrypted
  - open single files ?

* Ex01/Lx01 read support
  - add generic way to parse file object data ?
  - make sure to correctly deal with aligment padding (should only affect one of the Lx01 tables)

* Ex01 write support
  - enable secion data integrity hash support
  - add support for write resume

* Lx01 write support
  - make sure to correctly deal with aligment padding (should only affect one of the Lx01 tables)

* Ex01/Lx01 extensions
  - implement SHA256 hash section
  - allow pattern fill to have a size
  - hash based imaging?

* pyewf
  - update (+/-)
  - file entry data ? instead of file entry
* ewfmount
  - check access rights on fuse mount point
  - add stat info on readdir (see regfmount)
  - provide means to have a similar naming approach as mount_ewf.py
* check performance


Performance:
* map full RAW storage media data into memory?
  - libsigsegv?
  - mmap?

Deployment:
* spec file
  - make flex, yacc include flexible ?
* add BORLANDC deprecated function support, unable to find suitable macro
  #pragma deprecated( function )
  __declspec(deprecated)
* windows build
  - some of the CRT functions are MVC 2005 or later specific
  - figure out a way to warn about the compiler used
  - add VC_EXTRALEAN to msvscpp config
* integrating libewf.rc into mingw dll ?

Known issues:
* error acquiring USB floppy with bad blocks under FreeBSD - no problem with internal floppy drive
  virtualisation could be part of the problem
* check chunk size for > 2 GiB table offsets
  the current assumption is that these chunks only can be uncompressed
  because the MSB is normally used to mark if the chunk is compressed
  This actually is a bug in EnCase 6.7.1 the behavior is gone in EnCase 6.8
* MS Visual Studio C++ does not allow pre compiler macro in macro definitions
* Cygwin compilation missing __xpg_strerror_r - bug in recent Cygwin 1.7.8

Closed:
* libewf_single_files refactor to use UTF-16 split string
  - does not really matter still need to do conversion to runtime UTF-16
* write resume, allow resume from volume/disk section
  - why bother no media data has been written anyway
* EWF2 can handle more then UINT16_MAX segment files - add support for this?
  - having more than 65000 segment files is a bit overkill,
    also the naming schema does not support this without creating odd names
    for now not adding support for this, unless there is a good reason to do so

20060820
* see `git log' for more recent change log
* ewfacquire: small change to confirmation

20060819
* lifted the 255 segment file limit on write, however it uses an (horrible) increment based naming scheme much like EnCase
  write: allow more than 255 segment files - completed
* write: worked on SMART (EWF-S01) support
* write: SMART (EWF-S01) - build in support for table2
* all: add SMART support - complete
* worked on - all: due to 20060505 fix give all value definitions a default value - looks nicer
* due to changes to the read and write logic the library was retested
* read: error in percentage in libewf_read_random - works on FC5
* table size limitation does not seem present in EWF-E01 only in EWF-S01
* use autoconf/automake cross compilation parameters for platform determination - completed

20060818
* read: worked on SMART (EWF-S01) support
* alternative read method was removed, the assumption it was based on is no longer valid
  read: remove alternative read method the assumption behind is fails for the SMART format - completed
* renamed alternative write method to wipe block on read error
  write: rename the alternative write method to wipe_block_on_error (or something) - completed
* read random: chunk based size check now is a buffer based size check
* removed handle->chunk_crc no longer needed
* for the EWF-S01 format the entries in table2 are appended to the offset table - have yet to test a file with a table2 section
* write: worked on SMART (EWF-S01) support

20060817
* read: worked on SMART (EWF-S01) support

20060816
* ewfacquire: add overview to confirm user input, otherwise restart input query - completed
* ewfacquire: make encase5 file option default - completed
* fixed incorrect order for values in header2 on write - also in documentation

20060812
* made the number of read error retries flexible and user adjustable through ewfacquire
* correction of "int result" in file_write to be C89 code compliant
* new file size algorithm to fill up closer to max file size when compressing
  write: change algorithm to fill files upto file size - completed
* added empty block compression
  write: always compress empty chunks - completed as an option

20060809
* fixed size bug for large images on Cygwin - fix of 20060610 not sufficient for Cygwin
* fixed invalid value in printf volume read verbose messages
* added autoconf/automake structure

20060708
* renamed endian.h to libewf_endian.h for compatibility issues

20060617
* put conditions around debug code in eqfacquire

20060611
* fixed - one off bug in header value end of string

20060610
* ewfacquire test on multiple platforms
* added support for SMART format volume section - equals the original EWF volume section definition
* fixed - type casting error in data size
* fixed - missing header for encase1 format in write
* fixed - bug in writing encase4 and encase5 header2 in write - error in ascii to utf16 routine
* fixed - bug in writing encase4 and enacse5 header in write - logical formatting error
* write: timestamp is now generated only once
* write: fixed incorrect free in header values
* write: fixed memory corruption bug in ewfacquire - required explicit zero of string

20060605
* fix of data corruption bug - one off
* ewfacquire: sanitize input provided by user
* ewfinfo/eqwfacquire ? 1970 date when no case data is provided - bug in split function regarding first empty value
* removed file cache - no longer required

20060527
* started analysis of a data corruption bug

20060526
* fixed write bug when read is successful after error with small error granualirity
* fixed write bug with error granularity 1
* ewfacquire: add chunk size input possibility
* ewfacquire: add error granularity size input possibility
* ewfacquire: memory corruption bug in user input - repaced some code - looks OK now - but hard to reproduce
* fixed write bug with block size 128 error granularity 16 in test
* write: improve reading of bad sectors and writing a error2 section to file for encase5 file option

20060520
* fixed media size bug - old method of calculation was still in there :-(
* worked on ewfacquire read on error
* added alternative write method to test acquired images against existing tooling
* ewfacquire: able to produce exact media data as linen
* write: what if read error is in last chunk and it is smaller than chunk size !?

20060517
* removed duplicate include from notify.c

20060516
* added media size to ewfinfo
* worked on ewfacquire read on error

20060513
* minor adjustment in libewf.h - ewf_close renamed to libewf_close

20060505
* Fix for older compilers like gcc 2.95 - rewrite to older C statements

20060424
* added warning macro to allow overriding some fatal errors
* allow overriding invalid CRC's
* refactored crc check in ewf_section to file_read - for scope of handle and uniformity within the code
* further analysis of unknown values in hash section - also found difference in signature value Encase4 and Encase5

20060423
* Fix for Solaris in definitions.h

20060422
* Fixed missing PRI definitions on OpenBSD - now defines in definitions.h
* changed sprintf to snprintf
* added device size determination for OpenBSD
* fixed bug in header values parsing for empty first value empty
* added compression level definition
* ewfinfo: add compression media information in ewfinfo - Encase5 format only
* write: moved compression type functionality from ewfacquire into library

20060419
* Adjusted write method - to fix filesize problem
* Correction of verbose print of last chunk number
* the write function works only when a data section is at the start of the file - why ? not a bug in write
  but in read the first offset in the segment table was filled with 0 instead of 13 which is the size
  of the file header

20060418
* Tested on FreeBSD 6.0 needed to add include header in ewf_sectors.h
* Adjusted Make to be more generic and to work on FreeBSD 6.0

20060415
* changed u_int to uint
* removed LIBEWF_ types
* zlib uses a long which varies per 32 and 64 bit architecture using an intermediate variable
  fixes not able to read compressed images on a x86_64 architecure
* Memory corruption bug in segment table filename fixed

20060414
* Fixed previous offset size
* removed memcpy assignments
* type fixed malloc assignments
* removed (void *) from memset
* reduced the number of int definions
* changed printf formaters
* added status indication to ewfexport using callback function in read_from_file_descriptor
* added quiet option to ewfexport and ewfacquire
* removed dependency on libiconv - created a naive utf16 to ascii conversion
* moved header string generation code to header_values
* added time string generation code to header_values
* added acquire options for compression and file type to ewfacquire
* code now partially fixed for x86_64 systems fixed malloc statement, printf sizes, and some type definitions
* additional changes to memcpy statements made
* Use header_values for write
* ewfacquire fix error print on correct fixed value
* ewfacquire fix for empty input values
* ewfacquire fix strange image file rights
* ewfacquire fix image filename weird characters - one byte off allocation problem in ewfacquire
* ewfacquire allow for a default value - for fixed values
* ewfacquire can now acquire devices on works on Mac OS-X/Darwin
* fix write crc error in chunk on mac os x
* ewfacquire add EWF file size input possibility - check upper bound 2 GiB, lower bound 1440k
* fix write bug with small file size 1440k - for floppy image - this has to do if
  both the filesize and the ewf size are the same - write will adjust with one additional segment file

20060413
* Used LIBEWF_VERSION in executables
* Added file signature function to library for integration in The SleuthKit
* Added md5.h for allowing to do with different MD5 calculation code bases
  like the one integrated in the The SleuthKit

20060412
* Created separate file for header value logic - moved code from ewfinfo

20060411
* Nearing release of first version, so started logging relevant changes
* Removed memory leak with duplicate header assignment to handle
* Renamed ewfmd5sum to ewfverify - still have to enhance functionality
* Renamed ewfcat to ewfexport - still have to enhance functionality
* Added time parsing functionality to ewfinfo - date format is currently fixed in MM/DD/YYYY
* Enhanced password and compression header value output in ewfinfo