Goals:
- provide C API to FreeArc functionality, similar to 7z.dll
- provide C APIs for each stage of command processing
- make these APIs stable and accessible from any language by using ProtoBuf to serialize all parameters
- provide alternative low-level C APIs for all these functions with direct access to C structures
- may be, provide some extra functionality to 7z.dll:
- add ProtoBuf interfaces to 7z.dll functions
- combine FreeArc cmdline processing with 7z.dll archive processing, essentially allowing FreeArc to process any archive type
We can classify provided APIs by the level of micro-management:
- Command - highest-level, executes an entire archiver command
- Operation - provides individual operations from which a command can be constructed
We also discuss:
- Fast vs Stable API - internal and external API provided by the library
- ProtoBuf decoder - minimal ProtoBuf implementation required for the Stable API
The implementation plan:
- Move main() to Lua side
- Modify FreearcCommand() to use fields in the Command instead of global variables
- Implement PortableFreearcCommand() that uses ProtoBuf for all parameters, including callbacks
- Modify Lua main() code to work with internal Command and then call PortableFreearcCommand()
- Further split FreearcCommand() into operations and provide ProtoBuf APIs to them
- Provide ProtoBuf APIs to 7z.dll functions and combine Lua cmdline processing with 7z.dll archive processing
FreearcCommand():
- executes an entire archiver command, i.e. its only effect is the modification of disk files and/or some callback calls
- returns almost nothing, probably only error code - all other information is passed back via Callback
- thus has C ABI, even if unstable. Also, it looks pretty similar to 7z.dll API
- stable API requires only unpacking Command and packing Callback parameters
- its execution creates, modifies and finally destroys a FileDB object (see Operation)
Call parameters:
- version = high<<16 + low
- high=0, incremented at incompatible API changes
- low = sizeof(Command), thus allowing to copy passed Command over initialized actual Command structure that may contain extra fields at the end
- Command* - all command parameters, established by the processing of cmdline with config files
- i.e. {char* cmd = "a"; char* arcname = "a.arc"; char** filenames = {"."}; char** cmethods = {"lzma"}...}
- void* - passed to callback
int(*)(void*, int, message*)
- callback called for UI/log, warnings/errors, file/method filtering and so on- int - message type
- message is a union of structures providing extra info depending on the message
Portable proxy to FreearcCommand(), generated from ProtoBuf description of Command via ProtoBuf reflection:
- declaration of C++ struct Command, including default values of fields
- PortableFreearcCommand() decoding wire format of Command and then calling FreearcCommand()
- Lua/Python/Ruby... code encoding Command{cmd,arcname...} object into the wire format and then calling PortableFreearcCommand()
All modifications required to the current code:
- move main() to the Lua side, so it parses options and then calls FreeArcCommand(Command,Callback)
- make all C++ APIs, provided to Lua, global, i.e. independent of the C++ Program object
- well, inside Callback execution we may need some instance-specific info, calling back for it once again
- modify Lua code to work with internal
command
variable before passing it to FreeArcCommand() - modify Callback API to make it language-agnostic (now, it depends on C++/Lua bridge facilities)
- we have a few message types, each having a fixed set of parameters - this calls for a serialization library, again
Managing a database:
- db = Create() - create an empty database.
- Clear(db) - reset the database to an empty state.
- Close(db) - free memory occupied by the database.
Adding files to an archive:
- Scan(db, scan_settings) - scan the disk according to the settings, adding found files to the database
- Sort(db, sort_criteria) - sort files in the database
- Add(db, compression_settings) - create an archive from the files in the database
Extracting files from an archive:
- ReadDir(db, readdir_settings) - read the archive directory, adding these files to the database
- Size(db) - return the number of files in the database
- List(db, range) - list files in the given range
- Extract(db, extraction_settings) - extract files listed in the database from the archive
Two API sets:
- Fast API: the actual C++ implementation, employing multiple function parameters, C structs, and C++ exceptions. It isn't intended to be consumed directly, except for reaching maximum speed in some cases (custom filters, custom sort order, listing files).
- Stable API: a thin layer over the Fast API, marshaling most arguments/results/exceptions in ProtoBuf messages. It thus provides a stable, language-agnostic API that can be consumed by static or dynamic linking. It semi-automatically maps to the Fast API.
Notes:
- Fast API is almost the existing one, except for the need to break the monolithic program_options structure into ~5 smaller structures with parameters of each specific operation
- Stable API is mostly straightforward as far as we define those small structures via ProtoBuf
- but the existing ProtoBuf implementations seem over-bloated, do we need N+1 one?
- alternatives: Thrift, FlexBuffers/FlatBuffers
- the remaining FreeArc'Next code is already mostly written in Lua, but we will still need to translate C++ remnants
- the existing program driver should be modified to run Lua main and to allow loading FreeArcLib from DLL