Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fcli * action: Add support for file appends & record writers #543

Open
rsenden opened this issue May 21, 2024 · 0 comments
Open

fcli * action: Add support for file appends & record writers #543

rsenden opened this issue May 21, 2024 · 0 comments
Labels
effort:medium Medium effort needed to implement/fix fcli-actions feature A feature that's on the roadmap or in development prio:high High priority; important new feature or bug fix

Comments

@rsenden
Copy link
Contributor

rsenden commented May 21, 2024

With the current action framework, output data is collected in memory and then written at once to an output file or console using the write instruction. At the moment, the write instruction overwrites any existing output file; there's no support for appending to an existing file.

We should consider adding an explicit append: true|false property to write instructions, to allow existing files to be appended to. If we do so, we also need to have an ability to clear (contents of) an existing file, i.e., on action start we'd clear (contents of) any existing file, and then in later steps append data to this clean file.

In addition or alternative to the above, we should also consider adding support for record writers that allow for appending individual records in various output formats (JSON, CSV, YAML, table, ...), similar to the existing fcli output framework. This serves 2 main purposes:

  • Data can be gradually written during action execution, rather than having to collect all data in memory first, and therefore results in lower memory consumption.
  • Data can be easily written to various output formats, for example actions won't need to manually construct CSV records and then use generic text output to write these records.

Potentially we could reuse functionality provided by the fcli output framework for implementing this feature.

If we implement this functionality, ideally we should also provide support for writing partial output to temporary files, and then combining these files to generate the full output. For example, for SARIF files, we separately collect rules and results, which now get combined in memory when writing the output. Ideally, we'd have separate JSON record writers for rules & results, and then stream the contents of those temporary files into the final SARIF output file.

@rsenden rsenden added feature A feature that's on the roadmap or in development fcli-actions labels May 22, 2024
@rsenden rsenden added prio:high High priority; important new feature or bug fix effort:medium Medium effort needed to implement/fix labels May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
effort:medium Medium effort needed to implement/fix fcli-actions feature A feature that's on the roadmap or in development prio:high High priority; important new feature or bug fix
Projects
None yet
Development

No branches or pull requests

1 participant