Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[compression] Enable compression through ros2bag cli #263

Merged
merged 27 commits into from
Feb 7, 2020
Merged

[compression] Enable compression through ros2bag cli #263

merged 27 commits into from
Feb 7, 2020

Conversation

zmichaels11
Copy link
Contributor

@zmichaels11 zmichaels11 commented Jan 22, 2020 •
edited
Loading

Changes

  • Add --compression-mode and --compression-format arguments for ros2bag
  • rosbag2_transport instantiates SequentialCompressionReader or SequentialCompressionWriter if both compression-mode and compression-format are specified.

Testing

  • Add e2e test to verify bagfiles are compressed
  • Add e2e test to verify compressed bagfiles can be read

@zmichaels11 zmichaels11 force-pushed the compression-cli branch 2 times, most recently from a6c2213 to db94ab0 Compare January 22, 2020 17:43
@zmichaels11 zmichaels11 marked this pull request as ready for review January 22, 2020 18:07
@zmichaels11 zmichaels11 mentioned this pull request Jan 22, 2020
Closed
2 tasks
piraka9011
piraka9011 approved these changes Jan 23, 2020
View reviewed changes
Copy link
Contributor

@piraka9011 piraka9011 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor changes but LGTM.

dabonnie
dabonnie approved these changes Jan 24, 2020
View reviewed changes
Copy link
Contributor

@dabonnie dabonnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than some nits seems fine.

@zmichaels11 zmichaels11 force-pushed the compression-cli branch 2 times, most recently from 96deb6d to 089c971 Compare January 24, 2020 00:30
@zmichaels11
Copy link
Contributor Author

Rebased onto master and resolved redundancy issue.

@zmichaels11
Copy link
Contributor Author

Any comments?
@Karsten1987
@thomas-moulard
@mm318
@emersonknapp

I'd like to merge this tonight if possible.

thomas-moulard
thomas-moulard reviewed Jan 24, 2020
View reviewed changes
rosbag2_tests/test/rosbag2_tests/test_rosbag2_record_end_to_end.cpp
// TODO(zmichaels11): Remove when stop_execution properly SIGINT on Windows.
// This is required since stop_execution hard kills the proces on Windows,
// which prevents the metadata from being written.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean this feature is broken on Windows? Did we test this on this platform?

@thomas-moulard
Copy link
Contributor

Somehow I can't approve this PR, but this LGTM. Let's make sure we test rosbag2 on windows and OS X too. The TODO in the test is worrying.

thomas-moulard
thomas-moulard approved these changes Jan 24, 2020
View reviewed changes
Copy link
Contributor

@thomas-moulard thomas-moulard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment.

@zmichaels11
Copy link
Contributor Author

@ros2/aws-oncall - please run this CI job
Gist: https://gist.githubusercontent.com/zmichaels11/0f8a5a8cf63f22aced353085c523f9ef/raw/77155bc93aa546d2e8c8b901f8d13e98785a4e42/ros2.repos
BUILD args: --packages-up-to rosbag2_tests rosbag2_compression rosbag2_transport rosbag2_cpp
TEST args: --packages-select rosbag2_tests rosbag2_compression rosbag2_transport rosbag2_cpp
Job: ci_launcher

@zmichaels11
Copy link
Contributor Author

zmichaels11 commented Jan 24, 2020 •
edited
Loading

  • Linux Build Status
  • Linux-aarch64 Build Status
  • macOS Build Status
  • Windows Build Status

@zmichaels11
Copy link
Contributor Author

This was failing earlier due to me configuring the build wrong.
It looks like its passing now. I'm going to merge this tonight if all CI is green.

@zmichaels11
Copy link
Contributor Author

The e2e tests on windows are failing due to the SIGINT workaround for Windows.
The last bagfile is compressed in the dtor. Because of this, it gets skipped in record_end_to_end_test_with_zstd_file_compression. I'm going to rewrite that test to ensure the bagfile splits at least once and verify that the first bagfile is compressed properly.

Similarly, record_end_to_end_with_splitting_bagsize_split_is_at_least_specified_size is unable to pass since the last bagfile is not compressed. I'll relax this test and skip checking that the last bagfile is < uncompressed size.

Karsten1987
Karsten1987 approved these changes Jan 28, 2020
View reviewed changes
Copy link
Collaborator

@Karsten1987 Karsten1987 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do approve as I think the tests cover the new functionality (please make sure CI turns out to be green).
But I am getting a bit concerned about this windows workaround as well, as we copied this one all over the place now within the end to end tests.

@zmichaels11
Copy link
Contributor Author

zmichaels11 commented Jan 28, 2020 •
edited
Loading

  • Linux Build Status
  • Linux-aarch64 Build Status
  • macOS Build Status
  • Windows Build Status

@zmichaels11
Copy link
Contributor Author

zmichaels11 commented Feb 6, 2020 •
edited
Loading

Ran successfully on local machine.

  • Windows Build Status
    Edit: fixed linters
  • Windows Build Status

@zmichaels11
Fix linting
c7d3b94 
Signed-off-by: Zachary Michaels <[email protected]>
@zmichaels11 zmichaels11 mentioned this pull request Feb 6, 2020
Merged
@zmichaels11
Copy link
Contributor Author

zmichaels11 commented Feb 6, 2020 •
edited by piraka9011
Loading

This is working now. I'm going to refactor the work done into single-focus PRs
Edit: Refactored into:

This PR should be rebased onto master once both of those PRs are merged in.

@zmichaels11 zmichaels11 mentioned this pull request Feb 6, 2020
Merged
Anas Abou Allaban and others added 12 commits February 7, 2020 10:57
Add compression flags to CLI
9d251f2 
Signed-off-by: Anas Abou Allaban <[email protected]>
@zmichaels11
ament export rosbag2_compression
5f7c960 
Signed-off-by: Zachary Michaels <[email protected]>
@zmichaels11
Add e2e test for recording compressed bagfiles
3d5d868 
Signed-off-by: Zachary Michaels <[email protected]>
@zmichaels11
Add e2e test verifying compressed bagfiles are readable
c2a81e1 
Signed-off-by: Zachary Michaels <[email protected]>
@zmichaels11
Remove redundant initialization of Reader and Writer
fc3b6aa 
Signed-off-by: Zachary Michaels <[email protected]>
@zmichaels11
Disable e2e compression test on Windows that depends heavily on metadata
dbc986b 
Signed-off-by: Zachary Michaels <[email protected]>
@zmichaels11
Update test to skip last split on windows
f71737e 
Signed-off-by: Zachary Michaels <[email protected]>
@zmichaels11
Add log statements to expects
77c1a28 
Signed-off-by: Zachary Michaels <[email protected]>
@zmichaels11
Assert only that bagfile split with compression
1cf8d38 
Signed-off-by: Zachary Michaels <[email protected]>
@zmichaels11
Rebase onto master
7d932e1 
Signed-off-by: Zachary Michaels <[email protected]>
@zmichaels11
Fix linting
db778a8 
Signed-off-by: Zachary Michaels <[email protected]>
Fix merge conflicts, linting, and rebase
4a9feb3 
Signed-off-by: Anas Abou Allaban <[email protected]>
@piraka9011
Copy link
Contributor

piraka9011 commented Feb 7, 2020 •
edited by zmichaels11
Loading

Kicking off CI after rebase:

  • Linux Build Status
  • Linux-aarch64 Build Status
  • macOS Build Status
  • Windows Build Status
  • Windows-container Build Status

@zmichaels11
Copy link
Contributor Author

zmichaels11 commented Feb 7, 2020 •
edited
Loading

It looks OSX test is failing due to the test fixture trying to access SQLite mid-compression.
Rewrote that test to not depend on querying SQLite while publishing.

  • Linux Build Status
  • Linux-aarch64 Build Status
  • macOS Build Status

Edit: It looks like its passing. Rerunning CI on Linux only since the test is not included in Windows.

@piraka9011 piraka9011 merged commit 714d4c8 into ros2:master Feb 7, 2020
@zmichaels11 zmichaels11 deleted the compression-cli branch February 7, 2020 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@emersonknapp emersonknapp emersonknapp left review comments

@piraka9011 piraka9011 piraka9011 approved these changes

@thomas-moulard thomas-moulard thomas-moulard approved these changes

@Karsten1987 Karsten1987 Karsten1987 approved these changes

@dabonnie dabonnie dabonnie approved these changes

@mm318 mm318 Awaiting requested review from mm318

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zmichaels11 @thomas-moulard @piraka9011 @emersonknapp @Karsten1987 @dabonnie