hydro_deploy v0.5.1

Chore

• manually set lockstep-versioned crates (and lattices) to version 0.5.1
  Setting manually since
  frewsxcv/rust-crates-index#159 is messing with
  smart-release

Documentation

• actually committing empty CHANGELOG.md is required

New Features

• improve build error message debuggability
• pass subgraph ID through deploy metadata
• improve API naming and eliminate wire API for builders
• improve Rust API for defining services
• split Rust core from Python bindings

Bug Fixes

• avoid inflexible \\?\ canonical paths on windows to mitigate / separator errors

• race conditions when handshake channels capture other outputs
  Timeouts in Hydroflow+ tests were being caused by a race condition in Hydro Deploy where stdout sent after a handshake message would sometimes be sent to the cli_stdout channel for handshakes.

  This PR adjusts the handshake channels to always be oneshot, so that the broadcaster immediately knows when to send data to the regular stdout channels.

  Also refactors Hydro Deploy sources to split up more modules.

• don't vendor openssl and fix docker build

• fix docs and remove unnecessary async_trait

Commit Statistics

• 11 commits contributed to the release over the course of 39 calendar days.
• 11 commits were understood as conventional.
• 9 unique issues were worked on: #1010, #1014, #986, #987, #992, #994, #995, #996, #999

Commit Details

view details

• #1010
  • Improve build error message debuggability (20fd1e5)
• #1014
  • Avoid inflexible \\?\ canonical paths on windows to mitigate / separator errors (d23c229)
• #986
  • Split Rust core from Python bindings (c50ca12)
• #987
  • Improve Rust API for defining services (53d7aee)
• #992
  • Fix docs and remove unnecessary async_trait (1d8adc1)
• #994
  • Don't vendor openssl and fix docker build (1ae27de)
• #995
  • Improve API naming and eliminate wire API for builders (b7aafd3)
• #996
  • Pass subgraph ID through deploy metadata (46d87fa)
• #999
  • Race conditions when handshake channels capture other outputs (f8a0b95)
• Uncategorized
  • Actually committing empty CHANGELOG.md is required (3b36020)
  • Manually set lockstep-versioned crates (and lattices) to version 0.5.1 (1b555e5)

lattices v0.5.0

Chore

• Fix clippy::implied_bounds_in_impls from latest nightlies

Documentation

• fix lattice math link

New Features

• Add serde derives to collections
• Add UnionFind lattice

Bug Fixes (BREAKING)

• fix some types and semantics for atomization

Commit Statistics

• 6 commits contributed to the release over the course of 41 calendar days.
• 56 days passed between releases.
• 5 commits were understood as conventional.
• 2 unique issues were worked on: #915, #922

Commit Details

view details

• #915
  • Add UnionFind lattice (35c2606)
  • Fix some types and semantics for atomization (18e9cfa)
• #922
  • Add serde derives to collections (488d6dd)
• Uncategorized
  • Release hydroflow_lang v0.5.0, hydroflow_datalog_core v0.5.0, hydroflow_datalog v0.5.0, hydroflow_macro v0.5.0, lattices v0.5.0, hydroflow v0.5.0, hydro_cli v0.5.0, safety bump 4 crates (2e2d8b3)
  • Fix lattice math link (6b82126)
  • Fix clippy::implied_bounds_in_impls from latest nightlies (e788989)

hydroflow_macro v0.5.0

Chore

• bump proc-macro2 min version to 1.0.63
• refactor collect-String into fold for clippy::clippy::format_collect lint
  using .map(|| format!()).collect() results in a heap String allocation for each item in the iterator, the fold only does one. Doesn't really matter for this case, just appeasing the lint

New Features

• initial technically working version of demux_enum with very bad error messages
  Technically does not check port names at all, just depends on their order.
• add import!() expression
• Add monotonic_fn and morphism macros, update snapshots for flow props.

Bug Fixes

• separate internal compiler operators in docs name/category/sort order
• properly feature gate macro_invocation_path source_file() #937

Refactor

• demux_enum requires enum type name, add better error handling

New Features (BREAKING)

• Simplify graph printing code, add delta/cumul green edges, allow hiding of vars/subgraphs

Commit Statistics

• 10 commits contributed to the release over the course of 55 calendar days.
• 56 days passed between releases.
• 9 commits were understood as conventional.
• 6 unique issues were worked on: #881, #882, #884, #898, #932, #938

Commit Details

view details

• #881
  • Refactor collect-String into fold for clippy::clippy::format_collect lint (2eac8ce)
• #882
  • Add monotonic_fn and morphism macros, update snapshots for flow props. (7714403)
• #884
  • Separate internal compiler operators in docs name/category/sort order (5ac9dde)
• #898
  • Add import!() expression (f013c3c)
• #932
  • Simplify graph printing code, add delta/cumul green edges, allow hiding of vars/subgraphs (9ed0ce0)
• #938
  • Properly feature gate macro_invocation_path source_file() #937 (80d985f)
• Uncategorized
  • Release hydroflow_lang v0.5.0, hydroflow_datalog_core v0.5.0, hydroflow_datalog v0.5.0, hydroflow_macro v0.5.0, lattices v0.5.0, hydroflow v0.5.0, hydro_cli v0.5.0, safety bump 4 crates (2e2d8b3)
  • Bump proc-macro2 min version to 1.0.63 (f19eccc)
  • demux_enum requires enum type name, add better error handling (1126266)
  • Initial technically working version of demux_enum with very bad error messages (b3d1148)

hydroflow_lang v0.5.0

Chore

• cleanup cli quotes, loose TODO comment
• bump proc-macro2 min version to 1.0.63
• ignore clippy::unwrap_or_default in fold_keyed codegen
• Fix clippy::implied_bounds_in_impls from latest nightlies

Documentation

• add docs for the hydroflow surface syntax compilation process

New Features

• Add --no-handoffs option to graphwrite args

• Add --op-short-text and --no-pull-push graphwrite args

• add kvs with replication example
  have both kvs_replicated and kvs, separate examples

  add flow_props_fns to cross_join (same as join), and filter

• update snapshot tests for previous two commits

• update propegate_flow_props fn to reach fixed-point

• Add flow_prop_fns to many operators

  • _upcast (new)
  • demux_enum
  • for_each
  • inspect
  • join

• open mermaid/dot graph in browser

  • HydroflowGraph::open_mermaid() opens https://mermaid.live/
  • HydroflowGraph::open_dot() opens https://dreampuf.github.io/GraphvizOnline/

  Behind a new hydroflow/debugging/hydroflow_lang/debugging feature
  gate.

• Update documentation and improve error messages for demux_enum operator

• add type guard to demux_enum codegen

• initial technically working version of demux_enum with very bad error messages
  Technically does not check port names at all, just depends on their order.

• Implement partition operator
  Supports both named ports and numeric indices.

• add context.is_first_time_subgraph_is_scheduled to simplify replaying operators

• make lattice_fold and lattice_reduce consistent with fold/reduce

• add import!() expression

• Implement flow_prop_fn for union()

• Add Cumul flow_prop_fns for lattice_fold() and lattice_reduce()

• Update dot/graphviz rendering of delta/cumul and defer_tick edges

• Make propegate_flow_props fallible, cleanup flow_prop_fn definition.

• add cast operator

• Update mermaid rendering of hydroflow graph to show flow properties

• Add monotonic_fn and morphism macros, update snapshots for flow props.

• Move structs into separate flow_props module, make flow_prop_fn return Options, impl for map

• Add source_iter_delta op for testing, basic flow props test, cleanups.

• Implement basic flow prop traversal (untested)

• Setup structure for tracking flow properties

Bug Fixes

• restore in-subgraph rendering of self-handoffs
• fix handling of whitespace in cast expressions
• clippy warning on multiline string in hydro_cli, py_udf
• fix demux error messages and add tests
• clippy redundant to_string() in print! lints
• Clean up degenerate subgraph error message for consistency
  Makes the pinned and latest nightly version have the same stderr output
  for consistent testing.
• separate internal compiler operators in docs name/category/sort order

Refactor

• cleanup kvs example more
  Add persist flow_prop_fn
• demux_enum requires enum type name, add better error handling
• combine topo_sort and scc_kosaraju into topo_sort_scc

New Features (BREAKING)

• Simplify graph printing code, add delta/cumul green edges, allow hiding of vars/subgraphs

Commit Statistics

• 41 commits contributed to the release over the course of 53 calendar days.
• 56 days passed between releases.
• 41 commits were understood as conventional.
• 14 unique issues were worked on: #882, #883, #884, #892, #896, #898, #902, #906, #923, #924, #926, #932, #933, #935

Commit Details

view details

• #882
  • Add Cumul flow_prop_fns for lattice_fold() and lattice_reduce() (63c435c)
  • Update dot/graphviz rendering of delta/cumul and defer_tick edges (9baf80c)
  • Make propegate_flow_props fallible, cleanup flow_prop_fn definition. (fd89cb4)
  • Add cast operator (38346cf)
  • Update mermaid rendering of hydroflow graph to show flow properties (9ab7cf8)
  • Add monotonic_fn and morphism macros, update snapshots for flow props. (7714403)
  • Move structs into separate flow_props module, make flow_prop_fn return Options, impl for map (008b980)
  • Add source_iter_delta op for testing, basic flow props test, cleanups. (fd5cdb5)
  • Implement basic flow prop traversal (untested) (b2ca4b7)
  • Setup structure for tracking flow properties (686c275)
• #883
  • Combine topo_sort and scc_kosaraju into topo_sort_scc (2e61c62)
  • Add docs for the hydroflow surface syntax compilation process (a6f3c64)
• #884
  • Separate internal compiler operators in docs name/category/sort order (5ac9dde)
• #892
  • Clean up degenerate subgraph error message for consistency (159a262)
• #896
  • Ignore clippy::unwrap_or_default in fold_keyed codegen (1fb753e)
• #898
  • Add import!() expression (f013c3c)
• #902
  • Make lattice_fold and lattice_reduce consistent with fold/reduce (1ce5f01)
• #906
  • Add context.is_first_time_subgraph_is_scheduled to simplify replaying operators (d254e2d)
• #923
  • Open mermaid/dot graph in browser (e7ea6d8)
• #924
  • Update snapshot tests for previous two commits (21140f0)
  • Update propegate_flow_props fn to reach fixed-point (9686ae8)
  • Add flow_prop_fns to many operators (cff7e48)
• #926
  • Cleanup cli quotes, loose TODO comment (5943758)
  • Add kvs with replication example (d38ec08)
  • Cleanup kvs example more (cb90ae1)
• #932
  • Add --no-handoffs option to graphwrite args (13fab15)
  • Add --op-short-text and --no-pull-push graphwrite args (6dbbf35)
  • Simplify graph printing code, add delta/cumul green edges, allow hiding of vars/subgraphs (9ed0ce0)
• #933
  • Fix handling of whitespace in cast expressions (f67b9f0)
• #935
  • Restore in-subgraph rendering of self-handoffs (2edf779)
• Uncategorized
  • Bump proc-macro2 min version to 1.0.63 (f19eccc)
  • Clippy warning on multiline string in hydro_cli, py_udf (a927dc6)
  • Update documentation and improve error messages for demux_enum operator (9646ca0)
  • demux_enum requires enum type name, add better error handling (1126266)
  • Add type guard to demux_enum codegen (02fddd2)
  • Initial technically working version of demux_enum with very bad error messages (b3d1148)
  • Implement partition operator (fc25433)
  • Fix demux error messages and add tests (5a7e1b1)
  • Clippy redundant to_string() in print! lints (51a200a)
  • Implement flow_prop_fn for union() (1bdbf73)
  • Fix clippy::implied_bounds_in_impls from latest nightlies (e788989)

hydroflow_datalog_core v0.5.0

Chore

• bump proc-macro2 min version to 1.0.63
• ignore clippy::unwrap_or_default in fold_keyed codegen
• Replace or_insert_with(Vec::new) with or_default()
  Clippy lint unwrap_or_default complaining on latest nightly

New Features

• update snapshot tests for previous two commits
• add context.is_first_time_subgraph_is_scheduled to simplify replaying operators
• add import!() expression
• Implement flow_prop_fn for union()
• Make propegate_flow_props fallible, cleanup flow_prop_fn definition.
• Add monotonic_fn and morphism macros, update snapshots for flow props.

Test

• update snapshots

Commit Statistics

• 10 commits contributed to the release over the course of 42 calendar days.
• 56 days passed between releases.
• 10 commits were understood as conventional.
• 7 unique issues were worked on: #882, #893, #896, #898, #906, #924, #926

Commit Details

view details

• #882
  • Make propegate_flow_props fallible, cleanup flow_prop_fn definition. (fd89cb4)
  • Add monotonic_fn and morphism macros, update snapshots for flow props. (7714403)
• #893
  • Replace or_insert_with(Vec::new) with or_default() (7c7eea7)
• #896
  • Ignore clippy::unwrap_or_default in fold_keyed codegen (1fb753e)
• #898
  • Add import!() expression (f013c3c)
• #906
  • Add context.is_first_time_subgraph_is_scheduled to simplify replaying operators (d254e2d)
• #924
  • Update snapshot tests for previous two commits (21140f0)
• #926
  • Update snapshots (9144dd9)
• Uncategorized
  • Bump proc-macro2 min version to 1.0.63 (f19eccc)
  • Implement flow_prop_fn for union() (1bdbf73)

hydroflow_datalog v0.5.0

Chore

• bump proc-macro2 min version to 1.0.63

Commit Statistics

• 1 commit contributed to the release over the course of 8 calendar days.
• 56 days passed between releases.
• 1 commit was understood as conventional.
• 0 issues like '(#ID)' were seen in commit messages

Commit Details

view details

• Uncategorized
  • Bump proc-macro2 min version to 1.0.63 (f19eccc)

hydroflow v0.5.0

Chore

• update snapshots for previous commit
• cleanup cli quotes, loose TODO comment
• Fix string join clippy::format_collect lint
• Fix clippy::implied_bounds_in_impls from latest nightlies
• Replace or_insert_with(Vec::new) with or_default()
  Clippy lint unwrap_or_default complaining on latest nightly

New Features

• add kvs with replication example
  have both kvs_replicated and kvs, separate examples

  add flow_props_fns to cross_join (same as join), and filter

• update snapshot tests for previous two commits

• open mermaid/dot graph in browser

  • HydroflowGraph::open_mermaid() opens https://mermaid.live/

• HydroflowGraph::open_dot() opens https://dreampuf.github.io/GraphvizOnline/

Bug Fixes

• restore in-subgraph rendering of self-handoffs
• fix example kvs/kvs_replicated tests, improve comments
• surface python snapshot tests
• clippy warning on multiline string in hydro_cli, py_udf
• fix demux error messages and add tests
• Clean up degenerate subgraph error message for consistency
  Makes the pinned and latest nightly version have the same stderr output
  for consistent testing.
• Make deadlock detector filter out spruious UDP ConnectionResets on Windows
• Ignore PermissionDenied on already-closed socket on Windows
• add context.current_tick_start() without breaking the scheduler this time
• add context.current_tick_start()
• separate internal compiler operators in docs name/category/sort order

Refactor

• kvs_bench graphwrite
• cleanup kvs example more
  Add persist flow_prop_fn
• cleanup kvs example with lattice properties
  Open graphs when --graph is specified
• refactor kvs example, cleanup, use demux_enum
• update chat example to use demux_enum
• demux_enum requires enum type name, add better error handling

Test

• record snapshots for surface_scheduling.rs
• Add flow prop tests

New Features (BREAKING)

• Simplify graph printing code, add delta/cumul green edges, allow hiding of vars/subgraphs

Commit Statistics

• 45 commits contributed to the release over the course of 49 calendar days.
• 56 days passed between releases.
• 43 commits were understood as conventional.
• 19 unique issues were worked on: #882, #884, #885, #886, #887, #892, #893, #896, #897, #898, #902, #906, #918, #919, #923, #924, #926, #932, #935

Commit Details

view details

• #882
  • Add Cumul flow_prop_fns for lattice_fold() and lattice_reduce() (63c435c)
  • Update dot/graphviz rendering of delta/cumul and defer_tick edges (9baf80c)
  • Record snapshots for surface_scheduling.rs (96e2f4c)
  • Add flow prop tests (95ed7a5)
  • Make propegate_flow_props fallible, cleanup flow_prop_fn definition. (fd89cb4)
  • Add cast operator (38346cf)
  • Update mermaid rendering of hydroflow graph to show flow properties (9ab7cf8)
  • Add monotonic_fn and morphism macros, update snapshots for flow props. (7714403)
  • Add source_iter_delta op for testing, basic flow props test, cleanups. (fd5cdb5)
• #884
  • Separate internal compiler operators in docs name/category/sort order (5ac9dde)
• #885
  • Revert "fix: add context.current_tick_start()" (6844366)
  • Add context.current_tick_start() (9918c78)
• #886
  • Revert "fix: add context.current_tick_start()" (6844366)
• #887
  • Add context.current_tick_start() without breaking the scheduler this time (3f45ec1)
• #892
  • Clean up degenerate subgraph error message for consistency (159a262)
• #893
  • Replace or_insert_with(Vec::new) with or_default() (7c7eea7)
• #896
  • Fix string join clippy::format_collect lint (58bf7ba)
• #897
  • Move hydroflow_expect_warnings into library, use in tests/surface_flow_props.rs (d03ffe7)
• #898
  • Add import!() expression (f013c3c)
• #902
  • Make lattice_fold and lattice_reduce consistent with fold/reduce (1ce5f01)
• #906
  • Add context.is_first_time_subgraph_is_scheduled to simplify replaying operators (d254e2d)
• #918
  • Refactor kvs example, cleanup, use demux_enum (b0c43c7)
  • Update chat example to use demux_enum (3c8dbd7)
• #919
  • Surface python snapshot tests (a6708bb)
• #923
  • Open mermaid/dot graph in browser (e7ea6d8)
• #924
  • Cleanup kvs example with lattice properties (db9f270)
  • Update snapshot tests for previous two commits (21140f0)
• #926
  • Cleanup cli quotes, loose TODO comment (5943758)
  • Add kvs with replication example (d38ec08)
  • Cleanup kvs example more (cb90ae1)
• #932
  • Fix example kvs/kvs_replicated tests, improve comments (3ac9b16)
  • Update snapshots for previous commit (cc15bab)
  • Simplify graph printing code, add delta/cumul green edges, allow hiding of vars/subgraphs (9ed0ce0)
• #935
  • Restore in-subgraph rendering of self-handoffs (2edf779)
  • Kvs_bench graphwrite (e519fb1)
• Uncategorized
  • Release hydroflow_lang v0.5.0, hydroflow_datalog_core v0.5.0, hydroflow_datalog v0.5.0, hydroflow_macro v0.5.0, lattices v0.5.0, hydroflow v0.5.0, hydro_cli v0.5.0, safety bump 4 crates (2e2d8b3)
  • Clippy warning on multiline string in hydro_cli, py_udf (a927dc6)
  • Update documentation and improve error messages for demux_enum operator (9646ca0)
  • demux_enum requires enum type name, add better error handling (1126266)
  • Initial technically working version of demux_enum with very bad error messages (b3d1148)
  • Implement partition operator (fc25433)
  • Fix demux error messages and add tests (5a7e1b1)
  • Implement flow_prop_fn for union() (1bdbf73)
  • Fix clippy::implied_bounds_in_impls from latest nightlies (e788989)
  • Make deadlock detector filter out spruious UDP ConnectionResets on Windows (e8b027b)
  • Ignore PermissionDenied on already-closed socket on Windows (d9a1350)

hydro_cli v0.5.0

Chore

• Add clippy::needless_pass_by_ref_mut false-positive workaround

Bug Fixes

• better error message when using wrong port types
• clippy warning on multiline string in hydro_cli, py_udf

Commit Statistics

• 4 commits contributed to the release over the course of 41 calendar days.
• 56 days passed between releases.
• 3 commits were understood as conventional.
• 1 unique issue was worked on: #931

Commit Details

view details

• #931
  • Better error message when using wrong port types (5d77694)
• Uncategorized
  • Release hydroflow_lang v0.5.0, hydroflow_datalog_core v0.5.0, hydroflow_datalog v0.5.0, hydroflow_macro v0.5.0, lattices v0.5.0, hydroflow v0.5.0, hydro_cli v0.5.0, safety bump 4 crates (2e2d8b3)
  • Clippy warning on multiline string in hydro_cli, py_udf (a927dc6)
  • Add clippy::needless_pass_by_ref_mut false-positive workaround (2b95a6d)

pusherator v0.0.3

Chore

• fix lint, format errors for latest nightly version (without updated pinned)
  For nightly version (d9c13cd45 2023-07-05)

New Features

• rename assert => assert_eq, add assert, change underlying implementation to work across ticks

Commit Statistics

• 2 commits contributed to the release over the course of 39 calendar days.
• 42 days passed between releases.
• 2 commits were understood as conventional.
• 2 unique issues were worked on: #822, #835

Commit Details

view details

• #822
  • Fix lint, format errors for latest nightly version (without updated pinned) (f60053f)
• #835
  • Rename assert => assert_eq, add assert, change underlying implementation to work across ticks (8f306e2)

lattices v0.4.0

Chore

• fix lint, format errors for latest nightly version (without updated pinned)
  For nightly version (d9c13cd45 2023-07-05)

Documentation

• Improve Atomize docs

New Features

• formalize Default::default() as returning bottom for lattice types
  Not a breaking change since changed names were introduced only since last release
• Implement SimpleKeyedRef for map types
• Add atomize trait, impls, tests

Refactor

• fix new clippy lints on latest nightly 1.73.0-nightly (db7ff98a7 2023-07-31)

• Change Atomize to require returning empty iff lattice is bottom
  Previously was the opposite, Atomize always had to return non-empty.

  Not breaking since Atomize has not yet been published.

New Features (BREAKING)

• Add bottom (+top) collapsing, implement IsBot/IsTop for all lattice types
  • WithBot(Some(BOTTOM)) and WithBot(None) are now considered to both be bottom, equal. Also, MapUnion({}) and MapUnion({key: BOTTOM}) are considered to both be bottom, equal.
  • WithTop(Some(TOP)) and WithTop(None) are now considered to both be top, equal.
  • check_lattice_bot/top now check that is_bot and is_top must be consistent among all equal elements

Refactor (BREAKING)

• Rename Seq -> VecUnion

Commit Statistics

• 9 commits contributed to the release over the course of 39 calendar days.
• 42 days passed between releases.
• 9 commits were understood as conventional.
• 8 unique issues were worked on: #822, #849, #854, #860, #865, #866, #867, #879

Commit Details

view details

• #822
  • Fix lint, format errors for latest nightly version (without updated pinned) (f60053f)
• #849
  • Rename Seq -> VecUnion (7b0485b)
• #854
  • Add atomize trait, impls, tests (8ec75c6)
• #860
  • Improve Atomize docs (a8b0d2d)
• #865
  • Add bottom (+top) collapsing, implement IsBot/IsTop for all lattice types (7b752f7)
• #866
  • Implement SimpleKeyedRef for map types (b240699)
• #867
  • Change Atomize to require returning empty iff lattice is bottom (262166e)
• #879
  • Formalize Default::default() as returning bottom for lattice types (7282457)
• Uncategorized
  • Fix new clippy lints on latest nightly 1.73.0-nightly (db7ff98a7 2023-07-31) (6a2ad6b)