From 537ca895861f4afd0a03f5fcf5ec850c0a0a9532 Mon Sep 17 00:00:00 2001 From: Joe Hellerstein Date: Thu, 19 Dec 2024 21:28:47 -0800 Subject: [PATCH 1/2] chore: hydroflow+ => hydro, hydroflow => DFIR --- README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 90b16524653..7b5ed81dd76 100644 --- a/README.md +++ b/README.md @@ -18,16 +18,16 @@ We often describe Hydro via a metaphor: *LLVM for the cloud*. Like LLVM, Hydro i ## The Language (and the Low-Level IR) -Hydro provides a high-level language for the majority of developers called [Hydroflow+](https://hydro.run/docs/hydroflow_plus). Hydroflow+ allows you to program an entire fleet of processes from a single program, and then launch your fleet locally or in the cloud via [Hydro Deploy](https://hydro.run/docs/deploy). Get started with Hydroflow+ via the Hydroflow+ [documentation](https://hydro.run/docs/hydroflow_plus) and [examples](https://github.com/hydro-project/hydroflow/tree/main/hydroflow_plus_test/examples). +Hydro provides a [high-level language](https://hydro.run/docs/hydro) that allows you to program an entire fleet of processes from a single program, and then launch your fleet locally or in the cloud via [Hydro Deploy](https://hydro.run/docs/deploy). Get started with Hydro via the language [documentation](https://hydro.run/docs/hydro) and [examples](https://github.com/hydro-project/hydroflow/tree/main/hydroflow_plus_test/examples). -> Internally, the Hydro stack compiles Hydroflow+ programs into a low-level Internal Representation (IR) language called [Hydroflow](https://hydro.run/docs/hydroflow); each process corresponds to a separate Hydroflow program. In rare cases you may want to compose one or more processes in Hydroflow by hand; see the Hydroflow [documentation](https://hydro.run/docs/hydroflow) or [examples](https://github.com/hydro-project/hydroflow/tree/main/hydroflow/examples) for details. +> Internally, the Hydro stack compiles Hydro programs into a low-level Dataflow Internal Representation (IR) language called [DFIR](https://hydro.run/docs/dfir); each process corresponds to a separate DFIR program. In rare cases you may want to compose one or more processes in DFIR by hand; see the DFIR [documentation](https://hydro.run/docs/dfir) or [examples](https://github.com/hydro-project/hydroflow/tree/main/hydroflow/examples) for details. ## Development Setup -See the [quickstart section of the Hydroflow+ book](https://hydro.run/docs/hydro/quickstart/) for instructions on installing Rust and getting started with the Hydroflow+ template. +See the [quickstart section of the Hydro book](https://hydro.run/docs/hydro/quickstart/) for instructions on installing Rust and getting started with the Hydro template. # A New Approach to Distributed Programming -There have been many frameworks and platforms for distributed programming over the years, with significant tradeoffs. These include: +There have been many frameworks and platforms for distributed programming over the years, with significant tradeoffs: **Higher level frameworks** have been designed to serve specialized distributed use cases. These including *Client-Server (Monolith)* frameworks (e.g. Ruby on Rails + DBMS), parallel *Bulk Dataflow* frameworks (e.g. Spark, Flink, etc.), and step-wise *Workflows / Pipelines / Serverless / μservice Orchestration* frameworks (e.g. Kafka, Airflow). All of these frameworks offer limited expressibility and are inefficient outside their sweet spot. Each one ties developers' hands in different ways. @@ -37,20 +37,20 @@ There have been many frameworks and platforms for distributed programming over t ## Towards a more comprehensive approach What's wanted, we believe, is a proper language stack addressing distributed concerns: -- **Broad Expressivity**: The stack should support a spectrum of performance regimes from lightweight, low-latency async event handling to high-throughput dataflow. It should also support a full range of architectural configurations, from SIMD to more heterogeneous architectures. +- **Broad Expressivity**: The stack should support a spectrum of performance regimes from lightweight, low-latency async event handling to high-throughput dataflow. It should also support a full range of architectural configurations, from [SPMD](https://en.wikipedia.org/wiki/Single_program,_multiple_data) to more heterogeneous architectures. - **Familiarity**: The distributed aspects of the language should be integrated into a familiar mature programming language and environment, including libraries, IDEs and other tooling. A mature compiler should optimize local code to be fast and lean. (Hydro embraces Rust and LLVM for these attributes.) - **Performance control**: The ability to program a fleet of machines "globally" should not prevent software engineers from optimizing the code that executes locally at each node. - **Distributed Typechecking**: The type system of the language should enforce distributed correctness in the compiler, in ways that are visible in an IDE at time of authoring. For example, the types of data items should include their abstract locations, so that two items materialized at different nodes cannot be referenced together without an intervening construct for (async) communication. - **Distributed Optimizations**: The compiler should be able to correctly optimize (transform) programs for distributed concerns: removing bottlenecks by flexibly assigning compute and data to different processes or clusters, while preserving program semantics. -- **Modularity**: The standard modularity of traditional programming—e.g. function calling abstractions—should work for distriuted logic. For example, it should be possible to wrap a common cross-node construct like "heartbeats and timeouts", and invoke it as simply as one invokes a sequential function. -- **Native Testing Tools** for correctness. In today's standard practice, formal specification languages for testing (e.g. TLA+) are separate from languages of implementation. We believe it should be possible to perform many kinds of formal testing (e.g. model checking) on the same code that is used in deployment. +- **Modularity**: The standard modularity of traditional programming—e.g. function calling abstractions—should work for distributed logic. For example, it should be possible to wrap a common cross-node construct like "heartbeats and timeouts", and invoke it as simply as one invokes a sequential function. +- **Native Testing Tools** for correctness. In today's standard practice, formal specification languages for testing (e.g. [TLA+](https://en.wikipedia.org/wiki/TLA%2B)) are separate from languages of implementation. We believe it should be possible to perform many kinds of formal testing (e.g. model checking) on the same code that is used in deployment. -[Hydro](https://hydro.run) is a Rust library for distributed programming that is designed to address these goals. +[Hydro](https://hydro.run) is a Rust library for distributed programming that is designed to address these goals. # Learning More The Hydro project's main website is at [https://hydro.run](https://hydro.run). -- **Docs**: There are docs for the [high-level Hydroflow+ language](https://hydro.run/docs/hydroflow_plus/) and the [low-level Hydroflow IR](https://hydro.run/docs/hydroflow), as well as the [HydroDeploy](https://hydro.run/docs/deploy) framework for launching Hydroflow+ programs. +- **Docs**: There are docs for the [high-level Hydro language](https://hydro.run/docs/hydro/) and the low-level dataflow IR, [DFIR](https://hydro.run/docs/dfir), as well as the [HydroDeploy](https://hydro.run/docs/deploy) framework for launching Hydroflow+ programs. - **Research Papers**: Our [research publications](https://hydro.run/research) are available on the project website. Some notable selections: - The original Hydro vision paper from CIDR 2021: [New Directions in Cloud Programming](https://hydro.run/papers/new-directions.pdf) From 6948250c1bb7214fe61b779e50da41d5abd8a2f2 Mon Sep 17 00:00:00 2001 From: Mingwei Samuel Date: Sun, 22 Dec 2024 09:17:38 -0800 Subject: [PATCH 2/2] add redirects for old paths --- docs/docs/hydroflow/index.mdx | 3 +++ docs/docs/hydroflow_plus/index.mdx | 3 +++ 2 files changed, 6 insertions(+) create mode 100644 docs/docs/hydroflow/index.mdx create mode 100644 docs/docs/hydroflow_plus/index.mdx diff --git a/docs/docs/hydroflow/index.mdx b/docs/docs/hydroflow/index.mdx new file mode 100644 index 00000000000..a45bd07fe6e --- /dev/null +++ b/docs/docs/hydroflow/index.mdx @@ -0,0 +1,3 @@ +import {Redirect} from '@docusaurus/router'; + + diff --git a/docs/docs/hydroflow_plus/index.mdx b/docs/docs/hydroflow_plus/index.mdx new file mode 100644 index 00000000000..ff293e52387 --- /dev/null +++ b/docs/docs/hydroflow_plus/index.mdx @@ -0,0 +1,3 @@ +import {Redirect} from '@docusaurus/router'; + +