Merge pull request #5182 from FederatedAI/dev-2.0.0-beta

Update fate-v2.0.0-beta

.gitmodules

@@ -1,12 +1,16 @@
 [submodule "eggroll"]
 	path = eggroll
 	url = https://github.com/WeBankFinTech/eggroll
-	branch = v2.5.0-alpha
+	branch = v3.0.0-beta
 [submodule "fate_client"]
 	path = fate_client
 	url = https://github.com/FederatedAI/FATE-Client
-	branch = v2.0.0-alpha
+	branch = v2.0.0-beta
 [submodule "fate_flow"]
 	path = fate_flow
 	url = https://github.com/FederatedAI/FATE-Flow
-	branch = v2.0.0-alpha
+	branch = v2.0.0-beta
+[submodule "fate_test"]
+	path = fate_test
+	url = https://github.com/FederatedAI/FATE-Test.git
+	branch = v2.0.0-beta

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/pycqa/isort
-    rev: 5.5.2
+    rev: 5.12.0
     hooks:
       - id: isort
   - repo: https://github.com/psf/black

Makefile

@@ -3,16 +3,20 @@ mkfile_dir := $(dir $(mkfile_path))
 .DEFAULT_GOAL:=help
 
 ##@ Dev
-.PHONY: install-rust_paillier
+.PHONY: install-rust
 install-rust_paillier: ## Install rust_paillier.
-	@echo "install rust_paillier"
-	@cd ${mkfile_dir} && maturin develop --release -m rust/tensor/rust_paillier/Cargo.toml
+	@echo "install fate_utils"
+	@cd ${mkfile_dir} && \
+		. venv/bin/activate.sh && \
+		maturin develop --release -m rust/fate_utils/Cargo.toml --target-dir build
 
 ##@ Build
-.PHONY: build-rust_paillier
-build-rust_paillier: ## Build rust_paillier.
-	@echo "build rust_paillier"
-	@cd ${mkfile_dir} && maturin build --release -m rust/tensor/rust_paillier/Cargo.toml --out dist --target-dir build
+.PHONY: build-rust
+build-rust: ## Build fate_utils.
+	@echo "build fate_utils"
+	@cd ${mkfile_dir} && \
+		. ${mkfile_dir}venv/bin/activate && \
+		maturin build --release -m rust/fate_utils/crates/fate_utils/Cargo.toml --out dist --target-dir build
 
 .PHONY: build-fate
 build-fate: ## Build fate

README.md

@@ -25,8 +25,18 @@ FATE is an open source project hosted by Linux Foundation. The [Technical Charte
 ### Version < 2.0
 Releases history can be found in [releases](https://github.com/FederatedAI/FATE/releases), deployment resources can be found on [wiki](https://github.com/FederatedAI/FATE/wiki/Download)
 
+### Version == 2.0.0-beta
+#### Standalone deployment
+- Deploying FATE on a single node via PyPI, pre-built docker images or installers. It is for simple testing purposes. Refer to this [guide](./deploy/standalone-deploy/).
+
+### Cluster deployment
+Deploying FATE to multiple nodes to achieve scalability, reliability and manageability.
+
+- [Cluster deployment by CLI](./deploy/cluster-deploy): Using CLI to deploy a FATE cluster.
+
 ### Quick Start
-- [Training Demo With Installing From Pypi](doc/2.0/quick_start.md)
+- [Training Demo With Installing FATE AND FATE-Flow From Pypi](doc/2.0/quick_start.md)
+- [Training Demo With Installing FATE Only From Pypi](doc/2.0/fate/ml)
 
 ## Related Repositories (Projects)
 - [KubeFATE](https://github.com/FederatedAI/KubeFATE): An operational tool for the FATE platform using cloud native technologies such as containers and Kubernetes.
@@ -38,6 +48,7 @@ Releases history can be found in [releases](https://github.com/FederatedAI/FATE/
 - [AnsibleFATE](https://github.com/FederatedAI/AnsibleFATE): A tool to optimize and automate the configuration and deployment operations via Ansible.
 - [FATE-Builder](https://github.com/FederatedAI/FATE-Builder): A tool to build package and docker image for FATE and KubeFATE.
 - [FATE-Client](https://github.com/FederatedAI/FATE-Client): A tool to enable fast federated modeling tasks for FATE.
+- [FATE-Test](https://github.com/FederatedAI/FATE-Test): An automated testing tool for FATE, including tests and benchmark comparisons.
 
 ## Governance 
 

README_zh.md

@@ -23,8 +23,16 @@ FATE于2019年2月首次对外开源，并成立
 ### 2.0以前的版本
 FATE 2.0以前的版本在[发布页](https://github.com/FederatedAI/FATE/releases), 下载资源汇总页在[wiki](https://github.com/FederatedAI/FATE/wiki/Download)
 
+### 2.0.0-beta 版本
+#### 单机版部署
+在单节点上部署FATE单机版，支持从 PyPI 直接安装，docker，主机安装包三种方式。
+- [单机版部署教程](./deploy/standalone-deploy)
+#### 集群
+- [原生集群安装](./deploy/cluster-deploy): Using CLI to deploy a FATE cluster.
+
 ### 快速开始
-- [从Pypi下载安装并启动训练任务示例](doc/2.0/quick_start.md)
+- [从 PyPI 下载安装 FATE 和 FATE-Flow 并启动训练任务示例](doc/2.0/quick_start.md)
+- [从 PyPI 下载安装 FATE，并启动训练任务示例](doc/2.0/fate/ml)
 
 ## 关联仓库
 - [KubeFATE](https://github.com/FederatedAI/KubeFATE)
@@ -37,6 +45,7 @@ FATE 2.0以前的版本在[发布页](https://github.com/FederatedAI/FATE/releas
 - [AnsibleFATE](https://github.com/FederatedAI/AnsibleFATE)
 - [FATE-Builder](https://github.com/FederatedAI/FATE-Builder)
 - [FATE-Client](https://github.com/FederatedAI/FATE-Client)
+- [FATE-Test](https://github.com/FederatedAI/FATE-Test)
 
 ## 社区治理  
 

RELEASE.md

@@ -1,3 +1,51 @@
+## Release 2.0.0-beta
+### Major Features and Improvements
+> Arch 2.0：Building Unified and Standardized API for Heterogeneous Computing Engines Interconnection
+* Framework: PSI-ECDH protocol support, single entry for histogram statistical computation 
+* Protocol: Support for ECDH, Secure Aggregation protocols
+* Tensor: abstracted PHETensor, smooth switch between various underlying PHE implementations through standard interface
+* DataFrame: New data block manager supports mixed-type columns & feature anonymization; added 30+ operator interfaces for statistics, including comparison, indexing, data binning, and transformation, etc.
+* Enhanced workflow: Support for Cross Validation workflow
+
+> Components 2.0: Building Standardized Algorithm Components for different Scheduling Engines
+* Input-Output: Further decoupling of FATE-Flow, providing standardized black-box calling processes 
+* Component Definition: Support for typing-based definition, automatic checking for component parameters, support for multiple types of data and model input and output, in addition to multiple inputs
+
+> ML:-2.0:  Major functionality migration from FATE-v1.x, decoupling call hierarchy 
+* Data preprocessing: Added DataFrame Transformer, Union and DataSplit migration completed
+* Feature Engineering: Migrated HeteroFederatedBinning, HeteroFeatureSelection, DataStatistics, Sampling, FeatureScale
+3. Federated Training: Migrated HeteroSecureBoost, HomoNN, vertical CoordinatedLogisticRegression, and CoordinatedLinearRegression
+4. Evaluation: Migrated Evaluation
+
+> OSX(Open Site Exchange) 1.0: Building Open Platform for Cross-Site Communication Interconnection 
+* Improved HTTP/1.X protocol support, support for GRPC-to-HTTP transmission 
+* Support for TLS secure transmission protocol 
+* Added routing table configuration interface 
+* Added routing table connectivity automatic check 
+* Improved transmission function in cluster mode 
+* Enhanced flow control in cluster mode 
+* Support for simple interface authentication
+
+> FATE Flow 2.0: Building Open and Standardized Scheduling Platform for Scheduling Interconnection
+* Migrated functions: data upload/download, process scheduling, component output data/model/metric management, multi-storage adaptation for models, authentication, authorization, feature anonymization, multi-computing/storage/communication engine adaptation, and system high availability
+* Optimized process scheduling, with scheduling separated and customizable, and added priority scheduling
+* Optimized algorithm component scheduling, dividing execution steps into preprocessing, running, and post-processing
+* Optimized multi-version algorithm component registration, supporting registration for mode of components
+* Optimized client authentication logic, supporting permission management for multiple clients
+* Optimized RESTful interface, making parameter fields and types, return fields, and status codes clearer
+* Decoupling the system layer from the algorithm layer, with system configuration moved from the FATE repository to the Flow repository
+* Published FATE Flow package to PyPI and added service-level CLI for service management
+
+> Fate-Client 2.0: Building Scalable Federated DSL for Application Layer Interconnection And Providing Tools For Fast Federated Modeling.
+* Migrated Flow CLI and Flow SDK
+* Updated federated DSL IR: enhance IR, add DataWarehouse and ModelWarehouse to load data and model from other sources
+* Update component definitions to support Fate-v2.0.0-beta
+* Flow CLI and PipeLine share configuration
+
+> Fate-Test: FATE Automated Testing Tool
+* Migrated automated testing for functionality, performance, and correctness
+
+
 ## Release 2.0.0-alpha
 ### Feature Highlights
 > Arch 2.0：Building Unified and Standardized API for Heterogeneous Computing Engines Interconnection