From 705b2d470116f8191310b104f90f693f6f808827 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=81=93=E8=BE=95?= Date: Mon, 25 Dec 2023 10:35:53 +0800 Subject: [PATCH] init readme --- README.md | 372 ++++++++---------------------------------------------- 1 file changed, 55 insertions(+), 317 deletions(-) diff --git a/README.md b/README.md index 0abadd033..0c4606aaf 100644 --- a/README.md +++ b/README.md @@ -1,339 +1,77 @@ -

- federatedscope-logo -

+# FedKSeed +**The codes will be released soon after internal review and collation.** -![](https://img.shields.io/badge/language-python-blue.svg) -![](https://img.shields.io/badge/license-Apache-000000.svg) -[![Website](https://img.shields.io/badge/website-FederatedScope-0000FF)](https://federatedscope.io/) -[![Playground](https://shields.io/badge/JupyterLab-Enjoy%20Your%20FL%20Journey!-F37626?logo=jupyter)](https://try.federatedscope.io/) -[![Contributing](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://federatedscope.io/docs/contributor/) +This repository contains the official implementation for the work “**Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes**”. See more details in our [paper](https://arxiv.org/abs/2312.06353). -FederatedScope is a comprehensive federated learning platform that provides convenient usage and flexible customization for various federated learning tasks in both academia and industry. Based on an event-driven architecture, FederatedScope integrates rich collections of functionalities to satisfy the burgeoning demands from federated learning, and aims to build up an easy-to-use platform for promoting learning safely and effectively. +> Pre-trained large language models (LLMs) require fine-tuning to improve their responsiveness to natural language instructions. Federated learning (FL) offers a way to perform fine-tuning using the abundant data on end devices without compromising data privacy. Most existing federated fine-tuning methods for LLMs rely on parameter-efficient fine-tuning techniques, which may not reach the performance heights possible with full-parameter tuning. However, the communication overhead associated with full-parameter tuning is prohibitively high for both servers and clients. This work introduces FedKSeed, a novel approach that employs zeroth-order optimization (ZOO) with a set of random seeds. It enables federated full-parameter tuning of billion-sized LLMs directly on devices. Our method significantly reduces transmission requirements between the server and clients to just a few scalar gradients and random seeds, amounting to only a few thousand bytes. Building on this, we develop a strategy to assess the significance of ZOO perturbations for FL, allowing for probability-differentiated seed sampling. This prioritizes perturbations that have a greater impact on model accuracy. Experiments across six scenarios with different LLMs, datasets and data partitions demonstrate that our approach outperforms existing federated LLM fine-tuning methods in terms of both communication efficiency and new task generalization. -A detailed tutorial is provided on our website: [federatedscope.io](https://federatedscope.io/) +The purpose of this implementation is to evaluate the accuracy of our approach. To reduce the time consumed by the experiments, we did not have all clients compute the global model in each round. Instead, we performed such a computation only once, and allowed all clients to copy this model directly using `copy.deepcopy()` (please refer to line 145 of `main.py`). -You can try FederatedScope via [FederatedScope Playground](https://try.federatedscope.io/) or [Google Colab](https://colab.research.google.com/github/alibaba/FederatedScope). - -| [Code Structure](#code-structure) | [Quick Start](#quick-start) | [Advanced](#advanced) | [Documentation](#documentation) | [Publications](#publications) | [Contributing](#contributing) | - -## News -- ![new](https://img.alicdn.com/imgextra/i4/O1CN01kUiDtl1HVxN6G56vN_!!6000000000764-2-tps-43-19.png) [05-17-2023] Our paper [FS-REAL](https://arxiv.org/abs/2303.13363) has been accepted by KDD'2023! -- ![new](https://img.alicdn.com/imgextra/i4/O1CN01kUiDtl1HVxN6G56vN_!!6000000000764-2-tps-43-19.png) [05-17-2023] Our benchmark paper for FL backdoor attacks [Backdoor Attacks Bench](https://arxiv.org/abs/2302.01677) has been accepted by KDD'2023! -- ![new](https://img.alicdn.com/imgextra/i4/O1CN01kUiDtl1HVxN6G56vN_!!6000000000764-2-tps-43-19.png) [05-17-2023] Our paper [Communication Efficient and Differentially Private Logistic Regression under the Distributed Setting]() has been accepted by KDD'2023! -- ![new](https://img.alicdn.com/imgextra/i4/O1CN01kUiDtl1HVxN6G56vN_!!6000000000764-2-tps-43-19.png) [04-25-2023] Our paper [pFedGate](https://arxiv.org/abs/2305.02776) has been accepted by ICML'2023! -- ![new](https://img.alicdn.com/imgextra/i4/O1CN01kUiDtl1HVxN6G56vN_!!6000000000764-2-tps-43-19.png) [04-25-2023] Our benchmark paper for FedHPO [FedHPO-Bench](https://arxiv.org/abs/2206.03966) has been accepted by ICML'2023! -- ![new](https://img.alicdn.com/imgextra/i4/O1CN01kUiDtl1HVxN6G56vN_!!6000000000764-2-tps-43-19.png) [04-03-2023] We release FederatedScope v0.3.0! -- [02-10-2022] Our [paper](https://arxiv.org/pdf/2204.05011.pdf) elaborating on FederatedScope is accepted by VLDB'23! -- [10-05-2022] Our benchmark paper for personalized FL, [pFL-Bench](https://arxiv.org/abs/2206.03655) has been accepted by NeurIPS'22, Dataset and Benchmark Track! -- [08-18-2022] Our KDD 2022 [paper](https://arxiv.org/abs/2204.05562) on federated graph learning receives the KDD Best Paper Award for ADS track! -- [07-30-2022] We release FederatedScope v0.2.0! -- [06-17-2022] We release **pFL-Bench**, a comprehensive benchmark for personalized Federated Learning (pFL), containing 10+ datasets and 20+ baselines. [[code](https://github.com/alibaba/FederatedScope/tree/master/benchmark/pFL-Bench), [pdf](https://arxiv.org/abs/2206.03655)] -- [06-17-2022] We release **FedHPO-Bench**, a benchmark suite for studying federated hyperparameter optimization. [[code](https://github.com/alibaba/FederatedScope/tree/master/benchmark/FedHPOBench), [pdf](https://arxiv.org/abs/2206.03966)] -- [06-17-2022] We release **B-FHTL**, a benchmark suit for studying federated hetero-task learning. [[code](https://github.com/alibaba/FederatedScope/tree/master/benchmark/B-FHTL), [pdf](https://arxiv.org/abs/2206.03436)] -- [06-13-2022] Our project was receiving an attack, which has been resolved. [More details](https://github.com/alibaba/FederatedScope/blob/master/doc/news/06-13-2022_Declaration_of_Emergency.txt). -- [05-25-2022] Our paper [FederatedScope-GNN](https://arxiv.org/abs/2204.05562) has been accepted by KDD'2022! -- [05-06-2022] We release FederatedScope v0.1.0! - -## Code Structure +## Project Structure +```Markdown +. +├── optimizers +│ ├── mezo_bias_optimizer.py // implementation of FedKSeed-Pro +│ └── mezo_optimizer.py // implementation of FedKSeed +├── utils_data +│ ├── default_tokens.py // definitions of some special tokens +│ ├── llm_dataset.py // utilities to load Dolly-15K +│ ├── load_data.py // entrance to get dataloaders +│ ├── natural_instruction_loader.py // utilities to load Natural Instructions +│ └── partition_data.py // utilities to partition datasets in Dirichlet distribution +├── client.py +├── evaluations.py +├── main.py +└── server.py ``` -FederatedScope -├── federatedscope -│   ├── core -│   | ├── workers # Behaviors of participants (i.e., server and clients) -│   | ├── trainers # Details of local training -│   | ├── aggregators # Details of federated aggregation -│   | ├── configs # Customizable configurations -│   | ├── monitors # The monitor module for logging and demonstrating -│   | ├── communication.py # Implementation of communication among participants -│   | ├── fed_runner.py # The runner for building and running an FL course -│   | ├── ... .. -│   ├── cv # Federated learning in CV -│   ├── nlp # Federated learning in NLP -│   ├── gfl # Graph federated learning -│   ├── autotune # Auto-tunning for federated learning -│   ├── vertical_fl # Vertical federated learning -│   ├── contrib -│   ├── main.py -│   ├── ... ... -├── scripts # Scripts for reproducing existing algorithms -├── benchmark # We release several benchmarks for convenient and fair comparisons -├── doc # For automatic documentation -├── environment # Installation requirements and provided docker files -├── materials # Materials of related topics (e.g., paper lists) -│   ├── notebook -│   ├── paper_list -│   ├── tutorial -│   ├── ... ... -├── tests # Unittest modules for continuous integration -├── LICENSE -└── setup.py -``` - -## Quick Start -We provide an end-to-end example for users to start running a standard FL course with FederatedScope. +## Requirements +Please see `requirements.txt`. -### Step 1. Installation +## Data Preparation +1. Natural Instructions +To run experiments on [Natural Instructions](https://github.com/allenai/natural-instructions), you need to unzip the downloaded dataset in directory `./data`. -First of all, users need to clone the source code and install the required packages (we suggest python version >= 3.9). You can choose between the following two installation methods (via docker or conda) to install FederatedScope. +2. Dolly-15K +To run experiments on [Dolly-15K](https://github.com/databrickslabs/dolly), you need to download the corresponding dataset in directory `./data`, with its name as `databricks-dolly-15k.jsonl`. -```bash -git clone https://github.com/alibaba/FederatedScope.git -cd FederatedScope +## Running Examples +We provide some example scripts to conduct the experiments. +The arguments can be adjusted according to the `help` information in their definitions. +1. FedKSeed on Natural Instructions +```Shell +# On Natural Instructions, the number of clients `num_clients` does not require manual setting. +# It will be automatically adjusted to the number of tasks in `splits/default/train_tasks.txt`. +python main.py --rounds 40 --model datajuicer/LLaMA-1B-dj-refine-150B --use_prompts --dataset instruct --lr 0.0000003 -K 1024 -m 0.05 --log ``` -#### Use Docker - -You can build docker image and run with docker env (cuda 11 and torch 1.10): +2. FedKSeed on Dolly-15K with $\alpha=0.5$ +```Shell +python main.py --rounds 60 --model datajuicer/LLaMA-1B-dj-refine-150B --use_prompts --dataset dolly --iid dir0.5 --num_clients 200 --lr 0.0000003 -K 1024 -m 0.05 --log ``` -docker build -f environment/docker_files/federatedscope-torch1.10.Dockerfile -t alibaba/federatedscope:base-env-torch1.10 . -docker run --gpus device=all --rm -it --name "fedscope" -w $(pwd) alibaba/federatedscope:base-env-torch1.10 /bin/bash -``` -If you need to run with down-stream tasks such as graph FL, change the requirement/docker file name into another one when executing the above commands: -``` -# environment/requirements-torch1.10.txt -> -environment/requirements-torch1.10-application.txt - -# environment/docker_files/federatedscope-torch1.10.Dockerfile -> -environment/docker_files/federatedscope-torch1.10-application.Dockerfile -``` -Note: You can choose to use cuda 10 and torch 1.8 via changing `torch1.10` to `torch1.8`. -The docker images are based on the nvidia-docker. Please pre-install the NVIDIA drivers and `nvidia-docker2` in the host machine. See more details [here](https://github.com/alibaba/FederatedScope/tree/master/environment/docker_files). - -#### Use Conda -We recommend using a new virtual environment to install FederatedScope: -```bash -conda create -n fs python=3.9 -conda activate fs +3. FedKSeed-Pro on Natural Instructions +```Shell +# On Natural Instructions, the number of clients `num_clients` does not require manual setting. +# It will be automatically adjusted to the number of tasks in `splits/default/train_tasks.txt`. +python main.py --rounds 40 --bias_sampling --model datajuicer/LLaMA-1B-dj-refine-150B --use_prompts --dataset instruct --lr 0.0000003 -K 1024 -m 0.05 --log ``` -If your backend is torch, please install torch in advance ([torch-get-started](https://pytorch.org/get-started/locally/)). For example, if your cuda version is 11.3 please execute the following command: - -```bash -conda install -y pytorch=1.10.1 torchvision=0.11.2 torchaudio=0.10.1 torchtext=0.11.1 cudatoolkit=11.3 -c pytorch -c conda-forge -``` - -For users with Apple M1 chips: -```bash -conda install pytorch torchvision torchaudio -c pytorch -# Downgrade torchvision to avoid segmentation fault -python -m pip install torchvision==0.11.3 -``` - -Finally, after the backend is installed, you can install FederatedScope from `source`: - -##### From source - -```bash -# Editable mode -pip install -e . - -# Or (developers for dev mode) -pip install -e .[dev] -pre-commit install +4. FedKSeed-Pro on Dolly-15K with $\alpha=0.5$ +```Shell +python main.py --rounds 60 --bias_sampling --model datajuicer/LLaMA-1B-dj-refine-150B --use_prompts --dataset dolly --iid dir0.5 --num_clients 200 --lr 0.0000003 -K 1024 -m 0.05 --log ``` -Now, you have successfully installed the minimal version of FederatedScope. (**Optinal**) For application version including graph, nlp and speech, run: - -```bash -bash environment/extra_dependencies_torch1.10-application.sh -``` - -### Step 2. Prepare datasets - -To run an FL task, users should prepare a dataset. -The DataZoo provided in FederatedScope can help to automatically download and preprocess widely-used public datasets for various FL applications, including CV, NLP, graph learning, recommendation, etc. Users can directly specify `cfg.data.type = DATASET_NAME`in the configuration. For example, - -```bash -cfg.data.type = 'femnist' -``` - -To use customized datasets, you need to prepare the datasets following a certain format and register it. Please refer to [Customized Datasets](https://federatedscope.io/docs/own-case/#data) for more details. - -### Step 3. Prepare models - -Then, users should specify the model architecture that will be trained in the FL course. -FederatedScope provides a ModelZoo that contains the implementation of widely adopted model architectures for various FL applications. Users can set up `cfg.model.type = MODEL_NAME` to apply a specific model architecture in FL tasks. For example, - -```yaml -cfg.model.type = 'convnet2' -``` - -FederatedScope allows users to use customized models via registering. Please refer to [Customized Models](https://federatedscope.io/docs/own-case/#model) for more details about how to customize a model architecture. - -### Step 4. Start running an FL task - -Note that FederatedScope provides a unified interface for both standalone mode and distributed mode, and allows users to change via configuring. - -#### Standalone mode - -The standalone mode in FederatedScope means to simulate multiple participants (servers and clients) in a single device, while participants' data are isolated from each other and their models might be shared via message passing. - -Here we demonstrate how to run a standard FL task with FederatedScope, with setting `cfg.data.type = 'FEMNIST'`and `cfg.model.type = 'ConvNet2'` to run vanilla FedAvg for an image classification task. Users can customize training configurations, such as `cfg.federated.total_round_num`, `cfg.dataloader.batch_size`, and `cfg.train.optimizer.lr`, in the configuration (a .yaml file), and run a standard FL task as: - -```bash -# Run with default configurations -python federatedscope/main.py --cfg scripts/example_configs/femnist.yaml -# Or with custom configurations -python federatedscope/main.py --cfg scripts/example_configs/femnist.yaml federate.total_round_num 50 dataloader.batch_size 128 -``` - -Then you can observe some monitored metrics during the training process as: - -``` -INFO: Server has been set up ... -INFO: Model meta-info: . -... ... -INFO: Client has been set up ... -INFO: Model meta-info: . -... ... -INFO: {'Role': 'Client #5', 'Round': 0, 'Results_raw': {'train_loss': 207.6341676712036, 'train_acc': 0.02, 'train_total': 50, 'train_loss_regular': 0.0, 'train_avg_loss': 4.152683353424072}} -INFO: {'Role': 'Client #1', 'Round': 0, 'Results_raw': {'train_loss': 209.0940284729004, 'train_acc': 0.02, 'train_total': 50, 'train_loss_regular': 0.0, 'train_avg_loss': 4.1818805694580075}} -INFO: {'Role': 'Client #8', 'Round': 0, 'Results_raw': {'train_loss': 202.24929332733154, 'train_acc': 0.04, 'train_total': 50, 'train_loss_regular': 0.0, 'train_avg_loss': 4.0449858665466305}} -INFO: {'Role': 'Client #6', 'Round': 0, 'Results_raw': {'train_loss': 209.43883895874023, 'train_acc': 0.06, 'train_total': 50, 'train_loss_regular': 0.0, 'train_avg_loss': 4.1887767791748045}} -INFO: {'Role': 'Client #9', 'Round': 0, 'Results_raw': {'train_loss': 208.83140087127686, 'train_acc': 0.0, 'train_total': 50, 'train_loss_regular': 0.0, 'train_avg_loss': 4.1766280174255375}} -INFO: ----------- Starting a new training round (Round #1) ------------- -... ... -INFO: Server: Training is finished! Starting evaluation. -INFO: Client #1: (Evaluation (test set) at Round #20) test_loss is 163.029045 -... ... -INFO: Server: Final evaluation is finished! Starting merging results. -... ... -``` - -#### Distributed mode - -The distributed mode in FederatedScope denotes running multiple procedures to build up an FL course, where each procedure plays as a participant (server or client) that instantiates its model and loads its data. The communication between participants is already provided by the communication module of FederatedScope. - -To run with distributed mode, you only need to: - -- Prepare isolated data file and set up `cfg.data.file_path = PATH/TO/DATA` for each participant; -- Change `cfg.federate.model = 'distributed'`, and specify the role of each participant by `cfg.distributed.role = 'server'/'client'`. -- Set up a valid address by `cfg.distribute.server_host/client_host = x.x.x.x` and `cfg.distribute.server_port/client_port = xxxx`. (Note that for a server, you need to set up `server_host` and `server_port` for listening messages, while for a client, you need to set up `client_host` and `client_port` for listening as well as `server_host` and `server_port` for joining in an FL course) - -We prepare a synthetic example for running with distributed mode: - -```bash -# For server -python federatedscope/main.py --cfg scripts/distributed_scripts/distributed_configs/distributed_server.yaml data.file_path 'PATH/TO/DATA' distribute.server_host x.x.x.x distribute.server_port xxxx - -# For clients -python federatedscope/main.py --cfg scripts/distributed_scripts/distributed_configs/distributed_client_1.yaml data.file_path 'PATH/TO/DATA' distribute.server_host x.x.x.x distribute.server_port xxxx distribute.client_host x.x.x.x distribute.client_port xxxx -python federatedscope/main.py --cfg scripts/distributed_scripts/distributed_configs/distributed_client_2.yaml data.file_path 'PATH/TO/DATA' distribute.server_host x.x.x.x distribute.server_port xxxx distribute.client_host x.x.x.x distribute.client_port xxxx -python federatedscope/main.py --cfg scripts/distributed_scripts/distributed_configs/distributed_client_3.yaml data.file_path 'PATH/TO/DATA' distribute.server_host x.x.x.x distribute.server_port xxxx distribute.client_host x.x.x.x distribute.client_port xxxx -``` - -An executable example with generated toy data can be run with (a script can be found in `scripts/run_distributed_lr.sh`): -```bash -# Generate the toy data -python scripts/distributed_scripts/gen_data.py - -# Firstly start the server that is waiting for clients to join in -python federatedscope/main.py --cfg scripts/distributed_scripts/distributed_configs/distributed_server.yaml data.file_path toy_data/server_data distribute.server_host 127.0.0.1 distribute.server_port 50051 - -# Start the client #1 (with another process) -python federatedscope/main.py --cfg scripts/distributed_scripts/distributed_configs/distributed_client_1.yaml data.file_path toy_data/client_1_data distribute.server_host 127.0.0.1 distribute.server_port 50051 distribute.client_host 127.0.0.1 distribute.client_port 50052 -# Start the client #2 (with another process) -python federatedscope/main.py --cfg scripts/distributed_scripts/distributed_configs/distributed_client_2.yaml data.file_path toy_data/client_2_data distribute.server_host 127.0.0.1 distribute.server_port 50051 distribute.client_host 127.0.0.1 distribute.client_port 50053 -# Start the client #3 (with another process) -python federatedscope/main.py --cfg scripts/distributed_scripts/distributed_configs/distributed_client_3.yaml data.file_path toy_data/client_3_data distribute.server_host 127.0.0.1 distribute.server_port 50051 distribute.client_host 127.0.0.1 distribute.client_port 50054 -``` - -And you can observe the results as (the IP addresses are anonymized with 'x.x.x.x'): - -``` -INFO: Server: Listen to x.x.x.x:xxxx... -INFO: Server has been set up ... -Model meta-info: . -... ... -INFO: Client: Listen to x.x.x.x:xxxx... -INFO: Client (address x.x.x.x:xxxx) has been set up ... -Client (address x.x.x.x:xxxx) is assigned with #1. -INFO: Model meta-info: . -... ... -{'Role': 'Client #2', 'Round': 0, 'Results_raw': {'train_avg_loss': 5.215108394622803, 'train_loss': 333.7669372558594, 'train_total': 64}} -{'Role': 'Client #1', 'Round': 0, 'Results_raw': {'train_total': 64, 'train_loss': 290.9668884277344, 'train_avg_loss': 4.54635763168335}} ------------ Starting a new training round (Round #1) ------------- -... ... -INFO: Server: Training is finished! Starting evaluation. -INFO: Client #1: (Evaluation (test set) at Round #20) test_loss is 30.387419 -... ... -INFO: Server: Final evaluation is finished! Starting merging results. -... ... -``` - - -## Advanced - -As a comprehensive FL platform, FederatedScope provides the fundamental implementation to support requirements of various FL applications and frontier studies, towards both convenient usage and flexible extension, including: - -- **Personalized Federated Learning**: Client-specific model architectures and training configurations are applied to handle the non-IID issues caused by the diverse data distributions and heterogeneous system resources. -- **Federated Hyperparameter Optimization**: When hyperparameter optimization (HPO) comes to Federated Learning, each attempt is extremely costly due to multiple rounds of communication across participants. It is worth noting that HPO under the FL is unique and more techniques should be promoted such as low-fidelity HPO. -- **Privacy Attacker**: The privacy attack algorithms are important and convenient to verify the privacy protection strength of the design FL systems and algorithms, which is growing along with Federated Learning. -- **Graph Federated Learning**: Working on the ubiquitous graph data, Graph Federated Learning aims to exploit isolated sub-graph data to learn a global model, and has attracted increasing popularity. -- **Recommendation**: As a number of laws and regulations go into effect all over the world, more and more people are aware of the importance of privacy protection, which urges the recommender system to learn from user data in a privacy-preserving manner. -- **Differential Privacy**: Different from the encryption algorithms that require a large amount of computation resources, differential privacy is an economical yet flexible technique to protect privacy, which has achieved great success in database and is ever-growing in federated learning. -- ... - -More supports are coming soon! We have prepared a [tutorial](https://federatedscope.io/) to provide more details about how to utilize FederatedScope to enjoy your journey of Federated Learning! - -Materials of related topics are constantly being updated, please refer to [FL-Recommendation](https://github.com/alibaba/FederatedScope/tree/master/materials/paper_list/FL-Recommendation), [Federated-HPO](https://github.com/alibaba/FederatedScope/tree/master/materials/paper_list/Federated_HPO), [Personalized FL](https://github.com/alibaba/FederatedScope/tree/master/materials/paper_list/Personalized_FL), [Federated Graph Learning](https://github.com/alibaba/FederatedScope/tree/master/materials/paper_list/Federated_Graph_Learning), [FL-NLP](https://github.com/alibaba/FederatedScope/tree/master/materials/paper_list/FL-NLP), [FL-Attacker](https://github.com/alibaba/FederatedScope/tree/master/materials/paper_list/FL-Attacker), [FL-Incentive-Mechanism](https://github.com/alibaba/FederatedScope/tree/master/materials/paper_list/FL-Incentive), [FL-Fairness](https://github.com/alibaba/FederatedScope/tree/master/materials/paper_list/FL-Fiarness) and so on. - -## Documentation - -The classes and methods of FederatedScope have been well documented so that users can generate the API references by: - -```shell -cd doc -pip install -r requirements.txt -make html -``` -NOTE: -* The `doc/requirements.txt` is only for documentation of API by Sphinx, which can be automatically generated by Github actions `.github/workflows/sphinx.yml`. (Trigger by pull request if `DOC` in the title.) -* Download via Artifacts in Github actions. - -We put the API references on our [website](https://federatedscope.io/refs/index). - -Besides, we provide documents for [executable scripts](https://github.com/alibaba/FederatedScope/tree/master/scripts) and [customizable configurations](https://github.com/alibaba/FederatedScope/tree/master/federatedscope/core/configs). - ## License - -FederatedScope is released under Apache License 2.0. - -## Publications -If you find FederatedScope useful for your research or development, please cite the following paper: -``` -@article{federatedscope, - title = {FederatedScope: A Flexible Federated Learning Platform for Heterogeneity}, - author = {Xie, Yuexiang and Wang, Zhen and Gao, Dawei and Chen, Daoyuan and Yao, Liuyi and Kuang, Weirui and Li, Yaliang and Ding, Bolin and Zhou, Jingren}, - journal={Proceedings of the VLDB Endowment}, - volume={16}, - number={5}, - pages={1059--1072}, - year={2023} +This project adopts the Apache-2.0 License. +If the implementations and/or our paper were useful to you, please consider citing this [work](https://arxiv.org/abs/2312.06353): +```latex +@article{qin2023federated, + title={Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes}, + author={Zhen Qin and Daoyuan Chen and Bingchen Qian and Bolin Ding and Yaliang Li and Shuiguang Deng}, + journal={arXiv preprint arXiv:2312.06353} + year={2023} } ``` -More publications can be found in the [Publications](https://federatedscope.io/pub/). - -## Contributing - -We **greatly appreciate** any contribution to FederatedScope! We provide a developer version of FederatedScope with additional pre-commit hooks to perform commit checks compared to the official version: - -```bash -# Install the developer version -pip install -e .[dev] -pre-commit install - -# Or switch to the developer version from the official version -pip install pre-commit -pre-commit install -pre-commit run --all-files -``` - -You can refer to [Contributing to FederatedScope](https://federatedscope.io/docs/contributor/) for more details. - -Welcome to join in our [Slack channel](https://join.slack.com/t/federatedscopeteam/shared_invite/zt-1apmfjqmc-hvpYbsWJdm7D93wPNXbqww), or DingDing group (please scan the following QR code) for discussion. -federatedscope-logo