repo-sync-2024-02-22T16:58:09+0800 (#88)

secretflow · Feb 22, 2024 · ad7a4e5 · ad7a4e5
1 parent 025c680
commit ad7a4e5
Show file tree

Hide file tree

Showing 49 changed files with 945 additions and 814 deletions.
diff --git a/RELEASE.md b/RELEASE.md
@@ -5,6 +5,9 @@
 > - `[API]` prefix for API changes.
 > - `[Improvement]` prefix for implementation improvement.
 
+## v0.2.0.dev240222
+- [API] expose PIR API.
+
 ## v0.2.0.dev240219
 
 - [Feature] add ecdh logger for debug purposes.

diff --git a/docs/_static/rr22_psi.png b/docs/_static/rr22_psi.png
diff --git a/docs/development/psi_protocol_intro.rst b/docs/development/psi_protocol_intro.rst
@@ -9,6 +9,7 @@ SecretFlow SPU implements the following PSI protocols,
 - Semi-honest OT-based two-party PSI protocol (with improved communication efficiency) [BC22]_
 - Differentially Private (DP) PSI Protocol [DP-PSI]_
 - Unbalanced PSI Protocol
+- Semi-honest and Malicious VOLE-based two-party PSI protocol [RS21]_ [RR22]_
 
 ECDH-PSI
 --------
@@ -355,6 +356,44 @@ Labeled PSI Parameters
 |                   | coeff_modulus_bits   | {48} / {48, 30, 30} / {56, 56, 56, 50}                              |
 +-------------------+----------------------+---------------------------------------------------------------------+
 
+RR22 Blazing Fast PSI
+--------------
+
+[RS21]_ introduced an efficient PSI protocol based on OKVS and VOLE. [RR22]_ present significant improvements
+to the OKVS data structure along with new techniquesfor further reducing the communication overhead of [RS]21.
+
+Oblivous Key-Value Stores(OKVS) consists of algorithms Encode and Decode. Encode takes a list of key-value (k,v)
+pairs as input and returns an abstract data structure S. Decode takes such a data structure S and a key k' as
+input, and gives some output v'.
+
+Pseudorandom correlation generators(PCGs) allow for the efficient generation of
+oblivious transfer (OT) and vector oblivious linear evaluations (VOLE)
+with sublinear communication and concretely good computational overhead.
+PCG makes use of a so-called LPN-friendly errorcorrecting code.
+`secretflow/YACL <https://github.com/secretflow/yacl>`_  provides VOLE code implementation.
+LPN-friendly coeds now support [CRR21]_ silver codes(LDPC) and [BCGI+22]_ Expand-Accumulate Codes.
+Silver is Most efficient, but not recommended to use due to its security flaw.
+
+Semi-honest Protocol:
+
+.. figure:: ../_static/rr22_psi.png
+
+1. The Receiver samples :math:`r \leftarrow \{0,1\}^\kappa` and computes
+   :math:`\vec{P} :=  \mathrm{Encode} (L,r)` where
+   :math:`L := \{(H^{n*m}(x,r),H(x))|x \in X\}`.
+
+2. Sender and Receiver Run (sub)VOLE protocol, Sender gets :math:`\Delta` and
+   :math:`\vec{B}`, Receiver gets :math:`\vec{A}` and :math:`\vec{C}`, such that:
+   :math:`\vec{C}=\Delta *\vec{A'}+\vec{B}`.
+
+3. Receiver sends :math:`r, \vec{A}=\vec{A'}+\vec{P}` to Sender. Sender defines
+   :math:`\vec{K}=\vec{B}+\Delta \cdot \vec{A}`.
+
+4. Sender sends :math:`Y'=H^{n*m}(\vec{Y},r)\cdot \vec{K}-\Delta \cdot H(\vec{Y})`
+   to the Receiver.
+
+5. Receiver compares :math:`X'=H^{n*m}(\vec{X},r)\cdot \vec{C}` and :math:`Y'`, outputs
+   intersection result :math:`X \cap Y`.
 
 Reference
 ------------
@@ -375,9 +414,12 @@ Reference
 .. [Ber06] Daniel J. Bernstein. Curve25519: new diffie-hellman speed records. In In Public
    Key Cryptography (PKC), Springer-Verlag LNCS 3958, page 2006, 2006. (Cited on page 4.)
 
+.. [BCGI+22] Elette Boyle, Geoffroy Couteau, Niv Gilboa, Yuval Ishai, Lisa Kohl, Nicolas Resch, Peter Scholl.
+   Correlated Pseudorandomness from Expand-Accumulate Codes. Crypto2022.
+
 .. [BBCD+11] Baldi, P., Baronio, R., Cristofaro, E.D., Gasti, P., Tsudik, G.: Countering GATTACA:
    Efficient and Secure Testing of Fully-sequenced Human Genomes. In: ACM
-   Conference on Computer and Communications Security. pp. 691–702. ACM (2011)
+   Conference on Computer and Communications Security. pp. 691–702. ACM (2011).
 
 .. [CIK+20] G. Couteau, Y. Ishai, L. Kohl, E. Boyle, P. Scholl, and N. Gilboa. Efficient pseudorandom
    correlation generators from ring-lpn. Springer-Verlag, 2020.
@@ -394,6 +436,9 @@ Reference
    Michael Rosenberg. Labeled PSI from Homomorphic Encryption with Reduced Computation and Communication
    CCS'21: Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications SecurityNovember 2021
 
+.. [CRR21] Geoffroy Couteau, Peter Rindal, and Srinivasan Raghuraman. Silver: Silent VOLE and Oblivious Transfer
+   from Hardness of Decoding Structured LDPC Codes. Crypto2021.
+
 .. [DP-PSI] Differentially-Private PSI https://arxiv.org/pdf/2208.13249.pdf
 
 .. [FourQ] Costello, C., Longa, P.: Fourq: four-dimensional decompositions on a q-curve over the mersenne prime.
@@ -422,6 +467,13 @@ Reference
 .. [RA18] Resende, A.C.D., Aranha, D.F.: Faster unbalanced private set intersection. In: Meiklejohn, S.,
    Sako, K. (eds.) FC2018. LNCS, vol. 10957, pp. 203{221. Springer, Heidelberg (Feb / Mar 2018)
 
+.. [RR22] Srinivasan Raghuraman and Peter Rindal. Blazing Fast PSI from Improved OKVS and Subfield VOLE. CCS'22.
+
+.. [RRT23] Srinivasan Raghuraman, Peter Rindal, Titouan Tanguy. Expand-Convolute Codes for Pseudorandom
+   Correlation Generators from LPN. Crypto2023.
+
+.. [RS21] Peter Rindal and Phillipp Schoppmann. VOLE-PSI: fast OPRF and circuit-psi from vector-ole. EUROCRYPT2021.
+
 .. [SEAL] Microsoft SEAL (release 4.0). https://github.com/Microsoft/SEAL (Sep 2022),
    microsoft Research, Redmond, WA.
 

diff --git a/docs/reference/index.rst b/docs/reference/index.rst
@@ -8,4 +8,5 @@ This page covers all Protocol Buffers message as APIs.
 
    psi_config
    psi_v2_config
+   pir_config
    launch_config
diff --git a/docs/reference/launch_config.md b/docs/reference/launch_config.md
@@ -44,6 +44,7 @@ Please check psi.v2.PsiConfig and psi.v2.UbPsiConfig at **PSI v2 Configuration**
 | [**oneof**](https://developers.google.com/protocol-buffers/docs/proto3#oneof) runtime_config.legacy_psi_config | [ BucketPsiConfig](#bucketpsiconfig) | Please check at psi.proto. |
 | [**oneof**](https://developers.google.com/protocol-buffers/docs/proto3#oneof) runtime_config.psi_config | [ v2.PsiConfig](#v2psiconfig) | Please check at psi_v2.proto. |
 | [**oneof**](https://developers.google.com/protocol-buffers/docs/proto3#oneof) runtime_config.ub_psi_config | [ v2.UbPsiConfig](#v2ubpsiconfig) | Please check at psi_v2.proto. |
+| [**oneof**](https://developers.google.com/protocol-buffers/docs/proto3#oneof) runtime_config.pir_config | [ PirConfig](#pirconfig) | Please check at pir.proto. |
  <!-- end Fields -->
  <!-- end HasFields -->
  <!-- end messages -->

diff --git a/docs/reference/pir_config.md b/docs/reference/pir_config.md
@@ -0,0 +1,149 @@
+# PIR Configuration
+
+## Table of Contents
+
+
+
+- Messages
+    - [ApsiServerConfig](#apsiserverconfig)
+    - [PirClientConfig](#pirclientconfig)
+    - [PirConfig](#pirconfig)
+    - [PirResultReport](#pirresultreport)
+    - [PirServerConfig](#pirserverconfig)
+
+
+
+- Enums
+    - [PirConfig.Mode](#pirconfigmode)
+    - [PirProtocol](#pirprotocol)
+
+
+
+- [Scalar Value Types](#scalar-value-types)
+
+
+
+ <!-- end services -->
+
+## Messages
+
+
+### ApsiServerConfig
+Server config for APSI protocol.
+
+
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+| oprf_key_path | [ string](#string) | The path of oprf_key file path. This field is not required for MODE_SERVER_FULL. |
+| num_per_query | [ uint32](#uint32) | The number of per query. |
+| compressed | [ bool](#bool) | compressed Seal ciphertext |
+| max_items_per_bin | [ uint32](#uint32) | max items per bin, i.e. Interpolate polynomial max degree. optional. |
+ <!-- end Fields -->
+ <!-- end HasFields -->
+
+
+### PirClientConfig
+Client configs.
+
+
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+| input_path | [ string](#string) | The input csv file path of pir. |
+| key_columns | [repeated string](#string) | The key columns name of input data. |
+| output_path | [ string](#string) | The path of query output csv file path. |
+ <!-- end Fields -->
+ <!-- end HasFields -->
+
+
+### PirConfig
+The config for PIR. This is the entrypoint for all PIR tasks.
+
+
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+| mode | [ PirConfig.Mode](#pirconfigmode) | none |
+| pir_protocol | [ PirProtocol](#pirprotocol) | The PIR protocol. |
+| pir_server_config | [ PirServerConfig](#pirserverconfig) | Required for MODE_SERVER_SETUP, MODE_SERVER_ONLINE and MODE_SERVER_FULL. |
+| pir_client_config | [ PirClientConfig](#pirclientconfig) | Required for MODE_CLIENT. |
+ <!-- end Fields -->
+ <!-- end HasFields -->
+
+
+### PirResultReport
+The report of pir result.
+
+
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+| data_count | [ int64](#int64) | The data count of input/query. |
+ <!-- end Fields -->
+ <!-- end HasFields -->
+
+
+### PirServerConfig
+Server configs.
+setup_path is only required field for MODE_SERVER_ONLINE.
+setup_path is not required for MODE_SERVER_FULL.
+
+
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+| input_path | [ string](#string) | The input csv file path. |
+| setup_path | [ string](#string) | The path of setup output path. |
+| key_columns | [repeated string](#string) | The key columns name of input data. |
+| label_columns | [repeated string](#string) | The label columns name of input data. |
+| label_max_len | [ uint32](#uint32) | The max number bytes of label. |
+| bucket_size | [ uint32](#uint32) | split data bucket to do pir query |
+| apsi_server_config | [ ApsiServerConfig](#apsiserverconfig) | For APSI protocol only |
+ <!-- end Fields -->
+ <!-- end HasFields -->
+ <!-- end messages -->
+
+## Enums
+
+
+### PirConfig.Mode
+
+
+| Name | Number | Description |
+| ---- | ------ | ----------- |
+| MODE_UNSPECIFIED | 0 | none |
+| MODE_SERVER_SETUP | 1 | Server with setup stage. |
+| MODE_SERVER_ONLINE | 2 | Server with online stage. |
+| MODE_SERVER_FULL | 3 | Server with both online and offline stages. |
+| MODE_CLIENT | 4 | Client |
+
+
+
+
+### PirProtocol
+The algorithm type of pir.
+
+| Name | Number | Description |
+| ---- | ------ | ----------- |
+| PIR_PROTOCOL_UNSPECIFIED | 0 | none |
+| PIR_PROTOCOL_KEYWORD_PIR_APSI | 1 | Keyword PIR APSI Reference: https://github.com/microsoft/APSI |
+
+
+ <!-- end Enums -->
+ <!-- end Files -->
+
+## Scalar Value Types
+
+| .proto Type | Notes | C++ Type | Java Type | Python Type |
+| ----------- | ----- | -------- | --------- | ----------- |
+| <div><h4 id="double" /></div><a name="double" /> double |  | double | double | float |
+| <div><h4 id="float" /></div><a name="float" /> float |  | float | float | float |
+| <div><h4 id="int32" /></div><a name="int32" /> int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int |
+| <div><h4 id="int64" /></div><a name="int64" /> int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long |
+| <div><h4 id="uint32" /></div><a name="uint32" /> uint32 | Uses variable-length encoding. | uint32 | int | int/long |
+| <div><h4 id="uint64" /></div><a name="uint64" /> uint64 | Uses variable-length encoding. | uint64 | long | int/long |
+| <div><h4 id="sint32" /></div><a name="sint32" /> sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int |
+| <div><h4 id="sint64" /></div><a name="sint64" /> sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long |
+| <div><h4 id="fixed32" /></div><a name="fixed32" /> fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int |
+| <div><h4 id="fixed64" /></div><a name="fixed64" /> fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long |
+| <div><h4 id="sfixed32" /></div><a name="sfixed32" /> sfixed32 | Always four bytes. | int32 | int | int |
+| <div><h4 id="sfixed64" /></div><a name="sfixed64" /> sfixed64 | Always eight bytes. | int64 | long | int/long |
+| <div><h4 id="bool" /></div><a name="bool" /> bool |  | bool | boolean | boolean |
+| <div><h4 id="string" /></div><a name="string" /> string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode |
+| <div><h4 id="bytes" /></div><a name="bytes" /> bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str |
diff --git a/docs/reference/pir_config_md.tmpl b/docs/reference/pir_config_md.tmpl
@@ -0,0 +1,75 @@
+# PIR Configuration
+
+## Table of Contents
+{{range .Files}}
+{{if .HasServices}}
+- Services
+  {{range .Services}}  - [{{.Name}}](#{{.FullName | lower | replace "." ""}})
+  {{end}}
+{{end}}
+{{if .HasMessages}}
+- Messages
+  {{range .Messages}}  - [{{.LongName}}](#{{.LongName | lower | replace "." ""}})
+  {{end}}
+{{end}}
+{{if .HasEnums}}
+- Enums
+  {{range .Enums}}  - [{{.LongName}}](#{{.LongName | lower | replace "." ""}})
+  {{end}}
+{{end}}
+{{end}}
+- [Scalar Value Types](#scalar-value-types)
+
+{{range .Files}}
+
+{{range .Services -}}
+## {{.Name}} {#{{.FullName | lower | replace "." ""}}}
+{{.Description}}
+
+{{range .Methods -}}
+### {{.Name}}
+
+> **rpc** {{.Name}}([{{.RequestLongType}}](#{{.RequestLongType | lower | replace "." ""}}))
+    [{{.ResponseLongType}}](#{{.ResponseLongType | lower | replace "." ""}})
+
+{{ .Description}}
+{{end}} <!-- end methods -->
+{{end}} <!-- end services -->
+
+## Messages
+{{range .Messages}}
+
+### {{.LongName}}
+{{.Description}}
+
+{{if .HasFields}}
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+{{range .Fields -}}
+	| {{if .IsOneof}}[**oneof**](https://developers.google.com/protocol-buffers/docs/proto3#oneof) {{.OneofDecl}}.{{end}}{{.Name}} | [{{if .IsMap}}map {{else}}{{.Label}} {{end}}{{.LongType}}](#{{.LongType | lower | replace "." ""}}) | {{if .Description}}{{nobr .Description}}{{if .DefaultValue}} Default: {{.DefaultValue}}{{end}}{{else}}none{{end}} |
+{{end}} <!-- end Fields -->
+{{end}} <!-- end HasFields -->
+{{end}} <!-- end messages -->
+
+## Enums
+{{range .Enums}}
+
+### {{.LongName}}
+{{.Description}}
+
+| Name | Number | Description |
+| ---- | ------ | ----------- |
+{{range .Values -}}
+	| {{.Name}} | {{.Number}} | {{if .Description}}{{nobr .Description}}{{else}}none{{end}} |
+{{end}}
+
+{{end}} <!-- end Enums -->
+{{end}} <!-- end Files -->
+
+## Scalar Value Types
+
+| .proto Type | Notes | C++ Type | Java Type | Python Type |
+| ----------- | ----- | -------- | --------- | ----------- |
+{{range .Scalars -}}
+  | <div><h4 id="{{.ProtoType | lower | replace "." ""}}" /></div><a name="{{.ProtoType}}" /> {{.ProtoType}} | {{.Notes}} | {{.CppType}} | {{.JavaType}} | {{.PythonType}} |
+{{end}}
diff --git a/docs/reference/psi_v2_config.md b/docs/reference/psi_v2_config.md
@@ -343,8 +343,8 @@ config for unbalanced psi.
 | role | [ Role](#role) | Required for all modes except MODE_OFFLINE_GEN_CACHE. |
 | input_config | [ IoConfig](#ioconfig) | Config for origin input. Servers: Required for MODE_OFFLINE_GEN_CACHE, MODE_OFFLINE, MODE_FULL. Clients: Required for MODE_ONLINE and MODE_FULL. |
 | keys | [repeated string](#string) | Join keys. Servers: Required for MODE_OFFLINE_GEN_CACHE, MODE_OFFLINE, MODE_FULL. Clients: Required for MODE_ONLINE and MODE_FULL. |
-| server_secret_key_config | [ IoConfig](#ioconfig) | Servers: Required for MODE_OFFLINE_GEN_CACHE, MODE_OFFLINE, MODE_ONLINE and MODE_FULL. |
-| cache_config | [ IoConfig](#ioconfig) | Required. |
+| server_secret_key_path | [ string](#string) | Servers: Required for MODE_OFFLINE_GEN_CACHE, MODE_OFFLINE, MODE_ONLINE and MODE_FULL. |
+| cache_path | [ string](#string) | Required. |
 | server_get_result | [ bool](#bool) | none |
 | client_get_result | [ bool](#bool) | none |
 | disable_alignment | [ bool](#bool) | It true, output is not promised to be aligned. Valid if both server_get_result and client_get_result are true. |

diff --git a/docs/update_data.sh b/docs/update_data.sh
@@ -6,6 +6,12 @@ docker run --rm -v $(pwd)/reference/:/out \
                 pseudomuto/protoc-gen-doc \
                 --doc_opt=/out/psi_config_md.tmpl,psi_config.md psi/proto/psi.proto
 
+# pir_config_md.tmpl is adapted from https://github.com/pseudomuto/protoc-gen-doc/blob/master/examples/templates/grpc-md.tmpl.
+docker run --rm -v $(pwd)/reference/:/out \
+                -v $(pwd)/../:/protos \
+                pseudomuto/protoc-gen-doc \
+                --doc_opt=/out/pir_config_md.tmpl,pir_config.md psi/proto/pir.proto
+
 
 # psi_v2_config_md.tmpl is adapted from https://github.com/pseudomuto/protoc-gen-doc/blob/master/examples/templates/grpc-md.tmpl.
 docker run --rm -v $(pwd)/reference/:/out \