对于星核杯隐匿查询案例代码运行有关Ray报错 #126
Comments
|
@integrationex01 你好，我们的CPU/Memory推荐最低配置是 8C16G，建议使用所要求的机型。 |
我主要是想要知道隐语对于Ray集群的详细配置操作是否只有对sf.init()中的几项参数可以操作，还是说可以直接对ray集群进行操作。 |
可以参考集群仿真模式，先启动一个 ray 集群，再让 secretflow 连接到已启动的集群。 此外，ray.init() 的大部分参数可以用过 sf.init() 传入：https://github.com/secretflow/secretflow/blob/main/secretflow/device/driver.py#L519 |
|
修改对应核心教以及内存容量后报错： 目前运行环境为： (SPURuntime(device id=None, party=alice) pid=5496) 2024-04-28 14:51:36.840 [error] [global.cpp:BRPC:306] external/com_google _protobuf/src/google/protobuf/message_lite.cc:480 在WSL环境： 查询耗时：2976.5s 请问这个错误如何处理呢？ |
|
辛苦将贴一下运行代码 |
代码如下： import os
import sys
import time
import logging
import multiprocessing
from absl import app
import spu
import secretflow as sf
import pandas as pd
from pathlib import Path
#import random
# init log
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
# SPU settings
# alice as pir client
# bob as pir server
cluster_def = {
'nodes': [
{
'party': 'alice',
'id': 'local:0',
'address': f'127.0.0.1:17268',
# 'tls_opts': {
# 'server_ssl_opts': {
# 'certificate_path': 'alice servercert.pem',
# 'private_key_path': 'alice serverkey.pem',
# # The options used for verify peer's client certificate
# 'ca_file_path': 'cacert.pem',
# # Maximum depth of the certificate chain for verification
# 'verify_depth': 1
# },
# 'client_ssl_opts': {
# 'certificate_path': 'alice clientcert.pem',
# 'private_key_path': 'alice clientkey.pem',
# # The options used for verify peer's server certificate
# 'ca_file_path': 'cacert.pem',
# # Maximum depth of the certificate chain for verification
# 'verify_depth': 1
# }
# }
},
{
'party': 'bob',
'id': 'local:1',
'address': f'127.0.0.1:17269',
# 'tls_opts': {
# 'server_ssl_opts': {
# 'certificate_path': 'bob servercert.pem',
# 'private_key_path': 'bob serverkey.pem',
# # The options used for verify peer's client certificate
# 'ca_file_path': 'cacert.pem',
# # Maximum depth of the certificate chain for verification
# 'verify_depth': 1
# },
# 'client_ssl_opts': {
# 'certificate_path': 'bob clientcert.pem',
# 'private_key_path': 'bob clientkey.pem',
# # The options used for verify peer's server certificate
# 'ca_file_path': 'cacert.pem',
# # Maximum depth of the certificate chain for verification
# 'verify_depth': 1
# }
# }
},
],
'runtime_config': {
'protocol': spu.spu_pb2.SEMI2K,
'field': spu.spu_pb2.FM128,
},
}
link_desc = {
'recv_timeout_ms': 3600000,
}
def main(_):
# sf init
sf.init(['alice','bob'],address='local',log_to_driver=True,omp_num_threads=multiprocessing.cpu_count())
# init log
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
alice = sf.PYU('alice')
bob = sf.PYU('bob')
key_columns = ['name']
label_columns = ['age']
spu = sf.SPU(cluster_def, link_desc)
pir_input_path = f"{str(Path.home())}/pythonpath/alice_pir_input_1kw.csv"
pir_oprf_key_path = f"{str(Path.home())}/pythonpath/server_secret_key.bin"
pir_setup_path = f"{str(Path.home())}/pythonpath/alice_1kw_setup"
bob_df = pd.DataFrame({
# "name": ["tony", "bob"],
"name": ["李建华"],
})
bob_df.to_csv(f"{str(Path.home())}/pythonpath/bob_pir_query_1kw.csv", index=False)
bob_input_path = f"{str(Path.home())}/pythonpath/bob_pir_query_1kw.csv"
start = time.time()
# server setup
reports = spu.pir_setup(
server='alice',
input_path=pir_input_path,
key_columns=key_columns,
label_columns=label_columns,
oprf_key_path=pir_oprf_key_path,
setup_path=pir_setup_path,
num_per_query=1,
label_max_len=18,
)
#print(f"psi reports: {reports}")
logging.info(f"offline psi reports: {reports}")
logging.info(f"cost time: {time.time() - start}")
# client config
bob_config = {
'input_path': bob_input_path,
'key_columns': key_columns,
'output_path': f"{str(Path.home())}/pythonpath/bob_pir_result_1kw.csv",
}
# server config
alice_config = {
'oprf_key_path': pir_oprf_key_path,
'setup_path': pir_setup_path,
}
query_config = {
alice: alice_config,
bob: bob_config,
}
start = time.time()
reports = spu.pir_query(
server='alice',
config=query_config,
)
logging.info(f"online pir reports: {reports}")
logging.info(f"cost time: {time.time() - start}")
sf.shutdown()
if __name__ == '__main__':
app.run(main) |
|
|
@Chrisdehe 您好，请问该参数设置时是否有对应的规则可以参考，即该参数与setup的数据量的关系。或者说是数据量setup大小与内存的关系。 |
|
你现在的client是要查询1000万条嘛？ |
@6fj |
|
|
@6fj |
|
目前看到的问题是，brpc的message中间超出了serialization的限制，label_max_len 看看能不能小一点。当然也不能太小，但如果太小了装不下会有明确的报错。 |
@6fj 是的，我目前遇到的就是这个问题（太大超出限制，太小装不下），在测试环境针对该问题进行了调试，预估在35-38附近，但是遇到了在int情况下不能满足需求的情况，所以就想要知道这个label_max_len和数据量之间的关系。目前我使用的机器是16C64G不知道增加内存是否能解决该问题呢？ |
|
你遇到这个问题，应该不是现有配置可以解决的，我们需要优化代码或者暴露更多的BFV参数。能否将你这边的数据脱敏传上来呢？ |
请问如何发送给您这边呢。压缩后39M（input+query+数据生成python文件） |
@6fj GitHub上传文件大小限制在了25M，所以需要其他方式。还有就是上述代码中的查询文件生成使用的名字“李建华”可能在alice_pir_input_1kw.csv文件中不存在，所以请先用excel对该文件进行预览。 |
|
@integrationex01 您可以通过邮箱（[email protected]）或者微信（技术支持：secretflow02）的方式来传输文件。 |
|
@Chrisdehe |
|
@6fj @Chrisdehe 您好，请问这个问题现在有解决方案了吗？ |
|
我们确实发现了一个bug，我们将会立刻修复。但是同时，你的数据中也存在重复key的情况，需要你处理一下，感谢。 |
@6fj |
|
https://github.com/secretflow/psi/blob/c80ac38fd8e0df9860001d46bc064153c9e203c6/examples/pir/generate_pir_data.cc |
|
hi,请问您的gcc版本号是多少，这边建议使用11.4再尝试下 |
@lq0404510 您好，以将gcc升级到11.4版本，bazel编译还是有如下问题： WARNING: Download from https://golang.org/dl/?mode=json&include=all failed: class java.io.IOException connect timed out
INFO: Analyzed target //examples/pir:generate_pir_data (82 packages loaded, 6802 targets configured).
INFO: Found 1 target...
ERROR: /home/chenhengji/cppPath/secretflow/psi/examples/pir/BUILD.bazel:19:14: Linking examples/pir/generate_pir_data failed: (Exit 1): gcc failed: error executing command (from target //examples/pir:generate_pir_data) /usr/bin/gcc @bazel-out/k8-opt/bin/examples/pir/generate_pir_data-2.params
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o):dso_dlfcn.c:function dlfcn_globallookup: error: undefined reference to 'dlopen'
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o):dso_dlfcn.c:function dlfcn_globallookup: error: undefined reference to 'dlsym'
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o):dso_dlfcn.c:function dlfcn_globallookup: error: undefined reference to 'dlclose'
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o):dso_dlfcn.c:function dlfcn_pathbyaddr: error: undefined reference to 'dladdr'
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o):dso_dlfcn.c:function dlfcn_pathbyaddr: error: undefined reference to 'dlerror'
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o):dso_dlfcn.c:function dlfcn_bind_func: error: undefined reference to 'dlsym'
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o):dso_dlfcn.c:function dlfcn_bind_func: error: undefined reference to 'dlerror'
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o):dso_dlfcn.c:function dlfcn_load: error: undefined reference to 'dlopen'
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o):dso_dlfcn.c:function dlfcn_load: error: undefined reference to 'dlclose'
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o):dso_dlfcn.c:function dlfcn_load: error: undefined reference to 'dlerror'
bazel-out/k8-opt/bin/external/com_github_openssl_openssl/openssl/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o):dso_dlfcn.c:function dlfcn_unload: error: undefined reference to 'dlclose'
collect2: error: ld returned 1 exit status
Target //examples/pir:generate_pir_data failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 339.269s, Critical Path: 124.49s
INFO: 157 processes: 23 internal, 134 linux-sandbox.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target``` |
|
hi！您可以先在psi目录下的.bazelrc文件中添加一行命令：build --linkopt=-ldl，再尝试下 |
@6fj @Chrisdehe 您好，我对代码进行了修改，通过Index以及name共同作为keylabel进行查询，目前生成测试数据的代码以及运行隐匿查询的代码已经发送到对应邮箱。 |
@lq0404510 感谢，成功了。 |
hi，咨询两个问题 1、这个问题您使用的gcc版本是多少？2、这两次gcc的安装您是通过什么途径安装的（yum、conda或者其他方式） |
hi，您可以将您的conda环境中的python升级至python3.10，然后进行安装pip install secretflow==1.6.1b0 |
Issue Type
Bug
Source
binary
Secretflow Version
secretflow 1.0.0b3
OS Platform and Distribution
Asianux
Python version
3.8.16
Bazel version
No response
GCC/Compiler version
No response
What happend and What you expected to happen.
Reproduction code to reproduce the issue.
The text was updated successfully, but these errors were encountered: