feat(functions): Add support for REST based remote functions #10911

Joe-Abraham · 2024-09-02T05:36:22Z

netlify · 2024-09-02T05:36:42Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`97483aa`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/675bf65783bdd90008633603

pedroerp · 2024-09-09T15:19:16Z

Pretty cool! I see the PR is still as draft, but I can help review when it's ready. Would also be nice to add some documentation on how to use it, the configs parameters, etc.

czentgr · 2024-11-19T23:43:19Z

@Joe-Abraham There are build errors. Can you please take a look?
Also please note that the PR title needs adjusting to the new format. I updated it.

czentgr

Initial review.

czentgr · 2024-11-19T23:50:28Z

velox/functions/remote/client/CMakeLists.txt

@@ -16,11 +16,23 @@ velox_add_library(velox_functions_remote_thrift_client ThriftClient.cpp)
 velox_link_libraries(velox_functions_remote_thrift_client
                     PUBLIC remote_function_thrift FBThrift::thriftcpp2)

+find_package(CURL REQUIRED)


Curl is a dependency that comes in from a sepaarte dependency but you can re-use this

set(curl_SOURCE BUNDLED) velox_resolve_dependency(curl)

This should set up Curl to be used as a dependency for use in velox_link_libraries.
You can look at other dependencies that are introduced like this in the main CMake.

Updated the code.

czentgr · 2024-11-19T23:51:50Z

velox/functions/remote/client/Remote.cpp

 #include "velox/vector/VectorStream.h"

+#include <fmt/format.h>


System includes need to go before velox dependencies in alphabetical order (except for the header on line 17 that is for this cpp file).

Updated the code.

czentgr · 2024-11-19T23:52:21Z

velox/functions/remote/client/Remote.cpp

+#include <sstream>
+#include <string>
+
+#include "RestClient.h"


This needs to be a relative path with the root in the velox main dir and then get into the alphabetical list for the velox includes.

Updated the code.

czentgr · 2024-11-20T00:04:20Z

velox/functions/remote/server/CMakeLists.txt

@@ -24,3 +24,18 @@ add_executable(velox_functions_remote_server_main RemoteFunctionServiceMain.cpp)
 target_link_libraries(
  velox_functions_remote_server_main velox_functions_remote_server
  velox_functions_prestosql)
+
+add_library(velox_functions_remote_server_rest RemoteFunctionRestService.cpp)


Ok, so the idea is that a user can implement Velox functions and add them to this REST server and serve them up for execution though a remote velox engine?

Yes, that's the idea. And it also used by test file. Similar to the ThriftServer.

Joe-Abraham · 2024-11-28T03:53:00Z

@aditi-pandit Can you please review the changes?

aditi-pandit · 2024-12-02T11:56:37Z

velox/functions/remote/client/Remote.h

-/// (non-remote) function registered with the same name. The `overwrite` flag
-/// controls whether to overwrite in these cases.
+/// (non-remote) function registered with the same name. The `overwrite`
+/// flagwrite controls whether to overwrite in these cases.


Nit : wording... maybe write is not needed here.

aditi-pandit · 2024-12-02T11:57:59Z

velox/functions/remote/client/Remote.cpp

 #include <folly/io/async/EventBase.h>
+#include <sstream>
+#include <string>
+#include "velox/common/memory/ByteStream.h"


Add empty line between the system and velox includes.

Added new line

aditi-pandit · 2024-12-02T11:58:34Z

velox/functions/remote/client/Remote.cpp

 #include "velox/vector/VectorStream.h"

+#include "velox/functions/remote/client/RestClient.h"


Move this include to the correct alphabetical order in the previous velox includes.

corrected the inclusion list

aditi-pandit · 2024-12-02T12:00:03Z

velox/functions/remote/client/Remote.h

-  /// Network address of the servr to communicate with. Note that this can hold
-  /// a network location (ip/port pair) or a unix domain socket path (see
+  /// URL of the HTTP/REST server for remote function.
+  /// Or Network address of the servr to communicate with. Note that this can


Nit : spelling "server"

Corrected it

aditi-pandit · 2024-12-02T14:51:37Z

velox/functions/remote/client/RestClient.cpp

+}
+size_t writeCallback(char* ptr, size_t size, size_t nmemb, void* userdata) {
+  auto* outputBuf = static_cast<IOBufQueue*>(userdata);
+  size_t total_size = size * nmemb;


nit : Use camelCase naming -> totalSize

Corrected it

aditi-pandit · 2024-12-02T14:52:31Z

velox/functions/remote/client/RestClient.cpp

+  return totalCopied;
+}
+size_t writeCallback(char* ptr, size_t size, size_t nmemb, void* userdata) {
+  auto* outputBuf = static_cast<IOBufQueue*>(userdata);


Use camel case "userData"

Corrected it

aditi-pandit · 2024-12-02T14:53:53Z

velox/functions/remote/client/RestClient.cpp

+using namespace folly;
+namespace facebook::velox::functions {
+namespace {
+size_t readCallback(char* dest, size_t size, size_t nmemb, void* userp) {


Please write comments explaining the signature and the parameters.

Added the documentation

aditi-pandit · 2024-12-02T14:54:45Z

velox/functions/remote/client/RestClient.h

+
+class RestClient : public HttpClient {
+ public:
+  std::unique_ptr<folly::IOBuf> performCurlRequest(


Please write the documentation for this API. What is it for ? What do the parameters mean ?

Added the documentation

aditi-pandit · 2024-12-02T15:07:22Z

velox/functions/remote/server/RemoteFunctionRestService.h

+      memory::memoryManager()->addLeafPool()};
+};
+
+class listener : public std::enable_shared_from_this<listener> {


Can you add some documentation about these classes and what are they for ?

Added documentation

aditi-pandit · 2024-12-02T15:19:42Z

velox/functions/remote/server/RemoteFunctionRestService.cpp

+        // called to use the functions mentioned in this map
+};
+
+TypePtr deserializeType(const std::string& input) {


There is a lot of repetition between this code and https://github.com/facebookincubator/velox/blob/main/velox/functions/remote/server/RemoteFunctionService.cpp. Please can you refactor.

introduced the RemoteFunctionHelper.h and moved the duplicate code.

Co-authored-by: Wills Feng <[email protected]>

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 2, 2024

Joe-Abraham changed the title ~~Add support for REST based remote functions~~ [WIP] Add support for REST based remote functions Sep 2, 2024

Joe-Abraham force-pushed the udf branch 12 times, most recently from b88d136 to b85e0e6 Compare September 4, 2024 09:32

Yuhta requested review from pedroerp and mbasmanova September 4, 2024 18:28

Joe-Abraham force-pushed the udf branch 6 times, most recently from 0cd4510 to 74023dc Compare September 9, 2024 08:10

Joe-Abraham force-pushed the udf branch 3 times, most recently from abe87e1 to 6c1606e Compare September 13, 2024 05:06

Joe-Abraham force-pushed the udf branch 3 times, most recently from 05115f4 to 2ffec26 Compare September 20, 2024 05:18

Joe-Abraham changed the title ~~[WIP] Add support for REST based remote functions~~ Add support for REST based remote functions Nov 4, 2024

Joe-Abraham force-pushed the udf branch 5 times, most recently from a541614 to 8609347 Compare November 12, 2024 07:18

Joe-Abraham force-pushed the udf branch 7 times, most recently from 4e0627d to b589f78 Compare November 19, 2024 08:06

czentgr changed the title ~~Add support for REST based remote functions~~ feat(functions): Add support for REST based remote functions Nov 19, 2024

czentgr reviewed Nov 20, 2024

View reviewed changes

Joe-Abraham force-pushed the udf branch from b589f78 to 5f6bcdb Compare November 24, 2024 10:35

Joe-Abraham requested a review from czentgr November 25, 2024 06:23

Joe-Abraham mentioned this pull request Nov 27, 2024

Add e2e testcases for REST based remote functions prestodb/presto#23777

Open

Joe-Abraham marked this pull request as ready for review November 28, 2024 03:52

Joe-Abraham requested review from assignUser and majetideepak as code owners November 28, 2024 03:52

Joe-Abraham mentioned this pull request Nov 28, 2024

[native] Add support for REST based remote function prestodb/presto#23568

Open

aditi-pandit reviewed Dec 2, 2024

View reviewed changes

Joe-Abraham force-pushed the udf branch from 9d0787d to 37e6d84 Compare December 3, 2024 08:57

Joe-Abraham requested a review from aditi-pandit December 3, 2024 10:17

Joe-Abraham force-pushed the udf branch from 37e6d84 to 8278ff8 Compare December 3, 2024 10:17

feat(functions): Add support for REST based remote functions

97483aa

Co-authored-by: Wills Feng <[email protected]>

Joe-Abraham force-pushed the udf branch from 8278ff8 to 97483aa Compare December 13, 2024 08:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(functions): Add support for REST based remote functions #10911

feat(functions): Add support for REST based remote functions #10911

Joe-Abraham commented Sep 2, 2024 •

edited

Loading

netlify bot commented Sep 2, 2024 •

edited

Loading

pedroerp commented Sep 9, 2024

czentgr commented Nov 19, 2024

czentgr left a comment

czentgr Nov 19, 2024

Joe-Abraham Nov 24, 2024

czentgr Nov 19, 2024

Joe-Abraham Nov 24, 2024

czentgr Nov 19, 2024

Joe-Abraham Nov 24, 2024

czentgr Nov 20, 2024

Joe-Abraham Nov 24, 2024

Joe-Abraham commented Nov 28, 2024

aditi-pandit Dec 2, 2024

aditi-pandit Dec 2, 2024

Joe-Abraham Dec 3, 2024

aditi-pandit Dec 2, 2024

Joe-Abraham Dec 3, 2024

aditi-pandit Dec 2, 2024

Joe-Abraham Dec 3, 2024

aditi-pandit Dec 2, 2024

Joe-Abraham Dec 3, 2024

aditi-pandit Dec 2, 2024

Joe-Abraham Dec 3, 2024

aditi-pandit Dec 2, 2024

Joe-Abraham Dec 3, 2024

aditi-pandit Dec 2, 2024

Joe-Abraham Dec 3, 2024

aditi-pandit Dec 2, 2024

Joe-Abraham Dec 3, 2024

aditi-pandit Dec 2, 2024

Joe-Abraham Dec 3, 2024

		#include "velox/vector/VectorStream.h"

		#include <fmt/format.h>

		#include "velox/vector/VectorStream.h"

		#include "velox/functions/remote/client/RestClient.h"

feat(functions): Add support for REST based remote functions #10911

Are you sure you want to change the base?

feat(functions): Add support for REST based remote functions #10911

Conversation

Joe-Abraham commented Sep 2, 2024 • edited Loading

netlify bot commented Sep 2, 2024 • edited Loading

✅ Deploy Preview for meta-velox canceled.

pedroerp commented Sep 9, 2024

czentgr commented Nov 19, 2024

czentgr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Joe-Abraham commented Nov 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Joe-Abraham commented Sep 2, 2024 •

edited

Loading

netlify bot commented Sep 2, 2024 •

edited

Loading