[CELEBORN-1513] Support wildcard bind in dual stack environments #2713
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
this is a duplicate of #2633 -- I had forgotten to update it for a long time and it became stale with a lot of conflicts, hence I just made a new branch/PR. |
|
cc @FMX @SteNicholas @pan3793 @RexXiong @turboFei please review, thank you :) |
RexXiong
reviewed
Sep 2, 2024
common/src/main/scala/org/apache/celeborn/common/util/Utils.scala
Outdated
Show resolved
Hide resolved
added 2 commits
September 2, 2024 15:03
|
thanks @RexXiong , I have addressed your comments. Please take another look , thanks!! :) |
RexXiong
reviewed
Sep 3, 2024
common/src/main/scala/org/apache/celeborn/common/util/Utils.scala
Outdated
Show resolved
Hide resolved
added 2 commits
September 3, 2024 00:58
RexXiong
approved these changes
Sep 5, 2024
|
@RexXiong @SteNicholas are we able to merge this? :) |
FMX
approved these changes
Sep 6, 2024
|
thanks everyone for the reviews!! |
pan3793
reviewed
Sep 12, 2024
pan3793
reviewed
Sep 12, 2024
common/src/main/java/org/apache/celeborn/common/protocol/TransportModuleConstants.java
Show resolved
Hide resolved
pan3793
pushed a commit
that referenced
this pull request
Sep 13, 2024
### What changes were proposed in this pull request? enrich the docs for supporting wildcard address bind in this [PR](#2713). ### Why are the changes needed? better docs ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? N/A - just docs change Closes #2736 from akpatnam25/CELEBORN-1513-doc-followup. Authored-by: Aravind Patnam <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
s0nskar
pushed a commit
to s0nskar/celeborn
that referenced
this pull request
Sep 16, 2024
### What changes were proposed in this pull request? Support wildcard bind for RPC and HTTP servers. When wildcard address is used, the service is able to listen to both ipv4 and ipv6 traffic in dual-stack environments. The specific scenario where this becomes relevant is as follows: If some of the compute infrastructure is IPv4 only, some v6 only and others dual stack - the way we can have a single Celeborn infra to cater to all is by: a) Set bind.preferip to false - so that advertised address is the host and not IP. b) bind to wild card address With both in place, the v4 only compute nodes will resolve the v4 address and connect to v4 ip/port. Likewise, for v6 only. Dual stack compute nodes will use prefer ipv6 Java flag to resolve to either v4 or v6. This is how we are handling the combination of scenarios internally. ### Why are the changes needed? see above. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tested on a server using netstat, and tried connecting to via `nc -4` and `nc -6` to ensure connection was there. Closes apache#2713 from akpatnam25/CELEBORN-1513-fix. Authored-by: Aravind Patnam <[email protected]> Signed-off-by: mingji <[email protected]>
s0nskar
pushed a commit
to s0nskar/celeborn
that referenced
this pull request
Sep 16, 2024
### What changes were proposed in this pull request? enrich the docs for supporting wildcard address bind in this [PR](apache#2713). ### Why are the changes needed? better docs ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? N/A - just docs change Closes apache#2736 from akpatnam25/CELEBORN-1513-doc-followup. Authored-by: Aravind Patnam <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
FMX
pushed a commit
that referenced
this pull request
Oct 8, 2024
…e log in startup document ### What changes were proposed in this pull request? Update `advertised endpoint` of master service log in startup document. ### Why are the changes needed? #2713 has changed the startup log of of master service in `NettyRpcEnv`, which should update the log of startup document. ``` logInfo(s"Starting RPC Server [${config.name}] on ${config.bindAddress}:$actualPort " + s"with advertised endpoint ${config.advertiseAddress}:$actualPort") ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No. Closes #2773 from SteNicholas/CELEBORN-1513. Authored-by: SteNicholas <[email protected]> Signed-off-by: mingji <[email protected]>
turboFei
pushed a commit
to turboFei/incubator-celeborn
that referenced
this pull request
Oct 10, 2024
…e log in startup document ### What changes were proposed in this pull request? Update `advertised endpoint` of master service log in startup document. ### Why are the changes needed? apache#2713 has changed the startup log of of master service in `NettyRpcEnv`, which should update the log of startup document. ``` logInfo(s"Starting RPC Server [${config.name}] on ${config.bindAddress}:$actualPort " + s"with advertised endpoint ${config.advertiseAddress}:$actualPort") ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No. Closes apache#2773 from SteNicholas/CELEBORN-1513. Authored-by: SteNicholas <[email protected]> Signed-off-by: mingji <[email protected]>
wankunde
pushed a commit
to wankunde/celeborn
that referenced
this pull request
Oct 11, 2024
### What changes were proposed in this pull request? Support wildcard bind for RPC and HTTP servers. When wildcard address is used, the service is able to listen to both ipv4 and ipv6 traffic in dual-stack environments. The specific scenario where this becomes relevant is as follows: If some of the compute infrastructure is IPv4 only, some v6 only and others dual stack - the way we can have a single Celeborn infra to cater to all is by: a) Set bind.preferip to false - so that advertised address is the host and not IP. b) bind to wild card address With both in place, the v4 only compute nodes will resolve the v4 address and connect to v4 ip/port. Likewise, for v6 only. Dual stack compute nodes will use prefer ipv6 Java flag to resolve to either v4 or v6. This is how we are handling the combination of scenarios internally. ### Why are the changes needed? see above. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tested on a server using netstat, and tried connecting to via `nc -4` and `nc -6` to ensure connection was there. Closes apache#2713 from akpatnam25/CELEBORN-1513-fix. Authored-by: Aravind Patnam <[email protected]> Signed-off-by: mingji <[email protected]>
wankunde
pushed a commit
to wankunde/celeborn
that referenced
this pull request
Oct 11, 2024
### What changes were proposed in this pull request? enrich the docs for supporting wildcard address bind in this [PR](apache#2713). ### Why are the changes needed? better docs ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? N/A - just docs change Closes apache#2736 from akpatnam25/CELEBORN-1513-doc-followup. Authored-by: Aravind Patnam <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
HolyLow
pushed a commit
to HolyLow/celeborn
that referenced
this pull request
Oct 14, 2024
…e log in startup document ### What changes were proposed in this pull request? Update `advertised endpoint` of master service log in startup document. ### Why are the changes needed? apache#2713 has changed the startup log of of master service in `NettyRpcEnv`, which should update the log of startup document. ``` logInfo(s"Starting RPC Server [${config.name}] on ${config.bindAddress}:$actualPort " + s"with advertised endpoint ${config.advertiseAddress}:$actualPort") ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No. Closes apache#2773 from SteNicholas/CELEBORN-1513. Authored-by: SteNicholas <[email protected]> Signed-off-by: mingji <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Support wildcard bind for RPC and HTTP servers. When wildcard address is used, the service is able to listen to both ipv4 and ipv6 traffic in dual-stack environments.
The specific scenario where this becomes relevant is as follows:
If some of the compute infrastructure is IPv4 only, some v6 only and others dual stack - the way we can have a single Celeborn infra to cater to all is by:
a) Set bind.preferip to false - so that advertised address is the host and not IP.
b) bind to wild card address
With both in place, the v4 only compute nodes will resolve the v4 address and connect to v4 ip/port.
Likewise, for v6 only.
Dual stack compute nodes will use prefer ipv6 Java flag to resolve to either v4 or v6.
This is how we are handling the combination of scenarios internally.
Why are the changes needed?
see above.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Tested on a server using netstat, and tried connecting to via
nc -4andnc -6to ensure connection was there.