client routes: add exclusive-proxy config to control fallback behavior by nikagra · Pull Request #881 · scylladb/java-driver

nikagra · 2026-04-28T14:42:58Z

Summary

Adds advanced.client-routes.exclusive-proxy (boolean, default false) to control whether the driver falls back to a node's broadcast address when no route entry exists in system.client_routes
When false (default): nodes without a route entry are reached directly via their broadcast address — backward-compatible, supports mixed proxy/direct topologies
When true: any node without a route is treated as unreachable; resolve() throws IllegalStateException with a WARN log, the node stays DOWN, and the reconnection loop retries until CLIENT_ROUTES_CHANGE populates the route

Motivation

Two requirements needed to coexist:

Mixed proxy/direct topologies (original design): some nodes are behind the private endpoint, others are not — the driver connects directly to unrouted nodes. The default exclusive-proxy = false preserves this behavior.
Strict proxy enforcement (new): when a new node is added there is a race window before its system.client_routes entry is published. With exclusive-proxy = true the driver refuses to bypass the proxy during this window, matching the behavior of the Rust driver fix (failing address translation on missing entries). The node goes DOWN and reconnects once the route is available.

Changes

DefaultDriverOption: new CLIENT_ROUTES_EXCLUSIVE_PROXY enum constant
TypedDriverOption: new typed Boolean constant
reference.conf: advanced.client-routes.exclusive-proxy = false with documentation
ClientRoutesEndPoint: restored fallbackEndPoint field; resolve() uses the fallback when exclusive-proxy = false, throws IllegalStateException + WARN when exclusive-proxy = true
ClientRoutesTopologyMonitor: reads config flag in constructor; passes fallback endpoint (null when exclusive-proxy is true) and flag to ClientRoutesEndPoint
Tests: constructor calls updated; separate test cases for exclusive and non-exclusive modes; createHandlerWithExclusiveProxy() helper in the topology monitor test

Not in this PR

Graceful migration (soft-drop of existing direct connections when a route is added for a node) — tracked as a follow-up

dkropachev · 2026-04-28T15:05:25Z

@nikagra , I would prefere to have an option to enable it and handle case when host switches from direct connection to client routes, you can make host connection pool either softly drop all the connections that used to directly, check if it won't blow the PR.

Copilot

Pull request overview

Adds an advanced.client-routes.exclusive-proxy configuration flag to control whether the driver may fall back to a node’s broadcast address when no entry exists in system.client_routes.

Changes:

Introduces CLIENT_ROUTES_EXCLUSIVE_PROXY as a new built-in driver option (typed + defaulted).
Updates client-routes endpoint resolution to either fall back (default) or throw when a route is missing (exclusive mode).
Updates reference configuration documentation and expands tests to cover both exclusive/non-exclusive behaviors.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
core/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesTopologyMonitor.java	Reads the new flag and passes fallback endpoint as `null` in exclusive mode; adjusts endpoint construction behavior.
core/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesEndPoint.java	Adds exclusive-proxy behavior in `resolve()` (fallback vs WARN+throw) and updates constructor contract.
core/src/main/java/com/datastax/oss/driver/api/core/config/DefaultDriverOption.java	Adds `CLIENT_ROUTES_EXCLUSIVE_PROXY` option path.
core/src/main/java/com/datastax/oss/driver/api/core/config/TypedDriverOption.java	Adds typed `Boolean` option constant for exclusive-proxy.
core/src/main/java/com/datastax/oss/driver/api/core/config/OptionsMap.java	Registers default value (`false`) for the new typed option.
core/src/main/resources/reference.conf	Documents `advanced.client-routes.exclusive-proxy` with default `false`.
core/src/test/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesTopologyMonitorTest.java	Adds/updates tests for exclusive vs non-exclusive fallback behavior and host_id-null handling.
core/src/test/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesEndPointTest.java	Updates constructor calls and adds tests for exclusive mode throwing vs fallback mode.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ssing When a node has client-routes configured but no matching entry is found, fall back to the broadcast address by default (backward-compatible). When advanced.client-routes.exclusive-proxy=true, throw instead of falling back so the node stays DOWN until a CLIENT_ROUTES_CHANGE event provides the route. The constructor of ClientRoutesEndPoint now enforces that fallbackEndPoint is non-null when exclusiveProxy=false, making the invariant explicit at construction time rather than silently failing at resolve() time. The WARN log on null host_id no longer mentions exclusive-proxy behaviour since that path fires regardless of the flag; the message now focuses on the actual cause (corrupted system tables).

…cement Introduces advanced.client-routes.exclusive-proxy (default: false). When false (default), the driver falls back to the node broadcast address when no client-route entry exists — preserving backward compatibility with mixed proxy/direct topologies. When true, no fallback is attempted: the node stays DOWN until a CLIENT_ROUTES_CHANGE event publishes the route. This prevents silent bypass of the proxy infrastructure during node additions.

dkropachev

Pl

dkropachev · 2026-05-08T03:16:55Z

+   *
+   * <p>Value type: boolean
+   */
+  CLIENT_ROUTES_EXCLUSIVE_PROXY("advanced.client-routes.exclusive-proxy");


Suggested change

CLIENT_ROUTES_EXCLUSIVE_PROXY("advanced.client-routes.exclusive-proxy");

CLIENT_ROUTES_DIRECT_CONNECTION_FALLBACK("advanced.client-routes.direct-connection-fallback");

dkropachev · 2026-05-08T03:18:41Z

+      @Nullable EndPoint fallbackEndPoint,
+      boolean exclusiveProxy) {


You can just keep fallbackEndPoint for the case when fallback is not allowed

dkropachev · 2026-05-08T03:26:17Z

    } catch (IOException e) {
      throw new UncheckedIOException("DNS resolution failed for host_id=" + hostId, e);
    }
-    return fallbackEndPoint.resolve();


public class UnknownHostException extends IOException, so fallback wont work on broken DNS, which is exactly what we want.

dkropachev · 2026-05-08T03:36:44Z

+   * Whether the driver connects exclusively through proxies when client routes are configured.
+   *
+   * <p>When {@code true}, any node whose {@code host_id} does not appear in the {@code
+   * system.client_routes} table is treated as unreachable; the driver will never fall back to the
+   * node's broadcast address. The node stays DOWN and the reconnection loop retries until a {@code
+   * CLIENT_ROUTES_CHANGE} event populates the route.
+   *
+   * <p>When {@code false} (the default), nodes that have no route entry are contacted directly
+   * using their broadcast address, preserving backward-compatible mixed proxy/direct topologies.


Actually, fallback is needed for cases where a node has a record in client-routes, but it cannot connect to (or, in a semi-perfect world, resolve) the proxy address specified in that client-routes record.

Also, we should mention that when this feature is enabled, the driver does not actively destroy existing direct connections once the proxy connection becomes available again. As a result, old connections may continue to operate, and users might notice that some node pools still maintain direct connections to the node.

nikagra force-pushed the fix/client-routes-disable-fallback branch from b870f1d to 90b8684 Compare April 28, 2026 14:52

nikagra changed the title ~~client routes: disable fallback to broadcast address when route is missing~~ client routes: add exclusive-proxy config to control fallback behavior Apr 28, 2026

nikagra force-pushed the fix/client-routes-disable-fallback branch from 1d9717b to 805ec7f Compare April 28, 2026 18:24

nikagra marked this pull request as ready for review April 28, 2026 20:16

nikagra requested review from Copilot and dkropachev April 28, 2026 20:16

Copilot AI reviewed Apr 28, 2026

View reviewed changes

Comment thread ...rc/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesTopologyMonitor.java Outdated

Comment thread core/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesEndPoint.java

nikagra added 2 commits April 29, 2026 23:52

nikagra force-pushed the fix/client-routes-disable-fallback branch from 805ec7f to d341fef Compare April 29, 2026 21:52

dkropachev requested changes May 8, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

client routes: add exclusive-proxy config to control fallback behavior#881

client routes: add exclusive-proxy config to control fallback behavior#881
nikagra wants to merge 2 commits intoscylladb:scylla-4.xfrom
nikagra:fix/client-routes-disable-fallback

nikagra commented Apr 28, 2026 •

edited

Loading

Uh oh!

dkropachev commented Apr 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

dkropachev left a comment

Uh oh!

dkropachev May 8, 2026

Uh oh!

dkropachev May 8, 2026

Uh oh!

dkropachev May 8, 2026

Uh oh!

dkropachev May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	CLIENT_ROUTES_EXCLUSIVE_PROXY("advanced.client-routes.exclusive-proxy");
	CLIENT_ROUTES_DIRECT_CONNECTION_FALLBACK("advanced.client-routes.direct-connection-fallback");

		@Nullable EndPoint fallbackEndPoint,
		boolean exclusiveProxy) {

Conversation

nikagra commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Changes

Not in this PR

Uh oh!

dkropachev commented Apr 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

dkropachev left a comment

Choose a reason for hiding this comment

Uh oh!

dkropachev May 8, 2026

Choose a reason for hiding this comment

Uh oh!

dkropachev May 8, 2026

Choose a reason for hiding this comment

Uh oh!

dkropachev May 8, 2026

Choose a reason for hiding this comment

Uh oh!

dkropachev May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nikagra commented Apr 28, 2026 •

edited

Loading