Skip to content

11622 : OutlierDetection should use Ticker, not TimeProvider #12110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

11622 : OutlierDetection should use Ticker, not TimeProvider #12110

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ec44974
11622 : OutlierDetection should use Ticker, not TimeProvider
vimanikag May 29, 2025
8f513b7
11622 : OutlierDetection should use Ticker, not TimeProvider
vimanikag May 29, 2025
debbd7a
11243 : RLS cleanups
vimanikag Jun 3, 2025
3735a6c
11622 : OutlierDetection should use Ticker, not TimeProvider
vimanikag Jun 3, 2025
c831f1e
11622 : OutlierDetection should use Ticker, not TimeProvider
vimanikag Jun 4, 2025
b8c0e0c
11622 : OutlierDetection should use Ticker, not TimeProvider
vimanikag Jun 6, 2025
aaf5016
11622 : OutlierDetection should use Ticker, not TimeProvider
vimanikag Jun 9, 2025
e336fa6
Merge branch 'grpc:master' into vimanikag-FIX_11622_OTP
vimanikag Jun 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions util/src/main/java/io/grpc/util/OutlierDetectionLoadBalancer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import static java.util.concurrent.TimeUnit.NANOSECONDS;

import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Ticker;
import com.google.common.collect.ForwardingMap;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
Expand All @@ -39,7 +40,6 @@
import io.grpc.Status;
import io.grpc.SynchronizationContext;
import io.grpc.SynchronizationContext.ScheduledHandle;
import io.grpc.internal.TimeProvider;
import java.net.SocketAddress;
import java.util.ArrayList;
import java.util.Collection;
Expand Down Expand Up @@ -82,7 +82,7 @@ public final class OutlierDetectionLoadBalancer extends LoadBalancer {
private final SynchronizationContext syncContext;
private final Helper childHelper;
private final GracefulSwitchLoadBalancer switchLb;
private TimeProvider timeProvider;
private Ticker ticker;
private final ScheduledExecutorService timeService;
private ScheduledHandle detectionTimerHandle;
private Long detectionTimerStartNanos;
Expand All @@ -95,14 +95,14 @@ public final class OutlierDetectionLoadBalancer extends LoadBalancer {
/**
* Creates a new instance of {@link OutlierDetectionLoadBalancer}.
*/
public OutlierDetectionLoadBalancer(Helper helper, TimeProvider timeProvider) {
public OutlierDetectionLoadBalancer(Helper helper, Ticker ticker) {
logger = helper.getChannelLogger();
childHelper = new ChildHelper(checkNotNull(helper, "helper"));
switchLb = new GracefulSwitchLoadBalancer(childHelper);
endpointTrackerMap = new EndpointTrackerMap();
this.syncContext = checkNotNull(helper.getSynchronizationContext(), "syncContext");
this.timeService = checkNotNull(helper.getScheduledExecutorService(), "timeService");
this.timeProvider = timeProvider;
this.ticker = ticker;
logger.log(ChannelLogLevel.DEBUG, "OutlierDetection lb created.");
}

Expand Down Expand Up @@ -151,7 +151,7 @@ public Status acceptResolvedAddresses(ResolvedAddresses resolvedAddresses) {
// If a timer has started earlier we cancel it and use the difference between the start
// time and now as the interval.
initialDelayNanos = Math.max(0L,
config.intervalNanos - (timeProvider.currentTimeNanos() - detectionTimerStartNanos));
config.intervalNanos - (ticker.read() - detectionTimerStartNanos));
}

// If a timer has been previously created we need to cancel it and reset all the call counters
Expand Down Expand Up @@ -201,7 +201,7 @@ class DetectionTimer implements Runnable {

@Override
public void run() {
detectionTimerStartNanos = timeProvider.currentTimeNanos();
detectionTimerStartNanos = ticker.read();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you need to audit fully, if somewhere in inner methods it is being used in unwanted way - like it is mentioned in javadoc. The comparison should always be done doing subtraction.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have taken the report using grep commands and verified with the IDE search with ticker.read() but I did not see the scenario where the ticker.read() is directly used in finding the difference without doing the subtraction and We are frequently using the this ticker.read() in junit's but not seen it's used in unwanted way

I can see the ticker.read() in a few below implementation classes but not seen it's used in an unwanted way , please find the attached Audit_TR.txt for Your reference.

Aduit_ticker_read.txt

AbstractNettyHandler.java
NettyServerHandler.java
CachingRlsLbClient.java
LinkedHashLruCache.java
AdaptiveThrottler.java
PingTracker.java
OutlierDetectionLoadBalancer.java

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to audit all the files. This single file has such a bug. I'd outright tell you, but then I have to audit it for any other missed cases, because clearly you aren't finding them, and that is literally the only interesting part of this PR.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review and the guidance , I have done a careful audit on the OutlierDetectionLoadBalancer method calls and noticed ticker.read() has invoked two api's

  1. acceptResolvedAddresses(ResolvedAddresses resolvedAddresses)

We have already had discussion on the below API and we are good with existing code as it's already been using subtraction to find the time difference instead of using > or < directly in finding the time difference and we are intended to the System.nanoTime() documentation

initialDelayNanos = Math.max(0L,
config.intervalNanos - (ticker.read() - detectionTimerStartNanos));

  1. maxEjectionTimeElapsed(long currentTimeNanos)

I'm hoping our long discussions on finding the bug in this API and I have observed the maybeUnejectOutliers method has invoked it in run() with detectionTimerStartNanos and which is assigned with ticker.read() and using the currentTimeNanos > maxEjectionTimeNanos expression while returning the boolean value if the currentTimeNanos is after the maxEjectionTimeNanos on maxEjectionTimeElapsed

endpointTrackerMap.maybeUnejectOutliers(detectionTimerStartNanos);

grpc-java/util/src/main/java/io/grpc/util/OutlierDetectionLoadBalancer.java

Line 212 in b8c0e0c

endpointTrackerMap.maybeUnejectOutliers(detectionTimerStartNanos);

I have addressed this issue in the latest commit , please review it and let me know Your thoughts if I have missed to find any other bugs in the OutlierDetectionLoadBalancer


endpointTrackerMap.swapCounters();

Expand Down Expand Up @@ -638,7 +638,7 @@ public boolean maxEjectionTimeElapsed(long currentTimeNanos) {
config.baseEjectionTimeNanos * ejectionTimeMultiplier,
maxEjectionDurationSecs);

return currentTimeNanos > maxEjectionTimeNanos;
return currentTimeNanos - maxEjectionTimeNanos > 0;
Copy link
Member

@shivaspeaks shivaspeaks Jun 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. It is still not solving the complete problem.

As it was pointed before, you just need follow the variable in which you're storing ticker.read() and you will find where it is being used in an incorrect way. Oh I see, you reached to this point, it's just 2 lines above where you need to see. ejectionTimeNanos , and we are adding something into it! That seems to be a problem right? Probably we can get rid of that maxEjectionTimeNanos and you just need to do algebraic manipulation here.

Probably there could be some more places, idk you need to check.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've not looked really looked at all the changes yet, but this does have the correct shape and is the case I had noticed.

@shivaspeaks, what's happening here is the subtraction may overflow/underflow in order to produce the correct result, as the nanotime itself is allowed to overflow during the execution of the process. But the difference will still be accurate. Doing the subtraction essentially "removes" nanotime from the result ((nanotime + A) - (nanoTime + B) = A - B), independent of all overflow/underflow, as long as A-B itself isn't too large/small to fit in a long.

Copy link
Member

@shivaspeaks shivaspeaks Jun 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but that subtraction needs to be done in one equation. Storing in a variable itself can also overflow. That's what I was referring to by saying algebraic manipulation. It's just we are trying to avoid any addition and storing in long (which potentially can overflow/underflow).

We can do something like this:
return currentTimeNanos - ejectionTimeNanos + Math.min(config.baseEjectionTimeNanos * ejectionTimeMultiplier, maxEjectionDurationSecs) > 0;
Or
return currentTimeNanos > ejectionTimeNanos - Math.min(config.baseEjectionTimeNanos * ejectionTimeMultiplier, maxEjectionDurationSecs);

Both works fine. The first one will also work since it's in one equation.

Copy link
Member

@shivaspeaks shivaspeaks Jun 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh the second one will not work, we should avoid that. We are doing same mistake as original if we go by the second way.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, so let me put it this way. There are two problems here overlapped.

  1. The one Eric was talking about. Comparison should be done in a - b > 0 way and not a > b.
  2. Adding something to long potentially may overflow.
    long maxEjectionTimeNanos = ejectionTimeNanos + Math.min( config.baseEjectionTimeNanos * ejectionTimeMultiplier, maxEjectionDurationSecs);

I over looked 1st one. I had it first but this time while reviewing I totally over looked what javadoc was saying. So yes, Eric is right about it. This change does solve that. But should we also try to solve 2nd concern here? @ejona86

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Storing it in a variable has no impact to the result, as long as the variable is not smaller than the original operands. Floating point sometimes can get messy with the order of operations, but integer math is very safe for addition/subtraction (in Java).

Try it out:

jshell> Integer.MAX_VALUE + 100 - Integer.MAX_VALUE;
$1 ==> 100

jshell> int overflow = Integer.MAX_VALUE + 100;
overflow ==> -2147483549

jshell> overflow - Integer.MAX_VALUE;
$3 ==> 100

Signed addition/subtraction is identical in twos complement as unsigned addition/subtraction. So there's actually not any data lost in the above; just some of the result is encoded in the sign bit. But the same actually holds when the high bits are truncated. If you reverse the operation, information is not lost.

jshell> (byte) 200;
$1 ==> -56

jshell> byte b = (byte)(Integer.MAX_VALUE + 200);
b ==> -57

jshell> (byte)(((int) b) - Integer.MAX_VALUE)
$3 ==> -56

jshell> Byte.toUnsignedInt(b)
$4 ==> 199

jshell> (byte)(Byte.toUnsignedInt(b) - Integer.MAX_VALUE)
$5 ==> -56

jshell> b & 0xFF
$6 ==> 199

jshell> (byte)((b & 0xFF) - Integer.MAX_VALUE)
$7 ==> -56

The (int) b cast does a sign extension, so the high bits will be 1s there. The Byte.toUnsignedInt(b) does not do sign extension, which is the same as b & 0xFF (the sign extension happens, and then those bits are thrown away; I assume the JIT recognizes the pattern and optimizes the sign extension away). As you can see, the high bits don't actually impact the result (as long as you truncate back down to byte).

(Note that C++ allows unsigned integer overflow/underflow but considers signed integer overflow/underflow undefined behavior. So in other languages, you might have to be careful. The process is fine with all this, but there can be per-language details.)

You may be confused some by Java auto-promoting types to integer (or long, or double). Those rules are confusing, but it isn't relevant in this case because the intermediate results are long.

jshell> byte b = (byte) 100 + (byte) 100
|  Error:
|  incompatible types: possible lossy conversion from int to byte
|  byte b = (byte) 100 + (byte) 100;
|           ^---------------------^

jshell> byte b = (byte) 100 + 100L
|  Error:
|  incompatible types: possible lossy conversion from long to byte
|  byte b = (byte) 100 + 100L;
|           ^---------------^

The first shows + promotes to int. The second shows how a long operand makes the result long. I'm very thankful that Golang chose not to retain this from C. Auto-promotion/coercion makes things harder to understand.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My most recent really-long comment was without having read Shiva's most recent comment.

Adding something to long potentially may overflow.

I think I ended up answering this, but basically, "that's okay and expected." Case (1) and case (2) are actually the same case, as case (1) is necessary because of overflow. And for the same reasons case (1) works out, case (2) will be fine too.

... As long as the final result actually fits within 63 bits. baseEjectionTimeNanos, maxEjectionDurationSecs, maxEjectionTimeNanos could be quite large... or small. The service config code path allows negatives. JsonUtil itself makes sure to saturate instead of overflowing. Xds does check that the value isn't negative (and note that we don't use the service config parsing flow for xds). Protobuf requires durations to be less than 10000 years, which is useless to us, but it does throw on overflow when converting to nanoseconds. So in both cases we could be dealing with Long.MAX_VALUE, and one case allows Long.MIN_VALUE.

Thinking through those cases is generally pretty useless. Instead it is much more reliable to clamp all the inputs to 10 or 100 years and then not deal with it at all here. But that should be done separately, as it is unrelated to Ticker vs TimeProvider.

}

/** Tracks both successful and failed call counts. */
Expand Down
4 changes: 2 additions & 2 deletions util/src/main/java/io/grpc/util/OutlierDetectionLoadBalancerProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,14 @@

package io.grpc.util;

import com.google.common.base.Ticker;
import io.grpc.Internal;
import io.grpc.LoadBalancer;
import io.grpc.LoadBalancer.Helper;
import io.grpc.LoadBalancerProvider;
import io.grpc.NameResolver.ConfigOrError;
import io.grpc.Status;
import io.grpc.internal.JsonUtil;
import io.grpc.internal.TimeProvider;
import io.grpc.util.OutlierDetectionLoadBalancer.OutlierDetectionLoadBalancerConfig;
import io.grpc.util.OutlierDetectionLoadBalancer.OutlierDetectionLoadBalancerConfig.FailurePercentageEjection;
import io.grpc.util.OutlierDetectionLoadBalancer.OutlierDetectionLoadBalancerConfig.SuccessRateEjection;
Expand All @@ -34,7 +34,7 @@ public final class OutlierDetectionLoadBalancerProvider extends LoadBalancerProv

@Override
public LoadBalancer newLoadBalancer(Helper helper) {
return new OutlierDetectionLoadBalancer(helper, TimeProvider.SYSTEM_TIME_PROVIDER);
return new OutlierDetectionLoadBalancer(helper, Ticker.systemTicker());
}

@Override
Expand Down
2 changes: 1 addition & 1 deletion util/src/test/java/io/grpc/util/OutlierDetectionLoadBalancerTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ public Void answer(InvocationOnMock invocation) throws Throwable {
when(mockStreamTracerFactory.newClientStreamTracer(any(),
any())).thenReturn(mockStreamTracer);

loadBalancer = new OutlierDetectionLoadBalancer(mockHelper, fakeClock.getTimeProvider());
loadBalancer = new OutlierDetectionLoadBalancer(mockHelper, fakeClock.getTicker());
}

@Test
Expand Down