Backport OffsetDateTime changes from 4.0 #3090

xiazcy · 2025-04-04T19:09:37Z

OffsetDateTime is now the default Date type in Gremlin.
Changes were back-ported from Java/Python. For Go/JS/.NET, the OffsetDateTime serializers were added.
Made OffsetDateTime the default serializer for dates in GLVs, the Date type will only deserialize responses.

VOTE +1.

…ateTime is now the default Date type in Gremlin; added OffsetDateTime serializers to Go/JS/.NET. Made OffsetDateTime the default serializer for date types in GLVs (Date type will only deserialize).

Cole-Greer · 2025-04-04T19:39:53Z

...e/src/main/java/org/apache/tinkerpop/gremlin/language/translator/PythonTranslateVisitor.java

-        final Date dt = DatetimeHelper.parse(removeFirstAndLastCharacters(dtString));
-        sb.append("datetime.datetime.utcfromtimestamp(" + dt.getTime() + " / 1000.0)");
+        final OffsetDateTime dt = DatetimeHelper.parse(removeFirstAndLastCharacters(dtString));
+        sb.append("datetime.datetime.fromtimestamp(" + dt.toEpochSecond() + ").astimezone(datetime.timezone.utc)");


Are we unnecessarily losing precision by using dt.toEpochSecond()? Python's datetime.datetime supports microsecond precision: https://docs.python.org/3/library/datetime.html#datetime.datetime. Perhaps fromisoformat is preferable.

Same comment applies for the old PythonTranslator.

Cole-Greer · 2025-04-04T19:55:24Z

...-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/DateAddStep.java

+        OffsetDateTime date = (OffsetDateTime) object;
+        OffsetDateTime new_date;
+
+        switch (dateToken) {


nit: dateToken and value should arguably be converted to a Duration in the constructor for this step so this conversion does not need to happen each time the step is iterated.

Cole-Greer · 2025-04-04T20:07:08Z

...-core/src/main/java/org/apache/tinkerpop/gremlin/language/translator/GoTranslateVisitor.java

-        final Date dt = DatetimeHelper.parse(removeFirstAndLastCharacters(dtString));
-        sb.append("time.UnixMilli(" + dt.getTime() + ")");
+        final OffsetDateTime dt = DatetimeHelper.parse(removeFirstAndLastCharacters(dtString));
+        sb.append("time.UnixMilli(" +  dt.toInstant().toEpochMilli() + ")");


We may be able to preserve more precision as well as the timezone offset via https://pkg.go.dev/time#Date

Cole-Greer · 2025-04-04T20:07:26Z

...rc/main/java/org/apache/tinkerpop/gremlin/process/traversal/translator/GolangTranslator.java

@@ -138,6 +139,11 @@ protected String getSyntax(final Date o) {
            return "time.UnixMilli(" + o.getTime() + ")";
        }

+        @Override
+        protected String getSyntax(final OffsetDateTime o) {
+            return "time.UnixMilli(" + o.toInstant().toEpochMilli() + ")";


We may be able to preserve more precision as well as the timezone offset via https://pkg.go.dev/time#Date

...ain/java/org/apache/tinkerpop/gremlin/process/traversal/translator/JavascriptTranslator.java

Cole-Greer · 2025-04-04T20:17:26Z

gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/util/DatetimeHelper.java

@@ -93,45 +97,41 @@ public static String format(final Instant d) {
     * </ul>>
     *
     */
-    public static Date parse(final String d) {
+    public static OffsetDateTime parse(final String d) {


There's a lot going on in this method. Based on the javadoc comment, could it simply be replaced with OffsetDateTime.parse(String) or OffsetDateTime.from(TemporalAccessor)?

Not quite, dates without time or time without dates cannot be transformed into OffsetDateTime, so we'll still need the manual parsing. But you have a point that I can combine the last two cases.

Cole-Greer · 2025-04-04T20:33:19Z

...e/src/test/java/org/apache/tinkerpop/gremlin/language/grammar/GeneralLiteralVisitorTest.java

-            final Date dt = (Date) new GenericLiteralVisitor(new GremlinAntlrToJava()).visitDateLiteral(ctx);
-            assertTrue(new Date().getTime() - dt.getTime() < 1000);
+            final OffsetDateTime dt = (OffsetDateTime) new GenericLiteralVisitor(new GremlinAntlrToJava()).visitDateLiteral(ctx);
+            assertTrue(OffsetDateTime.now(UTC).toEpochSecond() - dt.toEpochSecond() < 1000);


The < 1000 is too broad here as the units have changed from millis to seconds.

Suggested change

assertTrue(OffsetDateTime.now(UTC).toEpochSecond() - dt.toEpochSecond() < 1000);

assertTrue(OffsetDateTime.now(UTC).toInstant().toEpochMilli() - dt.toInstant().toEpochMilli() < 1000);

Cole-Greer · 2025-04-04T21:33:21Z

gremlin-go/driver/graphBinary.go

+	offset := readIntSafe(data, i)
+	// only way to pass offset info, timezone display is fixed to UTC as consequence (offset is calculated properly)
+	loc := time.FixedZone("UTC", int(offset))
+	datetime := time.Date(int(year), time.Month(month), int(day), 0, 0, 0, int(ns), loc)


int(ns) may be an issue for times which are late in the day. According to the io docs, the "time" value can be from 0 to 86,399,999,999,999. The max size of int is only guaranteed to be 2,147,483,647, although on most 64-bit machines it will be a 64 bit integer in practice.

Unfortunately the Date constructor requires int, it's more like a caveat of go, not sure if it's avoidable.

kmcginnes · 2025-04-04T22:21:07Z

...c/main/java/org/apache/tinkerpop/gremlin/language/translator/JavascriptTranslateVisitor.java

        sb.append("new Date(");
-        sb.append(dt.getTime());
+        sb.append(dt.toInstant().toEpochMilli());


This should work, and it seems equivalent to what was there before. As long as you've tested this with various time zones it should be fine.

However, I'd wager that if someone is using the translator to translate a query, I doubt they started with a date value as the millisecond since unix epoch. More likely they started with a string version of the date time.

Perhaps a better approach would be to use the ISO string value as the input to new Date(). The precision should be the same (milliseconds), and it supports time zone offsets or UTC. It's just a more human friendly translation.

kmcginnes · 2025-04-04T22:35:05Z

...-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/DateAddStep.java

-        cal.add(DTtoCalendar.get(dateToken), value);
-
-        return cal.getTime();
+        if (!(object instanceof OffsetDateTime)) throw new IllegalArgumentException("dateAdd accept only DateTime.");


Should the error be:

dateAdd accept only OffsetDateTime.

We changed it a general DateTime type in 4.0, but you are right it prob make more sense to be OffsetDateTime here. I will also make it accept Date as well for compatibility reasons.

kmcginnes · 2025-04-04T22:51:00Z

...-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/DateAddStep.java

Is it intentional for this step to only have second precision?

The original design of the step uses the dateToken enums, which only goes to second precision. I suppose we could add support for additional precisions, but that would be outside the scope of this PR.

I too find it strange that date add and date diff do not work with millisecond precision.

I agree that the available precision here is odd, although this is really out of scope for this PR and should be taken as a separate discussion. I think there should be come consideration of taking a Duration as input instead of an int and DT enum, although some consideration is needed around how to make Duration's constructable in gremlin-lang.

kmcginnes · 2025-04-04T22:53:47Z

...core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/DateDiffStep.java


-        return (((Date) object).getTime() - otherDateMs) / 1000;
+        return (((OffsetDateTime) object).toEpochSecond() - otherDateMs);


Did you mean to use .toEpochMilli()?

If you did intend to use seconds instead of milliseconds, then you should change the variable name otherDateMs to be otherDateSeconds or something.

No this step meant to return epoch time in seconds, so I'm just directly using it.

kmcginnes · 2025-04-04T23:10:16Z

...e/src/test/java/org/apache/tinkerpop/gremlin/language/grammar/GeneralLiteralVisitorTest.java

-            final Date dt = (Date) new GenericLiteralVisitor(new GremlinAntlrToJava()).visitDateLiteral(ctx);
-            assertTrue(new Date().getTime() - dt.getTime() < 1000);
+            final OffsetDateTime dt = (OffsetDateTime) new GenericLiteralVisitor(new GremlinAntlrToJava()).visitDateLiteral(ctx);
+            assertTrue(OffsetDateTime.now(UTC).toEpochSecond() - dt.toEpochSecond() < 1000);


It looks like we are dropping from millisecond to second precision here.

kmcginnes · 2025-04-04T23:16:29Z

gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/Translator.java

@@ -310,6 +316,9 @@ protected Script convertToScript(final Object object) {
                } else if (object instanceof Date) {
                    final Object objectOrWrapper = withParameters ? object : getSyntax((Date) object);
                    return script.getBoundKeyOrAssign(withParameters, objectOrWrapper);
+                } else if (object instanceof OffsetDateTime) {
+                    final Object objectOrWrapper = withParameters ? object : getSyntax((OffsetDateTime) object);
+                    return script.getBoundKeyOrAssign(withParameters, objectOrWrapper);


Now that this exists, should the else if (object instanceof Date) above be removed?

I believe it should stay because both Date and OffsetDateTime types are still supported, just now the default is OffsetDateTime.

andreachild · 2025-04-07T16:42:32Z

.../main/java/org/apache/tinkerpop/gremlin/language/translator/AnonymizedTranslatorVisitor.java

@@ -23,6 +23,7 @@

 import java.math.BigDecimal;
 import java.math.BigInteger;
+import java.time.OffsetDateTime;
 import java.util.Date;


This import is now unused

andreachild · 2025-04-07T16:43:59Z

...e/src/main/java/org/apache/tinkerpop/gremlin/language/translator/DotNetTranslateVisitor.java

@@ -27,6 +27,7 @@
 import org.apache.tinkerpop.gremlin.structure.VertexProperty;
 import org.apache.tinkerpop.gremlin.util.DatetimeHelper;

+import java.time.OffsetDateTime;
 import java.util.Date;


This import is now unused

andreachild · 2025-04-07T18:16:01Z

...-core/src/main/java/org/apache/tinkerpop/gremlin/language/translator/GoTranslateVisitor.java

@@ -27,6 +27,7 @@
 import org.apache.tinkerpop.gremlin.structure.VertexProperty;
 import org.apache.tinkerpop.gremlin.util.DatetimeHelper;

+import java.time.OffsetDateTime;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Date;


This import is now unused

andreachild · 2025-04-07T18:16:53Z

...ore/src/main/java/org/apache/tinkerpop/gremlin/language/translator/JavaTranslateVisitor.java

@@ -25,6 +25,7 @@
 import org.apache.tinkerpop.gremlin.structure.util.reference.ReferenceVertex;
 import org.apache.tinkerpop.gremlin.util.DatetimeHelper;

+import java.time.OffsetDateTime;
 import java.util.Date;


This import is now unused

andreachild · 2025-04-07T18:30:47Z

...e/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.java

@@ -204,6 +204,7 @@
 import org.apache.tinkerpop.gremlin.structure.VertexProperty;
 import org.apache.tinkerpop.gremlin.util.function.ConstantSupplier;

+import java.time.OffsetDateTime;


Date import is now unused

andreachild · 2025-04-07T18:32:27Z

gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/__.java

@@ -37,6 +37,7 @@
 import org.apache.tinkerpop.gremlin.structure.Vertex;
 import org.apache.tinkerpop.gremlin.structure.VertexProperty;

+import java.time.OffsetDateTime;
 import java.util.Collection;
 import java.util.Date;


andreachild · 2025-04-07T18:50:07Z

...n-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/AsDateStep.java

-            return (Date) object;
+            throw new IllegalArgumentException("Can't parse null as OffsetDateTime.");
+        if (object instanceof OffsetDateTime)
+            return (OffsetDateTime) object;
        if (object instanceof Byte || object instanceof Short || object instanceof Integer || object instanceof Long)
            // numbers handled as milliseconds since January 1, 1970, 00:00:00 GMT.


Comment should be updated to UTC instead of GMT which are slightly different.

andreachild · 2025-04-07T18:53:16Z

...n-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/AsDateStep.java

        final Object object = traverser.get();
        if (object == null)
-            throw new IllegalArgumentException("Can't parse null as Date.");
-        if (object instanceof Date)


Technically since Date types are supported isn't it possible to use the asDate step on a traversal that produces Date? In that case should the Date be converted to OffsetDateTime?

Yes, this is part of the change I'll revert along with other changes to make it compatible with Date as type.

andreachild · 2025-04-07T19:45:27Z

...core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/DateDiffStep.java

@@ -25,6 +25,7 @@
 import org.apache.tinkerpop.gremlin.process.traversal.util.TraversalUtil;
 import org.apache.tinkerpop.gremlin.structure.util.StringFactory;

+import java.time.OffsetDateTime;
 import java.util.Collections;
 import java.util.Date;


andreachild · 2025-04-07T21:50:04Z

...core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/DateDiffStep.java


-        return (((Date) object).getTime() - otherDateMs) / 1000;
+        return (((OffsetDateTime) object).toEpochSecond() - otherDateMs);


Suggestion:

return otherDate == null ? 0L : Duration.between(otherDate, (OffsetDateTime) object).getSeconds();

I like the simplification, not quite the right logic but I can use it, i.e. if otherDate is null we consider it as 0L and return the epoch seconds of date, instead of returning 0L.

andreachild · 2025-04-07T22:38:57Z

gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/util/DatetimeHelper.java

@@ -22,7 +22,11 @@
 import java.time.LocalDate;
 import java.time.LocalDateTime;
 import java.time.LocalTime;
+import java.time.Month;
+import java.time.OffsetDateTime;
+import java.time.Year;
 import java.time.YearMonth;


Unused imports

andreachild · 2025-04-07T23:27:11Z

gremlin-dotnet/src/Gremlin.Net/Structure/IO/GraphBinary/DataType.cs

@@ -78,6 +78,7 @@ public class DataType : IEquatable<DataType>
        // TODO: Support metrics and traversal metrics
        public static readonly DataType Char = new DataType(0x80);
        public static readonly DataType Duration = new DataType(0x81);
+        public static readonly DataType OffsetDateTime = new DataType(0x88);


Just wondering why OffsetDateTime was not previously supported as a type for .NET? This PR does not add the type to TInkerPop, only changes the default date type.

Not just .NET, it actually wasn't supported in any of the GLVs because it's an extended serialization type.

andreachild · 2025-04-08T00:28:28Z

gremlin-dotnet/src/Gremlin.Net/Structure/IO/GraphBinary/Types/OffsetDateTimeSerializer.cs

+            var s = value.Second;
+            var ms = value.Millisecond; 
+            // Note there will be precision loss as microsecond and nanosecond access was added after .net 7
+            var ns = h * 60 * 60 * 1e9 + m * 60 * 1e9 + s * 1e9 + ms * 1e6;


Q says obtaining nanoseconds is possible using Ticks:

// Convert the time portion of DateTimeOffset to nanoseconds since midnight static long TimeToNanosecondsSinceMidnight(DateTimeOffset dto) { // Get the time of day as TimeSpan TimeSpan timeOfDay = dto.TimeOfDay; // Convert ticks to nanoseconds (1 tick = 100 nanoseconds in .NET) return timeOfDay.Ticks * 100; }

…e Date as parameter to dateDiff, update date precision in GLV translators.

Backport OffsetDateTime changes from Java/Python on 4.0 where OffsetD…

c515c6d

…ateTime is now the default Date type in Gremlin; added OffsetDateTime serializers to Go/JS/.NET. Made OffsetDateTime the default serializer for date types in GLVs (Date type will only deserialize).

Cole-Greer reviewed Apr 4, 2025

View reviewed changes

...ain/java/org/apache/tinkerpop/gremlin/process/traversal/translator/JavascriptTranslator.java Outdated Show resolved Hide resolved

Cole-Greer reviewed Apr 4, 2025

View reviewed changes

kmcginnes reviewed Apr 4, 2025

View reviewed changes

andreachild reviewed Apr 7, 2025

View reviewed changes

andreachild reviewed Apr 8, 2025

View reviewed changes

Retain ability to manipulate java.until.Date for date steps, deprecat…

de16613

…e Date as parameter to dateDiff, update date precision in GLV translators.

	assertTrue(OffsetDateTime.now(UTC).toEpochSecond() - dt.toEpochSecond() < 1000);
	assertTrue(OffsetDateTime.now(UTC).toInstant().toEpochMilli() - dt.toInstant().toEpochMilli() < 1000);


		return (((Date) object).getTime() - otherDateMs) / 1000;
		return (((OffsetDateTime) object).toEpochSecond() - otherDateMs);

Backport OffsetDateTime changes from 4.0 #3090

Are you sure you want to change the base?

Backport OffsetDateTime changes from 4.0 #3090

Conversation

xiazcy commented Apr 4, 2025 • edited Loading

Cole-Greer Apr 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiazcy commented Apr 4, 2025 •

edited

Loading

Cole-Greer Apr 4, 2025 •

edited

Loading