Portable Time type (Java + model changes) - #39804
Conversation
Adds TIME = 10 to the LogicalTypes enum with URN beam:logical_type:time:v1, registers Time in SchemaTranslation's standard logical types, and derives Time.IDENTIFIER from the enum rather than hardcoding the string. Mirrors the portable Date type added in apache#38077. As there, the Python side follows in a separate PR so the URN reaches a snapshot container before the cross-language tests need it.
|
In practice a "portable" type effectively defined an "abstract" type asking each SDK to implement. In general |
|
That is a fair bar, and I would rather the model stay small too. Let me put the specific numbers here, because I think this case is narrower than adding a new abstract type.
So today Java occupies the portable namespace for this URN, emits it with an empty payload — and cannot read its own output back. Any SDK receiving it has a URN that looks standard, is documented nowhere, and carries nothing to reconstruct from. That is what makes me read this as documenting an obligation the Java SDK already imposes, rather than creating one. The enum entry costs other SDKs nothing they are not already paying: an SDK that does not implement If the preference is the other direction, the consistent alternative is to stop squatting on the portable namespace — change Either way the present state seems worth not leaving alone. Whichever you prefer, I will follow. |
|
Assigning reviewers: R: @chamikaramj for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
This type may not be a straightforward redo of the Date type, since conceptually it can be described by different precisions. Using nanosecond precision makes sense today, but one could argue it's arbitrary, and future applications may require a higher precision. Check out the timestamp logical type (design doc + Java implementation + Python implementation), where we include a "precision" parameter. I think it makes sense to do something similar here. Python already has a custom Timestamp type. I think we can do with a new custom Time type that behaves similarly, but others may have opinions. I think at the very least, this topic deserves a discussion in the mailing dev list. Feel free to send a message there so others can pitch in |
|
That is a better framing than the one I brought, and I am happy to redo it that way. Having read through Precision as an argument, same as Representation is where it need not follow. The tradeoff is symmetry against simplicity: a One useful property either way. With On Python: agreed that a custom type mirroring the I will start the dev@ thread with the above and link back here. Meanwhile I will hold this PR rather than push a half-converted version — happy to close and reopen against whatever the list settles on, if you would prefer that to leaving it open. |
Adds the portable
Timelogical type, so aLocalTimefield can cross an SDK boundary the wayDatecan since #38077.Timealready exists in the Java SDK but is not portable: its identifier is a hardcoded string that no entry inLogicalTypes.Enumbacks, and it is absent fromSchemaTranslation.STANDARD_LOGICAL_TYPES. A schema arriving from another SDK carryingbeam:logical_type:time:v1therefore comes back asUnknownLogicalTyperather than asTime.This is a direct mirror of #38077:
schema.proto—TIME = 10, representationINT64, nanoseconds since midnight, matching whatTime.toBaseTypealready produces;SchemaTranslation— registerTime.IDENTIFIER -> Time.class;Time.java— takeIDENTIFIERfrom the enum, and returnnullfromgetArgumentType()instead of theSTRING/""pair that was marked// unused.Following the same split @Abacn suggested on #37830, the Python changes go in a separate PR so the new URN is in a snapshot container before the cross-language tests need it.
JdbcUtilis deliberately left out too: theDatePR added aDate.valueOf(LocalDate)setter, but theTimeequivalent,Time.valueOf(LocalTime), silently truncates sub-second precision, and this type's whole base representation is nanoseconds. That deserves its own change rather than a copy of theDateline.Testing
SchemaTranslationTest— 89 tests, 0 failures.spotlessJavaCheck,checkstyleMainandcheckstyleTestare clean.The two entries added to the existing parameterised lists are not on their own enough, and it is worth saying why.
LogicalTypesTest.testLogicalTypeFromToProtoCorrectlybranches on the registry it is meant to be exercising:Drop the
SchemaTranslationregistration and theelsebranch simply takes over — the suite stays green. I checked: with that one line removed, all 87 pre-existing tests still passed.So this adds
PortableLogicalTypeFromUrnTest, which asserts the recovery directly and consults nothing:With the registration removed this fails as it should —
— while its
Datesibling stays green, so the failure is the missing registration and not the test class itself.Addresses part of #25946. @ahmedabu98 you offered to review this one back in March — PTAL.