GH-50901: [C++] Replace RapidJSON with simdjson in tensor extension types by Reranko05 · Pull Request #50874 · apache/arrow

Reranko05 · 2026-08-15T10:43:02Z

Rationale for this change

This PR continues the simdjson migration by replacing RapidJSON usage in the FixedShapeTensorType and VariableShapeTensorType extension types with simdjson's DOM API.

Changes

Replace RapidJSON parsing in FixedShapeTensorType::Deserialize with simdjson.
Replace RapidJSON parsing in VariableShapeTensorType::Deserialize with simdjson.
Centralize the reusable simdjson DOM helpers in arrow/util/simdjson_internal.h.
Add helpers for parsing JSON objects, retrieving optional fields, and validating JSON arrays and their element types.
Remove the obsolete RapidJSON-specific helpers and includes from the tensor extension utilities.
Preserve existing deserialization behavior and error validation, including the ordering of metadata-level validation errors.
Update deserialization tests to match simdjson JSON type names.

GitHub Issue: [C++] Replace RapidJSON with simdjson in tensor extension types #50901

github-actions · 2026-08-15T10:43:23Z

Thanks for opening a pull request!

This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format.

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

After updating the title, you can mark the pull request as ready for review.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

pitrou

This looks much better now!

pitrou · 2026-08-25T07:52:49Z

@HuaHuaY @rok Do you want to give this another look?

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

rok

Other then the BIGINT - LGTM!

HuaHuaY

Looks good to me.

pitrou · 2026-08-25T13:04:39Z

@github-actions crossbow submit -g cpp

github-actions · 2026-08-25T13:07:33Z

Revision: 6614716

Submitted crossbow builds: ursacomputing/crossbow @ actions-f28d5d23af

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

pitrou · 2026-08-25T13:47:36Z

CI failures are unrelated - except for example-cpp-tutorial which was already failing.

rok · 2026-08-25T13:50:01Z

CI failures are unrelated - except for example-cpp-tutorial which was already failing.

pinged #35460 (comment)

Migrate FixedShapeTensorType deserialization to simdjson

9c5ff53

github-actions Bot added Component: C++ awaiting review Awaiting review labels Aug 15, 2026

Reranko05 added the CI: Extra: C++ Run extra C++ CI label Aug 15, 2026

Replace RapidJSON with simdjson in VariableShapeTensor

3332bfb

Reranko05 changed the title ~~[WIP] GH-50873: [C++] Migrate FixedShapeTensorType deserialization to simdjson~~ GH-50901: [C++] Replace RapidJSON with simdjson in tensor extension types Aug 18, 2026

Reranko05 marked this pull request as ready for review August 18, 2026 10:57

Reranko05 requested a review from pitrou as a code owner August 18, 2026 10:57

Copilot AI lite review requested due to automatic review settings August 18, 2026 10:57

Copilot AI reviewed Aug 18, 2026

Reranko05 requested review from kou and rok August 18, 2026 10:57

Remove obsolete RapidJSON tensor helper

bbcaaec

Copilot AI review requested due to automatic review settings August 18, 2026 18:49

Copilot AI reviewed Aug 18, 2026

kou reviewed Aug 18, 2026

View reviewed changes

Comment thread cpp/src/arrow/extension/fixed_shape_tensor.cc Outdated

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Aug 18, 2026

Address Feedback

ec82dd0

Copilot AI review requested due to automatic review settings August 19, 2026 07:48

Copilot AI reviewed Aug 19, 2026

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 19, 2026

Reranko05 requested a review from kou August 19, 2026 07:49

rok reviewed Aug 19, 2026

View reviewed changes

Comment thread cpp/src/arrow/extension/tensor_extension_array_test.cc

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Aug 19, 2026

kou reviewed Aug 19, 2026

View reviewed changes

Comment thread cpp/src/arrow/extension/fixed_shape_tensor.cc Outdated

Comment thread cpp/src/arrow/extension/tensor_extension_array_test.cc

Copilot AI review requested due to automatic review settings August 20, 2026 08:32

github-actions Bot added the awaiting change review Awaiting change review label Aug 24, 2026

rok requested a balanced review from Copilot August 24, 2026 18:10

Copilot started reviewing on behalf of rok August 24, 2026 18:11 View session

Remove unused helper

9f4cf1a

Copilot AI reviewed Aug 24, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/simdjson_internal.h

Copilot AI review requested due to automatic review settings August 24, 2026 18:14

Copilot AI reviewed Aug 24, 2026

Reranko05 requested a review from pitrou August 24, 2026 18:57

pitrou reviewed Aug 25, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/simdjson_internal.h Outdated

pitrou mentioned this pull request Aug 25, 2026

GH-50936: [C++][Integration] Replace RapidJSON with simdjson #50937

Merged

Update cpp/src/arrow/util/simdjson_internal.h

6614716

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Copilot AI review requested due to automatic review settings August 25, 2026 10:23

Copilot AI reviewed Aug 25, 2026

rok reviewed Aug 25, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/simdjson_internal.h Outdated

rok approved these changes Aug 25, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes awaiting merge Awaiting merge and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Aug 25, 2026

HuaHuaY approved these changes Aug 25, 2026

View reviewed changes

HuaHuaY reviewed Aug 25, 2026

View reviewed changes

Comment thread cpp/src/arrow/extension/variable_shape_tensor.cc

github-actions Bot removed the awaiting merge Awaiting merge label Aug 25, 2026

fallback BIGINT to return unknown

2a29050

Copilot AI reviewed Aug 25, 2026

pitrou mentioned this pull request Aug 25, 2026

[C++] Replace RapidJSON with simdjson in tensor extension types #50901

Closed

Uh oh!

Conversation

Reranko05 commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Changes

Uh oh!

github-actions Bot commented Aug 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rok left a comment

Choose a reason for hiding this comment

Uh oh!

HuaHuaY left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

rok commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Reranko05 commented Aug 15, 2026 •

edited

Loading