Repository navigation
Avoid temporary serialization for shared-string lookups - #9410
kunaldrafts25 wants to merge 2 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
There was a problem hiding this comment.
Code Review
This pull request optimizes FlatBufferBuilder::CreateSharedString by utilizing heterogeneous lookup (__cpp_lib_generic_associative_lookup) when available, which prevents temporary serialization and unnecessary buffer growth for duplicate strings. It also includes new benchmarks and unit tests to validate the performance and correctness of this optimization. The feedback suggests using str->data() instead of str->c_str() in the transparent comparison operators to maintain consistency with the existing codebase.
| bool operator()(const Offset<String>& a, const StringLookup& b) const { | ||
| auto str = reinterpret_cast<const String*>(buf_->data_at(a.o)); | ||
| return StringLessThan(str->c_str(), str->size(), b.data, b.size); | ||
| } | ||
| bool operator()(const StringLookup& a, const Offset<String>& b) const { | ||
| auto str = reinterpret_cast<const String*>(buf_->data_at(b.o)); | ||
| return StringLessThan(a.data, a.size, str->c_str(), str->size()); | ||
| } |
There was a problem hiding this comment.
For consistency with the existing operator()(const Offset<String>&, const Offset<String>&) which uses stra->data(), please use str->data() instead of str->c_str() in the transparent comparison operators.
bool operator()(const Offset<String>& a, const StringLookup& b) const {
auto str = reinterpret_cast<const String*>(buf_->data_at(a.o));
return StringLessThan(str->data(), str->size(), b.data, b.size);
}
bool operator()(const StringLookup& a, const Offset<String>& b) const {
auto str = reinterpret_cast<const String*>(buf_->data_at(b.o));
return StringLessThan(a.data, a.size, str->data(), str->size());
}
CreateSharedString currently serializes every input before checking its offset pool. On a duplicate, it discards the newly written bytes, but any buffer growth has already happened. For example, a repeated 1 KiB string in a nearly full builder increases the buffer allocation count from one to three even though the returned offset and final buffer size are unchanged.
Use heterogeneous lookup when the standard library supports it (C++14 and newer), comparing the input bytes directly against stored strings. Return the existing offset on a hit, and reuse the lower_bound insertion hint on a miss. Keep the existing implementation for C++11 libraries. Preserve the nesting assertion, exact-length comparison, and empty-input handling.
Add an allocation regression test and benchmarks for duplicate and unique strings. The test also covers empty inputs, prefixes, buffer growth, and reuse after Clear; existing shared-string tests cover embedded NULs.
Validation: