Skip to content

[C API] Implement memory estimation functions for index build and search - #364

Open
rfsaliev wants to merge 6 commits into
mainfrom
rfsaliev/c-api-estimate-index-size
Open

[C API] Implement memory estimation functions for index build and search#364
rfsaliev wants to merge 6 commits into
mainfrom
rfsaliev/c-api-estimate-index-size

Conversation

@rfsaliev

@rfsaliev rfsaliev commented Aug 13, 2026

Copy link
Copy Markdown
Member

This pull request introduces a set of enhancements to the C API and internal implementation to support memory usage estimation for index building and search operations. The primary focus is on adding new estimation functions, implementing the underlying logic for various data builders, and integrating these features with both static and dynamic Vamana index types. The changes also include updates to the build system to incorporate new source files.

New C API for Memory Estimation:

  • Added several new API functions to estimate memory usage for index building and search operations, including support for both static and dynamic indices, as well as search queries. These functions allow users to obtain detailed memory breakdowns before performing expensive operations.

Implementation of Estimation Logic:

  • Implemented estimate_size methods for all major data builders (SimpleDataBuilder, LVQDataBuilder, LeanVecDataBuilder, SQDataBuilder) to provide accurate memory usage estimation based on builder configuration, number of vectors, and dimension. [1] [2] [3] [4]
  • Created data_builder.cpp and corresponding header to dispatch and specialize estimation logic for different data types and block sizes. [1] [2]

Integration with Vamana Index Types:

  • Added dispatcher functions to estimate memory usage for both static and dynamic Vamana indices, providing structured breakdowns of graph, data, and metadata memory requirements. [1] [2] [3] [4]

Build System and Dependency Updates:

  • Updated CMakeLists.txt to include the new data_builder.cpp source file, ensuring the new logic is compiled and linked.
  • Added necessary includes and dependencies in implementation files to support the new estimation features. [1] [2] [3]

Summary of Most Important Changes:

1. API Additions

  • Added new C API functions for estimating memory usage of index building and search operations, including dynamic and static variants.

2. Memory Estimation Logic

  • Implemented estimate_size methods for all major data builders to enable accurate memory usage estimation. [1] [2] [3] [4]
  • Created data_builder.cpp and header to centralize and dispatch estimation logic for various data types/blocking strategies. [1] [2]

3. Vamana Index Integration

  • Added dispatcher functions for memory estimation of static and dynamic Vamana indices, including detailed breakdowns. [1] [2] [3] [4]

4. Build and Dependency Updates

  • Included new source files in the build system and updated includes for proper compilation and linkage. [1] [2] [3] [4]

@rfsaliev

Copy link
Copy Markdown
Member Author

This PR contains number of workarounds/TODOs which should be resolved out-of the bindings/c directory.
To avoid potential conflicts, keeping the PR as 'Draft' until getting bindings/c on the main branch.

@rfsaliev
rfsaliev force-pushed the dev/c-api branch 2 times, most recently from e0cbd2f to c57d2ad Compare August 24, 2026 10:27
Base automatically changed from dev/c-api to main August 24, 2026 12:01
@rfsaliev
rfsaliev force-pushed the rfsaliev/c-api-estimate-index-size branch from cc12653 to 1d381ec Compare August 24, 2026 12:05
@rfsaliev
rfsaliev marked this pull request as ready for review August 25, 2026 13:28
@rfsaliev

Copy link
Copy Markdown
Member Author

@klmckeig, please review this PR in accordance to #333

@rfsaliev rfsaliev changed the title [C API] Implement memory estimation functions for index builders and tests [C API] Implement memory estimation functions for index build and search Aug 27, 2026
@rfsaliev
rfsaliev requested a review from yuejiaointel August 27, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C API] Add memory estimation functions for Vamana index build and search

1 participant