Repository navigation
feat: IPTC domain taxonomy, TOC keywords, and image region metadata - #464
Merged
Merged
Conversation
- Added support for extra metadata in the MarkdownParseState to store image region information. - Updated the append_row method to accept additional metadata and store it in a new dictionary. - Modified the parse_md function to include image region metadata when processing markdown content. - Integrated image region loading in the PDF parsing workflow to ensure accurate page number mapping after TOC removal. - Adjusted tests to validate the new metadata structure and ensure consistency across chunk and zip formats.
… parsing - Introduced `extra_metadata` parameter in `build_image_asset_row` to accommodate additional image region information. - Updated `ParsedRow` and related methods to handle the new `extra_metadata` field. - Modified markdown and pdf parsing functions to integrate image region metadata, ensuring accurate representation in output. - Adjusted tests to validate the inclusion of `extra_metadata` in various parsing scenarios.
…oved metadata - Updated `DocumentProfile` to support a list of categories and an additional `category_label` for better classification. - Modified the coarse profiling process to extract domain IDs and category labels from input data, ensuring accurate document classification. - Enhanced the parsing instructions to reflect the new structure, requiring domain IDs and category labels for document classification. - Adjusted related functions and tests to accommodate the new multi-category and label features, ensuring consistency across document types.
- Changed document profiling to utilize IPTC Media Topics instead of ISIC codes for categorization. - Updated `DocumentProfile` and related parsing instructions to reflect the new category structure. - Adjusted coarse profiling prompts and functions to align with the IPTC classification system. - Modified tests to validate the new category codes and ensure consistency across document types.
Move exhibit index to the global line-start keyword set, and keep finance-only index to/of prefixes for SEC-style statement indexes. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
categorywith IPTC Media Topics top-level codes (multi-select) pluscategory_label, with invalid picks falling back toOTHER.exhibit index; finance-onlyindex to/index of.Test plan
make check(ruff + pyright)Made with Cursor