Repository navigation
Conversation
A source column's name also no longer breaks its profile.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A field with a declared type now reads each cell as that type, deterministically, by what is declared and never by the
machine's locale, and a transform either conforms to its destination schema or is not built. This replaces
coerce_to_schemaand the pandas-era value parsers.What changes
ReadingParser, in Polars, reads numbers, integers, booleans, strings, dates, times,datetimes, periods, and durations. Each cell is a value, a blank, or a refusal with its reason.
policyholds CSV on the Web'sbaseandformat, withwhyqd's own
rulesbeside them. A date pattern may be typed as Excel,strptime, or UAX 35 writes it, and is savedas UAX 35's.
£5,(5),5%,1,234.50,1e3,13/04/2024,1 April 2024, and a bare45295in a date field. Each warning says what was read, and declaring it confirms thereading and tightens the field.
01/02/2024,1,234,24as a year,<5,TBC. A column never votes, so one cellnever settles another.
report, raising
ConformanceError, aValueError.process(strict=True)treats each warning as a fault.CATEGORISEassigningTruetakes any value of the field's type. A blank is neverpresent, and a zero only where
zeroAsTrue, set byset_categories(..., zero_as_true=True), counts it.whyqd/config/names.json, so no library or machinechanges a read.
Breaking changes
coerce_to_schemaand theDataSourceParservalue parsers (parse_dates,parse_float, and the rest) are removed.what halted.
TransformDefinition.reportsays what and where.usdateupgrades to adatewith month-first patterns, and saves back to 1.x as it was.month,quarter, oryearreads as the date it ends on.Tested
tests/test_reading.pyreads every case intests/data/synthetic-typing/typing-key.json, 427 of them, and converts13 synthetic files written as different locales' publishers write, under three machine locales.
tests/test_conformance.pycovers halting, warnings, constraints, actions, presence, and a stored sample.The wrangler's account is
docs/strategies/reading.md.