diff --git a/docs/API/errors.md b/docs/API/errors.md new file mode 100644 index 0000000..17123d0 --- /dev/null +++ b/docs/API/errors.md @@ -0,0 +1,5 @@ +--- +icon: lucide/triangle-alert +--- + +::: dstrack.errors diff --git a/docs/getting_started.md b/docs/getting_started.md index 71d8e76..813ffd5 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -129,22 +129,16 @@ dataset's path, and `dstrack` works out which dataset that is: dstrack log data.csv ``` -```text -● a587afd9 Customers HEAD -│ 2 minutes ago by you -│ 4 rows +1 3 cols -│ data.csv -│ -● 605eb940 Customers -│ 5 minutes ago by you -│ 3 rows +1 3 cols -│ data.csv -│ -● 0c82297f Customers - 1 hour ago by you - 2 rows 3 cols - data.csv -``` +
● e9d6f0d7 sample HEAD
+│ 1 week ago by menjivar
+│ 54 rows +4 8 cols +1
+│ data/sample.csv
+│
+● e83517d4 sample
+ 1 week ago by menjivar
+ 50 rows 7 cols
+ data/sample.csv
+
Each node is one snapshot, showing how its row and column counts changed from the
snapshot below it. `HEAD` marks the latest.
@@ -226,48 +220,83 @@ for column in result.columns:
print(column.name, column.distribution_shift.score)
```
-??? example "Exemplary bash diff output"
- ```bash title="Command lind output"
- ✔ Wrote the comparison report to report.pdf
- ╭────────────────── dstrack diff ───────────────────╮
- │ before after │
- │ snapshot 0d86445c 6f6577b3 │
- │ name sample sample │
- │ path data/sample.csv data/sample.csv │
- │ taken 1 minute ago 55 seconds ago │
- │ by menjivar menjivar │
- │ rows 50 54 │
- │ columns 7 8 │
- │ source 2dfa4d9c36ae… 5887d2bc5a31… changed │
- ╰─────────────────── CRITICAL ────────────────────╯
-
- Schema
- column before after
- ────────────────────────────────────────────────
- + country — string not null
- ~ age int64 not null int64 null
-
- Dataset
- • num rows 50 54 +4 (+8.0%)
- ⚠ num columns 7 8 +1 (+14.3%)
-
- Columns
- Numeric columns
- column nulls mean std p50 min max uniq shift
- ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
- ⚠ id 0 25.5 → 28.11 14.43 → 16.31 25.5 → 27.5 1 50 → 56 50 → 54 PSI 0.23
- ⚠ age 0 → 0.09259 44.14 → 45.37 18.31 → 18.94 37.5 → 39 18 75 33 → 31 PSI 0.15
- ❗ balance 0 1,235 → 2,213 684.5 → 1,180 1,226 → 2,192 97.28 → 170.2 2,403 → 4,202 50 → 54 PSI 1.09
- • is_active 0 0.5 0.5 0.5 0 1 2 PSI 0.00
- Text columns
- column nulls uniq coverage avg len shift
- ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
- • name 0 28 1 13.88 → 13.93 TVD 0.10 +Margaret Hopper -Katherine Torvalds
- ⚠ plan 0 3 1 5.9 → 7 TVD 0.18
- • country (new) — → 0 — → 5 — → 1 — → 2 —
-
- +4 rows +1 columns 4 drifted
- ```
+╭────────────────── dstrack diff ───────────────────╮
+│ before after │
+│ snapshot e83517d4 e9d6f0d7 │
+│ name sample sample │
+│ path data/sample.csv data/sample.csv │
+│ taken 1 week ago 1 week ago │
+│ by menjivar menjivar │
+│ rows 50 54 │
+│ columns 7 8 │
+│ source 2dfa4d9c36ae… 1746ddc049a2… changed │
+╰─────────────────── CRITICAL ────────────────────╯
+
+Schema
+ column before after
+─────────────────────────────────────────────────────────
+ + country — string not null
+ ~ age int64 not null int64 null
+ ~ signup_date datetime64 not null datetime64 null
+
+Dataset
+• num rows 50 54 +4 (+8.0%)
+⚠ num columns 7 8 +1 (+14.3%)
+
+Column sets
+⚠ constant columns +is_active
+⚠ high null columns +signup_date
+
+Columns
+ Numeric columns
+ column nulls mean std p50 min max uniq shift
+────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
+ ⚠ id 0 25.5 → 29.22 14.43 → 16.76 25.5 → 29.5 1 50 → 57 50 → 54 PSI 0.22
+ ⚠ age 0 → 0.09259 44.14 → 44.94 18.31 → 17.45 37.5 → 39 18 75 33 → 35 PSI 0.10
+ ❗ balance 0 1,235 → 1,678 684.5 → 920.2 1,226 → 1,748 97.28 → 80.57 2,403 → 3,190 50 → 54 PSI 0.84
+ ❗ is_active 0 0.5 → 1 0.5 → 0 0.5 → 1 0 → 1 1 2 → 1 pctl 1.00
+ Text columns
+ column nulls uniq coverage avg len shift
+─────────────────────────────────────────────────────────────────────────────────────────────────────
+ • name 0 28 → 30 1 13.88 → 13.83 TVD 0.08 +Ada Turing +Alan Johnson
+ ⚠ plan 0 3 1 5.9 → 6.944 TVD 0.16
+ • country (new) — → 0 — → 5 — → 1 — → 2 —
+ Datetime columns
+ column nulls range (d)
+───────────────────────────────────────────
+ ❗ signup_date 0 → 0.7407 360 → 353
+
+Other column changes
+• id percentiles.p5 3.45 → 3.65 +0.2 (+5.8%)
+• id percentiles.p25 13.25 → 14.5 +1.25 (+9.4%)
+⚠ id percentiles.p75 37.75 → 43.75 +6 (+15.9%)
+⚠ id percentiles.p95 47.55 → 54.35 +6.8 (+14.3%)
+⚠ id percentiles.p99 49.51 → 56.47 +6.96 (+14.1%)
+⚠ age null_count 0 → 5 +5
+• age percentiles.p5 20.9 → 20.8 -0.1 (-0.5%)
+• age percentiles.p25 29 → 30 +1 (+3.4%)
+• age percentiles.p75 60.75 → 59 -1.75 (-2.9%)
+⚠ balance percentiles.p5 154.1 → 179.6 +25.58 (+16.6%)
+❗ balance percentiles.p25 601.7 → 992.7 +391 (+65.0%)
+⚠ balance percentiles.p75 1,828 → 2,504 +676 (+37.0%)
+⚠ balance percentiles.p95 2,237 → 2,965 +727.2 (+32.5%)
+⚠ balance percentiles.p99 2,385 → 3,164 +779.3 (+32.7%)
+⚠ is_active percentiles.p5 0 → 1 +1
+⚠ is_active percentiles.p25 0 → 1 +1
+• country null_count — → 0
+• country min_char_length — → 2
+• country max_char_length — → 2
+• country avg_token_count — → 1
+• country min_token_count — → 1
+• country max_token_count — → 1
+⚠ signup_date null_count 0 → 40 +40
+• signup_date min 2023-01-02 00:00:00 → 2023-01-09 00:00:00 +6.048e+05 (+0.0%)
+
++4 rows +1 columns 6 columns drifted +1 constant columns +1 high null columns
+
+