Initial Implementation of the Parquet Writer - #215

Merged

mchav merged 40 commits into

DataHaskell:mainfrom

sharmrj:parquet-writer

Aug 30, 2026

Commits on Aug 5, 2026

WIP: Initial sketch of the parquet writer design
sharmrj
committed
85c6e4a

Commits on Aug 18, 2026

WIP Parquet Writer (does not compile)
sharmrj
committed
89d7b6d

Commits on Aug 19, 2026

Implemented HasBuffer instance for the file backed buffer; Cleaned up imports and exports
sharmrj
committed
e8c2b85
Tests for `HasBuffer` instances
sharmrj
committed
c3fc6c3
Changes to make Writer.hs compile so we can run tests
sharmrj
committed
f292f51
fix conflicts with main
sharmrj
committed
00f105b
Fixed ensureCapacity so that it works correctly with pinned ByteArrays and added some helper functions.
sharmrj
committed
167ab08
implement a cleaned up Parquet Writer
sharmrj
committed
6fd2e80
added some comments
sharmrj
committed
f76f518
added benchmarks and a stress test
sharmrj
committed
ad47e3b
Optimized hotspots
sharmrj
committed
237a139
ran fourmolu
sharmrj
committed
995fc7b
hlint
sharmrj
committed
74ff879
fourmolu again
sharmrj
committed
ddab22c
parquet.thrift was committed by accident
sharmrj
committed
bb46712
made sure test parquet files can be recognized in CI.
sharmrj
committed
db19108
Changed dataframe-parquet.cabal after the recommendations from CI's cabal check
sharmrj
committed
31b054a

Commits on Aug 20, 2026

Commits on Aug 21, 2026

Commits on Aug 23, 2026

Writer Refactor WIP
sharmrj
committed
c486e02
removed dead code (refactor WIP)
sharmrj
committed
bf8f486
Refactored the Parquet Writer for readability
sharmrj
committed
c38f2fa
Merge remote-tracking branch 'refs/remotes/upstream/main' into parquet-writer
sharmrj
committed
af73988
Fixed a compile issue after pulling from main
sharmrj
committed
09e0ce0
Reverted changes to the stress test
sharmrj
committed
bddd38e
Fixed a `Bitmap` import
sharmrj
committed
80dd2f3

Commits on Aug 24, 2026

`writeByteStringToFile` no longer instantiates a new buffer.
sharmrj
committed
4d37dc0

Commits on Aug 30, 2026

Added `Int` and `Integer` cases to `buildEncoder`
sharmrj
committed
6344c23
Added `INLINE` pragmas
sharmrj
committed
f130410
Limit `Integer` values fed into the Encoder to the `Int64` range
sharmrj
committed
29083f9
Remove `writeInteger64` from Encoder.hs
sharmrj
committed
a6b6609
Write to a temp file and rename when the whole file is written successfully
sharmrj
committed
682b5ef
Merge remote-tracking branch 'refs/remotes/upstream/main' into parquet-writer
sharmrj
committed
e894427
Fixed dataframe-core bounds on the benchmarks in dataframe-parquet.cabal
sharmrj
committed
4aabd8e
Fixed PackedText import after merging `main` into `parquet-writer`
sharmrj
committed
66281ba
Fix build failure
sharmrj
committed
208390d
small fix
sharmrj
committed
38d42a3