[qilitang]

Summary

• stream ORC and Parquet source files without applying text line delimiters or UTF-8 replacement
• keep every binary source file intact in its own Stream Load request while using workers for file-level concurrency
• preserve retry selection, LZ4 compression, FE redirects, progress reporting, and per-file load results
• document the binary format behavior and add regression coverage

Problem

The CLI forwards format:orc and format:parquet to Doris, but the local reader still processes those files with ReadBytes(lineDelimiter) and enables UTF-8 validation by default. Binary columnar files can therefore be modified by replacement characters, rejected at EOF when they do not end in a text delimiter, or split across independent Stream Load requests.

This is the same root cause reported in #34 for Parquet and also affects ORC.

Implementation

ORC and Parquet now use a file-streaming path:

• one complete source file per Stream Load request
• bounded concurrency across files, never within a file
• no UTF-8 inspection or text batching
• streaming LZ4 compression when requested
• retry bookkeeping remains keyed by the source file index

CSV and other text-format behavior remains on the existing batching path.

Verification

• go test ./... -count=1
• go test -race ./... -count=1
• go vet ./...
• go build

Regression tests verify byte-for-byte preservation, per-file request boundaries, selected-file retry, LZ4 compression, and FE HTTP 307 redirects.

Fixes #34