Missing error handling when writing to unknown Clickhouse columns

[fschoell]

When the sink receives a database entry for a column that doesn't exist in Clickhouse, the sink crashes with a nil pointer exception:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xf99e6e8]

goroutine 137 [running]:
github.com/streamingfast/substreams-sink-sql/db.convertOpToClickhouseValues(0xc000e12330)
	/Users/work/GoLand/substreams-sink-sql/db/dialect_clickhouse.go:172 +0x208
github.com/streamingfast/substreams-sink-sql/db.clickhouseDialect.Flush({}, {0x15?, 0x1?}, {0x107da570, 0xc000e12870}, 0xc00073a700, {0x107da570?, 0x106b9400?}, 0xc0005ab840?)
	/Users/work/GoLand/substreams-sink-sql/db/dialect_clickhouse.go:64 +0x9c5
github.com/streamingfast/substreams-sink-sql/db.(*Loader).Flush(0xc00073a700, {0xfb36cf3?, 0xbe?}, {0xc000d7b530, 0x28}, 0xc001002130, 0xbe)
	/Users/work/GoLand/substreams-sink-sql/db/flush.go:29 +0x279
github.com/streamingfast/substreams-sink-sql/sinker.(*SQLSinker).HandleBlockScopedData(0xc0006be600, {0x107da5a8, 0xc000836dc0}, 0xc00093b100, 0x119a51aa, 0xc001002130)
	/Users/work/GoLand/substreams-sink-sql/sinker/sinker.go:125 +0x5c5
github.com/streamingfast/substreams-sink.(*Sinker).doRequest(0xc0006e44d0, {0x107da5a8, 0xc000836dc0}, 0xc000820dd0, 0xc00073ab00, {0x107b78e0, 0xc000066270}, {0xc0000661a0, 0x1, 0x1}, ...)
	/Users/work/.go/pkg/mod/github.com/streamingfast/substreams-sink@v0.3.3-0.20231222213313-af36d45db340/sinker.go:472 +0x1974
github.com/streamingfast/substreams-sink.(*Sinker).run(0xc0006e44d0, {0x107da5a8, 0xc000836dc0}, 0xc000820dd0, {0x107cf668, 0xc0006be600})
	/Users/work/.go/pkg/mod/github.com/streamingfast/substreams-sink@v0.3.3-0.20231222213313-af36d45db340/sinker.go:274 +0x867
github.com/streamingfast/substreams-sink.(*Sinker).Run(0xc0006e44d0, {0x107da5a8, 0xc000836dc0}, 0xc000820dd0, {0x107cf668, 0xc0006be600})
	/Users/work/.go/pkg/mod/github.com/streamingfast/substreams-sink@v0.3.3-0.20231222213313-af36d45db340/sinker.go:199 +0x488
github.com/streamingfast/substreams-sink-sql/sinker.(*SQLSinker).Run(0xc0006be600, {0x107da5a8, 0xc000836dc0})
	/Users/work/GoLand/substreams-sink-sql/sinker/sinker.go:94 +0x7c5
created by github.com/streamingfast/cli.(*Application).SuperviseAndStart in goroutine 1
	/Users/work/.go/pkg/mod/github.com/streamingfast/cli@v0.0.4-0.20230825151644-8cc84512cd80/application.go:102 +0x37a

Seems like there is no check if the column actually exists before accessing it in here.

[GabrielCartier]

Heads up: this repository is deprecated (see #133). The SQL sink was folded into the main substreams CLI as substreams sink postgres / substreams sink clickhouse, shipped in substreams v1.20.2. This repo gets security fixes only for one year, then goes read-only.

Leaving this open because the bug is real and the code still runs — it just lives at sink/sql/db_changes/ in streamingfast/substreams now. Any fix has to land there, so please refile or follow up upstream if you are still hitting it.