export_csv builds the column list from the first record only:
if fields is None:
fields = list(data[0].keys())
Rows are then built by looking up fields on each record, so a key that appears on any later record is never written and nothing is raised.
from brightdata.datasets.utils import export_csv
data = [
{"url": "a.com", "price": 10},
{"url": "b.com", "price": 20, "discount": "50%"},
]
export_csv(data, "out.csv")
out.csv:
url,price
a.com,10
b.com,20
discount is gone with no warning. Scraper results are commonly heterogeneous, since optional fields are absent when a page does not have them, so this loses real data on ordinary output. export_json and export_jsonl keep every key, so the same result set exports differently depending on format.
Taking the union of keys across all records, in first-seen order, would fix it. Happy to send a PR.
export_csvbuilds the column list from the first record only:Rows are then built by looking up
fieldson each record, so a key that appears on any later record is never written and nothing is raised.out.csv:
discountis gone with no warning. Scraper results are commonly heterogeneous, since optional fields are absent when a page does not have them, so this loses real data on ordinary output.export_jsonandexport_jsonlkeep every key, so the same result set exports differently depending on format.Taking the union of keys across all records, in first-seen order, would fix it. Happy to send a PR.