Skip to content

export_csv drops fields that are missing from the first record #53

Description

@chiruu12

export_csv builds the column list from the first record only:

if fields is None:
    fields = list(data[0].keys())

Rows are then built by looking up fields on each record, so a key that appears on any later record is never written and nothing is raised.

from brightdata.datasets.utils import export_csv

data = [
    {"url": "a.com", "price": 10},
    {"url": "b.com", "price": 20, "discount": "50%"},
]
export_csv(data, "out.csv")

out.csv:

url,price
a.com,10
b.com,20

discount is gone with no warning. Scraper results are commonly heterogeneous, since optional fields are absent when a page does not have them, so this loses real data on ordinary output. export_json and export_jsonl keep every key, so the same result set exports differently depending on format.

Taking the union of keys across all records, in first-seen order, would fix it. Happy to send a PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions