Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ e este projeto adere ao [Versionamento Semântico](https://semver.org/lang/pt-BR

- A criação das colunas de saída usa a opção `mode.chained_assignment` só no pandas 2, preservando entradas fatiadas sem emitir aviso de depreciação no pandas 3.1 (#269).

- A documentação, os READMEs e os notebooks de exemplo deixam de descrever o que o código não faz. A referência da API traz `perguntas` na 4ª posição da assinatura, diz que a entrada `pd.DataFrame` é modificada no lugar e que a saída é sempre DataFrame; `basic-usage.md` e os notebooks 04 e 08 deixam de passar a uma segunda chamada o DataFrame que a primeira já processou, e o trace completo da busca web usa um modelo sem configuração por campo. A página de exceções lista `ProviderAbortError` e `ProviderUsageLimitError`, que param a execução; a FAQ mostra a mensagem de dependência que o código emite e diz que só o `codex` recusa `api_key`; o extra `all`, as colunas de tokens da saída e o resumo de throughput, que só sai com `parallel_requests > 1`, aparecem como são (#200).
- No provider `codex`, a thread daemon que pede o `thread/unsubscribe` da linha abortada pela morte do app-server deixa de ficar presa para sempre. O processo morto ainda aceita a escrita enquanto termina, e o SDK registra a espera da resposta depois de o leitor do stdout ter acordado os pedidos em voo; agora o fechamento do cliente acorda essa espera (#250).
- `read_df` com argumento de leitura que faz o pandas devolver outra coisa que um DataFrame, como `chunksize`, `iterator`, `sheet_name=None` ou lista e o `chunksize` do JSON em linhas, devolve o `TextFileReader`, o `JsonReader` ou o dict de abas como o pandas o devolve, sem normalizar, com ou sem modelo. Antes, sem modelo, ou com modelo que tem campo de lista, dict ou modelo aninhado, e com `normalize=True`, o padrão, a função levantava `AttributeError` ao procurar as colunas no objeto do leitor (#249).
- Com entrada polars, a saída deixa de trocar sem aviso o valor que o modelo devolveu quando o pyarrow tipa a coluna pelo primeiro valor e converte os demais. Sai `Object`, com o aviso das colunas que o polars não reúne, o campo cuja anotação junta `timedelta` a outro tipo, `float` a `bool` ou `bytes` a `str`, e a coluna cujos valores trazem essas combinações ou as de data e hora, também em `Any`, onde o escalar do numpy, como o `np.int64`, conta como o número do Python, nos itens de lista e no campo de modelo aninhado, além da coluna cujos dicts do mesmo campo têm chaves diferentes e da que traz `datetime` com e sem fuso, `time` com fuso ou `Decimal` NaN. O NaN de campo `float` sai NaN. Antes, em `Union[timedelta, int]` o 5 virava cinco microssegundos, em `Union[float, bool]` o `True` virava `1.0`, o `datetime` sem fuso ganhava o fuso do primeiro, o `time` perdia o fuso, o `Decimal` NaN e o NaN de campo `float` viravam nulo, `{}` ao lado de `{"a": 1}` virava `{"a": None}`, e em `Union[NewType sobre date, int]` o 5 virava `1970-01-06`. Com `resume=True`, a coluna do campo cuja anotação é `int`, um `Literal` com inteiro ou uma união com um deles é lida com os inteiros do polars, e o inteiro acima de 2**53 deixa de voltar arredondado em `Float64`. O `Decimal` NaN e o NaN de campo `float` de uma linha processada deixam de ser acusados como incompatíveis na retomada, também com entrada pandas, e o reprocessamento campo a campo os repassa às condições como valor; na coluna float64, como a do checkpoint relido, o NaN não se distingue do nulo e segue contando como ausente. O `NewType` e o apelido `type`, inclusive o apelido genérico usado com argumentos, como `Opcional[int]` de `type Opcional[T] = T | None`, passam a contar pelo tipo a que se referem na saída polars, na retomada e em `read_df`: o `NewType` sobre `dict` sai em texto JSON, como o campo `dict`, e é retomado, e o campo `NewType` sobre `str` ou sobre `int` é relido do CSV como texto cru ou como inteiro exato (#241).
Expand Down
4 changes: 3 additions & 1 deletion README.es.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ pip install dataframeit[anthropic] # Anthropic Claude
pip install dataframeit[groq] # Groq
pip install dataframeit[codex] # SDK oficial de Codex (experimental)
pip install dataframeit[claude-code] # Claude Code mediante el Claude Agent SDK
pip install dataframeit[all] # todos los proveedores excepto Codex, búsqueda web, Polars y Excel
pip install dataframeit[all] # los proveedores de arriba salvo Codex, más búsqueda web, Polars y Excel
```

Extras opcionales: `search` (Tavily), `search-exa` (Exa), `search-all`, `polars`, `excel`.
Expand Down Expand Up @@ -70,6 +70,8 @@ print(resultado)
| Pésima atención... | negativo | alta |
| La entrega bien... | neutro | media |

Con `track_tokens=True`, el valor por defecto, la salida también trae las columnas de uso de tokens `_input_tokens`, `_cached_input_tokens`, `_output_tokens` y `_reasoning_tokens`, omitidas en la tabla de arriba.

Los nombres de clases y campos son arbitrarios: los notebooks de [`example/`](example/) usan nombres en portugués.

## Funcionalidades
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ pip install dataframeit[anthropic] # Anthropic Claude
pip install dataframeit[groq] # Groq
pip install dataframeit[codex] # Official Codex SDK (experimental)
pip install dataframeit[claude-code] # Claude Code via the Claude Agent SDK
pip install dataframeit[all] # all providers except Codex, web search, Polars and Excel
pip install dataframeit[all] # the providers above except Codex, plus web search, Polars and Excel
```

Optional extras: `search` (Tavily), `search-exa` (Exa), `search-all`, `polars`, `excel`.
Expand Down Expand Up @@ -70,6 +70,8 @@ print(result)
| Terrible service... | negative | high |
| Delivery was fine... | neutral | medium |

With `track_tokens=True`, the default, the output also has the token usage columns `_input_tokens`, `_cached_input_tokens`, `_output_tokens` and `_reasoning_tokens`, left out of the table above.

Field and class names are arbitrary; the examples in the [`example/`](https://github.com/bdcdo/dataframeit/tree/main/example) notebooks use Portuguese ones.

## Features
Expand Down
4 changes: 3 additions & 1 deletion README.pt-BR.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ pip install dataframeit[anthropic] # Anthropic Claude
pip install dataframeit[groq] # Groq
pip install dataframeit[codex] # Codex SDK oficial (experimental)
pip install dataframeit[claude-code] # Claude Code pelo Claude Agent SDK
pip install dataframeit[all] # todos os providers exceto Codex, busca web, Polars e Excel
pip install dataframeit[all] # os providers acima menos o Codex, mais busca web, Polars e Excel
```

Extras opcionais: `search` (Tavily), `search-exa` (Exa), `search-all`, `polars`, `excel`.
Expand Down Expand Up @@ -70,6 +70,8 @@ print(resultado)
| Péssimo atendimento... | negativo | alta |
| Entrega ok... | neutro | media |

Com `track_tokens=True`, o padrão, a saída traz também as colunas de uso de tokens `_input_tokens`, `_cached_input_tokens`, `_output_tokens` e `_reasoning_tokens`, omitidas da tabela acima.

Os nomes de classes e campos são arbitrários: os notebooks de [`example/`](example/) usam nomes em português.

## Funcionalidades
Expand Down
2 changes: 2 additions & 0 deletions docs/en/getting-started/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,8 @@ print(result)
2 Delivery ok, average product. neutral medium
```

With `track_tokens=True`, the default, the output also has the token usage columns `_input_tokens`, `_cached_input_tokens`, `_output_tokens` and `_reasoning_tokens`, left out above; see [Added Columns](../reference/api.md#added-columns).

## Complete Example

```python
Expand Down
16 changes: 9 additions & 7 deletions docs/en/guides/basic-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

Practical examples for common use cases.

`dataframeit()` writes the results and the status into the pandas DataFrame it receives; see [Input modified in place](../reference/api.md#input-modified-in-place). That is why the examples on this page pass `df.copy()`: without the copy, each example would find the `df` already processed by the previous one.

## Sentiment Analysis

```python
Expand All @@ -22,7 +24,7 @@ df = pd.DataFrame({
]
})

result = dataframeit(df, Sentiment, "Analyze the sentiment of the text.")
result = dataframeit(df.copy(), Sentiment, "Analyze the sentiment of the text.")
```

## Category Classification
Expand All @@ -36,7 +38,7 @@ class Category(BaseModel):
subcategory: str = Field(description="More specific subcategory")

result = dataframeit(
df,
df.copy(),
Category,
"Classify the text into the most appropriate category."
)
Expand All @@ -59,7 +61,7 @@ Extract all named entities from the text.
If there are no entities of some type, return an empty list.
"""

result = dataframeit(df, Entities, PROMPT)
result = dataframeit(df.copy(), Entities, PROMPT)
```

## Text Summarization
Expand All @@ -77,7 +79,7 @@ Analyze the text and extract a concise summary.
Identify the main points and central topic.
"""

result = dataframeit(df, Summary, PROMPT)
result = dataframeit(df.copy(), Summary, PROMPT)
```

## Using Different Input Types
Expand Down Expand Up @@ -113,18 +115,18 @@ result = dataframeit(series, Sentiment, PROMPT)

```python
# OpenAI with gpt-6-luna (default)
result = dataframeit(df, Model, PROMPT)
result = dataframeit(df.copy(), Sentiment, PROMPT)

# Google Gemini
result = dataframeit(
df, Model, PROMPT,
df.copy(), Sentiment, PROMPT,
provider='google_genai',
model='gemini-3.8-flash'
)

# Anthropic Claude
result = dataframeit(
df, Model, PROMPT,
df.copy(), Sentiment, PROMPT,
provider='anthropic',
model='claude-sonnet-5'
)
Expand Down
10 changes: 6 additions & 4 deletions docs/en/guides/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,21 @@

### `ImportError` asking you to install a package

The chosen provider needs the corresponding extra. The message says which one to install, for example:
The chosen provider needs its package, and the message says what to install. For LangChain providers, it suggests the provider's package or the `all` extra:

```bash
pip install dataframeit[anthropic]
pip install langchain-anthropic
# or
pip install dataframeit[all]
```

The extras are listed in [Installation](../getting-started/installation.md). LangChain providers without their own extra, such as Cohere and Mistral, need the `langchain-<provider>` package.
The provider's extra, such as `dataframeit[anthropic]`, installs the same package. `all` brings the providers that have an extra, except Codex; LangChain providers without their own extra, such as Cohere and Mistral, need the `langchain-<provider>` package, which `all` does not include. For `claude_code` and `codex`, the message asks for the provider's extra (`dataframeit[claude-code]`, `dataframeit[codex]`), and for web search, the search provider's package. The extras are listed in [Installation](../getting-started/installation.md).

### Authentication error (401 or 403)

The API key was not found or was rejected. Check the provider's environment variable (`OPENAI_API_KEY`, `GOOGLE_API_KEY`, `ANTHROPIC_API_KEY`, `GROQ_API_KEY`) in the same process that runs Python; in a notebook, set it before calling `dataframeit()`. An authentication error gets no new attempt: every row ends with status `'error'`.

The `codex` and `claude_code` providers use the local authentication of those tools and do not accept `api_key`; see [Providers](providers.md).
The `codex` and `claude_code` providers use the local authentication of those tools. `codex` rejects `api_key` with `ProviderConfigurationError`, and `claude_code` ignores it; see [Providers](providers.md).

## Result

Expand Down
4 changes: 2 additions & 2 deletions docs/en/guides/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ result = dataframeit(
)
```

When the provider reports a failure that prevents any further row, such as an exhausted usage quota (`ProviderUsageLimitError`) or a terminated `codex` runtime (`ProviderAbortError`), execution stops dispatching rows, saves the checkpoint, and warns how many rows were left without status. The control columns stay in the output, and running again with `resume=True` processes only the pending rows. With `reprocess_columns`, rows the interruption left without reprocessing keep their previous values and are marked in `_error_details`.
When the provider reports a failure that prevents any further row, such as an exhausted usage quota (`ProviderUsageLimitError`) or a terminated `codex` runtime (`ProviderAbortError`), execution stops dispatching rows, saves the checkpoint, and warns how many rows were left without status. The control columns stay in the output. When running again with `resume=True` processes only the pending rows, and what happens with `reprocess_columns`, is covered in [Failures that stop the run](../reference/exceptions.md#failures-that-stop-the-run).

## Token Tracking

Expand Down Expand Up @@ -155,7 +155,7 @@ Requisicoes: 100
============================================================
```

Cache and reasoning lines appear when the provider reports those tokens. With web search, the summary gains a section on searches and credits; with `claude_code`, the cost reported by the SDK.
The throughput section appears only with `parallel_requests > 1`. Cache and reasoning lines appear when the provider reports those tokens. With web search, the summary gains a section on searches and credits; with `claude_code`, the cost reported by the SDK.

To silence the summary and keep the usage columns, raise the logger level:

Expand Down
32 changes: 18 additions & 14 deletions docs/en/guides/web-search.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,13 @@ export EXA_API_KEY="your-exa-key" # https://exa.ai/
| Exa | `"exa"` | ignores `search_depth` | 1 credit (US$ 0.005) per search with up to 25 results, 5 above that |

```python
result = dataframeit(df, Model, PROMPT, use_search=True, search_provider="exa")
result = dataframeit(df.copy(), Model, PROMPT, use_search=True, search_provider="exa")
```

## Basic Usage

`dataframeit()` writes the results and the status into the pandas DataFrame it receives; see [Input modified in place](../reference/api.md#input-modified-in-place). That is why the examples on this page pass `df.copy()`: without the copy, each example would find the `df` already processed by the previous one.

```python
from pydantic import BaseModel, Field
from typing import Literal
Expand All @@ -61,7 +63,7 @@ extract information about the mentioned company.

# Enable web search with use_search=True
result = dataframeit(
df,
df.copy(),
CompanyInfo,
PROMPT,
use_search=True, # Enable web search
Expand Down Expand Up @@ -90,7 +92,7 @@ With search, the output gains the `_search_credits` column, with the credits spe

```python
result = dataframeit(
df, Model, PROMPT,
df.copy(), Model, PROMPT,
use_search=True
)
```
Expand All @@ -101,7 +103,7 @@ When the model has many fields, it can be useful to give each field its own agen

```python
result = dataframeit(
df, Model, PROMPT,
df.copy(), Model, PROMPT,
use_search=True,
search_per_field=True # One search agent per model field
)
Expand All @@ -112,7 +114,7 @@ result = dataframeit(
```python
# More detailed search (slower, more expensive)
result = dataframeit(
df, Model, PROMPT,
df.copy(), Model, PROMPT,
use_search=True,
search_depth='advanced',
max_results=10
Expand Down Expand Up @@ -162,7 +164,7 @@ class MedicationInfo(BaseModel):
)

result = dataframeit(
df,
df.copy(),
MedicationInfo,
"Analyze the medication: {texto}",
use_search=True,
Expand Down Expand Up @@ -243,12 +245,14 @@ To debug and audit agent reasoning, use the `save_trace` parameter.

### Example: Full Trace

With one agent per row, the trace goes to `_trace`. The example uses `CompanyInfo` from [Basic Usage](#basic-usage), which has no per-field configuration:

```python
import json

result = dataframeit(
df,
MedicationInfo,
df.copy(),
CompanyInfo,
PROMPT,
use_search=True,
save_trace=True # or "full"
Expand All @@ -267,7 +271,7 @@ For audits where only the search queries matter:

```python
result = dataframeit(
df, Model, PROMPT,
df.copy(), Model, PROMPT,
use_search=True,
save_trace="minimal" # Excludes search result content
)
Expand All @@ -277,7 +281,7 @@ result = dataframeit(

```python
result = dataframeit(
df,
df.copy(),
MedicationInfo,
PROMPT,
use_search=True,
Expand Down Expand Up @@ -329,7 +333,7 @@ class DrugRegulatory(BaseModel):
manufacturer: str = Field(description="Manufacturer")

result = dataframeit(
df,
df.copy(),
DrugRegulatory,
"Research the drug: {texto}",
use_search=True,
Expand Down Expand Up @@ -368,7 +372,7 @@ With `save_trace=True`, traces are organized by group:

```python
result = dataframeit(
df, Model, PROMPT,
df.copy(), Model, PROMPT,
use_search=True,
search_per_field=True,
search_groups={"regulatory": {"fields": ["fda_status", "ema_approval"]}},
Expand Down Expand Up @@ -494,14 +498,14 @@ With `parallel_requests=20` and `search_per_field=True` on a 4-field model, 80 a
```python
# Safe settings for Tavily with multiple fields
result = dataframeit(
df, Model, PROMPT,
df.copy(), Model, PROMPT,
use_search=True, search_per_field=True,
parallel_requests=3, rate_limit_delay=0.5,
)

# Higher throughput with Exa
result = dataframeit(
df, Model, PROMPT,
df.copy(), Model, PROMPT,
use_search=True, search_provider="exa",
search_per_field=True,
parallel_requests=5, rate_limit_delay=0.3,
Expand Down
6 changes: 3 additions & 3 deletions docs/en/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Automatic retry with exponential backoff, configurable rate limiting and periodi

### Performance

Parallel processing with auto-adjustment. Real-time throughput metrics.
Parallel processing that reduces the workers on rate limits. With `parallel_requests > 1`, the summary at the end of the run includes throughput metrics.
</div>

<div class="feature-card" markdown>
Expand All @@ -87,7 +87,7 @@ Tavily or Exa to enrich data with information from the internet, with per-field

### Multiple Inputs

pandas or Polars DataFrame and Series, list and dictionary. The output comes back in the same format as the input.
pandas or Polars DataFrame and Series, list and dictionary. The output is a DataFrame, a Polars one when the input is Polars.
</div>

</div>
Expand All @@ -98,7 +98,7 @@ pandas or Polars DataFrame and Series, list and dictionary. The output comes bac
pip install dataframeit[openai] # OpenAI (default provider)
pip install dataframeit[google] # Google Gemini
pip install dataframeit[anthropic] # Anthropic
pip install dataframeit[all] # all providers, web search, Polars and Excel
pip install dataframeit[all] # providers with an extra except Codex, plus web search, Polars and Excel
```

## Next Steps
Expand Down
Loading
Loading