Skip to content

Commit 9274c51

Browse files
committed
Document cited chat and Gemini's free-tier limits
- How a question is answered, the settings, the 20-a-day limit and what readers see when it runs out - Migration 0002 must reach Neon before this code deploys - How to run the evaluation
1 parent 0c0a36c commit 9274c51

2 files changed

Lines changed: 59 additions & 2 deletions

File tree

‎docs/setup/cited-chat.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Cited chat and Research Workspaces
2+
3+
How questions get answered, what they need, and how to measure retrieval quality.
4+
5+
## What a question does
6+
7+
1. `POST /api/chat` checks the session, the chat rate limit (10 a minute and 100 an hour), the input and access to the document or workspace, then reserves one question from the month's allowance.
8+
2. A follow-up is rewritten into a standalone question, if it needs it. This is the only step that reads earlier messages, and its result can't change which documents are searched.
9+
3. Jina embeds the question. Postgres finds up to 20 passages by meaning and up to 20 by words, inside the versions this user may read right now.
10+
4. Jina reranks the best 30, and up to 8 passages go to Gemini.
11+
5. Gemini's answer streams to the browser. Then the question and answer are saved, every citation is mapped to a real passage, and the question is counted.
12+
6. A second, cheaper Gemini call checks each cited sentence against its passage. Citations it can't support show "Not clearly supported by this source".
13+
14+
If nothing relevant is found, FolioMind says so at step 4 without calling Gemini. A failed or stopped answer isn't counted.
15+
16+
## Settings
17+
18+
All of these have working defaults.
19+
20+
| Variable | Default | Used for |
21+
| --------------------------------------------- | ----------------------- | -------------------------------------- |
22+
| `GEMINI_CHAT_MODEL` | `gemini-3.6-flash` | Answers |
23+
| `GEMINI_FAST_MODEL` | `gemini-3.5-flash-lite` | Follow-up rewrites and citation checks |
24+
| `JINA_EMBEDDING_MODEL`, `JINA_RERANKER_MODEL` | See `.env.example` | Must match the index for embeddings |
25+
26+
Google applies rate limits per model and per project, so the fast model keeps rewrites and checks from using up the quota for answers.
27+
28+
Search settings such as passage counts and the abstention threshold live in `packages/ai/src/retrieval/config.ts`.
29+
30+
## Gemini's free tier
31+
32+
The free tier allows **20 requests a day for each model, across the whole Google project** (`GenerateRequestsPerDayPerProjectPerModel-FreeTier`, measured 2026-11-05), plus a per-minute limit. Quotas reset at midnight Pacific time, and AI Studio shows your project's current limits.
33+
34+
So that one busy day doesn't stop answers, FolioMind answers on `GEMINI_CHAT_MODEL` first and on `GEMINI_FAST_MODEL` once that model's quota is spent. When both are out, readers see "FolioMind has used today's free AI answers" with **Try again**, and the question isn't counted. For a brief per-minute limit they see "FolioMind's AI is busy right now" instead.
35+
36+
Turning on billing for the Google project lifts these limits. Local testing and the deployed site share them if they use the same project, so a long `pnpm eval:retrieval` run can use up the day's answers on the live site. Give local development its own project to avoid that.
37+
38+
## Deploying a schema change
39+
40+
Migration `0002_chat_citation_support` adds two columns the chat reads and writes. Apply it to Neon **before** deploying this code, following [Neon (staging and production)](database.md#neon-staging-and-production). It only adds columns, so the site that's already running keeps working.
41+
42+
## Measuring retrieval
43+
44+
```sh
45+
pnpm eval:retrieval --retrieval-only # Jina only: quick, uses no Gemini quota
46+
pnpm eval:retrieval # The whole set, at a pace the free tier allows
47+
pnpm eval:retrieval --kind adversarial # One kind of question
48+
pnpm eval:retrieval --delay 13 # Seconds between Gemini answers
49+
pnpm eval:retrieval --threshold 0.15 # Try another abstention threshold
50+
pnpm eval:retrieval --json out.json # Keep every per-question result
51+
```
52+
53+
`GEMINI_CHAT_MODEL` chooses the model under test, with no fallback, so results belong to that model. A whole run is more answers than one free model allows in a day: it stops cleanly at the limit and reports what it finished.
54+
55+
It builds its own local database (`foliomind_eval_test`) with the production chunker and real embeddings, and it refuses to run against any database that isn't on your machine. The documents and questions are in `packages/ai/src/eval/`. They describe invented places and figures, so a correct answer can only come from retrieval.
56+
57+
Run it before and after changing a retrieval setting, and compare the embedding-only, fusion and reranked rows.

‎docs/setup/third-party-services.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -121,8 +121,8 @@ All `AUTH0_*` variables go in `apps/web/.env.local`. The Google/GitHub OAuth sec
121121
### Google Gemini
122122

123123
1. Create an API key in Google AI Studio → `GEMINI_API_KEY` (web and Worker B).
124-
2. `GEMINI_CHAT_MODEL` and `GEMINI_GENERATION_MODEL` default to the pinned `gemini-3.6-flash`. Avoid aliases such as `gemini-flash-latest`: they move to the newest model, which can be overloaded (on 2026-10-30 it pointed at `gemini-3.8-flash` and returned 503 "high demand"). Google may also serve a newer model than the one requested (`gemini-3.5-flash` requests are served by `gemini-3.6-flash`), so AI runs record the response's `modelVersion` (Section 2). `pnpm check:services` fails if a configured model is not available to your key.
125-
3. The free tier allows 5 generate requests per minute per model, which is enough for the smoke check but not for real use. Enable billing on the Google Cloud project behind the key before staging or production.
124+
2. `GEMINI_CHAT_MODEL` and `GEMINI_GENERATION_MODEL` default to the pinned `gemini-3.6-flash`. Avoid aliases such as `gemini-flash-latest`: they move to the newest model, which can be overloaded (on 2026-10-30 it pointed at `gemini-3.8-flash` and returned 503 "high demand"). Google may also serve a newer model than the one requested (`gemini-3.5-flash` requests are served by `gemini-3.6-flash`), so AI runs record the response's `modelVersion` (Section 2). `pnpm check:services` fails if a configured model is not available to your key. `GEMINI_FAST_MODEL` (default `gemini-3.5-flash-lite`) handles follow-up rewrites and citation checks, and answers once the chat model's free quota is spent; see [cited chat](cited-chat.md).
125+
3. The free tier allows 5 generate requests a minute and 20 a day, per model and per project. That is enough for a side project with few readers, and [cited chat](cited-chat.md) explains how FolioMind handles running out. Enable billing on the Google Cloud project behind the key if it needs to serve more.
126126

127127
### Stripe (test mode)
128128

0 commit comments

Comments
 (0)