Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 3 additions & 0 deletions docs/admin/keys-and-permissions.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ The two permissions required by a connected Failproof AI machine are independent
- `events:add` sends events and session data.
- `policies:pull` retrieves assigned policy deployments.

To run [Jev policies through FailproofAI Cloud](/policies/jev), select the **machine** key preset. It adds `jev:evaluate` to both permissions above. Cloud Jev cannot run with a key that lacks it.

Key secrets are shown when created or regenerated. Store them in a secret manager and rotate them without reusing an operator's interactive credentials.

## Permission catalog
Expand All @@ -64,6 +66,7 @@ Key secrets are shown when created or regenerated. Store them in a secret manage
| Audits | `audits:read`, `audits:write` |
| Policies | `policies:read`, `policies:write`, `policies:pull` |
| Usage | `usage:read` |
| Jev | `jev:evaluate` (requires `events:add` and `policies:pull`) |

`orgs:admin` is reserved for the instance operator and cannot be granted to an organization key or ordinary member. Retired `incidents:*` and `alerts:ack` tokens are accepted for compatibility and normalize to current `issues:*` permissions.

Expand Down
61 changes: 61 additions & 0 deletions docs/admin/product-analytics.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: "Product analytics"
description: "The usage telemetry Failproof AI Cloud sends about itself, what is never collected, and how to turn it off."
icon: "chart-no-axes-column"
---

Failproof AI Cloud sends usage telemetry **about the software itself** to its maintainers. It never includes your agents' data. On a self-hosted install it's on by default, and you can turn it off.

## What is collected

Every event carries five dimensions and nothing that identifies a person:

| Dimension | Example | What it is |
| --- | --- | --- |
| `product` | `agenteye` | Which product sent it |
| `component` | `dashboard`, `server`, `cli` | Which part of the product sent it |
| `app_version` | `0.0.1-beta.81` | The build you are running |
| `deployment_kind` | `self_hosted` | Whether this is a customer install, our demo, our QA, or a dev machine |
| `org_id` | a UUID | Which organization the action happened in, as an opaque identifier |

On top of those, events record **what was used, never what was in it**: which page was viewed (as a route pattern such as `/:org/sessions/:id`, never the real URL), how many filters of which kind were applied, whether an action succeeded, and how long it took.

The Slack app records workspace connects and disconnects, account links, questions asked, charts posted, approval decisions, and failures by a fixed category. It never includes Slack workspace, user, or channel IDs, email, questions, answers, SQL, or tokens.

### Your organization's name

Alongside the opaque `org_id`, we record your organization's name, URL slug, and dashboard address once, on an organization record rather than on every event. It lets us tell which account a pattern belongs to when we look at adoption or investigate a problem you reported. This is the one piece of identifying information the product sends about you.

### Your usage totals

The server reports each organization's metered usage totals, the same numbers you see on the Usage page. These are counts only: events, sessions, agents, environments, evaluation runs, scores, audit runs, saved queries, dashboards, alerts, issues, API keys, and members, plus the metering window's dates. They contain no identifiers, names, or query text, and they arrive whether or not anyone is signed in.

## What is never collected

- **No prompts, model responses, or event content**, not as text, a prefix, or a hash
- **No session, agent, or trace identifiers**
- **No evaluator score names**
- **No error messages, exception text, or response bodies**, only a status code and a category from a fixed list
- **No tool arguments, SQL, or query text**
- **No connected-app details.** Connecting or revoking an MCP client sends a permission count at most, never the app's name or any token
- **No email addresses**
- **No referrer URLs or query strings**
- **No screen recording, DOM capture, or click and cursor tracking**

## Turn it off on a self-hosted install

Set `DO_NOT_TRACK=1` on both the dashboard and the server. They decide separately, so setting it on only one leaves the other reporting. Restart the server for the change to apply.

Your release README describes the other opt-out settings and how to confirm telemetry is off.

## One ping when you opt out

If you opt out with a product setting rather than `DO_NOT_TRACK`, the dashboard may send **one anonymous ping** recording that, with a probability of 1 in 1000 per process start. It tells us how many operators turn telemetry off. It contains no identifiers: no organization, user, hostname, URL, or IP, only the product, component, version, deployment kind, and reason. An opted-out server sends nothing.

It is never sent when you use `DO_NOT_TRACK`, which is why that's the setting to use if your policy forbids any outbound telemetry.

## Where it goes

To PostHog (`us.i.posthog.com`) over TLS on port 443. It's one hostname, so you can also block it with a single egress rule.

For how we handle personal information more broadly, see the [privacy policy](https://befailproof.ai/privacy/).
141 changes: 141 additions & 0 deletions docs/agenteye/opentelemetry.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
---
title: "OpenTelemetry"
description: "Send coding-agent and application telemetry to FailproofAI Cloud with OTLP."
icon: "radio"
---

OpenTelemetry lets you send agent traces, logs, and metrics to FailproofAI Cloud using the standard OTLP/HTTP protocol. It is optional. Your existing FailproofAI integrations continue working without enabling it.

## Turn on ingest for your organisation

An organisation admin opens **Settings → OpenTelemetry** and turns on ingest. Use a Cloud API key with `events:add` permission, the same permission used for agent reporting.

If ingest is off, Cloud refuses OTLP exports, so `failproofai otel enable` changes nothing and your agents keep sending sessions the usual way. Run `failproofai otel status` to check the organisation switch, and ask an admin to turn it on.

## Connect a coding agent

Connect your machine with `failproofai config`, then choose an agent:

```bash
failproofai otel enable claude
failproofai otel enable codex
failproofai otel enable copilot
```

The setup command supports Claude Code, Codex, and Copilot and chooses Cloud or a local authenticated relay as appropriate. Copilot uses the local relay automatically. Start your configured FailproofAI daemon before using the relay.

If FailproofAI already sends an agent's session transcripts, setup asks whether to switch that agent to OTEL. The default is No. Accepting stops transcript uploads for that agent; it does not change policy enforcement or reporting from other agents.

## Connect an application

Use your application's OpenTelemetry instrumentation with these environment variables:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=https://app.befailproof.ai/otel
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer%20<key>'
export OTEL_SERVICE_NAME=my-agent
```

The endpoint is Cloud's OTLP base, `https://app.befailproof.ai/otel`; your exporter appends `/v1/traces`, `/v1/logs`, or `/v1/metrics` itself. Per-signal variables such as `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` are used as-is, so they need the full path, for example `https://app.befailproof.ai/otel/v1/logs`.

Replace `<key>` with your API key. Keep the secret out of shell history, screen recordings, and committed configuration. For a configured machine, `failproofai otel env --service my-agent` prints the block with its existing key. Its output contains a credential.

Cloud accepts OTLP/HTTP exports for traces, logs, and metrics, as protobuf or JSON, including gzip-compressed requests. It does not accept OTLP/gRPC.

Metrics are acknowledged for exporter compatibility but are not stored or displayed. Supported agent traces and coding-agent logs become sessions; fields the agent does not export remain absent.

## Capture prompt text

<Warning>
Text capture can include prompts, tool arguments, and other sensitive content. `failproofai otel enable` turns it on. Add `--no-content` to keep prompt and tool-detail capture off.
</Warning>

```bash
failproofai otel enable claude --no-content
failproofai otel enable codex --no-content
```

For an agent you configure manually, use its own content-capture controls:

<Tabs>
<Tab title="Claude Code">
Add these variables to your agent's environment alongside the application block above:

```bash
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_LOGS_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOG_USER_PROMPTS=1
export OTEL_LOG_TOOL_DETAILS=1
```

Set both content variables to `0` to avoid capturing prompts and tool details.
</Tab>
<Tab title="Codex">
In Codex's user configuration, use its OTEL settings:

```toml
[otel]
log_user_prompt = true
exporter = { otlp-http = { endpoint = "https://app.befailproof.ai/otel/v1/logs", protocol = "json", headers = { Authorization = "Bearer <key>" } } }
```

Codex conversation content is in logs; leave internal runtime traces and metrics disabled. Codex uses the complete endpoint you provide. Set `log_user_prompt` to `false` to leave prompts out; the setup command preserves an explicit `false`.
</Tab>
<Tab title="Gemini CLI">
Use Gemini's telemetry settings with the local relay:

```json
{
"telemetry": {
"enabled": true,
"target": "local",
"otlpEndpoint": "http://127.0.0.1:4318",
"otlpProtocol": "http",
"useCollector": true,
"logPrompts": true
}
}
```

Gemini CLI is not supported by `failproofai otel enable`. Configure it manually with an authenticated collector of your own. Set `logPrompts` to `false` for no prompt capture.
</Tab>
<Tab title="Copilot in VS Code">
Use these VS Code user settings:

```json
{
"github.copilot.chat.otel.enabled": true,
"github.copilot.chat.otel.exporterType": "otlp-http",
"github.copilot.chat.otel.otlpEndpoint": "http://127.0.0.1:4318",
"github.copilot.chat.otel.captureContent": true
}
```

Run `failproofai otel enable copilot` to enable the authenticated relay. Set `captureContent` to `false` to omit content.
</Tab>
</Tabs>

## Check or undo setup

The relay retains telemetry through temporary network and server failures and
retries after recovery. A permanent `400` rejection, including an event that is
too large to store, requires correcting the payload. The relay keeps the
rejected batch for inspection rather than repeatedly sending it.

After a `401` or `403`, replace the credential with a key granting `events:add`
for the same organisation. Reconnecting does not automatically resend retained
OTLP exports refused for credentials; ask your operator to recover those batches.
Keep them in their original organisation.

```bash
failproofai otel status
failproofai otel disable all
```

Status distinguishes settings managed by FailproofAI from telemetry you configured elsewhere. Disable restores the previous managed settings and the agent's previous transcript-upload choice.

<Card title="OpenTelemetry CLI" href="/reference/otel" icon="terminal">
Command options, local relay setup, and troubleshooting.
</Card>
61 changes: 61 additions & 0 deletions docs/ar/admin/product-analytics.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: "تحليلات المنتج"
description: "بيانات المراقبة الاستخدام التي يرسلها Failproof AI Cloud عن نفسه، وما لا يتم جمعه أبدًا، وكيفية إيقافه."
icon: "chart-no-axes-column"
---

يرسل Failproof AI Cloud بيانات مراقبة الاستخدام **عن البرنامج نفسه** إلى المسؤولين عنه. لا تتضمن أبدًا بيانات وكلائك. في التثبيت الذاتي، يكون مفعلًا بشكل افتراضي، ويمكنك إيقافه.

## ما يتم جمعه

كل حدث يحمل خمسة أبعاد ولا شيء يحدد شخصًا:

| البعد | مثال | ما هو |
| --- | --- | --- |
| `product` | `agenteye` | أي منتج أرسله |
| `component` | `dashboard`, `server`, `cli` | أي جزء من المنتج أرسله |
| `app_version` | `0.0.1-beta.81` | الإصدار الذي تقوم بتشغيله |
| `deployment_kind` | `self_hosted` | ما إذا كان هذا تثبيتًا من العميل أو عرضنا التوضيحي أو فريق ضمان الجودة لدينا أو جهاز تطوير |
| `org_id` | معرّف فريد عام | أي تنظيم حدثت الإجراءات فيه، كمعرّف معتم |

بالإضافة إلى هذا، تسجل الأحداث **ما تم استخدامه، وليس ما كان فيه أبدًا**: أي صفحة تم عرضها (كنمط مسار مثل `/:org/sessions/:id`، وليس العنوان الفعلي)، كم عدد المرشحات من أي نوع تم تطبيقه، ما إذا كان الإجراء ناجحًا، وكم من الوقت استغرق.

يسجل تطبيق Slack عمليات الاتصال والقطع بـ Slack، وربط الحسابات، والأسئلة المطروحة، والمخططات المنشورة، وقرارات الموافقة، والأخطاء حسب فئة ثابتة. لا يتضمن أبدًا معرّفات Slack Workspace أو المستخدم أو القنوات أو البريد الإلكتروني أو الأسئلة أو الإجابات أو SQL أو الرموز.

### اسم تنظيمك

إلى جانب `org_id` المعتم، نسجل اسم تنظيمك وعنوان URL slug وعنوان لوحة التحكم مرة واحدة، في سجل التنظيم بدلاً من كل حدث. يتيح لنا معرفة أي حساب ينتمي إليه نمط ما عند النظر في الاعتماد أو التحقيق في مشكلة أبلغت عنها. هذا هو الجزء الوحيد من المعلومات المحددة للهوية الذي يرسله المنتج عنك.

### إجمالي استخدامك

يبلغ الخادم عن إجمالي الاستخدام المقاس لكل تنظيم، وهي نفس الأرقام التي تراها في صفحة الاستخدام. هذه أرقام فقط: الأحداث والجلسات والوكلاء والبيئات وعمليات التقييم والنقاط وعمليات التدقيق والاستعلامات المحفوظة ولوحات التحكم والتنبيهات والمشاكل ومفاتيح API والأعضاء، بالإضافة إلى تواريخ نافذة الحد. لا تحتوي على معرّفات أو أسماء أو نص الاستعلام، وتصل سواء كان أي شخص مسجل دخول أم لا.

## ما لا يتم جمعه أبدًا

- **لا توجد مطالبات أو ردود نموذج أو محتوى الحدث**، وليس كنص أو بادئة أو تجزئة
- **لا معرّفات جلسة أو وكيل أو تتبع**
- **لا أسماء درجات المقيّم**
- **لا رسائل خطأ أو نص استثناء أو أجسام ردود**، فقط رمز الحالة وفئة من قائمة ثابتة
- **لا حجج أداة أو SQL أو نص استعلام**
- **لا تفاصيل التطبيقات المتصلة.** يرسل الاتصال أو إلغاء الإذن لعميل MCP عدد الأذونات على الأكثر، وليس اسم التطبيق أو أي رمز
- **لا عناوين بريد إلكتروني**
- **لا عناوين URL أو سلاسل الاستعلام للمرجع**
- **لا تسجيل الشاشة أو التقاط DOM أو تتبع النقر والمؤشر**

## إيقافه في التثبيت الذاتي

اضبط `DO_NOT_TRACK=1` على لوحة التحكم والخادم معًا. يقرران بشكل منفصل، لذلك إعدادها على واحد فقط يترك الآخر يبلغ. أعد تشغيل الخادم لتطبيق التغيير.

يصف ملف README الخاص بإصدارك الإعدادات الأخرى للإلغاء وكيفية التأكد من إيقاف المراقبة.

## ping واحد عند إلغاء الاشتراك

إذا قمت بإلغاء الاشتراك باستخدام إعداد المنتج بدلاً من `DO_NOT_TRACK`، فقد ترسل لوحة التحكم **ping مجهول واحد** يسجل ذلك، بحتمالية 1 من 1000 لكل بدء عملية. يخبرنا بعدد المشغلين الذين يقفلون المراقبة. لا يحتوي على معرّفات: لا توجد منظمة أو مستخدم أو اسم مضيف أو عنوان URL أو IP، فقط المنتج والمكون والإصدار ونوع النشر والسبب. لا يرسل الخادم الذي تم إلغاء الاشتراك فيه أي شيء.

لا يتم أبدًا إرساله عند استخدام `DO_NOT_TRACK`، وهذا هو السبب في أن هذا هو الإعداد الذي يجب استخدامه إذا كانت سياستك تمنع أي مراقبة صادرة.

## حيث يذهب

إلى PostHog (`us.i.posthog.com`) عبر TLS على المنفذ 443. إنه اسم مضيف واحد، لذا يمكنك أيضًا حظره باستخدام قاعدة خروج واحدة.

لمعرفة كيفية تعاملنا مع المعلومات الشخصية بشكل عام، راجع [سياسة الخصوصية](https://befailproof.ai/privacy/).
Loading
Loading