Skip to content

HTTP-API.md

Latest commit

 

History

History
1218 lines (944 loc) · 49.2 KB

HTTP-API.md

File metadata and controls

1218 lines (944 loc) · 49.2 KB
Raw

ScreenKit HTTP API

Languages: English · 中文

Local HTTP API: OCR endpoints, plus ASR / TTS / ITN / barcode / face extensions.

LAN access is on by default. Turn it off to listen on this PC only. Do not expose this port on untrusted networks.

1. Enable and base URL

Turn on HTTP under Settings, or edit config.toml next to the executable:

http_enabled = true
http_ocr = true
http_tts = true
http_asr = true
http_translate = true
http_chat = true
http_face = true
http_rtsp = true
http_lan = true
http_port = 1224

Each http_* module defaults to on. Turning one off leaves that path out of GET /api and makes the path return code 810. http_rtsp = false refuses WebSocket /rtsp and does not upgrade the connection. /api/status, /api/toast, /api/zhconv, /api/pinyin, /api/calendar, /api/jpyomi, /api/text, /api/fx, /api/ipcity, /api/mac, /api/usb, /api/qrmake, /api/qrscan, and /api/cast/stop stay available. GET / is the local tools page. WebSocket /chat is the tools-page chat room. The first text frame is {"op":"join","nick":"name","gender":"m|f|"}, then {"op":"say","text":"..."}. An omitted or other gender means private. POST /chat/file uploads one image or file (multipart fields file and kind=image|file, 8 MB). The bytes go to tmp/chat- plus an id beside the program and are deleted on the next start. The client then sends {"op":"file","file":"<id>"}. GET /chat/file/{id} downloads it. Optional to is a peer id and priv:true delivers that message only to the two people. {"op":"offer","file":"<id>","to":peer} asks that person to accept; {"op":"take","xfer":n} or {"op":"drop","xfer":n} answers. The file id is sent to the receiver only after take. Heartbeats are every 5 seconds; a missed heartbeat closes the socket. http_enabled = false turns the whole server off. OCR also covers /api/qr and the Umi-OCR compatible /api/qrcode. ASR also covers /api/itn and /api/asr/models.

Item Description
Base URL http://127.0.0.1:{http_port}. With http_lan = true, LAN clients use this PC’s NIC address on the same port
Default Port 1224, http_lan = true (this PC and the LAN). http_lan = false listens on 127.0.0.1 only
Content-Type Prefer application/json; charset=utf-8 for JSON request/response
JSON text UTF-8 CJK as-is (not \uXXXX escapes)
CORS * allowed for local web debugging
OPTIONS Preflight supported (204)

The bottom status bar keeps the listener beside the memory summary, for example HTTP 0.0.0.0:1224 or HTTP 127.0.0.1:1224. Per-NIC file-transfer sockets without the API listener show HTTP LAN:1224. If the API, file transfer, and cast are all off it shows HTTP off. If one of them is on but nothing is listening it shows HTTP not started and the error. Click the HTTP text to open the web home in the browser. Click the model summary to open Memory. The empty space to the right does nothing.

2. Response conventions

Failures also return HTTP 200. Success or failure is only the JSON code and data. An unknown path, a wrong method, and an uncaught exception are still 200, with the error code in the JSON (404, 805, 900). OPTIONS preflight stays 204. A successful WebSocket upgrade stays 101.

2.1 Success (with payload)

{
  "code": 100,
  "data": { },
  "time": 12,
  "timestamp": 1710000000
}
Field Type Description
code int 100 success; 101 OCR found no text; other values are errors
data any Result payload; on failure often an error message string
time int Optional processing time (ms)
timestamp long Optional Unix seconds

2.2 Error example

{
  "code": 802,
  "data": "Missing base64 field in request."
}

2.3 Common business codes

code Meaning (summary)
100 Success
101 OCR: no text; barcode: none found; Face: no face
404 Unknown path
800 Request parse failure / empty body
801 Empty request
802 Missing required field (e.g. base64, text)
803 Invalid field type/length
804 Failed to interpret options
805 Method not allowed (HTTP status stays 200)
806 base64 decode failed
810 That API module is turned off (Settings → API)
900 Internal error
901 OCR recognition failed
910+ ASR-related
920+ TTS-related (921 Sherpa unavailable, 922 no model/voice, 923 empty audio, 924 synth failed, 925 unknown engine, 927 Chrome voice pack not installed)
930+ Face-related (930 no models, 931 failed, 932 missing det/rec files)
950 Barcode / QR scan failed

3. Endpoint overview

Method Path Description
GET / Local tools page (HTML). Assets /sk/tools.css, /sk/tools.js, /sk/fa.css, /sk/fa-solid-900.woff2, /sk/viewer.min.css, /sk/viewer.min.js
GET /api Enabled routes: method, summary, parameters. ?lang= picks the language; omitted uses the current UI language
GET /api/status · /api/health Service and capability status
GET/POST /api/toast Bottom-of-screen toast (not tied to a module switch)
GET/POST /api/zhconv Simplified/traditional (not tied to a module switch)
GET/POST /api/pinyin Chinese to pinyin (not tied to a module switch; needs NPinyin.dll)
GET/POST /api/calendar Gregorian to another calendar (not tied to a module switch)
GET/POST /api/jpyomi Japanese reading (not tied to a module switch)
GET/POST /api/text Text encode/decode (not tied to a module switch)
GET/POST /api/fx Currency conversion (not tied to a module switch)
GET/POST /api/ipcity IP city lookup (not tied to a module switch)
GET/POST /api/mac MAC vendor lookup (not tied to a module switch)
GET/POST /api/usb USB ID lookup: vendor, product, model, and interfaces (not tied to a module switch)
GET/POST /api/qrmake Make a QR code or barcode (not tied to a module switch)
POST /api/qrscan Read a QR code or barcode (not tied to a module switch)
GET /api/ocr/models OCR engines and models (same as the main window)
GET /api/ocr/get_options OCR option descriptors
POST /api/ocr Image OCR
POST /api/qr · /api/qrcode · /api/barcode · /api/barcodes Barcode / QR only (no OCR). /api/qrcode matches Umi-OCR
GET /api/asr/models List ASR models
POST /api/asr Speech recognition
GET /api/tts/engines List TTS engines only
GET /api/tts/models List TTS models. engine filters one engine; a full list is cached
GET /api/tts/devices List playback devices
POST /api/tts Speech synthesis (WAV base64)
POST /api/tts/speak Synthesize and play here. play=queue waits; play=now interrupts
POST /api/itn Inverse text normalization (WeText + rules)
GET /api/translate/langs Source languages and the targets for each, plus the engines and local ONNX models
POST /api/translate · /api/translate/batch Batch translate; engine is auto / onnx / chrome / llm
POST /api/chat LLM chat (text or audio in; optional TTS wav_base64)
GET/POST /api/cast/stop Close the screencast viewer and drop the current receive session
WebSocket /cast Screencast media (same HTTP port; SCST frames as binary messages)
GET /api/face/models List face ONNX files
POST /api/face Face detect / embedding / compare two images

Paths are case-insensitive; a trailing / is optional. A module turned off under Settings → API is omitted from GET /api and returns code 810.

4. GET / and GET /api

GET / and HEAD / return the local tools page: calendars, text (including simplified/traditional and Japanese yomi), QR codes, and barcodes. The page loads /sk/tools.css, /sk/tools.js, /sk/fa.css (Font Awesome, font /sk/fa-solid-900.woff2), /sk/viewer.min.css, and /sk/viewer.min.js, and calls /api/zhconv, /api/pinyin, /api/calendar, /api/jpyomi, /api/text, /api/fx, /api/qrmake, and /api/qrscan. The desktop file manager is /files.

GET /api returns the service name, the short endpoints list, and apis. Each apis item has method (GET, POST, or GET/POST), path, summary, and params (name and description). That text is the same as the HTTP tab templates. ?lang=zh|en|ja|ko selects the language. Omitting it uses the app's current UI language and does not change the UI. A module that is off is omitted. langs lists the choices.

Response example:

{
  "code": 100,
  "data": {
    "name": "ScreenKit HTTP API",
    "lang": "en",
    "langs": [{ "code": "en", "name": "English" }],
    "endpoints": ["GET  /api/status"],
    "apis": [
      {
        "method": "GET",
        "path": "/api/status",
        "example": "/api/status",
        "summary": "Whether the server is listening, and whether OCR, speech, translate, chat, and face are on.\nGET only. No parameters.\nThe response includes http_enabled, http_ocr, ocr_engine, asr_models, llm_chat, and similar fields.",
        "params": [],
        "doc": "Whether the server is listening, and whether OCR, speech, translate, chat, and face are on.\nGET only. No parameters.\nThe response includes http_enabled, http_ocr, ocr_engine, asr_models, llm_chat, and similar fields.",
        "aliases": ["/api/health"]
      }
    ]
  }
}

5. GET /api/status · /api/health

Health check and capability probe.

data fields:

Field Type Description
app string "ScreenKit"
http_enabled bool Whether HTTP is enabled in config
http_ocr bool OCR and barcode paths enabled
http_tts bool TTS paths enabled
http_asr bool ASR and ITN paths enabled
http_translate bool Translate path enabled
http_chat bool Chat path enabled
http_face bool Face paths enabled
http_rtsp bool RTSP relay enabled
ocr_engine bool OCR runner available
asr_engine bool ASR engine injected
tts_engine bool TTS (Sherpa) engine injected
tts_sapi bool Local SAPI voices available
tts_winrt bool Windows (WinRT / OneCore) voices available
tts_edge bool Edge online voice feature is built in (actual synthesis still needs network access)
asr_models int Scanned ASR model count
tts_models int Scanned Sherpa TTS model count
face_ready bool Whether facemodels has det+rec ONNX
face_models int Scanned face ONNX count
barcode bool Barcode / QR endpoint is built in (no extra model). Scanning still needs ZXing.dll, which is not in the release archive
itn bool WeText ITN available
itn_error string Reason when ITN is unavailable
llm_translate bool Translate LLM configured
llm_chat bool Chat LLM configured

Example:

curl -s "http://127.0.0.1:1224/api/status"

---

## 5.1 GET/POST `/api/toast`

Shows a toast at the bottom of the primary monitor work area. It does not take focus. Duration is clamped to 800–8000 ms.

**GET:** `/api/toast?text=Copied&ms=1900`

**POST** `application/json`

| Field | Type | Description |
|-------|------|-------------|
| `text` | string | Message. `message` is an alias |
| `ms` | number | Visible milliseconds, default 1900 |

```json
{ "text": "Copied", "ms": 1900 }

Response data: text and ms (clamped to 800–8000). An empty text returns code 802.

curl -s -X POST "http://127.0.0.1:1224/api/toast" -H "Content-Type: application/json" -d "{\"text\":\"Copied\",\"ms\":1900}"

The CLI shows the same toast without HTTP:

ScreenKit --toast "Copied" --ms 1900

---

## 5.2 GET/POST `/api/zhconv`

`LCMapStringEx` simplified/traditional conversion. Traditional uses `zh-TW`. The default direction is traditional. GET query strings are percent-encoded and decoded as UTF-8.

**GET:** `/api/zhconv?text=软件&to=trad`

**POST** `application/json`

| Field | Type | Description |
|-------|------|-------------|
| `text` | string | Source. `message` is an alias |
| `to` | string | `trad` (default) or `simp` |

```json
{ "text": "软件", "to": "trad" }

Response data: text, to. An empty text returns 802.

ScreenKit --zhconv "软件"
ScreenKit --zhconv "國發" --simp

5.2.1 GET/POST /api/pinyin

Chinese to pinyin with optional NPinyin.dll (about 75 KB). Install it from Help → Install features → Tools. It is not in the slim package. There are no tone marks. Common polyphonic words are listed in pinyin.toml beside the program (重庆 is chong qing, 银行 is yin hang). The same word in pinyin.user.toml overrides it. An empty reading drops that system word. Updates replace pinyin.toml and leave pinyin.user.toml. The letter ü is written as u. Letters and digits are spaced one by one.

GET: /api/pinyin?text=中国人民银行

POST application/json

Field Type Description
text string Source. message is an alias. Empty or omitted returns only data.installed (code 100) and does not return pinyin or py 
{ "text": "中国人民银行" }

Response data: installed, pinyin (spaced, no tone marks), and py (one lowercase initial per character). 中国人民银行 is pinyin zhong guo ren min yin hang and py zgrmyh. 重庆 is chong qing and cq. Text longer than 20000 characters returns 500. Text present while the DLL is missing returns 803.

ScreenKit --pinyin "重庆"
ScreenKit --pinyin "重庆" --init

5.3 GET/POST /api/calendar

Windows.Globalization.Calendar. Chinese lunar includes the sexagenary year and leap month. The date is converted at local noon. Omitted date is today; omitted calendar is lunar.

GET: /api/calendar?date=2024-02-10&cal=lunar

POST application/json

Field Type Description
date string yyyy-MM-dd. Omitted means today
cal string lunar (default), gregorian, jp, jplunar, tw, ko, vilunar, he, hijri, umalqura, fa, th, julian 
{ "date": "2024-02-10", "cal": "lunar" }

Response data: cal, gregorian, ganzhi, era, era_num, year, year_num, month, month_num, month_count, day, day_num, week, leap. A bad date or calendar returns 802.

ScreenKit --calendar 2024-02-10 --cal lunar

5.4 GET/POST /api/jpyomi

JapanesePhoneticAnalyzer. The Japanese language feature must be installed. Windows has no equivalent Chinese pinyin API.

GET: /api/jpyomi?text=東京は晴れです

POST application/json

Field Type Description
text string Japanese. message is an alias
mono bool true reads character by character. Default is by word 
{ "text": "東京は晴れです", "mono": false }

Response data: ruby, yomi, mono. An empty text returns 802.

ScreenKit --jpyomi "東京は晴れです"
ScreenKit --jpyomi "東京" --mono

5.5 GET/POST /api/text

Text helpers. Not tied to a module switch. text may be empty. Unknown op returns 802. Bad Base64, hex, or JSON returns 500.

Field Type Description
text string Source. message is an alias
op string b64enc b64dec urlenc urldec utf8hex utf8unhex gbkhex gbkunhex uesc uunesc upper lower collapse dropempty json stats 
{ "text": "hi", "op": "b64enc" }

Response data: text is the result (aGk= for the example). op echoes the operation. stats puts a short multiline count in text.

5.5.1 GET/POST /api/fx

Currency conversion. Not tied to a module switch. Foreign units use the Bank of China middle rate (中行折算价 per 100, divided by 100). A successful page is cached for 10 minutes. A timeout, an empty page, or a missing currency uses the built-in 2026-10-10 table and that failure is cached for 10 minutes too. There is no jiao or fen. An unknown unit or a non-number returns 802.

Field Type Description
amount number or string Amount. n is an alias
from string Source unit. Default CNY. USD and the other listed units
to string Target unit, or all (default) to list every unit 
{ "amount": 1, "from": "USD", "to": "CNY" }

Response data: text is the multiline result, including the rate note. live is false when the default table was used.

5.5.2 GET/POST /api/ipcity

City for an IP address or domain. Not tied to a module switch. An empty ip looks up this PC's public address. The request uses the app HTTP proxy setting and tries ip-api.com, then ipinfo.io, then api.ip.sb. A private or invalid address returns 802.

Field Type Description
ip string IP or domain. Empty means this PC. host and query are aliases 
{ "ip": "8.8.8.8" }

Response data: ip, country, region, city, isp, and text. lat and lon are present when the provider returned a point.

5.5.3 GET/POST /api/mac

IEEE registrant for a MAC address. Not tied to a module switch. Colons, dashes, dots, or plain hex are accepted, from the first six digits through a full address. The request uses the app HTTP proxy and tries macaddresslookup.app, then api.macvendors.com, then macverify.com. A locally administered address (the second-least bit of the first octet) is not registered and returns 802 without a network call. An unknown prefix also returns 802.

Field Type Description
mac string MAC or prefix. oui, host, and query are aliases 
{ "mac": "00:1A:2B:3C:4D:5E" }

Response data: mac, company, country, address, block (MA-L, MA-M, or MA-S when known), local, multicast, and text.

5.5.4 GET/POST /api/usb

Vendor and product name for a USB ID. Not tied to a module switch. The list is the public usb.ids file. It is downloaded through the app HTTP proxy, stored as tmp/usb.ids beside the program, and used again for 7 days. Accepted forms: 046d, 046d:c52b, 0x046d, and VID_046D&PID_C52B. An unknown vendor returns 802. An unknown product still returns the vendor.

Field Type Description
vid string Vendor ID, four hex digits
pid string Product ID, optional
id string vid or vid:pid. usb, host, and query are aliases 
{ "vid": "046d", "pid": "c52b" }

Response data: vid, pid, vendor, product, and text.

5.6 GET/POST /api/qrmake · POST /api/qrscan

Make or read a QR code or barcode. Not tied to a module switch. Making uses ZXing. Reading does not run OCR.

Make GET/POST /api/qrmake

Field Type Description
text string Payload. message is an alias
format string qr (default), datamatrix, code128, code39, ean13, ean8, upca
encoding string utf8 (default), gbk, hex. Linear codes use the characters as written 
{ "text": "hello", "format": "qr", "encoding": "utf8" }

Response data: png is the PNG as Base64, plus format and encoding. Empty text returns 802.

Read POST /api/qrscan

Send the image as base64, path, or multipart, the same as POST /api/qr. Request format is dict (default, with boxes) or text. Each hit also has a format field (QRCode for a QR code). No code returns code 101.

6. OCR

6.0 GET /api/ocr/models

Lists the same engines as the main window. The first packs[] item is Windows OCR (id=winocr; models are installed languages, id is the BCP-47 tag). Later items are ONNX packs under ocrmodels (models[].id is the variant title). current is the main window's pack, language, and device (cpu / gpu / intel).

POST /api/ocr options accepts ocr.pack (pack id). When set, ocr.language is matched only inside that pack.

6.1 GET /api/ocr/get_options

Returns option descriptors: each entry has title / toolTip / default / optional optionsList.

Key Meaning Default
ocr.angle Text orientation (cls) true
ocr.maxSideLen Detection side limit 1024
ocr.engine Engine: empty = main window, winocr = Windows OCR, onnx = ONNX pack ""
ocr.language ONNX variant title, or a Windows OCR language "" (main window model / language)
ocr.device Device: cpu / gpu / intel cpu
ocr.barcode Also scan barcodes / QR codes false (barcode-only: POST /api/qr)
tbpu.parser Layout mode (kept for compatibility) multi_line
data.format Response format: dict or text dict

6.2 POST /api/ocr

Recognize text in an image. Supported input modes:

  1. JSON: base64 + optional options
  2. multipart/form-data: file field + optional options

JSON body

{
  "base64": "<image base64; data:image/png;base64, prefix allowed>",
  "options": {
    "ocr.angle": true,
    "ocr.maxSideLen": 1600,
    "ocr.engine": "winocr",
    "ocr.language": "zh-Hans-CN",
    "ocr.device": "gpu",
    "data.format": "dict"
  }
}
Field Required Description
base64 Yes Image encoding; formats OpenCV can decode (png/jpg/bmp/webp, …)
options No Object; missing keys use get_options defaults, then merge with main-window model config

Common options keys:

Key Type Description
data.format string dict: line list with boxes; text: plain text (\n between lines)
ocr.angle bool/string Orientation classification on/off
ocr.maxSideLen int Detection side length, roughly 320–4096
ocr.engine string Empty follows the main window. winocr / windows / Windows 系统 OCR uses Windows OCR and does not load ONNX. onnx switches back to the first ONNX pack when the main window is Windows OCR
ocr.language string ONNX: match a variant title (fuzzy contains). Windows OCR: BCP-47 or a language name (zh-Hans-CN, en-US, 简体中文, 英语, 韩语). Setting this field to winocr also selects Windows OCR. An unknown installed language fails
ocr.device string cpu / gpu (CUDA) / intel (DirectML)
ocr.detThresh number Detection threshold (extension)
ocr.detBoxThresh number Box score threshold (extension)
ocr.barcode bool/string When true, also scan barcodes/QR; aliases: ocr.qr, ocr.codes

Multipart request

Part name Description
file / image / img / upload / pic or a part with a filename Image binary
base64 / image_base64 Base64 text also accepted
options JSON object string, or key=value&...
Any ocr.xxx / data.format field Single option

Success (data.format = dict)

{
  "code": 100,
  "data": [
    {
      "text": "Hello",
      "score": 0.98,
      "box": [[10.0, 20.0], [100.0, 20.0], [100.0, 50.0], [10.0, 50.0]],
      "end": "\n"
    }
  ],
  "time": 45,
  "timestamp": 1710000000
}
Field Description
text Line text
score Confidence
box Four-point box [[x,y],…] in original image pixels (detection may downscale by maxSideLen; coordinates are mapped back)
end Line ending; fixed "\n"

Success with barcodes (ocr.barcode = true)

When barcode scanning is enabled, the response includes a top-level barcodes array (always present if the option is on). OCR lines stay in data. If there is no OCR text but barcodes are found, code is still 100 and data may be an empty array (dict) or barcode plain text (text format).

{
  "code": 100,
  "data": [],
  "barcodes": [
    {
      "type": "QR_CODE",
      "text": "https://example.com",
      "box": [[12.0, 40.0], [180.0, 40.0], [180.0, 210.0], [12.0, 210.0]]
    },
    {
      "type": "EAN_13",
      "text": "6901234567892",
      "box": [[20.0, 300.0], [220.0, 300.0], [220.0, 340.0], [20.0, 340.0]]
    }
  ],
  "time": 55,
  "timestamp": 1710000000
}
Field Description
type Symbology name from ZXing, e.g. QR_CODE, EAN_13, CODE_128, DATA_MATRIX, PDF_417, AZTEC, UPC_A, CODE_39, …
text Decoded payload
box Corner points [[x,y],…] in image pixels (may be 2–4+ points)

Supported formats (ZXingCpp / zxing-cpp + light OpenCV QR fallback): QR, Aztec, Data Matrix, PDF417, EAN-8/13, UPC-A/E, Code 39/93/128, Codabar, ITF, etc.

Success (data.format = text)

{
  "code": 100,
  "data": "line one\nline two",
  "time": 40,
  "timestamp": 1710000000
}

With ocr.barcode=true, barcodes is also returned; if OCR is empty, data is filled with barcode lines like [QR_CODE] payload.

No text detected

{
  "code": 101,
  "data": "No text detected",
  "time": 30,
  "timestamp": 1710000000
}

With ocr.barcode=true and nothing found: data is "未检测到文字或条码" and barcodes is [].

Call examples

PowerShell (JSON base64):

$b64 = [Convert]::ToBase64String([IO.File]::ReadAllBytes("D:\sample.png"))
$body = @{
  base64 = $b64
  options = @{
    "data.format" = "text"
    "ocr.maxSideLen" = 1600
  }
} | ConvertTo-Json -Depth 5
Invoke-RestMethod -Uri "http://127.0.0.1:1224/api/ocr" -Method Post `
  -ContentType "application/json; charset=utf-8" -Body $body

curl (multipart):

curl -s -X POST "http://127.0.0.1:1224/api/ocr" \
  -F "file=@sample.png" \
  -F 'options={"data.format":"dict","ocr.angle":true}'

Python:

import base64, json, urllib.request

with open("sample.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode("ascii")

req = urllib.request.Request(
    "http://127.0.0.1:1224/api/ocr",
    data=json.dumps({
        "base64": b64,
        "options": {"data.format": "text", "ocr.maxSideLen": 1600},
    }).encode("utf-8"),
    headers={"Content-Type": "application/json"},
    method="POST",
)
print(urllib.request.urlopen(req).read().decode("utf-8"))

7. ASR (speech recognition)

Uses either an offline model under asrmodels or an installed Windows system recognizer (System.Speech). Sherpa streaming models are for live paths only; HTTP uses offline recognition.

7.1 GET /api/asr/models

{
  "code": 100,
  "data": [
    {
      "name": "sensevoice-small",
      "type": "SenseVoice",
      "streaming": false,
      "sample_rate": 16000,
      "culture": ""
    }
  ],
  "count": 1
}

type is Windows for a system recognizer, otherwise SenseVoice / Paraformer / Transducer / Whisper / ZipformerCtc. Windows entries include a BCP-47 culture. Use streaming for live vs offline.

7.2 POST /api/asr

Request JSON:

Field Required Description
base64 One of two Audio base64 (wav/mp3/flac/webm, …; decoded server-side)
path One of two Server-local absolute audio path (local debugging only)
filename No Hint for extension, e.g. a.wav
model / asr_model No Model display name; accepts a Sherpa model or Windows recognizer; default = config / first offline entry
lang No Default auto (SenseVoice: zh/en/ja/ko/yue, …)
itn No Model ITN, default true
postprocess No Rule post-process after recognition, default true; false to skip
device / compute No auto / gpu / cpu / igpu; ignored by Windows ASR

Success response:

{
  "code": 100,
  "data": {
    "text": "recognized text",
    "model": "sensevoice-small",
    "provider": "CUDA",
    "sample_rate": 16000,
    "audio_sec": 3.2,
    "load_ms": 120,
    "recognize_ms": 80,
    "postprocess": true
  },
  "time": 250,
  "timestamp": 1710000000
}

Example:

# Local path readable by the server process
curl -s -X POST "http://127.0.0.1:1224/api/asr" \
  -H "Content-Type: application/json" \
  -d "{\"path\":\"D:/audio/test.wav\",\"lang\":\"zh\"}"

8. TTS (speech synthesis)

Four engines: Sherpa (ONNX packs under ttsmodels), SAPI (classic System.Speech, including x86-only voices via x86host.exe), Windows (engine=winrt, WinRT / OneCore neural voices), and Edge online (engine=edge, no model/API key, Internet required). Omitting engine keeps the Sherpa path when a Sherpa model exists; otherwise it uses Windows speech when a voice is installed, or Edge online when none is.

Chrome (engine=chrome) speaks in a hidden local Chrome and does not return WAV. Speech and translation share that one process, and several requests run at the same time. Speaking starts immediately. If Chrome cannot speak, that language's offline voice pack is installed and the text is spoken once more. If it still cannot speak, the call returns an error. The desktop shows a progress window you can cancel during that install.

8.1 GET /api/tts/engines · GET /api/tts/models

GET /api/tts/engines returns the four engines and does not scan models or voices.

engine name
sherpa Sherpa
sapi SAPI
winrt Windows
edge Edge Online

GET /api/tts/models lists models and voices. When the desktop app is open, the list is the one the speech tab already loaded, including every speaker, and the call returns at once with cached true. Otherwise, with no parameters, it scans everything and keeps that result. The next full query returns the cache. refresh=1 (or true) scans again. A full rescan replaces the cache.

engine returns one engine: sherpa, sapi, winrt (windows is accepted), or edge. If a full list is already cached, the filter uses that cache. An unknown engine returns 802.

{
  "code": 100,
  "data": [
    {
      "name": "vits-zh",
      "engine": "sherpa",
      "type": "Vits",
      "speakers": [
        { "id": 0, "name": "speaker0", "label": "speaker0", "lang": "zh", "gender": "" }
      ]
    },
    {
      "name": "SAPI",
      "engine": "sapi",
      "type": "Sapi",
      "speakers": [
        { "id": 0, "name": "Microsoft Huihui Desktop", "label": "Microsoft Huihui Desktop · zh-CN · 女", "lang": "zh", "gender": "female", "key": "sapi:Microsoft Huihui Desktop" }
      ]
    },
    {
      "name": "Windows",
      "engine": "winrt",
      "type": "WinRt",
      "speakers": [
        { "id": 0, "name": "{voice-id}", "label": "Microsoft Hanhan · zh-TW · 女", "lang": "zh", "gender": "female", "key": "winrt:{voice-id}" }
      ]
    },
    {
      "name": "Edge Online",
      "engine": "edge",
      "type": "Edge",
      "speakers": [
        { "id": 0, "name": "ko-KR-SunHiNeural", "label": "SunHi · ko-KR · 女", "lang": "ko", "gender": "female", "key": "edge:ko-KR-SunHiNeural" }
      ]
    }
  ],
  "count": 4,
  "cached": false
}

label is the same readable name as the desktop list. name stays the raw id used for synthesis; for Windows voices that id is the voice id. Sherpa lists at most 64 speakers per model. SAPI / Windows list all installed voices; the Edge online catalog is fetched from Microsoft. x86-only SAPI voices have key prefix sapi-x86:.

8.2 POST /api/tts

Field Required Description
text Yes Text to synthesize, max 20000 characters
engine No sherpa / sapi / winrt / edge (plus aliases). Empty = infer from model / voice; Sherpa if a model is present, otherwise Windows speech or Edge online
model No Sherpa display name, or SAPI / Windows / Edge Online
voice / speaker No System/online voice name or key (sapi:… / sapi-x86:… / winrt:… / edge:…)
speaker_id / sid No Speaker index (Sherpa sid, or index into that engine’s list). Default 0
speed No Rate 0.5–2.0, default 1.0
volume No 0–100, default 100 (SAPI / Windows / Edge; ignored by Sherpa)
device / compute No Sherpa only: auto / gpu / cpu / igpu

Success response:

{
  "code": 100,
  "data": {
    "format": "wav",
    "sample_rate": 22050,
    "samples": 44100,
    "wav_base64": "<base64 of full WAV file>",
    "engine": "winrt",
    "model": "Windows",
    "voice": "winrt:{voice-id}",
    "speaker_id": 0,
    "provider": "WinRT"
  },
  "time": 300,
  "timestamp": 1710000000
}

provider is Sherpa’s EP (CPU / CUDA / …), or SAPI / SAPI x86 / WinRT / Edge Online. Decode wav_base64 to obtain standard WAV bytes.

import base64, json, urllib.request

req = urllib.request.Request(
    "http://127.0.0.1:1224/api/tts",
    data=json.dumps({"text": "Hello, world", "engine": "winrt", "speed": 1.0}).encode("utf-8"),
    headers={"Content-Type": "application/json"},
    method="POST",
)
data = json.loads(urllib.request.urlopen(req).read().decode("utf-8"))
open("out.wav", "wb").write(base64.b64decode(data["data"]["wav_base64"]))

CLI: ScreenKit --test-http-tts starts a loopback server, checks SAPI / Windows / Edge WAV output, and checks playback devices plus direct play.

8.3 GET /api/tts/devices · POST /api/tts/speak

GET /api/tts/devices lists active playback devices.

{
  "code": 100,
  "data": [
    { "id": "{0.0.0.00000000}.{device-id}", "name": "Speakers (Realtek)", "index": 0, "default": true }
  ],
  "count": 1
}

POST /api/tts/speak uses the same synthesis fields as /api/tts, then plays the audio on this PC. The response has no WAV. It returns when the clip is queued, not when playback ends.

Field Required Description
text Yes Same as /api/tts
engine / model / voice / speaker_id / speed / volume / device No Same as /api/tts. device is still the Sherpa compute device (auto / gpu / cpu / igpu), not a speaker
devices / outputs No One or more playback devices. Each item is an id, name, or index from the list above, or a unique fragment of the name. A single string may separate several with commas. Omit to use the default device
play / mode No queue (default: after the current clip and anything already waiting) or now (stop the clip that is playing and start this one; clips already waiting stay behind it)

Success response:

{
  "code": 100,
  "data": {
    "play": "queue",
    "ahead": 0,
    "devices": [
      { "id": "{0.0.0.00000000}.{device-id}", "name": "Speakers (Realtek)", "index": 0, "default": true }
    ],
    "sample_rate": 22050,
    "samples": 12000,
    "engine": "sapi",
    "model": "SAPI",
    "voice": "sapi:Microsoft Huihui Desktop",
    "speaker_id": 0,
    "provider": "SAPI"
  },
  "time": 180
}

ahead is how many clips are in front of this one. 0 means it starts now (the queue was empty, or this call is now). An unknown device, a name that matches more than one device, or a play value other than queue / now returns 802.

9. POST /api/itn

Inverse text normalization (WeText; rule post-process may still run if WeText is unavailable).

Request:

{ "text": "二零二六年七月二十五日" }

Response:

{
  "code": 100,
  "data": {
    "text": "2026年7月25日",
    "input": "二零二六年七月二十五日",
    "wetext": true
  },
  "time": 5,
  "timestamp": 1710000000
}
Field Description
text Normalized text
input Original input
wetext Whether WeText binary/resources are available

10. POST /api/translate · /api/translate/batch

LLM batch translate (grouped by translate_llm_batch, default 8; missing indexes retried one-by-one). No cap on how many items one request may send. Uses [[llm]] / translate_llm (or request llm) when that endpoint is ready. When it is not, the same request uses an installed local Opus-MT model for src→dst. Request engine forces a path — onnx (local model, picked by model), chrome, or llm; the default auto is the rule above. /api/translate/batch is the same handler. Large lists take time proportional to the number of groups (about 90 s max per group).

GET /api/translate/langs returns data.llm, data.src (language codes, auto first), data.dst (each source code maps to its target codes), plus the auto pair data.auto_native / data.auto_learn and its display name data.auto_label. auto translates between the native and learning languages: text that is not native goes to the native language, native text goes to the learning language. Defaults to Chinese ↔ English; change it in Settings → Translate. The same response also carries data.engines for pickers (auto / onnx / chrome, plus every LLM endpoint configured in Settings) and data.models (local ONNX models only, each with key, src, dst, name, ready).

Request:

{
  "src": "zh",
  "dst": "en",
  "items": [
    "单击目标窗口 · Esc 取消",
    "按 Ctrl+Alt+V 开始听写"
  ]
}
Field Description
items / texts String array, unlimited length. Elements may also be {"text":"…"}
text Single string; treated as a one-item batch
src / dst Language codes (zh / en / ja / ko / fr / de / es / ru / ar / th / cht / …). May be omitted or set to auto; the first non-empty item is detected against the native/learning pair
native / learn Optional overrides for the configured native / learning language (e.g. native: "zh", learn: "ja"); used only when src/dst are missing or auto
dir Optional pair such as zh-en
chunk Items per LLM call. Omitted uses translate_llm_batch (default 8). Range 1–64
engine Optional: auto (default), onnx, chrome, llm. onnx uses a local model even when an LLM is ready; llm requires a ready translate LLM
model With engine: "onnx", the local model: direction key (zh-en) or folder name (opus-mt-zh-en-onnx); the model's direction wins over src/dst. Otherwise an alias of llm
llm Optional [[llm]] display name or model id; default translate_llm, else first list entry

Response: data.items[] with i / text / out. data.miss is how many non-empty inputs came back empty. GET /api/status → llm_translate is true when a translate LLM is configured.

11. POST /api/chat

LLM chat. Text or audio user message; always returns text; WAV wav_base64 only when request tts: true (ignores config and other field names). Needs [[llm]] (chat_llm or request llm).

Field Required Description
text / message / content text or audio Current user text (max 8000)
base64 / path text or audio User speech; text wins if both set
messages no Prior user/assistant turns (not including current user)
tts no Only this field controls TTS; true → wav_base64, default/false → text only
agent no Tool agent; default from chat_agent
llm no Endpoint display name / model id
TTS / ASR fields no Same as /api/tts and /api/asr (engine, voice, asr_model, lang, …)

Success data: text/reply, user_text, llm_ms / asr_ms / tts_ms, optional wav_base64 + format/sample_rate/engine. TTS failure still returns text with tts_error. Codes: 960 no LLM, 961 missing input, 962 ASR fail, 963 chat fail. Status flag: llm_chat.

curl -s -X POST "http://127.0.0.1:1224/api/chat" \
  -H "Content-Type: application/json" \
  -d "{\"text\":\"hello\",\"tts\":true,\"engine\":\"sapi\"}"

12. Face

Needs det+rec ONNX under facemodels/. Download InsightFace buffalo_l from Help → Install features. Aliases: POST /api/face/compare, POST /api/face/extract (same handler).

12.1 GET /api/face/models

{
  "code": 100,
  "data": {
    "root": ".../facemodels",
    "ready": true,
    "det": ["det_10g.onnx"],
    "rec": ["w600k_r50.onnx"],
    "landmark": ["2d106det.onnx", "1k3d68.onnx"],
    "attr": ["genderage.onnx"]
  },
  "count": 5
}

12.2 POST /api/face

One image: detect the largest face and extract an embedding. Two images: cosine similarity and same-person decision.

JSON:

Field Required Description
base64 / image / path at least one First image (base64 or server-local path)
base64_b / image_b / path_b for compare Second image
det / reg no Det/rec file names (fuzzy; default config or first in folder)
threshold no Compare threshold (config default, about 0.5)
device / compute no auto / gpu / cpu / igpu
attr / genderage no Run gender/age (true by default if genderage.onnx exists)
include_feature no Include embedding vector when true

multipart: file + file2 (or image / image_b), plus optional scalar fields.

One-image success:

{
  "code": 100,
  "data": {
    "faces": 1,
    "det": "det_10g.onnx",
    "reg": "w600k_r50.onnx",
    "provider": "CUDA",
    "face": {
      "score": 0.91,
      "box": [80.0, 40.0, 220.0, 210.0],
      "landmarks5": [[x, y], "..."],
      "gender": "女",
      "age": 22
    }
  },
  "time": 80
}

Two-image compare:

{
  "code": 100,
  "data": {
    "similarity": 0.7234,
    "match": true,
    "threshold": 0.5,
    "det": "det_10g.onnx",
    "reg": "w600k_r50.onnx",
    "provider": "CUDA",
    "left": { "score": 0.91, "box": [80, 40, 220, 210], "gender": "女", "age": 22 },
    "right": { "score": 0.88, "box": [70, 30, 200, 200], "gender": "男", "age": 26 }
  },
  "time": 150
}

No face: code=101.

curl -s -X POST "http://127.0.0.1:1224/api/face" \
  -F "file=@left.jpg" -F "file2=@right.jpg" -F "threshold=0.5"

12. QR / barcode

Dedicated barcode scan — does not run OCR. Same ZXingCpp pipeline as the result-panel Barcode tab (QR, Aztec, Data Matrix, PDF417, EAN-8/13, UPC-A/E, Code 39/93/128, Codabar, ITF, plus a light OpenCV QR fallback). ZXing.dll is not in the release archive. Help → Install features downloads it. A request without that file returns code 950. Aliases: POST /api/qrcode (Umi-OCR’s QR path), POST /api/barcode, POST /api/barcodes (same handler). The body is still base64 or path. Each hit has both type and format; a QR code’s value is QRCode. Combined OCR+barcode remains POST /api/ocr with options.ocr.barcode.

12.1 POST /api/qr

JSON:

{
  "base64": "<image base64>",
  "path": "D:\\sample.png",
  "format": "dict"
}
Field Required Description
base64 one of Image bytes (png/jpg/bmp/webp, …)
path one of Server-local file path (same caveat as ASR/face)
image / img — Aliases of base64
format no dict (default) or text; also data.format or options.data.format

multipart: file field file / image / img / upload, same as OCR.

Success (format=dict):

{
  "code": 100,
  "data": [
    {
      "type": "QRCode",
      "format": "QRCode",
      "text": "https://example.com",
      "box": [[12.0, 40.0], [180.0, 40.0], [180.0, 210.0], [12.0, 210.0]]
    }
  ],
  "count": 1,
  "time": 18,
  "timestamp": 1710000000
}

type / format / text / box match the barcodes[] objects on POST /api/ocr. Umi-OCR clients read data[].format and data[].text.

Success (format=text): data is lines like [QRCode] https://example.com.

None found: code=101, count=0; data is "未检测到条码或二维码" (dict) or "" (text). Scan errors: 950.

curl -s -X POST "http://127.0.0.1:1224/api/qr" -F "file=@code.png"
import json, base64, urllib.request
b64 = base64.b64encode(open("code.png", "rb").read()).decode("ascii")
req = urllib.request.Request(
    "http://127.0.0.1:1224/api/qr",
    data=json.dumps({"base64": b64}).encode("utf-8"),
    headers={"Content-Type": "application/json"},
    method="POST",
)
print(json.loads(urllib.request.urlopen(req).read().decode("utf-8")))

13. Relationship to the main window

Behavior Description
OCR model With ocr.engine and ocr.language omitted, follows the main window. ocr.engine=winocr (or ocr.language=winocr) uses Windows OCR. Its language is ocr.language, or the main window win_ocr_langs when omitted, and no ONNX model is loaded. When the main window is already Windows OCR, ocr.engine=onnx or an ONNX variant title switches back to a model pack. ocr.maxSideLen applies only to ONNX. Windows OCR does not shrink to it. The language still defaults to the main window win_ocr_langs
Device May override with ocr.device; otherwise uses main-window device
Service mode service_mode = true preloads engines for frequent API calls and does not unload them on idle
Idle unload onnx_unload_min (default 5) unloads OCR, translation, face, speech recognition, and speech synthesis after that many idle minutes. 0 disables it. Ignored while service mode is on
Parameter changes Changing model/device in the UI invalidates engines; next request reloads

14. Security notes

  1. Default is http_lan = true (every interface). Set http_lan = false to listen on 127.0.0.1 only. Do not expose the port on an untrusted network; there is no firewall rule or authentication built in.
  2. No auth, no HTTPS — trust only the local machine or a controlled LAN.
  3. POST /api/asr, POST /api/face, and POST /api/qr path / path_b read server-local files; never expose this to untrusted clients.
  4. Large images / long audio use CPU/GPU and memory; watch concurrency (requests run via Task.Run; engines use locks).

15. PC file transfer (LAN, pairing required)

Shares the HTTP API port (default 1224). Enable under Settings → API → PC file transfer. Phones and browsers on the LAN also need Allow LAN access (http_lan = true). UDP discovery 17531. Files are restricted to sendfile/ next to the exe.

GET /apk needs no pairing (phone scans the QR before the app is installed). GET /files and GET /m are the web file manager (desktop / phone). GET / is the tools page, not the file manager. GET /f/<rel> downloads without login. Web upload/list/delete need a login (cookie sk_web or X-Web-Token). Other phone routes after pairing: X-Device-Id + Authorization: Bearer <token>.

Discovery: UDP broadcast SCREENKIT_DISCOVER to port 17531; the PC replies with JSON {v,name,httpPort,pcId}.

Method Path Notes
GET /files Desktop web page (mobile UA redirects to /m; ?pc=1 forces desktop)
GET /index.html Redirects to /files
GET /m Phone web page
GET /web/app.js · /web/d.css · /web/m.css Page assets
GET /f/<rel> Public download (no login; path must stay inside sendfile/)
GET /d?p= Public download (query)
GET /apk APK bytes from this PC (no pairing; File sync Install on phone QR)
POST /api/web/login body {password, keep?} → cookie sk_web + {token}. keep=true stores a 30-day cookie (Max-Age and Expires) on disk and still works after the program restarts
GET/POST /api/web/zip?path= · ?paths=a|b Zip download (login required; file or folder)
POST /api/web/logout Clear session
GET /api/web/me Whether signed in
GET /api/web/list?path= List (login)
POST /api/web/upload?path= Raw-body upload (login)
POST /api/web/mkdir body {path} (login)
POST /api/web/rename body {from,to} (login)
DELETE /api/web/delete?path= Delete (login)
GET /api/web/text Current text on the PC File sync page {text} (login)
POST /api/web/text body {text} writes that box; an empty string clears it (login, max 65536)
POST /api/sendfile/pair body {id,name}; first time shows a confirm dialog on the PC
GET /api/sendfile/info name / pcId
GET /api/sendfile/list?path=&deep= deep=1 recursive
GET /api/sendfile/download?path= octet-stream
POST /api/sendfile/upload?path= raw body
POST /api/sendfile/mkdir body {path}
DELETE /api/sendfile/delete?path= file or directory
POST /api/sendfile/text body {text} → File sync tab text list
GET /api/sendfile/text?since= messages from PC to this phone
GET /api/sendfile/pull files queued from the PC for this phone {items:[{id,path,rel,name,size}]} (path is for download)
POST /api/sendfile/pulldone body {id}; ack and delete the PC staging copy

JSON code 100 = success; 401/403 unpaired; 410 bad path.

Android app: android/README.md.

16. Related

  • Implementation: ScreenKit/Ocr/HttpOcrServer.cs · HttpOcrServer.Face.cs · HttpOcrServer.Translate.cs · HttpOcrServer.Qr.cs
  • Config: config.toml (http_enabled / http_lan / http_port / service_mode)
  • Overview: README.md · README.zh.md