Feature request: AI next-word suggestions (on-device first)

[Lloyd-Jackman-UKPL]

Is your feature request related to a problem?
Next-word suggestions come only from the AOSP n-gram engine, which takes weeks/months of personal typing before suggesting anything beyond dictionary headwords. The AI runtime LeanType already ships (cloud providers + on-device GGUF via llama.cpp) is used only for reactive tasks (proofreading, translation, custom AI keys) and never for prediction. LLM-based next-word prediction would give contextually relevant suggestions from day one (names, places, per-app topics) instead of after long usage.

Describe the solution you'd like
An async AI candidate source that produces next-word candidates, fed into the existing suggestion pipeline like any other dictionary source (UserBinaryDictionary pattern). No behavior change when no model/key is configured — the feature is simply off.

Primary: on-device (Offline flavor)

• Reuse the GGUF/llama.cpp runtime already shipped in the Offline build (Settings → Advanced → GGUF Model) — zero network, zero cost, nothing leaves the phone.
• Default model: a small Q4_K_M GGUF (~1–1.5GB), e.g. Qwen2.5-1.5B (the docs already recommend this class).
• Users may drop in their own fine-tuned GGUF (e.g. QLoRA on their own message history) — no code needed, just a model file.

Fallback: existing AI provider slot (provider-agnostic)

• Same feature works through the OpenAI-compatible provider on standard builds — Gemini, Groq, OpenRouter, or a LAN Ollama endpoint — no extra infrastructure.

Trigger design (gesture-typing friendly)

• Fire on word-commit events (space, suggestion tap, gesture-typed word) — exactly when the existing AOSP next-word hook runs; the LLM source augments that same stage, no separate event handling.
• The suggested candidate is pinned to the current position and stays visible until the next commit, so fast swipe typers can still catch it mid-way through the next word; it is dropped and re-fired on the next commit.
• In-flight requests are cancelled when the position moves on.
• Minimum 1.5s between requests to avoid spam on rapid typing.
• Never fires in sensitive fields (password, etc.); per-app opt-out list.
• Strictly no network in the Offline flavor (it has none by design); async and non-blocking everywhere — AOSP suggestions always remain primary.

Phase 2 — per-app context memory

• Rolling, device-local buffer per package name (last ~2 completed messages + current field), following the per-app keying already used for remembered locales.
• A short window (not the archive) is injected as context — minimal exposure even with a cloud provider.

Use case

• Personal vocabulary and names (family, pets, places — e.g. "Burze", "Dobrogoszcz") suggested immediately, instead of after months of learning.
• Per-app continuity: in Signal we're talking about the vet, in Slack about the deploy.
• Offline-first users get prediction that never leaves the device.

Alternatives considered

• Dictionary import / bigram seeding files — words only, no bigrams parsed, not context-aware.
• Existing "prioritize personal dictionary / next-word boost" settings — helpful, but n-gram-bound.
• Third-party keyboards with LLM prediction — none exist in FOSS; Gboard's neural predictor is proprietary, context-blind.

App version & Flavor
v4.1.1 (currently F-Droid standard; targeting the offline flavor for on-device).

[LeanBitLab]

I’m concerned about the potential performance impact of running on-device AI during every word commit. If the LLM inference isn’t handled with extreme efficiency, won't it introduce input lag or stuttering that degrades the typing experience?

[AZADAYAZ]

We need to test whether phones with high RAM can handle this.
Even if it is positive, by restricting the use of this feature to the minimum RAM requirement, low RAM users are prevented from using it.

[Lloyd-Jackman-UKPL]

So honestly, I've withdrawn the PR as whilst the predictions genuinely helped in topping up the AOSP suggestions, I noticed some unexpected behaviour where under some conditions it would auto commit the AI suggestion over the input, which is unforgivable for a keyboard app.

The performance though was remarkably good in terms of speed and utility running Qwen 2.5 1.5B so a fairly lightweight model. I could certainly have gone with a larger model on my hardware (S24+) but that wasn't my intention.

For now I need to deal with that auto commit behaviour as whilst it only affected users with the feature turned on (normal behaviour resumed when turned off) I didn't want to risk that bug going any further.

[LeanBitLab]

So honestly, I've withdrawn the PR as whilst the predictions genuinely helped in topping up the AOSP suggestions, I noticed some unexpected behaviour where under some conditions it would auto commit the AI suggestion over the input, which is unforgivable for a keyboard app.

The performance though was remarkably good in terms of speed and utility running Qwen 2.5 1.5B so a fairly lightweight model. I could certainly have gone with a larger model on my hardware (S24+) but that wasn't my intention.

For now I need to deal with that auto commit behaviour as whilst it only affected users with the feature turned on (normal behaviour resumed when turned off) I didn't want to risk that bug going any further.

• since it always need to run while typing and stay loaded in background, unlike the voice input and proofreading, ai suggestion will create memory leak/crashing issues on long run,
• maybe we could try if there is an lightweight model specific to the suggestion task or something released later
• for now ill try to find an very lightweight way to improve the suggestion, gonna start with optimising the current suggestion methods first, may we can add an slider which can control how personal the suggestion would be or something, we ll find a way