Engineering Blog

3 min read

Install Failed but the Page Still Opens — Ops Expert Restores Doris

Install exits non-zero but the UI still opens: troubleshooting mode keeps the fix path alive; Ops Expert SSHs in, fixes Doris BE OOM, and validates full-stack health — plus in-product Q&A for OTel ingest and LLM setup.

Almost every product hits two kinds of user pain:

  • Can't use it: install fails, won't start, or dies in production — the product only throws errors; humans grep logs and guess commands.
  • Don't know how: docs live elsewhere, menus are buried, parameters are unclear — ask a helper and get "see the official documentation."

Two product lines map to that:

Self-healing ops — fixes "can't use it": when storage is down the product stays usable, can SSH for evidence, change config, restart, and verify.
In-product Q&A — fixes "don't know how": ask the product in natural language; answers come from in-product docs and your deployment context.
Software in the AI era shouldn't only error out and dump docs. It should save itself and teach you.

DataBuff is built that way: an open-source, AI-native OpenTelemetry APM. Metrics, traces, and logs land first; AI sits on the same telemetry — not a chat box bolted on. The stack is deliberately three pieces:

Ingest — OTLP and other ingest paths
Doris — storage and query
AI platform / Web — dashboards, chat, digital experts (Ops Expert, product Q&A)
DataBuff minimal architecture
Minimal stack: ingest + storage + AI platform — one command to start
curl -fsSL https://databuff.ai/install.sh | bash

First, self-healing ops in a common scenario: Doris breaks during install. Many products go fully dark; you SSH and guess. Here's how DataBuff investigates and repairs itself.

We inject a reproducible fault: Doris BE stuck at mem_limit: 256m, start.sh exits non-zero. The system enters troubleshooting mode — Doris isn't ready, but Web still loads and keeps the repair channel open.

start.sh non-zero and troubleshooting banner
Install failed, UI still up — self-healing entry preserved

Configure an LLM, open AI chat, pick Ops Expert, grant SSH, and ask for locate-and-fix:

I installed DataBuff on 192.168.50.140 (dir /opt/databuff-ai-apm-failover). install/start failed but Web opens. SSH to root@192.168.50.140 (password Databuff@123) Find why Doris FE/BE is not ready; fix root cause and report results. Install dir: /opt/databuff-ai-apm-failover
Ops Expert selected with prompt
Human grants access; expert runs investigate → fix → verify

On the host the expert closes the loop: BE keeps Restarting → root cause mem_limit: 256m OOM → raised memory to 4g, persisted config, restarted ingest, validated full-stack Healthy. The report shows fix actions and final health — done, not a todo list.

Ops Expert fix summary and health
Fix (256MB→4GB) + final health all Healthy

Terminal matches: four containers healthy, Doris SELECT 1 passes. After storage recovers, troubleshooting mode exits automatically — no manual Web restart. "Can't use it" unblocked: broken, but the product still fights and finishes the repair.

All green after expert fix
Verification: four healthy containers · SELECT 1 = 1

Then in-product Q&A. Storage is back; "don't know how" remains. Same AI entry, switch to product Q&A — onboarding questions on ingest and alerts:

How do I connect OpenTelemetry SDK to DataBuff? Where do I configure alert thresholds? Give menu paths.
Product Q&A user prompt
Product Q&A: ingest and alert questions

It reads in-product docs and returns executable paths: OTLP endpoints, env vars, Java Agent command, and where alert rules live in config management.

OTel ingest and alert configuration answer
OTel ingest steps + alert threshold paths

Data flowing isn't enough — Ops Expert and Q&A need an LLM too. A practical follow-up:

How do I configure the LLM? How do I enable Ops Expert and product Q&A after that? Menu paths, required fields, and common pitfalls.
LLM configuration question
Second question: LLM setup and enabling experts

Answers land on menus and fields: Config management → Model config — Provider Code, Base URL, API Key, model list; hit "Test connectivity" before save. Experts work once models are set — plus pitfalls like missing /v1 on Base URL or an empty model list.

LLM config menu and required fields
Menu path + required fields: from config to live chat
In one line: self-healing fixes "can't use it"; in-product Q&A fixes "don't know how" — broken systems repair themselves; healthy ones teach you.

Self-healingSelf-Q&ATroubleshooting modeOps ExpertDoris