分類
LLM evals & observability — traces, scoring, live monitoring
Treat LLM apps like production systems: offline evals, live traces, and metrics at scale. This hub covers major eval/observability platforms and OSS options.
The gap between “I tuned this prompt once” and “I can ship a change and watch 200 regressions pass plus P99 latency stay flat” is filled by this category. Key comparisons: custom scorer support, unified **trace + eval + replay**, and whether offline/online share one dataset. Depth of LangChain/LlamaIndex/OpenAI SDK integration is another frequent deciding factor.
檢索與補充說明
LangSmith vs Langfuse vs Braintrust
LangSmith leans into LangChain; Langfuse is self-hostable OSS; Braintrust is eval-first. Run one real pipeline through two of them for two weeks and see which tab engineers actually open.
How do I eval a RAG stack?
Typically retrieval metrics (recall/precision/nDCG) + generation scores (correctness, faithfulness, groundedness), topped with human spot-checks. Look for built-in LLM-as-judge and golden dataset management.
Monitoring LLMs in production
Track P50/P95 latency, token cost distribution, failure rate, and PII leakage. Confirm log retention and training-use clauses on each vendor site.
本類工具
簡介與官網以各工具詳情頁為準；可在同類條目間交叉瀏覽。
LangChain 團隊的 LLM 評測與 trace 平台——資料集、評分器、線上監控與人工標註，與 LangChain/LangGraph 整合最深。
開源 LLM 可觀測與評測平台：trace、資料集、評分器與提示管理；可 Docker 自部署，把資料留在內網。
Braintrust：常見的 AI 產品——功能、價格、支援地區、資料處理與最新模型，請以官網說明為準。
Arize Phoenix：常見的 AI 產品——功能、價格、支援地區、資料處理與最新模型，請以官網說明為準。
Helicone：常見的 AI 產品——功能、價格、支援地區、資料處理與最新模型，請以官網說明為準。
Galileo：常見的 AI 產品——功能、價格、支援地區、資料處理與最新模型，請以官網說明為準。
Patronus AI：常見的 AI 產品——功能、價格、支援地區、資料處理與最新模型，請以官網說明為準。