LANGSMITH

LangSmith LangChain LLM observability + eval + test.

Définition

LangSmith key features : (1) Tracing : auto-instruments LangChain + LlamaIndex via environment variables (LANGCHAIN_TRACING_V2 + LANGCHAIN_API_KEY), instruments LLM calls + chains + tools + agents + retrievers, web UI shows tree-based traces with latency + tokens + cost + inputs/outputs per node. (2) Datasets : create + manage labeled datasets for evaluation, import from JSON + CSV + Hugging Face + production traces, support paired comparison datasets. (3) Evaluations : run evaluations against datasets (test set), evaluator types : (a) Built-in evaluators (Exact Match + COT correctness + criteria-based LLM-as-Judge), (b) Custom evaluators Python functions, (c) Pairwise evaluators (compare 2 versions side-by-side), (d) LLM-as-Judge templates (helpfulness + correctness + conciseness + harmfulness). (4) Prompt Hub : prompt versioning + sharing, A/B testing prompts, integrate prompts directly LangChain code via langsmith_client.pull_prompt. (5) Monitoring : production metrics dashboards (latency p50/p95/p99 + cost + errors + custom metrics) per project, alerting rules. (6) Annotation Queues : Human-in-the-Loop queue review + label production traces, integrate dataset for re-training. (7) Self-hosted : Enterprise tier supports self-hosted deployment Kubernetes Helm chart. Customers : ~10000+ paying customers 2024, OpenAI internal projects + Anthropic + Apple + Square + Doordash.

Origine

LangSmith announced juillet 2023 ; private beta -> public general availability fevrier 2024 ; ~10000+ paying customers 2024 ; pricing free tier 5K traces/month + paid tiers $39/seat/month Plus + Enterprise custom.

Exemple en contexte

Enterprise customer support AI uses LangChain agent + LangSmith production : 5M+ user queries/month, LangSmith traces every agent execution (tools called + LLM costs + latency), evaluations run nightly on golden dataset 1000 examples GPT-4 vs Claude responses, Prompt Hub manages production prompt versions A/B test, Annotation Queue reviews 100/day low-confidence traces.

Termes liés

• Helicone — alternative LLM observability OSS.