LANGSMITH

LangSmith LangChain LLM observability + eval + test.

Definition

LangSmith key features: (1) Tracing: auto-instruments LangChain + LlamaIndex via environment variables (LANGCHAIN_TRACING_V2 + LANGCHAIN_API_KEY), instruments LLM calls + chains + tools + agents + retrievers, web UI shows tree-based traces with latency + tokens + cost + inputs/outputs per node. (2) Datasets: create + manage labeled datasets for evaluation, import from JSON + CSV + Hugging Face + production traces, support paired comparison datasets. (3) Evaluations: run evaluations against datasets (test set), evaluator types: (a) Built-in evaluators (Exact Match + COT correctness + criteria-based LLM-as-Judge), (b) Custom evaluators Python functions, (c) Pairwise evaluators (compare 2 versions side-by-side), (d) LLM-as-Judge templates (helpfulness + correctness + conciseness + harmfulness). (4) Prompt Hub: prompt versioning + sharing, A/B testing prompts, integrate prompts directly LangChain code via langsmith_client.pull_prompt. (5) Monitoring: production metrics dashboards (latency p50/p95/p99 + cost + errors + custom metrics) per project, alerting rules. (6) Annotation Queues: Human-in-the-Loop queue review + label production traces, integrate dataset for re-training. (7) Self-hosted: Enterprise tier supports self-hosted deployment Kubernetes Helm chart. Customers: ~10000+ paying customers 2024, OpenAI internal projects + Anthropic + Apple + Square + Doordash.

Origin

LangSmith announced July 2023 ; private beta -> public general availability February 2024 ; ~10000+ paying customers 2024 ; pricing free tier 5K traces/month + paid tiers $39/seat/month Plus + Enterprise custom.

Example in context

Enterprise customer support AI uses LangChain agent + LangSmith production: 5M+ user queries/month, LangSmith traces every agent execution (tools called + LLM costs + latency), evaluations run nightly on golden dataset 1000 examples GPT-4 vs Claude responses, Prompt Hub manages production prompt versions A/B test, Annotation Queue reviews 100/day low-confidence traces.

Related terms

• Helicone — alternative LLM observability OSS.