MLFLOW-LLM

MLflow LLM tracking + eval + serving Databricks.

Definition

MLflow LLM features: (1) MLflow Tracking LLM: log LLM runs with prompts + responses + parameters (temperature + max_tokens + top_p) + metrics (latency + token counts + cost estimate), MLflow Trace API (auto-instrument LangChain + LlamaIndex + OpenAI + Anthropic clients) for distributed tracing. (2) MLflow Models LLM: package LLM models for deployment (mlflow.openai.log_model, mlflow.transformers.log_model HuggingFace, mlflow.langchain.log_model entire chain, etc.), serve via mlflow models serve or deploy Databricks Model Serving. (3) MLflow Evaluate: evaluate LLM applications via mlflow.evaluate(data, model_type='question-answering') with metrics: exact_match, token_count, ari_grade_level (readability), flesch_kincaid_grade_level, toxicity (HuggingFace toxicity model), perplexity, custom metrics support via make_metric. (4) MLflow Deployments Gateway: unified API gateway access LLM providers (OpenAI + Azure OpenAI + Anthropic + Cohere + AI21 + AWS Bedrock + GCP Vertex AI + custom), token tracking + rate limiting + caching. (5) MLflow Prompt Engineering UI: Databricks Notebook UI for iterative prompt experimentation, side-by-side prompts comparison. (6) Integration: Databricks Mosaic AI (ML platform), AzureML (similar via Microsoft), AWS SageMaker (MLflow Open Source compatible).

Origin

MLflow initial release June 2018 by Databricks (Matei Zaharia + Spark team) ; MLflow LLM extension 2.4 May 2023 ; MLflow 2.10 February 2024 LLM enhancements ; ~17000 GitHub stars 2024.

Example in context

Databricks customer enterprise AI engineering team uses MLflow LLM Tracking + Evaluate: prompts iterated in Prompt Engineering UI, evaluations run via mlflow.evaluate against eval dataset (1000+ QA examples), tracks 50+ experiments versions prompts + models (GPT-4 vs Claude vs Llama 70B fine-tuned), best version deployed via Databricks Model Serving + MLflow Deployments Gateway.

Related terms

• Weights & Biases — alternative ML tracking.