ARGILLA

Argilla LLM data labelling Hugging Face acquired.

Definition

Argilla key features: (1) Workspaces + Datasets: organize labeling projects by team workspace, multiple datasets per workspace. (2) Question types: labelling questions various types - TextQuestion (free text), RatingQuestion (1-5 stars Likert), LabelQuestion (single choice classification), MultiLabelQuestion (multiple choice), RankingQuestion (rank items by preference), SpanQuestion (annotate spans within text NER-style). (3) Records: individual examples to label, support fields (text + multiple texts comparison + Markdown content + images soon). (4) Guidelines: labelling guidelines written per dataset, shown labelers context. (5) Status: Draft / Submitted / Discarded states per record per labeler. (6) Suggestions: LLM-generated suggestions (zero-shot prompt result), labeler accepts/rejects/edits, accelerates labelling 5-10x. (7) Distributed labelling: multiple labelers same dataset, agreement metrics + Inter-Annotator Agreement (IAA) Cohen Kappa scores. (8) Integration Hugging Face: Argilla datasets pushable to HF Hub + AutoTrain fine-tuning + Spaces deploys. (9) RLHF preference data: RatingQuestion + ranking for fine-tuning DPO Direct Preference Optimization + PPO + RLHF Reward Models. Use cases: custom fine-tuning data curation, RLHF preference labelling, evaluation dataset building.

Origin

Argilla SAS founded 2021 in Madrid Spain by Daniel Vila + Francisco Aranda (ex-Recogn.io) ; Seed funding 2022 ; acquired by Hugging Face June 2024 (terms undisclosed) ; ~3000+ GitHub stars 2024.

Example in context

Open-source LLM fine-tuning project uses Argilla: curate ~50000 instruction-tuning examples for fine-tuning Mistral 7B model, multiple labelers review + rate + edit synthetic LLM outputs, push curated dataset Hugging Face Hub, AutoTrain fine-tunes model + deploys HF Spaces ; collaborative iterative process model quality improvement.

Related terms

• MLflow — MLOps tool complement.