ARGILLA

Argilla LLM data labelling Hugging Face acquired.

Définition

Argilla key features : (1) Workspaces + Datasets : organize labeling projects by team workspace, multiple datasets per workspace. (2) Question types : labelling questions various types - TextQuestion (free text), RatingQuestion (1-5 stars Likert), LabelQuestion (single choice classification), MultiLabelQuestion (multiple choice), RankingQuestion (rank items by preference), SpanQuestion (annotate spans within text NER-style). (3) Records : individual examples to label, support fields (text + multiple texts comparison + Markdown content + images soon). (4) Guidelines : labelling guidelines written per dataset, shown labelers context. (5) Status : Draft / Submitted / Discarded states per record per labeler. (6) Suggestions : LLM-generated suggestions (zero-shot prompt result), labeler accepts/rejects/edits, accelerates labelling 5-10x. (7) Distributed labelling : multiple labelers same dataset, agreement metrics + Inter-Annotator Agreement (IAA) Cohen Kappa scores. (8) Integration Hugging Face : Argilla datasets pushable to HF Hub + AutoTrain fine-tuning + Spaces deploys. (9) RLHF preference data : RatingQuestion + ranking for fine-tuning DPO Direct Preference Optimization + PPO + RLHF Reward Models. Use cases : custom fine-tuning data curation, RLHF preference labelling, evaluation dataset building.

Origine

Argilla SAS fondee 2021 a Madrid Espagne par Daniel Vila + Francisco Aranda (ex-Recogn.io) ; Seed funding 2022 ; acquise par Hugging Face juin 2024 (terms undisclosed) ; ~3000+ GitHub stars 2024.

Exemple en contexte

Open-source LLM fine-tuning project uses Argilla : curate ~50000 instruction-tuning examples for fine-tuning Mistral 7B model, multiple labelers review + rate + edit synthetic LLM outputs, push curated dataset Hugging Face Hub, AutoTrain fine-tunes model + deploys HF Spaces ; collaborative iterative process model quality improvement.

Termes liés

• MLflow — MLOps tool complement.