Web & Dev

AI Dataset Formats

Common dataset formats used for training and evaluating AI — JSONL, Parquet, HuggingFace datasets, and more.

Formats

FormatBest forNotes
CSVSmall tabular dataSimple, universal; type info lost
TSVText with commasSame as CSV but tab-delimited
JSONNested or hierarchical dataVerbose; whole-file parse
JSONL (ND-JSON)Streaming records, LLM fine-tuningOne JSON object per line
ParquetColumnar analyticsCompressed, efficient column reads
Feather / Arrow IPCFast in-memory handoffZero-copy to pandas / polars
HDF5Multidimensional arraysCommon in older ML (Keras)
NumPy .npy / .npzRaw arraysSimple dump / restore
TFRecordTensorFlow pipelinesProtocol Buffer records
WebDataset (tar)Large-scale vision / multimodalStreams shards from disk or cloud
HuggingFace datasetsTraining / evaluationMemory-mapped Arrow backend
MLflow model / signatureServingMetadata + artifacts
SafetensorsModel weights (not data)Secure alternative to pickle

LLM fine-tuning formats

// OpenAI chat fine-tune (JSONL)
{"messages": [
  {"role": "system", "content": "You are a helpful assistant."},
  {"role": "user", "content": "What's the capital of France?"},
  {"role": "assistant", "content": "Paris."}
]}

// Alpaca-style (JSONL)
{"instruction": "...", "input": "...", "output": "..."}

// Preference (DPO)
{"prompt": "...", "chosen": "...", "rejected": "..."}

Picking a format

  • Tabular, < 1 GB: CSV or Parquet.
  • Tabular, > 1 GB: Parquet — columnar reads + compression.
  • Records, streaming: JSONL.
  • Images / video at scale: WebDataset shards in tar.
  • Cross-team reproducibility: HuggingFace datasets or versioned Parquet.

Worked example: why JSONL and Parquet

A JSON file holding one array of 1,000,000 records must be parsed as a single document, so the whole thing sits in memory before the first record is usable. A JSONL file stores one object per line, so a loader can read, validate and discard one line at a time, and a corrupt line costs one record rather than the file. Line breaks inside a record must be escaped inside strings, so a pretty-printed JSON object spread over several lines is not valid JSONL.

Parquet stores data by column. If a table has 50 columns of similar width and a job needs two, a columnar reader touches roughly 2 / 50 = 4 percent of the bytes, while a CSV reader must scan every row in full. Compression also works better on a single column of similar values. The trade-off is that Parquet is not human-readable and not suited to appending one row at a time.

Common mistakes

  • Losing leading zeros. A ZIP code such as 02134 read from CSV as a number becomes 2134. Declare identifier columns as text.
  • Mixed encodings. Save text datasets as UTF-8 without a byte-order mark, or the first field name may carry an invisible character.
  • Duplicates across splits. If near-identical records appear in both training and evaluation data, scores are inflated. Remove duplicates before splitting.
  • Loading untrusted pickles. Python pickle files can run code when loaded; prefer Safetensors for weights and Parquet or JSONL for data.
  • Inconsistent schemas. In JSONL, a field that is a string in one row and a number in another breaks many loaders. Validate every line.
Was this article helpful?