Formats
| Format | Best for | Notes |
|---|---|---|
| CSV | Small tabular data | Simple, universal; type info lost |
| TSV | Text with commas | Same as CSV but tab-delimited |
| JSON | Nested or hierarchical data | Verbose; whole-file parse |
| JSONL (ND-JSON) | Streaming records, LLM fine-tuning | One JSON object per line |
| Parquet | Columnar analytics | Compressed, efficient column reads |
| Feather / Arrow IPC | Fast in-memory handoff | Zero-copy to pandas / polars |
| HDF5 | Multidimensional arrays | Common in older ML (Keras) |
| NumPy .npy / .npz | Raw arrays | Simple dump / restore |
| TFRecord | TensorFlow pipelines | Protocol Buffer records |
| WebDataset (tar) | Large-scale vision / multimodal | Streams shards from disk or cloud |
| HuggingFace datasets | Training / evaluation | Memory-mapped Arrow backend |
| MLflow model / signature | Serving | Metadata + artifacts |
| Safetensors | Model weights (not data) | Secure alternative to pickle |
LLM fine-tuning formats
// OpenAI chat fine-tune (JSONL)
{"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What's the capital of France?"},
{"role": "assistant", "content": "Paris."}
]}
// Alpaca-style (JSONL)
{"instruction": "...", "input": "...", "output": "..."}
// Preference (DPO)
{"prompt": "...", "chosen": "...", "rejected": "..."}Picking a format
- Tabular, < 1 GB: CSV or Parquet.
- Tabular, > 1 GB: Parquet — columnar reads + compression.
- Records, streaming: JSONL.
- Images / video at scale: WebDataset shards in tar.
- Cross-team reproducibility: HuggingFace datasets or versioned Parquet.
Worked example: why JSONL and Parquet
A JSON file holding one array of 1,000,000 records must be parsed as a single document, so the whole thing sits in memory before the first record is usable. A JSONL file stores one object per line, so a loader can read, validate and discard one line at a time, and a corrupt line costs one record rather than the file. Line breaks inside a record must be escaped inside strings, so a pretty-printed JSON object spread over several lines is not valid JSONL.
Parquet stores data by column. If a table has 50 columns of similar width and a job needs two, a columnar reader touches roughly 2 / 50 = 4 percent of the bytes, while a CSV reader must scan every row in full. Compression also works better on a single column of similar values. The trade-off is that Parquet is not human-readable and not suited to appending one row at a time.
Common mistakes
- Losing leading zeros. A ZIP code such as 02134 read from CSV as a number becomes 2134. Declare identifier columns as text.
- Mixed encodings. Save text datasets as UTF-8 without a byte-order mark, or the first field name may carry an invisible character.
- Duplicates across splits. If near-identical records appear in both training and evaluation data, scores are inflated. Remove duplicates before splitting.
- Loading untrusted pickles. Python pickle files can run code when loaded; prefer Safetensors for weights and Parquet or JSONL for data.
- Inconsistent schemas. In JSONL, a field that is a string in one row and a number in another breaks many loaders. Validate every line.