Xvc for Data Versioning

Xvc keeps large data files out of Git while still versioning them alongside your code. Git stores small metadata files that record the content of each dataset, and the data itself lives in Xvc's content-addressed cache and, optionally, in a remote storage.

This guide walks through a typical data-versioning workflow. Every command has a dedicated page in the Command Reference if you want more detail.

Initialize a repository

Data versioning works inside an Xvc repository. Initialize one on top of Git:

$ git init
$ xvc init

By default Xvc handles the Git operations for the metadata it creates, so you don't have to stage or commit anything under .xvc/ yourself. See Turn off Git Integration if you'd rather manage Git yourself.

Track your data

$ xvc file track data/

Xvc computes a content digest (BLAKE-3 by default) for each file, moves the content into the cache under .xvc/b3/, and rechecks (checks out) a copy into your workspace. Because the address is derived from the content, changing a file produces a new version without overwriting the old one.

Use xvc file list to see what is tracked and whether your workspace matches the recorded content:

$ xvc file list data/

Save new versions

When you change a tracked file, carry the new content into the cache with xvc file carry-in:

$ xvc file carry-in data/measurements.csv

Xvc records the new digest and commits the updated metadata to Git, so the history of your data follows the history of your code.

Save space with links

Copying every tracked file into the workspace duplicates its content. If you don't need writable copies, recheck files as links instead:

$ xvc file recheck data/ --as symlink

Symlinks and hardlinks point at the read-only cache and consume no extra space. Reflinks give you writable copy-on-write files on filesystems that support them. See xvc file recheck and the recheck concept page.

To back up your data or share it with collaborators, configure a storage. Any directory, SSH host, or S3-compatible bucket will do; here is a local directory used as a shared cache:

$ xvc storage new local --name backup --path /mnt/backup/xvc

Send tracked content to it with xvc file send:

$ xvc file send --to backup

Retrieve data elsewhere

Anyone who clones the Git repository can fetch the data from the storage with xvc file bring:

$ git clone <your-repository-url>
$ cd <your-repository>
$ xvc file bring data/ --from backup

They don't need to run xvc init again — the repository is already initialized — but they do need read access to the storage. Xvc never stores your credentials; it reads them from the environment when a network operation is needed.

Next steps

Xvc for Machine Learning adds pipelines on top of data versioning.
The (Remote) Storages reference lists every supported backend, including AWS S3, Google Cloud Storage, MinIO, Cloudflare R2, Wasabi, Digital Ocean Spaces, rsync, and rclone.

The Xvc Book