Hugging Face · AI

HF Buckets: Upload Only What Changed with Xet

If you work with large datasets, you're probably re-uploading the same bytes every time the data changes. In this video I upload a 191 MB NYC taxi CSV to a Hugging Face Bucket, append 100 rows, and upload it again: Xet, Hugging Face's storage layer, sends only the new pieces. Then I convert the data to Parquet, prepend 100 rows, and sync again: with PyArrow's use_content_defined_chunking=True, only 8.6 MB of the 52.9 MB file goes over the wire. F

Introductie van de bron.