Smart - only syncs new or updated files.Multi-threaded - syncs the files faster by utilizing multiple threads.$ aws s3 sync s3://your-bucket-name /home/ubuntu/s3/your-bucket-name/ Benefits: It provides a useful sync command that quickly and easily downloads the remote bucket files to your local filesystem. This is Amazon's official command line interface for interacting with their different cloud services, S3 included. In the meantime, I set out to find another alternative.Īnd then I found awscli. I opened an issue ( bloomreach/s4cmd/#46) to report this strange behavior. It should check whether the remote file already exists locally (hash/filesize checking would be neat) and skip it in the next sync run on the same target directory. That is not the kind of behavior I was expecting from the sync command. Looked even more promising, however, I noticed that it kept re-downloading files that were already present on the local filesystem. The newer, multi-threaded alternative to s3cmd. Since it did not work for huge buckets, I set out to find an alternative. It did work fine on small buckets, though. However, after trying it on my enormous S3 bucket - it failed to scale, erroring out with a Segmentation fault. This can be easily achieved by utilizing multiple command line utilities that make it possible to sync a remote S3 bucket to the local filesystem.Īt first, s3cmd looked extremely promising.
Originally posted on my blog: Sync Your S3 Bucket to an EC2 Server Periodically
0 kommentar(er)
