All Questions
16
questions
0
votes
0
answers
232
views
How suitable is a S3 backend for rsync operations?
I run a large server providing open source software (https://ftp.halifax.rwth-aachen.de), currently serving more than 30 TByte of data with Multi-Gigabit throughput. Data is synchronized and kept up-...
- 334
6
votes
1
answer
7k
views
Does AWS cli do a data integrity check on sync with s3? [closed]
I've been looking into using AWS cli for data integrity checks to verify a backup has been transferred from a Linux file server correctly to AWS s3. Likewise, I would like to verify when restoring a ...
- 965
1
vote
1
answer
1k
views
Clever way to reduce rsync/rclone load with backups to Amazon S3?
I started running rclone to keep a set of files on my laptop that are synced up to the cloud and encrypted. I wanted to 'control' the encryption because of trust issues. Rclone does this pretty well, ...
- 13
1
vote
1
answer
2k
views
How to copy more than 300GB files one AWS EC2 server to another EC2
I need to copy more than 300 GB files (most them are images) one EC2 instance to another EC2 instance, what will be the best method for doing this?
EBS Snapshot will be not work in my situation ...
- 13
53
votes
3
answers
78k
views
Rsync to AWS S3 bucket
For a server I am hosting a website on I want to backup the data and settings to an S3 bucket. I found out that you can't directly use rsync to backup to an S3 bucket. Is there another way to achieve ...
- 653
2
votes
3
answers
4k
views
How to transfer data between s3 buckets of the different aws account using command line
How to transfer data between s3 buckets of the different aws account using s3cmd ?
command will be like this: s3cmd rsync s3://acc1_bucket/folder/ s3://acc2_bucket/folder --recursive
But then how ...
- 277
3
votes
0
answers
523
views
Solution for file data backup, versioning and replication
We have a growing set of data files (.wav files, image files etc) which are data, i.e not part of the application code - uploaded and modified by users. The number of files is in the 1000s and the ...
- 227
2
votes
3
answers
2k
views
Should I use ec2 as a file server?
I need to be able to share user uploaded content across multiple EC2 application servers. I have looked at rsync, mounted NFS, and S3 as potential options of being able to share this data in almost ...
- 113
5
votes
1
answer
3k
views
Linux server sync to an Amazon S3 bucket
I am looking for a stable solution to replace a classic server backup to another server using rsync.
I have to sync a whole filesystem (more than 1Tb) to Amazon S3.
Where am I?
Solution 1:
I mapped ...
- 141
0
votes
1
answer
180
views
Subsequent runs of rsync locally don't reduce data transferred
I have an EC2 instance with data I want to sync to a mounted, but remote, volume, as a backup.
rsync seems like the way to go with this, so as a test I took my test file (a Postgres pg_dump file) and ...
- 115
47
votes
7
answers
64k
views
How can one efficiently use S3 to back up files incrementally?
I understand how rsync works on a high-level, but there are 2 sides. With S3 there is no daemon to speak of — well there is, but it's basically just HTTP.
There look to be a few approaches.
s3rsync ...
- 571
0
votes
1
answer
248
views
Alternative storage for SVN backup with rsync
I've been backing up my SVN repo and trac environment using a weekly cron job and 2 custom scripts. This has been going to Joyent Strongspace; but since they have discontinued the service, I need an ...
- 23
2
votes
1
answer
1k
views
Best way to mirror data across two-node apache cluster
I'm architecting a new server structure and I plan to have two apache workhorses and a sql database behind both of them. I was wondering what the best way to mirror the data between the apache servers ...
- 121
1
vote
1
answer
659
views
Does librsync really reduce bandwidth for Amazon S3
I am looking to do some backups to S3 and want to reduce the bandwidth as much as possible. I am looking at a few options. It seems like librsync is the best solution for low bandwidth remote backups.
...
- 1,189
4
votes
4
answers
9k
views
s3fs Input/output error
I'm trying to set up a backup system with s3fs and the Amazon S3 service.
I followed this this guide and this guide.
Tailing /var/log/messages I get:
Aug 28 13:37:46 server s3fs:###response=403
I ...
- 376
42
votes
7
answers
66k
views
Using RSYNC with Amazon S3 [closed]
I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools)
Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and ...
- 1,720