All Questions
31
questions
0
votes
1
answer
841
views
cannot Delete objects on s3 ceph bucket
I have a ceph cluster and running a few s3 bucket on it, 'gitlab-s3-api' user has full permission on everything on ( users=;buckets=;metadata=;usage=;zone=* ) but very oddly it cannot delete any file ...
0
votes
1
answer
701
views
Where is the AWS S3 metadata stored written by s3cmd? [closed]
I was looking for a AWS S3 solution for backing up a Linux file system so that it would preserve the timestamp and permissions. It appears that s3cmd allows this by writing this information to the ...
- 965
0
votes
1
answer
3k
views
How to upload multiple files from directory to S3?
I have a directory on an Ubuntu, with 340K images, 45GB of total size! Is there an efficient way to transfer them all to an S3 of DigitalOcean?
I thought of using s3cmd put or s3cmd sync but I'm ...
- 198
0
votes
1
answer
227
views
Transfer image files from MongoDB to S3
I have a NodeJS app that stores all of its data, even images, to MongoDB. Now I want to use an S3 to store my app's files (images, 43gb) and so, I need to transfer there all files that I have in my ...
- 198
1
vote
0
answers
73
views
Spectra S3 Java CLI (DS3) difficulty verifying checksum
We have a SpectraLogic BlackPearl+T950 which uses S3 style commands to put/get/sync etc.
To access the Spectra flavour of S3 commands we use a Java CLI client to perform these operations.
Supposedly ...
- 141
1
vote
1
answer
2k
views
How to copy more than 300GB files one AWS EC2 server to another EC2
I need to copy more than 300 GB files (most them are images) one EC2 instance to another EC2 instance, what will be the best method for doing this?
EBS Snapshot will be not work in my situation ...
- 13
0
votes
1
answer
110
views
can I attach IAM role to instance after launching it?
Getting error :
root@omp_affiliate_backend ~]# s3cmd get s3://admarvel-noc/all-users/nrpe-plugins.tar.gz .
s3://admarvel-noc/all-users/nrpe-plugins.tar.gz -> ./nrpe-plugins.tar.gz [1 of 1]
ERROR:...
- 277
0
votes
1
answer
2k
views
How to use s3cmd put when there are spaces in the filename?
for file in $(find . -type f -mmin -240)
do
filename=$(echo "$file" | cut -d'.' --complement -s -f1)
s3cmd put $file s3://${bucketname}/$filename
done
I am trying to ...
- 919
1
vote
1
answer
2k
views
s3cmd error on sync: S3 error: 400 (InvalidRequest)
I'm trying to configure s3cmd to backup some directories on my server to my AWS S3 bucket. My problem is that when I do the sync command: s3cmd sync -r -p --no-delete-removed /path/to/dir s3://...
- 653
0
votes
1
answer
1k
views
s3cmd restore recursive not working right
I am running the following command to restore the data from Amazon S3 to the web server.
s3cmd restore --recursive s3://my-bucket/Weekly-Sunday/dbexports/
and getting this error:
Restore is not ...
- 348
2
votes
2
answers
3k
views
How to upload a large file using aws commandline when connection may be unreliable?
I've been having trouble uploading a large (800MB) file to s3, using the aws commandline tool. The first attempt completed (after many hours) but was not visible, and I was advised (here) that it has ...
- 730
2
votes
3
answers
4k
views
How to transfer data between s3 buckets of the different aws account using command line
How to transfer data between s3 buckets of the different aws account using s3cmd ?
command will be like this: s3cmd rsync s3://acc1_bucket/folder/ s3://acc2_bucket/folder --recursive
But then how ...
- 277
2
votes
2
answers
796
views
Logrotate script not uploading to s3
I am trying to use logrotate to upload log files from my rails and nginx commands once a day. When I run the logrotate command manually the files successfully upload, but it never does it on its own. ...
- 123
2
votes
2
answers
4k
views
How best way to rsync from aws s3 to gcloud?
I must execute a rsync ( not only copy or move ) from all my buckets from s3 to google cloud.
I need create the same infraestructure of s3 to google.
I have like 2Tera in s3, and all days I receive ...
google-cloud-platform
1
vote
1
answer
478
views
s3 service using s3cmd tool through URL based firewalls
I am behind a URL based application firewall and currently using s3cmd tool to access amazon's s3 service. Doing a bit of searching has asked me to whitelist s3.amazonaws.com but I see that s3cmd uses ...
- 111
3
votes
0
answers
523
views
Solution for file data backup, versioning and replication
We have a growing set of data files (.wav files, image files etc) which are data, i.e not part of the application code - uploaded and modified by users. The number of files is in the 1000s and the ...
- 227
4
votes
1
answer
4k
views
Sync files from local filesystem with S3 - one way
I have a process that generates log files to the local filesystem, and files that are more than 2 weeks old are automatically purged. I want to use the aws command line tools to sync my folder with ...
- 143
1
vote
1
answer
758
views
Redirect warning when running s3cmd sync
When running s3cmd I get a redirect warning as follows:
WARNING: Redirected to: xyz.s3-eu-west-1.amazonaws.com
The script that produces the warning is this:
s3cmd sync /var/backups/db/ s3://xyz/...
- 189
1
vote
1
answer
3k
views
Setting cache for already uploaded amazon s3 files
In amazon s3 for our site through s3cmd tool we uploaded around 60GB of files including mp3, jpg to a bucket. But now when we checked site slowness files are not caching and further checking we came ...
- 31
9
votes
3
answers
22k
views
s3cmd: Deleting files or folder does not work
s3cmd s3://MYBUCKET/FOLDER does not work. (folder is empty)
s3cmd s3://MYBUCKET/FILE does not work.
Output of both commands is "File FILE sucessfully deleted", yet the file remains on the bucket.
...
user54883
10
votes
5
answers
12k
views
how to pipe a mysql dump to s3cmd
I want to transfer a mysql dump, compressed, to s3.
I tried:
mysqldump -u root -ppassword -all-databases | gzip -9 | s3cmd put s3://bucket/sql/databases.sql.gz
but then I get:
ERROR: Not enough ...
- 329
0
votes
0
answers
3k
views
s3cmd 403 Access Denied error on put but not ls
I can access s3 using s3cmd on an ec2 instance to do s3cmd ls s3://myc
bucketname but not using 'put' to add a file. It gives a 403 access denied error.
I am using IAM and have group setup to access ...
- 239
4
votes
2
answers
4k
views
Amazon S3 - private file is still downloadable for everyone?
I have some secret files which I want to restrict anonymous user from viewing/downloading. I have tried to make it private by running something like:
s3cmd setacl --acl-private s3://bucket/some/path/*...
- 51.6k
2
votes
1
answer
4k
views
How to copy all s3 buckets to another AWS account?
I'm copying all EC2 instances to another AWS account (creating the AMIs, sharing them with that account and running the instances from AMIs there). I also need to move my S3 buckets since the ...
- 121
5
votes
1
answer
3k
views
Linux server sync to an Amazon S3 bucket
I am looking for a stable solution to replace a classic server backup to another server using rsync.
I have to sync a whole filesystem (more than 1Tb) to Amazon S3.
Where am I?
Solution 1:
I mapped ...
- 141
4
votes
1
answer
7k
views
s3cmd with --delete-removed
I'm currently writing a script to sync files in s3 buckets with s3cmd.
I check the document and it says:
s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR
also I find a ...
- 43
0
votes
1
answer
870
views
How continue sync with S3CMD after broken connection?
I'm using s3cmd to backup some logs into Amazon S3 buckets. Which is a great tool.
But I notice that, very often, my sync's are broken, they just cancel and try to upload again, until they can upload ...
- 155
2
votes
1
answer
2k
views
How backup files with logrotate and Amazon S3?
I would like to create a new standard of two things:
how long would be generated and be rotated the logs generated by my applications ?
how to transfer the logs to Amazon S3, as a backup server ?.
...
- 155
4
votes
1
answer
4k
views
s3cmd put only files that do not exist in destination bucket
I've been using s3cmd to sync local files on our server with the S3 bucket. There are over 200,000 files on the local server. Can't seem to find any documentation on the site that explains how to ...
- 155
1
vote
1
answer
2k
views
One Way Sync of a Bucket With Local Directory
I have a local directory that I would like to synchronize with an S3 bucket.
I have two specific requirements:
If local file is the same as the remote, do not re-transfer it to the bucket.
If some ...
- 131
1
vote
3
answers
536
views
Backing up data (including mysqldumps) to S3
We have a web app on a number of servers and we want to add an additional layer of redundancy by backing up the key data to S3. The key data is the MySQL database and a folder containing dynamically ...
- 990
Related Tags
amazon-web-services × 10
linux × 6
backup × 5
rsync × 4
amazon-ec2 × 2
logging × 2
logrotate × 2
aws-cli × 2
digital-ocean × 2
mysql × 1
cron × 1
replication × 1
mongodb × 1
synchronization × 1
upload × 1
database-backup × 1
amazon-iam × 1
pipe × 1
ceph × 1
s3fs × 1
amazon-glacier × 1
object-storage × 1
more related tags
Hot Network Questions
- Does the expansion and contraction of a variable star affect the measured radial velocity?
- Does becoming a warlock mean said warlock will become a "Soul Coin" upon death?
- Can philosophy be useful?
- Was it legal according to Ukrainian law to kill Illya Kyva?
- What was the role of Schmidt in derivation of the Gram-Schmidt process?
- Diploria labyrinthiformis - can we reproduce Ernst Haeckel's drawing of a brain coral?
- What happens when an anti-electron collides with a neutrino?
- How bad is topping out a suspension fork?
- Negotiated conditions not met after taking back resignation
- Can we hit the spaceship
- Why is SQL server dropping the seconds of values passed to a SMALLDATETIME field?
- Did Huawei modem just tried to do a Man-In-The-Middle attack on me?
- Can One Solve This Scary Sequence?
- Why do many Arabic letters look exactly like other letters except for dots, yet have no similarity in sound?
- How does one express the ordering of thoughts?
- bullet resistant space armor material?
- ASCII-art milk carton
- How to let a recruiter know you're not seriously considering another job but you are exploring your options
- Indonesia visa misprint by consulate
- Why "quod" and not "quo" is used here?
- Issue with the Hook Modifier in Curve Distortion
- Why do we sometimes lose solutions when solving equations?
- May a juror refuse to answer questions?
- Correctness of mixed signed/unsigned arithmetic