All Questions
38
questions
0
votes
0
answers
52
views
Maximize duplicity efficiency
I'm creating incremental backups for a database. The size is around 8 GB per dump.
I have a backup creation policy, that looks like this: (don't worry, I already avoided collisions)
An increment ...
1
vote
0
answers
142
views
How do I block TCP ACK causing retransmissions
I am trying to replicate a issue when making requests to S3 that will lead to tcp_retries2 leading to 900 seconds of retransmission attempts before giving up. How would I do this? Is it sufficient to ...
- 11
0
votes
1
answer
841
views
cannot Delete objects on s3 ceph bucket
I have a ceph cluster and running a few s3 bucket on it, 'gitlab-s3-api' user has full permission on everything on ( users=;buckets=;metadata=;usage=;zone=* ) but very oddly it cannot delete any file ...
0
votes
0
answers
192
views
Copy old data from local Centos to AWS S3 bucket
I have the following task to complete. I hope someone can provide the feedback to get me started.
We have a directory with thousands of images in many folders and sub-folders
The goal is to have a ...
1
vote
1
answer
384
views
AWS EC2 deleted but still shows current usage
I am using AWS EC2 free tier now.
I deleted EC2 instances but my billing dashboard shows current usage space is 11 GB-MO.
Here is my billing dashboard image.
And here is EC2 dashboard.
Can any one ...
- 109
2
votes
1
answer
3k
views
Amazon s3 bucket mount point getting unmounted automatically
I use s3fs on aws ubuntu instance to mount s3 bucket. Following is the command I used to mount the bucket.
sudo fusermount -u /s3/project1
sudo s3fs -o allow_other project1 /s3/project1
The ...
- 141
3
votes
2
answers
948
views
Migrating 300GB Data from Linux Server to S3 Bucket
I have a linux dedicated server that has 300GB of uploaded files that I need to transfer to AWS Storage S3 since I am now changing the uploads to be saved in S3 bucket instead of local disk. I read ...
- 1,451
2
votes
1
answer
1k
views
SSHFS mount to AWS Transfer for SFTP: cannot create regular file
I'm trying to setup an SSHFS mount point via Amazon's new service Transfer for SFTP. I can sftp into the endpoint just fine using sftp and can get/put files.
I can mount it using sshfs. I'm having ...
- 551
2
votes
1
answer
960
views
Backing up Apache access logs to S3
I am trying to backup Apache 2.4.27 logs on Amazon Linux 2016.03. I am trying to do so without writing a shell script, and I do not want to configure AWS access key and secret key on my Linux machine.
- 23
0
votes
1
answer
2k
views
How to use s3cmd put when there are spaces in the filename?
for file in $(find . -type f -mmin -240)
do
filename=$(echo "$file" | cut -d'.' --complement -s -f1)
s3cmd put $file s3://${bucketname}/$filename
done
I am trying to ...
- 919
1
vote
2
answers
9k
views
AWS S3 no space left on device
I am mounting a S3 bucket on a amazon EC2 instance using S3FS. Things have been working great for now over 18 months.
Today I've tried to download a 5GB file using wget directly on the S3 bucket (...
- 121
0
votes
1
answer
2k
views
How to suppress file output from aws s3 ls?
I am using the aws cli to summarize the number of files and the total size of an s3 bucket using the following command (documentation):
aws s3 ls s3://mybucket --recursive --human-readable --...
- 155
0
votes
0
answers
437
views
When s3cmd checks a bucket with thousands of files to sync it, does this count as one list request or thousands?
Trying to figure out pricing here so I don't get overcharged. I daily run a backup script where s3cmd checks a folder with over one hundred thousand files to see if any have changed (which by the way, ...
- 919
2
votes
2
answers
7k
views
Is there any way to restore an entire previous version of a folder in an s3 bucket?
I know folders are not folders within S3, but it would be extremely useful if you could restore a previous version of one when you have versioning enabled. I've been looking at it and I can only see ...
- 919
3
votes
2
answers
6k
views
aws s3 sync: how to exclude proc, sys directories
I can't seem to get the aws sync s3 command to exclude directories. I am backing up a mounted filesystem from OpenVZ which is located at /vz/root/100.
My command is:
aws s3 sync /vz/root/100 s3://[...
- 800
2
votes
4
answers
3k
views
Duplicity backup to S3: BackendException
I'm trying to configure my Duplicity installation to backup to an Amazon S3 bucket. I keep getting a BackendException error with no further details.
EDIT: I've removed my Duply configuration from ...
- 163
0
votes
1
answer
614
views
Load times of S3 Static Website
We have recently decided to move our static single page website to AWS S3, the previous server was hosted by Godaddy(cpanel) in singapore. We are using Synthetics New Relic to monitor our website and ...
- 151
1
vote
1
answer
1k
views
What is the best way to backup logs in Amazon S3 in real time? [closed]
I want to backup all my logs in Amazon S3 in real time, which tool I should use?
- 2,164
4
votes
3
answers
12k
views
s3fs "could not determine how to establish security credentials"
I'm trying to use s3fs to mount an S3 bucket on to a standard AWS Amazon Linux AMI (with all the necessary dependencies installed). However when following this tutorial when I run: s3fs mybucketname -...
- 141
0
votes
1
answer
1k
views
rebuild yum index on aws s3
I am trying to rebuild yum repo on aws S3 after adding new packages.
Here are few commands I am trying, but it is not helping.
[root@chucks ~]$ createrepo --baseurl http://rpmcopy.xxxxx.com.s3-...
- 531
9
votes
3
answers
22k
views
s3cmd: Deleting files or folder does not work
s3cmd s3://MYBUCKET/FOLDER does not work. (folder is empty)
s3cmd s3://MYBUCKET/FILE does not work.
Output of both commands is "File FILE sucessfully deleted", yet the file remains on the bucket.
...
user54883
2
votes
3
answers
2k
views
Should I use ec2 as a file server?
I need to be able to share user uploaded content across multiple EC2 application servers. I have looked at rsync, mounted NFS, and S3 as potential options of being able to share this data in almost ...
- 113
8
votes
2
answers
13k
views
I'm trying to use "aws s3 sync" on my EC2 instance. Is the '--exclude' option broken?
I'm trying to backup my EC2 instance to S3 using Amazon's official tools that come preinstalled on the EC2 instance. I'm having difficulty getting the sync command to exclude directories from the sync....
- 91
0
votes
1
answer
870
views
How continue sync with S3CMD after broken connection?
I'm using s3cmd to backup some logs into Amazon S3 buckets. Which is a great tool.
But I notice that, very often, my sync's are broken, they just cancel and try to upload again, until they can upload ...
- 155
2
votes
1
answer
2k
views
How backup files with logrotate and Amazon S3?
I would like to create a new standard of two things:
how long would be generated and be rotated the logs generated by my applications ?
how to transfer the logs to Amazon S3, as a backup server ?.
...
- 155
4
votes
1
answer
4k
views
s3cmd put only files that do not exist in destination bucket
I've been using s3cmd to sync local files on our server with the S3 bucket. There are over 200,000 files on the local server. Can't seem to find any documentation on the site that explains how to ...
- 155
2
votes
2
answers
5k
views
s3fs changing s3 permissions?
My developer believes that s3fs is changing my bucket's permissions. Is this possible? I want my bucket to be public, but it keeps reverting back to private.
Here's my fstab.
s3fs#production /mnt/...
- 169
0
votes
4
answers
2k
views
Easiest to use backup software for live linux server [closed]
I need a backup software that is capable of:
doing a full HDD image (from live server, if that's possible at all?)or just copy everything except /dev, /tmp, etc
updates the backup incremental just ...
- 49
4
votes
3
answers
7k
views
Uploading 200GB of files to S3
What's the best way to upload 200GB tar.gz files to S3 in Linux? On researching I found that S3 limit on objects has been increased to 5TB and came to know about the multi part upload mechanism to ...
7
votes
7
answers
13k
views
s3cmd fails too many times
It used to be my favorite backup transport agent but now I frequently get this result from s3cmd on the very same Ubuntu server/network:
root@server:/home/backups# s3cmd put bkup.tgz s3://mybucket/
...
- 3,137
0
votes
1
answer
248
views
Alternative storage for SVN backup with rsync
I've been backing up my SVN repo and trac environment using a weekly cron job and 2 custom scripts. This has been going to Joyent Strongspace; but since they have discontinued the service, I need an ...
- 23
4
votes
2
answers
2k
views
Linux filesystem or CDN for millions of files with replication
Please tell me solution for this scenario:
several millions files, located in one directory ("img/8898f6152a0ecd7997a68631768fb72e9ac2efe1_1.jpg")
~80k file size in average
90% random read access
...
- 95
1
vote
1
answer
156
views
Store files in the cloud to be accessible by any VPS
I often create/delete online VPS machines so storing files there will not be so good.
I want a dedicated online storage like S3 for storing files (scripts, videos etc) to be accessible by any VPS ...
- 1,245
5
votes
5
answers
1k
views
What sort of web server hardware do you use to handle 100 Mbps+ of static files?
I currently use Amazon S3 for much of my static file serving needs but my monthly bill is getting very expensive. I did some rough calculations using the logs and at peak times, my most expensive ...
- 1,290
9
votes
3
answers
3k
views
What's the easiest way to back up EC2 instances automatically?
I'm worried I might log in one day and my fully configured instance has been removed. I would like to be able to back it up daily to S3 but frustratingly this is not a simple setting in the console.
...
- 2,609
1
vote
3
answers
1k
views
Amazon S3 tools for Debian?
I need to (programmatically, in a shell script) upload an EAR file to an Amazon S3 bucket on Debian (5.0.4). What, if any, Debian package provides simple, scriptable tools for that?
Ideally this ...
- 2,981
0
votes
1
answer
73
views
Backing Up User Data when data is not in use. Should I be concerned?
This may be a dumb question. I would like to use duplicity to make backups to Amazon S3 of directories, each of which contains a different user's data. Each directory could be written to at any time. ...
- 914
10
votes
8
answers
1k
views
S3 Interfaces for Backup, etc [closed]
Since none of the S3 "file systems" are compatible right now, what s3 gateway tool do you recommend for mounting a remote S3 bucket for backups? I've been using S3FS, but I've heard a lot of people ...
- 2,741