All Questions
28
questions
0
votes
1
answer
82
views
Best method to copy image files from AWS EC2 instance to S3 bucket based on query
What is the most efficient method to copy selected image files from a EC2 instance (Ubuntu 20.04) in production to a S3 bucket while checking if the file exists in EC2?
This is a one-time operation. ...
- 1
-1
votes
2
answers
73
views
Backup of multiple sites on Ubuntu
I have multiple sites in my /var/www/html/ folder like following:-
/var/www/html/site-1
/var/www/html/site-2
/var/www/html/site-3
/var/www/html/site-4
i want to zip each site on my server and to ...
- 2,009
2
votes
0
answers
390
views
gsutil rsync fails on Ubuntu with UnicodeDecodeError
I'm trying to rsync between google cloud storage and amazon s3. On my local machine (MacOS) this command works
gsutil rsync -d -r gs://mira-internal/data/exports/20191108 s3://mira-temp/raw/20191108
...
- 71
0
votes
0
answers
214
views
architecture binary-amd64 not recognized in ubuntu mirror
I am using preseed to install ubuntu 16.04 and I have a setup that works fine with the regular ubuntu mirror if I specify
d-i mirror/http/hostname string archive.ubuntu.com
d-i mirror/http/directory ...
- 245
1
vote
1
answer
923
views
Moodledata on shared storage on AWS
I am in the process of setting up a simple Moodle cluster on AWS for production use. The plan is to use a loadblancer, two EC2 instances and a RDS. I have worked extensively with Moodle before but ...
- 421
1
vote
2
answers
9k
views
AWS S3 no space left on device
I am mounting a S3 bucket on a amazon EC2 instance using S3FS. Things have been working great for now over 18 months.
Today I've tried to download a 5GB file using wget directly on the S3 bucket (...
- 121
0
votes
2
answers
912
views
What is the most efficient way to transfer files from AWS S3 to S3?
I have a large number of files stored in an AWS S3 bucket. I need to get those files to another S3 bucket owned by another user and in a different region. The files are stored in different locations ...
- 155
0
votes
1
answer
2k
views
How to suppress file output from aws s3 ls?
I am using the aws cli to summarize the number of files and the total size of an s3 bucket using the following command (documentation):
aws s3 ls s3://mybucket --recursive --human-readable --...
- 155
5
votes
2
answers
13k
views
SYNC a folder on Ubuntu server with amazon S3 bucket automatically
I have an app running on Digital Ocean server and there are two folders which are being updated with user uploaded images.
Is it possible to transfer the new uploads from the server folder to s3 ...
- 161
-1
votes
1
answer
85
views
Best and cheapest way of expanding storage on a server
I provide photo editing services, so customers send their photos to me through website, and I send them back through website once edited. I need about 1TB storage when I keep files for 1 month. I'm ...
2
votes
4
answers
3k
views
Duplicity backup to S3: BackendException
I'm trying to configure my Duplicity installation to backup to an Amazon S3 bucket. I keep getting a BackendException error with no further details.
EDIT: I've removed my Duply configuration from ...
- 163
2
votes
2
answers
2k
views
Use Apt Inside AWS VPC Without NAT Instance
I have several Ubuntu machines inside VPCs. For many of them, the only network access they need whatsoever is for "apt-get update && apt-get upgrade" and NTP. For NTP, I can simply have an NTP ...
- 1,157
1
vote
0
answers
453
views
Duplicity -> S3 backup failing with "[Errno 105] No buffer space available"
I have a nightly script on my Ubuntu 10.04 VPS which uses duplicity (0.6.24) to run incremental, encrypted backups to Amazon S3. This script had been working until a month or so ago when it started ...
- 239
4
votes
2
answers
5k
views
Move files from ftp server to s3
I would like to set up an ftp server, where users will upload files,
and for each file, put it on s3 storage, and delete it from the ftp server. (the server runs on ec2 ubuntu)
Here are the stuff I ...
- 171
1
vote
0
answers
574
views
Amazon S3 sync theory
I'm setting up a backup of my vital data (database and images, mainly) to S3 from my Ubuntu box. I've set up Amazon S3, and installed S3FS to mount the bucket on my machine. This is working well, but ...
- 193
2
votes
2
answers
5k
views
Create an Amazon EC2 instance with automatic access to private S3
I was wondering. Say I have a file in a private S3 bucket, and I then deploy an EC2 instance which has something like that in its rc.local:
GET http://privateamazons3url.com
How can I make it so ...
- 1,123
0
votes
1
answer
638
views
configure s3fs mount for write-only
I am using s3fs to mount a S3 buckets on an Ubunutu server to store my daily backups there.
Considering security this would allow a potential hacker to delete all my backups from the mounted bucket.
...
- 288
15
votes
5
answers
24k
views
How to make s3fs work with IAM roles?
We are using IAM roles to maintain the machines. We now planning to mount the s3 to our ec2 instances and do the processing as per our need. We been using s3fs tool for mounting.
But that doesn't ...
- 287
0
votes
1
answer
868
views
EC2 user-data and private s3 resources
I would like to set a user-data script on my EC2 instances like this:
#include
http://mybucket.s3.amazonaws.com/myapp/environment/aws_init.sh
However the resource is acl private and cannot be ...
- 285
0
votes
1
answer
235
views
How to place my audit file in Amazon S3?
I have installed and configured the auditd package in Amazon Ec2 instance. I wish I need to place all the auditd log files in Amazon S3.
I'm very new to Ubuntu can anyone explain me how to do it?
- 321
0
votes
1
answer
400
views
Is uploading a 500mb file to s3 an expensive operation? Any way to limit cpu usage etc?
On a production server, I'm planning on uploading a single file or a set of files that may be 500 megs in size.
I havent' test it just yet, but just want some opinions on whether you think this will ...
- 513
1
vote
4
answers
400
views
Best practice for synchronizing sources across servers (Amazon EC2)
I'm moving from single app server to load balanced config.
In the single server config I synchronized the sources by S3 cmd sync s3 every few minutes and on hostup (Scalr event) I sync the sources ...
- 1,441
1
vote
1
answer
2k
views
Mounting an S3 bucket onto a AWS Ubuntu instance issues
i'm using s3fuse to mount an s3 bucket onto ubuntu, so that i can take advantage of the storage space.
here are some environment info:
System Info:
Linux ip-xxx-xxx-xxx-xxx 2.6.38-8-virtual #42-...
- 259
0
votes
1
answer
515
views
How do I backup files stored in Rackspace Cloud Files container to an Amazon s3 bucket
I'm currently storing PDF reports in a Rackspace cloud files container. These files are generated from a web application. I'm keen to find a way to sync these files to an Amazon s3 container as a form ...
- 41
3
votes
3
answers
5k
views
s3fs: how to force remount on errors?
I use s3fs 1.33 on Ubuntu 9.10.
Regularily it gives me errors like this:
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: close failed on "/mnt/s3/...
- 2,443
7
votes
1
answer
14k
views
How to force s3fs mount on boot
I'm using s3fs 1.33 on Ubuntu 9.10 x86.
I mount it manually with this command:
sudo /usr/bin/s3fs mybucket /mnt/s3/mybucket -ouse_cache=/tmp
How can I force the mount on boot?
I guess that I have ...
- 2,443
0
votes
1
answer
600
views
S3sync not working
I managed to get s3sync to upload my test folder to Amazon S3 and can see it in the MWS Managment Console.
Downloading the data back to a test folder results in the following error message:
root@...
- 21
2
votes
1
answer
6k
views
Should use EXT4 or XFS to be able to 'sync'/backup to S3?
It's my first message here, so bear with me...
(I have already checked quite a few of the "Related Questions" suggested by the editor)
Here's the setup,
a brand new dedicated server (8GB RAM, some ...
- 144