All Questions
39
questions
0
votes
1
answer
293
views
Bandwidth consumption of Amazon S3 bucket mounted with S3fs on a VPS
I have :
VPS server on Vultr with web server and Wordpress installed.
Uploads folder is mounted on Amazon S3 bucket using s3fs.
If a user downloads a 1GB file from my website will the traffic be ...
- 1
0
votes
2
answers
1k
views
Unable to mount a folder in Ubuntu to as AWS S3 Bucket
I am attempting to mount a folder on a fresh AWS EC2 instance running Ubuntu 18.04, to AWS S3.
I've been following instructions I found at https://cloud.netapp.com/blog/amazon-s3-as-a-file-system and ...
- 101
1
vote
3
answers
1k
views
mysqldump using a lot of space
I have a server and the / partition is 20GB in size.
Databases are stored in /mnt/mysql-data partition is 500GB in size.
Now here's the problem. Whenever I run mysqldump it fills up / partition to ...
- 129
6
votes
0
answers
1k
views
How does S3FS (or any other S3 FUSE filsystem) compare to AWS FSx for Lustre + S3
I remember trying s3fs a year back, trying to use some S3 bucket as a FUSE filesystem. I remember it being quite laggy, especially when coupled with git operations on it (an oblivious system architect ...
- 245
2
votes
1
answer
3k
views
Amazon s3 bucket mount point getting unmounted automatically
I use s3fs on aws ubuntu instance to mount s3 bucket. Following is the command I used to mount the bucket.
sudo fusermount -u /s3/project1
sudo s3fs -o allow_other project1 /s3/project1
The ...
- 141
2
votes
1
answer
589
views
Use S3FS Mount As HTTP Server Document Root
My proposed setup:
An EC2 server running Apache or nginx.
An S3 Bucket in the same region/availability zone mounted with S3FS.
The Apache or nginx document root set to the S3FS mount.
My question: ...
- 414
3
votes
1
answer
834
views
How to share a EFS/s3 with code in a auto scaling group?
the main problem is:
Infrastructure:
Autoscaling (min 1 - max 3)
RDS
ELB
elasticache (redis)
elasticsearch
I want to share a volume (EFS / S3) with the code of my application;
The size of it is ...
- 71
7
votes
2
answers
19k
views
Can not unmount an S3 directory mounted with s3fs-fuse
I have an s3fs-fuse "device" mounted at /mnt/s3. I want to unmount it so that I can remount it with the mp_umask flag so that chroot will work properly.
The command should be:
fusermount -u /mnt/s3
...
3
votes
1
answer
2k
views
How to use SAMBA on an Amazon S3 mounted directory on Ubuntu VM
I had successfully configured a SAMBA share on /home/myusername on an Ubuntu 17.10 VM. I then used s3fs-fuse to mount an S3 bucket on that exact directory, i.e. /home/myusername. It worked too, any ...
- 161
1
vote
1
answer
583
views
S3 Bucket for hosting media, s3fs mount catch-22
I'm attempting to use an S3 bucket to host media files (images, video and static files) for a small web application and came across a strange catch-22...
According to the S3 documentation, to serve a ...
- 121
1
vote
0
answers
107
views
Subdomain pointing to deleted S3 bucket [duplicate]
I am stuck in my present (AP-SOUTH) and deleted (US-EAST) S3 bucket.
I had mistakenly created a bucket in US-EAST (cdn.example.com.s3-website-us-east-1.amazonaws.com) and mapped my subdomain - cdn....
- 11
2
votes
0
answers
736
views
rpc.mountd[pid]: /xxx and /yyy have same filehandle for host, using first
I have an S3 bucket mounted using s3fs-fuse at /yyy and a normal drive mounted at /xxx. I am exporting both with NFS however when a client attempts mount, I see the following error:
rpc.mountd[pid]: /...
- 491
1
vote
2
answers
9k
views
AWS S3 no space left on device
I am mounting a S3 bucket on a amazon EC2 instance using S3FS. Things have been working great for now over 18 months.
Today I've tried to download a 5GB file using wget directly on the S3 bucket (...
- 121
1
vote
1
answer
736
views
Best manner to access files in S3 Bucket from EC2 instance
I have tried several packages (goofys, s3fs, fuse) and have not been able to find a solution to my problem.
I have an S3 Bucket with several thousand files currently in it and will continue to grow. ...
- 277
0
votes
0
answers
932
views
S3FS and Fuse, scale out or scale up
I have an infra where I have multiple machines (eg FTP servers, get/put and Samba/NFS servers) that need to share out the same (HA) storage. So I figured to use s3fs and fuse. I got this all setup and ...
- 133
0
votes
1
answer
1k
views
Is S3 a proper choice to keep live linux user home directories?
We are building an RDP cluster using Amazon EC2/S3. Users' home directories are on S3 (single bucket) and all EC2 nodes mount the same S3 bucket using S3FS. Each EC2 node uses the appropriate home ...
- 23
4
votes
3
answers
12k
views
s3fs "could not determine how to establish security credentials"
I'm trying to use s3fs to mount an S3 bucket on to a standard AWS Amazon Linux AMI (with all the necessary dependencies installed). However when following this tutorial when I run: s3fs mybucketname -...
- 141
0
votes
1
answer
1k
views
AWS: Best solution for syncing files across autoscaled EC2 instances?
I'm working on setting up autoscaling on EC2; Instances will be running a web app with big traffic spikes. The first solution I tried for syncing files across instances (since the web app will write ...
- 511
1
vote
0
answers
574
views
Amazon S3 sync theory
I'm setting up a backup of my vital data (database and images, mainly) to S3 from my Ubuntu box. I've set up Amazon S3, and installed S3FS to mount the bucket on my machine. This is working well, but ...
- 193
5
votes
1
answer
3k
views
Linux server sync to an Amazon S3 bucket
I am looking for a stable solution to replace a classic server backup to another server using rsync.
I have to sync a whole filesystem (more than 1Tb) to Amazon S3.
Where am I?
Solution 1:
I mapped ...
- 141
0
votes
1
answer
211
views
autoscale static content on ec2 using local cache
Hello Serverfault members.
I want to achieve the following setup on EC2:
A load balancer starts and stops instances which only serve static content. (a javascript based application)
I do not want to ...
- 113
0
votes
1
answer
180
views
Subsequent runs of rsync locally don't reduce data transferred
I have an EC2 instance with data I want to sync to a mounted, but remote, volume, as a backup.
rsync seems like the way to go with this, so as a test I took my test file (a Postgres pg_dump file) and ...
- 115
0
votes
1
answer
638
views
configure s3fs mount for write-only
I am using s3fs to mount a S3 buckets on an Ubunutu server to store my daily backups there.
Considering security this would allow a potential hacker to delete all my backups from the mounted bucket.
...
- 288
15
votes
5
answers
24k
views
How to make s3fs work with IAM roles?
We are using IAM roles to maintain the machines. We now planning to mount the s3 to our ec2 instances and do the processing as per our need. We been using s3fs tool for mounting.
But that doesn't ...
- 287
2
votes
2
answers
5k
views
Can a S3 mount be used as the document root for Apache?
Has anyone been successful in having their DocumentRoot reside on an S3 mount (using s3fs)?
I currently have a mounted bucket at /mnt/s3. I can read and write files to it no problem.
In my httpd....
- 21
2
votes
0
answers
593
views
s3fs-mounted S3 bucket shows bad filesizes
Background:
I have an Amazon S3 bucket with data uploaded using the AWS web interface. Now I need to give permissions-based access to paths within my buckets to FTP users. To do this, I'm running ...
- 131
2
votes
2
answers
5k
views
s3fs changing s3 permissions?
My developer believes that s3fs is changing my bucket's permissions. Is this possible? I want my bucket to be public, but it keeps reverting back to private.
Here's my fstab.
s3fs#production /mnt/...
- 169
4
votes
3
answers
8k
views
How to list s3fs mounts?
How can I list the s3fs mounts that exist on an ubuntu system?
I'd like to know to which bucket each mount is mapped.
Specifically, I have a specific mount (e.g. ~/s3/mymount), and would like to know ...
- 5,970
9
votes
2
answers
4k
views
EC2 - Shared Storage - S3FS or EBS?
Building my web service onto EC2 right now and have a single instance behind a load balancer. I will of course cater for multiple instances.
My initial idea was to run all the instances a dumb slaves,...
- 344
1
vote
1
answer
2k
views
Mounting an S3 bucket onto a AWS Ubuntu instance issues
i'm using s3fuse to mount an s3 bucket onto ubuntu, so that i can take advantage of the storage space.
here are some environment info:
System Info:
Linux ip-xxx-xxx-xxx-xxx 2.6.38-8-virtual #42-...
- 259
2
votes
1
answer
3k
views
Writing to an s3 mounted directory
hopefully someone can advise.
I've recently mounted an s3 bucket to a directory /s3 on my ubuntu amazon ec2 instance -following: http://code.google.com/p/s3fs/wiki/FuseOverAmazon. This was set up in ...
- 1,167
5
votes
2
answers
1k
views
S3 based file system capable of requesting only part of file
I'm storing large datasets in s3, but on a given computer in my cluster, my program only needs to read a small subset of the data.
I first tried s3fs, but it downlooads the entire file first, which ...
- 1,116
0
votes
1
answer
3k
views
s3fs and fuse: failed to open /dev/fuse: Permission denied
Just installed Debian 6 box. I wanted to mount my S3 bucket for some backups.
So I followed instructions for http://code.google.com/p/s3fs/ .
1) I installed all perquisites asked in installation ...
- 103
0
votes
3
answers
1k
views
Bacula over s3fs
We are constantly running out of space on our backup server so I want to move the backups to S3. Does it make sense to run Bacula on top of s3fs? Does anybody have experience with that? Is it reliable?...
- 7,478
3
votes
1
answer
3k
views
s3fs or EBS through nfs? [closed]
Which one is better?
I need to store a lot of static files ... and I wonder which one is better, cheaper and faster.
- 201
3
votes
3
answers
5k
views
s3fs: how to force remount on errors?
I use s3fs 1.33 on Ubuntu 9.10.
Regularily it gives me errors like this:
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: close failed on "/mnt/s3/...
- 2,443
7
votes
1
answer
14k
views
How to force s3fs mount on boot
I'm using s3fs 1.33 on Ubuntu 9.10 x86.
I mount it manually with this command:
sudo /usr/bin/s3fs mybucket /mnt/s3/mybucket -ouse_cache=/tmp
How can I force the mount on boot?
I guess that I have ...
- 2,443
4
votes
4
answers
5k
views
Best remote mounted filesystem based on S3 [closed]
I need to know what the best (read: highly performant and fault tolerant) filesystem solution based on Amazon's S3 is. I have looked a s3fs, but I notice that it has not been worked on for a while now,...
- 927
4
votes
4
answers
9k
views
s3fs Input/output error
I'm trying to set up a backup system with s3fs and the Amazon S3 service.
I followed this this guide and this guide.
Tailing /var/log/messages I get:
Aug 28 13:37:46 server s3fs:###response=403
I ...
- 376