Questions tagged [amazon-s3]
This is for questions on Amazon's cloud storage service.
280
questions with no upvoted or accepted answers
8
votes
1
answer
1k
views
Amazon S3 appears to be blocking cloudflare IP addresses. How do I fix it?
I have static content that is being served by Cloudflare. Cloudflare points to Amazon S3 to pick up the static content and serves it via a CNAME (cdn.mydomain.com). The bucket is setup properly and ...
- 181
6
votes
0
answers
1k
views
How does S3FS (or any other S3 FUSE filsystem) compare to AWS FSx for Lustre + S3
I remember trying s3fs a year back, trying to use some S3 bucket as a FUSE filesystem. I remember it being quite laggy, especially when coupled with git operations on it (an oblivious system architect ...
- 245
6
votes
1
answer
2k
views
S3/Cloudfront CORS working on some files only
I'm really struggling to get Cloudfront and S3 to add Access-Control-Allow-Origin: * to the headers of video files stored on S3 (for inline video on iPhones - seems to be working everywhere else ...
- 225
5
votes
0
answers
5k
views
How to handle trailing slash in a redirect rule for an AWS S3 website?
I'm trying to build up some landing pages in my s3 hosted web site.
For example: http://www.example.com/products
That should redirect to http://www.example.com/products.html
To accomplish this, I ...
- 365
5
votes
1
answer
894
views
Syncing / backup from S3 buckets to openstack swift containers
I need to backup/sync few hundred GB of files in S3 buckets to openstack swift containers.
I have thousands of files in several S3 buckets I want to copy/sync over openstack swift containers but I ...
- 51
4
votes
0
answers
1k
views
Understand S3 cost at folder level
I am planning for a use case wherein which my S3 bucket is used by 10 different users. All these users has separate folders within this bucket, to where they'll store their files. Now I want to know ...
- 764
4
votes
0
answers
3k
views
How to remove orphaned s3 bucket
I have an s3 bucket that I can see in the console or with aws s3 ls/aws s3api list-buckets but the bucket appears to be in some sort of orphaned state.
It's missing the Access and Region fields in ...
- 141
4
votes
1
answer
3k
views
Does CloudFront support S3 signature version 4 for KMS encrypted objects?
I'm using Cloudfront with an S3 origin that is using KMS to encrypt objects. I'm getting the following error when sending a GET request for an object in the S3 bucket.
Requests specifying Server ...
- 191
4
votes
0
answers
196
views
S3 + Cloudflare vs S3 + 2 Web Server + Cloudflate
Currently, I run two websites and each site own it own big server. Currently, all of those two sites has roughly 300GB of images. Its grow rate is 15GB of images per month. I know that eventually, one ...
- 405
3
votes
0
answers
335
views
Downloading fails at random for our users from S3, need help figuring out what's wrong
We have a file sharing service where users can upload, share and download their files. We are using AWS S3 to store our files.
We're facing an issue where small files download fine, but when files ...
- 131
3
votes
2
answers
272
views
AWS Opsworks Chef deployment using S3 Archive without making archive public
I'm trying to use the S3 Archive method to do a Chef deployment in Opsworks. I put the Chef archive in S3 and copied the address of https://s3.us-east-2.amazonaws.com/redacted/redacted.tar.gz. I gave ...
3
votes
1
answer
1k
views
AWS: SES Rule: S3 Action: fails when choosing bucket that has encryption enabled
I can not create a SES rule to put emails into a S3 bucket that has encryption enabled (on the bucket).
Created a bucket and enabled encryption.
Add a SESPut bucket policy to allow SES. https://docs....
- 419
3
votes
0
answers
627
views
How to have frontend, on s3, and backend on another server with the same domain
I have the frontend part of an application on Amazon S3 and the backend on cloud66, with NGinx. The DNS have 'www.website.com' pointing on S3 and 'api.website.com' pointing on nginx. I want to use the ...
- 167
3
votes
0
answers
2k
views
CloudFront + S3 Access Denied Using Signed Cookies
I'm trying to integrate signed cookies for a CloudFront distribution that's fed off of an S3 bucket but getting access denied error messages:
<Error>
<Code>AccessDenied</Code>
...
- 31
3
votes
0
answers
776
views
Set up a different index document per directory with S3 static website hosting
I am using S3 static website hosting, and trying to get the following redirects to work:
example.com --> example.com/index.html
example.com/foo --> example.com/foo/foo.html
example.com/bar --&...
- 147
3
votes
0
answers
523
views
Solution for file data backup, versioning and replication
We have a growing set of data files (.wav files, image files etc) which are data, i.e not part of the application code - uploaded and modified by users. The number of files is in the 1000s and the ...
- 227
2
votes
1
answer
356
views
Since S3 charges by request, couldn't a malicious hacker cause a huge AWS bill just by spamming requests?
What would stop them from doing so, against, say, a static website hosted using S3? Is there a good way to deny some requests such that one avoids getting billed for them?
(Context: I want to host a ...
- 121
2
votes
0
answers
2k
views
Access denied for S3 bucket with redirect access via CloudFront
I have public S3 bucket exposed via CloudFront to S3 A record for www.mezinamiridici.cz.
https://www.mezinamiridici.cz/ - works
http://www.mezinamiridici.cz/ - redirects to https://www.mezinamiridici....
- 305
2
votes
0
answers
390
views
gsutil rsync fails on Ubuntu with UnicodeDecodeError
I'm trying to rsync between google cloud storage and amazon s3. On my local machine (MacOS) this command works
gsutil rsync -d -r gs://mira-internal/data/exports/20191108 s3://mira-temp/raw/20191108
...
- 71
2
votes
0
answers
388
views
S3 GetBucketAccelerateConfiguration operation: Access Denied
I need to grant user access to get AccelerateConfiguration for one bucket.
I set next permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "...
- 21
2
votes
1
answer
589
views
Use S3FS Mount As HTTP Server Document Root
My proposed setup:
An EC2 server running Apache or nginx.
An S3 Bucket in the same region/availability zone mounted with S3FS.
The Apache or nginx document root set to the S3FS mount.
My question: ...
- 414
2
votes
0
answers
683
views
Server access logging on AWS S3
Delving into the AWS world, I have created two buckets in my AWS. The first one called BucketA (holding the files) and the second one called BucketB (holding the log files). BucketA will be hosting ...
- 167
2
votes
1
answer
2k
views
Cloudfront (w/ S3) Static Website Doesn't utilize index.html from sub directories
I have my website hosted on S3 with CloudFront as a CDN, and I need these two URLs to behave the same and to serve the index.html file within the directory:
example.com/directory
example.com/...
2
votes
1
answer
107
views
Serving backup/fail-safe object on AWS S3
What would be the easiest way to serve backup/fail-safe object when the originally requested object is not available in the bucket?
For example, currently our users can upload custom profile photos ...
- 21
2
votes
0
answers
826
views
log cloud foundry syslog to s3 using the syslog protocol
I want to send my syslog entries to an S3 bucket. Security restrictions have meant that this is the only ways that I can do this.
With cloud foundry, you can create a user provided server where I ...
- 247
2
votes
1
answer
839
views
AWS SES Encryption vs S3 bucket encryption
SES
Configuration for a SES rule allows one to have the email be put into a S3 bucket.
Under this configuration, the option "KMS Key" is available, which would have SES encrypt the email before ...
- 419
2
votes
0
answers
21
views
Proper way to handle file serving
We have a website where we our daily bandwidth use is roughly 10GB of user generated contents.
The software layer is simply Heroku + some addons for DB and monitoring,
and we are handling the uploads ...
- 21
2
votes
0
answers
895
views
How to get clean URLs in CloudFront
I'm hosting my static website on AWS S3, with Cloudfront as a CDN, and I'm wondering how I can get clean URLs working.
I currently have to go to example.com/about.html to get the about page. I'd ...
2
votes
0
answers
736
views
rpc.mountd[pid]: /xxx and /yyy have same filehandle for host, using first
I have an S3 bucket mounted using s3fs-fuse at /yyy and a normal drive mounted at /xxx. I am exporting both with NFS however when a client attempts mount, I see the following error:
rpc.mountd[pid]: /...
- 491
2
votes
0
answers
772
views
Reverse Proxy on cloudfront
I have an angular site / web application (Lets call this www.mysite.com) hosted on Amazon S3 as a static website.
This s3 bucket serves as the origin for a cloudfront distribution and it works ...
- 121
2
votes
1
answer
676
views
Route 53 configuration for Cloudfront distribution linked to S3 bucket for naked domain
I am using route 53 for a domain (example.com) linked to my app on Heroku. To handle naked domain redirection, I am using a S3 bucket as explained here:
https://aws.amazon.com/blogs/aws/root-domain-...
- 121
2
votes
1
answer
2k
views
How backup files with logrotate and Amazon S3?
I would like to create a new standard of two things:
how long would be generated and be rotated the logs generated by my applications ?
how to transfer the logs to Amazon S3, as a backup server ?.
...
- 155
2
votes
1
answer
1k
views
Amazon S3 Website & MySQL Backup
I have a website (a digital asset management system/gallery - http://www.resourcespace.org) that has a huge amount of images. The total size of the website, including the images is approximately 6gb.
...
user991830
2
votes
0
answers
593
views
s3fs-mounted S3 bucket shows bad filesizes
Background:
I have an Amazon S3 bucket with data uploaded using the AWS web interface. Now I need to give permissions-based access to paths within my buckets to FTP users. To do this, I'm running ...
- 131
2
votes
0
answers
321
views
Gluster with s3ql mounted brick?
I am trying to test run Gluster with a volume consisting of two local bricks, the first being a plain directory and the second being a s3ql based mount.
Having just the plain directory as a unique ...
- 848
2
votes
0
answers
937
views
Web-based file manager like File Thingie, but for S3?
At work, we use a little PHP script called File Thingie, which is a web-based file manager. Basically, an FTP replacement. It allows users to upload and download files, and most importantly, it has a ...
- 233
2
votes
1
answer
739
views
How to access an Eucalyptus Walrus(S3) using Amazon AWS SDK
I have downloaded the Amazon AWS SDK for C#, I have no problem accessing the EC2 part of our private cloud running Eucalyptus, I can list, Images, Instances, Zones ...
This is working fine :
...
- 110
1
vote
0
answers
348
views
Amazon S3 life cycle rule for sub folder contents?
With this reference link
My s3 bucket name say testbucket.
If I create prefix for a rule say.
subfolder1/
All the objects in it are applied this rule.
But it is not getting applied for objects inside ...
- 155
1
vote
0
answers
317
views
AWS S3 Bucket Static Website Index/ Home Access Denied
So i have problem with AWS S3 Bucket Static when i try to access my domain without filename.
http://customdomain/ >> Error access denied
http://customdomain/index.html >> normal
but it's ...
- 23
1
vote
0
answers
196
views
making all S3 uploads to digitaloceans spaces default to being public
I'm using DigitalOcean Spaces to upload a bunch of files in different directories. In particular, I'm using WinSCP to do the upload via the S3 protocol.
The problem is that, per WinSCP, all the files ...
- 317
1
vote
0
answers
60
views
Reduce Query Syntax for ends_with()
I have the following query:
--query "Contents[?ends_with(Key, 'jpg') || ends_with(Key, 'NEF') || ends_with(Key, 'nef') || ends_with(Key, 'CR2') || ends_with(Key, 'cr2')]"
Is there a way to ...
- 111
1
vote
0
answers
158
views
Digital Ocean Space Cors with private files
I am unsure whether I am misunderstanding CORS or whether this is possible within a Digital Ocean Space.
Basically, I want to set a file to private on the DO Space. I only want a certain domain to be ...
- 11
1
vote
0
answers
840
views
Unable to resolve AWS S3 and other DNS from Docker container
We have an API that puts an object to AWS S3 using the Java API. It works fine from the locations below:
From our local dev workstations
From within production docker containers in EC2 machines.
But,...
- 111
1
vote
1
answer
1k
views
S3 access control based on bucket tags
i hope you can help me out. I have read a couple of docs now, and I am still unsure whether this actually works.
I want to give access in different levels to AWS users based on S3 tags.
Example:
S3 ...
- 203
1
vote
0
answers
1k
views
CloudFront CORS request using signed cookies and withCredentials, not sending back Access-Control-Allow-Credentials unless I include some extra header
I'm having a very strange issue that I can't seem to crack. I configured a private CloudFront distribution to serve content from a private S3 bucket. I am using signed cookies to grant access to the ...
- 61
1
vote
1
answer
110
views
S3 policy to allow by IP can't be changed
We have an S3 bucket with website content in it that's served to the public through a proxy cache. The access to the bucket is limited to the IPs used by the proxy. This is all working fine.
New IPs ...
- 633
1
vote
0
answers
104
views
Calculation of S3’s byte-hour
S3 uses the concept of Byte-Hour in billing: https://aws.amazon.com/s3/faqs/#Billing
Unfortunately (in typical Amazon AWS style) they left some details unsaid, namely what happens in the case of ...
- 11
1
vote
0
answers
142
views
How do I block TCP ACK causing retransmissions
I am trying to replicate a issue when making requests to S3 that will lead to tcp_retries2 leading to 900 seconds of retransmission attempts before giving up. How would I do this? Is it sufficient to ...
- 11
1
vote
1
answer
1k
views
S3 life cycle policy - move to SIA only if size bigger than 128KB
Usecase description:
I'm planning to set a life cycle policiy to move objects from frequent access to infrequent access automatically after 30 days. but in the bucket, there are some small files that ...
1
vote
0
answers
323
views
S3 through Nginx Proxy doesn't allow more than 10 files to be downloaded simultaneously
I setup a Nginx Proxy in front of my S3 object storage and I'm able to download files without any issues so far.
However, if I try to download the same file from, let's say, 30 different servers, at ...
- 111