Questions tagged [amazon-s3]
This is for questions on Amazon's cloud storage service.
1,124
questions
2
votes
1
answer
733
views
What is the purpose of version dates in AWS policy documents
AWS policies contain a date version. What is its purpose? In this example a bucket policy and a user policy both contain "Version": "2012-10-17" in the JSON. I don't see the ...
0
votes
0
answers
158
views
CloudFront Access Logs Issue
I was tried querying our CloudFront logs using Athena for an error i'm getting but haven't received any results. The query i'm trying using is:
SELECT *
FROM cloudfront_logs
WHERE request_id =
> ...
- 81
1
vote
0
answers
60
views
Reduce Query Syntax for ends_with()
I have the following query:
--query "Contents[?ends_with(Key, 'jpg') || ends_with(Key, 'NEF') || ends_with(Key, 'nef') || ends_with(Key, 'CR2') || ends_with(Key, 'cr2')]"
Is there a way to ...
- 111
0
votes
0
answers
728
views
S3 Fatal error: Failed to connect to proxy URL: "Proxy url"
Good Afternoon,
I have been trying to remove some files from S3, last week I did not have any issues accessing my s3 bucket however now any s3 cmd fails with the error message above, I cant even run ...
- 9
0
votes
0
answers
232
views
How suitable is a S3 backend for rsync operations?
I run a large server providing open source software (https://ftp.halifax.rwth-aachen.de), currently serving more than 30 TByte of data with Multi-Gigabit throughput. Data is synchronized and kept up-...
- 334
2
votes
1
answer
356
views
Since S3 charges by request, couldn't a malicious hacker cause a huge AWS bill just by spamming requests?
What would stop them from doing so, against, say, a static website hosted using S3? Is there a good way to deny some requests such that one avoids getting billed for them?
(Context: I want to host a ...
- 121
1
vote
1
answer
515
views
Why would anyone disable S3 KMS "bucket key" option?
As per the documentation:
Amazon S3 Bucket Keys reduce the cost of Amazon S3 server-side
encryption using AWS Key Management Service (SSE-KMS). This new
bucket-level key for SSE can reduce AWS KMS ...
- 563
1
vote
1
answer
1k
views
RDS PostgreSQL Import+Export to/from S3
Is it possible to enable both importing and exporting with an RDS PostgreSQL instance to an S3 bucket? I've been able to use the following pattern to enable one or the other with consistent success:
...
- 394
3
votes
1
answer
2k
views
Google Cloud S3 & rclone: Error 400: Cannot insert legacy ACL for an object when uniform bucket-level access is enabled
I'm trying to configure rclone to access a google cloud bucket (which happens to be managed by terra.bio. My rclone config reads as follows:
[terra]
type = google cloud storage
object_acl = private
...
google-cloud-platform
0
votes
0
answers
73
views
Serve cephFS files over HTTP (or S3?)
I have a CEPH+LXD cluster configuration.
And I would like to serve the files in the container over HTTP (S3,whatever doesn't matter).
Is there a way to do this using the rgw service?
Like can I point ...
- 11
1
vote
1
answer
316
views
AWS s3 sftp without AWS transfer family
Are there any other ways to upload files from an Ubuntu server to AWS s3 without AWS Transfer Family? I have to make a MYSQL dump every five minutes and upload to the bucket but the Transfer Family's ...
- 13
0
votes
1
answer
87
views
Rails 6 - Understanding AWS Pricing for CDN video on demand application
I am working on an Rails 6 application, already on production which is heavily dependent on AWS Services like S3, Lambda, Cloudtrail, SNS and CDN to serve a video-on-demand app, where users can upload ...
- 101
0
votes
1
answer
190
views
Backup from S3 to S3
I'd like to backup multiple S3 buckets into another S3 buckets. It seems like most backup tools only support S3 as a destination, but not as a source e.g. restic. The backup should support pruning, ...
- 129
-2
votes
1
answer
272
views
Storing Blog posts in Amazon s3 storage vs Mysql
I am planning to create a SASS like blogger.com , In future there may be million and millions of posts.What is the best way to store this.
Example of my Mysql columns
Title , Date , author , Content
...
- 97
1
vote
0
answers
158
views
Digital Ocean Space Cors with private files
I am unsure whether I am misunderstanding CORS or whether this is possible within a Digital Ocean Space.
Basically, I want to set a file to private on the DO Space. I only want a certain domain to be ...
- 11
1
vote
0
answers
840
views
Unable to resolve AWS S3 and other DNS from Docker container
We have an API that puts an object to AWS S3 using the Java API. It works fine from the locations below:
From our local dev workstations
From within production docker containers in EC2 machines.
But,...
- 111
0
votes
0
answers
463
views
Create Windows 10 AMI for EC2 AWS
I have case to create AMI based on windows 10 OS, the AMI file will be used on AWS ec2 to run some our 3rd party programs for client. I have tried to follow several tutorial and guidelines from google ...
0
votes
0
answers
1k
views
How to read files from a directory having name "/" in S3 bucket?
Code:
val df = spark.read.csv("s3a://sample_bucket//csvFiles/file.csv");
Error:
22/06/23 20:02:57 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system....
1
vote
1
answer
793
views
Fargate task from service with Public IP disabled can't download env file from S3
We have a Fargate service that should be exposed to the internet via a load balancer, and since for tests we had used so far the random Public IP of the task, we decided to disable the Public IP, so ...
- 111
0
votes
1
answer
673
views
nginx proxy_pass fallback location
I'm using nginx as a reverse proxy for my web application with proxy_pass for the api.
This is my current config, where I've nginx directive location /data/ where images are being displayed from the /...
- 1
0
votes
0
answers
918
views
Breakdown of AWS S3 DataTransfer-Out-Bytes costs per object
I have an app where various customers upload files to a single S3 bucket in my AWS account. Over time the costs of DataTransfer-Out-Bytes have grown and I want to see if there's some way to understand ...
- 1
1
vote
1
answer
1k
views
S3 access control based on bucket tags
i hope you can help me out. I have read a couple of docs now, and I am still unsure whether this actually works.
I want to give access in different levels to AWS users based on S3 tags.
Example:
S3 ...
- 203
0
votes
0
answers
2k
views
Unable to access few objects in S3
Why some objects are accessible in S3 bucket and some are not. Following are the below errors we faced. Though I checked the access and it seems to be fine. Exception while connecting S3: An error ...
- 1
1
vote
1
answer
10k
views
How do I extract a list of all folders of my AWS S3 storage bucket or directories?
Does Amazon provide an easy way extract a list of all folders that have files greater than 500 MB from a s3 bucket? want to limit the scope to the '/files/ftp_upload/' directories also This is so I ...
- 59
0
votes
1
answer
814
views
AWS Policy to Read/write RDS
In my scenario , I want a policy that will allow reading and writing of abc-database-backups/rds/postgresql-backup on S3? We'll want the my servers to have that access added.
Is creating a role and ...
- 1
0
votes
1
answer
169
views
How do I open a private static website using boto3?
I have a static website consisting of multiple HTML, CSS, and media files uploaded to a private S3 bucket. I would like to access the website using my web browser.
I'm currently using boto3 to ...
- 101
0
votes
1
answer
695
views
Trying to understand S3 Lifecycle rule
I am working on an e-commerce website.
I store product photos in an s3 bucket. Once the product is deleted, I also delete the photos from s3 bucket.
I have S3 bucket versioning enabled. I am not ...
- 528
2
votes
2
answers
2k
views
What is the difference between S3 Multi-Region Access Points and AWS CloudFront?
Can anyone explain the difference between S3 Multi-Region Access Points and AWS CloudFront?
The documentation for a multi-region access point states:
Multi-Region Access Points offer a global S3 ...
- 121
0
votes
1
answer
3k
views
How to download millions of s3 files and compress them on the fly?
I have an S3 bucket with millions of files, and I want to download all of them.
Since I don't have enough storage, I would like to download them, compress them on the fly and only then save them. How ...
- 1
1
vote
0
answers
1k
views
CloudFront CORS request using signed cookies and withCredentials, not sending back Access-Control-Allow-Credentials unless I include some extra header
I'm having a very strange issue that I can't seem to crack. I configured a private CloudFront distribution to serve content from a private S3 bucket. I am using signed cookies to grant access to the ...
- 61
0
votes
1
answer
293
views
Bandwidth consumption of Amazon S3 bucket mounted with S3fs on a VPS
I have :
VPS server on Vultr with web server and Wordpress installed.
Uploads folder is mounted on Amazon S3 bucket using s3fs.
If a user downloads a 1GB file from my website will the traffic be ...
- 1
1
vote
1
answer
110
views
S3 policy to allow by IP can't be changed
We have an S3 bucket with website content in it that's served to the public through a proxy cache. The access to the bucket is limited to the IPs used by the proxy. This is all working fine.
New IPs ...
- 633
0
votes
1
answer
150
views
S3 Logs event Issue
Is there a way to see what actions the 'g2' IAM user is performing in S3, and which IP(s) they are running from? I have already enabled the logging of S3 actions.
One point I’m still not able to ...
- 81
0
votes
1
answer
60
views
Sending push notification to s3 origin website from lambda
I want to send a websockets notification from lambda function to clients, on a website hosted via s3 origin with cloudfront.
Could someone help me with this, please? Thanks.
- 3
0
votes
1
answer
364
views
aws cloudfront custom ssl pricing for subdomain and www
I have created two s3 origin website linked with cloudfront and used acm certificate.
But I recently found out pricing are different between https://admin.example.com and https://www.example.com.
www....
- 3
0
votes
0
answers
688
views
CloudFront delivery is slow in particular location
In our scenario, I found that our CloudFront delivery particularly in the Japan area is very slow for a few days. which lead our site to become slow in the time of loading the media contents .our s3 ...
- 81
1
vote
0
answers
104
views
Calculation of S3’s byte-hour
S3 uses the concept of Byte-Hour in billing: https://aws.amazon.com/s3/faqs/#Billing
Unfortunately (in typical Amazon AWS style) they left some details unsaid, namely what happens in the case of ...
- 11
0
votes
1
answer
191
views
DNS - GoDaddy for mail and AWS for rest
I have a GoDaddy hosting account but for some of my websites I would like to host them in AWS S3. My question is how to set DNS entries in Route 53 so that I keep web traffic served from S3 but I ...
- 1
1
vote
0
answers
142
views
How do I block TCP ACK causing retransmissions
I am trying to replicate a issue when making requests to S3 that will lead to tcp_retries2 leading to 900 seconds of retransmission attempts before giving up. How would I do this? Is it sufficient to ...
- 11
0
votes
1
answer
18
views
Setting up an architecture for analyses flows of time stream data from multiple sources [closed]
What would be the best practice in terms of AWS for the following:
Many IOT medical devices gather data at around 256kBps
The data is a time series data (a matrix of [Channels X Samples], there can ...
- 1
0
votes
2
answers
1k
views
Unable to mount a folder in Ubuntu to as AWS S3 Bucket
I am attempting to mount a folder on a fresh AWS EC2 instance running Ubuntu 18.04, to AWS S3.
I've been following instructions I found at https://cloud.netapp.com/blog/amazon-s3-as-a-file-system and ...
- 101
1
vote
1
answer
785
views
Will aws s3 be still accessible using pre-signed urls if we create a vpc endpoint gateway?
I have a need to add a vpc endpoint for s3 so i can access it on lambda, but when i try to create the endpoint I get a warning
Warning
When you use an endpoint, the source IP addresses from your ...
- 109
1
vote
1
answer
1k
views
S3 life cycle policy - move to SIA only if size bigger than 128KB
Usecase description:
I'm planning to set a life cycle policiy to move objects from frequent access to infrequent access automatically after 30 days. but in the bucket, there are some small files that ...
0
votes
2
answers
2k
views
Terraform: "S3:path_to_terraform.tfstate": x509: cannot validate certificate for <IP> because it doesn't contain any IP SANs
I'm using s3 to save my state. since this is a custom S3 (and not AWS) I configured it as shown below (using IP + Port).
when running terraform init I received this error:
Successfully configured the ...
- 103
0
votes
0
answers
19
views
AWS S3 bucket policy does not define Put, yet it is allowed [duplicate]
I have a public static website which images can be uploaded to. For this I enabled static website hosting and set Block public access to OFF. I then added this policy
{
"Version": "...
- 133
0
votes
0
answers
2k
views
AWS S3 bucket policy for get and upload
I have two buckets each with their own goals. So far I cannot understand the complicated configuration of bucket settings. It seems there are three things to configure
Block public access
Bucket ...
- 133
2
votes
2
answers
1k
views
AWS S3 Static Website vs CloudFront OIA
It seems there are kind of two ways to host a static website on S3/CloudFront.
Create S3 bucket with full public access and Static Website set to Enabled + CloudFront config
Create S3 bucket with no ...
- 133
0
votes
1
answer
351
views
API gateway cache every hour
My API is set up as follows:
API Gateway > Lambda function > S3
Every hour the S3 content is updated at the same time. How do I configure API gateway to cache the endpoint's responses but serve ...
- 107
0
votes
1
answer
530
views
how to download archive in aws s3 glacier
first, sorry for my bad english.
i create a vault in glacier service and then i uploaded an archive into vault and in out-put, aws gave back me a archive id. then using the archive-id, i created a job ...
- 89
0
votes
0
answers
64
views
AWS/S3/Cloudfront/GoDadday and subdomains
I am wondering and hoping this is possible.
On AWS I have two s3 buckets to separate a client facing application and an administrative application
mysite-app
mysite-admin
I have a cloudfront ...
- 133