Questions tagged [amazon-s3]
This is for questions on Amazon's cloud storage service.
1,124
questions
0
votes
1
answer
331
views
Backing up an SQL backup file from EC2 to Glacier
I have an EC2 server running Sendy (bulk mailing) and I wanted to take backups of the SQL db. Currently, I can download the backup.sql file manually via Virtualmin but I wanted to automate this.
I've ...
- 103
3
votes
1
answer
2k
views
Trying to receive emails AND store them into an S3 bucket
I'm trying to store emails i receive into an s3 bucket, i followed this tutorial and multiple others :
https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-getting-started.html
My MX ...
- 129
1
vote
1
answer
1k
views
Flask app on ECS that pull images from s3 bucket
I have a flask app that I want to pull images to display from an s3 bucket.
How do I implement the credentials parts? what I mean is:
Do I use boto3 with aws credentials to pull the images from the ...
- 113
0
votes
0
answers
30
views
Can I create a lifecycle rule to AWS S3 that moves objects that are already in GLACIER class to DEEP_ARCHIVE after 180 days?
My company has probably close to a petabyte of data in s3 and we've been using S3's GLACIER class to get some cost savings. The files that are already in glacier were manually chosen to be put there ...
- 363
1
vote
1
answer
5k
views
Using aws cli sync include and exclude with delete option [closed]
aws s3 sync /WordProcessing/DOCUMENTS s3://mybigbucket --delete --include "*" --exclude ".DS_Store"
If somehow .DS_Store made its way to the AWS S3 mybigbucket (from a previous backup), using the ...
- 965
-1
votes
3
answers
1k
views
aws life cycle rules to transition files from s3 bucket to Glacier storage class
I am creating life cycle rules to move files from bucket to the glacier storage class.
I was wondering if there is an option to copy files from S3 bucket to Glacier using either CLI or console?
I ...
- 109
6
votes
1
answer
7k
views
Does AWS cli do a data integrity check on sync with s3? [closed]
I've been looking into using AWS cli for data integrity checks to verify a backup has been transferred from a Linux file server correctly to AWS s3. Likewise, I would like to verify when restoring a ...
- 965
0
votes
1
answer
1k
views
Is there an additional cost to write metadata to objects in the bucket? [closed]
I was looking into writing an MD5sum to the object after it has been uploaded to AWS S3 and doing a data integrity check as described here:
https://aws.amazon.com/premiumsupport/knowledge-center/data-...
- 965
0
votes
1
answer
954
views
AWS FSx for lustre with S3 vs EMR (with EMRFS) for spark jobs
We are currently using EMR for easy job submission for our spark jobs.
Recently I came across the "FSx lustre + S3" solution that is being advertised as ideal for HPC situations.
EMRFS however is also ...
- 245
-1
votes
2
answers
73
views
Backup of multiple sites on Ubuntu
I have multiple sites in my /var/www/html/ folder like following:-
/var/www/html/site-1
/var/www/html/site-2
/var/www/html/site-3
/var/www/html/site-4
i want to zip each site on my server and to ...
- 2,009
1
vote
1
answer
179
views
Export 20TB from Amazon S3
I have recently inherited 20TB of data after a company takeover. The files are currently stored on an Amazon S3 account which is due to close.
What would be the most cost effective method of ...
- 113
0
votes
1
answer
701
views
Where is the AWS S3 metadata stored written by s3cmd? [closed]
I was looking for a AWS S3 solution for backing up a Linux file system so that it would preserve the timestamp and permissions. It appears that s3cmd allows this by writing this information to the ...
- 965
0
votes
1
answer
2k
views
Access Denied using S3 upload function
I have a lambda function with the following policy
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:ListBucket"
],
"...
- 121
6
votes
0
answers
1k
views
How does S3FS (or any other S3 FUSE filsystem) compare to AWS FSx for Lustre + S3
I remember trying s3fs a year back, trying to use some S3 bucket as a FUSE filesystem. I remember it being quite laggy, especially when coupled with git operations on it (an oblivious system architect ...
- 245
0
votes
1
answer
764
views
EBS vs EFs vs FSx vs S3 for feeding data to a distributed machine learning model?
Lets says there are 1TB of data to be simultaneously fed to 100 EC2 consumer instances. Each instance accesses random chunks of data of size 10MB. What would be the considerations to take into account ...
- 101
2
votes
1
answer
285
views
Best strategy for daily back up of terabytes of data with millions of files
Currently I create an NFS and mount it on a special server that does the actual backups and uploads it to s3. My main concern is I feel like overcomplicated this simple task. I schedule cron job to ...
- 174
0
votes
1
answer
3k
views
How to upload multiple files from directory to S3?
I have a directory on an Ubuntu, with 340K images, 45GB of total size! Is there an efficient way to transfer them all to an S3 of DigitalOcean?
I thought of using s3cmd put or s3cmd sync but I'm ...
- 198
0
votes
1
answer
227
views
Transfer image files from MongoDB to S3
I have a NodeJS app that stores all of its data, even images, to MongoDB. Now I want to use an S3 to store my app's files (images, 43gb) and so, I need to transfer there all files that I have in my ...
- 198
1
vote
1
answer
3k
views
s3 static site: Amazon using wrong bucket
I'm setting up a static site, hosted on S3, using this set of directions:
https://www.linode.com/docs/platform/object-storage/host-static-site-object-storage/
I've set my bucket up with the name ...
- 13
1
vote
0
answers
73
views
Spectra S3 Java CLI (DS3) difficulty verifying checksum
We have a SpectraLogic BlackPearl+T950 which uses S3 style commands to put/get/sync etc.
To access the Spectra flavour of S3 commands we use a Java CLI client to perform these operations.
Supposedly ...
- 141
2
votes
2
answers
864
views
Redirecting HTTPS -> HTTPS on AWS?
I have a static website w/ S3 + Route53 + Cloudfront set up. I am successfully able to do the following redirects:
http://example.com -> https://example.com
http://www.example.com -> https://example....
- 121
0
votes
1
answer
301
views
Grant access to role in another AWS account to all objects in my bucket
So I want to grant a role that exists in account A the get-object permission on items in a bucket owned by account B.
I read The AWS docs' example policies and created a bucket policy like this:
{
...
- 101
2
votes
0
answers
390
views
gsutil rsync fails on Ubuntu with UnicodeDecodeError
I'm trying to rsync between google cloud storage and amazon s3. On my local machine (MacOS) this command works
gsutil rsync -d -r gs://mira-internal/data/exports/20191108 s3://mira-temp/raw/20191108
...
- 71
0
votes
0
answers
16
views
Traffic handling capacity for 1 EC2 instance for t3.2xlarge [duplicate]
Protocol exchange between phone server and IOT device(inter regional). say there are 100000 users and each are requesting or using 12 - 20 times a day may be approximately at the same time(protocols ...
2
votes
1
answer
6k
views
What is the maximum bandwidth (upload and download) and number of connections for an s3 bucket?
I'm using s3 to feet multiple terabytes of data into a bioinformatics pipeline. I've got 20 nodes all in the same subnet copying a file from s3://mybucket.com, processing it, and uploading the results ...
- 363
3
votes
1
answer
842
views
Guidance on Offline/Online Backup for AWS EC2, RDS & S3 [closed]
I have some EC2 instances (Linux and Windows) with attached EBS Volumes, some MySQL Database and S3 buckets in an AWS account. I am in a situation where
I won't have time to work on this project ...
- 33
1
vote
2
answers
2k
views
Digital Ocean Spaces S3: Is it possible to add access-control-allow-origin: * to all requests?
If you curl my image
curl -i https://local-spaces.fra1.digitaloceanspaces.com/test.jpg
you will see that there is no Access-Control-Allow-Origin header in the response.
If you curl like this, the ...
- 165
1
vote
0
answers
206
views
Mixing private and public static files in AWS is either fully private or fully public
I am trying to move my static assets to AWS. I am following many blog posts but most of all this one: https://simpleisbetterthancomplex.com/tutorial/2017/08/01/how-to-setup-amazon-s3-in-a-django-...
- 231
0
votes
1
answer
1k
views
How to route to Cloudfront+s3 and rest to ALB in Route53?
The project has a domain name foobar.com pointing to a legacy system, hosting and serving mainly static files. The project got more complex over time and in AWS ALB certain /routes are pointed to ...
- 221
1
vote
1
answer
584
views
Understanding how AWS charges S3
This is not a troubleshooting question, rather a more generic one. I have read about how AWS S3 is priced - in terms of storage used, data transfer, requests and other features used. And wrt storage ...
- 764
2
votes
2
answers
3k
views
AWS Deny NotPrincipal Bucket Policy
I have an AWS S3 bucket called test33333 I need to lock down to minimum necessary permissions. I've created a bucket policy to Deny all except user account MyUser a role MyRole. (account name xxx out)...
- 31
0
votes
1
answer
1k
views
Amazon CloudFront default SSL certificate for multiple domain names
A few years ago I setup an S3 bucket and enabled hosting. It's got an index.html file, which I use as a simple landing page. I then setup AWS CloudFront to use this S3 bucket as an origin server and ...
0
votes
1
answer
784
views
What permissions are required to run AWS TransferUtility?
Using the AWS .NET SDK's TransferUtility, I can download files using an access key ID for myself, but when I set up a restricted user with read-only access, I get "The remote server returned an error: ...
- 91
0
votes
2
answers
2k
views
nginx proxying S3 public bucket hosted by minio service
I'm trying to do what I would have thought was the easiest proxy directive in the world, proxying a public S3 bucket.
This is my config:
server {
listen 80;
listen [::]:80;
server_name ...
4
votes
0
answers
1k
views
Understand S3 cost at folder level
I am planning for a use case wherein which my S3 bucket is used by 10 different users. All these users has separate folders within this bucket, to where they'll store their files. Now I want to know ...
- 764
0
votes
1
answer
436
views
Local Container Registry using S3 Bucket and Proxy Server
I am trying to deploy a local docker registry on AWS EC2 instances. The instances must reach out through a foraward proxy (apache) server. However, I am unable to reach the s3 bucket from within the ...
- 121
1
vote
1
answer
334
views
S3 hosted website with Cloudflare returns 404 status code for any non root route
I have an S3 hosted website working well behind Cloudflare with the following:
example.com/ works fine
example.com/test also works but the document itself in the network tab is returning 404, ...
- 143
0
votes
1
answer
189
views
Can you limit the operations to log in s3 buckets?
I've been using the s3 logs to track downloads for a podcast (mp3 files) I currently have 23 files I actually care about and I only need to know information about GET operations (download requests ...
- 103
2
votes
1
answer
3k
views
Amazon s3 bucket mount point getting unmounted automatically
I use s3fs on aws ubuntu instance to mount s3 bucket. Following is the command I used to mount the bucket.
sudo fusermount -u /s3/project1
sudo s3fs -o allow_other project1 /s3/project1
The ...
- 141
-1
votes
1
answer
182
views
Data Backup to AWS S3 using Backup Agent
I am having cloudberry backup client, using which I need to backup some data. Consider the data to be backed up is 100 GB on a daily basis. I have 2 different use cases, to understand the number of ...
- 764
3
votes
2
answers
948
views
Migrating 300GB Data from Linux Server to S3 Bucket
I have a linux dedicated server that has 300GB of uploaded files that I need to transfer to AWS Storage S3 since I am now changing the uploads to be saved in S3 bucket instead of local disk. I read ...
- 1,451
3
votes
1
answer
11k
views
AWS S3 upload speeds very slow
I have a 20 Mbps symmetrical pipe. Upload speed to AWS S3 tops out at 2.3 Mbps. Tried the multipart upload but even with 10 concurrent threads, the total speed remains the same, just gets split ...
- 430
1
vote
0
answers
249
views
How do I redirect https://example.com to https://www.example.com?
I recently transferred my website to AWS and added an SSL certificate. The transfer was successful, however, I can't figure out how to redirect https://example.com to https://www.example.com
Only ...
- 11
1
vote
1
answer
800
views
How to use AWS CloudFront for serving both Images and Website?
I have an e-commerce website, an ASP.NET website running on Windows/IIS. Currently I have all my product photos on the local server's File System but since the number of photos are growing I have ...
- 528
1
vote
1
answer
2k
views
Performance consideration for loading Images from S3 bucket
I am working on e-commerce website where users can upload images for the product that they want to sell on the website. The website is running on a windows/IIS server.
At the moment I am using ...
- 528
1
vote
1
answer
219
views
If I already have files in an S3 bucket in the Glacier storage class, what happens if I apply a bucket policy to move to Glacier after 1 day?
We have a pre-existing S3 bucket that we are looking to add a lifecycle policy to in order to move large files in S3 standard to Glacier. The bucket already contains items with the Glacier storage ...
0
votes
1
answer
99
views
Storing assets like css, js, images automatically to s3
I have an ecommerce web app which constantly gets updated with modified css, js and new images that are being part of visuals like banners images, icons, promotion images etc. At present all of these ...
- 763
0
votes
2
answers
795
views
AWS s3, hide/disable serving the web.config file
Currently we have a site setup in an s3 bucket.
This site makes use of a web.config file.
However we can navigate to the file via the url:
for example, www.mysite.com/web.config will serve the file.
...
- 105
1
vote
0
answers
402
views
Restore AWS RDS Database with memory optimized Filegroup in Amazon S3
Can't restore AWS RDS database that has memory-optimized data.
I have a backup in S3 and when I try to do a restore into RDS:
exec msdb.dbo.rds_restore_database
@restore_db_name='MyDatabase',
...
2
votes
1
answer
3k
views
Copy large amounts of data from S3 Bucket to Bucket, different regions
Alright, so I'm setting up an off-site backup of my S3 data and have set up Cross-Region Replication to a new bucket in another region. However I have a lot of data in the original bucket which needs ...
- 467