All Questions
62
questions
0
votes
0
answers
65
views
Enable MFA S3 delete
I have THALES TOTP hardware tokens (MFA) which I would like to use as and additional protection against the accidental S3 object deletion (https://docs.aws.amazon.com/AmazonS3/latest/userguide/...
- 1
0
votes
1
answer
137
views
Does AWS CLI bring data across local network when doing `aws s3 mv` between two S3 locations?
When moving data between two S3 locations using the AWS CLI, for example:
aws s3 mv s3://bucket/prefix s3://bucket/other-prefix
Is the data downloaded to your local computer and then re-uploaded? Or ...
- 131
0
votes
1
answer
4k
views
Why does an S3 to S3 copy care about VPCs? Related to error: "VPC endpoints do not support cross-region requests."
Goal: Get files
from Bucket 1 in ca-central-1 in Account A
to Bucket 2 in us-east-1 in Account B
using the AWS CLI from a third machine using an the IAM role with correct S3 read and write ...
- 103
1
vote
0
answers
60
views
Reduce Query Syntax for ends_with()
I have the following query:
--query "Contents[?ends_with(Key, 'jpg') || ends_with(Key, 'NEF') || ends_with(Key, 'nef') || ends_with(Key, 'CR2') || ends_with(Key, 'cr2')]"
Is there a way to ...
- 111
0
votes
0
answers
728
views
S3 Fatal error: Failed to connect to proxy URL: "Proxy url"
Good Afternoon,
I have been trying to remove some files from S3, last week I did not have any issues accessing my s3 bucket however now any s3 cmd fails with the error message above, I cant even run ...
- 9
1
vote
1
answer
10k
views
How do I extract a list of all folders of my AWS S3 storage bucket or directories?
Does Amazon provide an easy way extract a list of all folders that have files greater than 500 MB from a s3 bucket? want to limit the scope to the '/files/ftp_upload/' directories also This is so I ...
- 59
0
votes
1
answer
3k
views
How to download millions of s3 files and compress them on the fly?
I have an S3 bucket with millions of files, and I want to download all of them.
Since I don't have enough storage, I would like to download them, compress them on the fly and only then save them. How ...
- 1
0
votes
1
answer
150
views
S3 Logs event Issue
Is there a way to see what actions the 'g2' IAM user is performing in S3, and which IP(s) they are running from? I have already enabled the logging of S3 actions.
One point I’m still not able to ...
- 81
0
votes
1
answer
530
views
how to download archive in aws s3 glacier
first, sorry for my bad english.
i create a vault in glacier service and then i uploaded an archive into vault and in out-put, aws gave back me a archive id. then using the archive-id, i created a job ...
- 89
3
votes
2
answers
9k
views
How to *actually* exclude a directory in AWS S3 sync?
The aws s3 sync command has an --exclude flag which lets you exclude a folder from the sync. However, even though the files are not uploaded from that directory, the command still looks at and ...
- 171
0
votes
1
answer
296
views
AWS Glacier and Ransomware
I'm trying to understand the structure of how AWS Glacier works because I have a problem.
Problem:
I have a NAS that backs up to Glacier about once a week. About two weeks ago the NAS got infected ...
0
votes
0
answers
192
views
Copy old data from local Centos to AWS S3 bucket
I have the following task to complete. I hope someone can provide the feedback to get me started.
We have a directory with thousands of images in many folders and sub-folders
The goal is to have a ...
0
votes
1
answer
92
views
VM ram requirement for aws s3 sync job
I have a VM setup to sync 15TB of data split in 25GB volumes to S3 using the latest awscli v.2. I have a 100Mbps upload capacity, and the job will probably take 16 days. I haven't had any experience ...
- 215
1
vote
0
answers
560
views
How to access a S3 bucket accessible only through a EKS Pod
We've got a S3 bucket we use to store files used by an application deployed on EKS, to access the bucket we have a secret web identity token.
The application maintenance guys sometime need to browse ...
- 111
1
vote
1
answer
358
views
Amazon SES data via AWS S3: Is there a simple way to list and download a folder and get line counts?
I've set up Amazon SES to send a company announcement to a list of about 1,000 contacts. I set up the Kenesis Firehose to log all e-mail events (e.g., Send, Bounce, Click) to a bucket in S3. SES seems ...
- 113
0
votes
0
answers
30
views
Did AWS stop allowing PUTs to S3 locations more than one directory deep through the AWS CLI?
On our various servers that have a product that includes a database, I create a job that runs a small script that makes a backup of the database, compresses the file, and sends it to S3. I have one ...
- 1
1
vote
1
answer
1k
views
What is the format of S3 lifecycle configuration?
My Problem
An S3 lifecycle configuration changes object storage classes. A typical usage is periodically moving objects to cheaper storage classes after a specified retention period.
I am trying to ...
- 13.4k
0
votes
1
answer
4k
views
aws s3 ls --summarize switch is broken when trying to get the size of a specific prefix in a bucket
Am I stupid or is this command just totally broken or unclear in usage
I want to see the size of a specific prefix in a bucket
aws s3 ls --summarize --human-readable --recursive 's3://mybucket/...
- 4,233
23
votes
2
answers
42k
views
Deleting S3 files with a given prefix only
We have a bucket with more than 500,000 objects in it.
I'm assigned a job where I've to delete files which have a specific prefix. There are around 300,000 files with the given prefix in the bucket.
...
- 471
2
votes
1
answer
4k
views
S3 PUT directly to glacier
This AWS post from 2018 makes it sound like it should be possible to make S3 style PUT requests to create archives in glacier now:
I'm interpreting that to mean that you could use the s3 CLI or sdk ...
- 123
1
vote
1
answer
5k
views
Using aws cli sync include and exclude with delete option [closed]
aws s3 sync /WordProcessing/DOCUMENTS s3://mybigbucket --delete --include "*" --exclude ".DS_Store"
If somehow .DS_Store made its way to the AWS S3 mybigbucket (from a previous backup), using the ...
- 965
-1
votes
3
answers
1k
views
aws life cycle rules to transition files from s3 bucket to Glacier storage class
I am creating life cycle rules to move files from bucket to the glacier storage class.
I was wondering if there is an option to copy files from S3 bucket to Glacier using either CLI or console?
I ...
- 109
6
votes
1
answer
7k
views
Does AWS cli do a data integrity check on sync with s3? [closed]
I've been looking into using AWS cli for data integrity checks to verify a backup has been transferred from a Linux file server correctly to AWS s3. Likewise, I would like to verify when restoring a ...
- 965
0
votes
1
answer
1k
views
Is there an additional cost to write metadata to objects in the bucket? [closed]
I was looking into writing an MD5sum to the object after it has been uploaded to AWS S3 and doing a data integrity check as described here:
https://aws.amazon.com/premiumsupport/knowledge-center/data-...
- 965
0
votes
0
answers
3k
views
aws s3 cp 'Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')
I am actually writing a simple script to zip a local game save then upload the zip file into my s3 bucket.
Somehow it gives me this error though.
The most I can find googling around is about filesize ...
- 341
2
votes
2
answers
15k
views
How can I 'aws s3 sync' two buckets, which are located in different accounts
I'm trying to use 'aws s3 sync' on the awscli between two accounts.
Account A, I own. Account B, Owned by a third party.
Account B has given a user:jon on account A permission to a bucket through a ...
- 21
2
votes
1
answer
120
views
AWS S3 synced files transformed into hex files
I used AWS cli to sync my S3 bucket to my computer before deleting the bucket. I did not notice it right after the sync but I now realized that most of my files have been turned into hex files (?). ...
- 123
1
vote
2
answers
2k
views
aws s3 sync - skip "system" folders on windows
I'm attempting to use aws s3 sync to push a bunch of backup files from my Windows server to an s3 bucket. The problem that I'm running into is that it appears to be picking up the Recycle Bin and ...
- 3,024
0
votes
1
answer
1k
views
AWS S3 sync command stalls and rung for a long time even when nothing new to sync
I have a cronjob that executes every minute that uses awscli s3 sync command for syncing my website with a S3 bucket.
It seems the command sometimes run for a very long time for no apparent reason : ...
- 131
3
votes
1
answer
2k
views
AWS S3: How to change an HTTP Header of a single object in a S3 bucket via CLI?
As the question implies I want to change/set some HTTP headers for a single object in a S3 bucket via CLI. This is what I have tried so far:
aws s3 sync --delete --acl public-read --cache-control \
...
- 6,259
2
votes
3
answers
3k
views
How to automatically archive AWS S3 whole buckets to S3 Glacier?
I'm required to archive around 200 AWS S3 buckets to S3 Glacier and I would like to do it automatically but I can't find how it can be done with aws-cli.
The only method I found, is through AWS UI... ...
- 10.7k
2
votes
1
answer
1k
views
How to check which IAM performed an object modification of an s3 bucket
Is there a way to see which IAM did a modification on a aws s3 bucket object (with versioning enabled)?
I know that there is the aws s3api list-object-versions, but it only shows the owner's ...
- 245
0
votes
1
answer
3k
views
aws s3 cp files from the root of a subdirectory without getting subfolders as well? Powershell
s3://bucketname/yyyymmdd/foo/bar I have some files in the root directory and I want to copy them. I do not want to copy any subdirectories to the root (foo). I'm trying this in powershell:
aws s3 cp "...
- 1
3
votes
2
answers
6k
views
`Access Denied` for some files, when syncing buckets
I'm using awscli to sync 2 buckets (same account):
aws s3 sync --only-show-errors s3://bucket-1 s3://bucket-2
but for some files I get permission errors:
copy failed: s3://bucket-1/dirname/file....
- 131
0
votes
1
answer
2k
views
Download files from S3 without listing
I have a s3 bucket with more than a million files and about a thousand files added each day by various applications into various folders.
I would like to download and keep all the files locally as ...
- 421
1
vote
1
answer
2k
views
How to copy more than 300GB files one AWS EC2 server to another EC2
I need to copy more than 300 GB files (most them are images) one EC2 instance to another EC2 instance, what will be the best method for doing this?
EBS Snapshot will be not work in my situation ...
- 13
0
votes
1
answer
417
views
How to identify which account a given S3 bucket lives in?
I come across many S3 buckets... is there a way to tell which AWS account a given S3 bucket was created in? I can only think of iterating through each account I have access to, but I don't have keys ...
- 1,549
0
votes
2
answers
2k
views
Moving 1TB / 1 million files across buckets in same region cheaply
Currently we need to migrate a bucket of about 1 TB but it contains a lot of files split into many layers of subdirectories. If I understand the pricing correctly moving data between buckets in same ...
- 3
3
votes
1
answer
2k
views
AWS CLI put-lifecycle-configuration between buckets
I have a bucket that I manually created an object lifecycle policy for; now I'd like to apply that policy to other buckets. I run the following command to get the existing policy:
aws s3api get-...
- 31
1
vote
0
answers
434
views
How to download an S3 object using awless?
Given that the credentials are present in ~/.aws/credentials and region is set in ~/.aws/config, how can I download an S3 object via S3 URI using awless tool?
- 111
1
vote
2
answers
4k
views
Cloud-init is failing to execute awscli command in User Data
Need help to fix the error and enable the instance to run issue free User Data script.
Issue: cloud-init is unable to execute awscli command. I have mentioned a custom script (which will be present ...
- 23
1
vote
1
answer
295
views
CTRL+C on 200k files aws s3 sync
I'm syncing a directory containing 200k files (140GB) to a bucket using the following command
aws s3 sync target s3:/awesomebucket/key/prefix/ --region ap-southeast-2
I took along time. The network ...
7
votes
1
answer
16k
views
Access Denied when syncing between s3 buckets on different AWS accounts
I'm trying to setup sync between two buckets on different AWS accounts.
I got cp working with:
@ubuntu:~$ s3cmd cp -v s3://src/dir/ s3://dest/folder --recursive
I am the user who owns /src/dir and ...
- 182
0
votes
0
answers
350
views
How to not conflict between policies on AWS?
I'm trying to make my AWS environment safer. So, I just created a bucket, no modifications at all, on S3, and want to give a minimal set of permission to my application users.
So, I have created an ...
- 190
1
vote
2
answers
636
views
How to take EBS Volume snapshot and store to S3 Reduced Redundancy?
By default, EBS Volume snapshots on AWS are stored to S3 Standard storage. I would like to figure out how to direct those snapshots to Reduced Redundancy storage instead. I've been trudging through ...
- 75
0
votes
1
answer
2k
views
How to suppress file output from aws s3 ls?
I am using the aws cli to summarize the number of files and the total size of an s3 bucket using the following command (documentation):
aws s3 ls s3://mybucket --recursive --human-readable --...
- 155
1
vote
0
answers
33
views
Enabling S3 bucket as website using cli
I figured out how to create an S3 bucket using aws command line. Is it possible to enable the s3 bucket I created as website using aws cli? The reason I'm doing this is because we're automating ...
- 181
2
votes
1
answer
1k
views
AWS CLI copy life-cycle configuration between buckets
I am trying to read-modify-write an existing life-cycle configuration. So the simpler thing I am staring out with is copying a life-cycle configuration between buckets. From reading through the ...
- 203
1
vote
1
answer
164
views
Is there any way to transfer S3 data between AWS locations and not be charged?
We need to transfer our São Paulo buckets to US East so we can make use of glacier.
Those buckets have more than 4Tb each. We moved the first using AWS API using an instance at US East to send those ...
- 178
5
votes
1
answer
17k
views
Why can my IAM user create a bucket but not upload to it?
UPDATE: everything works fine the next day!?! So I think the answer might be that you have to wait some period of time, either after creating a new IAM user, or after creating a new bucket, before ...
- 730