How to recursively list files in AWS S3 bucket using AWS SDK for Python?

I am trying to replicate the AWS CLI ls command to recursively list files in an AWS S3 bucket. For example, I would use the following command to recursively list all of the files in the "location2" bucket.

aws s3 ls s3://location2 --recursive

What is the AWS SDK for Python (i.e. boto3) equivalent of aws s3 ls s3://location2 --recursive?


ANSWERS:


There is no need to use the --recursive option while using the AWS SDK as it lists all the objects in the bucket using the list_objects method.

import boto3 
client = boto3.client('s3')
client.list_objects(Bucket='MyBucket')

Using the higher level API and use resources is the way to go.

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('location2')
bucket_files = [x.key for x in bucket.objects.all()]

You can also use minio-py client library, its open source & compatible with AWS S3.

list_objects.py example below, you can refer to the docs for additional information.

from minio import Minio

client = Minio('s3.amazonaws.com',
               access_key='YOUR-ACCESSKEYID',
               secret_key='YOUR-SECRETACCESSKEY')

# List all object paths in bucket that begin with my-prefixname.
objects = client.list_objects('my-bucketname', prefix='my-prefixname',
                              recursive=True)
for obj in objects:
    print(obj.bucket_name, obj.object_name.encode('utf-8'), obj.last_modified,
          obj.etag, obj.size, obj.content_type)

Hope it helps.

Disclaimer: I work for Minio



 MORE:


 ? How do you increase the token expiration time for a boto3.s3.transfer download?
 ? boto - delete fails because of schema mismatch
 ? Using boto3 to update a cloudformation template
 ? Boto3 : Can we use actual data buffer as parameter instaed of file name to upload file in s3?
 ? create a snapshot for EBS without a reboot using python boto module
 ? is it possible to get s3 ObjectSummary of specific key?
 ? boto3 rds modify_db_snapshot_attribute not doing anything
 ? How do I find who created a CloudFormation stack?
 ? Boto3: How to list of Subnets which are not associated with any Route tables.?
 ? when to generate pre-signed urls