How to check if a specified key exists in a given S3 bucket using Java

I would like to check if a key exists in a given bucket using Java. I looked at the API but there aren't any methods that are useful. I tried to use getObject but it threw an exception.


ANSWERS:


Use the jets3t library. Its a lot more easier and robust than the AWS sdk. Using this library you can call, s3service.getObjectDetails(). This will check and retrieve only the details of the object (not the contents) of the object. It will throw a 404 if the object is missing. So you can catch that exception and deal with it in your app.

But in order for this to work, you will need to have ListBucket access for the user on that bucket. Just GetObject access will not work. The reason being, Amazon will prevent you from checking for the presence of the key if you dont have ListBucket access. Just knowing whether a key is present or not, will also suffice for malicious users in some cases. Hence unless they have ListBucket access they will not be able to do so.


There's now a doesObjectExist method in the official Java API.

Enjoy!


Use errorCode.equals("NoSuchKey")

try {
    AmazonS3 s3 = new AmazonS3Client(new ClasspathPropertiesFileCredentialsProvider());
    String bucketName = getBucketName();
    s3.createBucket(bucketName);
    S3Object object = s3.getObject(bucketName, getKey());
} catch (AmazonServiceException e) {
    String errorCode = e.getErrorCode();
    if (!errorCode.equals("NoSuchKey")) {
        throw e;
    }
    Logger.getLogger(getClass()).debug("No such key!!!", e);
}

Note about the exception: I know exceptions should not be used for flow control. The problem is that Amazon didn't provide any api to check this flow - just documentation about the exception.


Using the AWS SDK use the getObjectMetadata method. The method will throw an AmazonServiceException if the key doesn't exist.

private AmazonS3 s3;
...
public boolean exists(String path, String name) {
    try {
        s3.getObjectMetadata(bucket, getS3Path(path) + name); 
    } catch(AmazonServiceException e) {
        return false;
    }
    return true;
}

In Amazon Java SDK 1.10+, you can use getStatusCode() to get the status code of the HTTP response, which will be 404 if the object does not exist.

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.AmazonS3Exception;
import org.apache.http.HttpStatus;

try {
    AmazonS3 s3 = new AmazonS3Client();
    S3Object object = s3.getObjectMetadata("my-bucket", "my-client");
} catch (AmazonS3Exception e) {
    if (e.getStatusCode() == HttpStatus.SC_NOT_FOUND) {
        // bucket/key does not exist 
    } else {
        throw e;
    }
}

getObjectMetadata() consumes fewer resources, and the response doesn't need to be closed like getObject().


In previous versions, you can use getErrorCode() and check for the appropriate string (depends on the version).


Use ListObjectsRequest setting Prefix as your key.

.NET code:

 public bool Exists(string key)
    {

        using (Amazon.S3.AmazonS3Client client = (Amazon.S3.AmazonS3Client)Amazon.AWSClientFactory.CreateAmazonS3Client(m_accessKey, m_accessSecret))
        {
            ListObjectsRequest request = new ListObjectsRequest();
            request.BucketName = m_bucketName;
            request.Prefix = key;
            using (ListObjectsResponse response = client.ListObjects(request))
            {

                foreach (S3Object o in response.S3Objects)
                {
                    if( o.Key == key )
                        return true;
                }
                return false;
            }
        }
    }.

This java code checks if the key (file) exists in s3 bucket.

public static boolean isExistS3(String accessKey, String secretKey, String bucketName, String file) {

    // Amazon-s3 credentials
    AWSCredentials myCredentials = new BasicAWSCredentials(accessKey, secretKey); 
    AmazonS3Client s3Client = new AmazonS3Client(myCredentials); 

    ObjectListing objects = s3Client.listObjects(new ListObjectsRequest().withBucketName(bucketName).withPrefix(file));

    for (S3ObjectSummary objectSummary: objects.getObjectSummaries()) {
        if (objectSummary.getKey().equals(file)) {
            return true;
        }
    }
    return false;
}

For PHP (I know the question is Java, but Google brought me here), you can use stream wrappers and file_exists

$bucket = "MyBucket";
$key = "MyKey";
$s3 = Aws\S3\S3Client->factory([...]);
$s3->registerStreamWrapper();
$keyExists = file_exists("s3://$bucket/$key");

Break your path into bucket and object. Testing the bucket using the method doesBucketExist, Testing the object using the size of the listing (0 in case not exist). So this code will do:

String bucket = ...;
String objectInBucket = ...;
AmazonS3 s3 = new AmazonS3Client(...);
return s3.doesBucketExist(bucket) 
       && !s3.listObjects(bucket, objectInBucket).getObjectSummaries().isEmpty();

Using Object isting. Java function to check if specified key exist in AWS S3.

boolean isExist(String key)
    {
        ObjectListing objects = amazonS3.listObjects(new ListObjectsRequest().withBucketName(bucketName).withPrefix(key));

        for (S3ObjectSummary objectSummary : objects.getObjectSummaries())
        {
            if (objectSummary.getKey().equals(key))
            {
                return true;
            }

        }
        return false;
    }

There is an easy way to do it using jetS3t API's isObjectInBucket() method.

Sample code:

ProviderCredentials awsCredentials = new AWSCredentials(
                awsaccessKey,
                awsSecretAcessKey);

        // REST implementation of S3Service
        RestS3Service restService = new RestS3Service(awsCredentials);

        // check whether file exists in bucket
        if (restService.isObjectInBucket(bucket, objectKey)) {

            //your logic

        }

Alternatively you can use Minio-Java client library, its Open Source and compatible with AWS S3 API.

You can use Minio-Java StatObject.java examples for the same.

import io.minio.MinioClient;
import io.minio.errors.MinioException;

import java.io.InputStream;
import java.io.IOException;
import java.security.NoSuchAlgorithmException;
import java.security.InvalidKeyException;

import org.xmlpull.v1.XmlPullParserException;


public class GetObject {
  public static void main(String[] args)
    throws NoSuchAlgorithmException, IOException, InvalidKeyException, XmlPullParserException, MinioException {
    // Note: YOUR-ACCESSKEYID, YOUR-SECRETACCESSKEY and my-bucketname are
    // dummy values, please replace them with original values.
    // Set s3 endpoint, region is calculated automatically
    MinioClient s3Client = new MinioClient("https://s3.amazonaws.com", "YOUR-ACCESSKEYID", "YOUR-SECRETACCESSKEY");
    InputStream stream = s3Client.getObject("my-bucketname", "my-objectname");

    byte[] buf = new byte[16384];
    int bytesRead;
    while ((bytesRead = stream.read(buf, 0, buf.length)) >= 0) {
      System.out.println(new String(buf, 0, bytesRead));
    }

    stream.close();
  }
}

I hope it helps.

Disclaimer : I work for Minio



 MORE:


 ? Amazon S3: How to get a list of folders in the bucket?
 ? How to troubleshooting Amazon S3 GET request signature mismatch?
 ? How to resolve Network Failure net::ERR_NAME_NOT_RESOLVED
 ? How to empty the bucket?
 ? AWS API: How to write code that checks whether a file exists in a specific bucket?
 ? Fatest way to retrieve S3 objects metadata (NodeJS API)
 ? Time to live for SimpleDB or DynamoDB
 ? Time to live for SimpleDB or DynamoDB
 ? Time to live for SimpleDB or DynamoDB
 ? Scaling MongoDB on EC2 or should I just switch to DynamoDB?