Paging Through Files in an S3 Bucket with .NET
Want to learn more about AWS Lambda and .NET? Check out my A Cloud Guru course on ASP.NET Web API and Lambda.
If you have used S3 for a while, you will know that the number of files in a bucket can get quite large. If you want to get a list of these files, you can make a request with s3Client.ListObjectsV2Async(request), but that will return only the first 1000, and if there are more you won’t see them.
Also, if you would like to break the list of files into pages, that command by itself won’t do it for you.
Fortunately, there are two ways to page through the list of files in an S3 bucket.
The below application shows those ways, the first using a ContinuationToken, the second using Paginators.
Both approaches can be combined with the MaxKeys property on the ListObjectsV2Request allowing you to specify how many results you want to get at a time.
I’ve included some comments within the code.
1using Amazon.S3;
2using Amazon.S3.Model;
3
4public static class Program
5{
6 public static async Task Main()
7 {
8 await PageWithContinuationAsync();
9 await PageWithPaginatorAsync();
10 }
11
12 private static async Task PageWithContinuationAsync()
13 {
14 var s3Client = new AmazonS3Client(); // create the S3 client
15
16 ListObjectsV2Request request = new ListObjectsV2Request
17 {
18 BucketName = "your-bucket-name",
19 MaxKeys = 50 // the number of files you want per page
20 };
21
22 ListObjectsV2Response response;
23 int pageNumber = 1;
24 do
25 {
26 response = await s3Client.ListObjectsV2Async(request); // make the request
27
28 Console.WriteLine($"page {pageNumber++}, results {response.KeyCount}");
29
30 foreach (var item in response.S3Objects)
31 {
32 Console.WriteLine(item.Key);
33 }
34
35 request.ContinuationToken = response.NextContinuationToken; // set the continuation token for the next request
36 } while (response.IsTruncated);
37 }
38
39
40 private static async Task PageWithPaginatorAsync()
41 {
42 var s3Client = new AmazonS3Client(); // create the S3 client
43
44 int pageNumber = 1;
45 var request = new ListObjectsV2Request
46 {
47 BucketName = "your-bucket-name",
48 MaxKeys = 50 // the number of files you want per page
49 };
50
51 var paginatorResponse = s3Client.Paginators.ListObjectsV2(request); // make the request, but using the paginator this time
52
53 await foreach (var page in paginatorResponse.Responses) // get each "page" from the paginatorResponse
54 {
55 Console.WriteLine($"page {pageNumber++}, results {page.KeyCount}");
56 foreach (var item in page.S3Objects)
57 {
58 Console.WriteLine(item.Key);
59 }
60 }
61
62 Console.WriteLine("done");
63 }
64}If you want to create thousands of files in an S3 bucket, use the below in a bash shell, then upload them via the AWS Console.
for i in 000{1..9} 00{10..99} 0{100..999} {1000..2000}; do touch $i.txt; done