Listing Files via the API with Pagination

Overview

This article covers how to list files and directories using the StorageLink API filesystem endpoint, including how to paginate through large directory listings.

This documentation is accurate for v1.1.1 and later versions. The core pagination functionality (cursor, size, nextCursor) has been consistent since v1.1.0.

View the API documentation

Here is a link to the StorageLink API documentation file (it will open in your browser): https://thorntech-public-documents.s3.amazonaws.com/storagelink/KB/storagelink-api-documentation-swagger.json

This file is in OpenAPI JSON format, which is designed to be viewed using Swagger. Go to http://editor.swagger.io/, an online Swagger viewer.

Paste in the JSON API. When prompted to convert your JSON to YAML, click OK. You will then see the API documentation for StorageLink.

info

When running these commands from your workstation/Postman, make sure your IP address is whitelisted for the port 443 rule in your Security Group/Firewall since that's the port we're sending the requests to.

Generating the OAuth token

In order to make the necessary API calls, we need to obtain an OAuth token, which can be generated via Postman or cURL.

To generate the OAuth token, you will need 4 values:

security.client-id
security.client-secret
Web User Username
Web User Password

To get the security client id & secret, they are located on the server at /opt/swiftgw/application.properties and to show these values you can run the command:

cat /opt/swiftgw/application.properties

info

Despite the name, the client-id & secret aren't sensitive information and can be found in an unauthenticated web browser visit (just go to view source, and it's there in clear text in the Javascript). The reason why we have the client-id and secret is because it's part of the OAuth spec. So, feel free to ask for these values from the admin managing the server if they haven't already been provided.

Using Postman

In Postman, use the POST method and set the endpoint as this value (Use your own IP/Hostname):

https://your-storagelink-server/backend/login

Under the Authorization tab, select Basic Auth and put your client id as the Username and client secret as the Password.

Next, go to the Body tab and use the x-www-form-urlencoded option. Then, enter in these Keys & Values:

grant_type: The literal string: password
username: The username of the web user
password: The password of the web user

If you click Send, this will then generate the OAuth access_token needed to make requests.

Using cURL

curl --location 'https://your-storagelink-server/backend/login' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--header 'Authorization: Basic <base64-encoded-client-id:client-secret>' \
--data-urlencode 'grant_type=password' \
--data-urlencode 'username=your-username' \
--data-urlencode 'password=your-password'

Listing Files Endpoint

GET /1.0.0/filesystem/{path}

The above endpoint points to the root / directory when {path} is empty, which is the directory you'll first see when signing in as that Web User in the UI.

If you'd like to list a specific path, specify it after the last forward slash, for example:

https://your-storagelink-server/backend/1.0.0/filesystem/Uploads/Bryce

Query Parameters for Pagination

Parameter	Type	Required	Default	Description
cursor	string	No	null	The continuation token from the previous response's nextCursor field. Pass this to retrieve the next page of results.
size	integer	No	250	Preferred number of items per page. See maximum values per cloud provider below.
action	string	No	auto-detected	Set to LIST for directory listing. If omitted, the API auto-detects based on the path type.

Maximum Page Size by Cloud Provider

The size parameter is passed directly to the underlying cloud provider's API. Each provider has its own maximum limit:

Cloud Provider	Maximum size Value	Notes
AWS S3	1000	S3 ListObjectsV2 API limit
Azure Blob Storage	5000	Azure Blob Service API limit
Azure File Share	5000	Azure Files API limit
Google Cloud Storage	1000	GCP Objects.list API limit

If you specify a size larger than the provider's maximum, the provider will return at most its maximum number of items per page. For best compatibility across all providers, use a size of 1000 or less.

Response Structure

{
  "nextCursor": "string or null",
  "content": [
    {
      "name": "filename.txt",
      "size": 1024,
      "type": "REGULAR",
      "lastModified": "2025-01-15T10:30:00Z",
      "contentType": "text/plain",
      "absolutePath": "/test_upload/filename.txt",
      "exists": true,
      "isCurrentFolder": false
    }
  ]
}

Key Behavior Notes

1. Cursor Scope: The cursor is directory-scoped - it is tied to the specific directory being listed. It is actually the underlying cloud provider's continuation token (S3 ContinuationToken, Azure Blob continuation marker, or GCP page token).

2. First Page Behavior: On the first request (no cursor), the response includes an additional item at the beginning of content with "isCurrentFolder": true representing the current directory's metadata.

3. Subsequent Pages: When a cursor is provided, the response does NOT include the current folder metadata, only the files/folders for that page.

4. End of Results: When nextCursor is null, you have reached the end of the directory listing.

5. Ordering: Results are sorted alphabetically with folders appearing before files.

6. No Expiration: The cursor does not expire (it's a cloud provider token), but it becomes invalid if the underlying directory structure changes significantly.

7. Invalid Cursor: Passing an invalid or malformed cursor returns HTTP 400 Bad Request.

Sample Code - Complete Pagination Examples

cURL Example

# First request - get initial page
curl -X GET "https://your-storagelink-server/backend/1.0.0/filesystem/test_upload?action=LIST&size=250" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Content-Type: application/json"

# Response (first page):
# {
#   "nextCursor": "eyJjb250aW51YXRpb25Ub2tlbiI6IjFIa...",
#   "content": [
#     {"name": "test_upload", "isCurrentFolder": true, ...},
#     {"name": "file001.txt", ...},
#     {"name": "file002.txt", ...},
#     ...
#   ]
# }

# Second request - use nextCursor from previous response
curl -X GET "https://your-storagelink-server/backend/1.0.0/filesystem/test_upload?action=LIST&size=250&cursor=eyJjb250aW51YXRpb25Ub2tlbiI6IjFIa..." \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Content-Type: application/json"

# Continue until nextCursor is null

Python Example

import requests

def list_all_files(base_url, access_token, directory_path, page_size=250):
    """
    Iterate through all files in a directory using cursor-based pagination.

    Args:
        base_url: StorageLink server URL (e.g., "https://your-server.com")
        access_token: OAuth2 Bearer token
        directory_path: Directory to list (e.g., "/test_upload" or "test_upload")
        page_size: Number of items per page (default 250)

    Returns:
        List of all files in the directory
    """
    all_files = []
    cursor = None
    page_number = 1

    headers = {
        "Authorization": f"Bearer {access_token}",
        "Content-Type": "application/json"
    }

    while True:
        # Build request parameters
        params = {
            "action": "LIST",
            "size": page_size
        }

        # Add cursor for subsequent pages
        if cursor:
            params["cursor"] = cursor

        # Make the request
        url = f"{base_url}/backend/1.0.0/filesystem/{directory_path.lstrip('/')}"
        response = requests.get(url, headers=headers, params=params)
        response.raise_for_status()

        data = response.json()

        # Process files from this page
        for item in data.get("content", []):
            # Skip the current folder marker (only on first page)
            if item.get("isCurrentFolder"):
                continue
            all_files.append(item)

        print(f"Page {page_number}: Retrieved {len(data.get('content', []))} items, "
              f"Total so far: {len(all_files)}")

        # Check if there are more pages
        cursor = data.get("nextCursor")
        if not cursor:
            print("Reached end of directory listing")
            break

        page_number += 1

    return all_files


# Usage example
if __name__ == "__main__":
    BASE_URL = "https://your-storagelink-server.com"
    ACCESS_TOKEN = "your_oauth_access_token"
    DIRECTORY = "/test_upload"

    files = list_all_files(BASE_URL, ACCESS_TOKEN, DIRECTORY)
    print(f"\nTotal files retrieved: {len(files)}")
    for f in files[:5]:  # Print first 5 as sample
        print(f"  - {f['name']} ({f['size']} bytes)")

JavaScript/Node.js Example

async function listAllFiles(baseUrl, accessToken, directoryPath, pageSize = 250) {
  const allFiles = [];
  let cursor = null;
  let pageNumber = 1;

  while (true) {
    // Build URL with query parameters
    const params = new URLSearchParams({
      action: 'LIST',
      size: pageSize.toString()
    });

    if (cursor) {
      params.append('cursor', cursor);
    }

    const url = `${baseUrl}/backend/1.0.0/filesystem/${directoryPath.replace(/^\//, '')}?${params}`;

    const response = await fetch(url, {
      method: 'GET',
      headers: {
        'Authorization': `Bearer ${accessToken}`,
        'Content-Type': 'application/json'
      }
    });

    if (!response.ok) {
      throw new Error(`HTTP ${response.status}: ${response.statusText}`);
    }

    const data = await response.json();

    // Process files from this page (skip current folder marker)
    const files = (data.content || []).filter(item => !item.isCurrentFolder);
    allFiles.push(...files);

    console.log(`Page ${pageNumber}: Retrieved ${data.content?.length || 0} items, Total: ${allFiles.length}`);

    // Check for next page
    cursor = data.nextCursor;
    if (!cursor) {
      console.log('Reached end of directory listing');
      break;
    }

    pageNumber++;
  }

  return allFiles;
}

// Usage
const files = await listAllFiles(
  'https://your-storagelink-server.com',
  'your_access_token',
  '/test_upload'
);
console.log(`Total files: ${files.length}`);

Sample Request/Response Flow

Request 1 (First Page)

GET /1.0.0/filesystem/test_upload?action=LIST&size=250 HTTP/1.1
Host: your-storagelink-server.com
Authorization: Bearer eyJhbGciOiJSUzI1NiIs...
Content-Type: application/json

Response 1

{
  "nextCursor": "1/RkSIFBN3BoREU4dzVVUkM4QUNpVEFPVnRDOXVCSG...",
  "content": [
    {
      "name": "test_upload",
      "absolutePath": "/test_upload",
      "contentType": "folder",
      "type": "DIRECTORY",
      "isCurrentFolder": true,
      "exists": true
    },
    {
      "name": "document001.pdf",
      "absolutePath": "/test_upload/document001.pdf",
      "size": 45678,
      "contentType": "application/pdf",
      "type": "REGULAR",
      "lastModified": "2025-01-10T14:30:00Z",
      "exists": true
    }
  ]
}

Request 2 (Second Page)

GET /1.0.0/filesystem/test_upload?action=LIST&size=250&cursor=1/RkSIFBN3BoREU4dzVVUkM4QUNpVEFPVnRDOXVCSG... HTTP/1.1
Host: your-storagelink-server.com
Authorization: Bearer eyJhbGciOiJSUzI1NiIs...
Content-Type: application/json

Response 2

{
  "nextCursor": "2/QWxsUGFnZXNDb21wbGV0ZWQhU3RpbGxNb3Jl...",
  "content": [
    {
      "name": "document251.pdf",
      "absolutePath": "/test_upload/document251.pdf",
      "size": 12345,
      "contentType": "application/pdf",
      "type": "REGULAR",
      "lastModified": "2025-01-11T09:15:00Z",
      "exists": true
    }
  ]
}

Final Page (nextCursor is null)

{
  "nextCursor": null,
  "content": [
    {
      "name": "document5001.pdf",
      "absolutePath": "/test_upload/document5001.pdf",
      "size": 9876,
      "contentType": "application/pdf",
      "type": "REGULAR",
      "lastModified": "2025-01-12T16:45:00Z",
      "exists": true
    }
  ]
}

Quick Reference Summary

Question	Answer
Parameter name for cursor	cursor (query parameter)
Cursor scope	Directory-scoped (cloud provider continuation token)
Default page size	250 items
Max page size (S3, GCP)	1000 items
Max page size (Azure)	5000 items
Recommended max for cross-provider	1000 items
Ordering	Alphabetical, folders before files
Cursor expiration	No expiration, but invalid if directory changes
End of results indicator	nextCursor is null

Known Issues by Version

Version	Issue	Description
v1.1.1 - v1.1.3	Azure Blob error handling	Azure Blob Storage may return unhelpful error messages if the Azure SDK throws a RuntimeException during listing. Fixed in v1.1.4 (SWIFT-1053).

note

Note for v1.1.1 users: The pagination functionality works correctly. The Azure Blob issue only affects error scenarios (e.g., network failures, permission issues), not normal pagination operations.