Thorn Tech Marketing Ad
Skip to main content
Version: 1.3.1

Listing Files via the API with Pagination

TLDR

Purpose: List a StorageLink folder through the REST API and page through large listings with a cursor

Endpoint: GET https://<host>/backend/1.0.0/filesystem/<path>?action=LIST&size=250 with a Bearer token from POST /backend/login

Pagination:

  • Pass the previous response's nextCursor as the cursor query parameter to get the next page
  • nextCursor is null on the last page
  • Only the first page starts with the folder itself ("isCurrentFolder": true), so skip that item

Page Size: size defaults to 250. AWS S3 returns at most 1000 per page, so use 1000 or less across providers. Local File Storage folders aren't paged: the whole folder comes back at once

Azure File Share: In StorageLink 1.3.1, a listing with more items than size returns the first size items and nextCursor: null. Set size higher than the folder's item count (up to 5000)

Note: A cursor belongs to the directory it came from. An invalid cursor returns HTTP 400 on S3, Azure Blob and Google Cloud Storage

Product: StorageLink by Thorn Technologies — cloud storage gateway for secure file sharing

Overview​

This article covers how to list files and directories using the StorageLink API filesystem endpoint, including how to paginate through large directory listings.

This documentation is accurate for v1.1.1 and later versions. The core pagination functionality (cursor, size, nextCursor) has been consistent since v1.1.0.

View the API documentation​

The StorageLink API reference describes every endpoint of StorageLink 1.3.1 in OpenAPI 3.1 format:

https://thorntech-public-documents.s3.amazonaws.com/storagelink/KB/storagelink-api-documentation-swagger.json

To browse it, download the file, go to https://editor.swagger.io, and click File > Import File. See Uploading Files via the API for a screenshot. Endpoint paths are relative to https://<your-host>/backend.

info

When running these commands from your workstation/Postman, make sure your IP address is whitelisted for the port 443 rule in your Security Group/Firewall since that's the port we're sending the requests to.

Generating the OAuth token​

In order to make the necessary API calls, we need to obtain an OAuth token, which can be generated via Postman or cURL.

To generate the OAuth token, you need the user's username and password.

You can also send the application's client ID and secret, security.client-id and security.client-secret, with HTTP Basic Auth. This is optional: StorageLink issues the token without them, but rejects the request with 401 invalid_client if you send values that don't match. An admin can show them with this command on the server:

sudo grep -E '^security.client-(id|secret)=' /opt/swiftgw/application.properties
info

Despite the name, the client-id & secret aren't sensitive information and can be found in an unauthenticated web browser visit: the StorageLink web page loads them in clear text from https://<your-host>/webconfig.js. The reason why we have the client-id and secret is because it's part of the OAuth spec. So, feel free to ask for these values from the admin managing the server if they haven't already been provided.

Using Postman​

In Postman, use the POST method and set the endpoint as this value (Use your own IP/Hostname):

https://your-storagelink-server/backend/login

Optionally, under the Authorization tab, select Basic Auth and put your client id as the Username and client secret as the Password.

Next, go to the Body tab and use the x-www-form-urlencoded option. Then, enter in these Keys & Values:

grant_type: The literal string: password
username: The username of the web user
password: The password of the web user

If you click Send, this will then generate the OAuth access_token needed to make requests.

Using cURL​

curl --location 'https://your-storagelink-server/backend/login' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=password' \
--data-urlencode 'username=your-username' \
--data-urlencode 'password=your-password'

To send the client ID and secret as well, add --user '<client-id>:<client-secret>'.

Listing Files Endpoint​

GET /1.0.0/filesystem/{path}

The above endpoint points to the root / directory when {path} is empty, which is the directory you'll first see when signing in as that Web User in the UI.

If you'd like to list a specific path, specify it after the last forward slash, for example:

https://your-storagelink-server/backend/1.0.0/filesystem/Uploads/Bryce

Query Parameters for Pagination​

ParameterTypeRequiredDefaultDescription
cursorstringNonullThe continuation token from the previous response's nextCursor field. Pass this to retrieve the next page of results.
sizeintegerNo250Preferred number of items per page. See maximum values per cloud provider below.
actionstringNoauto-detectedSet to LIST for directory listing. If omitted, the API auto-detects based on the path type.

Maximum Page Size by Storage Type​

The size parameter is passed to the underlying storage, and each storage type has its own limit. Tested on StorageLink 1.3.1 with a folder of 1,102 items:

StorageMaximum size ValueNotes
AWS S31000S3 ListObjectsV2 API limit; size=2000 returns 1000 items and a nextCursor
Azure Blob Storage (including Data Lake / HNS)5000Azure Blob Service API limit; size=2000 returned all 1,102 items
Azure File Share5000Azure Files API limit. See the note below
Google Cloud StorageNo fixed limit observedsize=2000 returned all 1,102 items in one page
Local File StorageNot pagedThe whole folder comes back in one response; size and cursor are ignored

If you specify a size larger than the provider's maximum, the provider will return at most its maximum number of items per page. For best compatibility across all providers, use a size of 1000 or less.

Azure File Share folders with more items than size

In StorageLink 1.3.1, listing an Azure File Share folder that holds more items than size returns the first size items with nextCursor: null, as if the listing were complete. A client that follows nextCursor misses the rest of the folder. Until this is fixed, list Azure File Share folders with a size larger than the number of items in the folder, up to 5000.

Response Structure​

{
"nextCursor": "string or null",
"content": [
{
"name": "filename.txt",
"size": 1024,
"type": "REGULAR",
"lastModified": "2025-01-15T10:30:00Z",
"contentType": "text/plain",
"absolutePath": "/test_upload/filename.txt",
"exists": true,
"isCurrentFolder": false
}
]
}

Key Behavior Notes​

  1. Cursor Scope: The cursor is directory-scoped - it is tied to the specific directory being listed. It is actually the underlying cloud provider's continuation token (S3 ContinuationToken, Azure Blob continuation marker, or GCP page token).

  2. First Page Behavior: On the first request (no cursor), the response includes an additional item at the beginning of content with "isCurrentFolder": true representing the current directory's metadata.

  3. Subsequent Pages: When a cursor is provided, the response does NOT include the current folder metadata, only the files/folders for that page.

  4. End of Results: When nextCursor is null, you have reached the end of the directory listing.

  5. Ordering: Within each page, folders come before files. Across pages, items follow the storage provider's own order, which is by name on S3, Azure Blob and Google Cloud Storage, so a folder appears on the page where its name falls rather than on the first page. Local File Storage returns all folders first, then all files, sorted by name.

  6. No Expiration: The cursor does not expire (it's a cloud provider token), but it becomes invalid if the underlying directory structure changes significantly.

  7. Invalid Cursor: Passing an invalid or malformed cursor returns HTTP 400 Bad Request on S3, Azure Blob and Google Cloud Storage. Azure File Share and Local File Storage ignore the cursor and return the first page.

Sample Code - Complete Pagination Examples​

cURL Example​

# First request - get initial page
curl -X GET "https://your-storagelink-server/backend/1.0.0/filesystem/test_upload?action=LIST&size=250" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json"

# Response (first page):
# {
# "nextCursor": "eyJjb250aW51YXRpb25Ub2tlbiI6IjFIa...",
# "content": [
# {"name": "test_upload", "isCurrentFolder": true, ...},
# {"name": "file001.txt", ...},
# {"name": "file002.txt", ...},
# ...
# ]
# }

# Second request - use nextCursor from previous response
curl -X GET "https://your-storagelink-server/backend/1.0.0/filesystem/test_upload?action=LIST&size=250&cursor=eyJjb250aW51YXRpb25Ub2tlbiI6IjFIa..." \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json"

# Continue until nextCursor is null

Python Example​

import requests

def list_all_files(base_url, access_token, directory_path, page_size=250):
"""
Iterate through all files in a directory using cursor-based pagination.

Args:
base_url: StorageLink server URL (e.g., "https://your-server.com")
access_token: OAuth2 Bearer token
directory_path: Directory to list (e.g., "/test_upload" or "test_upload")
page_size: Number of items per page (default 250)

Returns:
List of all files in the directory
"""
all_files = []
cursor = None
page_number = 1

headers = {
"Authorization": f"Bearer {access_token}",
"Content-Type": "application/json"
}

while True:
# Build request parameters
params = {
"action": "LIST",
"size": page_size
}

# Add cursor for subsequent pages
if cursor:
params["cursor"] = cursor

# Make the request
url = f"{base_url}/backend/1.0.0/filesystem/{directory_path.lstrip('/')}"
response = requests.get(url, headers=headers, params=params)
response.raise_for_status()

data = response.json()

# Process files from this page
for item in data.get("content", []):
# Skip the current folder marker (only on first page)
if item.get("isCurrentFolder"):
continue
all_files.append(item)

print(f"Page {page_number}: Retrieved {len(data.get('content', []))} items, "
f"Total so far: {len(all_files)}")

# Check if there are more pages
cursor = data.get("nextCursor")
if not cursor:
print("Reached end of directory listing")
break

page_number += 1

return all_files


# Usage example
if __name__ == "__main__":
BASE_URL = "https://your-storagelink-server.com"
ACCESS_TOKEN = "your_oauth_access_token"
DIRECTORY = "/test_upload"

files = list_all_files(BASE_URL, ACCESS_TOKEN, DIRECTORY)
print(f"\nTotal files retrieved: {len(files)}")
for f in files[:5]: # Print first 5 as sample
print(f" - {f['name']} ({f['size']} bytes)")

JavaScript/Node.js Example​

This example uses fetch and a top-level await, so it needs Node.js 18 or later, saved as an ES module, for example list_all_files.mjs.

async function listAllFiles(baseUrl, accessToken, directoryPath, pageSize = 250) {
const allFiles = [];
let cursor = null;
let pageNumber = 1;

while (true) {
// Build URL with query parameters
const params = new URLSearchParams({
action: 'LIST',
size: pageSize.toString()
});

if (cursor) {
params.append('cursor', cursor);
}

const url = `${baseUrl}/backend/1.0.0/filesystem/${directoryPath.replace(/^\//, '')}?${params}`;

const response = await fetch(url, {
method: 'GET',
headers: {
'Authorization': `Bearer ${accessToken}`,
'Content-Type': 'application/json'
}
});

if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}

const data = await response.json();

// Process files from this page (skip current folder marker)
const files = (data.content || []).filter(item => !item.isCurrentFolder);
allFiles.push(...files);

console.log(`Page ${pageNumber}: Retrieved ${data.content?.length || 0} items, Total: ${allFiles.length}`);

// Check for next page
cursor = data.nextCursor;
if (!cursor) {
console.log('Reached end of directory listing');
break;
}

pageNumber++;
}

return allFiles;
}

// Usage
const files = await listAllFiles(
'https://your-storagelink-server.com',
'your_access_token',
'/test_upload'
);
console.log(`Total files: ${files.length}`);

Sample Request/Response Flow​

Request 1 (First Page)​

GET /1.0.0/filesystem/test_upload?action=LIST&size=250 HTTP/1.1
Host: your-storagelink-server.com
Authorization: Bearer eyJhbGciOiJSUzI1NiIs...
Content-Type: application/json

Response 1​

{
"nextCursor": "1/RkSIFBN3BoREU4dzVVUkM4QUNpVEFPVnRDOXVCSG...",
"content": [
{
"name": "test_upload",
"absolutePath": "/test_upload",
"contentType": "folder",
"type": "DIRECTORY",
"isCurrentFolder": true,
"exists": true
},
{
"name": "document001.pdf",
"absolutePath": "/test_upload/document001.pdf",
"size": 45678,
"contentType": "application/pdf",
"type": "REGULAR",
"lastModified": "2025-01-10T14:30:00Z",
"exists": true
}
]
}

Request 2 (Second Page)​

GET /1.0.0/filesystem/test_upload?action=LIST&size=250&cursor=1/RkSIFBN3BoREU4dzVVUkM4QUNpVEFPVnRDOXVCSG... HTTP/1.1
Host: your-storagelink-server.com
Authorization: Bearer eyJhbGciOiJSUzI1NiIs...
Content-Type: application/json

Response 2​

{
"nextCursor": "2/QWxsUGFnZXNDb21wbGV0ZWQhU3RpbGxNb3Jl...",
"content": [
{
"name": "document251.pdf",
"absolutePath": "/test_upload/document251.pdf",
"size": 12345,
"contentType": "application/pdf",
"type": "REGULAR",
"lastModified": "2025-01-11T09:15:00Z",
"exists": true
}
]
}

Final Page (nextCursor is null)​

{
"nextCursor": null,
"content": [
{
"name": "document5001.pdf",
"absolutePath": "/test_upload/document5001.pdf",
"size": 9876,
"contentType": "application/pdf",
"type": "REGULAR",
"lastModified": "2025-01-12T16:45:00Z",
"exists": true
}
]
}

Quick Reference Summary​

QuestionAnswer
Parameter name for cursorcursor (query parameter)
Cursor scopeDirectory-scoped (cloud provider continuation token)
Default page size250 items
Max page size (S3)1000 items
Max page size (Azure)5000 items
Max page size (Google Cloud Storage)No fixed limit observed
Local File StorageNot paged
Recommended max for cross-provider1000 items
OrderingFolders before files within each page; provider order across pages
Cursor expirationNo expiration, but invalid if directory changes
End of results indicatornextCursor is null

Known Issues by Version​

VersionIssueDescription
v1.1.1 - v1.1.3Azure Blob error handlingAzure Blob Storage may return unhelpful error messages if the Azure SDK throws a RuntimeException during listing. Fixed in v1.1.4 (SWIFT-1053).
v1.3.1Azure File Share paginationA listing with more items than size returns the first size items and nextCursor: null. Use a size larger than the folder's item count (up to 5000). Earlier versions not checked.
note

Note for v1.1.1 users: The pagination functionality works correctly. The Azure Blob issue only affects error scenarios (e.g., network failures, permission issues), not normal pagination operations.