Download Files via API

TLDR

Purpose: Download the files in a StorageLink folder with a Python script that calls the REST API

Requirements:

• Python 3 with the requests library
• The client ID and secret: security.client-id and security.client-secret in /opt/swiftgw/application.properties
• A StorageLink username and password

Quick Start:

1. Set BASE_URL, CLIENT_ID, CLIENT_SECRET, USERNAME, PASSWORD and DOWNLOAD_PATH (the local folder to save into)
2. Set DIRECTORY_PATH, the StorageLink folder to download (starting with /), adding it to the variables if it is missing
3. Run python download_files.py

How It Works: Gets an OAuth token, lists the folder and downloads each regular file (not subfolders), adding a counter to the name if the local file already exists

Product: StorageLink by Thorn Technologies — cloud storage gateway for secure file sharing

How to Use the Python Script to Download Files

This article guides you through the process of using a Python script to download files from an API. The script requires specific credentials that you can obtain from your server's configuration files.

Obtaining CLIENT_ID and CLIENT_SECRET

To authenticate and interact with the API, you will need a CLIENT_ID and CLIENT_SECRET. These can be obtained by accessing your server's configuration files: SSH into your server where the API is hosted. Once logged in, open the configuration file by running:

cat /opt/swiftgw/application.properties

Locate the security.client-id and security.client-secret entries within the file. Their values are the CLIENT_ID and CLIENT_SECRET your script uses to authenticate API requests.

Using the Provided Script

Below is the Python script you will use to download files from the API. It is pre-configured with placeholder values for the variables you will replace with actual credentials and paths:

import requests
import base64
import urllib3
import os
import re
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


# Variables to be configured
BASE_URL = "https://3.238.156.250/backend"
CLIENT_ID = "Your_CLIENT_ID_here"
CLIENT_SECRET = "Your_CLIENT_SECRET_here"
USERNAME = "Your_Username_here"
PASSWORD = "Your_Password_here"
DOWNLOAD_PATH = '/Users/Name/Documents/downloadscripttest'

def get_oauth_token():
    url = f"{BASE_URL}/login"
    credentials = f"{CLIENT_ID}:{CLIENT_SECRET}"
    auth_header = base64.b64encode(credentials.encode()).decode()
    headers = {
        "Content-Type": "application/x-www-form-urlencoded",
        "Authorization": f"Basic {auth_header}"
    }
    data = {
        "grant_type": "password",
        "username": USERNAME,
        "password": PASSWORD
    }
    response = requests.post(url, headers=headers, data=data, verify=False)
    if response.status_code == 200:
        return response.json().get("access_token")
    else:
        print(f"Error getting token: {response.text}")
        return None
    

def download_files(token):
    """Download all files from a specific directory in StorageLink."""
    url = f"{BASE_URL}/1.0.0/filesystem{DIRECTORY_PATH}"
    headers = {
        "Authorization": f"Bearer {token}"
    }
    response = requests.get(url, headers=headers, verify=False)
    if response.status_code == 200:
        files = response.json().get('content', [])
        for file in files:
            if file['type'] == 'REGULAR':  # Ensure to download only regular files
                file_url = f"{BASE_URL}/1.0.0/filesystem{file['absolutePath']}"
                original_name = file['name']
                base_name, extension = os.path.splitext(original_name)
                file_name = original_name
                file_path = os.path.join(DOWNLOAD_PATH, file_name)

                # Handle files with existing counters
                match = re.match(r"(.*) \((\d+)\)$", base_name)
                if match:
                    base_name = match.group(1)
                    start_counter = int(match.group(2))
                else:
                    start_counter = 1

                # Increment filename counter if file exists
                while os.path.exists(file_path):
                    new_file_name = f"{base_name} ({start_counter}){extension}"
                    file_path = os.path.join(DOWNLOAD_PATH, new_file_name)
                    start_counter += 1

                file_response = requests.get(file_url, headers=headers, verify=False)
                if file_response.status_code == 200:
                    with open(file_path, 'wb') as f:
                        f.write(file_response.content)
                    print(f"Downloaded {file_path}")
                else:
                    print(f"Error downloading {file_name}: {file_response.text}")
    else:
        print(f"Error downloading files: {response.text}")

if __name__ == "__main__":
    token = get_oauth_token()
    if token:
        download_files(token)

Replace the placeholder values in the script with the credentials you obtained and the specific paths relevant to your API's environment.

Executing the Script

To execute the script:

Save the script to a local file with a .py extension, for example, download_files.py. Open a command line interface (CLI), navigate to the directory where your script is saved.

Run the script by typing:

python download_files.py

This will execute the script, and it should output the authentication status followed by confirmation of each file downloaded from the specified directory, or an error message if unsuccessful.

Understanding and Configuring Script Variables

The script contains several variables that you need to configure according to your environment. Here's what each variable means and how you can adjust them:

• BASE_URL: The URL of the API backend. Replace this with the URL provided to you by your API administrator.
• CLIENT_ID and CLIENT_SECRET: These are credentials provided to you by the API provider. They are used to authenticate your script's access to the API.
• USERNAME and PASSWORD: Your API user credentials. These may differ depending on your role or the permissions assigned to you.
• DOWNLOAD_PATH: The path to the download whose files you want to download. Change this path based on where the files are stored in your API's file system.