Version 2 Rewrite

.gitattributes

@@ -1,2 +0,0 @@
-# Auto detect text files and perform LF normalization
-* text=auto

.gitignore

@@ -1,5 +1,32 @@
-.vscode
+# Node modules
-node_modules
+node_modules/
-output
+
-package-lock.json
+# Project outputs
+output/
+coverage/
+dist/
+
+# Credentials and config (do NOT commit secrets)
+credentials.json
+credentials-*.json
 config.json
+config-*.json
+.env
+.env.*
+
+# Logs
+logs/
+*.log
+npm-debug.log
+yarn-error.log
+
+# Editor directories
+.vscode/
+.idea/
+
+# OS files
+.DS_Store
+Thumbs.db
+
+# Misc
+.cache/

GUIDE_API_KEY.md

@@ -0,0 +1,60 @@
+# API Key Setup Guide
+
+This guide explains how to create and configure API keys for use with the Google Sheets Rate Assistant. The application expects an `apiKeys` array in `config.json` (even a single key should be provided as a single-element array).
+
+## Overview
+
+- The app supports using API keys instead of a service account by providing an `apiKeys` array in `config.json`.
+- Each entry in `apiKeys` may be a string (the key) or an object with `{ key, rateLimitPerMinute }`.
+
+## Create an API key
+
+1. Go to the Google Cloud Console: https://console.cloud.google.com/
+2. Select or create a project.
+3. Enable the **Google Sheets API** (and **Drive API** if you need it) under "APIs & Services > Library".
+4. Go to "APIs & Services > Credentials" and click **Create credentials > API key**. Copy the key value.
+
+## Secure the key
+
+- Click **Restrict key** on the credentials page and set:
+  - Application restrictions (IP addresses or HTTP referrers) where possible.
+  - API restrictions: select **Google Sheets API** only.
+- Do not leave keys unrestricted in production.
+
+## Example `config.json` snippet
+
+To configure one key:
+
+```json
+{
+  "apiKeys": [
+    "YOUR_API_KEY_1"
+  ]
+}
+```
+
+To configure multiple keys with per-key rate limits:
+
+```json
+{
+  "apiKeys": [
+    { "key": "API_KEY_1", "rateLimitPerMinute": 60 },
+    { "key": "API_KEY_2", "rateLimitPerMinute": 30 }
+  ]
+}
+```
+
+The app will round-robin requests across configured keys and respect `rateLimitPerMinute` values.
+
+## Storing keys safely
+
+- Do not commit `config.json` with keys to version control. Add `config.json` to `.gitignore`.
+- Alternatively, keep a `config.json.example` in the repo and place the real keys in a local `config.json`.
+- If you prefer to use environment variables, you can have a small loader script that reads an env var and writes `config.json` before running the app.
+
+## Troubleshooting
+
+- If you get quota or 403 errors, check API restrictions, project quotas, and whether the key is restricted to the wrong IP/referrer.
+- Use multiple keys to spread requests if you hit per-key quotas.
+
+*** End of guide ***

GUIDE_SERVICE_ACCOUNT.md

@@ -0,0 +1,61 @@
+# Google Service Account Setup for Google Sheets Rate Assistant
+
+This guide explains how to create a Google service account, enable the Sheets API, and configure the project to use service account credentials.
+
+## 1. Enable the Google Sheets API
+
+1. Open the Google Cloud Console: https://console.cloud.google.com/
+2. Select an existing project or create a new one.
+3. In the left menu, open **APIs & Services > Library**.
+4. Search for **Google Sheets API** and click it.
+5. Click **Enable**.
+
+## 2. Create a service account
+
+1. In the Cloud Console, open **IAM & Admin > Service Accounts**.
+2. Click **Create Service Account**.
+3. Enter a name and description.
+4. Click **Create and continue**.
+5. Skip granting optional roles or add a minimal role if required.
+6. Click **Done**.
+
+## 3. Create and download credentials
+
+1. Find the service account you created in the list.
+2. Click the service account name.
+3. Open the **Keys** tab.
+4. Click **Add Key > Create new key**.
+5. Choose **JSON** and click **Create**.
+6. Save the downloaded JSON file securely in your project folder, for example `credentials.json`.
+
+## 4. Share your Google Sheets with the service account
+
+1. Open the Google Sheet you want to export.
+2. Click **Share**.
+3. Add the service account email address, which looks like `...@...iam.gserviceaccount.com`.
+4. Give it **Viewer** access.
+
+## 5. Configure the project
+
+In `config.json`, set the `credentialsPath` field to the JSON file path, and remove any `apiKey` or `apiKeys` entries if you are using service account auth.
+
+Example:
+
+```json
+{
+  "credentialsPath": "credentials.json",
+  "rateLimitPerMinute": 50,
+  "documents": [
+    {
+      "documentId": "YOUR_SPREADSHEET_ID",
+      "outputDir": "output/your-document",
+      "sheets": [
+        {
+          "name": "Sheet1",
+          "outputFilename": "sheet1.csv"
+        }
+      ]
+    }
+  ]
+}
+```

config-example.json

@@ -1,37 +0,0 @@
-{
-    "outputFolder": "output",
-    "apis": [
-        {
-            "apiKey": "--- API key ---",
-            "documents": [
-                {
-                    "googleDocId": "--- Google Sheet ID ---",
-                    "sheets": [
-                        "-- Sheet Name ---",
-                        "-- 2nd Sheet Name ---"
-                    ],
-                    "pollRate": 1500
-                },
-                {
-                    "googleDocId": "--- 2nd Google Sheet ID ---",
-                    "sheets": [
-                        "-- Sheet Name ---"
-                    ],
-                    "pollRate": 10000
-                }
-            ]
-        },
-        {
-            "apiKey": "--- 2nd API Key ---",
-            "documents": [
-                {
-                    "googleDocId": "--- 3rd Google Sheet ID ---",
-                    "sheets": [
-                        "-- Sheet Name ---"
-                    ],
-                    "pollRate": 5000
-                }
-            ]
-        }
-    ]
-}

config.example.json

@@ -0,0 +1,30 @@
+{
+  "apiKeys": [
+    {
+      "key": "YOUR_API_KEY_1",
+      "rateLimitPerMinute": 50
+    },
+    {
+      "key": "YOUR_API_KEY_2",
+      "rateLimitPerMinute": 50
+    }
+  ],
+  "serviceAccounts": [
+    {
+      "path": "credentials.json",
+      "rateLimitPerMinute": 50
+    }
+  ],
+  "documents": [
+    {
+      "documentId": "YOUR_SPREADSHEET_ID",
+      "outputDir": "output/your-document",
+      "sheets": [
+        {
+          "name": "Sheet1",
+          "outputFilename": "sheet1.csv"
+        }
+      ]
+    }
+  ]
+}

index.js

@@ -1,84 +1,358 @@
-const { GoogleSpreadsheet } = require('google-spreadsheet');
+// Google Sheets Rate Assistant
+// Orchestrates periodic fetches of Google Sheets and writes CSV outputs.
+// Uses `./lib/config`, `./lib/rate`, and `./lib/auth` for configuration,
+// rate-limiting, and authentication/fetch helpers respectively.
+// === Imports & constants ===
 const fs = require('fs');
-const readline = require('readline').createInterface({
+const path = require('path');
-    input: process.stdin,
+const {google} = require('googleapis');
-    output: process.stdout,
+const stringifyModule = require('csv-stringify/sync');
-});
+const stringify = typeof stringifyModule === 'function'
-const config = require('./config.json');
+  ? stringifyModule
+  : stringifyModule.default || stringifyModule.stringify || stringifyModule;
+const {loadConfig, resolveOutputPath, validateConfig} = require('./lib/config');
+const {buildThrottle, applyRateLimitBuffer, getTotalRateLimitPerMinute} = require('./lib/rate');
+const authHelpers = require('./lib/auth');
+const {createApiKeyBatchFetcher, createServiceAccountBatchFetcher, getFirstApiKey, getFirstServiceAccountAuthClient, loadAuth} = authHelpers;
 
-const check = new Promise(function (resolve) {
+// === Status rendering / console display ===
-    var count = 0,
+// Renders per-sheet status lines when running in a TTY, updating in-place.
-        position = {};
-    config.apis.forEach(function (document, d) {
-        var totalRate = 0;
-        document.documents.forEach(function (item, i) {
-            item.sheets.forEach(function (sh, s) {
-                count += 1;
-                position[d + "" + i + "" + s] = count;
-            });
-            totalRate += (60 / (item.pollRate / 1000));
-            //config.apis[d].documents[i].sheets.itemNo = count;
-        });
-        config.position = position;
-        var stdin = process.openStdin();
-        if (totalRate > 50) {
-            console.log('\n---------- RATE LIMIT WARNING ----------');
-            readline.question(`\nYour poll rate will be ${totalRate} per minute, this is above the recommendation of 50 per minute.\nIf the GoogleAPI limit is reached (60 per min on free) you will receive no updates until a break period has passed.\n(API: ${document.apiKey})\n\nAre you sure you want to continue? [y/n]: `, answer => {
-                if (answer == 'y') {
-                    readline.close();
-                    resolve();
-                } else {
-                    process.exit();
-                }
-            });
-        } else if (config.apis.length - 1 == d) {
-            resolve();
-        }
-    });
-})
 
-check.then(function () {
+function mapDocumentToFilename(documentConfig, sheetName) {
-    console.clear();
+  const safeName = sheetName.replace(/[\\/:*?"<>|]/g, '_');
-    process.stdout.cursorTo(0, 1);
+  return `${documentConfig.documentId}-${safeName}.csv`;
-    console.log(`Ctrl+C to kill the application.`);
+}
-    config.apis.forEach(function (api, d) {
+
-        api.documents.forEach(function (docs, i) {
+const statusManager = {
-            const doc = new GoogleSpreadsheet(docs.googleDocId);
+  keys: [],
-            doc.useApiKey(api.apiKey);
+  statuses: new Map(),
-
+  lastRenderLines: 0,
-            setInterval(function () {
+  enabled: process.stdout && process.stdout.isTTY,
-                (async function () {
+};
-                    await doc.loadInfo();
+
-                    var titleClean = doc.title.replace(/[^a-zA-Z0-9 ]/g, '');
+// === Document grouping & mapping ===
-                    if (!fs.existsSync(`${config.outputFolder}/${titleClean}/`)) {
+// Map `config.documents` entries to internal groups with resolved output paths.
-                        fs.mkdirSync(`${config.outputFolder}/${titleClean}/`, { recursive: true });
+
-                    }
+function renderStatusBlock() {
-                    docs.sheets.forEach(function (sheet, s) {
+  if (!statusManager.enabled || statusManager.keys.length === 0) {
-                        (async function () {
+    return;
-                            downloadCSV = await doc.sheetsByTitle[sheet].downloadAsCSV();
+  }
-                            fs.writeFile(`${config.outputFolder}/${titleClean}/${sheet.replace(/[^a-zA-Z0-9 ]/g)}.csv`, downloadCSV, function (err) {
+
-                                if (err) {
+  if (statusManager.lastRenderLines > 0) {
-                                    return console.log(err);
+    process.stdout.write(`\x1B[${statusManager.lastRenderLines}F`);
-                                }
+  }
-                                const date = new Date();
+
-                                process.stdout.cursorTo(0, config.position[d + "" + i + "" + s] + 2);
+  for (const key of statusManager.keys) {
-                                process.stdout.clearLine();
+    const statusText = statusManager.statuses.get(key) || `${key} - pending`;
-                                console.log(`"${doc.title} - ${sheet}" last updated at ${pad(date.getHours(), 2)}:${pad(date.getMinutes(), 2)}:${pad(date.getSeconds(), 2)}`);
+    process.stdout.clearLine(0);
-                                process.stdout.cursorTo(31, 1);
+    process.stdout.cursorTo(0);
+    process.stdout.write(statusText + '\n');
+  }
+
+  statusManager.lastRenderLines = statusManager.keys.length;
+}
+
+function setTaskStatus(taskId, statusText) {
+  statusManager.statuses.set(taskId, statusText);
+  renderStatusBlock();
+}
+
+function initializeTaskStatuses(tasks) {
+  statusManager.keys = tasks.map(task => `${task.documentId}:${task.sheetName}`);
+  statusManager.statuses.clear();
+  statusManager.lastRenderLines = 0;
+
+  for (const key of statusManager.keys) {
+    statusManager.statuses.set(key, `${key} - pending`);
+  }
+
+  renderStatusBlock();
+}
+
+// Authentication helpers: see `./lib/auth` for credential loaders and
+// per-credential, throttled batch fetchers (API key and service account modes).
+
+async function fetchSpreadsheetTitle(context, documentId) {
+  const sheetOptions = {version: 'v4'};
+  if (context.authClient) {
+    sheetOptions.auth = context.authClient;
+  }
+
+  const sheets = google.sheets(sheetOptions);
+  const request = {
+    spreadsheetId: documentId,
+    fields: 'properties/title',
+  };
+
+  if (context.apiKey) {
+    request.key = context.apiKey;
+  }
+
+  const response = await sheets.spreadsheets.get(request);
+  return response.data.properties?.title || documentId;
+}
+
+function buildDocumentGroups(config) {
+  const groups = [];
+
+  if (!Array.isArray(config.documents)) {
+    throw new Error('`documents` must be an array in config.json');
+  }
+
+  for (const documentConfig of config.documents) {
+    if (!documentConfig.documentId || !Array.isArray(documentConfig.sheets)) {
+      throw new Error('Each document must include documentId and sheets array');
+    }
+
+    const documentOutputDir = documentConfig.outputDir || './output';
+    const sheets = [];
+
+    for (const sheetConfig of documentConfig.sheets) {
+      const name = typeof sheetConfig === 'string' ? sheetConfig : sheetConfig.name;
+      const outputFilename = sheetConfig.outputFilename || mapDocumentToFilename(documentConfig, name);
+      const outputPath = resolveOutputPath(documentOutputDir, outputFilename);
+
+      sheets.push({
+        sheetName: name,
+        outputPath,
+      });
+    }
+
+    groups.push({
+      documentId: documentConfig.documentId,
+      sheets,
+    });
+  }
+
+  return groups;
+}
+
+// === Batch fetching (fallback) ===
+// `index.js` provides a simple `batchFetchDocumentSheets` fallback used when
+// no per-credential batch fetcher is configured by `./lib/auth`.
+async function batchFetchDocumentSheets(context, documentId, sheetNames) {
+  const sheetOptions = {version: 'v4'};
+  if (context.authClient) {
+    sheetOptions.auth = context.authClient;
+  }
+
+  const sheets = google.sheets(sheetOptions);
+  const request = {
+    spreadsheetId: documentId,
+    ranges: sheetNames,
+  };
+
+  if (context.apiKey) {
+    request.key = context.apiKey;
+  }
+
+  const response = await sheets.spreadsheets.values.batchGet(request);
+  const valueRanges = response.data.valueRanges || [];
+  const rowsBySheet = {};
+
+  for (let i = 0; i < sheetNames.length; i += 1) {
+    const sheetName = sheetNames[i];
+    rowsBySheet[sheetName] = valueRanges[i]?.values || [];
+  }
+
+  return rowsBySheet;
+}
+
+function formatStatusTimestamp(date = new Date()) {
+  const pad = value => String(value).padStart(2, '0');
+  const year = date.getFullYear();
+  const month = pad(date.getMonth() + 1);
+  const day = pad(date.getDate());
+  const hours = pad(date.getHours());
+  const minutes = pad(date.getMinutes());
+  const seconds = pad(date.getSeconds());
+  return `${year}-${month}-${day} ${hours}:${minutes}:${seconds}`;
+}
+
+// === Execution / saving ===
+// Functions that execute per-document fetches and write CSV outputs.
+
+function writeCsv(outputPath, rows) {
+  const csv = stringify(rows, {
+    header: false,
+    quoted: true,
   });
-                        }());
+  fs.writeFileSync(outputPath, csv, 'utf8');
+}
+
+async function executeDocumentGroup(group, fetcher) {
+  const sheetNames = group.sheets.map(sheet => sheet.sheetName);
+
+  for (const sheet of group.sheets) {
+    const displayName = `${group.documentDisplayName || group.documentId}:${sheet.sheetName}`;
+    setTaskStatus(`${group.documentId}:${sheet.sheetName}`, `${displayName} - pulling...`);
+  }
+
+  const rowsBySheet = await fetcher(group.documentId, sheetNames);
+
+  for (const sheet of group.sheets) {
+    const rows = rowsBySheet[sheet.sheetName] || [];
+    writeCsv(sheet.outputPath, rows);
+    const relativePath = path.relative(process.cwd(), sheet.outputPath) || sheet.outputPath;
+    const timestamp = formatStatusTimestamp();
+    const displayName = `${group.documentDisplayName || group.documentId}:${sheet.sheetName}`;
+    setTaskStatus(
+      `${group.documentId}:${sheet.sheetName}`,
+      `${displayName} - saved to ${relativePath} (updated ${timestamp})`
+    );
+  }
+}
+
+function formatGoogleError(err) {
+  if (err && err.response && err.response.data) {
+    return `${err.message} (${JSON.stringify(err.response.data)})`;
+  }
+  return err.message || String(err);
+}
+
+function interpretGoogleSheetsError(err, documentId, sheetName) {
+  const message = formatGoogleError(err);
+  const notFound = err && err.response && err.response.status === 404;
+  const requestedEntity = message.includes('Requested entity was not found');
+
+  if (notFound || requestedEntity) {
+    let detail = `Spreadsheet not found or inaccessible: ${documentId}`;
+    if (sheetName) {
+      detail += `, sheet: ${sheetName}`;
+    }
+    detail += '. Check that the spreadsheet ID is correct, that the sheet exists, and that the document is shared with the service account or is publicly readable when using an API key.';
+    return detail;
+  }
+
+  return message;
+}
+
+function sleep(milliseconds) {
+  return new Promise(resolve => setTimeout(resolve, milliseconds));
+}
+
+// === Error formatting / interpretation ===
+
+// === Main scheduler ===
+// Orchestrates startup, selects the appropriate fetcher (API keys,
+// service accounts, or single credentials), and runs the periodic loop.
+async function scheduleRuns() {
+  const config = loadConfig();
+  try {
+    validateConfig(config);
+  } catch (e) {
+    console.error('Configuration error:', e.message);
+    process.exit(1);
+  }
+  const apiBatchFetcher = createApiKeyBatchFetcher(config);
+  const svcBatchFetcher = createServiceAccountBatchFetcher(config);
+  let fetcher;
+
+  if (apiBatchFetcher) {
+    fetcher = apiBatchFetcher;
+  } else if (svcBatchFetcher) {
+    fetcher = svcBatchFetcher;
+  } else {
+    const auth = await loadAuth(config);
+    const baseRate = config.rateLimitPerMinute || 50;
+    const bufferedRate = applyRateLimitBuffer(baseRate);
+    const throttle = buildThrottle(bufferedRate);
+    fetcher = throttle((documentId, sheetNames) => batchFetchDocumentSheets({authClient: auth}, documentId, sheetNames));
+  }
+
+  const groups = buildDocumentGroups(config);
+  if (groups.length === 0) {
+    throw new Error('No sheets configured to pull in config.json');
+  }
+
+  const firstApiKey = getFirstApiKey(config);
+  const titleMap = {};
+
+  // If running in single-auth mode, try to warm the token
+  if (!apiBatchFetcher && !svcBatchFetcher) {
+    try {
+      const singleAuth = await loadAuth(config);
+      if (singleAuth) {
+        try {
+          const at = await singleAuth.getAccessToken();
+          console.log('Service account token acquired (len=', (at && at.token) ? at.token.length : 'no', ')');
+        } catch (e) {
+          console.error('Failed to acquire initial access token from service account:', e && e.message);
+        }
+      }
+    } catch (_) {
+      // loadAuth will throw if missing credentials; continue and let title fetch handle errors
+    }
+  }
+
+  // build a context for fetching titles: prefer API key, then first service account, then single-auth if present
+  let titleContext = null;
+  if (firstApiKey) {
+    titleContext = {apiKey: firstApiKey};
+  } else if (svcBatchFetcher) {
+    const firstSvcAuth = getFirstServiceAccountAuthClient(config);
+    if (firstSvcAuth) titleContext = {authClient: firstSvcAuth};
+  } else {
+    try {
+      const singleAuth = await loadAuth(config);
+      if (singleAuth) titleContext = {authClient: singleAuth};
+    } catch (_) {
+      // ignore
+    }
+  }
+
+  for (const group of groups) {
+    try {
+      titleMap[group.documentId] = await fetchSpreadsheetTitle(titleContext || {}, group.documentId);
+    } catch (err) {
+      titleMap[group.documentId] = group.documentId;
+      console.error(`Failed to load title for ${group.documentId}: ${formatGoogleError(err)}`);
+    }
+  }
+
+  const tasks = groups.flatMap(group =>
+    group.sheets.map(sheet => ({
+      documentId: group.documentId,
+      sheetName: sheet.sheetName,
+    }))
+  );
+
+  tasks.forEach(task => {
+    task.documentDisplayName = titleMap[task.documentId] || task.documentId;
   });
-                }());
+
-            }, docs.pollRate)
+  groups.forEach(group => {
+    group.documentDisplayName = titleMap[group.documentId] || group.documentId;
   });
+
+  const totalRateLimit = getTotalRateLimitPerMinute(config);
+  const intervalMs = Math.max(1, Math.ceil(60000 / totalRateLimit));
+  console.log(`Configured for ${totalRateLimit} requests per minute across ${tasks.length} tasks (${intervalMs}ms interval)`);
+  initializeTaskStatuses(tasks);
+
+  let currentIndex = 0;
+  while (true) {
+    const group = groups[currentIndex];
+    currentIndex = (currentIndex + 1) % groups.length;
+
+    try {
+      await executeDocumentGroup(group, fetcher);
+    } catch (err) {
+      for (const sheet of group.sheets) {
+        setTaskStatus(
+          `${group.documentId}:${sheet.sheetName}`,
+          `${group.documentId}:${sheet.sheetName} - failed: ${interpretGoogleSheetsError(err, group.documentId, sheet.sheetName)}`
+        );
+      }
+    }
+
+    await sleep(intervalMs);
+  }
+}
 
+if (require.main === module) {
+  scheduleRuns().catch(err => {
+    console.error('Application error:', err.message || err);
+    process.exit(1);
   });
-}, function (err) {
-    console.log(err);
-})
-
-function pad(n, width, z) {
-    z = z || '0';
-    n = n + '';
-    return n.length >= width ? n : new Array(width - n.length + 1).join(z) + n;
 }

lib/auth.js

@@ -0,0 +1,199 @@
+const fs = require('fs');
+const path = require('path');
+const {google} = require('googleapis');
+const {applyRateLimitBuffer} = require('./rate');
+const pThrottleModule = require('p-throttle');
+const pThrottle = pThrottleModule.default || pThrottleModule;
+
+function loadApiKeys(config) {
+  const apiKeys = [];
+  const defaultRateLimit = config.rateLimitPerMinute || 50;
+
+  if (Array.isArray(config.apiKeys)) {
+    for (const entry of config.apiKeys) {
+      if (typeof entry === 'string') {
+        apiKeys.push({key: entry, rateLimitPerMinute: defaultRateLimit});
+      } else if (entry && typeof entry === 'object') {
+        if (!entry.key) {
+          throw new Error('Each entry in `apiKeys` must include `key`');
+        }
+        apiKeys.push({
+          key: entry.key,
+          rateLimitPerMinute: entry.rateLimitPerMinute || defaultRateLimit,
+        });
+      } else {
+        throw new Error('`apiKeys` entries must be strings or objects');
+      }
+    }
+  }
+
+  return apiKeys;
+}
+
+function loadServiceAccounts(config) {
+  const accounts = [];
+  const defaultRateLimit = config.rateLimitPerMinute || 50;
+
+  if (!Array.isArray(config.serviceAccounts)) {
+    return accounts;
+  }
+
+  for (const entry of config.serviceAccounts) {
+    if (typeof entry === 'string') {
+      accounts.push({path: entry, rateLimitPerMinute: defaultRateLimit});
+    } else if (entry && typeof entry === 'object') {
+      if (!entry.path && !entry.credentialsPath) {
+        throw new Error('Each entry in `serviceAccounts` must include `path` (or `credentialsPath`)');
+      }
+      const p = entry.path || entry.credentialsPath;
+      accounts.push({path: p, rateLimitPerMinute: entry.rateLimitPerMinute || defaultRateLimit});
+    } else {
+      throw new Error('`serviceAccounts` entries must be strings or objects');
+    }
+  }
+
+  return accounts;
+}
+
+function createServiceAccountBatchFetcher(config) {
+  const accounts = loadServiceAccounts(config);
+  if (accounts.length === 0) return null;
+
+  const fetchers = accounts.map(({path: credentialsPath, rateLimitPerMinute}) => {
+    const resolvedPath = path.isAbsolute(credentialsPath) ? credentialsPath : path.resolve(process.cwd(), credentialsPath);
+    if (!fs.existsSync(resolvedPath)) {
+      throw new Error(`Service account credentials not found: ${resolvedPath}`);
+    }
+
+    let credentials;
+    try {
+      credentials = JSON.parse(fs.readFileSync(resolvedPath, 'utf8'));
+    } catch (e) {
+      throw new Error(`Failed to read/parse service account credentials at ${resolvedPath}: ${e.message}`);
+    }
+
+    let authClient;
+    try {
+      authClient = google.auth.fromJSON(credentials);
+    } catch (e) {
+      throw new Error(`Failed to initialize auth client from credentials at ${resolvedPath}: ${e.message}`);
+    }
+    authClient.scopes = ['https://www.googleapis.com/auth/spreadsheets.readonly'];
+
+    const bufferedRateLimit = applyRateLimitBuffer(rateLimitPerMinute);
+    const interval = Math.ceil(60000 / bufferedRateLimit);
+    const throttled = pThrottle({limit: 1, interval})(async (documentId, sheetNames) => {
+      const sheets = google.sheets({version: 'v4', auth: authClient});
+      const request = {spreadsheetId: documentId, ranges: sheetNames};
+      const response = await sheets.spreadsheets.values.batchGet(request);
+      const valueRanges = response.data.valueRanges || [];
+      const rowsBySheet = {};
+      for (let i = 0; i < sheetNames.length; i += 1) {
+        const sheetName = sheetNames[i];
+        rowsBySheet[sheetName] = valueRanges[i]?.values || [];
+      }
+      return rowsBySheet;
+    });
+
+    return {authClient, fetch: throttled};
+  });
+
+  let currentIndex = 0;
+  return async (documentId, sheetNames) => {
+    const worker = fetchers[currentIndex];
+    currentIndex = (currentIndex + 1) % fetchers.length;
+    return worker.fetch(documentId, sheetNames);
+  };
+}
+
+function createApiKeyBatchFetcher(config) {
+  const apiKeys = loadApiKeys(config);
+  if (apiKeys.length === 0) {
+    return null;
+  }
+
+  const fetchers = apiKeys.map(({key, rateLimitPerMinute}) => {
+    const bufferedRateLimit = applyRateLimitBuffer(rateLimitPerMinute);
+    const interval = Math.ceil(60000 / bufferedRateLimit);
+    const throttled = pThrottle({limit: 1, interval})(async (documentId, sheetNames) => {
+      const sheets = google.sheets({version: 'v4'});
+      const request = {spreadsheetId: documentId, ranges: sheetNames, key};
+      const response = await sheets.spreadsheets.values.batchGet(request);
+      const valueRanges = response.data.valueRanges || [];
+      const rowsBySheet = {};
+      for (let i = 0; i < sheetNames.length; i += 1) {
+        const sheetName = sheetNames[i];
+        rowsBySheet[sheetName] = valueRanges[i]?.values || [];
+      }
+      return rowsBySheet;
+    });
+    return {key, fetch: throttled};
+  });
+
+  let currentIndex = 0;
+  return async (documentId, sheetNames) => {
+    const worker = fetchers[currentIndex];
+    currentIndex = (currentIndex + 1) % fetchers.length;
+    return worker.fetch(documentId, sheetNames);
+  };
+}
+
+function getFirstApiKey(config) {
+  if (Array.isArray(config.apiKeys) && config.apiKeys.length > 0) {
+    const firstEntry = config.apiKeys[0];
+    return typeof firstEntry === 'string' ? firstEntry : firstEntry.key;
+  }
+
+  return null;
+}
+
+function getFirstServiceAccountAuthClient(config) {
+  if (!Array.isArray(config.serviceAccounts) || config.serviceAccounts.length === 0) return null;
+  const firstEntry = config.serviceAccounts[0];
+  const entryObj = typeof firstEntry === 'string' ? {path: firstEntry} : firstEntry;
+  const credentialsPath = entryObj.path || entryObj.credentialsPath;
+  if (!credentialsPath) return null;
+
+  const resolvedPath = path.isAbsolute(credentialsPath) ? credentialsPath : path.resolve(process.cwd(), credentialsPath);
+  if (!fs.existsSync(resolvedPath)) return null;
+
+  try {
+    const raw = fs.readFileSync(resolvedPath, 'utf8');
+    const credentials = JSON.parse(raw);
+    const authClient = google.auth.fromJSON(credentials);
+    authClient.scopes = ['https://www.googleapis.com/auth/spreadsheets.readonly'];
+    return authClient;
+  } catch (e) {
+    console.error(`Failed to load first service account from ${resolvedPath}: ${e.message}`);
+    return null;
+  }
+}
+
+function loadAuth(config) {
+  if (!config || typeof config !== 'object') return null;
+
+  if (config.credentialsPath) {
+    const credentialsPath = config.credentialsPath;
+    const resolvedPath = path.isAbsolute(credentialsPath) ? credentialsPath : path.resolve(process.cwd(), credentialsPath);
+    if (!fs.existsSync(resolvedPath)) {
+      throw new Error(`Credentials file not found: ${resolvedPath}`);
+    }
+    const raw = fs.readFileSync(resolvedPath, 'utf8');
+    const credentials = JSON.parse(raw);
+    const authClient = google.auth.fromJSON(credentials);
+    authClient.scopes = ['https://www.googleapis.com/auth/spreadsheets.readonly'];
+    return authClient;
+  }
+
+  return getFirstServiceAccountAuthClient(config);
+}
+
+module.exports = {
+  loadApiKeys,
+  loadServiceAccounts,
+  createServiceAccountBatchFetcher,
+  createApiKeyBatchFetcher,
+  getFirstApiKey,
+  getFirstServiceAccountAuthClient
+  , loadAuth
+};

lib/config.js

@@ -0,0 +1,48 @@
+const fs = require('fs');
+const path = require('path');
+
+const CONFIG_PATH = path.resolve(__dirname, '..', 'config.json');
+
+function loadConfig() {
+  if (!fs.existsSync(CONFIG_PATH)) {
+    throw new Error(`Missing config file at ${CONFIG_PATH}`);
+  }
+  const json = fs.readFileSync(CONFIG_PATH, 'utf8');
+  return JSON.parse(json);
+}
+
+function resolveOutputPath(outputDir, filename) {
+  if (!path.isAbsolute(outputDir)) {
+    outputDir = path.resolve(process.cwd(), outputDir);
+  }
+  if (!fs.existsSync(outputDir)) {
+    fs.mkdirSync(outputDir, {recursive: true});
+  }
+  return path.join(outputDir, filename);
+}
+
+function validateConfig(config) {
+  if (!config || typeof config !== 'object') {
+    throw new Error('Missing or invalid configuration object');
+  }
+
+  if (!Array.isArray(config.documents) || config.documents.length === 0) {
+    throw new Error('`documents` must be a non-empty array in config.json');
+  }
+
+  const hasApiKeys = Array.isArray(config.apiKeys) && config.apiKeys.length > 0;
+  const hasSvcAccounts = Array.isArray(config.serviceAccounts) && config.serviceAccounts.length > 0;
+  const hasCredPath = !!config.credentialsPath;
+
+  if (!hasApiKeys && !hasSvcAccounts && !hasCredPath) {
+    throw new Error('Configuration must include at least one of: `apiKeys`, `serviceAccounts`, or `credentialsPath`');
+  }
+
+  // Basic validation of documents/sheets structure
+  for (const doc of config.documents) {
+    if (!doc.documentId) throw new Error('Each document must include a `documentId`');
+    if (!Array.isArray(doc.sheets) || doc.sheets.length === 0) throw new Error(`Document ${doc.documentId} must include a non-empty 'sheets' array`);
+  }
+}
+
+module.exports = {loadConfig, resolveOutputPath, validateConfig};

lib/rate.js

@@ -0,0 +1,45 @@
+const pThrottleModule = require('p-throttle');
+const pThrottle = pThrottleModule.default || pThrottleModule;
+
+function buildThrottle(maxRequestsPerMinute) {
+  const limit = Math.max(1, Math.floor(maxRequestsPerMinute));
+  const interval = Math.ceil(60000 / limit);
+  return pThrottle({limit: 1, interval});
+}
+
+function applyRateLimitBuffer(rateLimitPerMinute) {
+  return Math.max(1, Math.floor(rateLimitPerMinute * 0.95));
+}
+
+function getTotalRateLimitPerMinute(config) {
+  const defaultRateLimit = config.rateLimitPerMinute || 50;
+  let totalRateLimit = 0;
+
+  if (Array.isArray(config.apiKeys) && config.apiKeys.length > 0) {
+    for (const entry of config.apiKeys) {
+      if (typeof entry === 'string') {
+        totalRateLimit += applyRateLimitBuffer(defaultRateLimit);
+      } else if (entry && typeof entry === 'object') {
+        totalRateLimit += applyRateLimitBuffer(entry.rateLimitPerMinute || defaultRateLimit);
+      }
+    }
+  }
+
+  if (Array.isArray(config.serviceAccounts) && config.serviceAccounts.length > 0) {
+    for (const entry of config.serviceAccounts) {
+      if (typeof entry === 'string') {
+        totalRateLimit += applyRateLimitBuffer(defaultRateLimit);
+      } else if (entry && typeof entry === 'object') {
+        totalRateLimit += applyRateLimitBuffer(entry.rateLimitPerMinute || defaultRateLimit);
+      }
+    }
+  }
+
+  if (totalRateLimit === 0) {
+    totalRateLimit = applyRateLimitBuffer(defaultRateLimit);
+  }
+
+  return Math.max(1, totalRateLimit);
+}
+
+module.exports = {buildThrottle, applyRateLimitBuffer, getTotalRateLimitPerMinute};

package.json

@@ -1,14 +1,14 @@
 {
   "name": "google-sheets-rate-assistant",
-  "version": "0.0.1",
+  "version": "2.0.0",
+  "description": "Pull Google Sheets data and export to CSV with configurable schedules and rate limiting.",
   "main": "index.js",
   "scripts": {
-    "run": "node index.js"
+    "start": "node index.js"
   },
-  "author": "",
-  "license": "ISC",
-  "description": "Assists in keeping requests under the free rate limit for applications such as vMix.",
   "dependencies": {
-    "google-spreadsheet": "^3.3.0"
+    "csv-stringify": "^6.0.0",
+    "googleapis": "^173.0.0",
+    "p-throttle": "^8.1.0"
   }
 }

readme.md

@@ -1,45 +1,59 @@
-## Google Sheets Rate Assistant
+# Google Sheets Rate Assistant
 
-### Instructions
+This Node.js application pulls configured Google Sheets documents and exports them as CSV files.
 
-First fill out the config.json with the required information.
+## Setup
 
+1. Copy `config.example.json` to `config.json`.
+2. Configure authentication in `config.json` — choose one of:
 
-### Run
+- Service account: place your service account JSON in `credentials.json` and set `credentialsPath`, or add one or more entries to the `serviceAccounts` array (each entry may be a path string or an object with `path`/`credentialsPath` and optional `rateLimitPerMinute`).
-*Node must be installed on your system*
+- API keys: add one or more API keys to the `apiKeys` array in `config.json` (each entry may be a string or an object with `key` and optional `rateLimitPerMinute`).
 
-Download the .zip of the repository and extract into a folder.
+3. Install dependencies:
 
-1. Open Powershell in the directory
+  npm install
-2. run `npm install`
-3. run `node index`
 
-### Config Extract
+4. Run the application:
 
-```
+  node index.js
+
+## Setup Guides
+
+For detailed instructions on authentication methods, please refer to the following guides:
+
+- [Service Account Setup](./GUIDE_SERVICE_ACCOUNT.md)
+- [API Key Setup Guide](./GUIDE_API_KEY.md)
+
+### Sample `config.json`
+
+Here is an example `config.json` matching the repository's `config.example.json`:
+
+```json
 {
-    "apiKey": "--- API key ---",
+  "apiKeys": [
+    { "key": "YOUR_API_KEY_1", "rateLimitPerMinute": 50 },
+    { "key": "YOUR_API_KEY_2", "rateLimitPerMinute": 50 }
+  ],
+  "serviceAccounts": [
+    { "path": "credentials.json", "rateLimitPerMinute": 50 }
+  ],
   "documents": [
     {
-            "googleDocId": "--- Google Sheet ID ---",
+      "documentId": "YOUR_SPREADSHEET_ID",
+      "outputDir": "output/your-document",
       "sheets": [
-                "-- Sheet Name ---",
+        { "name": "Sheet1", "outputFilename": "sheet1.csv" }
-                "-- 2nd Sheet Name ---"
+      ]
-            ],
-            "pollRate": 1500
-        },
-        {
-            "googleDocId": "--- 2nd Google Sheet ID ---",
-            "sheets": [
-                "-- Sheet Name ---"
-            ],
-            "pollRate": 10000
     }
   ]
 }
 ```
 
-+ Lowest PollRate for a single API key is recommended at 1200 to keep these under the free tier of 60 per minute.
+## Rate limiting
-+ Each API key should be different, this is not checked.
+
-+ Multiple documents can be polled from a single key and the rate will be calculated.
+The app enforces a maximum of `rateLimitPerMinute` requests per minute to avoid exceeding API limits.
-+ Multiple sheets can be included under each 'googleDocId' and will not add to each APIKey rate limit.
+
+## Output
+
+CSV files are written to `output` by default. Individual documents may override this with their own `outputDir` in `config.json`.