Version 2 Rewrite
This commit is contained in:
@@ -1,2 +0,0 @@
|
||||
# Auto detect text files and perform LF normalization
|
||||
* text=auto
|
||||
+31
-4
@@ -1,5 +1,32 @@
|
||||
.vscode
|
||||
node_modules
|
||||
output
|
||||
package-lock.json
|
||||
# Node modules
|
||||
node_modules/
|
||||
|
||||
# Project outputs
|
||||
output/
|
||||
coverage/
|
||||
dist/
|
||||
|
||||
# Credentials and config (do NOT commit secrets)
|
||||
credentials.json
|
||||
credentials-*.json
|
||||
config.json
|
||||
config-*.json
|
||||
.env
|
||||
.env.*
|
||||
|
||||
# Logs
|
||||
logs/
|
||||
*.log
|
||||
npm-debug.log
|
||||
yarn-error.log
|
||||
|
||||
# Editor directories
|
||||
.vscode/
|
||||
.idea/
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Misc
|
||||
.cache/
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# API Key Setup Guide
|
||||
|
||||
This guide explains how to create and configure API keys for use with the Google Sheets Rate Assistant. The application expects an `apiKeys` array in `config.json` (even a single key should be provided as a single-element array).
|
||||
|
||||
## Overview
|
||||
|
||||
- The app supports using API keys instead of a service account by providing an `apiKeys` array in `config.json`.
|
||||
- Each entry in `apiKeys` may be a string (the key) or an object with `{ key, rateLimitPerMinute }`.
|
||||
|
||||
## Create an API key
|
||||
|
||||
1. Go to the Google Cloud Console: https://console.cloud.google.com/
|
||||
2. Select or create a project.
|
||||
3. Enable the **Google Sheets API** (and **Drive API** if you need it) under "APIs & Services > Library".
|
||||
4. Go to "APIs & Services > Credentials" and click **Create credentials > API key**. Copy the key value.
|
||||
|
||||
## Secure the key
|
||||
|
||||
- Click **Restrict key** on the credentials page and set:
|
||||
- Application restrictions (IP addresses or HTTP referrers) where possible.
|
||||
- API restrictions: select **Google Sheets API** only.
|
||||
- Do not leave keys unrestricted in production.
|
||||
|
||||
## Example `config.json` snippet
|
||||
|
||||
To configure one key:
|
||||
|
||||
```json
|
||||
{
|
||||
"apiKeys": [
|
||||
"YOUR_API_KEY_1"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
To configure multiple keys with per-key rate limits:
|
||||
|
||||
```json
|
||||
{
|
||||
"apiKeys": [
|
||||
{ "key": "API_KEY_1", "rateLimitPerMinute": 60 },
|
||||
{ "key": "API_KEY_2", "rateLimitPerMinute": 30 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The app will round-robin requests across configured keys and respect `rateLimitPerMinute` values.
|
||||
|
||||
## Storing keys safely
|
||||
|
||||
- Do not commit `config.json` with keys to version control. Add `config.json` to `.gitignore`.
|
||||
- Alternatively, keep a `config.json.example` in the repo and place the real keys in a local `config.json`.
|
||||
- If you prefer to use environment variables, you can have a small loader script that reads an env var and writes `config.json` before running the app.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- If you get quota or 403 errors, check API restrictions, project quotas, and whether the key is restricted to the wrong IP/referrer.
|
||||
- Use multiple keys to spread requests if you hit per-key quotas.
|
||||
|
||||
*** End of guide ***
|
||||
@@ -0,0 +1,61 @@
|
||||
# Google Service Account Setup for Google Sheets Rate Assistant
|
||||
|
||||
This guide explains how to create a Google service account, enable the Sheets API, and configure the project to use service account credentials.
|
||||
|
||||
## 1. Enable the Google Sheets API
|
||||
|
||||
1. Open the Google Cloud Console: https://console.cloud.google.com/
|
||||
2. Select an existing project or create a new one.
|
||||
3. In the left menu, open **APIs & Services > Library**.
|
||||
4. Search for **Google Sheets API** and click it.
|
||||
5. Click **Enable**.
|
||||
|
||||
## 2. Create a service account
|
||||
|
||||
1. In the Cloud Console, open **IAM & Admin > Service Accounts**.
|
||||
2. Click **Create Service Account**.
|
||||
3. Enter a name and description.
|
||||
4. Click **Create and continue**.
|
||||
5. Skip granting optional roles or add a minimal role if required.
|
||||
6. Click **Done**.
|
||||
|
||||
## 3. Create and download credentials
|
||||
|
||||
1. Find the service account you created in the list.
|
||||
2. Click the service account name.
|
||||
3. Open the **Keys** tab.
|
||||
4. Click **Add Key > Create new key**.
|
||||
5. Choose **JSON** and click **Create**.
|
||||
6. Save the downloaded JSON file securely in your project folder, for example `credentials.json`.
|
||||
|
||||
## 4. Share your Google Sheets with the service account
|
||||
|
||||
1. Open the Google Sheet you want to export.
|
||||
2. Click **Share**.
|
||||
3. Add the service account email address, which looks like `...@...iam.gserviceaccount.com`.
|
||||
4. Give it **Viewer** access.
|
||||
|
||||
## 5. Configure the project
|
||||
|
||||
In `config.json`, set the `credentialsPath` field to the JSON file path, and remove any `apiKey` or `apiKeys` entries if you are using service account auth.
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"credentialsPath": "credentials.json",
|
||||
"rateLimitPerMinute": 50,
|
||||
"documents": [
|
||||
{
|
||||
"documentId": "YOUR_SPREADSHEET_ID",
|
||||
"outputDir": "output/your-document",
|
||||
"sheets": [
|
||||
{
|
||||
"name": "Sheet1",
|
||||
"outputFilename": "sheet1.csv"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
@@ -1,37 +0,0 @@
|
||||
{
|
||||
"outputFolder": "output",
|
||||
"apis": [
|
||||
{
|
||||
"apiKey": "--- API key ---",
|
||||
"documents": [
|
||||
{
|
||||
"googleDocId": "--- Google Sheet ID ---",
|
||||
"sheets": [
|
||||
"-- Sheet Name ---",
|
||||
"-- 2nd Sheet Name ---"
|
||||
],
|
||||
"pollRate": 1500
|
||||
},
|
||||
{
|
||||
"googleDocId": "--- 2nd Google Sheet ID ---",
|
||||
"sheets": [
|
||||
"-- Sheet Name ---"
|
||||
],
|
||||
"pollRate": 10000
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"apiKey": "--- 2nd API Key ---",
|
||||
"documents": [
|
||||
{
|
||||
"googleDocId": "--- 3rd Google Sheet ID ---",
|
||||
"sheets": [
|
||||
"-- Sheet Name ---"
|
||||
],
|
||||
"pollRate": 5000
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"apiKeys": [
|
||||
{
|
||||
"key": "YOUR_API_KEY_1",
|
||||
"rateLimitPerMinute": 50
|
||||
},
|
||||
{
|
||||
"key": "YOUR_API_KEY_2",
|
||||
"rateLimitPerMinute": 50
|
||||
}
|
||||
],
|
||||
"serviceAccounts": [
|
||||
{
|
||||
"path": "credentials.json",
|
||||
"rateLimitPerMinute": 50
|
||||
}
|
||||
],
|
||||
"documents": [
|
||||
{
|
||||
"documentId": "YOUR_SPREADSHEET_ID",
|
||||
"outputDir": "output/your-document",
|
||||
"sheets": [
|
||||
{
|
||||
"name": "Sheet1",
|
||||
"outputFilename": "sheet1.csv"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,84 +1,358 @@
|
||||
const { GoogleSpreadsheet } = require('google-spreadsheet');
|
||||
// Google Sheets Rate Assistant
|
||||
// Orchestrates periodic fetches of Google Sheets and writes CSV outputs.
|
||||
// Uses `./lib/config`, `./lib/rate`, and `./lib/auth` for configuration,
|
||||
// rate-limiting, and authentication/fetch helpers respectively.
|
||||
// === Imports & constants ===
|
||||
const fs = require('fs');
|
||||
const readline = require('readline').createInterface({
|
||||
input: process.stdin,
|
||||
output: process.stdout,
|
||||
});
|
||||
const config = require('./config.json');
|
||||
const path = require('path');
|
||||
const {google} = require('googleapis');
|
||||
const stringifyModule = require('csv-stringify/sync');
|
||||
const stringify = typeof stringifyModule === 'function'
|
||||
? stringifyModule
|
||||
: stringifyModule.default || stringifyModule.stringify || stringifyModule;
|
||||
const {loadConfig, resolveOutputPath, validateConfig} = require('./lib/config');
|
||||
const {buildThrottle, applyRateLimitBuffer, getTotalRateLimitPerMinute} = require('./lib/rate');
|
||||
const authHelpers = require('./lib/auth');
|
||||
const {createApiKeyBatchFetcher, createServiceAccountBatchFetcher, getFirstApiKey, getFirstServiceAccountAuthClient, loadAuth} = authHelpers;
|
||||
|
||||
const check = new Promise(function (resolve) {
|
||||
var count = 0,
|
||||
position = {};
|
||||
config.apis.forEach(function (document, d) {
|
||||
var totalRate = 0;
|
||||
document.documents.forEach(function (item, i) {
|
||||
item.sheets.forEach(function (sh, s) {
|
||||
count += 1;
|
||||
position[d + "" + i + "" + s] = count;
|
||||
});
|
||||
totalRate += (60 / (item.pollRate / 1000));
|
||||
//config.apis[d].documents[i].sheets.itemNo = count;
|
||||
});
|
||||
config.position = position;
|
||||
var stdin = process.openStdin();
|
||||
if (totalRate > 50) {
|
||||
console.log('\n---------- RATE LIMIT WARNING ----------');
|
||||
readline.question(`\nYour poll rate will be ${totalRate} per minute, this is above the recommendation of 50 per minute.\nIf the GoogleAPI limit is reached (60 per min on free) you will receive no updates until a break period has passed.\n(API: ${document.apiKey})\n\nAre you sure you want to continue? [y/n]: `, answer => {
|
||||
if (answer == 'y') {
|
||||
readline.close();
|
||||
resolve();
|
||||
} else {
|
||||
process.exit();
|
||||
}
|
||||
});
|
||||
} else if (config.apis.length - 1 == d) {
|
||||
resolve();
|
||||
// === Status rendering / console display ===
|
||||
// Renders per-sheet status lines when running in a TTY, updating in-place.
|
||||
|
||||
function mapDocumentToFilename(documentConfig, sheetName) {
|
||||
const safeName = sheetName.replace(/[\\/:*?"<>|]/g, '_');
|
||||
return `${documentConfig.documentId}-${safeName}.csv`;
|
||||
}
|
||||
|
||||
const statusManager = {
|
||||
keys: [],
|
||||
statuses: new Map(),
|
||||
lastRenderLines: 0,
|
||||
enabled: process.stdout && process.stdout.isTTY,
|
||||
};
|
||||
|
||||
// === Document grouping & mapping ===
|
||||
// Map `config.documents` entries to internal groups with resolved output paths.
|
||||
|
||||
function renderStatusBlock() {
|
||||
if (!statusManager.enabled || statusManager.keys.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (statusManager.lastRenderLines > 0) {
|
||||
process.stdout.write(`\x1B[${statusManager.lastRenderLines}F`);
|
||||
}
|
||||
|
||||
for (const key of statusManager.keys) {
|
||||
const statusText = statusManager.statuses.get(key) || `${key} - pending`;
|
||||
process.stdout.clearLine(0);
|
||||
process.stdout.cursorTo(0);
|
||||
process.stdout.write(statusText + '\n');
|
||||
}
|
||||
|
||||
statusManager.lastRenderLines = statusManager.keys.length;
|
||||
}
|
||||
|
||||
function setTaskStatus(taskId, statusText) {
|
||||
statusManager.statuses.set(taskId, statusText);
|
||||
renderStatusBlock();
|
||||
}
|
||||
|
||||
function initializeTaskStatuses(tasks) {
|
||||
statusManager.keys = tasks.map(task => `${task.documentId}:${task.sheetName}`);
|
||||
statusManager.statuses.clear();
|
||||
statusManager.lastRenderLines = 0;
|
||||
|
||||
for (const key of statusManager.keys) {
|
||||
statusManager.statuses.set(key, `${key} - pending`);
|
||||
}
|
||||
|
||||
renderStatusBlock();
|
||||
}
|
||||
|
||||
// Authentication helpers: see `./lib/auth` for credential loaders and
|
||||
// per-credential, throttled batch fetchers (API key and service account modes).
|
||||
|
||||
async function fetchSpreadsheetTitle(context, documentId) {
|
||||
const sheetOptions = {version: 'v4'};
|
||||
if (context.authClient) {
|
||||
sheetOptions.auth = context.authClient;
|
||||
}
|
||||
|
||||
const sheets = google.sheets(sheetOptions);
|
||||
const request = {
|
||||
spreadsheetId: documentId,
|
||||
fields: 'properties/title',
|
||||
};
|
||||
|
||||
if (context.apiKey) {
|
||||
request.key = context.apiKey;
|
||||
}
|
||||
|
||||
const response = await sheets.spreadsheets.get(request);
|
||||
return response.data.properties?.title || documentId;
|
||||
}
|
||||
|
||||
function buildDocumentGroups(config) {
|
||||
const groups = [];
|
||||
|
||||
if (!Array.isArray(config.documents)) {
|
||||
throw new Error('`documents` must be an array in config.json');
|
||||
}
|
||||
|
||||
for (const documentConfig of config.documents) {
|
||||
if (!documentConfig.documentId || !Array.isArray(documentConfig.sheets)) {
|
||||
throw new Error('Each document must include documentId and sheets array');
|
||||
}
|
||||
|
||||
const documentOutputDir = documentConfig.outputDir || './output';
|
||||
const sheets = [];
|
||||
|
||||
for (const sheetConfig of documentConfig.sheets) {
|
||||
const name = typeof sheetConfig === 'string' ? sheetConfig : sheetConfig.name;
|
||||
const outputFilename = sheetConfig.outputFilename || mapDocumentToFilename(documentConfig, name);
|
||||
const outputPath = resolveOutputPath(documentOutputDir, outputFilename);
|
||||
|
||||
sheets.push({
|
||||
sheetName: name,
|
||||
outputPath,
|
||||
});
|
||||
}
|
||||
|
||||
groups.push({
|
||||
documentId: documentConfig.documentId,
|
||||
sheets,
|
||||
});
|
||||
}
|
||||
|
||||
return groups;
|
||||
}
|
||||
|
||||
// === Batch fetching (fallback) ===
|
||||
// `index.js` provides a simple `batchFetchDocumentSheets` fallback used when
|
||||
// no per-credential batch fetcher is configured by `./lib/auth`.
|
||||
async function batchFetchDocumentSheets(context, documentId, sheetNames) {
|
||||
const sheetOptions = {version: 'v4'};
|
||||
if (context.authClient) {
|
||||
sheetOptions.auth = context.authClient;
|
||||
}
|
||||
|
||||
const sheets = google.sheets(sheetOptions);
|
||||
const request = {
|
||||
spreadsheetId: documentId,
|
||||
ranges: sheetNames,
|
||||
};
|
||||
|
||||
if (context.apiKey) {
|
||||
request.key = context.apiKey;
|
||||
}
|
||||
|
||||
const response = await sheets.spreadsheets.values.batchGet(request);
|
||||
const valueRanges = response.data.valueRanges || [];
|
||||
const rowsBySheet = {};
|
||||
|
||||
for (let i = 0; i < sheetNames.length; i += 1) {
|
||||
const sheetName = sheetNames[i];
|
||||
rowsBySheet[sheetName] = valueRanges[i]?.values || [];
|
||||
}
|
||||
|
||||
return rowsBySheet;
|
||||
}
|
||||
|
||||
function formatStatusTimestamp(date = new Date()) {
|
||||
const pad = value => String(value).padStart(2, '0');
|
||||
const year = date.getFullYear();
|
||||
const month = pad(date.getMonth() + 1);
|
||||
const day = pad(date.getDate());
|
||||
const hours = pad(date.getHours());
|
||||
const minutes = pad(date.getMinutes());
|
||||
const seconds = pad(date.getSeconds());
|
||||
return `${year}-${month}-${day} ${hours}:${minutes}:${seconds}`;
|
||||
}
|
||||
|
||||
// === Execution / saving ===
|
||||
// Functions that execute per-document fetches and write CSV outputs.
|
||||
|
||||
function writeCsv(outputPath, rows) {
|
||||
const csv = stringify(rows, {
|
||||
header: false,
|
||||
quoted: true,
|
||||
});
|
||||
fs.writeFileSync(outputPath, csv, 'utf8');
|
||||
}
|
||||
|
||||
async function executeDocumentGroup(group, fetcher) {
|
||||
const sheetNames = group.sheets.map(sheet => sheet.sheetName);
|
||||
|
||||
for (const sheet of group.sheets) {
|
||||
const displayName = `${group.documentDisplayName || group.documentId}:${sheet.sheetName}`;
|
||||
setTaskStatus(`${group.documentId}:${sheet.sheetName}`, `${displayName} - pulling...`);
|
||||
}
|
||||
|
||||
const rowsBySheet = await fetcher(group.documentId, sheetNames);
|
||||
|
||||
for (const sheet of group.sheets) {
|
||||
const rows = rowsBySheet[sheet.sheetName] || [];
|
||||
writeCsv(sheet.outputPath, rows);
|
||||
const relativePath = path.relative(process.cwd(), sheet.outputPath) || sheet.outputPath;
|
||||
const timestamp = formatStatusTimestamp();
|
||||
const displayName = `${group.documentDisplayName || group.documentId}:${sheet.sheetName}`;
|
||||
setTaskStatus(
|
||||
`${group.documentId}:${sheet.sheetName}`,
|
||||
`${displayName} - saved to ${relativePath} (updated ${timestamp})`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function formatGoogleError(err) {
|
||||
if (err && err.response && err.response.data) {
|
||||
return `${err.message} (${JSON.stringify(err.response.data)})`;
|
||||
}
|
||||
return err.message || String(err);
|
||||
}
|
||||
|
||||
function interpretGoogleSheetsError(err, documentId, sheetName) {
|
||||
const message = formatGoogleError(err);
|
||||
const notFound = err && err.response && err.response.status === 404;
|
||||
const requestedEntity = message.includes('Requested entity was not found');
|
||||
|
||||
if (notFound || requestedEntity) {
|
||||
let detail = `Spreadsheet not found or inaccessible: ${documentId}`;
|
||||
if (sheetName) {
|
||||
detail += `, sheet: ${sheetName}`;
|
||||
}
|
||||
detail += '. Check that the spreadsheet ID is correct, that the sheet exists, and that the document is shared with the service account or is publicly readable when using an API key.';
|
||||
return detail;
|
||||
}
|
||||
|
||||
return message;
|
||||
}
|
||||
|
||||
function sleep(milliseconds) {
|
||||
return new Promise(resolve => setTimeout(resolve, milliseconds));
|
||||
}
|
||||
|
||||
// === Error formatting / interpretation ===
|
||||
|
||||
// === Main scheduler ===
|
||||
// Orchestrates startup, selects the appropriate fetcher (API keys,
|
||||
// service accounts, or single credentials), and runs the periodic loop.
|
||||
async function scheduleRuns() {
|
||||
const config = loadConfig();
|
||||
try {
|
||||
validateConfig(config);
|
||||
} catch (e) {
|
||||
console.error('Configuration error:', e.message);
|
||||
process.exit(1);
|
||||
}
|
||||
const apiBatchFetcher = createApiKeyBatchFetcher(config);
|
||||
const svcBatchFetcher = createServiceAccountBatchFetcher(config);
|
||||
let fetcher;
|
||||
|
||||
if (apiBatchFetcher) {
|
||||
fetcher = apiBatchFetcher;
|
||||
} else if (svcBatchFetcher) {
|
||||
fetcher = svcBatchFetcher;
|
||||
} else {
|
||||
const auth = await loadAuth(config);
|
||||
const baseRate = config.rateLimitPerMinute || 50;
|
||||
const bufferedRate = applyRateLimitBuffer(baseRate);
|
||||
const throttle = buildThrottle(bufferedRate);
|
||||
fetcher = throttle((documentId, sheetNames) => batchFetchDocumentSheets({authClient: auth}, documentId, sheetNames));
|
||||
}
|
||||
|
||||
const groups = buildDocumentGroups(config);
|
||||
if (groups.length === 0) {
|
||||
throw new Error('No sheets configured to pull in config.json');
|
||||
}
|
||||
|
||||
const firstApiKey = getFirstApiKey(config);
|
||||
const titleMap = {};
|
||||
|
||||
// If running in single-auth mode, try to warm the token
|
||||
if (!apiBatchFetcher && !svcBatchFetcher) {
|
||||
try {
|
||||
const singleAuth = await loadAuth(config);
|
||||
if (singleAuth) {
|
||||
try {
|
||||
const at = await singleAuth.getAccessToken();
|
||||
console.log('Service account token acquired (len=', (at && at.token) ? at.token.length : 'no', ')');
|
||||
} catch (e) {
|
||||
console.error('Failed to acquire initial access token from service account:', e && e.message);
|
||||
}
|
||||
});
|
||||
})
|
||||
}
|
||||
} catch (_) {
|
||||
// loadAuth will throw if missing credentials; continue and let title fetch handle errors
|
||||
}
|
||||
}
|
||||
|
||||
check.then(function () {
|
||||
console.clear();
|
||||
process.stdout.cursorTo(0, 1);
|
||||
console.log(`Ctrl+C to kill the application.`);
|
||||
config.apis.forEach(function (api, d) {
|
||||
api.documents.forEach(function (docs, i) {
|
||||
const doc = new GoogleSpreadsheet(docs.googleDocId);
|
||||
doc.useApiKey(api.apiKey);
|
||||
// build a context for fetching titles: prefer API key, then first service account, then single-auth if present
|
||||
let titleContext = null;
|
||||
if (firstApiKey) {
|
||||
titleContext = {apiKey: firstApiKey};
|
||||
} else if (svcBatchFetcher) {
|
||||
const firstSvcAuth = getFirstServiceAccountAuthClient(config);
|
||||
if (firstSvcAuth) titleContext = {authClient: firstSvcAuth};
|
||||
} else {
|
||||
try {
|
||||
const singleAuth = await loadAuth(config);
|
||||
if (singleAuth) titleContext = {authClient: singleAuth};
|
||||
} catch (_) {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
|
||||
setInterval(function () {
|
||||
(async function () {
|
||||
await doc.loadInfo();
|
||||
var titleClean = doc.title.replace(/[^a-zA-Z0-9 ]/g, '');
|
||||
if (!fs.existsSync(`${config.outputFolder}/${titleClean}/`)) {
|
||||
fs.mkdirSync(`${config.outputFolder}/${titleClean}/`, { recursive: true });
|
||||
}
|
||||
docs.sheets.forEach(function (sheet, s) {
|
||||
(async function () {
|
||||
downloadCSV = await doc.sheetsByTitle[sheet].downloadAsCSV();
|
||||
fs.writeFile(`${config.outputFolder}/${titleClean}/${sheet.replace(/[^a-zA-Z0-9 ]/g)}.csv`, downloadCSV, function (err) {
|
||||
if (err) {
|
||||
return console.log(err);
|
||||
}
|
||||
const date = new Date();
|
||||
process.stdout.cursorTo(0, config.position[d + "" + i + "" + s] + 2);
|
||||
process.stdout.clearLine();
|
||||
console.log(`"${doc.title} - ${sheet}" last updated at ${pad(date.getHours(), 2)}:${pad(date.getMinutes(), 2)}:${pad(date.getSeconds(), 2)}`);
|
||||
process.stdout.cursorTo(31, 1);
|
||||
});
|
||||
}());
|
||||
});
|
||||
}());
|
||||
}, docs.pollRate)
|
||||
});
|
||||
for (const group of groups) {
|
||||
try {
|
||||
titleMap[group.documentId] = await fetchSpreadsheetTitle(titleContext || {}, group.documentId);
|
||||
} catch (err) {
|
||||
titleMap[group.documentId] = group.documentId;
|
||||
console.error(`Failed to load title for ${group.documentId}: ${formatGoogleError(err)}`);
|
||||
}
|
||||
}
|
||||
|
||||
});
|
||||
}, function (err) {
|
||||
console.log(err);
|
||||
})
|
||||
const tasks = groups.flatMap(group =>
|
||||
group.sheets.map(sheet => ({
|
||||
documentId: group.documentId,
|
||||
sheetName: sheet.sheetName,
|
||||
}))
|
||||
);
|
||||
|
||||
function pad(n, width, z) {
|
||||
z = z || '0';
|
||||
n = n + '';
|
||||
return n.length >= width ? n : new Array(width - n.length + 1).join(z) + n;
|
||||
}
|
||||
tasks.forEach(task => {
|
||||
task.documentDisplayName = titleMap[task.documentId] || task.documentId;
|
||||
});
|
||||
|
||||
groups.forEach(group => {
|
||||
group.documentDisplayName = titleMap[group.documentId] || group.documentId;
|
||||
});
|
||||
|
||||
const totalRateLimit = getTotalRateLimitPerMinute(config);
|
||||
const intervalMs = Math.max(1, Math.ceil(60000 / totalRateLimit));
|
||||
console.log(`Configured for ${totalRateLimit} requests per minute across ${tasks.length} tasks (${intervalMs}ms interval)`);
|
||||
initializeTaskStatuses(tasks);
|
||||
|
||||
let currentIndex = 0;
|
||||
while (true) {
|
||||
const group = groups[currentIndex];
|
||||
currentIndex = (currentIndex + 1) % groups.length;
|
||||
|
||||
try {
|
||||
await executeDocumentGroup(group, fetcher);
|
||||
} catch (err) {
|
||||
for (const sheet of group.sheets) {
|
||||
setTaskStatus(
|
||||
`${group.documentId}:${sheet.sheetName}`,
|
||||
`${group.documentId}:${sheet.sheetName} - failed: ${interpretGoogleSheetsError(err, group.documentId, sheet.sheetName)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
await sleep(intervalMs);
|
||||
}
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
scheduleRuns().catch(err => {
|
||||
console.error('Application error:', err.message || err);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
|
||||
+199
@@ -0,0 +1,199 @@
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const {google} = require('googleapis');
|
||||
const {applyRateLimitBuffer} = require('./rate');
|
||||
const pThrottleModule = require('p-throttle');
|
||||
const pThrottle = pThrottleModule.default || pThrottleModule;
|
||||
|
||||
function loadApiKeys(config) {
|
||||
const apiKeys = [];
|
||||
const defaultRateLimit = config.rateLimitPerMinute || 50;
|
||||
|
||||
if (Array.isArray(config.apiKeys)) {
|
||||
for (const entry of config.apiKeys) {
|
||||
if (typeof entry === 'string') {
|
||||
apiKeys.push({key: entry, rateLimitPerMinute: defaultRateLimit});
|
||||
} else if (entry && typeof entry === 'object') {
|
||||
if (!entry.key) {
|
||||
throw new Error('Each entry in `apiKeys` must include `key`');
|
||||
}
|
||||
apiKeys.push({
|
||||
key: entry.key,
|
||||
rateLimitPerMinute: entry.rateLimitPerMinute || defaultRateLimit,
|
||||
});
|
||||
} else {
|
||||
throw new Error('`apiKeys` entries must be strings or objects');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return apiKeys;
|
||||
}
|
||||
|
||||
function loadServiceAccounts(config) {
|
||||
const accounts = [];
|
||||
const defaultRateLimit = config.rateLimitPerMinute || 50;
|
||||
|
||||
if (!Array.isArray(config.serviceAccounts)) {
|
||||
return accounts;
|
||||
}
|
||||
|
||||
for (const entry of config.serviceAccounts) {
|
||||
if (typeof entry === 'string') {
|
||||
accounts.push({path: entry, rateLimitPerMinute: defaultRateLimit});
|
||||
} else if (entry && typeof entry === 'object') {
|
||||
if (!entry.path && !entry.credentialsPath) {
|
||||
throw new Error('Each entry in `serviceAccounts` must include `path` (or `credentialsPath`)');
|
||||
}
|
||||
const p = entry.path || entry.credentialsPath;
|
||||
accounts.push({path: p, rateLimitPerMinute: entry.rateLimitPerMinute || defaultRateLimit});
|
||||
} else {
|
||||
throw new Error('`serviceAccounts` entries must be strings or objects');
|
||||
}
|
||||
}
|
||||
|
||||
return accounts;
|
||||
}
|
||||
|
||||
function createServiceAccountBatchFetcher(config) {
|
||||
const accounts = loadServiceAccounts(config);
|
||||
if (accounts.length === 0) return null;
|
||||
|
||||
const fetchers = accounts.map(({path: credentialsPath, rateLimitPerMinute}) => {
|
||||
const resolvedPath = path.isAbsolute(credentialsPath) ? credentialsPath : path.resolve(process.cwd(), credentialsPath);
|
||||
if (!fs.existsSync(resolvedPath)) {
|
||||
throw new Error(`Service account credentials not found: ${resolvedPath}`);
|
||||
}
|
||||
|
||||
let credentials;
|
||||
try {
|
||||
credentials = JSON.parse(fs.readFileSync(resolvedPath, 'utf8'));
|
||||
} catch (e) {
|
||||
throw new Error(`Failed to read/parse service account credentials at ${resolvedPath}: ${e.message}`);
|
||||
}
|
||||
|
||||
let authClient;
|
||||
try {
|
||||
authClient = google.auth.fromJSON(credentials);
|
||||
} catch (e) {
|
||||
throw new Error(`Failed to initialize auth client from credentials at ${resolvedPath}: ${e.message}`);
|
||||
}
|
||||
authClient.scopes = ['https://www.googleapis.com/auth/spreadsheets.readonly'];
|
||||
|
||||
const bufferedRateLimit = applyRateLimitBuffer(rateLimitPerMinute);
|
||||
const interval = Math.ceil(60000 / bufferedRateLimit);
|
||||
const throttled = pThrottle({limit: 1, interval})(async (documentId, sheetNames) => {
|
||||
const sheets = google.sheets({version: 'v4', auth: authClient});
|
||||
const request = {spreadsheetId: documentId, ranges: sheetNames};
|
||||
const response = await sheets.spreadsheets.values.batchGet(request);
|
||||
const valueRanges = response.data.valueRanges || [];
|
||||
const rowsBySheet = {};
|
||||
for (let i = 0; i < sheetNames.length; i += 1) {
|
||||
const sheetName = sheetNames[i];
|
||||
rowsBySheet[sheetName] = valueRanges[i]?.values || [];
|
||||
}
|
||||
return rowsBySheet;
|
||||
});
|
||||
|
||||
return {authClient, fetch: throttled};
|
||||
});
|
||||
|
||||
let currentIndex = 0;
|
||||
return async (documentId, sheetNames) => {
|
||||
const worker = fetchers[currentIndex];
|
||||
currentIndex = (currentIndex + 1) % fetchers.length;
|
||||
return worker.fetch(documentId, sheetNames);
|
||||
};
|
||||
}
|
||||
|
||||
function createApiKeyBatchFetcher(config) {
|
||||
const apiKeys = loadApiKeys(config);
|
||||
if (apiKeys.length === 0) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const fetchers = apiKeys.map(({key, rateLimitPerMinute}) => {
|
||||
const bufferedRateLimit = applyRateLimitBuffer(rateLimitPerMinute);
|
||||
const interval = Math.ceil(60000 / bufferedRateLimit);
|
||||
const throttled = pThrottle({limit: 1, interval})(async (documentId, sheetNames) => {
|
||||
const sheets = google.sheets({version: 'v4'});
|
||||
const request = {spreadsheetId: documentId, ranges: sheetNames, key};
|
||||
const response = await sheets.spreadsheets.values.batchGet(request);
|
||||
const valueRanges = response.data.valueRanges || [];
|
||||
const rowsBySheet = {};
|
||||
for (let i = 0; i < sheetNames.length; i += 1) {
|
||||
const sheetName = sheetNames[i];
|
||||
rowsBySheet[sheetName] = valueRanges[i]?.values || [];
|
||||
}
|
||||
return rowsBySheet;
|
||||
});
|
||||
return {key, fetch: throttled};
|
||||
});
|
||||
|
||||
let currentIndex = 0;
|
||||
return async (documentId, sheetNames) => {
|
||||
const worker = fetchers[currentIndex];
|
||||
currentIndex = (currentIndex + 1) % fetchers.length;
|
||||
return worker.fetch(documentId, sheetNames);
|
||||
};
|
||||
}
|
||||
|
||||
function getFirstApiKey(config) {
|
||||
if (Array.isArray(config.apiKeys) && config.apiKeys.length > 0) {
|
||||
const firstEntry = config.apiKeys[0];
|
||||
return typeof firstEntry === 'string' ? firstEntry : firstEntry.key;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function getFirstServiceAccountAuthClient(config) {
|
||||
if (!Array.isArray(config.serviceAccounts) || config.serviceAccounts.length === 0) return null;
|
||||
const firstEntry = config.serviceAccounts[0];
|
||||
const entryObj = typeof firstEntry === 'string' ? {path: firstEntry} : firstEntry;
|
||||
const credentialsPath = entryObj.path || entryObj.credentialsPath;
|
||||
if (!credentialsPath) return null;
|
||||
|
||||
const resolvedPath = path.isAbsolute(credentialsPath) ? credentialsPath : path.resolve(process.cwd(), credentialsPath);
|
||||
if (!fs.existsSync(resolvedPath)) return null;
|
||||
|
||||
try {
|
||||
const raw = fs.readFileSync(resolvedPath, 'utf8');
|
||||
const credentials = JSON.parse(raw);
|
||||
const authClient = google.auth.fromJSON(credentials);
|
||||
authClient.scopes = ['https://www.googleapis.com/auth/spreadsheets.readonly'];
|
||||
return authClient;
|
||||
} catch (e) {
|
||||
console.error(`Failed to load first service account from ${resolvedPath}: ${e.message}`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function loadAuth(config) {
|
||||
if (!config || typeof config !== 'object') return null;
|
||||
|
||||
if (config.credentialsPath) {
|
||||
const credentialsPath = config.credentialsPath;
|
||||
const resolvedPath = path.isAbsolute(credentialsPath) ? credentialsPath : path.resolve(process.cwd(), credentialsPath);
|
||||
if (!fs.existsSync(resolvedPath)) {
|
||||
throw new Error(`Credentials file not found: ${resolvedPath}`);
|
||||
}
|
||||
const raw = fs.readFileSync(resolvedPath, 'utf8');
|
||||
const credentials = JSON.parse(raw);
|
||||
const authClient = google.auth.fromJSON(credentials);
|
||||
authClient.scopes = ['https://www.googleapis.com/auth/spreadsheets.readonly'];
|
||||
return authClient;
|
||||
}
|
||||
|
||||
return getFirstServiceAccountAuthClient(config);
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
loadApiKeys,
|
||||
loadServiceAccounts,
|
||||
createServiceAccountBatchFetcher,
|
||||
createApiKeyBatchFetcher,
|
||||
getFirstApiKey,
|
||||
getFirstServiceAccountAuthClient
|
||||
, loadAuth
|
||||
};
|
||||
@@ -0,0 +1,48 @@
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const CONFIG_PATH = path.resolve(__dirname, '..', 'config.json');
|
||||
|
||||
function loadConfig() {
|
||||
if (!fs.existsSync(CONFIG_PATH)) {
|
||||
throw new Error(`Missing config file at ${CONFIG_PATH}`);
|
||||
}
|
||||
const json = fs.readFileSync(CONFIG_PATH, 'utf8');
|
||||
return JSON.parse(json);
|
||||
}
|
||||
|
||||
function resolveOutputPath(outputDir, filename) {
|
||||
if (!path.isAbsolute(outputDir)) {
|
||||
outputDir = path.resolve(process.cwd(), outputDir);
|
||||
}
|
||||
if (!fs.existsSync(outputDir)) {
|
||||
fs.mkdirSync(outputDir, {recursive: true});
|
||||
}
|
||||
return path.join(outputDir, filename);
|
||||
}
|
||||
|
||||
function validateConfig(config) {
|
||||
if (!config || typeof config !== 'object') {
|
||||
throw new Error('Missing or invalid configuration object');
|
||||
}
|
||||
|
||||
if (!Array.isArray(config.documents) || config.documents.length === 0) {
|
||||
throw new Error('`documents` must be a non-empty array in config.json');
|
||||
}
|
||||
|
||||
const hasApiKeys = Array.isArray(config.apiKeys) && config.apiKeys.length > 0;
|
||||
const hasSvcAccounts = Array.isArray(config.serviceAccounts) && config.serviceAccounts.length > 0;
|
||||
const hasCredPath = !!config.credentialsPath;
|
||||
|
||||
if (!hasApiKeys && !hasSvcAccounts && !hasCredPath) {
|
||||
throw new Error('Configuration must include at least one of: `apiKeys`, `serviceAccounts`, or `credentialsPath`');
|
||||
}
|
||||
|
||||
// Basic validation of documents/sheets structure
|
||||
for (const doc of config.documents) {
|
||||
if (!doc.documentId) throw new Error('Each document must include a `documentId`');
|
||||
if (!Array.isArray(doc.sheets) || doc.sheets.length === 0) throw new Error(`Document ${doc.documentId} must include a non-empty 'sheets' array`);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {loadConfig, resolveOutputPath, validateConfig};
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
const pThrottleModule = require('p-throttle');
|
||||
const pThrottle = pThrottleModule.default || pThrottleModule;
|
||||
|
||||
function buildThrottle(maxRequestsPerMinute) {
|
||||
const limit = Math.max(1, Math.floor(maxRequestsPerMinute));
|
||||
const interval = Math.ceil(60000 / limit);
|
||||
return pThrottle({limit: 1, interval});
|
||||
}
|
||||
|
||||
function applyRateLimitBuffer(rateLimitPerMinute) {
|
||||
return Math.max(1, Math.floor(rateLimitPerMinute * 0.95));
|
||||
}
|
||||
|
||||
function getTotalRateLimitPerMinute(config) {
|
||||
const defaultRateLimit = config.rateLimitPerMinute || 50;
|
||||
let totalRateLimit = 0;
|
||||
|
||||
if (Array.isArray(config.apiKeys) && config.apiKeys.length > 0) {
|
||||
for (const entry of config.apiKeys) {
|
||||
if (typeof entry === 'string') {
|
||||
totalRateLimit += applyRateLimitBuffer(defaultRateLimit);
|
||||
} else if (entry && typeof entry === 'object') {
|
||||
totalRateLimit += applyRateLimitBuffer(entry.rateLimitPerMinute || defaultRateLimit);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (Array.isArray(config.serviceAccounts) && config.serviceAccounts.length > 0) {
|
||||
for (const entry of config.serviceAccounts) {
|
||||
if (typeof entry === 'string') {
|
||||
totalRateLimit += applyRateLimitBuffer(defaultRateLimit);
|
||||
} else if (entry && typeof entry === 'object') {
|
||||
totalRateLimit += applyRateLimitBuffer(entry.rateLimitPerMinute || defaultRateLimit);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (totalRateLimit === 0) {
|
||||
totalRateLimit = applyRateLimitBuffer(defaultRateLimit);
|
||||
}
|
||||
|
||||
return Math.max(1, totalRateLimit);
|
||||
}
|
||||
|
||||
module.exports = {buildThrottle, applyRateLimitBuffer, getTotalRateLimitPerMinute};
|
||||
Generated
+1159
File diff suppressed because it is too large
Load Diff
+6
-6
@@ -1,14 +1,14 @@
|
||||
{
|
||||
"name": "google-sheets-rate-assistant",
|
||||
"version": "0.0.1",
|
||||
"version": "2.0.0",
|
||||
"description": "Pull Google Sheets data and export to CSV with configurable schedules and rate limiting.",
|
||||
"main": "index.js",
|
||||
"scripts": {
|
||||
"run": "node index.js"
|
||||
"start": "node index.js"
|
||||
},
|
||||
"author": "",
|
||||
"license": "ISC",
|
||||
"description": "Assists in keeping requests under the free rate limit for applications such as vMix.",
|
||||
"dependencies": {
|
||||
"google-spreadsheet": "^3.3.0"
|
||||
"csv-stringify": "^6.0.0",
|
||||
"googleapis": "^173.0.0",
|
||||
"p-throttle": "^8.1.0"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,45 +1,59 @@
|
||||
## Google Sheets Rate Assistant
|
||||
# Google Sheets Rate Assistant
|
||||
|
||||
### Instructions
|
||||
This Node.js application pulls configured Google Sheets documents and exports them as CSV files.
|
||||
|
||||
First fill out the config.json with the required information.
|
||||
## Setup
|
||||
|
||||
1. Copy `config.example.json` to `config.json`.
|
||||
2. Configure authentication in `config.json` — choose one of:
|
||||
|
||||
### Run
|
||||
*Node must be installed on your system*
|
||||
- Service account: place your service account JSON in `credentials.json` and set `credentialsPath`, or add one or more entries to the `serviceAccounts` array (each entry may be a path string or an object with `path`/`credentialsPath` and optional `rateLimitPerMinute`).
|
||||
- API keys: add one or more API keys to the `apiKeys` array in `config.json` (each entry may be a string or an object with `key` and optional `rateLimitPerMinute`).
|
||||
|
||||
Download the .zip of the repository and extract into a folder.
|
||||
3. Install dependencies:
|
||||
|
||||
1. Open Powershell in the directory
|
||||
2. run `npm install`
|
||||
3. run `node index`
|
||||
npm install
|
||||
|
||||
### Config Extract
|
||||
4. Run the application:
|
||||
|
||||
```
|
||||
node index.js
|
||||
|
||||
## Setup Guides
|
||||
|
||||
For detailed instructions on authentication methods, please refer to the following guides:
|
||||
|
||||
- [Service Account Setup](./GUIDE_SERVICE_ACCOUNT.md)
|
||||
- [API Key Setup Guide](./GUIDE_API_KEY.md)
|
||||
|
||||
### Sample `config.json`
|
||||
|
||||
Here is an example `config.json` matching the repository's `config.example.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"apiKey": "--- API key ---",
|
||||
"documents": [
|
||||
{
|
||||
"googleDocId": "--- Google Sheet ID ---",
|
||||
"sheets": [
|
||||
"-- Sheet Name ---",
|
||||
"-- 2nd Sheet Name ---"
|
||||
],
|
||||
"pollRate": 1500
|
||||
},
|
||||
{
|
||||
"googleDocId": "--- 2nd Google Sheet ID ---",
|
||||
"sheets": [
|
||||
"-- Sheet Name ---"
|
||||
],
|
||||
"pollRate": 10000
|
||||
}
|
||||
]
|
||||
"apiKeys": [
|
||||
{ "key": "YOUR_API_KEY_1", "rateLimitPerMinute": 50 },
|
||||
{ "key": "YOUR_API_KEY_2", "rateLimitPerMinute": 50 }
|
||||
],
|
||||
"serviceAccounts": [
|
||||
{ "path": "credentials.json", "rateLimitPerMinute": 50 }
|
||||
],
|
||||
"documents": [
|
||||
{
|
||||
"documentId": "YOUR_SPREADSHEET_ID",
|
||||
"outputDir": "output/your-document",
|
||||
"sheets": [
|
||||
{ "name": "Sheet1", "outputFilename": "sheet1.csv" }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
+ Lowest PollRate for a single API key is recommended at 1200 to keep these under the free tier of 60 per minute.
|
||||
+ Each API key should be different, this is not checked.
|
||||
+ Multiple documents can be polled from a single key and the rate will be calculated.
|
||||
+ Multiple sheets can be included under each 'googleDocId' and will not add to each APIKey rate limit.
|
||||
## Rate limiting
|
||||
|
||||
The app enforces a maximum of `rateLimitPerMinute` requests per minute to avoid exceeding API limits.
|
||||
|
||||
## Output
|
||||
|
||||
CSV files are written to `output` by default. Individual documents may override this with their own `outputDir` in `config.json`.
|
||||
|
||||
Reference in New Issue
Block a user