chore(scripts): redo cli audit script:

Moved to new tooling repo.
Removed package scripts for now.

Script Gets commands from source code and grep docs for commands- replaces the CLI audit script- searches tagged repo branch source code for CLI commands- searches docs content the commands- allows including and excluding "categories" of docs paths to search
pull/6282/head
Jason Stirnaman 2025-08-05 16:20:03 -05:00 committed by Jason Stirnaman
parent 92210137b2
commit 3f4ad5fb37
5 changed files with 1 additions and 2034 deletions

View File

@ -1,373 +0,0 @@
# InfluxDB 3 Monolith (Core and Enterprise) Helper Scripts
This directory contains helper scripts specifically for InfluxDB 3 Core and Enterprise (monolith deployments), as opposed to distributed/clustered deployments.
## Overview
These scripts help with documentation workflows for InfluxDB 3 Core and Enterprise, including CLI change detection, authentication setup, API analysis, and release preparation.
## Prerequisites
- **Docker and Docker Compose**: For running InfluxDB 3 containers
- **Node.js 16+**: For running JavaScript ESM scripts
- **Active containers**: InfluxDB 3 Core and/or Enterprise containers running via `docker compose`
- **Secret files**: Docker Compose secrets for auth tokens (`~/.env.influxdb3-core-admin-token` and `~/.env.influxdb3-enterprise-admin-token`)
## Scripts
### 🔐 Authentication & Setup
#### `setup-auth-tokens.sh`
Creates and configures authentication tokens for InfluxDB 3 containers.
**Usage:**
```bash
./setup-auth-tokens.sh [core|enterprise|both]
```
**What it does:**
- Checks existing tokens in secret files (`~/.env.influxdb3-core-admin-token` and `~/.env.influxdb3-enterprise-admin-token`)
- Starts containers if not running
- Creates admin tokens using `influxdb3 create token --admin`
- Updates appropriate secret files with new tokens
- Tests tokens to ensure they work
**Example:**
```bash
# Set up both Core and Enterprise tokens
./setup-auth-tokens.sh both
# Set up only Enterprise
./setup-auth-tokens.sh enterprise
```
### 🔍 CLI Documentation Audit
#### `audit-cli-documentation.js`
JavaScript ESM script that audits InfluxDB 3 CLI commands against existing documentation to identify missing or outdated content.
**Usage:**
```bash
node audit-cli-documentation.js [core|enterprise|both] [version|local]
```
**Features:**
- Compares actual CLI help output with documented commands
- Identifies missing documentation for new CLI options
- Finds documented options that no longer exist in the CLI
- Supports both released versions and local containers
- Generates detailed audit reports with recommendations
- Handles authentication automatically using Docker secrets
**Examples:**
```bash
# Audit Core documentation against local container
node audit-cli-documentation.js core local
# Audit Enterprise documentation against specific version
node audit-cli-documentation.js enterprise v3.2.0
# Audit both products against local containers
node audit-cli-documentation.js both local
```
**Output:**
- `../output/cli-audit/documentation-audit-{product}-{version}.md` - Detailed audit report
- `../output/cli-audit/parsed-cli-{product}-{version}.md` - Parsed CLI structure
- `../output/cli-audit/patches/{product}/` - Generated patches for missing documentation
### 🛠️ CLI Documentation Updates
#### `apply-cli-patches.js`
JavaScript ESM script that applies generated patches to update CLI documentation with missing options.
**Usage:**
```bash
node apply-cli-patches.js [core|enterprise|both] [--dry-run]
```
**Features:**
- Applies patches generated by `audit-cli-documentation.js`
- Updates CLI reference documentation with missing options
- Supports dry-run mode to preview changes
- Maintains existing documentation structure and formatting
- Creates backups before applying changes
**Examples:**
```bash
# Preview changes without applying (dry run)
node apply-cli-patches.js core --dry-run
# Apply patches to Enterprise documentation
node apply-cli-patches.js enterprise
# Apply patches to both products
node apply-cli-patches.js both
```
**Output:**
- Updates CLI reference documentation files in place
- Creates backup files with `.backup` extension
- Logs all changes made to the documentation
## Quick Start Guide
### 1. Initial Setup
```bash
# Navigate to the monolith scripts directory
cd helper-scripts/influxdb3-monolith
# Make scripts executable
chmod +x *.sh
# Set up authentication for both products
./setup-auth-tokens.sh both
# Restart containers to load new secrets
docker compose down && docker compose up -d influxdb3-core influxdb3-enterprise
```
### 2. CLI Documentation Audit
```bash
# Start your containers
docker compose up -d influxdb3-core influxdb3-enterprise
# Audit CLI documentation
node audit-cli-documentation.js core local
node audit-cli-documentation.js enterprise local
# Review the output
ls ../output/cli-audit/
```
### 3. Development Workflow
```bash
# Audit documentation for both products
node audit-cli-documentation.js both local
# Check the audit results
cat ../output/cli-audit/documentation-audit-core-local.md
cat ../output/cli-audit/documentation-audit-enterprise-local.md
# Apply patches if needed (dry run first)
node apply-cli-patches.js both --dry-run
```
### 4. Release Documentation Updates
For release documentation, use the audit and patch workflow:
```bash
# Audit against released version
node audit-cli-documentation.js enterprise v3.2.0
# Review missing documentation
cat ../output/cli-audit/documentation-audit-enterprise-v3.2.0.md
# Apply patches to update documentation
node apply-cli-patches.js enterprise
# Verify changes look correct
git diff content/influxdb3/enterprise/reference/cli/
```
## Container Integration
The scripts work with your Docker Compose setup:
**Expected container names:**
- `influxdb3-core` (port 8282)
- `influxdb3-enterprise` (port 8181)
**Docker Compose secrets:**
- `influxdb3-core-admin-token` - Admin token for Core (stored in `~/.env.influxdb3-core-admin-token`)
- `influxdb3-enterprise-admin-token` - Admin token for Enterprise (stored in `~/.env.influxdb3-enterprise-admin-token`)
- `INFLUXDB3_LICENSE_EMAIL` - Enterprise license email (set in `.env.3ent` env_file)
## Use Cases
### 📋 Release Documentation
1. **Pre-release audit:**
```bash
node audit-cli-documentation.js core v3.2.0
```
2. **Review audit results and update documentation**
3. **Apply patches for missing content**
4. **Test documented commands work correctly**
### 🔬 Development Testing
1. **Audit local development:**
```bash
node audit-cli-documentation.js enterprise local
```
2. **Verify new features are documented**
3. **Test authentication setup**
4. **Apply patches to keep docs current**
### 🚀 Release Preparation
1. **Final audit before release:**
```bash
node audit-cli-documentation.js both local
```
2. **Apply all pending patches**
3. **Update examples and tutorials**
4. **Verify all CLI commands work as documented**
## Output Structure
```
helper-scripts/
├── output/
│ └── cli-audit/
│ ├── documentation-audit-core-local.md # CLI documentation audit report
│ ├── documentation-audit-enterprise-v3.2.0.md # CLI documentation audit report
│ ├── parsed-cli-core-local.md # Parsed CLI structure
│ ├── parsed-cli-enterprise-v3.2.0.md # Parsed CLI structure
│ └── patches/
│ ├── core/ # Generated patches for Core
│ │ ├── influxdb3-cli-patch-001.md
│ │ └── influxdb3-cli-patch-002.md
│ └── enterprise/ # Generated patches for Enterprise
│ ├── influxdb3-cli-patch-001.md
│ └── influxdb3-cli-patch-002.md
└── influxdb3-monolith/
├── README.md # This file
├── setup-auth-tokens.sh # Auth setup
├── audit-cli-documentation.js # CLI documentation audit
└── apply-cli-patches.js # CLI documentation patches
```
## Error Handling
### Common Issues
**Container not running:**
```bash
# Check status
docker compose ps
# Start specific service
docker compose up -d influxdb3-core
```
**Authentication failures:**
```bash
# Recreate tokens
./setup-auth-tokens.sh both
# Test manually
docker exec influxdb3-core influxdb3 create token --admin
```
**Version not found:**
```bash
# Check available versions
docker pull influxdb:3-core:3.2.0
docker pull influxdb:3-enterprise:3.2.0
```
### Debug Mode
Enable debug output for troubleshooting:
```bash
DEBUG=1 node audit-cli-documentation.js core local
```
## Integration with CI/CD
### GitHub Actions Example
```yaml
- name: Audit CLI Documentation
run: |
cd helper-scripts/influxdb3-monolith
node audit-cli-documentation.js core ${{ env.VERSION }}
- name: Upload CLI Audit Results
uses: actions/upload-artifact@v3
with:
name: cli-audit
path: helper-scripts/output/cli-audit/
```
### CircleCI Example
```yaml
- run:
name: CLI Documentation Audit
command: |
cd helper-scripts/influxdb3-monolith
node audit-cli-documentation.js enterprise v3.2.0
- store_artifacts:
path: helper-scripts/output/cli-audit/
```
## Best Practices
### 🔒 Security
- Secret files (`~/.env.influxdb3-*-admin-token`) are stored in your home directory and not in version control
- Rotate auth tokens regularly by re-running `setup-auth-tokens.sh`
- Use minimal token permissions when possible
### 📚 Documentation
- Run audits early in release cycle
- Review all audit reports for missing content
- Apply patches to keep documentation current
- Test all documented commands work correctly
### 🔄 Workflow
- Use `local` version for development testing
- Audit against released versions for release prep
- Generate patches before documentation updates
- Validate changes with stakeholders
## Troubleshooting
### Script Permissions
```bash
chmod +x *.sh
```
### Missing Dependencies
```bash
# Node.js dependencies
node --version # Should be 16 or higher
# Docker Compose
docker compose version
```
### Container Health
```bash
# Check container logs
docker logs influxdb3-core
docker logs influxdb3-enterprise
# Test basic connectivity
docker exec influxdb3-core influxdb3 --version
```
## Contributing
When adding new scripts to this directory:
1. **Follow naming conventions**: Use lowercase with hyphens
2. **Add usage documentation**: Include help text in scripts
3. **Handle errors gracefully**: Use proper exit codes
4. **Test with both products**: Ensure Core and Enterprise compatibility
5. **Update this README**: Document new functionality
## Related Documentation
- [InfluxDB 3 Core CLI Reference](/influxdb3/core/reference/cli/)
- [InfluxDB 3 Enterprise CLI Reference](/influxdb3/enterprise/reference/cli/)

View File

@ -1,277 +0,0 @@
#!/usr/bin/env node
/**
* Apply CLI documentation patches generated by audit-cli-documentation.js
* Usage: node apply-cli-patches.js [core|enterprise|both] [--dry-run]
*/
import { promises as fs } from 'fs';
import { join, dirname } from 'path';
import { fileURLToPath } from 'url';
import { process } from 'node:process';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
// Color codes
const Colors = {
RED: '\x1b[0;31m',
GREEN: '\x1b[0;32m',
YELLOW: '\x1b[1;33m',
BLUE: '\x1b[0;34m',
NC: '\x1b[0m', // No Color
};
async function fileExists(path) {
try {
await fs.access(path);
return true;
} catch {
return false;
}
}
async function ensureDir(dir) {
await fs.mkdir(dir, { recursive: true });
}
async function extractFrontmatter(content) {
const lines = content.split('\n');
if (lines[0] !== '---') return { frontmatter: null, content };
const frontmatterLines = [];
let i = 1;
while (i < lines.length && lines[i] !== '---') {
frontmatterLines.push(lines[i]);
i++;
}
if (i >= lines.length) return { frontmatter: null, content };
const frontmatterText = frontmatterLines.join('\n');
const remainingContent = lines.slice(i + 1).join('\n');
return { frontmatter: frontmatterText, content: remainingContent };
}
async function getActualDocumentationPath(docPath, projectRoot) {
// Check if the documentation file exists and has a source field
const fullPath = join(projectRoot, docPath);
if (await fileExists(fullPath)) {
const content = await fs.readFile(fullPath, 'utf8');
const { frontmatter } = await extractFrontmatter(content);
if (frontmatter) {
// Look for source: field in frontmatter
const sourceMatch = frontmatter.match(/^source:\s*(.+)$/m);
if (sourceMatch) {
const sourcePath = sourceMatch[1].trim();
return sourcePath;
}
}
}
return docPath;
}
async function applyPatches(product, dryRun = false) {
const patchDir = join(
dirname(__dirname),
'output',
'cli-audit',
'patches',
product
);
const projectRoot = join(__dirname, '..', '..');
console.log(
`${Colors.BLUE}📋 Applying CLI documentation patches for ${product}${Colors.NC}`
);
if (dryRun) {
console.log(
`${Colors.YELLOW}🔍 DRY RUN - No files will be created${Colors.NC}`
);
}
console.log();
// Check if patch directory exists
if (!(await fileExists(patchDir))) {
console.log(`${Colors.YELLOW}No patches found for ${product}.${Colors.NC}`);
console.log("Run 'yarn audit:cli' first to generate patches.");
return;
}
// Read all patch files
const patchFiles = await fs.readdir(patchDir);
const mdFiles = patchFiles.filter((f) => f.endsWith('.md'));
if (mdFiles.length === 0) {
console.log(
`${Colors.YELLOW}No patch files found in ${patchDir}${Colors.NC}`
);
return;
}
console.log(`Found ${mdFiles.length} patch file(s) to apply:\n`);
// Map patch files to their destination
const baseCliPath = `content/influxdb3/${product}/reference/cli/influxdb3`;
const commandToFile = {
'create-database.md': `${baseCliPath}/create/database.md`,
'create-token.md': `${baseCliPath}/create/token/_index.md`,
'create-token-admin.md': `${baseCliPath}/create/token/admin.md`,
'create-trigger.md': `${baseCliPath}/create/trigger.md`,
'create-table.md': `${baseCliPath}/create/table.md`,
'create-last_cache.md': `${baseCliPath}/create/last_cache.md`,
'create-distinct_cache.md': `${baseCliPath}/create/distinct_cache.md`,
'show-databases.md': `${baseCliPath}/show/databases.md`,
'show-tokens.md': `${baseCliPath}/show/tokens.md`,
'delete-database.md': `${baseCliPath}/delete/database.md`,
'delete-table.md': `${baseCliPath}/delete/table.md`,
'query.md': `${baseCliPath}/query.md`,
'write.md': `${baseCliPath}/write.md`,
};
let applied = 0;
let skipped = 0;
for (const patchFile of mdFiles) {
const destinationPath = commandToFile[patchFile];
if (!destinationPath) {
console.log(
`${Colors.YELLOW}⚠️ Unknown patch file: ${patchFile}${Colors.NC}`
);
continue;
}
// Get the actual documentation path (handles source: frontmatter)
const actualPath = await getActualDocumentationPath(
destinationPath,
projectRoot
);
const fullDestPath = join(projectRoot, actualPath);
const patchPath = join(patchDir, patchFile);
// Check if destination already exists
if (await fileExists(fullDestPath)) {
console.log(
`${Colors.YELLOW}⏭️ Skipping${Colors.NC} ${patchFile} - destination already exists:`
);
console.log(` ${actualPath}`);
skipped++;
continue;
}
if (dryRun) {
console.log(`${Colors.BLUE}🔍 Would create${Colors.NC} ${actualPath}`);
console.log(` from patch: ${patchFile}`);
if (actualPath !== destinationPath) {
console.log(` (resolved from: ${destinationPath})`);
}
applied++;
} else {
try {
// Ensure destination directory exists
await ensureDir(dirname(fullDestPath));
// Copy patch to destination
const content = await fs.readFile(patchPath, 'utf8');
// Update the menu configuration based on product
let updatedContent = content;
if (product === 'enterprise') {
updatedContent = content
.replace('influxdb3/core/tags:', 'influxdb3/enterprise/tags:')
.replace(
'influxdb3_core_reference:',
'influxdb3_enterprise_reference:'
);
}
await fs.writeFile(fullDestPath, updatedContent);
console.log(`${Colors.GREEN}✅ Created${Colors.NC} ${actualPath}`);
console.log(` from patch: ${patchFile}`);
if (actualPath !== destinationPath) {
console.log(` (resolved from: ${destinationPath})`);
}
applied++;
} catch (error) {
console.log(
`${Colors.RED}❌ Error${Colors.NC} creating ${actualPath}:`
);
console.log(` ${error.message}`);
}
}
}
console.log();
console.log(`${Colors.BLUE}Summary:${Colors.NC}`);
console.log(`- Patches ${dryRun ? 'would be' : ''} applied: ${applied}`);
console.log(`- Files skipped (already exist): ${skipped}`);
console.log(`- Total patch files: ${mdFiles.length}`);
if (!dryRun && applied > 0) {
console.log();
console.log(
`${Colors.GREEN}✨ Success!${Colors.NC} Created ${applied} new ` +
'documentation file(s).'
);
console.log();
console.log('Next steps:');
console.log('1. Review the generated files and customize the content');
console.log('2. Add proper examples with placeholders');
console.log('3. Update descriptions and add any missing options');
console.log('4. Run tests: yarn test:links');
}
}
async function main() {
const args = process.argv.slice(2);
const product =
args.find((arg) => ['core', 'enterprise', 'both'].includes(arg)) || 'both';
const dryRun = args.includes('--dry-run');
if (args.includes('--help') || args.includes('-h')) {
console.log(
'Usage: node apply-cli-patches.js [core|enterprise|both] [--dry-run]'
);
console.log();
console.log('Options:');
console.log(
' --dry-run Show what would be done without creating files'
);
console.log();
console.log('Examples:');
console.log(
' node apply-cli-patches.js # Apply patches for both products'
);
console.log(
' node apply-cli-patches.js core --dry-run # Preview core patches'
);
console.log(
' node apply-cli-patches.js enterprise # Apply enterprise patches'
);
process.exit(0);
}
try {
if (product === 'both') {
await applyPatches('core', dryRun);
console.log();
await applyPatches('enterprise', dryRun);
} else {
await applyPatches(product, dryRun);
}
} catch (error) {
console.error(`${Colors.RED}Error:${Colors.NC}`, error.message);
process.exit(1);
}
}
// Run if called directly
if (import.meta.url === `file://${process.argv[1]}`) {
main();
}

File diff suppressed because it is too large Load Diff

View File

@ -1,164 +0,0 @@
#!/bin/bash
# Set up authentication tokens for InfluxDB 3 Core and Enterprise containers
# Usage: ./setup-auth-tokens.sh [core|enterprise|both]
set -e
# Color codes
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'
# Parse arguments
TARGET=${1:-both}
echo -e "${BLUE}🔐 InfluxDB 3 Authentication Setup${NC}"
echo "=================================="
echo ""
# Check for and load existing secret files
SECRET_CORE_FILE="$HOME/.env.influxdb3-core-admin-token"
SECRET_ENT_FILE="$HOME/.env.influxdb3-enterprise-admin-token"
if [ -f "$SECRET_CORE_FILE" ]; then
echo "✅ Found existing Core token secret file"
else
echo "📝 Creating new Core token secret file: $SECRET_CORE_FILE"
touch "$SECRET_CORE_FILE"
fi
if [ -f "$SECRET_ENT_FILE" ]; then
echo "✅ Found existing Enterprise token secret file"
else
echo "📝 Creating new Enterprise token secret file: $SECRET_ENT_FILE"
touch "$SECRET_ENT_FILE"
fi
echo ""
# Function to setup auth for a product
setup_auth() {
local product=$1
local container_name="influxdb3-${product}"
local port
local secret_file
case "$product" in
"core")
port="8282"
secret_file="$SECRET_CORE_FILE"
;;
"enterprise")
port="8181"
secret_file="$SECRET_ENT_FILE"
;;
esac
echo -e "${BLUE}Setting up $(echo ${product} | awk '{print toupper(substr($0,1,1)) tolower(substr($0,2))}') authentication...${NC}"
# Check if token already exists in secret file
if [ -s "$secret_file" ]; then
local existing_token=$(cat "$secret_file")
echo "✅ Token already exists in secret file"
echo " Token: ${existing_token:0:20}..."
# Test if the token works
echo -n "🧪 Testing existing token..."
if docker exec "${container_name}" influxdb3 show databases --token "${existing_token}" --host "http://localhost:${port}" > /dev/null 2>&1; then
echo -e " ${GREEN}✓ Working${NC}"
return 0
else
echo -e " ${YELLOW}⚠ Not working, will create new token${NC}"
fi
fi
# Check if container is running
if ! docker ps --format '{{.Names}}' | grep -q "^${container_name}$"; then
echo "🚀 Starting ${container_name} container..."
if ! docker compose up -d "${container_name}"; then
echo -e "${RED}❌ Failed to start container${NC}"
return 1
fi
echo -n "⏳ Waiting for container to be ready..."
sleep 5
echo -e " ${GREEN}${NC}"
else
echo "✅ Container ${container_name} is running"
fi
# Create admin token
echo "🔑 Creating admin token..."
local token_output
if token_output=$(docker exec "${container_name}" influxdb3 create token --admin 2>&1); then
# Extract the token from the "Token: " line
local new_token=$(echo "$token_output" | grep "^Token: " | sed 's/^Token: //' | tr -d '\r\n')
echo -e "${GREEN}Token created successfully!${NC}"
echo " Token: ${new_token:0:20}..."
# Update secret file
echo "${new_token}" > "$secret_file"
echo "📝 Updated secret file: $secret_file"
# Test the new token
echo -n "🧪 Testing new token..."
if docker exec "${container_name}" influxdb3 show databases --token "${new_token}" --host "http://localhost:${port}" > /dev/null 2>&1; then
echo -e " ${GREEN}✓ Working${NC}"
else
echo -e " ${YELLOW}⚠ Test failed, but token was created${NC}"
fi
else
echo -e "${RED}❌ Failed to create token${NC}"
echo "Error output: $token_output"
return 1
fi
echo ""
}
# Main execution
case "$TARGET" in
"core")
setup_auth "core"
;;
"enterprise")
setup_auth "enterprise"
;;
"both")
setup_auth "core"
setup_auth "enterprise"
;;
*)
echo "Usage: $0 [core|enterprise|both]"
exit 1
;;
esac
echo -e "${GREEN}🎉 Authentication setup complete!${NC}"
echo ""
echo "📋 Next steps:"
echo "1. Restart containers to load new secrets:"
echo " docker compose down && docker compose up -d influxdb3-core influxdb3-enterprise"
echo "2. Test CLI commands with authentication:"
echo " ./detect-cli-changes.sh core 3.1.0 local"
echo " ./detect-cli-changes.sh enterprise 3.1.0 local"
echo ""
echo "📄 Your secret files now contain:"
# Show Core tokens
if [ -f "$SECRET_CORE_FILE" ] && [ -s "$SECRET_CORE_FILE" ]; then
token_preview=$(head -c 20 "$SECRET_CORE_FILE")
echo " $SECRET_CORE_FILE: ${token_preview}..."
fi
# Show Enterprise tokens
if [ -f "$SECRET_ENT_FILE" ] && [ -s "$SECRET_ENT_FILE" ]; then
token_preview=$(head -c 20 "$SECRET_ENT_FILE")
echo " $SECRET_ENT_FILE: ${token_preview}..."
fi

View File

@ -55,12 +55,7 @@
"test:codeblocks:v2": "docker compose run --rm --name v2-pytest v2-pytest",
"test:codeblocks:stop-monitors": "./test/scripts/monitor-tests.sh stop cloud-dedicated-pytest && ./test/scripts/monitor-tests.sh stop clustered-pytest",
"test:e2e": "node cypress/support/run-e2e-specs.js",
"test:shortcode-examples": "node cypress/support/run-e2e-specs.js --spec \"cypress/e2e/content/article-links.cy.js\" content/example.md",
"audit:cli": "node ./helper-scripts/influxdb3-monolith/audit-cli-documentation.js both local",
"audit:cli:3core": "node ./helper-scripts/influxdb3-monolith/audit-cli-documentation.js core local",
"audit:cli:3ent": "node ./helper-scripts/influxdb3-monolith/audit-cli-documentation.js enterprise local",
"audit:cli:apply": "node ./helper-scripts/influxdb3-monolith/apply-cli-patches.js both",
"audit:cli:apply:dry": "node ./helper-scripts/influxdb3-monolith/apply-cli-patches.js both --dry-run"
"test:shortcode-examples": "node cypress/support/run-e2e-specs.js --spec \"cypress/e2e/content/article-links.cy.js\" content/example.md"
},
"type": "module",
"browserslist": [