docs-v2/helper-scripts/influxdb3-plugins

InfluxDB 3 Plugin Documentation Automation

This directory contains scripts and configuration for automating plugin documentation sync between influxdata/influxdb3_plugins (source) and influxdata/docs-v2 (documentation site).

Files in This Directory

• port_to_docs.js - Transforms plugin READMEs from influxdb3_plugins to docs-v2 format
• docs_mapping.yaml - Maps source plugin READMEs to destination documentation files
• README.md - This file: complete workflow documentation

Quick Start

Automated Sync (Recommended)

When you update plugin READMEs in influxdb3_plugins:

1. Commit your changes - The reminder workflow will automatically detect changes
2. Click the sync link - A comment will appear with a pre-filled link to create a sync request
3. Review the PR - The workflow creates a pull request with transformed content and screenshots

Manual Sync (Alternative)

To manually trigger synchronization:

1. Create sync request: Open sync request form
2. Fill in plugin names and source commit
3. Submit issue - This automatically triggers the sync workflow

Local Development

For local testing and development:

1. Validate source content (run from influxdb3_plugins directory):

   cd /path/to/influxdb3_plugins
   python scripts/validate_readme.py
   

2. Transform content (run from docs-v2 root directory):

   cd /path/to/docs-v2
   npm run sync-plugins
   

3. Review changes:

   git diff content/shared/influxdb3-plugins/
   

NPM Scripts

From the docs-v2 root directory, you can use these convenient npm scripts:

• npm run sync-plugins - Transform all plugin documentation
• npm run sync-plugins:dry-run - Preview changes without writing files
• npm run validate-plugin-config - Validate the mapping configuration

Overview

This workflow maintains consistent plugin documentation between repositories while ensuring bidirectional consistency and enabling future automation.

Key Principles

1. Single Source of Truth: Plugin READMEs in influxdb3_plugins are the canonical source
2. Consistency: All plugin READMEs follow a standardized template
3. Automation: Documentation updates flow automatically from source to docs
4. Validation: Both repositories validate documentation quality
5. No Divergence: Content differences are managed through transformation rules

Repository Responsibilities

influxdb3_plugins (Source)

• Contains canonical plugin documentation in README files
• Follows strict template structure
• Includes all technical details and examples
• Validates README compliance

docs-v2 (Documentation Site)

• Transforms and enhances source documentation
• Adds product-specific formatting and shortcodes
• Includes additional support sections
• Never directly edits plugin content

Phase 1: Standardize Source Documentation

1.1 README Template Structure

All plugin READMEs in influxdb3_plugins must follow this structure:

# [Plugin Name]

⚡ [trigger-type1, trigger-type2] 🏷️ [tag1, tag2, tag3] 🔧 InfluxDB 3 Core, InfluxDB 3 Enterprise

## Description

[2-4 sentences describing what the plugin does, its main capabilities, supported trigger types, and any special features]

## Configuration

Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin's `config_file_path` parameter.

If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use.

### Plugin metadata

This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin.

### Required parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `param_name` | string | required | Description of parameter |

### [Category] parameters

[Additional parameter tables organized by category]

### TOML configuration

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `config_file_path` | string | none | TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration) |

*To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3.

#### Example TOML configurations

- [config_file.toml](config_file.toml) - Description

For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3_plugins/README.md](/README.md).

## [Special Requirements Section - if applicable]

[For example: Data requirements, Schema requirements, Software requirements]

## Installation steps

1. Start InfluxDB 3 with the Processing Engine enabled:
   ```bash
   influxdb3 serve \
     --node-id node0 \
     --object-store file \
     --data-dir ~/.influxdb3 \
     --plugin-dir ~/.plugins

2. Install required Python packages (if any):

   influxdb3 install package package_name
   

Trigger setup

[Trigger Type 1]

[Description and example]

influxdb3 create trigger \
  --database mydb \
  --plugin-filename gh:influxdata/[plugin]/[plugin].py \
  --trigger-spec "[spec]" \
  --trigger-arguments '[arguments]' \
  trigger_name

[Trigger Type 2]

[Additional trigger examples as needed]

Example usage

Example 1: [Use case name]

[Description of the example]

# Create the trigger
influxdb3 create trigger \
  --database mydb \
  --plugin-filename gh:influxdata/[plugin]/[plugin].py \
  --trigger-spec "[spec]" \
  --trigger-arguments '[arguments]' \
  trigger_name

# Write test data
influxdb3 write \
  --database mydb \
  "measurement,tag=value field=1"

# Query results
influxdb3 query \
  --database mydb \
  "SELECT * FROM result_measurement"

Expected output

[Expected output format]

Details:

• Before: [input state]
• After: [output state]

Example 2: [Use case name]

[Additional examples following the same pattern]

Code overview

Files

• plugin_name.py: Main plugin code
• config.toml: Example configuration file

Main functions

function_name(params)

[Description of what the function does]

Key operations:

1. [Operation 1]
2. [Operation 2]
3. [Operation 3]

Key logic

[Description of the plugin's core logic and processing flow]

Troubleshooting

Common issues

Issue: [Problem description]

Solution: [How to fix it]

Issue: [Problem description]

Solution: [How to fix it]

Debugging tips

1. [Tip 1]: [Description]
2. [Tip 2]: [Description]

Performance considerations

• [Performance note 1]
• [Performance note 2]

Questions/Comments

For questions or comments about this plugin, please open an issue in the influxdb3_plugins repository.

### 1.2 Validation Requirements

Each README must:
- Include all required sections in the correct order
- Use consistent parameter table formatting
- Provide at least 2 complete usage examples
- Include expected output for examples
- Document all configuration parameters
- Link to TOML configuration files (if applicable)

## Phase 2: Transformation to docs-v2

### 2.1 Transformation Rules

When porting content from `influxdb3_plugins` to `docs-v2`:

#### Content Transformations

1. **Remove emoji metadata line** (already in plugin metadata)

2. **Convert relative links to GitHub URLs**:
   - `[file.toml](file.toml)` → `[file.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/[plugin]/file.toml)`
   - `/README.md` → `https://github.com/influxdata/influxdb3_plugins/blob/master/README.md`

3. **Add product shortcodes**:
   - Replace "InfluxDB 3" with `{{% product-name %}}`
   - Add `{{% token-link %}}` for token references
   - Use `{{% show-in "enterprise" %}}` for conditional content

4. **Enhance opening paragraph** (optional):
   - Can be expanded for better SEO and context
   - Must maintain factual accuracy

5. **Add docs-specific sections**:

```markdown
## Logging

Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs:

```bash
influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'"

Log columns:

• event_time: Timestamp of the log event
• trigger_name: Name of the trigger that generated the log
• log_level: Severity level (INFO, WARN, ERROR)
• log_text: Message describing the action or error

Report an issue

For plugin issues, see the Plugins repository issues page.

Find support for {{% product-name %}}

The InfluxDB Discord server is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise.

### 2.2 File Structure in docs-v2

Plugin documentation location:
- **Official plugins**: `/content/shared/influxdb3-plugins/plugins-library/official/[plugin-name].md`
- **Example plugins**: `/content/shared/influxdb3-plugins/plugins-library/examples/[plugin-name].md`
- **Community plugins**: `/content/shared/influxdb3-plugins/plugins-library/community/[plugin-name].md`

## Phase 3: Automation Implementation

### 3.1 Validation Script (scripts/validate_readme.py)

```python
#!/usr/bin/env python3
"""
Validates plugin README files against the standard template.
"""

import sys
import re
from pathlib import Path

REQUIRED_SECTIONS = [
    "# ",  # Title
    "## Description",
    "## Configuration",
    "### Required parameters",
    "## Installation steps",
    "## Trigger setup",
    "## Example usage",
    "## Code overview",
    "## Troubleshooting",
    "## Questions/Comments"
]

def validate_readme(readme_path):
    """Validate a README file against the template."""
    with open(readme_path, 'r') as f:
        content = f.read()
    
    errors = []
    
    # Check for required sections
    for section in REQUIRED_SECTIONS:
        if section not in content:
            errors.append(f"Missing required section: {section}")
    
    # Check for emoji metadata
    if not re.search(r'^⚡.*🏷️.*🔧', content, re.MULTILINE):
        errors.append("Missing emoji metadata line")
    
    # Check for parameter tables
    if '| Parameter |' not in content:
        errors.append("No parameter tables found")
    
    # Check for code examples
    if '```bash' not in content:
        errors.append("No bash code examples found")
    
    # Check for expected output
    if '### Expected output' not in content and '**Expected output' not in content:
        errors.append("No expected output section found")
    
    return errors

if __name__ == "__main__":
    # Validate all plugin READMEs
    plugin_dirs = Path('influxdata').glob('*/README.md')
    
    all_valid = True
    for readme_path in plugin_dirs:
        errors = validate_readme(readme_path)
        if errors:
            all_valid = False
            print(f"\n❌ {readme_path}:")
            for error in errors:
                print(f"  - {error}")
        else:
            print(f"✅ {readme_path}")
    
    sys.exit(0 if all_valid else 1)

3.2 Transformation Script (port_to_docs.py)

#!/usr/bin/env python3
"""
Transforms plugin READMEs from influxdb3_plugins to docs-v2 format.
"""

import re
import yaml
from pathlib import Path

def transform_content(content, plugin_name):
    """Transform README content for docs-v2."""
    
    # Remove emoji metadata line
    content = re.sub(r'^⚡.*🔧.*$', '', content, flags=re.MULTILINE)
    
    # Convert relative links to GitHub URLs
    base_url = f"https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/{plugin_name}/"
    content = re.sub(r'\[([^\]]+)\]\(([^)]+\.toml)\)', 
                     f'[\\1]({base_url}\\2)', content)
    
    # Replace InfluxDB 3 references
    content = content.replace('InfluxDB 3 Core/Enterprise', '{{% product-name %}}')
    content = content.replace('InfluxDB 3', '{{% product-name %}}')
    
    # Remove Questions/Comments section (will be replaced)
    content = re.sub(r'## Questions/Comments.*', '', content, flags=re.DOTALL)
    
    # Add docs-specific sections
    docs_sections = """
## Logging

Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs:

```bash
influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'"

Log columns:

• event_time: Timestamp of the log event
• trigger_name: Name of the trigger that generated the log
• log_level: Severity level (INFO, WARN, ERROR)
• log_text: Message describing the action or error

Report an issue

For plugin issues, see the Plugins repository issues page.

Find support for {{% product-name %}}

The InfluxDB Discord server is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. """

return content + docs_sections

def process_plugin(source_path, target_path, plugin_name): """Process a single plugin README.""" with open(source_path, 'r') as f: content = f.read()

transformed = transform_content(content, plugin_name)

# Ensure target directory exists
target_path.parent.mkdir(parents=True, exist_ok=True)

with open(target_path, 'w') as f:
    f.write(transformed)

print(f"✅ Processed {plugin_name}")

if name == "main": # Load mapping configuration with open('docs_mapping.yaml', 'r') as f: config = yaml.safe_load(f)

for plugin_name, mapping in config['plugins'].items():
    source = Path(mapping['source'])
    target = Path(mapping['target'])
    
    if source.exists():
        process_plugin(source, target, plugin_name)
    else:
        print(f"❌ Source not found: {source}")

### 3.3 Mapping Configuration (docs_mapping.yaml)

```yaml
plugins:
  basic_transformation:
    source: influxdata/basic_transformation/README.md
    target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/basic-transformation.md
    category: official
  
  downsampler:
    source: influxdata/downsampler/README.md
    target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/downsampler.md
    category: official
  
  forecast_error_evaluator:
    source: influxdata/forecast_error_evaluator/README.md
    target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/forecast-error-evaluator.md
    category: official
  
  influxdb_to_iceberg:
    source: influxdata/influxdb_to_iceberg/README.md
    target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/influxdb-to-iceberg.md
    category: official
  
  mad_check:
    source: influxdata/mad_check/README.md
    target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/mad-check.md
    category: official
  
  notifier:
    source: influxdata/notifier/README.md
    target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/notifier.md
    category: official
  
  prophet_forecasting:
    source: influxdata/prophet_forecasting/README.md
    target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/prophet-forecasting.md
    category: official
  
  state_change:
    source: influxdata/state_change/README.md
    target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/state-change.md
    category: official
  
  stateless_adtk_detector:
    source: influxdata/stateless_adtk_detector/README.md
    target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/stateless-adtk-detector.md
    category: official
  
  system_metrics:
    source: influxdata/system_metrics/README.md
    target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/system-metrics.md
    category: official
  
  threshold_deadman_checks:
    source: influxdata/threshold_deadman_checks/README.md
    target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/threshold-deadman-checks.md
    category: official

Phase 4: GitHub Actions Automation

4.1 Automated Workflow Architecture

The plugin documentation sync process uses a two-repository workflow:

1. influxdb3_plugins: Detects README changes and creates reminder comments
2. docs-v2: Handles the complete sync workflow via issue-triggered automation

4.2 Reminder System (influxdb3_plugins)

File: .github/workflows/remind-sync-docs.yml

• Triggers: On pushes to influxdata/**/README.md files
• Action: Creates commit comment with pre-filled link to docs-v2 sync form
• No secrets required: Pure URL-based integration

Flow:

Plugin README change → Commit → Automatic comment with sync link

4.3 Sync Automation (docs-v2)

Files:

• .github/workflows/sync-plugins.yml - Main sync workflow
• .github/ISSUE_TEMPLATE/sync-plugin-docs.yml - Issue template for requests

Triggers:

• Issue creation with sync-plugin-docs label (from template)
• Manual workflow dispatch

Process:

1. Parse request - Extract plugin names and source commit from issue
2. Validate source - Run scripts/validate_readme.py on specified plugins
3. Transform content - Apply docs-v2 formatting and enhancements
4. Generate screenshots - Capture Hugo-rendered plugin pages
5. Create PR - Submit changes with comprehensive summary
6. Update issue - Close with success/failure status

4.4 Key Features

• No cross-repo secrets - Uses URL parameters and public APIs
• Visual validation - Screenshots included in PRs
• Comprehensive error handling - Clear feedback on failures
• Selective sync - Individual plugins or "all"
• Progress tracking - Issue updates throughout process

Phase 5: Manual Workflow (Until Automation is Ready)

Step 1: Update Source README

1. Edit README in influxdb3_plugins/influxdata/[plugin]/
2. Run validation: python scripts/validate_readme.py
3. Fix any validation errors

Step 2: Transform Content

1. Change to docs-v2 directory: cd ../docs-v2
2. Run transformation: npm run sync-plugins
3. Review generated content: git diff content/shared/influxdb3-plugins/

Step 3: Manual Adjustments (if needed)

1. Add any docs-specific enhancements
2. Verify shortcodes and formatting
3. Test links and examples

Step 4: Commit and PR

1. Commit changes in influxdb3_plugins
2. Commit transformed content in docs-v2
3. Reference source commit in docs-v2 commit message

Maintenance Guidelines

For New Plugins

1. Create README following template in influxdb3_plugins
2. Add entry to docs_mapping.yaml
3. Run validation and transformation scripts
4. Review and merge generated documentation

For Updates

1. Update source README in influxdb3_plugins
2. Automation (or manual script) creates PR to docs-v2
3. Review for accuracy and merge

Exception Handling

• Use [no-sync] in commit message to skip automation
• Document special cases in exceptions.yaml
• Manual override allowed for urgent fixes

Quality Assurance Checklist

Before merging any plugin documentation:

• Source README passes validation (scripts/validate_readme.py)
• All required sections present and properly formatted
• At least 2 complete usage examples with expected output
• All configuration parameters documented
• Links to GitHub source files work
• TOML configuration documented (if applicable)
• Transformation script runs without errors
• Generated docs-v2 content is accurate
• Product shortcodes properly applied
• Support sections included

Benefits of This Workflow

1. Consistency: Single source of truth eliminates divergence
2. Automation: Reduces manual work and human error
3. Validation: Ensures quality standards are met
4. Scalability: Easy to add new plugins
5. Maintainability: Clear ownership and update process
6. Traceability: Git history shows relationship between repos

Future Enhancements

1. Automated testing of code examples
2. Version tracking for plugin compatibility
3. Changelog generation from commit history
4. Multi-language example generation
5. API documentation generation from metadata
6. Search optimization through automated tagging

Conclusion

This workflow establishes a sustainable process for maintaining high-quality, consistent plugin documentation across both repositories while enabling progressive automation. The standardized template and transformation rules ensure that documentation remains synchronized while allowing for repository-specific requirements.