docs-v2/content/shared/influxdb3-processing-engine.md

Processing engine

The Processing engine is an embedded Python virtual machine that runs inside an {{% product-name %}} database server. It executes Python code in response to triggers and database events without requiring external application servers or middleware.

How it works

Architecture

The Processing engine runs Python code directly within a {{% product-name %}} server process. This design provides high performance and direct access to database resources.

• Embedded execution: Code runs in the same process space as the database server
• Direct data access: Zero-copy access to data
• Event-driven: Responds to database writes, scheduled events, and HTTP requests
• Isolated contexts: Maintains separation between different plugin executions
• Cache integration: Access to system caches including Last values and Distinct values

Event processing flow

When events occur in the database, the Processing engine handles them through a consistent sequence:

1. A trigger activates when a specific event occurs
2. The engine loads the associated plugin (Python code)
3. The plugin receives relevant context and data
4. The code can process received data, query data, call third party tools, and write data to the database (for example, downsampled data) and caches.
5. Execution completes and the engine returns to waiting state

Key components

Trigger system

Triggers connect database events to Python code execution based on specific conditions:

• Data write triggers: Execute on WAL flush events, when data is written to the object store, for a specific table or all tables in a database
• Scheduled triggers: Run at intervals or according to cron expressions
• HTTP triggers: Respond to HTTP requests to custom endpoints

Plugin registry

The registry manages all Python code available to the Processing engine:

• Indexes plugins by filename and location
• Tracks which plugins are used by which triggers
• Manages plugin versioning and dependencies

Memory management

The Processing engine implements specialized memory handling to ensure stability and performance:

• Execution isolation: Each plugin runs in its own context
• Cache system: Maintains state between executions
• Resource limits: Controls memory usage and execution time

Performance characteristics

The Processing engine is designed for high-performance operation with minimal overhead:

• Low latency: Sub-millisecond trigger activation
• Efficient access: Direct database access without network overhead
• Controlled resources: Configurable memory and CPU limits
• Execution policies: Synchronous or asynchronous processing options

Reliability features

The Processing engine includes multiple features to ensure consistent and dependable execution:

• Error handling: Configurable behaviors (log, retry, or disable)
• Execution tracking: Monitoring of plugin performance
• State persistence: Cache management across server restarts

Extension capabilities

Extend and customize the Processing engine through several built-in mechanisms:

• Package management: Custom Python dependency installation
• Plugin distribution: Git-based plugin sharing
• Shared API: Consistent interface for database operations

For a step-by-step guide to setting up and using the Processing engine, see the Getting started with plugins documentation.