I have added e2e tests for agent dashboard page
It includes, tests like
- dashboard page loads successfully
- submit agent button works correctly
- agent table displays data correctly
- agent table actions work correctly
I’ve also updated the e2e test script to include some static agent
submissions, so I can test if it loads on the frontend.
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] All tests are working perfectly locally
<img width="469" height="177" alt="Screenshot 2025-08-08 at 12 13 42 PM"
src="https://github.com/user-attachments/assets/5e37afc3-c151-476a-84de-0a06f44a0722"
/>
In this PR, I’ve added library page tests.
### Changes
I’ve added 9 tests: 8 for normal flows and 1 for checking edge cases.
Test names are something like:
- Library navigation is accessible from the navbar.
- The library page loads successfully.
- Agents are visible, and cards work correctly.
- Pagination works correctly.
- Sorting works correctly.
- Searching works correctly.
- Pagination while searching works correctly.
- Uploading an agent works correctly.
- Edge case: Search edge cases and error handling behave correctly.
Other than that, I’ve added a new utility that uses the build page to
help us create users at the start, which we could use to test the
library page.
- All tests are passing locally
<img width="514" height="465" alt="Screenshot 2025-07-12 at 11 13 41 AM"
src="https://github.com/user-attachments/assets/7a46c437-7db5-458b-b99a-4fa0d479866f"
/>
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] All library tests are working locally and on CI perfectly.
This PR updates the existing E2E test data script to support the
creation of featured creators and featured agents. Previously, these
entities were not included, which limited our ability to fully test
certain flows during Playwright E2E testing.
### Changes
- Added logic to create featured creators
- Added logic to create featured agents
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] All tests are passing locally after updating the data script.
Need: The Gmail integration had several parsing issues that were causing
data loss and
workflow incompatibilities:
1. Email recipient parsing only captured the first recipient, losing
CC/BCC and multiple TO
recipients
2. Email body parsing was inconsistent between blocks, sometimes showing
"This email does
not contain a readable body" for valid emails
3. Type mismatches between blocks caused serialization issues when
connecting them in
workflows (lists being converted to string representations like
"[\"email@example.com\"]")
# Changes 🏗️
1. Enhanced Email Model:
- Added cc and bcc fields to capture all recipients
- Changed to field from string to list for consistency
- Now captures all recipients instead of just the first one
2. Improved Email Parsing:
- Updated GmailReadBlock and GmailGetThreadBlock to parse all recipients
using
getaddresses()
- Unified email body parsing logic across blocks with robust multipart
handling
- Added support for HTML to plain text conversion
- Fixed handling of emails with attachments as body content
3. Fixed Block Compatibility:
- Updated GmailSendBlock and GmailCreateDraftBlock to accept lists for
recipient fields
- Added validation to ensure at least one recipient is provided
- All blocks now consistently use lists for recipient fields, preventing
serialization
issues
4. Updated Test Data:
- Modified all test inputs/outputs to use the new list format for
recipients
- Ensures tests reflect the new data structure
# Checklist 📋
For code changes:
- I have clearly listed my changes in the PR description
- I have made a test plan
- I have tested my changes according to the test plan:
- Run existing Gmail block unit tests with poetry run test
- Create a workflow that reads emails with multiple recipients and
verify all TO, CC, BCC
recipients are captured
- Test email body parsing with plain text, HTML, and multipart emails
- Connect GmailReadBlock → GmailSendBlock in a workflow and verify
recipient data flows
correctly
- Connect GmailReplyBlock → GmailSendBlock and verify no serialization
errors occur
- Test sending emails with multiple recipients via GmailSendBlock
- Test creating drafts with multiple recipients via
GmailCreateDraftBlock
- Verify backwards compatibility by testing with single recipient
strings (should now
require lists)
- Create from scratch and execute an agent with at least 3 blocks
- Import an agent from file upload, and confirm it executes correctly
- Upload agent to marketplace
- Import an agent from marketplace and confirm it executes correctly
- Edit an agent from monitor, and confirm it executes correctly
# Breaking Change Note: The to field in GmailSendBlock and
GmailCreateDraftBlock now requires
a list instead of accepting both string and list. Existing workflows
using strings will
need to be updated to use lists (e.g., ["email@example.com"] instead of
"email@example.com").
---------
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
To allow for a simpler dev experience, the new SDK now auto discovers
providers and registers them. However, the OAuth system was still
requiring these credentials to be hardcoded in the settings object.
This PR changes that to verify the env var is present during
registration and then allows the OAuth system to load them from the env.
### Changes 🏗️
- **OAuth Registration**: Modified `ProviderBuilder.with_oauth(..)` to
check OAuth env vars exist during registration
- **OAuth Loading**: Updated OAuth system to load credentials from env
vars if not using secrets
- **Block Filtering**: Added `is_block_auth_configured()` function to
check if a block has valid authorization options configured at runtime
- **Test Updates**: Fixed failing SDK registry tests to properly mock
environment variables for OAuth registration
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Verified OAuth system checks that env vars exist during provider
registration
- [x] Confirmed OAuth system can use env vars directly without requiring
hardcoded secrets
- [x] Tested that blocks with unconfigured OAuth providers are filtered
out
- [x] All SDK registry tests pass with proper env var mocking
#### For configuration changes:
- [x] `.env.example` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)
- OAuth providers now require their client ID and secret env vars to be
set for registration
- No changes required to `.env.example` or `docker-compose.yml`
---------
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
## Overview
This PR adds comprehensive Airtable integration to the AutoGPT platform,
enabling users to seamlessly connect their Airtable bases with AutoGPT
workflows for powerful no-code automation capabilities.
## Why Airtable Integration?
Airtable is one of the most popular no-code databases used by teams for
project management, CRMs, inventory tracking, and countless other use
cases. This integration brings significant value:
- **Data Automation**: Automate data entry, updates, and synchronization
between Airtable and other services
- **Workflow Triggers**: React to changes in Airtable bases with
webhook-based triggers
- **Schema Management**: Programmatically create and manage Airtable
table structures
- **Bulk Operations**: Efficiently process large amounts of data with
batch create/update/delete operations
## Key Features
### 🔌 Webhook Trigger
- **AirtableWebhookTriggerBlock**: Listens for changes in Airtable bases
and triggers workflows
- Supports filtering by table, view, and specific fields
- Includes webhook signature validation for security
### 📊 Record Operations
- **AirtableCreateRecordsBlock**: Create single or multiple records (up
to 10 at once)
- **AirtableUpdateRecordsBlock**: Update existing records with upsert
support
- **AirtableDeleteRecordsBlock**: Delete single or multiple records
- **AirtableGetRecordBlock**: Retrieve specific record details
- **AirtableListRecordsBlock**: Query records with filtering, sorting,
and pagination
### 🏗️ Schema Management
- **AirtableCreateTableBlock**: Create new tables with custom field
definitions
- **AirtableUpdateTableBlock**: Modify table properties
- **AirtableAddFieldBlock**: Add new fields to existing tables
- **AirtableUpdateFieldBlock**: Update field properties
## Technical Implementation Details
### Authentication
- Supports both API Key and OAuth authentication methods
- OAuth implementation includes proper token refresh handling
- Credentials are securely managed through the platform's credential
system
### Webhook Security
- Added `credentials` parameter to WebhooksManager interface for proper
signature validation
- HMAC-SHA256 signature verification ensures webhook authenticity
- Webhook cursor tracking prevents duplicate event processing
### API Integration
- Comprehensive API client (`_api.py`) with full type safety
- Proper error handling and response validation
- Support for all Airtable field types and operations
## Changes 🏗️
### Added Blocks:
- AirtableWebhookTriggerBlock
- AirtableCreateRecordsBlock
- AirtableDeleteRecordsBlock
- AirtableGetRecordBlock
- AirtableListRecordsBlock
- AirtableUpdateRecordsBlock
- AirtableAddFieldBlock
- AirtableCreateTableBlock
- AirtableUpdateFieldBlock
- AirtableUpdateTableBlock
### Modified Files:
- Updated WebhooksManager interface to support credential-based
validation
- Modified all webhook handlers to support the new interface
## Test Plan 📋
### Manual Testing Performed:
1. **Authentication Testing**
- ✅ Verified API key authentication works correctly
- ✅ Tested OAuth flow including token refresh
- ✅ Confirmed credentials are properly encrypted and stored
2. **Webhook Testing**
- ✅ Created webhook subscriptions for different table events
- ✅ Verified signature validation prevents unauthorized requests
- ✅ Tested cursor tracking to ensure no duplicate events
- ✅ Confirmed webhook cleanup on block deletion
3. **Record Operations Testing**
- ✅ Created single and batch records with various field types
- ✅ Updated records with and without upsert functionality
- ✅ Listed records with filtering, sorting, and pagination
- ✅ Deleted single and multiple records
- ✅ Retrieved individual record details
4. **Schema Management Testing**
- ✅ Created tables with multiple field types
- ✅ Added fields to existing tables
- ✅ Updated table and field properties
- ✅ Verified proper error handling for invalid field types
5. **Error Handling Testing**
- ✅ Tested with invalid credentials
- ✅ Verified proper error messages for API limits
- ✅ Confirmed graceful handling of network errors
### Security Considerations 🔒
1. **API Key Management**
- API keys are stored encrypted in the credential system
- Keys are never logged or exposed in error messages
- Credentials are passed securely through the execution context
2. **Webhook Security**
- HMAC-SHA256 signature validation on all incoming webhooks
- Webhook URLs use secure ingress endpoints
- Proper cleanup of webhooks when blocks are deleted
3. **OAuth Security**
- OAuth tokens are securely stored and refreshed
- Scopes are limited to necessary permissions
- Token refresh happens automatically before expiration
## Configuration Requirements
No additional environment variables or configuration changes are
required. The integration uses the existing credential management
system.
## Checklist 📋
#### For code changes:
- [x] I have read the [contributing
instructions](https://github.com/Significant-Gravitas/AutoGPT/blob/master/.github/CONTRIBUTING.md)
- [x] Confirmed that `make lint` passes
- [x] Confirmed that `make test` passes
- [x] Updated documentation where needed
- [x] Added/updated tests for new functionality
- [x] Manually tested all blocks with real Airtable bases
- [x] Verified backwards compatibility of webhook interface changes
#### Security:
- [x] No hard-coded secrets or sensitive information
- [x] Proper input validation on all user inputs
- [x] Secure credential handling throughout
## Changes 🏗️
- Make docker + deps cache actually work on the FE CI
- Run the E2E test data script before Playwright
## Checklist 📋
### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] CI is faster in repeated runs ( _uses cache_ )
- [x] Test data script runs successfully
### For configuration changes:
None
Currently, we only create a library entry of the top-most graph when
importing the graph from an exported file.
This can cause some complications, as there is no way to remove the
library entry of it.
### Changes 🏗️
Create the library entry for all the subgraphs during the import
process.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] Export an agent with subgraphs and import it back.
### Changes
- Introduced a new script to generate test data for end-to-end (E2E)
tests using API functions, ensuring compatibility with future model
changes.
- The script creates test users, agent blocks, graphs, profiles, library
agents, presets, API keys, and store submissions.
- Utilizes external services for image and video URLs, and includes
error handling for data creation processes.
- Provides a summary of created data upon completion, enhancing the
testing framework for the AutoGPT platform.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Test scripts are working perfectly and not breaking anything. Data
is also correctly visible in the database.
This PR adds Excel file support to CSV processing and enhances text file
reading capabilities.
### Changes 🏗️
**ReadSpreadsheetBlock (formerly ReadCsvBlock):**
- Renamed `ReadCsvBlock` to `ReadSpreadsheetBlock` for better clarity
- Added Excel file support (.xlsx, .xls) with automatic conversion to
CSV using pandas
- Enhanced parameter `file_in` to `file_input` for consistency
- Excel files are automatically detected by extension and converted to
CSV format
- Maintains all existing CSV processing functionality (delimiters,
headers, etc.)
- Graceful error handling when pandas library is not available
**FileReadBlock:**
- Enhanced text file reading with advanced chunking capabilities
- Added parameters: `skip_size`, `skip_rows`, `row_limit`, `size_limit`,
`delimiter`
- Supports both character-based and row-based processing
- Chunked output for large files based on size limits
- Proper file handling with UTF-8 and latin-1 encoding fallbacks
- Uses `store_media_file` for secure file processing (URLs, data URIs,
local paths)
- Fixed test input to use data URI instead of non-existent file
**General Improvements:**
- Consistent parameter naming across blocks (`file_input`)
- Enhanced error handling and validation
- Comprehensive test coverage
- All existing functionality preserved
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Both ReadSpreadsheetBlock and FileReadBlock instantiate correctly
- [x] ReadSpreadsheetBlock processes CSV data with existing
functionality
- [x] FileReadBlock reads text files with data URI input
- [x] All block tests pass (457 passed, 83 skipped)
- [x] No linting errors in modified files
- [x] Excel support gracefully handles missing pandas dependency
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
*Note: No configuration changes required for this PR.*
<!-- Clearly explain the need for these changes: -->
The `GmailReadBlock._get_email_body()` method was only inspecting the
top-level payload and a single `text/plain` part, causing it to return
the fallback string "This email does not contain a text body." for most
Gmail messages. This occurred because Gmail messages are typically
wrapped in `multipart/alternative` or other multipart containers, which
the original implementation couldn't handle.
This critical issue made the Gmail integration unusable for reading
email body content, as virtually every real Gmail message uses multipart
MIME structures.
<!-- Concisely describe all of the changes made in this pull request:
-->
### Changes
#### Core Implementation:
- **Replaced simple `_get_email_body()` with recursive multipart
parser** that can walk through nested MIME structures
- **Added `_walk_for_body()` method** for recursive traversal of email
parts with depth limiting (max 10 levels)
- **Implemented safe base64 decoding** with automatic padding correction
in `_decode_base64()`
- **Added attachment body support** via `_download_attachment_body()`
for emails where body content is stored as attachments
#### Email Format Support:
- **HTML to text conversion** using `html2text` library for HTML-only
emails
- **Multipart/alternative handling** with preference for `text/plain`
over `text/html`
- **Nested multipart structure support** (e.g., `multipart/mixed`
containing `multipart/alternative`)
- **Single-part email support** (maintains backward compatibility)
#### Dependencies & Testing:
- **Added `html2text = "^2024.2.26"`** to `pyproject.toml` for HTML
conversion
- **Created comprehensive unit tests** in `test/blocks/test_gmail.py`
covering all email types and edge cases
- **Added error handling and graceful fallbacks** for malformed data and
missing dependencies
#### Security & Performance:
- **Recursion depth limiting** prevents infinite loops on malformed
email structures
- **Exception handling** ensures graceful degradation when API calls
fail
- **Efficient tree traversal** with early returns for better performance
### Checklist
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<details>
<summary>Test Plan</summary>
- **Single-part text/plain emails** - Verified correct extraction of
plain text content
- **Multipart/alternative emails** - Tested preference for plain text
over HTML when both available
- **HTML-only emails** - Confirmed HTML to text conversion works
correctly
- **Nested multipart structures** - Tested deeply nested
`multipart/mixed` containing `multipart/alternative`
- **Attachment-based body content** - Verified downloading and decoding
of body stored as attachments
- **Base64 padding edge cases** - Tested malformed base64 data with
missing padding
- **Recursion depth limits** - Confirmed protection against infinite
recursion
- **Error handling scenarios** - Tested graceful fallbacks for API
failures and missing dependencies
- **Backward compatibility** - Ensured existing functionality remains
unchanged for edge cases
- **Integration testing** - Ran standalone verification script with 100%
test pass rate
</details>
#### For configuration changes:
- [x] `.env.example` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Configuration Changes</summary>
- Added `html2text` dependency to `pyproject.toml` - no environment or
infrastructure changes required
- No changes to ports, services, secrets, or databases
- Fully backward compatible with existing Gmail API configuration
</details>
---------
Co-authored-by: Toran Bruce Richards <toran.richards@gmail.com>
Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co>
### Summary
Performance optimization for the platform's store and creator
functionality by adding targeted database indexes and implementing
materialized views to reduce query execution time.
### Changes 🏗️
**Database Performance Optimizations:**
- Added strategic database indexes for `StoreListing`,
`StoreListingVersion`, `StoreListingReview`, `AgentGraphExecution`, and
`Profile` tables
- Implemented materialized views (`mv_agent_run_counts`,
`mv_review_stats`) to cache expensive aggregation queries
- Optimized `StoreAgent` and `Creator` views to use materialized views
and improved query patterns
- Added automated refresh function with 15-minute scheduling for
materialized views (when pg_cron extension is available)
**Key Performance Improvements:**
- Filtered indexes on approved store listings to speed up marketplace
queries
- GIN index on categories for faster category-based searches
- Composite indexes for common query patterns (e.g., listing + version
lookups)
- Pre-computed agent run counts and review statistics to eliminate
expensive aggregations
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Verified migration runs successfully without errors
- [x] Confirmed materialized views are created and populated correctly
- [x] Tested StoreAgent and Creator view queries return expected results
- [x] Validated automatic refresh function works properly
- [x] Confirmed rollback migration successfully removes all changes
#### For configuration changes:
- [x] `.env.example` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)
**Note:** No configuration changes were required as this is purely a
database schema optimization.
## Block Development SDK - Simplifying Block Creation
### Problem
Currently, creating a new block requires manual updates to **5+ files**
scattered across the codebase:
- `backend/data/block_cost_config.py` - Manually add block costs
- `backend/integrations/credentials_store.py` - Add default credentials
- `backend/integrations/providers.py` - Register new providers
- `backend/integrations/oauth/__init__.py` - Register OAuth handlers
- `backend/integrations/webhooks/__init__.py` - Register webhook
managers
This creates significant friction for developers, increases the chance
of configuration errors, and makes the platform difficult to scale.
### Solution
This PR introduces a **Block Development SDK** that provides:
- Single import for all block development needs: `from backend.sdk
import *`
- Automatic registration of all block configurations
- Zero external file modifications required
- Provider-based configuration with inheritance
### Changes 🏗️
#### 1. **New SDK Module** (`backend/sdk/`)
- **`__init__.py`**: Unified exports of 68+ block development components
- **`registry.py`**: Central auto-registration system for all block
configurations
- **`builder.py`**: `ProviderBuilder` class for fluent provider
configuration
- **`provider.py`**: Provider configuration management
- **`cost_integration.py`**: Automatic cost application system
#### 2. **Provider Builder Pattern**
```python
# Configure once, use everywhere
my_provider = (
ProviderBuilder("my-service")
.with_api_key("MY_SERVICE_API_KEY", "My Service API Key")
.with_base_cost(5, BlockCostType.RUN)
.build()
)
```
#### 3. **Automatic Cost System**
- Provider base costs automatically applied to all blocks using that
provider
- Override with `@cost` decorator for block-specific pricing
- Tiered pricing support with cost filters
#### 4. **Dynamic Provider Support**
- Modified `ProviderName` enum to accept any string via `_missing_`
method
- No more manual enum updates for new providers
#### 5. **Application Integration**
- Added `sync_all_provider_costs()` to `initialize_blocks()` for
automatic cost registration
- Maintains full backward compatibility with existing blocks
#### 6. **Comprehensive Examples** (`backend/blocks/examples/`)
- `simple_example_block.py` - Basic block structure
- `example_sdk_block.py` - Provider with credentials
- `cost_example_block.py` - Various cost patterns
- `advanced_provider_example.py` - Custom API clients
- `example_webhook_sdk_block.py` - Webhook configuration
#### 7. **Extensive Testing**
- 6 new test modules with 30+ test cases
- Integration tests for all SDK features
- Cost calculation verification
- Provider registration tests
### Before vs After
**Before SDK:**
```python
# 1. Multiple complex imports
from backend.data.block import Block, BlockCategory, BlockOutput
from backend.data.model import SchemaField, CredentialsField
# ... many more imports
# 2. Update block_cost_config.py
BLOCK_COSTS[MyBlock] = [BlockCost(...)]
# 3. Update credentials_store.py
DEFAULT_CREDENTIALS.append(...)
# 4. Update providers.py enum
# 5. Update oauth/__init__.py
# 6. Update webhooks/__init__.py
```
**After SDK:**
```python
from backend.sdk import *
# Everything configured in one place
my_provider = (
ProviderBuilder("my-service")
.with_api_key("MY_API_KEY", "My API Key")
.with_base_cost(10, BlockCostType.RUN)
.build()
)
class MyBlock(Block):
class Input(BlockSchema):
credentials: CredentialsMetaInput = my_provider.credentials_field()
data: String = SchemaField(description="Input data")
class Output(BlockSchema):
result: String = SchemaField(description="Result")
# That's it\! No external files to modify
```
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Created new blocks using SDK pattern with provider configuration
- [x] Verified automatic cost registration for provider-based blocks
- [x] Tested cost override with @cost decorator
- [x] Confirmed custom providers work without enum modifications
- [x] Verified all example blocks execute correctly
- [x] Tested backward compatibility with existing blocks
- [x] Ran all SDK tests (30+ tests, all passing)
- [x] Created blocks with credentials and verified authentication
- [x] Tested webhook block configuration
- [x] Verified application startup with auto-registration
#### For configuration changes:
- [x] `.env.example` is updated or already compatible with my changes
(no changes needed)
- [x] `docker-compose.yml` is updated or already compatible with my
changes (no changes needed)
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)
### Impact
- **Developer Experience**: Block creation time reduced from hours to
minutes
- **Maintainability**: All block configuration in one place
- **Scalability**: Support hundreds of blocks without enum updates
- **Type Safety**: Full IDE support with proper type hints
- **Testing**: Easier to test blocks in isolation
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Abhimanyu Yadav <122007096+Abhi1992002@users.noreply.github.com>
Currently, we don't have a secure way to pass Authorization headers when
calling the `SendWebRequestBlock`.
This hinders the integration of third-party applications that do not yet
have native block support.
### Changes 🏗️
Add Host-scoped credentials support for the newly introduced
SendAuthenticatedWebRequestBlock.
<img width="1000" alt="image"
src="https://github.com/user-attachments/assets/0d3d577a-2b9b-4f0f-9377-0e00a069ba37"
/>
<img width="1000" alt="image"
src="https://github.com/user-attachments/assets/a59b9f16-c89c-453d-a628-1df0dfd60fb5"
/>
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] Uses `https://api.openai.com/v1/images/edits` through
SendWebRequestBlock by passing the api-key through host-scoped
credentials.
An anti-virus file scan step is added to each file upload step on the
platform before the file is sent to cloud storage or local machine
storage.
### Changes 🏗️
* Added ClamAV service
* Added AV file scan on each upload step
* Added tests & documentation
* Make the step mandatory even on local development
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] Tried using FileUploadBlock & AgentFileInputBlock
## Changes 🏗️
The test data script is not working locally, this should fix it 🤞🏽
- Fixed `agentId` → `agentGraphId` field references in preset matching
logic
- Fixed `agentId` → `agentGraphId` field references in store listing
graph lookup
- Added graph uniqueness logic to prevent duplicate library agents per
user
- Improved data consistency by ensuring proper foreign key relationships
## Checklist 📋
### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Verified script runs without database schema errors
- [x] Confirmed foreign key relationships are properly maintained
- [x] Tested that library agents use unique graphs per user
- [x] Validated preset matching uses correct field references
This pull request adds support for setting up (webhook-)triggered agents
in the Library. It contains changes throughout the entire stack to make
everything work in the various phases of a triggered agent's lifecycle:
setup, execution, updates, deletion.
Setting up agents with webhook triggers was previously only possible in
the Builder, limiting their use to the agent's creator only. To make it
work in the Library, this change uses the previously introduced
`AgentPreset` to store information on, instead of on the graph's nodes
to which only a graph's creator has access.
- Initial ticket: #10111
- Builds on #9786
### Changes 🏗️
Frontend:
- Amend the Library's `AgentRunDraftView` to handle creating and editing
Presets
- Add `hideIfSingleCredentialAvailable` parameter to `CredentialsInput`
- Add multi-select support to `TypeBasedInput`
- Add Presets section to `AgentRunsSelectorList`
- Amend `AgentRunSummaryCard` for use for Presets
- Add `AgentStatusChip` to display general agent status (for now: Active
/ Inactive / Error)
- Add Preset loading logic and create/update/delete handlers logic to
`AgentRunsPage`
- Rename `IconClose` to `IconCross`
API:
- Add `LibraryAgent` properties `has_external_trigger`,
`trigger_setup_info`, `credentials_input_schema`
- Add `POST /library/agents/{library_agent_id}/setup_trigger` endpoint
- Remove redundant parameters from `POST
/library/presets/{preset_id}/execute` endpoint
Backend:
- Add `POST /library/agents/{library_agent_id}/setup_trigger` endpoint
- Extract non-node-related logic from `on_node_activate` into
`setup_webhook_for_block`
- Add webhook-related logic to `update_preset` and `delete_preset`
endpoints
- Amend webhook infrastructure to work with AgentPresets
- Add preset trigger support to webhook ingress endpoint
- Amend executor stack to work with passed-in node input
(`nodes_input_masks`, generalized from `node_credentials_input_map`)
- Amend graph validation to work with passed-in node input
- Add `AgentPreset`->`IntegrationWebhook` relation
- Add `WebhookWithRelations` model
- Change behavior of `BaseWebhooksManager.get_manual_webhook(..)` to
avoid unnecessary changes of the webhook URL: ignore `events` to find
matching webhook, and update `events` if necessary.
- Fix & improve `AgentPreset` API, models, and DB logic
- Add `isDeleted` filter to get/list queries
- Add `user_id` attribute to `LibraryAgentPreset` model
- Add separate `credentials` property to `LibraryAgentPreset` model
- Fix `library_db.update_preset(..)` replacement of existing
`InputPresets`
- Make `library_db.update_preset(..)` more usage-friendly with separate
parameters for updateable properties
- Add `user_id` checks to various DB functions
- Fix error handling in various endpoints
- Fix cache race condition on `load_webhook_managers()`
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- Test existing functionality
- [x] Auto-setup and -teardown of webhooks on save in the builder still
works
- [x] Running an agent normally from the Library still works
- Test new functionality
- [x] Setting up a trigger in the Library
- [x] Updating a trigger in the Library
- [x] Disabling and re-enabling a trigger in the Library
- [x] Deleting a trigger in the Library
- [x] Triggers set up in the Library result in a new run when the
webhook receives a payload
This change introduced async execution for blocks and the execution
engine. Paralellism will be achieved through a single process
asynchronous execution instead of process concurrency.
### Changes 🏗️
* Support async execution for the graph executor
* Removed process creation for node execution
* Update all blocks to support async executions
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] Manual graph executions, tested many of the impacted blocks.
This pull request introduces a comprehensive backend testing guide and
adds new tests for analytics logging and various API endpoints, focusing
on snapshot testing. It also includes corresponding snapshot files for
these tests. Below are the most significant changes:
### Documentation Updates:
* Added a detailed `TESTING.md` file to the backend, providing a guide
for running tests, snapshot testing, writing API route tests, and best
practices. It includes examples for mocking, fixtures, and CI/CD
integration.
### Analytics Logging Tests:
* Implemented tests for logging raw metrics and analytics in
`analytics_test.py`, covering success scenarios, various input values,
invalid requests, and complex nested data. These tests utilize snapshot
testing for response validation.
* Added snapshot files for analytics logging tests, including responses
for success cases, various metric values, and complex analytics data.
[[1]](diffhunk://#diff-654bc5aa1951008ec5c110a702279ef58709ee455ba049b9fa825fa60f7e3869R1-R3)
[[2]](diffhunk://#diff-e0a434b107abc71aeffb7d7989dbfd8f466b5e53f8dea25a87937ec1b885b122R1-R3)
[[3]](diffhunk://#diff-dd0bc0b72264de1a0c0d3bd0c54ad656061317f425e4de461018ca51a19171a0R1-R3)
[[4]](diffhunk://#diff-63af007073db553d04988544af46930458a768544cabd08412265e0818320d11R1-R30)
### Snapshot Files for API Endpoints:
* Added snapshot files for various API endpoint tests, such as:
- Graph-related operations (`graphs_get_single_response`,
`graphs_get_all_response`, `blocks_get_all_response`).
[[1]](diffhunk://#diff-b25dba271606530cfa428c00073d7e016184a7bb22166148ab1726b3e113dda8R1-R29)
[[2]](diffhunk://#diff-1054e58ec3094715660f55bfba1676d65b6833a81a91a08e90ad57922444d056R1-R31)
[[3]](diffhunk://#diff-cfd403ab6f3efc89188acaf993d85e6f792108d1740c7e7149eb05efb73d918dR1-R14)
- User-related operations (`auth_get_or_create_user_response`,
`auth_update_email_response`).
[[1]](diffhunk://#diff-49e65ab1eb6af4d0163a6c54ed10be621ce7336b2ab5d47d47679bfaefdb7059R1-R5)
[[2]](diffhunk://#diff-ac1216f96878bd4356454c317473654d5d5c7c180125663b80b0b45aa5ab52cbR1-R3)
- Credit-related operations (`credits_get_balance_response`,
`credits_get_auto_top_up_response`, `credits_top_up_request_response`).
[[1]](diffhunk://#diff-189488f8da5be74d80ac3fb7f84f1039a408573184293e9ba2e321d535c57cddR1-R3)
[[2]](diffhunk://#diff-ba3c4a6853793cbed24030cdccedf966d71913451ef8eb4b2c4f426ef18ed87aR1-R4)
[[3]](diffhunk://#diff-43d7daa0c82070a9b6aee88a774add8e87533e630bbccbac5a838b7a7ae56a75R1-R3)
- Graph execution and deletion (`blocks_execute_response`,
`graphs_delete_response`).
[[1]](diffhunk://#diff-a2ade7d646ad85a2801e7ff39799a925a612548a1cdd0ed99b44dd870d1465b5R1-R12)
[[2]](diffhunk://#diff-c0d1cd0a8499ee175ce3007c3a87ba5f3235ce02d38ce837560b36a44fdc4a22R1-R3)##
Summary
- add pytest-snapshot to backend dev requirements
- snapshot server route response JSONs
- mention how to update stored snapshots
## Testing
- `poetry run format`
- `poetry run test`
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] run poetry run test
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co>
- Part of #9307
- ❗ Blocks #9541
### Changes 🏗️
Backend:
- Fix+improve `GET /library/presets` (`list_presets`) endpoint
- Fix pagination
- Add `graph_id` filter parameter
- Allow partial preset updates: `PUT /presets/{preset_id}` -> `PATCH
/presets/{preset_id}`
- Allow creating preset from graph execution through `POST /presets`
- Clean up models & DB functions
- Split `upsert_preset` into `create_preset` + `update_preset`
- Add `LibraryAgentPresetUpdatable`
- Replace `CreateLibraryAgentPresetRequest` with
`LibraryAgentPresetCreatable`
- Use `LibraryAgentPresetCreatable` as base class for
`LibraryAgentPreset`
- Remove redundant `set_is_deleted_for_library_agent(..)`
- Improve log statements
- Improve DB statements (e.g. by using unique keys where possible)
Frontend:
- Add timestamp parsing logic to library agent preset endpoints
- Brand `LibraryAgentPreset.id` + references
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] CI green
- Since these changes don't affect existing front-end functionality, no
additional testing is needed.
Suppose we have pint with list[list[int]] type, and we want directly
insert the a new value inside the first index of the first list e.g:
list[0][0] = X through a dynamic pin, this will be translated into
list_$_0_$_0, and the system does not currently support this.
### Changes 🏗️
Add support for nested dynamic pins for list, object, and dict.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] lots of unit tests
- [x] Tried inserting the value directly on the `value` nested field on
Google Sheets Write block.
<img width="371" alt="image"
src="https://github.com/user-attachments/assets/0a5e7213-b0e0-4fce-9e89-b39f7a583582"
/>
- Resolves#9987
### Changes 🏗️
- Split `pin_url(..)` out of `validate_url(..)` and call
`extra_url_validator` in between
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] GitHub Read Pull Request Block works with "Include PR Changes"
enabled
The Library Agent credentials UX (#9789) currently doesn't work for
sub-graphs.
### Changes 🏗️
- Include sub-graphs in generating `Graph.credentials_input_schema`
- Propagate `node_credentials_input_map` into `AgentExecutionBlock`
executions
- Fix: also apply `node_credentials_input_map` in `_enqueue_next_nodes`
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- Import a graph with sub-graphs that need credentials
- Run this agent from the Library
- [x] -> Should work
Using sync code in the async route often introduces a blocking
event-loop code that impacts stability.
The current RPC system only provides a synchronous client to call the
service endpoints.
The scope of this PR is to provide an entirely decoupled signature
between client and server, allowing the client can mix & match async &
sync options on the client code while not changing the async/sync nature
of the server.
### Changes 🏗️
* Add support for flexible async/sync RPC client.
* Migrate scheduler client to all-async client.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Scheduler route test.
- [x] Modified service_test.py
- [x] Run normal agent executions
<!-- Clearly explain the need for these changes: -->
We need a way to refund people who spend money on agents wihout making
manual db actions
### Changes 🏗️
- Adds a bunch for refunding users
- Adds reasons and admin id for actions
- Add admin to db manager
- Add UI for this for the admin panel
- Clean up pagination controls
<!-- Concisely describe all of the changes made in this pull request:
-->
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] Test by importing dev db as baseline
- [x] Add transactions on top for "refund", and make sure all existing
transactions work
---------
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
Currently the execution task is not properly distributed between
executors because we need to send the execution request to the execution
server.
The execution manager now accepts the execution request from the message
queue. Thus, we can remove the synchronous RPC system from this service,
let the system focus on executing the agent, and not spare any process
for the HTTP API interface.
This will also reduce the risk of the execution service being too busy
and not able to accept any add execution requests.
### Changes 🏗️
* Remove the RPC system in Agent Executor
* Allow the cancellation of the execution that is still waiting in the
queue (by avoiding it from being executed).
* Make a unified helper for adding an execution request to the system
and move other execution-related helper functions into
`executor/utils.py`.
* Remove non-db connections (redis / rabbitmq) in Database Manager and
let the client manage this by themselves.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Existing CI, some agent runs
Distilled from #9541 to reduce the scope of that PR.
- Part of #9307
- ❗ Blocks #9786
- ❗ Blocks #9541
### Changes 🏗️
- Fix `LibraryAgent` schema (for #9786)
- Fix relationships between `LibraryAgent`, `AgentGraph`, and
`AgentPreset`
- Impose uniqueness constraint on `LibraryAgent`
- Rename things that are called `agent` that actually refer to a
`graph`/`agentGraph`
- Fix singular/plural forms in DB schema
- Simplify reference names of closely related objects (e.g.
`AgentGraph.AgentGraphExecutions` -> `AgentGraph.Executions`)
- Eliminate use of `# type: ignore` in DB statements
- Add `typed` and `typed_cast` utilities to `backend.util.type`
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] CI static type checking (with all risky `# type: ignore` removed)
- [x] Check that column references in views are updated
This is a prerequisite infra change for
https://github.com/Significant-Gravitas/AutoGPT/issues/9714.
We will need a service where we can maintain our own client (db, redis,
rabbitmq, be it async/sync) and configure our own cadence of
initialization and cleanup.
While refactoring the service.py, an option to use Pyro as an RPC
protocol is also removed.
### Changes 🏗️
* Decouple resource initialization and cleanup from the parent
AppService logic.
* Removed Pyro.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] CI
This series of upgrades:
https://github.com/significant-gravitas/autogpt/pull/9727https://github.com/Significant-Gravitas/AutoGPT/pull/9728https://github.com/Significant-Gravitas/AutoGPT/pull/9560
Caused some code in the repo being deprecated, this PR addresses those.
### Changes 🏗️
Fix pydantic config, usage of field, usage of proper prisma
`CreateInput` type, pytest loop-scope.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] CI, manual test on running some agents.
---------
Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co>
- Resolves#9731
### Changes 🏗️
- feat: Add "Steps" showing `node_execution_count` to agent run view
- Add `GraphExecutionMeta.stats.node_exec_count` attribute
- feat(backend/executor): Send graph execution update after *every* node
execution (instead of only I/O node executions)
- Update graph execution stats after every node execution
- refactor: Move `GraphExecutionMeta` stats into sub-object
(`cost`, `duration`, `total_run_time` -> `stats.cost`, `stats.duration`,
`stats.node_exec_time`)
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- View an agent run with 1+ steps on `/library/agents/[id]`
- [x] "Info" section layout doesn't break
- [x] Number of steps is shown
- Initiate a new agent run
- [x] "Steps" increments in real time during execution
- Resolves#8782
### Changes 🏗️
- feat(frontend/library): Use WS subscription to get real-time execution
updates
- feat(backend/ws_api): Send `GraphExecutionUpdate` on all new agent I/O
- Include agent I/O in `GraphExecutionUpdate` (by subclassing
`GraphExecution`)
- Add `IO_BLOCK_IDs` to `.blocks.io`
- feat(backend/ws_api): Add `subscribe_graph_executions` method to
WebSocket API
- feat(backend): Withhold `GraphExecution.node_executions` from requests
by non-graph-owners
- Split `GraphExecutionWithNodes` off of `GraphExecution`
- Use `GraphExecution` as much as possible, as it's a much cheaper query
than `GraphExecutionWithNodes`
- refactor(frontend): Make `GraphExecution.node_executions` optional
- fix(frontend): Parse dates in responses of `/executions` and
`/graphs/{graph_id}/executions`
- refactor(frontend/library): Move sorting logic for agent runs list
from `AgentRunsPage` to `AgentRunsSelectorList`
- refactor(backend/ws_api): Clean up message handler implementations
- refactor(backend/tests): Use `.data.execution.get_graph_execution(..)`
directly instead of `AgentServer.test_get_graph_run_results(..)`
Out-of-scope changes:
- refactor(backend): Remove unnecessary query include from
`.data.graph.get_graph_metadata(..)`
Demo:
https://github.com/user-attachments/assets/8ea6225d-7334-49cb-a522-83f153d840da
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- Go to `/library/agents/[id]` for an agent with inputs and outputs
- Draft and run a new run
- [x] -> should appear in the list of runs at the top
- [x] -> should be selected as soon as the request finishes
- [x] -> new I/O should appear as it is generated
- [x] -> status should be updated in real-time (both in list and in
adjacent details view)
- Click "Run again"
- [x] -> should appear in the list of runs at the top
- [x] -> should be selected as soon as the request finishes
- [x] -> new I/O should appear as it is generated
- [x] -> status should be updated in real-time (both in list and in
adjacent details view)
- Click "Open in builder" under "Agent actions"; run the agent from the
builder
- [x] -> should work the same as before
- [x] -> node I/O should appear in real-time
- [x] -> node execution statuses should update in real-time
- Prep work for #8782
- Prep work for #8779
### Changes 🏗️
- refactor(platform): Differentiate graph/node execution events
- fix(platform): Subscribe to execution updates by `graph_exec_id`
instead of `graph_id`+`graph_version`
- refactor(backend): Move all execution related models and functions
from `.data.graph` to `.data.execution`
- refactor(backend): Reorganize & refactor `.data.execution`
- fix(libs): Remove `load_dotenv` in `.auth.config` to fix test config
issues
- dx: Bump version of `black` in pre-commit config to v24.10.0 to match
poetry.lock
- Other minor refactoring in both frontend and backend
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- Run an agent in the builder
- [x] -> works normally, node I/O is updated in real time
- Run an agent in the library
- [x] -> works normally
---------
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
The current block web requests utility has a logic to avoid the system
firing into blocklisted IPs.
However, the current logic is still prone to a few security issues:
* DNS rebinding attack: due to the lack of guarantee on the used IP not
being changed during the IP checking and firing step.
* Open redirect: due to the request sensitive request headers are still
being propagated throughout the web redirect.
### Changes 🏗️
* Uses IP pinning to request the web.
* Strip `Authorization`, `Proxy-Authorization`, `Cookie` upon web
redirects.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Test the web request block, add more tests with different
validation scenarios.
Blocks that are not defined in the block cost are pretty much free. The
lack of cost control makes it hard to control its quota. The scope of
this change is providing a way to charge any executions based on the
number of block being executed in real-time.
### Changes 🏗️
* Add execution charge logic based on the number of blocks executed,
controlled by these two configurations:
* `execution_cost_count_threshold`: We will charge the execution based
on the multiple of this number.
* `execution_cost_per_threshold`: The amount we are charging on its
threshold multiple.
* Make charging logic on the graph execution logic (as opposed to node
level) so it's being done serially and insufficient fund error is
guaranteed to stop the graph execution.
* Moved cost calculation logic into backend/executor/util.py
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Execute graph with configured threshold & cost and test the
balance being deducted on that.
- [x] Existing cost calculation is still being done without any issue.
- [x] Low balance stop the whole graph execution.
### Changes 🏗️
Added these types of input blocks:
* TextShort
* TextLong
* Number
* Date
* Time
* FileUpload
* Dropdown
* Toggle
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Test in respective block codes.
<!-- Clearly explain the need for these changes: -->
The store listing and submissions were previously just a best guess
without much implementation. This updates the database models and
queries and such to be based on discussion around what the process
should look like. It also adds and update the relevant routers for this
change
### Changes 🏗️
Store Listing
- change isApproved to hasApprovedVersion
- Move slug into store listing
- mark an active version in store listing
Store Version
- Move submissions into version
- make name optional
- have state transition timestamps for submitted and approved/rejected
- added a changes field
- added internal comments and clarified review comments field
SubmissionStatus
- Fixed DAFT to DRAFT
StoreListingSubmission
- Dropped table
Graph
- Used more modern format for the params for prisma -- no other changes
Added migrations for all the model movements
<!-- Concisely describe all of the changes made in this pull request:
-->
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] Use the store codepaths from the release testplan doc as the test
plan (claude I can't publish the testplan but I am a maintainer lol,
trust me here my guy, you're supposed to be lenient)
- [x] Check the db is used as appropriate following the rules
---------
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
Agents using Agent blocks should be seamlessly downloaded from the
marketplace to a file and imported from a file.
Requirements:
* A recursive export process that exports all the required agents to a
single file, no matter how many layers deep (taking care of potential
loops).
* An import process that expects and extracts several agents from a
single file into your library at once.
Considerations:
We need to ensure the reference IDs in the Agent Blocks match/are
updated to match the imported sub-agent ids to prevent broken
references.
### Changes 🏗️
* Add sub_graphs field on Graph model
* Improve graph creation query to support inserting graph + subgraphs in
batch
* Deprecate graph template & remove its column
* Update on marketplace download agent (unified the used method, with
more secure cleanup & proper ownership check).
* Fix failing test cases
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] Export graph with sub agents.
- [x] Import the exported graph with sub agents.
<!-- Clearly explain the need for these changes: -->
We don't want to spam the user with similar notification types so we
want to group them up over a timespan and handle that as a group of
notifications.
### Changes 🏗️
- Adds a batch queue
- Moves the ExecutionScheduleur to a generic Scheduler
- Makes the Agent run a batch operation
- Fixes various bugs in how we originally made the batch db models and
queries
<!-- Concisely describe all of the changes made in this pull request:
-->
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] Run 10 agents back to back
- [x] Notice how many emails you get
- [x] Wait a bit and you should after an hour (change the cron rule to
speed up testing this) you'll get an email and see all the batches in
your db are empty
---------
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
Now that the MVP for Smar Decision Block is available, we need to add
this info in the block output:
Messages (list) - The messages list sent to the LLM plus its generated
response as the latest assistant entry. This is a single list of
dictionaries in standard LLM API format).
### Changes 🏗️
* Add `conversations` output pin that populates the update
`conversation_history` input pin with the assistant response.
* Refactored `Smart Decision Block` to avoid downloading the whole graph
on each execution, remove code duplication, declutter data fetching.
* Minor UI issue on the smart decision block entry in the search bar.
### Checklist 📋
#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [ ] ...
<details>
<summary>Example test plan</summary>
- [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
- [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
- [ ] Edit an agent from monitor, and confirm it executes correctly
</details>
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Examples of configuration changes</summary>
- Changing ports
- Adding new services that need to communicate with each other
- Secrets or environment variable changes
- New or infrastructure changes such as databases
</details>
Co-authored-by: Swifty <craigswift13@gmail.com>
Originally we did not allow Blocks to be used as tools due to the
limitations of communicating the correct tool function signatures.
It has however, been decided to allow them to be used knowing that there
are limitations with them.
### Changes 🏗️
- Added ability to execute blocks as tools
### Checklist 📋
<img width="613" alt="Screenshot 2025-02-25 at 12 49 26"
src="https://github.com/user-attachments/assets/e614f56d-2bdc-46c9-8c2c-e56f80343bde"
/>
- create an agent with an SDM block and a block as a tool
- run agent and make sure the block can be called as a tool
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
---------
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
- Resolves#8774
- Resolves#8775
- Includes back-end work for #9168
- Partially implements #8776
- Partially implements #8777
### Changes 🏗️
- Add `/library` page
- Change target of "Library" navigation link from `/monitoring` to
`/library`
- Move `/agents/[id]` page to `/library/agents/[id]`
- Set application background color to `bg-neutral-50`
- Redirect to new library agent's "runs" page (`/library/agents/[id]`)
after adding from marketplace
Further (technical) frontend changes:
- Add types and client methods for all library API endpoints
- Added `primary` variant to `agptui/Button` component
Backend changes:
- Add functionality to library backend
- Aggregate agent status
- Image generation for use in library view
- Add `LibraryAgent.imageUrl` column to DB schema
- Sorting & pagination
- Explicit relation between library agents and their graph's creator
- Refactor & update API endpoints for DX
- Other minor refactoring
- Add missing but required `MEDIA_GCS_BUCKET_NAME` to `.env.example`
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- `/library`
- [x] Create agent from scratch -> should show up on `/library`
- [x] Add agent to library from marketplace -> should show up on
`/library`
- [x] Click on agent in `/library` -> should go to
`/library/agents/[id]`
- [x] Test sorting & pagination functionality
- `/library/agents/[id]`
- [x] Fill out inputs and click "Run" -> should run like normally
- [x] Select completed agent run -> should show all inputs & outputs
- [x] Click "run again" on a completed agent run -> should run
successfully with same input
- [x] `/monitoring` should still work the same as before
---------
Co-authored-by: abhi1992002 <abhimanyu1992002@gmail.com>
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co>
## Task
The SmartDecisionMakerBlock is a specialized block in a graph-based
system that leverages a language model (LLM) to make intelligent
decisions about which tools or functions to invoke based on a
user-provided prompt. It is designed to process input data, interact
with a language model, and dynamically determine the appropriate tools
to call from a set of available options, making it a powerful component
for AI-driven workflows.
## How It Works in Practice
- **Scenario:** Imagine a workflow where a user inputs, "Send an email
to John about the meeting." The SmartDecisionMakerBlock is connected to
tools like send_email, schedule_meeting, and search_contacts.
- **Execution:**
1. The block receives the prompt and system instructions (e.g., "Choose
a function to call").
2.It identifies the available tools from the graph and constructs their
signatures (e.g., send_email(recipient, subject, body)).
3. The LLM analyzes the prompt and decides to call send_email with
arguments like recipient: "John", subject: "Meeting", body: "Let’s
discuss...".
4. The block yields these tool-specific outputs, which can be picked up
by downstream nodes to execute the email-sending action.
## Changes 🏗️
- Add the Smart Decision Maker (SDM) block.
- Break circular imports in integration code.
## Work in Progress
⚠️ **Important note this is a temporary UX for the system - UX will be
addressed in a future PR** ⚠️
### Current Status
I’m currently focused on the smart decision logic. The main additions in
the ongoing PR include:
- Defining function signatures for OpenAI function-calling schemas based
on node links and the linked blocks.
- Adding tests for function signature generation.
- Force all tool calls to be made via an agent. (Need to uncomment)
- Restrict each tool call entry to a single node.
- simplify the output emission process, to emit each parameter one at a
time.
- Change test to use agents and hardcode output how I think it should
work to test it does actually work
- Hook up openai, in a simplified way, to test the function calling
(mock for testing)
- Once all the above is working, use credentials system and build of
llm.py
### What’s Next
- Review Process
### Reviewers Phase 1
This PR is now ready for review, during the first phase of reviews I'm
looking for comments on approach and logic.
Out of scope: code style and organization at this stage
### Reviewers Phase 2
Once we are all happy with the approach and logic. We can open the
review process to general code quality and nits, to be considered.
---------
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
- Resolves#8780
- Part of #8774
### Changes 🏗️
- Add new UI components
- Add `/agents/[id]` page, with sub-components:
- `AgentRunsSelectorList`
- `AgentRunSummaryCard`
- `AgentRunStatusChip`
- `AgentRunDetailsView`
- `AgentRunDraftView`
- `AgentScheduleDetailsView`
Backend improvements:
- Improve output of execution-related API endpoints: return
`GraphExecution` instead of `NodeExecutionResult[]`
- Reduce log spam from Prisma in tests
General frontend improvements:
- Hide nav link names on smaller screens to prevent navbar overflow
- Clean up styling and fix sizing of `agptui/Button`
Technical frontend improvements:
- Fix tailwind config size increments
- Rename `font-poppin` -> `font-poppins`
- Clean up component implementations and usages
- Yeet all occurrences of `variant="default"`
- Remove `default` button variant as duplicate of `outline`; make
`outline` the default
- Fix minor typing issues
DX:
- Add front end type-check step to `pre-commit` config
- Fix logging setup in conftest.py
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- `/agents/[id]` (new)
- Go to page -> list of runs loads
- Create new run -> runs; all I/O is visible
- Click "Run again" -> runs again with same input
- `/monitoring` (existing)
- Go to page -> everything loads
- Selecting agents and agent runs works
---------
Co-authored-by: Nicholas Tindle <nicktindle@outlook.com>
Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co>
Co-authored-by: Swifty <craigswift13@gmail.com>
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
- Blocked by #9267
This re-introduces changes from the following PRs with fixes:
- #9218
- #9211
### Changes 🏗️
- See #9218
- See #9211
Fixes:
- Fix Prisma query statements in `v2.library.db`
- Fix creation of (library) agents
- Fix test cleanup of (library) agents
- Fix handling and passing of `node_input` parameters
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Create & run a new agent
- [x] Update & run an existing agent
### Changes 🏗️
Due to the legacy of SQLite usage, some of the JSON columns are actually
a string column string a stringified JSON column.
The scope of this PR is migrating those columns into an actual JSON
column.
### Checklist 📋
#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [ ] ...
<details>
<summary>Example test plan</summary>
- [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
- [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
- [ ] Edit an agent from monitor, and confirm it executes correctly
</details>
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Examples of configuration changes</summary>
- Changing ports
- Adding new services that need to communicate with each other
- Secrets or environment variable changes
- New or infrastructure changes such as databases
</details>
Currently it's only possible to open latest graph from monitor and see
the node execution results only when manually running. This PR adds
ability to open running and finished graphs in builder.
### Changes 🏗️
Builder now handles graph version and execution ID in addition to graph
ID when opening a graph. When an execution ID is provided, node
execution results are fetched and subscribed to in real time. This makes
it possible to open a graph that is already executing and see both
existing node execution data and real-time updates (if it's still
running).
- Use graph version and execution id on the builder page and in
`useAgentGraph`
- Use graph version on the `execute_graph` endpoint
- Use graph version on the websockets to distinguish between versions
- Move `formatEdgeID` to utils; it's used in `useAgentGraph.ts` and in
`Flow.tsx`
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Opening finished execution restores node results
- [x] Opening running execution restores results and continues to run
properly
- [x] Results are separate for each graph across multiple tabs
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Examples of configuration changes</summary>
- Changing ports
- Adding new services that need to communicate with each other
- Secrets or environment variable changes
- New or infrastructure changes such as databases
</details>
---------
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
Currently, the test is failing on Jan 29 because 29th Feb does not
exist.
### Changes 🏗️
Force use the first day of the month for getting the current time.
### Checklist 📋
#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [ ] ...
<details>
<summary>Example test plan</summary>
- [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
- [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
- [ ] Edit an agent from monitor, and confirm it executes correctly
</details>
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Examples of configuration changes</summary>
- Changing ports
- Adding new services that need to communicate with each other
- Secrets or environment variable changes
- New or infrastructure changes such as databases
</details>
Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co>
Zamil Majdy
Abhimanyu Yadav
Nicholas Tindle
Swifty
Ubbe
Tejas Dharani
Reinier van der Leer
Krzysztof Czerwinski