AutoGPT/autogpt/tests/conftest.py

108 lines
2.6 KiB
Python
Raw Normal View History

feat(agent): Fully abstracted file storage access with `FileStorage` (#6931) * Rename `FileWorkspace` to `FileStorage` - `autogpt.file_workspace` -> `autogpt.file_storage` - `LocalFileWorkspace` -> `LocalFileStorage` - `S3FileWorkspace` -> `S3FileStorage` - `GCSFileWorkspace` -> `GCSFileStorage` * Rename `WORKSPACE_BACKEND` to `FILE_STORAGE_BACKEND` * Rename `WORKSPACE_STORAGE_BUCKET` to `STORAGE_BUCKET` * Rewrite `AgentManager` to use `FileStorage` rather than direct local file access * Rename `AgentManager.retrieve_state(..)` method to `load_agent_state` * Add docstrings to `AgentManager` * Create `AgentFileManagerMixin` to replace `AgentFileManager`, `FileWorkspaceMixin`, `BaseAgent.attach_fs(..)` * Replace `BaseAgentSettings.save_to_json_file(..)` method by `AgentFileManagerMixin.save_state()` * Replace `BaseAgent.set_id(..)` method by `AgentFileManagerMixin.change_agent_id(..)` * Remove `BaseAgentSettings.load_from_json_file(..)` * Remove `AgentSettings.agent_data_dir` * Update `AgentProtocolServer` to work with the new `FileStorage` system and `AgentFileManagerMixin` * Make `agent_id` and `file_storage` parameters for creating an Agent: - `create_agent`, `configure_agent_with_state`, `_configure_agent`, `create_agent_state` in `autogpt.agent_factory.configurators` - `generate_agent_for_task` in `autogpt.agent_factory.generators` - `Agent.__init__(..)` - `BaseAgent.__init__(..)` - Initialize and pass in `file_storage` in `autogpt.app.main.run_auto_gpt(..)` and `autogpt.app.main.run_auto_gpt_server(..)` * Add `clone_with_subroot` to `FileStorage` * Add `exists`, `make_dir`, `delete_dir`, `rename`, `list_files`, `list_folders` methods to `FileStorage` * Update `autogpt.commands.file_operations` to use `FileStorage` and `AgentFileManagerMixin` features * Update tests for `FileStorage` implementations and usages * Rename `workspace` fixture to `storage` * Update conftest.py
2024-03-11 21:26:14 +00:00
from __future__ import annotations
2023-06-09 22:18:56 +00:00
import os
import uuid
from pathlib import Path
import pytest
refactor(agent, forge): Move library code from `autogpt` to `forge` (#7106) Moved from `autogpt` to `forge`: - `autogpt.config` -> `forge.config` - `autogpt.processing` -> `forge.content_processing` - `autogpt.file_storage` -> `forge.file_storage` - `autogpt.logs` -> `forge.logging` - `autogpt.speech` -> `forge.speech` - `autogpt.agents.(base|components|protocols)` -> `forge.agent.*` - `autogpt.command_decorator` -> `forge.command.decorator` - `autogpt.models.(command|command_parameter)` -> `forge.command.(command|parameter)` - `autogpt.(commands|components|features)` -> `forge.components` - `autogpt.core.utils.json_utils` -> `forge.json.parsing` - `autogpt.prompts.utils` -> `forge.llm.prompting.utils` - `autogpt.core.prompting.(base|schema|utils)` -> `forge.llm.prompting.*` - `autogpt.core.resource.model_providers` -> `forge.llm.providers` - `autogpt.llm.providers.openai` + `autogpt.core.resource.model_providers.utils` -> `forge.llm.providers.utils` - `autogpt.models.action_history:Action*` -> `forge.models.action` - `autogpt.core.configuration.schema` -> `forge.models.config` - `autogpt.core.utils.json_schema` -> `forge.models.json_schema` - `autogpt.core.resource.schema` -> `forge.models.providers` - `autogpt.models.utils` -> `forge.models.utils` - `forge.sdk.(errors|utils)` + `autogpt.utils.(exceptions|file_operations_utils|validators)` -> `forge.utils.(exceptions|file_operations|url_validator)` - `autogpt.utils.utils` -> `forge.utils.const` + `forge.utils.yaml_validator` Moved within `forge`: - forge/prompts/* -> forge/llm/prompting/* The rest are mostly import updates, and some sporadic removals and necessary updates (for example to fix circular deps): - Changed `CommandOutput = Any` to remove coupling with `ContextItem` (no longer needed) - Removed unused `Singleton` class - Reluctantly moved `speech` to forge due to coupling (tts needs to be changed into component) - Moved `function_specs_from_commands` and `core/resource/model_providers` to `llm/providers` (resources were a `core` thing and are no longer relevant) - Keep tests in `autogpt` to reduce changes in this PR - Removed unused memory-related code from tests - Removed duplicated classes: `FancyConsoleFormatter`, `BelowLevelFilter` - `prompt_settings.yaml` is in both `autogpt` and `forge` because for some reason doesn't work when placed in just one dir (need to be taken care of) - Removed `config` param from `clean_input`, it wasn't used and caused circular dependency - Renamed `BaseAgentActionProposal` to `ActionProposal` - Updated `pyproject.toml` in `forge` and `autogpt` - Moved `Action*` models from `forge/components/action_history/model.py` to `forge/models/action.py` as those are relevant to the entire agent and not just `EventHistoryComponent` + to reduce coupling - Renamed `DEFAULT_ASK_COMMAND` to `ASK_COMMAND` and `DEFAULT_FINISH_COMMAND` to `FINISH_COMMAND` - Renamed `AutoGptFormatter` to `ForgeFormatter` and moved to `forge` Includes changes from PR https://github.com/Significant-Gravitas/AutoGPT/pull/7148 --------- Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
2024-05-15 22:37:53 +00:00
from forge.config.ai_profile import AIProfile
from forge.file_storage.local import (
feat(agent): Fully abstracted file storage access with `FileStorage` (#6931) * Rename `FileWorkspace` to `FileStorage` - `autogpt.file_workspace` -> `autogpt.file_storage` - `LocalFileWorkspace` -> `LocalFileStorage` - `S3FileWorkspace` -> `S3FileStorage` - `GCSFileWorkspace` -> `GCSFileStorage` * Rename `WORKSPACE_BACKEND` to `FILE_STORAGE_BACKEND` * Rename `WORKSPACE_STORAGE_BUCKET` to `STORAGE_BUCKET` * Rewrite `AgentManager` to use `FileStorage` rather than direct local file access * Rename `AgentManager.retrieve_state(..)` method to `load_agent_state` * Add docstrings to `AgentManager` * Create `AgentFileManagerMixin` to replace `AgentFileManager`, `FileWorkspaceMixin`, `BaseAgent.attach_fs(..)` * Replace `BaseAgentSettings.save_to_json_file(..)` method by `AgentFileManagerMixin.save_state()` * Replace `BaseAgent.set_id(..)` method by `AgentFileManagerMixin.change_agent_id(..)` * Remove `BaseAgentSettings.load_from_json_file(..)` * Remove `AgentSettings.agent_data_dir` * Update `AgentProtocolServer` to work with the new `FileStorage` system and `AgentFileManagerMixin` * Make `agent_id` and `file_storage` parameters for creating an Agent: - `create_agent`, `configure_agent_with_state`, `_configure_agent`, `create_agent_state` in `autogpt.agent_factory.configurators` - `generate_agent_for_task` in `autogpt.agent_factory.generators` - `Agent.__init__(..)` - `BaseAgent.__init__(..)` - Initialize and pass in `file_storage` in `autogpt.app.main.run_auto_gpt(..)` and `autogpt.app.main.run_auto_gpt_server(..)` * Add `clone_with_subroot` to `FileStorage` * Add `exists`, `make_dir`, `delete_dir`, `rename`, `list_files`, `list_folders` methods to `FileStorage` * Update `autogpt.commands.file_operations` to use `FileStorage` and `AgentFileManagerMixin` features * Update tests for `FileStorage` implementations and usages * Rename `workspace` fixture to `storage` * Update conftest.py
2024-03-11 21:26:14 +00:00
FileStorage,
FileStorageConfiguration,
LocalFileStorage,
feat(agent/workspace): Add GCS and S3 FileWorkspace providers (#6485) * refactor: Rename FileWorkspace to LocalFileWorkspace and create FileWorkspace abstract class - Rename `FileWorkspace` to `LocalFileWorkspace` to provide a more descriptive name for the class that represents a file workspace that works with local files. - Create a new base class `FileWorkspace` to serve as the parent class for `LocalFileWorkspace`. This allows for easier extension and customization of file workspaces in the future. - Update import statements and references to `FileWorkspace` throughout the codebase to use the new naming conventions. * feat: Add S3FileWorkspace + tests + test setups for CI and Docker - Added S3FileWorkspace class to provide an interface for interacting with a file workspace and storing files in an S3 bucket. - Updated pyproject.toml to include dependencies for boto3 and boto3-stubs. - Implemented unit tests for S3FileWorkspace. - Added MinIO service to Docker CI to allow testing S3 features in CI. - Added autogpt-test service config to docker-compose.yml for local testing with MinIO. * ci(docker): tee test output instead of capturing * fix: Improve error handling in S3FileWorkspace.initialize() - Do not tolerate all `botocore.exceptions.ClientError`s - Raise the exception anyways if the error is not "NoSuchBucket" * feat: Add S3 workspace backend support and S3Credentials - Added support for S3 workspace backend in the Autogpt configuration - Added a new sub-config `S3Credentials` to store S3 credentials - Modified the `.env.template` file to include variables related to S3 credentials - Added a new `s3_credentials` attribute on the `Config` class to store S3 credentials - Moved the `unmasked` method from `ModelProviderCredentials` to the parent `ProviderCredentials` class to handle unmasking for S3 credentials * fix(agent/tests): Fix S3FileWorkspace initialization in test_s3_file_workspace.py - Update the S3FileWorkspace initialization in the test_s3_file_workspace.py file to include the required S3 Credentials. * refactor: Remove S3Credentials and add get_workspace function - Remove `S3Credentials` as boto3 will fetch the config from the environment by itself - Add `get_workspace` function in `autogpt.file_workspace` module - Update `.env.template` and tests to reflect the changes * feat(agent/workspace): Make agent workspace backend configurable - Modified `autogpt.file_workspace.get_workspace` function to either take a workspace `id` or `root_path`. - Modified `FileWorkspaceMixin` to use the `get_workspace` function to set up the workspace. - Updated the type hints and imports accordingly. * feat(agent/workspace): Add GCSFileWorkspace for Google Cloud Storage - Added support for Google Cloud Storage as a storage backend option in the workspace. - Created the `GCSFileWorkspace` class to interface with a file workspace stored in a Google Cloud Storage bucket. - Implemented the `GCSFileWorkspaceConfiguration` class to handle the configuration for Google Cloud Storage workspaces. - Updated the `get_workspace` function to include the option to use Google Cloud Storage as a workspace backend. - Added unit tests for the new `GCSFileWorkspace` class. * fix: Unbreak use of non-local workspaces in AgentProtocolServer - Modify the `_get_task_agent_file_workspace` method to handle both local and non-local workspaces correctly
2023-12-07 13:46:08 +00:00
)
Set up unified pre-commit + CI w/ linting + type checking & FIX EVERYTHING (#7171) - **FIX ALL LINT/TYPE ERRORS IN AUTOGPT, FORGE, AND BENCHMARK** ### Linting - Clean up linter configs for `autogpt`, `forge`, and `benchmark` - Add type checking with Pyright - Create unified pre-commit config - Create unified linting and type checking CI workflow ### Testing - Synchronize CI test setups for `autogpt`, `forge`, and `benchmark` - Add missing pytest-cov to benchmark dependencies - Mark GCS tests as slow to speed up pre-commit test runs - Repair `forge` test suite - Add `AgentDB.close()` method for test DB teardown in db_test.py - Use actual temporary dir instead of forge/test_workspace/ - Move left-behind dependencies for moved `forge`-code to from autogpt to forge ### Notable type changes - Replace uses of `ChatModelProvider` by `MultiProvider` - Removed unnecessary exports from various __init__.py - Simplify `FileStorage.open_file` signature by removing `IOBase` from return type union - Implement `S3BinaryIOWrapper(BinaryIO)` type interposer for `S3FileStorage` - Expand overloads of `GCSFileStorage.open_file` for improved typing of read and write modes Had to silence type checking for the extra overloads, because (I think) Pyright is reporting a false-positive: https://github.com/microsoft/pyright/issues/8007 - Change `count_tokens`, `get_tokenizer`, `count_message_tokens` methods on `ModelProvider`s from class methods to instance methods - Move `CompletionModelFunction.schema` method -> helper function `format_function_def_for_openai` in `forge.llm.providers.openai` - Rename `ModelProvider` -> `BaseModelProvider` - Rename `ChatModelProvider` -> `BaseChatModelProvider` - Add type `ChatModelProvider` which is a union of all subclasses of `BaseChatModelProvider` ### Removed rather than fixed - Remove deprecated and broken autogpt/agbenchmark_config/benchmarks.py - Various base classes and properties on base classes in `forge.llm.providers.schema` and `forge.models.providers` ### Fixes for other issues that came to light - Clean up `forge.agent_protocol.api_router`, `forge.agent_protocol.database`, and `forge.agent.agent` - Add fallback behavior to `ImageGeneratorComponent` - Remove test for deprecated failure behavior - Fix `agbenchmark.challenges.builtin` challenge exclusion mechanism on Windows - Fix `_tool_calls_compat_extract_calls` in `forge.llm.providers.openai` - Add support for `any` (= no type specified) in `JSONSchema.typescript_type`
2024-05-28 03:04:21 +00:00
from forge.llm.providers import MultiProvider
refactor(agent, forge): Move library code from `autogpt` to `forge` (#7106) Moved from `autogpt` to `forge`: - `autogpt.config` -> `forge.config` - `autogpt.processing` -> `forge.content_processing` - `autogpt.file_storage` -> `forge.file_storage` - `autogpt.logs` -> `forge.logging` - `autogpt.speech` -> `forge.speech` - `autogpt.agents.(base|components|protocols)` -> `forge.agent.*` - `autogpt.command_decorator` -> `forge.command.decorator` - `autogpt.models.(command|command_parameter)` -> `forge.command.(command|parameter)` - `autogpt.(commands|components|features)` -> `forge.components` - `autogpt.core.utils.json_utils` -> `forge.json.parsing` - `autogpt.prompts.utils` -> `forge.llm.prompting.utils` - `autogpt.core.prompting.(base|schema|utils)` -> `forge.llm.prompting.*` - `autogpt.core.resource.model_providers` -> `forge.llm.providers` - `autogpt.llm.providers.openai` + `autogpt.core.resource.model_providers.utils` -> `forge.llm.providers.utils` - `autogpt.models.action_history:Action*` -> `forge.models.action` - `autogpt.core.configuration.schema` -> `forge.models.config` - `autogpt.core.utils.json_schema` -> `forge.models.json_schema` - `autogpt.core.resource.schema` -> `forge.models.providers` - `autogpt.models.utils` -> `forge.models.utils` - `forge.sdk.(errors|utils)` + `autogpt.utils.(exceptions|file_operations_utils|validators)` -> `forge.utils.(exceptions|file_operations|url_validator)` - `autogpt.utils.utils` -> `forge.utils.const` + `forge.utils.yaml_validator` Moved within `forge`: - forge/prompts/* -> forge/llm/prompting/* The rest are mostly import updates, and some sporadic removals and necessary updates (for example to fix circular deps): - Changed `CommandOutput = Any` to remove coupling with `ContextItem` (no longer needed) - Removed unused `Singleton` class - Reluctantly moved `speech` to forge due to coupling (tts needs to be changed into component) - Moved `function_specs_from_commands` and `core/resource/model_providers` to `llm/providers` (resources were a `core` thing and are no longer relevant) - Keep tests in `autogpt` to reduce changes in this PR - Removed unused memory-related code from tests - Removed duplicated classes: `FancyConsoleFormatter`, `BelowLevelFilter` - `prompt_settings.yaml` is in both `autogpt` and `forge` because for some reason doesn't work when placed in just one dir (need to be taken care of) - Removed `config` param from `clean_input`, it wasn't used and caused circular dependency - Renamed `BaseAgentActionProposal` to `ActionProposal` - Updated `pyproject.toml` in `forge` and `autogpt` - Moved `Action*` models from `forge/components/action_history/model.py` to `forge/models/action.py` as those are relevant to the entire agent and not just `EventHistoryComponent` + to reduce coupling - Renamed `DEFAULT_ASK_COMMAND` to `ASK_COMMAND` and `DEFAULT_FINISH_COMMAND` to `FINISH_COMMAND` - Renamed `AutoGptFormatter` to `ForgeFormatter` and moved to `forge` Includes changes from PR https://github.com/Significant-Gravitas/AutoGPT/pull/7148 --------- Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
2024-05-15 22:37:53 +00:00
from forge.logging.config import configure_logging
from autogpt.agents.agent import Agent, AgentConfiguration, AgentSettings
feat(forge): Component-specific configuration (#7170) Remove many env vars and use component-level configuration that could be loaded from file instead. ### Changed - `BaseAgent` provides `serialize_configs` and `deserialize_configs` that can save and load all component configuration as json `str`. Deserialized components/values overwrite existing values, so not all values need to be present in the serialized config. - Decoupled `forge/content_processing/text.py` from `Config` - Kept `execute_local_commands` in `Config` because it's needed to know if OS info should be included in the prompt - Updated docs to reflect changes - Renamed `Config` to `AppConfig` ### Added - Added `ConfigurableComponent` class for components and following configs: - `ActionHistoryConfiguration` - `CodeExecutorConfiguration` - `FileManagerConfiguration` - now file manager allows to have multiple agents using the same workspace - `GitOperationsConfiguration` - `ImageGeneratorConfiguration` - `WebSearchConfiguration` - `WebSeleniumConfiguration` - `BaseConfig` in `forge` and moved `Config` (now inherits from `BaseConfig`) back to `autogpt` - Required `config_class` attribute for the `ConfigurableComponent` class that should be set to configuration class for a component `--component-config-file` CLI option and `COMPONENT_CONFIG_FILE` env var and field in `Config`. This option allows to load configuration from a specific file, CLI option takes precedence over env var. - Added comments to config models ### Removed - Unused `change_agent_id` method from `FileManagerComponent` - Unused `allow_downloads` from `Config` and CLI options (it should be in web component config if needed) - CLI option `--browser-name` (the option is inside `WebSeleniumConfiguration`) - Unused `workspace_directory` from CLI options - No longer needed variables from `Config` and docs - Unused fields from `Config`: `image_size`, `audio_to_text_provider`, `huggingface_audio_to_text_model` - Removed `files` and `workspace` class attributes from `FileManagerComponent`
2024-06-19 08:14:01 +00:00
from autogpt.app.config import AppConfig, ConfigBuilder
refactor(agent, forge): Move library code from `autogpt` to `forge` (#7106) Moved from `autogpt` to `forge`: - `autogpt.config` -> `forge.config` - `autogpt.processing` -> `forge.content_processing` - `autogpt.file_storage` -> `forge.file_storage` - `autogpt.logs` -> `forge.logging` - `autogpt.speech` -> `forge.speech` - `autogpt.agents.(base|components|protocols)` -> `forge.agent.*` - `autogpt.command_decorator` -> `forge.command.decorator` - `autogpt.models.(command|command_parameter)` -> `forge.command.(command|parameter)` - `autogpt.(commands|components|features)` -> `forge.components` - `autogpt.core.utils.json_utils` -> `forge.json.parsing` - `autogpt.prompts.utils` -> `forge.llm.prompting.utils` - `autogpt.core.prompting.(base|schema|utils)` -> `forge.llm.prompting.*` - `autogpt.core.resource.model_providers` -> `forge.llm.providers` - `autogpt.llm.providers.openai` + `autogpt.core.resource.model_providers.utils` -> `forge.llm.providers.utils` - `autogpt.models.action_history:Action*` -> `forge.models.action` - `autogpt.core.configuration.schema` -> `forge.models.config` - `autogpt.core.utils.json_schema` -> `forge.models.json_schema` - `autogpt.core.resource.schema` -> `forge.models.providers` - `autogpt.models.utils` -> `forge.models.utils` - `forge.sdk.(errors|utils)` + `autogpt.utils.(exceptions|file_operations_utils|validators)` -> `forge.utils.(exceptions|file_operations|url_validator)` - `autogpt.utils.utils` -> `forge.utils.const` + `forge.utils.yaml_validator` Moved within `forge`: - forge/prompts/* -> forge/llm/prompting/* The rest are mostly import updates, and some sporadic removals and necessary updates (for example to fix circular deps): - Changed `CommandOutput = Any` to remove coupling with `ContextItem` (no longer needed) - Removed unused `Singleton` class - Reluctantly moved `speech` to forge due to coupling (tts needs to be changed into component) - Moved `function_specs_from_commands` and `core/resource/model_providers` to `llm/providers` (resources were a `core` thing and are no longer relevant) - Keep tests in `autogpt` to reduce changes in this PR - Removed unused memory-related code from tests - Removed duplicated classes: `FancyConsoleFormatter`, `BelowLevelFilter` - `prompt_settings.yaml` is in both `autogpt` and `forge` because for some reason doesn't work when placed in just one dir (need to be taken care of) - Removed `config` param from `clean_input`, it wasn't used and caused circular dependency - Renamed `BaseAgentActionProposal` to `ActionProposal` - Updated `pyproject.toml` in `forge` and `autogpt` - Moved `Action*` models from `forge/components/action_history/model.py` to `forge/models/action.py` as those are relevant to the entire agent and not just `EventHistoryComponent` + to reduce coupling - Renamed `DEFAULT_ASK_COMMAND` to `ASK_COMMAND` and `DEFAULT_FINISH_COMMAND` to `FINISH_COMMAND` - Renamed `AutoGptFormatter` to `ForgeFormatter` and moved to `forge` Includes changes from PR https://github.com/Significant-Gravitas/AutoGPT/pull/7148 --------- Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
2024-05-15 22:37:53 +00:00
from autogpt.app.main import _configure_llm_provider
pytest_plugins = [
"tests.integration.agent_factory",
]
@pytest.fixture()
def tmp_project_root(tmp_path: Path) -> Path:
return tmp_path
@pytest.fixture()
def app_data_dir(tmp_project_root: Path) -> Path:
refactor(agent/config): Modularize `Config` and revive Azure support (#6497) * feat: Refactor config loading and initialization to be modular and decentralized - Refactored the `ConfigBuilder` class to support modular loading and initialization of the configuration from environment variables. - Implemented recursive loading and initialization of nested config objects. - Introduced the `SystemConfiguration` base class to provide common functionality for all system settings. - Added the `from_env` attribute to the `UserConfigurable` decorator to provide environment variable mappings. - Updated the `Config` class and its related classes to inherit from `SystemConfiguration` and use the `UserConfigurable` decorator. - Updated `LoggingConfig` and `TTSConfig` to use the `UserConfigurable` decorator for their fields. - Modified the implementation of the `build_config_from_env` method in `ConfigBuilder` to utilize the new modular and recursive loading and initialization logic. - Updated applicable test cases to reflect the changes in the config loading and initialization logic. This refactor improves the flexibility and maintainability of the configuration loading process by introducing modular and recursive behavior, allowing for easier extension and customization through environment variables. * refactor: Move OpenAI credentials into `OpenAICredentials` sub-config - Move OpenAI API key and other OpenAI credentials from the global config to a new sub-config called OpenAICredentials. - Update the necessary code to use the new OpenAICredentials sub-config instead of the global config when accessing OpenAI credentials. - (Hopefully) unbreak Azure support. - Update azure.yaml.template. - Enable validation of assignment operations on SystemConfiguration and SystemSettings objects. * feat: Update AutoGPT configuration options and setup instructions - Added new configuration options for logging and OpenAI usage to .env.template - Removed deprecated configuration options in config/config.py - Updated setup instructions in Docker and general setup documentation to include information on using Azure's OpenAI services * fix: Fix image generation with Dall-E - Fix issue with image generation with Dall-E API Additional user context: This commit fixes an issue with image generation using the Dall-E API. The code now correctly retrieves the API key from the agent's legacy configuration. * refactor(agent/core): Refactor `autogpt.core.configuration.schema` and update docstrings - Refactor the `schema.py` file in the `autogpt.core.configuration` module. - Added docstring to `SystemConfiguration.from_env()` - Updated docstrings for functions `_get_user_config_values`, `_get_non_default_user_config_values`, `_recursive_init_model`, `_recurse_user_config_fields`, and `_recurse_user_config_values`.
2023-12-05 15:28:23 +00:00
dir = tmp_project_root / "data"
dir.mkdir(parents=True, exist_ok=True)
return dir
@pytest.fixture()
feat(agent): Fully abstracted file storage access with `FileStorage` (#6931) * Rename `FileWorkspace` to `FileStorage` - `autogpt.file_workspace` -> `autogpt.file_storage` - `LocalFileWorkspace` -> `LocalFileStorage` - `S3FileWorkspace` -> `S3FileStorage` - `GCSFileWorkspace` -> `GCSFileStorage` * Rename `WORKSPACE_BACKEND` to `FILE_STORAGE_BACKEND` * Rename `WORKSPACE_STORAGE_BUCKET` to `STORAGE_BUCKET` * Rewrite `AgentManager` to use `FileStorage` rather than direct local file access * Rename `AgentManager.retrieve_state(..)` method to `load_agent_state` * Add docstrings to `AgentManager` * Create `AgentFileManagerMixin` to replace `AgentFileManager`, `FileWorkspaceMixin`, `BaseAgent.attach_fs(..)` * Replace `BaseAgentSettings.save_to_json_file(..)` method by `AgentFileManagerMixin.save_state()` * Replace `BaseAgent.set_id(..)` method by `AgentFileManagerMixin.change_agent_id(..)` * Remove `BaseAgentSettings.load_from_json_file(..)` * Remove `AgentSettings.agent_data_dir` * Update `AgentProtocolServer` to work with the new `FileStorage` system and `AgentFileManagerMixin` * Make `agent_id` and `file_storage` parameters for creating an Agent: - `create_agent`, `configure_agent_with_state`, `_configure_agent`, `create_agent_state` in `autogpt.agent_factory.configurators` - `generate_agent_for_task` in `autogpt.agent_factory.generators` - `Agent.__init__(..)` - `BaseAgent.__init__(..)` - Initialize and pass in `file_storage` in `autogpt.app.main.run_auto_gpt(..)` and `autogpt.app.main.run_auto_gpt_server(..)` * Add `clone_with_subroot` to `FileStorage` * Add `exists`, `make_dir`, `delete_dir`, `rename`, `list_files`, `list_folders` methods to `FileStorage` * Update `autogpt.commands.file_operations` to use `FileStorage` and `AgentFileManagerMixin` features * Update tests for `FileStorage` implementations and usages * Rename `workspace` fixture to `storage` * Update conftest.py
2024-03-11 21:26:14 +00:00
def storage(app_data_dir: Path) -> FileStorage:
storage = LocalFileStorage(
FileStorageConfiguration(root=app_data_dir, restrict_to_root=False)
)
storage.initialize()
return storage
2023-04-23 23:40:53 +00:00
@pytest.fixture(scope="function")
def config(
tmp_project_root: Path,
app_data_dir: Path,
):
if not os.environ.get("OPENAI_API_KEY"):
os.environ["OPENAI_API_KEY"] = "sk-dummy"
refactor(agent/config): Modularize `Config` and revive Azure support (#6497) * feat: Refactor config loading and initialization to be modular and decentralized - Refactored the `ConfigBuilder` class to support modular loading and initialization of the configuration from environment variables. - Implemented recursive loading and initialization of nested config objects. - Introduced the `SystemConfiguration` base class to provide common functionality for all system settings. - Added the `from_env` attribute to the `UserConfigurable` decorator to provide environment variable mappings. - Updated the `Config` class and its related classes to inherit from `SystemConfiguration` and use the `UserConfigurable` decorator. - Updated `LoggingConfig` and `TTSConfig` to use the `UserConfigurable` decorator for their fields. - Modified the implementation of the `build_config_from_env` method in `ConfigBuilder` to utilize the new modular and recursive loading and initialization logic. - Updated applicable test cases to reflect the changes in the config loading and initialization logic. This refactor improves the flexibility and maintainability of the configuration loading process by introducing modular and recursive behavior, allowing for easier extension and customization through environment variables. * refactor: Move OpenAI credentials into `OpenAICredentials` sub-config - Move OpenAI API key and other OpenAI credentials from the global config to a new sub-config called OpenAICredentials. - Update the necessary code to use the new OpenAICredentials sub-config instead of the global config when accessing OpenAI credentials. - (Hopefully) unbreak Azure support. - Update azure.yaml.template. - Enable validation of assignment operations on SystemConfiguration and SystemSettings objects. * feat: Update AutoGPT configuration options and setup instructions - Added new configuration options for logging and OpenAI usage to .env.template - Removed deprecated configuration options in config/config.py - Updated setup instructions in Docker and general setup documentation to include information on using Azure's OpenAI services * fix: Fix image generation with Dall-E - Fix issue with image generation with Dall-E API Additional user context: This commit fixes an issue with image generation using the Dall-E API. The code now correctly retrieves the API key from the agent's legacy configuration. * refactor(agent/core): Refactor `autogpt.core.configuration.schema` and update docstrings - Refactor the `schema.py` file in the `autogpt.core.configuration` module. - Added docstring to `SystemConfiguration.from_env()` - Updated docstrings for functions `_get_user_config_values`, `_get_non_default_user_config_values`, `_recursive_init_model`, `_recurse_user_config_fields`, and `_recurse_user_config_values`.
2023-12-05 15:28:23 +00:00
config = ConfigBuilder.build_config_from_env(project_root=tmp_project_root)
2023-06-20 13:47:59 +00:00
config.app_data_dir = app_data_dir
2023-07-26 22:22:07 +00:00
config.noninteractive_mode = True
2023-08-22 05:29:56 +00:00
2023-04-23 23:40:53 +00:00
yield config
2023-08-24 14:09:25 +00:00
@pytest.fixture(scope="session")
feat(forge): Component-specific configuration (#7170) Remove many env vars and use component-level configuration that could be loaded from file instead. ### Changed - `BaseAgent` provides `serialize_configs` and `deserialize_configs` that can save and load all component configuration as json `str`. Deserialized components/values overwrite existing values, so not all values need to be present in the serialized config. - Decoupled `forge/content_processing/text.py` from `Config` - Kept `execute_local_commands` in `Config` because it's needed to know if OS info should be included in the prompt - Updated docs to reflect changes - Renamed `Config` to `AppConfig` ### Added - Added `ConfigurableComponent` class for components and following configs: - `ActionHistoryConfiguration` - `CodeExecutorConfiguration` - `FileManagerConfiguration` - now file manager allows to have multiple agents using the same workspace - `GitOperationsConfiguration` - `ImageGeneratorConfiguration` - `WebSearchConfiguration` - `WebSeleniumConfiguration` - `BaseConfig` in `forge` and moved `Config` (now inherits from `BaseConfig`) back to `autogpt` - Required `config_class` attribute for the `ConfigurableComponent` class that should be set to configuration class for a component `--component-config-file` CLI option and `COMPONENT_CONFIG_FILE` env var and field in `Config`. This option allows to load configuration from a specific file, CLI option takes precedence over env var. - Added comments to config models ### Removed - Unused `change_agent_id` method from `FileManagerComponent` - Unused `allow_downloads` from `Config` and CLI options (it should be in web component config if needed) - CLI option `--browser-name` (the option is inside `WebSeleniumConfiguration`) - Unused `workspace_directory` from CLI options - No longer needed variables from `Config` and docs - Unused fields from `Config`: `image_size`, `audio_to_text_provider`, `huggingface_audio_to_text_model` - Removed `files` and `workspace` class attributes from `FileManagerComponent`
2024-06-19 08:14:01 +00:00
def setup_logger():
configure_logging(
debug=True,
log_dir=Path(__file__).parent / "logs",
plain_console_output=True,
)
2023-08-24 14:09:25 +00:00
@pytest.fixture
feat(forge): Component-specific configuration (#7170) Remove many env vars and use component-level configuration that could be loaded from file instead. ### Changed - `BaseAgent` provides `serialize_configs` and `deserialize_configs` that can save and load all component configuration as json `str`. Deserialized components/values overwrite existing values, so not all values need to be present in the serialized config. - Decoupled `forge/content_processing/text.py` from `Config` - Kept `execute_local_commands` in `Config` because it's needed to know if OS info should be included in the prompt - Updated docs to reflect changes - Renamed `Config` to `AppConfig` ### Added - Added `ConfigurableComponent` class for components and following configs: - `ActionHistoryConfiguration` - `CodeExecutorConfiguration` - `FileManagerConfiguration` - now file manager allows to have multiple agents using the same workspace - `GitOperationsConfiguration` - `ImageGeneratorConfiguration` - `WebSearchConfiguration` - `WebSeleniumConfiguration` - `BaseConfig` in `forge` and moved `Config` (now inherits from `BaseConfig`) back to `autogpt` - Required `config_class` attribute for the `ConfigurableComponent` class that should be set to configuration class for a component `--component-config-file` CLI option and `COMPONENT_CONFIG_FILE` env var and field in `Config`. This option allows to load configuration from a specific file, CLI option takes precedence over env var. - Added comments to config models ### Removed - Unused `change_agent_id` method from `FileManagerComponent` - Unused `allow_downloads` from `Config` and CLI options (it should be in web component config if needed) - CLI option `--browser-name` (the option is inside `WebSeleniumConfiguration`) - Unused `workspace_directory` from CLI options - No longer needed variables from `Config` and docs - Unused fields from `Config`: `image_size`, `audio_to_text_provider`, `huggingface_audio_to_text_model` - Removed `files` and `workspace` class attributes from `FileManagerComponent`
2024-06-19 08:14:01 +00:00
def llm_provider(config: AppConfig) -> MultiProvider:
feat(agent/core): Add Anthropic Claude 3 support (#7085) - feat(agent/core): Add `AnthropicProvider` - Add `ANTHROPIC_API_KEY` to .env.template and docs Notable differences in logic compared to `OpenAIProvider`: - Merges subsequent user messages in `AnthropicProvider._get_chat_completion_args` - Merges and extracts all system messages into `system` parameter in `AnthropicProvider._get_chat_completion_args` - Supports prefill; merges prefill content (if any) into generated response - Prompt changes to improve compatibility with `AnthropicProvider` Anthropic has a slightly different API compared to OpenAI, and has much stricter input validation. E.g. Anthropic only supports a single `system` prompt, where OpenAI allows multiple `system` messages. Anthropic also forbids sequences of multiple `user` or `assistant` messages and requires that messages alternate between roles. - Move response format instruction from separate message into main system prompt - Fix clock message format - Add pre-fill to `OneShot` generated prompt - refactor(agent/core): Tweak `model_providers.schema` - Simplify `ModelProviderUsage` - Remove attribute `total_tokens` as it is always equal to `prompt_tokens + completion_tokens` - Modify signature of `update_usage(..)`; no longer requires a full `ModelResponse` object as input - Improve `ModelProviderBudget` - Change type of attribute `usage` to `defaultdict[str, ModelProviderUsage]` -> allow per-model usage tracking - Modify signature of `update_usage_and_cost(..)`; no longer requires a full `ModelResponse` object as input - Allow `ModelProviderBudget` zero-argument instantiation - Fix type of `AssistantChatMessage.role` to match `ChatMessage.role` (str -> `ChatMessage.Role`) - Add shared attributes and constructor to `ModelProvider` base class - Add `max_output_tokens` parameter to `create_chat_completion` interface - Add pre-filling as a global feature - Add `prefill_response` field to `ChatPrompt` model - Add `prefill_response` parameter to `create_chat_completion` interface - Add `ChatModelProvider.get_available_models()` and remove `ApiManager` - Remove unused `OpenAIChatParser` typedef in openai.py - Remove redundant `budget` attribute definition on `OpenAISettings` - Remove unnecessary `usage` in `OpenAIProvider` > `default_settings` > `budget` - feat(agent): Allow use of any available LLM provider through `MultiProvider` - Add `MultiProvider` (`model_providers.multi`) - Replace all references to / uses of `OpenAIProvider` with `MultiProvider` - Change type of `Config.smart_llm` and `Config.fast_llm` from `str` to `ModelName` - feat(agent/core): Validate function call arguments in `create_chat_completion` - Add `validate_call` method to `CompletionModelFunction` in `model_providers.schema` - Add `validate_tool_calls` utility function in `model_providers.utils` - Add tool call validation step to `create_chat_completion` in `OpenAIProvider` and `AnthropicProvider` - Remove (now redundant) command argument validation logic in agent.py and models/command.py - refactor(agent): Rename `get_openai_command_specs` to `function_specs_from_commands`
2024-05-04 18:33:25 +00:00
return _configure_llm_provider(config)
@pytest.fixture
feat(forge): Component-specific configuration (#7170) Remove many env vars and use component-level configuration that could be loaded from file instead. ### Changed - `BaseAgent` provides `serialize_configs` and `deserialize_configs` that can save and load all component configuration as json `str`. Deserialized components/values overwrite existing values, so not all values need to be present in the serialized config. - Decoupled `forge/content_processing/text.py` from `Config` - Kept `execute_local_commands` in `Config` because it's needed to know if OS info should be included in the prompt - Updated docs to reflect changes - Renamed `Config` to `AppConfig` ### Added - Added `ConfigurableComponent` class for components and following configs: - `ActionHistoryConfiguration` - `CodeExecutorConfiguration` - `FileManagerConfiguration` - now file manager allows to have multiple agents using the same workspace - `GitOperationsConfiguration` - `ImageGeneratorConfiguration` - `WebSearchConfiguration` - `WebSeleniumConfiguration` - `BaseConfig` in `forge` and moved `Config` (now inherits from `BaseConfig`) back to `autogpt` - Required `config_class` attribute for the `ConfigurableComponent` class that should be set to configuration class for a component `--component-config-file` CLI option and `COMPONENT_CONFIG_FILE` env var and field in `Config`. This option allows to load configuration from a specific file, CLI option takes precedence over env var. - Added comments to config models ### Removed - Unused `change_agent_id` method from `FileManagerComponent` - Unused `allow_downloads` from `Config` and CLI options (it should be in web component config if needed) - CLI option `--browser-name` (the option is inside `WebSeleniumConfiguration`) - Unused `workspace_directory` from CLI options - No longer needed variables from `Config` and docs - Unused fields from `Config`: `image_size`, `audio_to_text_provider`, `huggingface_audio_to_text_model` - Removed `files` and `workspace` class attributes from `FileManagerComponent`
2024-06-19 08:14:01 +00:00
def agent(
config: AppConfig, llm_provider: MultiProvider, storage: FileStorage
) -> Agent:
2023-10-08 01:14:52 +00:00
ai_profile = AIProfile(
ai_name="Base",
ai_role="A base AI",
ai_goals=[],
)
agent_settings = AgentSettings(
name=Agent.default_settings.name,
description=Agent.default_settings.description,
agent_id=f"AutoGPT-test-agent-{str(uuid.uuid4())[:8]}",
2023-10-08 01:14:52 +00:00
ai_profile=ai_profile,
config=AgentConfiguration(
fast_llm=config.fast_llm,
smart_llm=config.smart_llm,
allow_fs_access=not config.restrict_to_workspace,
use_functions_api=config.openai_functions,
),
history=Agent.default_settings.history.model_copy(deep=True),
)
agent = Agent(
settings=agent_settings,
llm_provider=llm_provider,
feat(agent): Fully abstracted file storage access with `FileStorage` (#6931) * Rename `FileWorkspace` to `FileStorage` - `autogpt.file_workspace` -> `autogpt.file_storage` - `LocalFileWorkspace` -> `LocalFileStorage` - `S3FileWorkspace` -> `S3FileStorage` - `GCSFileWorkspace` -> `GCSFileStorage` * Rename `WORKSPACE_BACKEND` to `FILE_STORAGE_BACKEND` * Rename `WORKSPACE_STORAGE_BUCKET` to `STORAGE_BUCKET` * Rewrite `AgentManager` to use `FileStorage` rather than direct local file access * Rename `AgentManager.retrieve_state(..)` method to `load_agent_state` * Add docstrings to `AgentManager` * Create `AgentFileManagerMixin` to replace `AgentFileManager`, `FileWorkspaceMixin`, `BaseAgent.attach_fs(..)` * Replace `BaseAgentSettings.save_to_json_file(..)` method by `AgentFileManagerMixin.save_state()` * Replace `BaseAgent.set_id(..)` method by `AgentFileManagerMixin.change_agent_id(..)` * Remove `BaseAgentSettings.load_from_json_file(..)` * Remove `AgentSettings.agent_data_dir` * Update `AgentProtocolServer` to work with the new `FileStorage` system and `AgentFileManagerMixin` * Make `agent_id` and `file_storage` parameters for creating an Agent: - `create_agent`, `configure_agent_with_state`, `_configure_agent`, `create_agent_state` in `autogpt.agent_factory.configurators` - `generate_agent_for_task` in `autogpt.agent_factory.generators` - `Agent.__init__(..)` - `BaseAgent.__init__(..)` - Initialize and pass in `file_storage` in `autogpt.app.main.run_auto_gpt(..)` and `autogpt.app.main.run_auto_gpt_server(..)` * Add `clone_with_subroot` to `FileStorage` * Add `exists`, `make_dir`, `delete_dir`, `rename`, `list_files`, `list_folders` methods to `FileStorage` * Update `autogpt.commands.file_operations` to use `FileStorage` and `AgentFileManagerMixin` features * Update tests for `FileStorage` implementations and usages * Rename `workspace` fixture to `storage` * Update conftest.py
2024-03-11 21:26:14 +00:00
file_storage=storage,
feat(forge): Component-specific configuration (#7170) Remove many env vars and use component-level configuration that could be loaded from file instead. ### Changed - `BaseAgent` provides `serialize_configs` and `deserialize_configs` that can save and load all component configuration as json `str`. Deserialized components/values overwrite existing values, so not all values need to be present in the serialized config. - Decoupled `forge/content_processing/text.py` from `Config` - Kept `execute_local_commands` in `Config` because it's needed to know if OS info should be included in the prompt - Updated docs to reflect changes - Renamed `Config` to `AppConfig` ### Added - Added `ConfigurableComponent` class for components and following configs: - `ActionHistoryConfiguration` - `CodeExecutorConfiguration` - `FileManagerConfiguration` - now file manager allows to have multiple agents using the same workspace - `GitOperationsConfiguration` - `ImageGeneratorConfiguration` - `WebSearchConfiguration` - `WebSeleniumConfiguration` - `BaseConfig` in `forge` and moved `Config` (now inherits from `BaseConfig`) back to `autogpt` - Required `config_class` attribute for the `ConfigurableComponent` class that should be set to configuration class for a component `--component-config-file` CLI option and `COMPONENT_CONFIG_FILE` env var and field in `Config`. This option allows to load configuration from a specific file, CLI option takes precedence over env var. - Added comments to config models ### Removed - Unused `change_agent_id` method from `FileManagerComponent` - Unused `allow_downloads` from `Config` and CLI options (it should be in web component config if needed) - CLI option `--browser-name` (the option is inside `WebSeleniumConfiguration`) - Unused `workspace_directory` from CLI options - No longer needed variables from `Config` and docs - Unused fields from `Config`: `image_size`, `audio_to_text_provider`, `huggingface_audio_to_text_model` - Removed `files` and `workspace` class attributes from `FileManagerComponent`
2024-06-19 08:14:01 +00:00
app_config=config,
)
return agent