feat(agent): Fully abstracted file storage access with `FileStorage` (#6931)
* Rename `FileWorkspace` to `FileStorage`
- `autogpt.file_workspace` -> `autogpt.file_storage`
- `LocalFileWorkspace` -> `LocalFileStorage`
- `S3FileWorkspace` -> `S3FileStorage`
- `GCSFileWorkspace` -> `GCSFileStorage`
* Rename `WORKSPACE_BACKEND` to `FILE_STORAGE_BACKEND`
* Rename `WORKSPACE_STORAGE_BUCKET` to `STORAGE_BUCKET`
* Rewrite `AgentManager` to use `FileStorage` rather than direct local file access
* Rename `AgentManager.retrieve_state(..)` method to `load_agent_state`
* Add docstrings to `AgentManager`
* Create `AgentFileManagerMixin` to replace `AgentFileManager`, `FileWorkspaceMixin`, `BaseAgent.attach_fs(..)`
* Replace `BaseAgentSettings.save_to_json_file(..)` method by `AgentFileManagerMixin.save_state()`
* Replace `BaseAgent.set_id(..)` method by `AgentFileManagerMixin.change_agent_id(..)`
* Remove `BaseAgentSettings.load_from_json_file(..)`
* Remove `AgentSettings.agent_data_dir`
* Update `AgentProtocolServer` to work with the new `FileStorage` system and `AgentFileManagerMixin`
* Make `agent_id` and `file_storage` parameters for creating an Agent:
- `create_agent`, `configure_agent_with_state`, `_configure_agent`, `create_agent_state` in `autogpt.agent_factory.configurators`
- `generate_agent_for_task` in `autogpt.agent_factory.generators`
- `Agent.__init__(..)`
- `BaseAgent.__init__(..)`
- Initialize and pass in `file_storage` in `autogpt.app.main.run_auto_gpt(..)` and `autogpt.app.main.run_auto_gpt_server(..)`
* Add `clone_with_subroot` to `FileStorage`
* Add `exists`, `make_dir`, `delete_dir`, `rename`, `list_files`, `list_folders` methods to `FileStorage`
* Update `autogpt.commands.file_operations` to use `FileStorage` and `AgentFileManagerMixin` features
* Update tests for `FileStorage` implementations and usages
* Rename `workspace` fixture to `storage`
* Update conftest.py
2024-03-11 21:26:14 +00:00
|
|
|
from __future__ import annotations
|
|
|
|
|
2023-06-09 22:18:56 +00:00
|
|
|
import os
|
2023-12-02 12:26:17 +00:00
|
|
|
import uuid
|
2023-04-23 19:36:04 +00:00
|
|
|
from pathlib import Path
|
|
|
|
|
|
|
|
import pytest
|
2024-05-15 22:37:53 +00:00
|
|
|
from forge.config.ai_profile import AIProfile
|
|
|
|
from forge.file_storage.local import (
|
feat(agent): Fully abstracted file storage access with `FileStorage` (#6931)
* Rename `FileWorkspace` to `FileStorage`
- `autogpt.file_workspace` -> `autogpt.file_storage`
- `LocalFileWorkspace` -> `LocalFileStorage`
- `S3FileWorkspace` -> `S3FileStorage`
- `GCSFileWorkspace` -> `GCSFileStorage`
* Rename `WORKSPACE_BACKEND` to `FILE_STORAGE_BACKEND`
* Rename `WORKSPACE_STORAGE_BUCKET` to `STORAGE_BUCKET`
* Rewrite `AgentManager` to use `FileStorage` rather than direct local file access
* Rename `AgentManager.retrieve_state(..)` method to `load_agent_state`
* Add docstrings to `AgentManager`
* Create `AgentFileManagerMixin` to replace `AgentFileManager`, `FileWorkspaceMixin`, `BaseAgent.attach_fs(..)`
* Replace `BaseAgentSettings.save_to_json_file(..)` method by `AgentFileManagerMixin.save_state()`
* Replace `BaseAgent.set_id(..)` method by `AgentFileManagerMixin.change_agent_id(..)`
* Remove `BaseAgentSettings.load_from_json_file(..)`
* Remove `AgentSettings.agent_data_dir`
* Update `AgentProtocolServer` to work with the new `FileStorage` system and `AgentFileManagerMixin`
* Make `agent_id` and `file_storage` parameters for creating an Agent:
- `create_agent`, `configure_agent_with_state`, `_configure_agent`, `create_agent_state` in `autogpt.agent_factory.configurators`
- `generate_agent_for_task` in `autogpt.agent_factory.generators`
- `Agent.__init__(..)`
- `BaseAgent.__init__(..)`
- Initialize and pass in `file_storage` in `autogpt.app.main.run_auto_gpt(..)` and `autogpt.app.main.run_auto_gpt_server(..)`
* Add `clone_with_subroot` to `FileStorage`
* Add `exists`, `make_dir`, `delete_dir`, `rename`, `list_files`, `list_folders` methods to `FileStorage`
* Update `autogpt.commands.file_operations` to use `FileStorage` and `AgentFileManagerMixin` features
* Update tests for `FileStorage` implementations and usages
* Rename `workspace` fixture to `storage`
* Update conftest.py
2024-03-11 21:26:14 +00:00
|
|
|
FileStorage,
|
|
|
|
FileStorageConfiguration,
|
|
|
|
LocalFileStorage,
|
2023-12-07 13:46:08 +00:00
|
|
|
)
|
Set up unified pre-commit + CI w/ linting + type checking & FIX EVERYTHING (#7171)
- **FIX ALL LINT/TYPE ERRORS IN AUTOGPT, FORGE, AND BENCHMARK**
### Linting
- Clean up linter configs for `autogpt`, `forge`, and `benchmark`
- Add type checking with Pyright
- Create unified pre-commit config
- Create unified linting and type checking CI workflow
### Testing
- Synchronize CI test setups for `autogpt`, `forge`, and `benchmark`
- Add missing pytest-cov to benchmark dependencies
- Mark GCS tests as slow to speed up pre-commit test runs
- Repair `forge` test suite
- Add `AgentDB.close()` method for test DB teardown in db_test.py
- Use actual temporary dir instead of forge/test_workspace/
- Move left-behind dependencies for moved `forge`-code to from autogpt to forge
### Notable type changes
- Replace uses of `ChatModelProvider` by `MultiProvider`
- Removed unnecessary exports from various __init__.py
- Simplify `FileStorage.open_file` signature by removing `IOBase` from return type union
- Implement `S3BinaryIOWrapper(BinaryIO)` type interposer for `S3FileStorage`
- Expand overloads of `GCSFileStorage.open_file` for improved typing of read and write modes
Had to silence type checking for the extra overloads, because (I think) Pyright is reporting a false-positive:
https://github.com/microsoft/pyright/issues/8007
- Change `count_tokens`, `get_tokenizer`, `count_message_tokens` methods on `ModelProvider`s from class methods to instance methods
- Move `CompletionModelFunction.schema` method -> helper function `format_function_def_for_openai` in `forge.llm.providers.openai`
- Rename `ModelProvider` -> `BaseModelProvider`
- Rename `ChatModelProvider` -> `BaseChatModelProvider`
- Add type `ChatModelProvider` which is a union of all subclasses of `BaseChatModelProvider`
### Removed rather than fixed
- Remove deprecated and broken autogpt/agbenchmark_config/benchmarks.py
- Various base classes and properties on base classes in `forge.llm.providers.schema` and `forge.models.providers`
### Fixes for other issues that came to light
- Clean up `forge.agent_protocol.api_router`, `forge.agent_protocol.database`, and `forge.agent.agent`
- Add fallback behavior to `ImageGeneratorComponent`
- Remove test for deprecated failure behavior
- Fix `agbenchmark.challenges.builtin` challenge exclusion mechanism on Windows
- Fix `_tool_calls_compat_extract_calls` in `forge.llm.providers.openai`
- Add support for `any` (= no type specified) in `JSONSchema.typescript_type`
2024-05-28 03:04:21 +00:00
|
|
|
from forge.llm.providers import MultiProvider
|
2024-05-15 22:37:53 +00:00
|
|
|
from forge.logging.config import configure_logging
|
|
|
|
|
|
|
|
from autogpt.agents.agent import Agent, AgentConfiguration, AgentSettings
|
2024-06-19 08:14:01 +00:00
|
|
|
from autogpt.app.config import AppConfig, ConfigBuilder
|
2024-05-15 22:37:53 +00:00
|
|
|
from autogpt.app.main import _configure_llm_provider
|
2023-04-27 17:16:56 +00:00
|
|
|
|
2023-06-06 17:48:49 +00:00
|
|
|
pytest_plugins = [
|
|
|
|
"tests.integration.agent_factory",
|
|
|
|
]
|
2023-05-26 19:33:49 +00:00
|
|
|
|
|
|
|
|
2023-04-23 19:36:04 +00:00
|
|
|
@pytest.fixture()
|
2023-10-08 08:04:53 +00:00
|
|
|
def tmp_project_root(tmp_path: Path) -> Path:
|
|
|
|
return tmp_path
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.fixture()
|
|
|
|
def app_data_dir(tmp_project_root: Path) -> Path:
|
2023-12-05 15:28:23 +00:00
|
|
|
dir = tmp_project_root / "data"
|
|
|
|
dir.mkdir(parents=True, exist_ok=True)
|
|
|
|
return dir
|
2023-10-08 08:04:53 +00:00
|
|
|
|
|
|
|
|
|
|
|
@pytest.fixture()
|
feat(agent): Fully abstracted file storage access with `FileStorage` (#6931)
* Rename `FileWorkspace` to `FileStorage`
- `autogpt.file_workspace` -> `autogpt.file_storage`
- `LocalFileWorkspace` -> `LocalFileStorage`
- `S3FileWorkspace` -> `S3FileStorage`
- `GCSFileWorkspace` -> `GCSFileStorage`
* Rename `WORKSPACE_BACKEND` to `FILE_STORAGE_BACKEND`
* Rename `WORKSPACE_STORAGE_BUCKET` to `STORAGE_BUCKET`
* Rewrite `AgentManager` to use `FileStorage` rather than direct local file access
* Rename `AgentManager.retrieve_state(..)` method to `load_agent_state`
* Add docstrings to `AgentManager`
* Create `AgentFileManagerMixin` to replace `AgentFileManager`, `FileWorkspaceMixin`, `BaseAgent.attach_fs(..)`
* Replace `BaseAgentSettings.save_to_json_file(..)` method by `AgentFileManagerMixin.save_state()`
* Replace `BaseAgent.set_id(..)` method by `AgentFileManagerMixin.change_agent_id(..)`
* Remove `BaseAgentSettings.load_from_json_file(..)`
* Remove `AgentSettings.agent_data_dir`
* Update `AgentProtocolServer` to work with the new `FileStorage` system and `AgentFileManagerMixin`
* Make `agent_id` and `file_storage` parameters for creating an Agent:
- `create_agent`, `configure_agent_with_state`, `_configure_agent`, `create_agent_state` in `autogpt.agent_factory.configurators`
- `generate_agent_for_task` in `autogpt.agent_factory.generators`
- `Agent.__init__(..)`
- `BaseAgent.__init__(..)`
- Initialize and pass in `file_storage` in `autogpt.app.main.run_auto_gpt(..)` and `autogpt.app.main.run_auto_gpt_server(..)`
* Add `clone_with_subroot` to `FileStorage`
* Add `exists`, `make_dir`, `delete_dir`, `rename`, `list_files`, `list_folders` methods to `FileStorage`
* Update `autogpt.commands.file_operations` to use `FileStorage` and `AgentFileManagerMixin` features
* Update tests for `FileStorage` implementations and usages
* Rename `workspace` fixture to `storage`
* Update conftest.py
2024-03-11 21:26:14 +00:00
|
|
|
def storage(app_data_dir: Path) -> FileStorage:
|
|
|
|
storage = LocalFileStorage(
|
|
|
|
FileStorageConfiguration(root=app_data_dir, restrict_to_root=False)
|
|
|
|
)
|
|
|
|
storage.initialize()
|
|
|
|
return storage
|
2023-04-23 23:40:53 +00:00
|
|
|
|
|
|
|
|
2024-01-17 18:56:19 +00:00
|
|
|
@pytest.fixture(scope="function")
|
2023-10-08 08:04:53 +00:00
|
|
|
def config(
|
|
|
|
tmp_project_root: Path,
|
|
|
|
app_data_dir: Path,
|
|
|
|
):
|
2023-06-24 04:15:20 +00:00
|
|
|
if not os.environ.get("OPENAI_API_KEY"):
|
|
|
|
os.environ["OPENAI_API_KEY"] = "sk-dummy"
|
2023-12-05 15:28:23 +00:00
|
|
|
config = ConfigBuilder.build_config_from_env(project_root=tmp_project_root)
|
2023-06-20 13:47:59 +00:00
|
|
|
|
2023-10-08 08:04:53 +00:00
|
|
|
config.app_data_dir = app_data_dir
|
2023-07-26 22:22:07 +00:00
|
|
|
|
2023-08-24 14:20:35 +00:00
|
|
|
config.noninteractive_mode = True
|
2023-08-22 05:29:56 +00:00
|
|
|
|
2023-04-23 23:40:53 +00:00
|
|
|
yield config
|
2023-04-27 17:16:56 +00:00
|
|
|
|
2023-04-25 18:12:24 +00:00
|
|
|
|
2023-08-24 14:09:25 +00:00
|
|
|
@pytest.fixture(scope="session")
|
2024-06-19 08:14:01 +00:00
|
|
|
def setup_logger():
|
2024-04-22 15:14:22 +00:00
|
|
|
configure_logging(
|
|
|
|
debug=True,
|
|
|
|
log_dir=Path(__file__).parent / "logs",
|
|
|
|
plain_console_output=True,
|
|
|
|
)
|
2023-08-24 14:09:25 +00:00
|
|
|
|
|
|
|
|
2023-06-10 22:48:50 +00:00
|
|
|
@pytest.fixture
|
2024-06-19 08:14:01 +00:00
|
|
|
def llm_provider(config: AppConfig) -> MultiProvider:
|
2024-05-04 18:33:25 +00:00
|
|
|
return _configure_llm_provider(config)
|
2023-09-21 14:32:19 +00:00
|
|
|
|
|
|
|
|
|
|
|
@pytest.fixture
|
2024-06-19 08:14:01 +00:00
|
|
|
def agent(
|
|
|
|
config: AppConfig, llm_provider: MultiProvider, storage: FileStorage
|
|
|
|
) -> Agent:
|
2023-10-08 01:14:52 +00:00
|
|
|
ai_profile = AIProfile(
|
2023-06-10 22:48:50 +00:00
|
|
|
ai_name="Base",
|
|
|
|
ai_role="A base AI",
|
|
|
|
ai_goals=[],
|
|
|
|
)
|
|
|
|
|
2023-09-21 14:46:13 +00:00
|
|
|
agent_settings = AgentSettings(
|
|
|
|
name=Agent.default_settings.name,
|
|
|
|
description=Agent.default_settings.description,
|
2023-12-02 12:26:17 +00:00
|
|
|
agent_id=f"AutoGPT-test-agent-{str(uuid.uuid4())[:8]}",
|
2023-10-08 01:14:52 +00:00
|
|
|
ai_profile=ai_profile,
|
2023-09-21 14:46:13 +00:00
|
|
|
config=AgentConfiguration(
|
|
|
|
fast_llm=config.fast_llm,
|
|
|
|
smart_llm=config.smart_llm,
|
2023-10-08 08:04:53 +00:00
|
|
|
allow_fs_access=not config.restrict_to_workspace,
|
2023-09-21 14:46:13 +00:00
|
|
|
use_functions_api=config.openai_functions,
|
|
|
|
),
|
2024-07-02 18:45:32 +00:00
|
|
|
history=Agent.default_settings.history.model_copy(deep=True),
|
2023-09-21 14:46:13 +00:00
|
|
|
)
|
|
|
|
|
2023-10-08 08:04:53 +00:00
|
|
|
agent = Agent(
|
2023-09-21 14:46:13 +00:00
|
|
|
settings=agent_settings,
|
2023-09-21 14:32:19 +00:00
|
|
|
llm_provider=llm_provider,
|
feat(agent): Fully abstracted file storage access with `FileStorage` (#6931)
* Rename `FileWorkspace` to `FileStorage`
- `autogpt.file_workspace` -> `autogpt.file_storage`
- `LocalFileWorkspace` -> `LocalFileStorage`
- `S3FileWorkspace` -> `S3FileStorage`
- `GCSFileWorkspace` -> `GCSFileStorage`
* Rename `WORKSPACE_BACKEND` to `FILE_STORAGE_BACKEND`
* Rename `WORKSPACE_STORAGE_BUCKET` to `STORAGE_BUCKET`
* Rewrite `AgentManager` to use `FileStorage` rather than direct local file access
* Rename `AgentManager.retrieve_state(..)` method to `load_agent_state`
* Add docstrings to `AgentManager`
* Create `AgentFileManagerMixin` to replace `AgentFileManager`, `FileWorkspaceMixin`, `BaseAgent.attach_fs(..)`
* Replace `BaseAgentSettings.save_to_json_file(..)` method by `AgentFileManagerMixin.save_state()`
* Replace `BaseAgent.set_id(..)` method by `AgentFileManagerMixin.change_agent_id(..)`
* Remove `BaseAgentSettings.load_from_json_file(..)`
* Remove `AgentSettings.agent_data_dir`
* Update `AgentProtocolServer` to work with the new `FileStorage` system and `AgentFileManagerMixin`
* Make `agent_id` and `file_storage` parameters for creating an Agent:
- `create_agent`, `configure_agent_with_state`, `_configure_agent`, `create_agent_state` in `autogpt.agent_factory.configurators`
- `generate_agent_for_task` in `autogpt.agent_factory.generators`
- `Agent.__init__(..)`
- `BaseAgent.__init__(..)`
- Initialize and pass in `file_storage` in `autogpt.app.main.run_auto_gpt(..)` and `autogpt.app.main.run_auto_gpt_server(..)`
* Add `clone_with_subroot` to `FileStorage`
* Add `exists`, `make_dir`, `delete_dir`, `rename`, `list_files`, `list_folders` methods to `FileStorage`
* Update `autogpt.commands.file_operations` to use `FileStorage` and `AgentFileManagerMixin` features
* Update tests for `FileStorage` implementations and usages
* Rename `workspace` fixture to `storage`
* Update conftest.py
2024-03-11 21:26:14 +00:00
|
|
|
file_storage=storage,
|
2024-06-19 08:14:01 +00:00
|
|
|
app_config=config,
|
2023-06-10 22:48:50 +00:00
|
|
|
)
|
2023-10-08 08:04:53 +00:00
|
|
|
return agent
|