* Implement syntax fault tolerant `json_loads` function using `dem3json`
- Add `dem3json` dependency
* Replace `json.loads` by `json_loads` in places where malformed JSON may occur
* Move `json_utils.py` to `autogpt/core/utils`
* Add tests for `json_utils`
---------
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
- Replace `session.prompt_async(..)` with `click.prompt(..)` in `clean_input` (autogpt/app/utils.py)
- Convert `clean_input` back to a synchronous function (and amend its usages accordingly)
- Remove `prompt-toolkit` dependency
This mitigates issues crashes in some shell environments on Windows.
- Improve error output for failure to load plugin
- Fix logic to determine qualified module name
- Use `importlib` rather than `__import__` magic function
This unbreaks `scan_plugins` on Windows.
- Add `AgentFinished` exception (subclass of `AgentTerminated`)
- Raise `AgentFinished` instead of `AgentTerminated` in `finish` method
- Remove resumption patch from PR #6990 in `BaseAgent`
- Clean up implementation of `finish` in `AgentProtocolServer`
- Add resumption mechanism in `run_auto_gpt` (main.py)
Fix resume-related exceptions
- CLI: prevent resumed agent to register action on already existing one
- Server: prevent trying to json() command without result
* Rename `FileWorkspace` to `FileStorage`
- `autogpt.file_workspace` -> `autogpt.file_storage`
- `LocalFileWorkspace` -> `LocalFileStorage`
- `S3FileWorkspace` -> `S3FileStorage`
- `GCSFileWorkspace` -> `GCSFileStorage`
* Rename `WORKSPACE_BACKEND` to `FILE_STORAGE_BACKEND`
* Rename `WORKSPACE_STORAGE_BUCKET` to `STORAGE_BUCKET`
* Rewrite `AgentManager` to use `FileStorage` rather than direct local file access
* Rename `AgentManager.retrieve_state(..)` method to `load_agent_state`
* Add docstrings to `AgentManager`
* Create `AgentFileManagerMixin` to replace `AgentFileManager`, `FileWorkspaceMixin`, `BaseAgent.attach_fs(..)`
* Replace `BaseAgentSettings.save_to_json_file(..)` method by `AgentFileManagerMixin.save_state()`
* Replace `BaseAgent.set_id(..)` method by `AgentFileManagerMixin.change_agent_id(..)`
* Remove `BaseAgentSettings.load_from_json_file(..)`
* Remove `AgentSettings.agent_data_dir`
* Update `AgentProtocolServer` to work with the new `FileStorage` system and `AgentFileManagerMixin`
* Make `agent_id` and `file_storage` parameters for creating an Agent:
- `create_agent`, `configure_agent_with_state`, `_configure_agent`, `create_agent_state` in `autogpt.agent_factory.configurators`
- `generate_agent_for_task` in `autogpt.agent_factory.generators`
- `Agent.__init__(..)`
- `BaseAgent.__init__(..)`
- Initialize and pass in `file_storage` in `autogpt.app.main.run_auto_gpt(..)` and `autogpt.app.main.run_auto_gpt_server(..)`
* Add `clone_with_subroot` to `FileStorage`
* Add `exists`, `make_dir`, `delete_dir`, `rename`, `list_files`, `list_folders` methods to `FileStorage`
* Update `autogpt.commands.file_operations` to use `FileStorage` and `AgentFileManagerMixin` features
* Update tests for `FileStorage` implementations and usages
* Rename `workspace` fixture to `storage`
* Update conftest.py
Poetry recently released v1.8.x containing a fix for the issue we were having earlier:
https://github.com/python-poetry/poetry/issues/8548
This means unavailable optional directory dependencies no longer break the docker build.
* Fix unmasking of `azure_endpoint` in `OpenAICredentials.get_api_access_kwargs()`
* Amend `ApiManager.get_models` to use `AzureOpenAI` client when `api_type` is set to `azure`
---------
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
* By default, allow requests originating from http://localhost:{AP_SERVER_PORT} instead of all origins
* Allow configuring allowed CORS origins through `AP_SERVER_CORS_ALLOWED_ORIGINS`
* Mitigate shell injection in `MacOSTTS._speech` implementation
* Mitigate shell command control bypassing in `execute_shell` and `execute_shell_popen` commands
- Improve implementation and docstring of `validate_command` function
---------
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
We have been seeing `AttributeError: 'NoneType' object has no attribute 'split'` in Sentry.
According to https://github.com/SergeyPirogov/webdriver_manager/issues/649, this occurs
when Chrome is not installed, or when no suitable ChromeDriver version is available for
the installed version of Chrome.
Instead of the `AttributeError`, we can print a better message for the agent and user to help them fix the issue.
---
Co-authored-by: kcze <kpczerwinski@gmail.com>
OpenAI requires the `id` property on tool calls in assistant messages.
We weren't storing it in the `AssistantChatMessage` that is created from the LLM's response,
causing an error when adding those messages back to the prompt.
Fix:
* Add `id` to `AssistantToolCall` and `AssistantToolCallDict` types in model_providers/schema.py
* Amend `_tool_calls_compat_extract_calls` to generate an ID for extracted tool calls
---
Co-authored-by: kcze <kpczerwinski@gmail.com>
OpenAI likes neither `tool_calls=[]` nor `tool_calls=None`. If no `tool_calls` are in the message, the key must be omitted.
This partially reverts commit 67bafa6302.
---
Co-authored-by: kcze <kpczerwinski@gmail.com>
In a `for item in list` loop, removing items from the list while iterating causes it to skip over the next item. Fix: refactor `interactively_revise_ai_settings` routine to use while loop for iterating over constraints, resources, and best practices.
---------
Co-authored-by: Kripanshu Jindal <polaris@Polaris.local>
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
- Added `task_cumulative_cost` and `task_total_cost` attributes to the `Step.additional_output` in the `AgentProtocolServer.execute_step` endpoint.
- Updated `agbenchmark` dependency in Agent and Forge
- Added a helper function `.app.utils.vcs_state_diverges_from_master()`. This function determines whether the relevant part of the codebase diverges from our `master`.
- Updated `.app.telemetry._setup_sentry()` to determine the default environment name using `vcs_state_diverges_from_master`.
OpenAI's newest models return JSON with markdown fences around it, breaking the `json.loads` parser.
This commit adds an `extract_list_from_response` function to json_utils/utilities.py and uses this function to replace `json.loads` in `_process_text`.
* Add Sentry integration for telemetry
- Add `sentry_sdk` dependency
- Add setup logic and config flow using `TELEMETRY_OPT_IN` environment variable
- Add app/telemetry.py with `setup_telemetry` helper routine
- Call `setup_telemetry` in `cli()` in app/cli.py
- Add `TELEMETRY_OPT_IN` to .env.template
- Add helper function `env_file_exists` and routine `set_env_config_value` to app/utils.py
- Add unit tests for `set_env_config_value` in test_utils.py
- Add prompt to startup to ask whether the user wants to enable telemetry if the env variable isn't set
* Add `capture_exception` statements for LLM parsing errors and command failures
- Change default `SMART_LLM` from `gpt-4` to `gpt-4-turbo-preview`
- Change default `FAST_LLM` from `gpt-3.5-turbo-16k` to `gpt-3.5-turbo-0125`
- Change default `EMBEDDING_MODEL` from `text-embedding-ada-002` to `text-embedding-3-small`
- Update .env.template, azure.yaml.template, and documentation accordingly
- Add `text-embedding-3-small` and `text-embedding-3-large` as `EMBEDDING_v3_S` and `EMBEDDING_v3_L` respectively
- Add `gpt-3.5-turbo-0125` as `GPT3_v4`
- Add `gpt-4-1106-vision-preview` as `GPT4_v3_VISION`
- Add GPT-4V models to info map
- Change chat model info mapping to derive info for aliases (e.g. `gpt-3.5-turbo`) from specific versions instead of the other way around
* Add `_sideload_chrome_extensions` subroutine to `open_page_in_browser` in web_selenium.py
* Sideloads uBlock Origin and I Still Don't Care About Cookies, downloading them if necessary
* Add 2-second delay to `open_page_in_browser` to allow time for handling cookie walls
Commit 956cdc7 "fix(agent/json_utils): Decode as JSON rather than Python objects" broke these unit tests because they generated "JSON" by stringifying a Python object.
* Compress steps in the prompt to reduce token usage, and to increase longevity when using models with limited context windows
* Move multiple copies of step formatting code to `Episode.format` method
* Add `EpisodicActionHistory.handle_compression` method to handle compression of new steps
* Implement `extract_information` function in `autogpt.processing.text` module. This function extracts pieces of information from a body of text based on a list of topics of interest.
* Add `topics_of_interest` and `get_raw_content` parameters to `read_webpage` commmand
* Limit maximum content length if `get_raw_content=true` is specified
* Replace `ast.literal_eval` with `json.loads` in `extract_dict_from_response`
This fixes a bug where boolean values could not be decoded because of their required capitalization in Python.
- Pydantic shallow-copies models when they are passed into a parent model, meaning they can't be updated through the original reference. This commit adds a fix for the resulting cost persistence issue.
- The `extract_dict_from_response` function, which is supposed to reliably extract a JSON object from an LLM's response, positively discriminated objects defined on a single line, causing issues.
- `summarize_text` and `QueryLanguageModel.__call__` still tried to access `response["content"]`, which isn't possible since upgrading to the OpenAI v1 client library.
- When an Artifact's file is modified by the agent, set its `agent_created` attribute to `True` instead of registering a new Artifact
- Update the `autogpt-forge` dependency to the newest version, in which `AgentDB.update_artifact` has been implemented
- Update `openai` dependency from ^v0.27.10 to ^v1.7.2
- Update poetry.lock
- Update code for changed endpoints and new output types of OpenAI library
- Replace uses of `AssistantChatMessageDict` by `AssistantChatMessage`
- Update `PromptStrategy`, `BaseAgent`, and all of their subclasses accordingly
- Update `OpenAIProvider`, `OpenAICredentials`, azure.yaml.template, .env.template and test_config.py to work with new separate `AzureOpenAI` client
- Remove `_OpenAIRetryHandler` and implement retry mechanism with `tenacity`
- Rewrite pytest fixture `cached_openai_client` (renamed from `patched_api_requestor`) for OpenAI v1 library
* docs: Add documentation on how to use Agent Protocol in the template
- Added documentation on how to use Agent Protocol and its settings in the `.env.template` file.
- An explanation is provided for the `AP_SERVER_PORT` and `AP_SERVER_DB_URL` settings.
- This change aims to improve the understanding and usage of Agent Protocol in the project.
* docs: Update usage.md with information about configuring the API port
- Update the documentation for the `serve` mode in `usage.md`
- Add information about configuring the port for the API server using the `AP_SERVER_PORT` environment variable.
---------
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
gitpython was installed as an indirect dependency via agbenchmark. The release builds don't contain agbenchmark and thus also lack the gitpython package, which breaks the image.
- Move `auto-gpt-plugin-template` from dev dependencies to regular dependencies in `pyproject.toml`.
- Fixes#6566 - No module named 'auto_gpt_plugin_template'.
- Add logging to capture errors raised during execution of actions in the Agent
- Move `fmt_kwargs` function from `agent_protocol_server.py` to `logs/utils.py`
- Update the `read_file` function in `file_operations.py` to pass the file's extension to the `decode_textual_file` function.
- Modify the `decode_textual_file` function in `file_operations_utils.py` to accept the file extension as an argument.
- Update the `content` property in the `FileContextItem` class in `context_item.py` to pass the file's extension to the `decode_textual_file` function.
- Update the `test_parsers` function in `test_text_file_parsers.py` to pass the file extension to the `decode_textual_file` function.
- Prevent the Agent from treating `AgentTerminated` like it would any other exception raised by a command.
- The agent should raise AgentTerminated exception to exit the loop.
- Fix the parsing of invalid LLM responses by appending an error message to the prompt and allowing the LLM to fix its mistakes.
- Update the `OpenAIProvider` to handle the self-correction process and limit the number of attempts to fix parsing errors.
- Update the `BaseAgent` to profit from the new pasing and parse-fixing mechanism.
This change ensures that the system can handle and recover from errors in parsing LLM responses.
Hopefully this fixes#1407 once and for all.
- Updated the quick links and user guide URLs in the bulletin
- Released v0.5.0 with cloud-readiness, a new UI, support for the newest Agent Protocol version, and other improvements
See the release notes on Github for more details: https://github.com/Significant-Gravitas/AutoGPT/releases
- Removed unnecessary print_attribute calls in configurators.py and configurator.py files
- Consolidated printing of configuration attributes in main.py for improved readability and reduced log spam in Agent Protocol mode
- Update GCSFileWorkspace.initialize() to handle cases where the bucket doesn't exist and create it if necessary
- Add logging to S3FileWorkspace.initialize() and GCSFileWorkspace.initialize()
- Update GCSFileWorkspace.list() and S3FileWorkspace.list() to correctly handle nested paths and return the relative paths of files
- Fix tests for GCSFileWorkspace and S3FileWorkspace to account for the changes in initialization and listing behavior
- Fix S3FileWorkspace.open_file() to correctly switch between binary and text mode
- Added tests to verify the fixes in workspace initialization and listing behavior
- Fixes#6553 (`web_search` command not working)
- v3.x.x of the duckduckgo-search library no longer works, so updating to v4.0.0 unbreaks the `web_search` command
feat: Add dependencies required to use PostgreSQL
- Added psycopg2-binary version 2.9.9 to the dependencies in pyproject.toml
- Updated the poetry.lock file with the new package information
- Update the signature of `FileWorkspace.open_file` and fix implementations in every workspace backend
- Replace `open()` with `workspace.open_file` in the `read_file` command to use the workspace's file opening functionality
- Fix the parametrization of the `test_text_file_parsers` test to correctly test text file parsers
- Adjusted path processing and use of `agent.workspace` in the file_operations.py module to prevent double path resolution.
- Updated the `is_duplicate_operation` and `log_operation` functions in file_operations.py to use the `make_relative` argument of the `sanitize_path_arg` decorator.
- Refactored the `write_to_file`, `list_folder`, and `list_files` functions in file_operations.py to accept both string and Path objects as the path argument.
- Modified the GCSFileWorkspace and S3FileWorkspace classes in the file_workspace module to ensure that the root path is always an absolute path.
This commit addresses issues with path processing in the file_operations.py module and across different workspace backend implementations. The changes ensure that relative paths are correctly converted to absolute paths where necessary and that the file operations logic functions consistently handle path arguments as strings or Path objects. Additionally, the GCSFileWorkspace and S3FileWorkspace classes now enforce that the root path is always an absolute path.
- Remove X- prefix from X-AutoGPT-UserID, X-AP-TaskID, and X-AP-StepID headers
- Refactor literal references to "ask_user" to `ask_user.__name__` in AgentProtocolServer
- Fix type annotation for `agent_id` in `BaseAgentSettings` class
- Add assertion to ensure `agent_id` is not an empty string in `AgentManager.get_agent_dir()` method
- Change type of `override_name` and `override_role` to be optional in `apply_overrides_to_ai_settings()` function
- Update `AgentProtocolServer` to include `X-AP-TaskID` and `X-AutoGPT-UserID` headers in outgoing requests for Agent Protocol tasks.
- Modify `ModelProvider` and `OpenAIProvider` to allow configuring extra headers to be added to all outgoing requests.
- Fix the type of the `task_id` parameter in `AgentProtocolServer.get_task`
- Update the return type of the `AgentProtocolServer.get_artifact` method to `StreamingResponse`.
- Fix the Content-Disposition header in the response to include quotes around the filename.
Krzysztof Czerwinski
Reinier van der Leer
edwardsp
Elias Hohl
Thunder Drag
Toran Bruce Richards
Nicholas Tindle