Commit Graph

65 Commits (757fb075b1a31e09530ad67250153dcbfa65107b)

Author SHA1 Message Date
Zamil Majdy 8938209d0d
fix(backend): Avoid long synchronous call to block FastAPI event-loop (#8429) 2024-10-27 23:54:38 +07:00
Zamil Majdy 1e620fdb13
feat(backend): RedisEventQueue into Pub/Sub (#8387) 2024-10-27 23:49:33 +07:00
Aarushi e2df6019fb
feat(blocks): Add pinecone and jina blocks (#8401)
* add pinecone and jina blocks

* udpate based on comments

* backend updates

* frontend updates

* type hint

* more type hints

* another type hint

* update run signature

* shared jina provider

* fix linting

* lockfile

* remove noqa

* remove noqa

* remove vector db folder

* line

* update pincone credentials provider

* fix imports

* formating

* update frontend

* Test (#8425)

* h

* Discard changes to autogpt_platform/backend/poetry.lock

* fix: broken dep

---------

Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co>
2024-10-24 21:01:53 +01:00
Zamil Majdy 1321faf9db
fix(backend): Fix error pin output not being propagated into the next nodes (#8408) 2024-10-24 17:13:58 +00:00
Zamil Majdy 9e411efd6e
fix(backend): Reverse pyro config refactor (#8410)
* fix(backend): Fix error pin output not being propagated into the next nodes

* fix(backend): Reverse pyro config refactor

* Revert "fix(backend): Fix error pin output not being propagated into the next nodes"

This reverts commit 2ff50a94ec.

---------

Co-authored-by: Aarushi <50577581+aarushik93@users.noreply.github.com>
2024-10-23 15:54:29 +00:00
Zamil Majdy 27c9ec5bcd
fix(platform): Enable auth on local-backend mode by default (#8405) 2024-10-23 03:39:15 +00:00
Zamil Majdy 17e79ad88d
fix(backend): Fix DatabaseManager usage by calling it on-demand (#8404) 2024-10-23 10:09:23 +07:00
Zamil Majdy 7f318685af Revert "fix(backend): Fix DatabaseManager usage by calling it on-demand"
This reverts commit e5f5005ab8.
2024-10-23 07:41:59 +07:00
Zamil Majdy e5f5005ab8 fix(backend): Fix DatabaseManager usage by calling it on-demand 2024-10-23 07:39:15 +07:00
Zamil Majdy f4dac22335
feat(backend): Track LLM token usage + LLM blocks cleanup (#8367) 2024-10-22 18:13:27 -05:00
Aarushi 81febb6589
feat(platform): Added latest claude version (#8397)
latest sonnet model
2024-10-22 16:51:23 +01:00
Nicholas Tindle 1622a4aaa8
refactor(backend): Move credentials storage to prisma user (#8283)
* feat(frontend,backend): testing

* feat: testing

* feat(backend): it works for reading email

* feat(backend): more docs on google

* fix(frontend,backend): formatting

* feat(backend): more logigin (i know this should be debug)

* feat(backend): make real the default scopes

* feat(backend): tests and linting

* fix: code review prep

* feat: sheets block

* feat: liniting

* Update route.ts

* Update autogpt_platform/backend/backend/integrations/oauth/google.py

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>

* Update autogpt_platform/backend/backend/server/routers/integrations.py

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>

* fix: revert opener change

* feat(frontend): add back opener

required to work on mac edge

* feat(frontend): drop typing list import from gmail

* fix: code review comments

* feat: code review changes

* feat: code review changes

* fix(backend): move from asserts to checks so they don't get optimized away in the future

* fix(backend): code review changes

* fix(backend): remove google specific check

* fix: add typing

* fix: only enable google blocks when oauth is configured for google

* fix: errors are real and valid outputs always when output

* fix(backend): add provider detail for debuging scope declines

* Update autogpt_platform/frontend/src/components/integrations/credentials-input.tsx

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>

* fix(frontend): enhance with comment, typeof error isn't known so this is best way to ensure the stringifyication will work

* feat: code review change requests

* fix: linting

* fix: reduce error catching

* fix: doc messages in code

* fix: check the correct scopes object 😄

* fix: remove double (and not needed) try catch

* fix: lint

* fix: scopes

* feat: handle the default scopes better

* feat: better email objectification

* feat: process attachements

turns out an email doesn't need a body

* fix: lint

* Update google.py

* Update autogpt_platform/backend/backend/data/block.py

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>

* fix: quit trying and except failure

* Update autogpt_platform/backend/backend/server/routers/integrations.py

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>

* feat: don't allow expired states

* fix: clarify function name and purpose

* feat: code links updates

* feat: additional docs on adding a block

* fix: type hint missing which means the block won't work

* fix: linting

* fix: docs formatting

* Update issues.py

* fix: improve the naming

* fix: formatting

* Update new_blocks.md

* Update new_blocks.md

* feat: better docs on what the args mean

* feat: more details on yield

* Update new_blocks.md

* fix: remove ignore from docs build

* feat: initial migration

* feat: migration tested with supabase-> prisma data location

* add custom migrations and script

* update migration command

* formatting and linting

* updated migration script

* add direct db url

* add find files

* rename

* use binary instead of source

* temp adding supabase

* remove unused functions

* adding missed merge

* fix: commit hash for lock

* ci: fix lint

* fix: minor bugs that prevented connecting and migrating to dbs and auth

* fix: linting

* fix: missed await

* fix(backend): phase one pr updates

* fix: handle error with returning user object from database_manager

* fix: linting

* Address comments

* Make the migration safe

* Update migration doc

* Move misplaced model functions

* Grammar

* Revert lock

* Remove irrelevant changes

* Remove irrelevant changes

* Avoid adding trigger on public schema

---------

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
Co-authored-by: Aarushi <aarushik93@gmail.com>
Co-authored-by: Aarushi <50577581+aarushik93@users.noreply.github.com>
2024-10-22 13:33:54 +01:00
Aarushi 6f3828fc99
fix(dockercompose): Fix db manager connection (#8377)
* add db host

* remove unused variable
2024-10-18 12:49:56 +00:00
Zamil Majdy 26b1bca033
refactor(backend): Make block fields consistently use SchemaField (#8360) 2024-10-18 10:22:05 +07:00
Zamil Majdy 2c0286e411
feat(backend): Add credit for Jina/Search & LLM blocks (#8361) 2024-10-17 20:51:10 +07:00
Aarushi ac33c1eb03
fix(platform): Include health router (#8362)
include health router
2024-10-17 11:09:33 +01:00
Aarushi 2d0e51fe28
security(platform/backend): Add health endpoint (#8341)
* add health endpoint

* fix linting

---------

Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co>
2024-10-16 14:57:23 +00:00
Zamil Majdy 6f07d24e93
feat(backend): Introduce executors shared DB connection (#8340) 2024-10-16 21:15:23 +07:00
Krzysztof Czerwinski bd5d2b1e86
feat(platform): List and revoke credentials in user profile (#8207)
Display existing credentials (OAuth and API keys) for all current providers: Google, Github, Notion and allow user to remove them. For providers that support it, we also revoke the tokens through the API: of the providers we currently have, Google and GitHub support it; Notion doesn't.

- Add credentials list and `Delete` button in `/profile`
- Add `revoke_tokens` abstract method to `BaseOAuthHandler` and implement it in each provider
- Revoke OAuth tokens for providers on `DELETE` `/{provider}/credentials/{cred_id}`, and return whether tokens could be revoked
   - Update `autogpt-server-api/baseClient.ts:deleteCredentials` with `CredentialsDeleteResponse` return type

Bonus:
- Update `autogpt-server-api/baseClient.ts:_request` to properly handle empty server responses
2024-10-14 17:50:55 +02:00
Reinier van der Leer 992989ee71
feat(backend): Ensure validity of OAuth credentials during graph execution (#8191)
- feat(backend/executor): Change credential injection mechanism to acquire credentials from `AgentServer` just before execution
  - Also locks the credentials for the duration of the execution

- feat(backend/server): Add thread-safe `IntegrationCredentialsManager` to handle and synchronize credentials-related operations

- feat(libs): Add mutexes to `SupabaseIntegrationCredentialsStore` to ensure thread-safety

Also:
- feat(backend): Added Pydantic model (de)serialization support to `@expose` decorator

Refactorings:
- refactor(backend, libs): Move `KeyedMutex` to `autogpt_libs.utils.synchronize`
- refactor(backend/server): Make `backend.server.integrations` module with `router`, `creds_manager`, and `utils` in it
2024-10-10 16:45:43 +00:00
Zamil Majdy 9ad5e1f808
fix(platform): Remove blind try-except for yielding error on block (#8287) 2024-10-10 23:25:29 +07:00
vishesh10 7b92bae942
Fix block execution status in case of error (#8267) 2024-10-10 02:59:26 +00:00
Aarushi c03e2fb949
tweak(platform): Remove importing templates from local dir (#8276)
* always filter on user id

* add user id to doc string

* fix linting

* fix imports function

* remove import templates from local directory
2024-10-09 23:13:46 +00:00
Nicholas Tindle e654aa1e7a
feat(backend): add the capibility to disable llm models in the cloud env (#8285)
* feat(backend): logic to disable enums based on python logic

* feat(backend): add behave as setting and clarify its purpose and APP_ENV

APP_ENV is used for not cloud vs local but the application environment such as local/dev/prod so we need BehaveAs as well

* fix(backend): various uses of AppEnvironment without the Enum or incorrectly

AppEnv in the logging library will never be cloud due to the restrictions applied when loading settings in by pydantic settings. This commit fixes this error, however the code path for logging may now be incorrect

* feat(backend): use a metaclass to disable ollama in the cloud environment

* fix: formatting

* fix(backend): typing improvements

* fix(backend): more linting 😭
2024-10-09 10:12:48 +01:00
Aarushi e37744b9f2
fix(platform): Update deletion of secret values to not do it in place (#8284)
update deletion of secret values to not do it in place
2024-10-08 22:43:36 +01:00
Aarushi bc1df92c29
fix(platform): Fix marketplace leaking secrets (#8281)
* add hide secrets param

* Update autogpt_platform/backend/backend/data/graph.py

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>

* Update autogpt_platform/frontend/src/lib/autogpt-server-api/baseClient.ts

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>

* rename hide secrets

* use builtin dict

* delete key

---------

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
2024-10-08 20:20:18 +01:00
Nicholas Tindle 04473cad1e
feat(docs): OAuth docs updates based on google block changes (#8243)
* feat(frontend,backend): testing

* feat: testing

* feat(backend): it works for reading email

* feat(backend): more docs on google

* fix(frontend,backend): formatting

* feat(backend): more logigin (i know this should be debug)

* feat(backend): make real the default scopes

* feat(backend): tests and linting

* fix: code review prep

* feat: sheets block

* feat: liniting

* Update route.ts

* Update autogpt_platform/backend/backend/integrations/oauth/google.py

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>

* Update autogpt_platform/backend/backend/server/routers/integrations.py

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>

* fix: revert opener change

* feat(frontend): add back opener

required to work on mac edge

* feat(frontend): drop typing list import from gmail

* fix: code review comments

* feat: code review changes

* feat: code review changes

* fix(backend): move from asserts to checks so they don't get optimized away in the future

* fix(backend): code review changes

* fix(backend): remove google specific check

* fix: add typing

* fix: only enable google blocks when oauth is configured for google

* fix: errors are real and valid outputs always when output

* fix(backend): add provider detail for debuging scope declines

* Update autogpt_platform/frontend/src/components/integrations/credentials-input.tsx

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>

* fix(frontend): enhance with comment, typeof error isn't known so this is best way to ensure the stringifyication will work

* feat: code review change requests

* fix: linting

* fix: reduce error catching

* fix: doc messages in code

* fix: check the correct scopes object 😄

* fix: remove double (and not needed) try catch

* fix: lint

* fix: scopes

* feat: handle the default scopes better

* feat: better email objectification

* feat: process attachements

turns out an email doesn't need a body

* fix: lint

* Update google.py

* Update autogpt_platform/backend/backend/data/block.py

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>

* fix: quit trying and except failure

* Update autogpt_platform/backend/backend/server/routers/integrations.py

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>

* feat: don't allow expired states

* fix: clarify function name and purpose

* feat: code links updates

* feat: additional docs on adding a block

* fix: type hint missing which means the block won't work

* fix: linting

* fix: docs formatting

* Update issues.py

* fix: improve the naming

* fix: formatting

* Update new_blocks.md

* Update new_blocks.md

* feat: better docs on what the args mean

* feat: more details on yield

* Update new_blocks.md

* fix: remove ignore from docs build

---------

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
2024-10-08 11:11:14 -05:00
Zamil Majdy 2a74381ae8
feat(platform): Add delete agent functionality (#8273) 2024-10-08 16:03:26 +00:00
Aarushi 2aed470d26
tweak(platform): Disable docs endpoint when not local (#8265)
* disable docs endpoint

* add to .env.example

* use enum for app env

* lint
2024-10-08 10:31:08 +01:00
Aarushi 61f1d0cdb5
fix(platform): Always filter on user id (#8275)
* always filter on user id

* add user id to doc string

* fix linting

* fix imports function
2024-10-07 15:47:49 +02:00
Aarushi d742019349
feat(platform,db): Add on deletes (#8272)
add on deletes
2024-10-07 14:44:37 +04:00
Swifty fd9968683c
tweak(platform): Limit templates to users (#8271)
limit templates to users

Co-authored-by: Aarushi <50577581+aarushik93@users.noreply.github.com>
2024-10-07 11:08:51 +01:00
Zamil Majdy daa054c79c
feat(backend): Make Redis connection Sync + Use Redis as Distributed Lock (#8197) 2024-10-07 15:39:32 +07:00
ymrohit a6f763db24
Updated ReplicateFluxAdvancedModel block with the latest Flux 1.1 Pro model (#8266)
Co-authored-by: Toran Bruce Richards <toran.richards@gmail.com>
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
2024-10-07 05:01:38 +00:00
Toran Bruce Richards 07ae912cfe
feat(blocks): Add text to speech block and Unreal Speech API key (#8264) 2024-10-04 14:35:56 +01:00
SuperComboGamer 7501089bbf
feat: Add Ideogram image generation block with customizable options (#8237)
* add ideogram ai image gen

* fixed revid secret api key being removed

* fixed auto checks errors

* Add AI Upscale option to IdeogramModelBlock

- Introduced an 'Upscale Image' option in the input schema to allow users to upscale generated images.
- Created the 'UpscaleOption' enum with options 'AI Upscale' and 'No Upscale'.
- Implemented the 'upscale_image' method to download the generated image into RAM and send it to the Ideogram AI upscale API without saving it to disk.
- Updated the 'run' method to handle the upscaling process based on the user's input.
- Ensured that the image processing is done entirely in memory (RAM) without writing to disk.
- Updated test inputs and mocks to reflect the new 'Upscale Image' option.

---------

Co-authored-by: Aarushi <50577581+aarushik93@users.noreply.github.com>
2024-10-04 10:25:51 +00:00
Zamil Majdy 1e3b1dad06
feat(platform): Add Graph Execution error data & status (#8250) 2024-10-04 09:15:54 +00:00
Nicholas Tindle 6dbc0f7270
feat(frontend,backend): fix google auth and add gmail, sheets (#8236)
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
2024-10-03 16:36:30 +00:00
Zamil Majdy 95e184c85f
fix(platform): Add block data db auto-update (#8249) 2024-10-03 10:10:45 +01:00
Zamil Majdy 78fe578177
feat(platform): Add implicit typing conversion of nested data-structure (#8231) 2024-10-02 20:57:13 +00:00
Toran Bruce Richards 723055ce1d
feat(blocks): Add text decoder block (#8248)
* Refactor search.py: Add option for raw content scraping in ExtractWebsiteContentBlock

* Add TextDecoderBlock for decoding escape sequences in text
2024-10-02 17:26:32 +01:00
vishesh10 24cc5131d7
refactor(backend): Replace deprecated event hooks with lifecycle handler (#8092)
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
2024-10-02 16:33:05 +02:00
Reinier van der Leer 8803740d47
refactor(blocks): Reassign non-random IDs (#8229)
- refactor(blocks): Assign new IDs to 13 blocks
   - Create DB migration to update block IDs in existing DB entities

- feat(frontend): Add `updateBlockIDs` "middleware" to `AgentImportForm` loader in front end
2024-10-01 15:08:42 +01:00
Toran Bruce Richards 1de99ca4df
tweak(blocks): Add option for simple raw content scraping to ExtractWebsiteContentBlock (#8228)
Refactor search.py: Add option for raw content scraping in ExtractWebsiteContentBlock
2024-10-01 01:48:07 +07:00
Toran Bruce Richards a8e5a0d98e
tweak(Blocks): Simplify iteration block to output list items (#8219)
Refactor iteration block to support iterating over dictionaries and to return individual list items.
2024-09-30 13:25:03 +00:00
Toran Bruce Richards 210c140a0f
feat(blocks): Add AI List Generator block (#8221)
Add AIListGeneratorBlock class
2024-09-30 08:05:26 -05:00
Toran Bruce Richards 0cf3b9f48b
feat(blocks): Add shortform video block (#8217)
* feat(blocks): Add AIShortformVideoCreatorBlock

- Added a new block called AIShortformVideoCreatorBlock to create shortform videos using revid.ai.
- The block takes input parameters such as script,  background music, and voice ID.
- It uses the revid.ai API to create the video and waits for completion using a webhook proxy service.
- Once the video is ready, it returns the URL of the created video.
- This block takes anywhere from 5 seconds to several minutes to complete depending on the length of the video.

Add revid.ai API key to Secrets

- Added a new field in the Secrets class to store the revid.ai API key.

* refactor(blocks): Remove unused webhook code in AIShortformVideoCreatorBlock

* Add background music track options.

* Add preset voice options

* Add generation preset and visual style configuration options.

* Remove "morpher" video type due to long generation times and low quality.

Plus extend timeout cut-off.

* Add audio track configuration options.

* refactor AudioTrack selection into single class

* format

* Add test mocks

* run format
2024-09-30 12:31:47 +01:00
Toran Bruce Richards 9dc7bb8497
Remove fluxdev from AI Generation block due to licence restrictions. (#8222) 2024-09-30 10:54:40 +01:00
ymrohit 22fb9bc635
Added Replicate Flux Blocks for image generation (#8216)
* Added Replicate Flux Blocks

* updated poetry lock file for replicate

* Refactor ReplicateFluxAdvancedModelBlock to use an enum for replicate_model_name rather than free strings.

* Refactor ReplicateFluxAdvancedModelBlock to use an enum for output_format instead of free strings

* Refactor ReplicateFluxAdvancedModelBlock to stop requiring people to type a random seed

* Refactor ReplicateFluxAdvancedModelBlock to stop requiring people to type a random seed

* run format

* poetry run format

* Delete ReplicateFluxBasicModelBlock

* Mark model name as not advanced

* tweak input order

* Fix test

---------

Co-authored-by: Toran Bruce Richards <toran.richards@gmail.com>
2024-09-29 22:24:42 +01:00
Bently d7e59966ef
Feat(Builder): Add Google Maps Search Block (#8162)
* Feat(Builder): Add Google Maps Search Block

* format

* Updates to google maps search block

* fixes

* format + updates again

* fix for pytest

* format again

* updates based on new comments

* fix for format?

---------

Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
2024-09-27 21:29:59 +01:00