AutoGPT/autogpt_platform/docker-compose.platform.yml

223 lines
7.7 KiB
YAML
Raw Normal View History

services:
migrate:
build:
context: ../
dockerfile: autogpt_platform/backend/Dockerfile
target: server
command: ["sh", "-c", "poetry run prisma migrate deploy"]
develop:
watch:
- path: ./
refactor(backend): Move credentials storage to prisma user (#8283) * feat(frontend,backend): testing * feat: testing * feat(backend): it works for reading email * feat(backend): more docs on google * fix(frontend,backend): formatting * feat(backend): more logigin (i know this should be debug) * feat(backend): make real the default scopes * feat(backend): tests and linting * fix: code review prep * feat: sheets block * feat: liniting * Update route.ts * Update autogpt_platform/backend/backend/integrations/oauth/google.py Co-authored-by: Reinier van der Leer <pwuts@agpt.co> * Update autogpt_platform/backend/backend/server/routers/integrations.py Co-authored-by: Reinier van der Leer <pwuts@agpt.co> * fix: revert opener change * feat(frontend): add back opener required to work on mac edge * feat(frontend): drop typing list import from gmail * fix: code review comments * feat: code review changes * feat: code review changes * fix(backend): move from asserts to checks so they don't get optimized away in the future * fix(backend): code review changes * fix(backend): remove google specific check * fix: add typing * fix: only enable google blocks when oauth is configured for google * fix: errors are real and valid outputs always when output * fix(backend): add provider detail for debuging scope declines * Update autogpt_platform/frontend/src/components/integrations/credentials-input.tsx Co-authored-by: Reinier van der Leer <pwuts@agpt.co> * fix(frontend): enhance with comment, typeof error isn't known so this is best way to ensure the stringifyication will work * feat: code review change requests * fix: linting * fix: reduce error catching * fix: doc messages in code * fix: check the correct scopes object :smile: * fix: remove double (and not needed) try catch * fix: lint * fix: scopes * feat: handle the default scopes better * feat: better email objectification * feat: process attachements turns out an email doesn't need a body * fix: lint * Update google.py * Update autogpt_platform/backend/backend/data/block.py Co-authored-by: Reinier van der Leer <pwuts@agpt.co> * fix: quit trying and except failure * Update autogpt_platform/backend/backend/server/routers/integrations.py Co-authored-by: Reinier van der Leer <pwuts@agpt.co> * feat: don't allow expired states * fix: clarify function name and purpose * feat: code links updates * feat: additional docs on adding a block * fix: type hint missing which means the block won't work * fix: linting * fix: docs formatting * Update issues.py * fix: improve the naming * fix: formatting * Update new_blocks.md * Update new_blocks.md * feat: better docs on what the args mean * feat: more details on yield * Update new_blocks.md * fix: remove ignore from docs build * feat: initial migration * feat: migration tested with supabase-> prisma data location * add custom migrations and script * update migration command * formatting and linting * updated migration script * add direct db url * add find files * rename * use binary instead of source * temp adding supabase * remove unused functions * adding missed merge * fix: commit hash for lock * ci: fix lint * fix: minor bugs that prevented connecting and migrating to dbs and auth * fix: linting * fix: missed await * fix(backend): phase one pr updates * fix: handle error with returning user object from database_manager * fix: linting * Address comments * Make the migration safe * Update migration doc * Move misplaced model functions * Grammar * Revert lock * Remove irrelevant changes * Remove irrelevant changes * Avoid adding trigger on public schema --------- Co-authored-by: Reinier van der Leer <pwuts@agpt.co> Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co> Co-authored-by: Aarushi <aarushik93@gmail.com> Co-authored-by: Aarushi <50577581+aarushik93@users.noreply.github.com>
2024-10-22 12:33:54 +00:00
target: autogpt_platform/backend/migrations
action: rebuild
depends_on:
db:
condition: service_healthy
environment:
- DATABASE_URL=postgresql://postgres:your-super-secret-and-long-postgres-password@db:5432/postgres?connect_timeout=60&schema=platform
networks:
- app-network
restart: on-failure
healthcheck:
test: ["CMD", "poetry", "run", "prisma", "migrate", "status"]
interval: 10s
timeout: 5s
retries: 5
redis:
image: redis:latest
command: redis-server --requirepass password
ports:
- "6379:6379"
networks:
- app-network
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 5s
retries: 5
rest_server:
build:
context: ../
dockerfile: autogpt_platform/backend/Dockerfile
target: server
command: ["python", "-m", "backend.rest"]
develop:
watch:
- path: ./
target: autogpt_platform/backend/
action: rebuild
depends_on:
redis:
condition: service_healthy
db:
condition: service_healthy
migrate:
condition: service_completed_successfully
environment:
- SUPABASE_URL=http://kong:8000
- SUPABASE_JWT_SECRET=your-super-secret-jwt-token-with-at-least-32-characters-long
feat(platform): OAuth support + API key management + GitHub blocks (#8044) ## Config - For Supabase, the back end needs `SUPABASE_URL`, `SUPABASE_SERVICE_ROLE_KEY`, and `SUPABASE_JWT_SECRET` - For the GitHub integration to work, the back end needs `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` - For integrations OAuth flows to work in local development, the back end needs `FRONTEND_BASE_URL` to generate login URLs with accurate redirect URLs ## REST API - Tweak output of OAuth `/login` endpoint: add `state_token` separately in response - Add `POST /integrations/{provider}/credentials` (for API keys) - Add `DELETE /integrations/{provider}/credentials/{cred_id}` ## Back end - Add Supabase support to `AppService` - Add `FRONTEND_BASE_URL` config option, mainly for local development use ### `autogpt_libs.supabase_integration_credentials_store` - Add `CredentialsType` alias - Add `.bearer()` helper methods to `APIKeyCredentials` and `OAuth2Credentials` ### Blocks - Add `CredentialsField(..) -> CredentialsMetaInput` ## Front end ### UI components - `CredentialsInput` for use on `CustomNode`: allows user to add/select credentials for a service. - `APIKeyCredentialsModal`: a dialog for creating API keys - `OAuth2FlowWaitingModal`: a dialog to indicate that the application is waiting for the user to log in to the 3rd party service in the provided pop-up window - `NodeCredentialsInput`: wrapper for `CredentialsInput` with the "usual" interface of node input components - New icons: `IconKey`, `IconKeyPlus`, `IconUser`, `IconUserPlus` ### Data model - `CredentialsProvider`: introduces the app-level `CredentialsProvidersContext`, which acts as an application-wide store and cache for credentials metadata. - `useCredentials` for use on `CustomNode`: uses `CredentialsProvidersContext` and provides node-specific credential data and provider-specific data/functions - `/auth/integrations/oauth_callback` route to close the loop to the `CredentialsInput` after a user completes sign-in to the external service - Add `BlockIOCredentialsSubSchema` ### API client - Add `isAuthenticated` method - Add methods for integration OAuth flow: `oAuthLogin`, `oAuthCallback` - Add CRD methods for credentials: `createAPIKeyCredentials`, `listCredentials`, `getCredentials`, `deleteCredentials` - Add mirrored types `CredentialsMetaResponse`, `CredentialsMetaInput`, `OAuth2Credentials`, `APIKeyCredentials` - Add GitHub blocks + "DEVELOPER_TOOLS" category - Add `**kwargs` to `Block.run(..)` signature to support additional kwargs - Add support for loading blocks from nested modules (e.g. `blocks/github/issues.py`) #### Executor - Add strict support for `credentials` fields on blocks - Fetch credentials for graph execution and pass them down through to the node execution
2024-09-25 22:36:29 +00:00
- SUPABASE_SERVICE_ROLE_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyAgCiAgICAicm9sZSI6ICJzZXJ2aWNlX3JvbGUiLAogICAgImlzcyI6ICJzdXBhYmFzZS1kZW1vIiwKICAgICJpYXQiOiAxNjQxNzY5MjAwLAogICAgImV4cCI6IDE3OTk1MzU2MDAKfQ.DaYlNEoUrrEn2Ig7tqibS-PHK5vgusbcbo7X36XVt4Q
- DATABASE_URL=postgresql://postgres:your-super-secret-and-long-postgres-password@db:5432/postgres?connect_timeout=60&schema=platform
- REDIS_HOST=redis
- REDIS_PORT=6379
- REDIS_PASSWORD=password
- ENABLE_AUTH=true
- PYRO_HOST=0.0.0.0
- EXECUTIONMANAGER_HOST=executor
- DATABASEMANAGER_HOST=executor
feat(platform): OAuth support + API key management + GitHub blocks (#8044) ## Config - For Supabase, the back end needs `SUPABASE_URL`, `SUPABASE_SERVICE_ROLE_KEY`, and `SUPABASE_JWT_SECRET` - For the GitHub integration to work, the back end needs `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` - For integrations OAuth flows to work in local development, the back end needs `FRONTEND_BASE_URL` to generate login URLs with accurate redirect URLs ## REST API - Tweak output of OAuth `/login` endpoint: add `state_token` separately in response - Add `POST /integrations/{provider}/credentials` (for API keys) - Add `DELETE /integrations/{provider}/credentials/{cred_id}` ## Back end - Add Supabase support to `AppService` - Add `FRONTEND_BASE_URL` config option, mainly for local development use ### `autogpt_libs.supabase_integration_credentials_store` - Add `CredentialsType` alias - Add `.bearer()` helper methods to `APIKeyCredentials` and `OAuth2Credentials` ### Blocks - Add `CredentialsField(..) -> CredentialsMetaInput` ## Front end ### UI components - `CredentialsInput` for use on `CustomNode`: allows user to add/select credentials for a service. - `APIKeyCredentialsModal`: a dialog for creating API keys - `OAuth2FlowWaitingModal`: a dialog to indicate that the application is waiting for the user to log in to the 3rd party service in the provided pop-up window - `NodeCredentialsInput`: wrapper for `CredentialsInput` with the "usual" interface of node input components - New icons: `IconKey`, `IconKeyPlus`, `IconUser`, `IconUserPlus` ### Data model - `CredentialsProvider`: introduces the app-level `CredentialsProvidersContext`, which acts as an application-wide store and cache for credentials metadata. - `useCredentials` for use on `CustomNode`: uses `CredentialsProvidersContext` and provides node-specific credential data and provider-specific data/functions - `/auth/integrations/oauth_callback` route to close the loop to the `CredentialsInput` after a user completes sign-in to the external service - Add `BlockIOCredentialsSubSchema` ### API client - Add `isAuthenticated` method - Add methods for integration OAuth flow: `oAuthLogin`, `oAuthCallback` - Add CRD methods for credentials: `createAPIKeyCredentials`, `listCredentials`, `getCredentials`, `deleteCredentials` - Add mirrored types `CredentialsMetaResponse`, `CredentialsMetaInput`, `OAuth2Credentials`, `APIKeyCredentials` - Add GitHub blocks + "DEVELOPER_TOOLS" category - Add `**kwargs` to `Block.run(..)` signature to support additional kwargs - Add support for loading blocks from nested modules (e.g. `blocks/github/issues.py`) #### Executor - Add strict support for `credentials` fields on blocks - Fetch credentials for graph execution and pass them down through to the node execution
2024-09-25 22:36:29 +00:00
- FRONTEND_BASE_URL=http://localhost:3000
- BACKEND_CORS_ALLOW_ORIGINS=["http://localhost:3000"]
- ENCRYPTION_KEY=dvziYgz0KSK8FENhju0ZYi8-fRTfAdlz6YLhdB_jhNw= # DO NOT USE IN PRODUCTION!!
ports:
- "8006:8006"
- "8003:8003" # execution scheduler
networks:
- app-network
executor:
build:
context: ../
dockerfile: autogpt_platform/backend/Dockerfile
target: server
command: ["python", "-m", "backend.exec"]
develop:
watch:
- path: ./
target: autogpt_platform/backend/
action: rebuild
depends_on:
redis:
condition: service_healthy
db:
condition: service_healthy
migrate:
condition: service_completed_successfully
environment:
feat(platform): OAuth support + API key management + GitHub blocks (#8044) ## Config - For Supabase, the back end needs `SUPABASE_URL`, `SUPABASE_SERVICE_ROLE_KEY`, and `SUPABASE_JWT_SECRET` - For the GitHub integration to work, the back end needs `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` - For integrations OAuth flows to work in local development, the back end needs `FRONTEND_BASE_URL` to generate login URLs with accurate redirect URLs ## REST API - Tweak output of OAuth `/login` endpoint: add `state_token` separately in response - Add `POST /integrations/{provider}/credentials` (for API keys) - Add `DELETE /integrations/{provider}/credentials/{cred_id}` ## Back end - Add Supabase support to `AppService` - Add `FRONTEND_BASE_URL` config option, mainly for local development use ### `autogpt_libs.supabase_integration_credentials_store` - Add `CredentialsType` alias - Add `.bearer()` helper methods to `APIKeyCredentials` and `OAuth2Credentials` ### Blocks - Add `CredentialsField(..) -> CredentialsMetaInput` ## Front end ### UI components - `CredentialsInput` for use on `CustomNode`: allows user to add/select credentials for a service. - `APIKeyCredentialsModal`: a dialog for creating API keys - `OAuth2FlowWaitingModal`: a dialog to indicate that the application is waiting for the user to log in to the 3rd party service in the provided pop-up window - `NodeCredentialsInput`: wrapper for `CredentialsInput` with the "usual" interface of node input components - New icons: `IconKey`, `IconKeyPlus`, `IconUser`, `IconUserPlus` ### Data model - `CredentialsProvider`: introduces the app-level `CredentialsProvidersContext`, which acts as an application-wide store and cache for credentials metadata. - `useCredentials` for use on `CustomNode`: uses `CredentialsProvidersContext` and provides node-specific credential data and provider-specific data/functions - `/auth/integrations/oauth_callback` route to close the loop to the `CredentialsInput` after a user completes sign-in to the external service - Add `BlockIOCredentialsSubSchema` ### API client - Add `isAuthenticated` method - Add methods for integration OAuth flow: `oAuthLogin`, `oAuthCallback` - Add CRD methods for credentials: `createAPIKeyCredentials`, `listCredentials`, `getCredentials`, `deleteCredentials` - Add mirrored types `CredentialsMetaResponse`, `CredentialsMetaInput`, `OAuth2Credentials`, `APIKeyCredentials` - Add GitHub blocks + "DEVELOPER_TOOLS" category - Add `**kwargs` to `Block.run(..)` signature to support additional kwargs - Add support for loading blocks from nested modules (e.g. `blocks/github/issues.py`) #### Executor - Add strict support for `credentials` fields on blocks - Fetch credentials for graph execution and pass them down through to the node execution
2024-09-25 22:36:29 +00:00
- SUPABASE_URL=http://kong:8000
- SUPABASE_JWT_SECRET=your-super-secret-jwt-token-with-at-least-32-characters-long
feat(platform): OAuth support + API key management + GitHub blocks (#8044) ## Config - For Supabase, the back end needs `SUPABASE_URL`, `SUPABASE_SERVICE_ROLE_KEY`, and `SUPABASE_JWT_SECRET` - For the GitHub integration to work, the back end needs `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` - For integrations OAuth flows to work in local development, the back end needs `FRONTEND_BASE_URL` to generate login URLs with accurate redirect URLs ## REST API - Tweak output of OAuth `/login` endpoint: add `state_token` separately in response - Add `POST /integrations/{provider}/credentials` (for API keys) - Add `DELETE /integrations/{provider}/credentials/{cred_id}` ## Back end - Add Supabase support to `AppService` - Add `FRONTEND_BASE_URL` config option, mainly for local development use ### `autogpt_libs.supabase_integration_credentials_store` - Add `CredentialsType` alias - Add `.bearer()` helper methods to `APIKeyCredentials` and `OAuth2Credentials` ### Blocks - Add `CredentialsField(..) -> CredentialsMetaInput` ## Front end ### UI components - `CredentialsInput` for use on `CustomNode`: allows user to add/select credentials for a service. - `APIKeyCredentialsModal`: a dialog for creating API keys - `OAuth2FlowWaitingModal`: a dialog to indicate that the application is waiting for the user to log in to the 3rd party service in the provided pop-up window - `NodeCredentialsInput`: wrapper for `CredentialsInput` with the "usual" interface of node input components - New icons: `IconKey`, `IconKeyPlus`, `IconUser`, `IconUserPlus` ### Data model - `CredentialsProvider`: introduces the app-level `CredentialsProvidersContext`, which acts as an application-wide store and cache for credentials metadata. - `useCredentials` for use on `CustomNode`: uses `CredentialsProvidersContext` and provides node-specific credential data and provider-specific data/functions - `/auth/integrations/oauth_callback` route to close the loop to the `CredentialsInput` after a user completes sign-in to the external service - Add `BlockIOCredentialsSubSchema` ### API client - Add `isAuthenticated` method - Add methods for integration OAuth flow: `oAuthLogin`, `oAuthCallback` - Add CRD methods for credentials: `createAPIKeyCredentials`, `listCredentials`, `getCredentials`, `deleteCredentials` - Add mirrored types `CredentialsMetaResponse`, `CredentialsMetaInput`, `OAuth2Credentials`, `APIKeyCredentials` - Add GitHub blocks + "DEVELOPER_TOOLS" category - Add `**kwargs` to `Block.run(..)` signature to support additional kwargs - Add support for loading blocks from nested modules (e.g. `blocks/github/issues.py`) #### Executor - Add strict support for `credentials` fields on blocks - Fetch credentials for graph execution and pass them down through to the node execution
2024-09-25 22:36:29 +00:00
- SUPABASE_SERVICE_ROLE_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyAgCiAgICAicm9sZSI6ICJzZXJ2aWNlX3JvbGUiLAogICAgImlzcyI6ICJzdXBhYmFzZS1kZW1vIiwKICAgICJpYXQiOiAxNjQxNzY5MjAwLAogICAgImV4cCI6IDE3OTk1MzU2MDAKfQ.DaYlNEoUrrEn2Ig7tqibS-PHK5vgusbcbo7X36XVt4Q
- DATABASE_URL=postgresql://postgres:your-super-secret-and-long-postgres-password@db:5432/postgres?connect_timeout=60&schema=platform
- REDIS_HOST=redis
- REDIS_PORT=6379
- REDIS_PASSWORD=password
- ENABLE_AUTH=true
- PYRO_HOST=0.0.0.0
- AGENTSERVER_HOST=rest_server
- DATABASEMANAGER_HOST=0.0.0.0
- ENCRYPTION_KEY=dvziYgz0KSK8FENhju0ZYi8-fRTfAdlz6YLhdB_jhNw= # DO NOT USE IN PRODUCTION!!
ports:
- "8002:8000"
networks:
- app-network
websocket_server:
build:
context: ../
dockerfile: autogpt_platform/backend/Dockerfile
target: server
command: ["python", "-m", "backend.ws"]
develop:
watch:
- path: ./
target: autogpt_platform/backend/
action: rebuild
depends_on:
db:
condition: service_healthy
redis:
condition: service_healthy
migrate:
condition: service_completed_successfully
environment:
- SUPABASE_JWT_SECRET=your-super-secret-jwt-token-with-at-least-32-characters-long
- DATABASE_URL=postgresql://postgres:your-super-secret-and-long-postgres-password@db:5432/postgres?connect_timeout=60&schema=platform
- REDIS_HOST=redis
- REDIS_PORT=6379
- REDIS_PASSWORD=password
- ENABLE_AUTH=true
- PYRO_HOST=0.0.0.0
- BACKEND_CORS_ALLOW_ORIGINS=["http://localhost:3000"]
ports:
- "8001:8001"
networks:
- app-network
market:
build:
context: ../
dockerfile: autogpt_platform/market/Dockerfile
develop:
watch:
- path: ./
target: autogpt_platform/market/
action: rebuild
depends_on:
db:
condition: service_healthy
market-migrations:
condition: service_completed_successfully
environment:
- SUPABASE_URL=http://kong:8000
- SUPABASE_JWT_SECRET=your-super-secret-jwt-token-with-at-least-32-characters-long
- SUPABASE_ANON_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyAgCiAgICAicm9sZSI6ICJhbm9uIiwKICAgICJpc3MiOiAic3VwYWJhc2UtZGVtbyIsCiAgICAiaWF0IjogMTY0MTc2OTIwMCwKICAgICJleHAiOiAxNzk5NTM1NjAwCn0.dc_X5iR_VP_qT0zsiyj_I_OZ2T9FtRU2BBNWN8Bu4GE
- DATABASE_URL=postgresql://postgres:your-super-secret-and-long-postgres-password@db:5432/postgres?connect_timeout=60&schema=market
- BACKEND_CORS_ALLOW_ORIGINS="http://localhost:3000,http://127.0.0.1:3000"
ports:
- "8015:8015"
networks:
- app-network
market-migrations:
build:
context: ../
dockerfile: autogpt_platform/market/Dockerfile
command: ["sh", "-c", "poetry run prisma migrate deploy"]
develop:
watch:
- path: ./
target: autogpt_platform/market/
action: rebuild
depends_on:
db:
condition: service_healthy
environment:
- SUPABASE_URL=http://kong:8000
- SUPABASE_JWT_SECRET=your-super-secret-jwt-token-with-at-least-32-characters-long
- SUPABASE_ANON_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyAgCiAgICAicm9sZSI6ICJhbm9uIiwKICAgICJpc3MiOiAic3VwYWJhc2UtZGVtbyIsCiAgICAiaWF0IjogMTY0MTc2OTIwMCwKICAgICJleHAiOiAxNzk5NTM1NjAwCn0.dc_X5iR_VP_qT0zsiyj_I_OZ2T9FtRU2BBNWN8Bu4GE
- DATABASE_URL=postgresql://postgres:your-super-secret-and-long-postgres-password@db:5432/postgres?connect_timeout=60&schema=market
networks:
- app-network
# frontend:
# build:
# context: ../
# dockerfile: autogpt_platform/frontend/Dockerfile
# target: dev
# depends_on:
# db:
# condition: service_healthy
# rest_server:
# condition: service_started
# websocket_server:
# condition: service_started
# migrate:
# condition: service_completed_successfully
# environment:
# - NEXT_PUBLIC_SUPABASE_URL=http://kong:8000
# - NEXT_PUBLIC_SUPABASE_ANON_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyAgCiAgICAicm9sZSI6ICJhbm9uIiwKICAgICJpc3MiOiAic3VwYWJhc2UtZGVtbyIsCiAgICAiaWF0IjogMTY0MTc2OTIwMCwKICAgICJleHAiOiAxNzk5NTM1NjAwCn0.dc_X5iR_VP_qT0zsiyj_I_OZ2T9FtRU2BBNWN8Bu4GE
# - DATABASE_URL=postgresql://agpt_user:pass123@postgres:5432/postgres?connect_timeout=60&schema=platform
# - NEXT_PUBLIC_AGPT_SERVER_URL=http://localhost:8006/api
# - NEXT_PUBLIC_AGPT_WS_SERVER_URL=ws://localhost:8001/ws
# - NEXT_PUBLIC_AGPT_MARKETPLACE_URL=http://localhost:8015/api/v1/market
# - NEXT_PUBLIC_BEHAVE_AS=LOCAL
# ports:
# - "3000:3000"
# networks:
# - app-network
networks:
app-network:
driver: bridge