b0cc2bda0d
In #7559 I introduced code to coalesce N queries for a given partition down to a single query, with the other callers (async) blocking to obtain the shared result. This is usually fine, but it is hypothetically possible for all callers to start the partition lookup and abort (cancelling their shared poll future) before the partition resolves, leaving it half complete. The problem is this half-complete future MAY be holding a connection from the catalog connection pool (internally a semaphore). This means this connection is "stuck", lent out to this future that isn't making progress (and may never make progress!). Given sufficient occurrences, it's possible this could lead to starving the system of catalog connections. This fix is simple - spawn the resolve future into the runtime so it completes irrespective of any caller polling it (they only poll for the result, not to drive it forward). NOTE: this isn't practically possible, because writes that traverse the WAL DmlSink currently become uncancellable - this may not always be the case however, so this underlying foot gun should be patched over! |
||
|---|---|---|
| .. | ||
| benches | ||
| src | ||
| tests | ||
| Cargo.toml | ||
| README.md | ||
README.md
ingester2
Quick run
Set-up empty catalog db:
mkdir -p /tmp/iox/{wal,obj}
createdb iox_shared
./target/debug/influxdb_iox catalog setup --catalog-dsn postgres:///iox_shared
# there has to exist one "topic", see https://github.com/influxdata/influxdb_iox/issues/6420
psql 'dbname=iox_shared options=-csearch_path=public,iox_catalog' -c "insert into topic (name) values ('iox-shared')"
Run ingester2:
INFLUXDB_IOX_RPC_MODE=2 ./target/debug/influxdb_iox run ingester2 --api-bind=127.0.0.1:8081 --grpc-bind=127.0.0.1:8042 --wal-directory /tmp/iox/wal --catalog-dsn postgres:///iox_shared --object-store=file --data-dir=/tmp/iox/obj -v
Run router2:
INFLUXDB_IOX_RPC_MODE=2 ./target/debug/influxdb_iox run router2 --api-bind=127.0.0.1:8080 --grpc-bind=127.0.0.1:8085 --ingester-addresses=127.0.0.1:8042 --catalog-dsn postgres:///iox_shared -v
Run querier:
INFLUXDB_IOX_RPC_MODE=2 ./target/debug/influxdb_iox run querier --ingester-addresses=http://127.0.0.1:8042 --api-bind 127.0.0.1:8083 --grpc-bind 127.0.0.1:8082 --catalog-dsn postgres:///iox_shared --object-store=file --data-dir=/tmp/iox/obj -v