Fixes a few issues:
* flux needs to write to the replication service, instead of the engine directly.
* the replication service incorrectly had value receiver methods, I think this
was just an accident. Pointer receivers make things easier to reason about. Also
with value receivers flux was not picking up the replication config properly.
* The flux to() function previously did not receive the org properly for internal
writes. Previously this was not necessary as the write path only needs the bucket
ID at this level (after authentication). But now we need the org id to look up
replications properly.
Closes#23183
This matches InfluxDB Cloud. The pagination was not exposed to the API,
but meant that API requests were limited to the default 20 pages.
Closes: #21407
* fix: forbid reading OSS buckets for a token with only write permissions
We previously enabled write tokens to also find DBRP buckets, in order to allow
the legacy /write (not /api/v2/write) endpoint to read the DBRP mappings and
find the real bucket id to write to.
This had the unintended consequency of allowing tokens with only write permissions
to read data in buckets via the legacy /query (not /api/v2/query) endpoint with
InfluxQL.
This change fixes the behaviour to allow writing to /write with a write-only
token, while forbidding reading from /query.
* fix: nanosecond precision in tests
* build(flux): update flux to v0.156.0
* chore(flux/schema): update schema tests to assert planner rules
The schema test where updated in Flux, this updates them here so that we
can assert that the planner rules are applied. See note about copied
data.
Co-authored-by: Nathaniel Cook <nvcook42@gmail.com>
* fix: remove nats for scraper processing
Scrapers now use go channels instead of NATS and interprocess communication.
This should fix#23085 .
Additionally, found and fixed#23106 .
* chore: fix formatting
* chore: fix static check and go.mod
* test: fix some flaky tests
* fix: mark NATS arguments as deprecated
* feat: api/v2/config endpoint for runtime config
* feat: use a type switch
* fix: add tests
* chore: add config key to returned json
* chore: update swagger ref
* feat: error out when config file contains invalid options
* feat: debug logging when loading a config file
* fix: only detect flags from 1.x
* test: update tests to use toml configs
* refactor: eliminate sqlite query in case of no configured replications
* refactor: updated write-related tests to reflect tracking of orgID and localBucket by the queue manager
* refactor: removed redundant trackedReplications field
* refactor: corrected slice init in GetReplications and added TestGetReplications
* refactor: eliminated tracked package and moved TrackedReplication struct to influxdb package via replication.go
* chore: ran make fmt
* fix: added closeRq function back in to address flaky tests
* refactor: small changes to queue manager test based on code review
When a file rename fails with EXDEV
(cross device or volume error), copy the
file and delete the original instead.
closes https://github.com/influxdata/influxdb/issues/22890
Co-authored-by: Chris Pahl <sahib@online.de>
* fix: advance replications queue after successful remote writes to prevent data duplication on errors
* fix: loop on sendwrite
* chore: remove flaky test
* chore: add TODO about future optimization
Fixes a race condition in the restore code path that could cause shard data restores to fail. When the race condition occurs, `Error while freeing cold shard resources` appears in the log files.
This is port of PR 22796 from master-1.x to master. Attempts at creating a test case for master failed, so the fix has ported without a corresponding unit test.
fixes#22957
Sam Arnold
Dane Strandboge
Andrew Charlton
Sean Brickley
Paul Hummer
wiedld
mcfarlm3
Jonathan A. Sternberg
Brandon Pfeifer
davidby-influx
William Baker
Christopher M. Wolff
Geoffrey Wossum