* feat: add the ability to replicate based on bucket name rather than bucket id.
- This adds compatibility with 1.x replication targets
* fix: improve error checking and add tests
* fix: add additional constraint to replications table
* fix: use OR not AND for constraint
* feat: delete invalid replications on downgrade
* fix: should be less than 2.4
* test: add test around down migration and cleanup migration code
* fix: use nil instead of platform.ID(1) for better consistency
* fix: fix tests
* fix: fix tests
* feat: start work on remotes/replications phone home data
* feat: add remotes/replications phone home data (no tests
* refactor: use erroring binary conversions
* style: gofmt
* refactor: improve some error handling
* style: cleanup
* feat: add tests
* refactor: just list remotes/replications rather than decrement
* chore: linting fix
Co-authored-by: DStrand1 <dstrandboge@influxdata.com>
Fixes a few issues:
* flux needs to write to the replication service, instead of the engine directly.
* the replication service incorrectly had value receiver methods, I think this
was just an accident. Pointer receivers make things easier to reason about. Also
with value receivers flux was not picking up the replication config properly.
* The flux to() function previously did not receive the org properly for internal
writes. Previously this was not necessary as the write path only needs the bucket
ID at this level (after authentication). But now we need the org id to look up
replications properly.
Closes#23183
* fix: remove nats for scraper processing
Scrapers now use go channels instead of NATS and interprocess communication.
This should fix#23085 .
Additionally, found and fixed#23106 .
* chore: fix formatting
* chore: fix static check and go.mod
* test: fix some flaky tests
* fix: mark NATS arguments as deprecated
* refactor: eliminate sqlite query in case of no configured replications
* refactor: updated write-related tests to reflect tracking of orgID and localBucket by the queue manager
* refactor: removed redundant trackedReplications field
* refactor: corrected slice init in GetReplications and added TestGetReplications
* refactor: eliminated tracked package and moved TrackedReplication struct to influxdb package via replication.go
* chore: ran make fmt
* fix: added closeRq function back in to address flaky tests
* refactor: small changes to queue manager test based on code review
* fix: advance replications queue after successful remote writes to prevent data duplication on errors
* fix: loop on sendwrite
* chore: remove flaky test
* chore: add TODO about future optimization
* feat: remote write function for replications
* chore: implement UpdateResponseInfo store method
* chore: only set gzip heading for non-empty requests
* fix: address review feedback
* feat: added replications queue management to launcher tasks
* refactor: separated sql logic into replications service rather than durable queue manager
* refactor: extended replications feature flag to launcher code and minor change to startup function param
* chore: added unit test coverage for replications server startup queue management
* refactor: made error messages reusable and factored out unecessary string from queue management tests
* refactor: changed queue management error names to pass linter check
Maintaining the current queue size in a SQL column would require
updating the DB on every queue operation. Avoid that contention by
instead looking up the current size on the in-memory durable queue
struct, which is already tracked & updated as data enters & leaves
the queue.
* feat: sql down migrations
* refactor: different name for up migrations
* chore: update migrations ref in svc tests
* build: add lint step to verify sql migration names match
* feat: added durable queue management to replications service
* refactor: improved mapping of replication streams to durable queues
* refactor: modified replication stream durable queues to use user-specified engine path
* chore: generated test mocks for replications DurableQueueManager
* chore: add test coverage for replications durable queue manager
* refactor: made changes based on code review, added mutex to durableQueueManager, improved error logging
* chore: ran make fmt
* refactor: further improvements to error logging
suitableZebraCaller
Jeffrey Smith II
Ole Kristian (Zee)
Dane Strandboge
Sam Arnold
William Baker
mcfarlm3
Dane Strandboge
Daniel Moran