Commit Graph

20 Commits (8576cad9e5080cf3e57f83b53bb2d97d4fb87bea)

Author SHA1 Message Date
Jeffrey Smith II b819edf095
fix: rename replication fields for better clarity (#24126)
* fix: rename replication fields for better clarity

* fix: dont rename, only add new field
2023-03-09 13:11:43 -05:00
suitableZebraCaller ec7fdd3a58
fix: Show Replication Queue size and Replication TCP Errors (#23960)
* feat: Show remaining replication queue size

* fix: Show non-http related error messages

* fix: Show non-http related error messages with backoff

* fix: Updates for replication tests

* chore: formatting

* chore: formatting

* chore: formatting

* chore: formatting

* chore: lowercase json field

---------

Co-authored-by: Geoffrey <suitableZebraCaller@users.noreply.github.com>
Co-authored-by: Jeffrey Smith II <jeffreyssmith2nd@gmail.com>
2023-02-02 09:47:45 -05:00
Dane Strandboge 6fc66acb0a
fix: do not require remoteOrgID in remote config/creation request (#23838) 2022-11-01 09:47:45 -05:00
Dane Strandboge 55b7d29e4f
fix: sql scan error on remote bucket id when replication to 1.x (#23826) 2022-10-19 14:51:48 -05:00
Jeffrey Smith II 6f50e70960
feat: replicate based on bucket name rather than id (#23638)
* feat: add the ability to replicate based on bucket name rather than bucket id.

- This adds compatibility with 1.x replication targets

* fix: improve error checking and add tests

* fix: add additional constraint to replications table

* fix: use OR not AND for constraint

* feat: delete invalid replications on downgrade

* fix: should be less than 2.4

* test: add test around down migration and cleanup migration code

* fix: use nil instead of platform.ID(1) for better consistency

* fix: fix tests

* fix: fix tests
2022-08-18 14:21:59 -04:00
Jeffrey Smith II 692b0d5153
feat: add instance-id flag for identifying edge nodes (#23447)
* feat: add instance-id flag for identifying edge nodes

* refactor: rename tag to _instance_id
2022-06-16 12:18:11 -04:00
Dane Strandboge 359fcc46b5
feat: add maximum age to replication queues (#23206)
Co-authored-by: Sam Arnold <sarnold@influxdata.com>
2022-03-25 13:06:05 -05:00
mcfarlm3 60234964d0
refactor: replications local write optimization (#22993)
* refactor: eliminate sqlite query in case of no configured replications

* refactor: updated write-related tests to reflect tracking of orgID and localBucket by the queue manager

* refactor: removed redundant trackedReplications field

* refactor: corrected slice init in GetReplications and added TestGetReplications

* refactor: eliminated tracked package and moved TrackedReplication struct to influxdb package via replication.go

* chore: ran make fmt

* fix: added closeRq function back in to address flaky tests

* refactor: small changes to queue manager test based on code review
2021-12-15 12:32:46 -08:00
William Baker 0e5b14fa5e
chore: increase replications batch size limits (#22983) 2021-12-13 11:02:38 -06:00
mcfarlm3 28bcd416b2
feat: batch replications remote writes to avoid payload limit errors (#22914)
* feat: batch replications remote writes appropriately to avoid payload limit errors

* chore: ran make fmt

* chore: fixed staticcheck failure

* refactor: removed batching code from queue manager

* refactor: batch writes before gzip compression

* fix: add in missing bracket after merge

* fix: removed duplicate lines of code from WritePoints function

* feat: add batching functionality for remote writes

* refactor: removed batch index variable
2021-12-02 12:04:10 -08:00
William Baker 9873ccd657
feat: remote write function for replications (#22942)
* feat: remote write function for replications

* chore: implement UpdateResponseInfo store method

* chore: only set gzip heading for non-empty requests

* fix: address review feedback
2021-11-30 15:33:42 -06:00
William Baker f47d514225
refactor: move replications store functionality to separate package (#22923)
* refactor: move replications store functionality to separate package

* fix: make opening all repls on startup work right
2021-11-24 11:45:19 -06:00
William Baker 3a81166812
feat: added metrics collection for replications (#22906)
* feat: added metrics collection for replications

* fix: fixed panic when restarting

* fix: fix panic pt2

* chore: self-review fixes

* chore: simplify test
2021-11-22 11:40:03 -06:00
William Baker ad52815e19
feat: add field for dropping data resulting in non-retryable errors to individual replications (#22885)
* feat: add field for dropping data resulting in non-retryable errors to individual replications
2021-11-16 13:41:54 -07:00
Daniel Moran 6b56af3c3f
feat: mirror writes to registered replications (#22833) 2021-11-10 08:25:47 -05:00
Daniel Moran 1aac92c5ee
refactor: remove replications.current_queue_size_bytes from sqlite (#22832)
Maintaining the current queue size in a SQL column would require
updating the DB on every queue operation. Avoid that contention by
instead looking up the current size on the in-memory durable queue
struct, which is already tracked & updated as data enters & leaves
the queue.
2021-11-05 14:35:12 -04:00
William Baker f7573f43a7
feat: sql migrator can do down migrations (#22806)
* feat: sql down migrations

* refactor: different name for up migrations

* chore: update migrations ref in svc tests

* build: add lint step to verify sql migration names match
2021-11-01 14:30:18 -06:00
mcfarlm3 8825cd5d50
feat: replication apis durable queue management (#22719)
* feat: added durable queue management to replications service

* refactor: improved mapping of replication streams to durable queues

* refactor: modified replication stream durable queues to use user-specified engine path

* chore: generated test mocks for replications DurableQueueManager

* chore: add test coverage for replications durable queue manager

* refactor: made changes based on code review, added mutex to durableQueueManager, improved error logging

* chore: ran make fmt

* refactor: further improvements to error logging
2021-10-26 12:14:29 -07:00
Daniel Moran 153a89dba0
feat: deleting a bucket also deletes all associated replications (#22424) 2021-09-09 15:22:36 -04:00
Daniel Moran 12c8fd28d2
feat: implement metadata management for replications (#22302) 2021-09-01 12:01:41 -04:00