Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
5,624 Commits
189 Branches
453 Tags
380 MiB
Commit Graph

72 Commits (e80b902d15b750b8322166c4de6cc2b6a3d36620)

Author SHA1 Message Date
Marco Neumann bc7244c48e chore: use Rust edition 2021 2021-10-25 10:58:20 +02:00
Edd Robinson fe5eea37da
refactor: downgrade logs to trace (#2912) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-20 12:41:51 +00:00
Raphael Taylor-Davies d6b7b56f16
refactor: pull lifecycle out of Db (#2242) (#2831) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-15 13:08:00 +00:00
Raphael Taylor-Davies 8a82f92c5d
refactor: add TimeProvider abstraction (#2722) (#2815) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-12 21:19:03 +00:00
Raphael Taylor-Davies 5b69bb0d72
feat: reduce lifecycle lock scope (#2242) (#2810) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-12 17:34:16 +00:00
Raphael Taylor-Davies 3dfe400e6b
feat: migrate write path to TimeProvider (#2722) (#2807) 2021-10-12 12:09:08 +00:00
Raphael Taylor-Davies b39e01f7ba
feat: migrate PersistenceWindows to TimeProvider (#2722) (#2798) 2021-10-11 20:40:00 +00:00
Marco Neumann ae0acf0024 refactor: remove `db_name` param from `select_persistable_chunks` 
This was only used for logging but is already part of `ChunkAddr`.
 2021-10-11 17:01:28 +02:00
Marco Neumann 8ab3ffde8c docs: explain error handling for `select_persistable_chunks` 2021-10-11 17:01:28 +02:00
Marco Neumann 9ff2213ecb fix: typo 
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
 2021-10-11 17:01:28 +02:00
Marco Neumann 3561dfa16c refactor: reduce vector re-allocations 
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2021-10-11 17:01:28 +02:00
Marco Neumann 8185feddb9 fix: do not break chunk ordering during persistence 
Fixes #2729.
 2021-10-11 17:01:28 +02:00
Marco Neumann d3de6bb6e4 refactor: `max_persisted_timestamp` => `flush_timestamp` 
There might be data left before this timestamp that wasn't persisted
(e.g. incoming data while the persistence was running).
 2021-10-08 12:36:23 +02:00
Marco Neumann 63d74be490 refactor: make `ChunkId` a UUID 2021-10-07 10:23:27 +02:00
Raphael Taylor-Davies 39157828b1
feat: remove remaining usages of Instant (#2722) (#2749) 
* feat: remove remaining usages of Instant (#2722)

* chore: review feedback

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-06 16:44:02 +00:00
Raphael Taylor-Davies ce5b24e65d
refactor: use DateTime<Utc> in PersistenceWindows (#2722) (#2743) 
* refactor: use DateTime<Utc> in PersistenceWindows (#2722)

* chore: fix benchmark

* chore: fmt

* chore: review feedback
 2021-10-06 09:39:32 +00:00
Raphael Taylor-Davies d0929e3a34
feat: persist no chunks (#2712) (#2718) 
* feat: persist no chunks (#2712)

* fix: persist partition

* fix: chunk ordering test

* chore: fix logical conflict

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-05 15:18:35 +00:00
Marco Neumann 97881079e8 refactor: make `ChunkOrder` non-zero 
This will make it easier to handle missing values.

Helps with #2633.
 2021-10-04 17:49:12 +02:00
Raphael Taylor-Davies b402423e9e
feat: remove move lifecycle action (#2674) 
* feat: remove move_chunk lifecycle action

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-30 16:58:05 +00:00
Marco Neumann cef5aeee52 refactor: introduce `ChunkId` type 2021-09-20 13:10:41 +02:00
Marco Neumann becef1c75f refactor: introduce `ChunkOrder` type 2021-09-14 17:10:23 +02:00
Marco Neumann 0dc5bd0921 fix: remove overstrict check for chunk orders in lifecycle policy 2021-09-14 13:00:55 +02:00
Marco Neumann 8e360f0c5a refactor: inline `sort_chunks` since it is only used once 2021-09-14 13:00:55 +02:00
Marco Neumann aaeb67ae5d refactor: make chunk iterations sorted by `order, ID` 2021-09-14 13:00:55 +02:00
Marco Neumann 804790711b refactor: isolate `sort_chunks` 2021-09-14 13:00:55 +02:00
Marco Neumann c28f38309a docs: improve chunk ordering docs 2021-09-14 13:00:55 +02:00
Marco Neumann 96618af6a2 fix: respect chunk order when invoking lifecycle actions 2021-09-14 13:00:55 +02:00
dependabot[bot] b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 09:11:38 +00:00
Marco Neumann 95a739272e refactor: move lifecycle policy away from `drop_non_persisted` and only consider `persist` 2021-08-17 16:53:45 +02:00
Dom 3de6b44e23
build: use new rustdoc lint name (#2261) 
* fix: nocache feature code rot

The MBChunk::snapshot code when using the "nocache" option no longer
compiles - this commit updates it to match the not(nocache) code.

* build: use updated broken_intra_doc_links name

The broken_intra_doc_links lint was renamed
rustdoc::broken_intra_doc_links

https://doc.rust-lang.org/rustdoc/lints.html
 2021-08-11 19:48:51 +00:00
Marco Neumann 6f163aba13 fix: remove unnecessary Copy requirement for Debug 2021-08-04 11:33:46 +02:00
Marco Neumann d1a4584dfc feat: easy way to suppress persitence from lifecycle policy 2021-08-04 11:33:46 +02:00
Marko Mikulicic fe7f65bfa7
feat(iox): Implement max_active_compactions_cpu_fraction 2021-07-28 17:31:17 +02:00
Carol (Nichols || Goulding) 8add00e761 feat: Make CatalogChunk first/last write times required 
Connects to #1927.
 2021-07-28 09:22:06 -04:00
Carol (Nichols || Goulding) 7c9a21632b refactor: Organize uses 2021-07-28 09:22:04 -04:00
Raphael Taylor-Davies 844a025c7c
feat: drop based on LRU (#2075) (#2092) 
* feat: drop based on LRU (#2075)

* chore: review feedback

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-23 08:31:28 +00:00
Edd Robinson cc0aaa58a7 test: ensure high enough limit 2021-07-20 12:43:10 +01:00
Edd Robinson 2d6a23f7c3 fix: avoid potential race 2021-07-19 14:02:29 +01:00
Edd Robinson dfda23f24a test: update e2e tests 2021-07-19 14:00:10 +01:00
Edd Robinson dd93b2cdec feat: add db-level compaction limiter 2021-07-19 14:00:10 +01:00
Marco Neumann 71b5030fc0 refactor: remove unused `LockableChunk::write_to_object_store` 2021-07-16 11:45:34 +02:00
Raphael Taylor-Davies a79c0b4e75
feat: add mub row count threshold to lifecycle rules (#1876) (#2016) 
* feat: add mub row count threshold to lifecycle rules (#1876)

* chore: update docstring

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-15 13:42:17 +00:00
Marco Neumann 77a9191a11 fix: chunk dropping over lifecycle policy should also respect the preserved catalog 2021-07-15 12:07:56 +02:00
Raphael Taylor-Davies f1c1620c84
feat: make persistence windows interface harder to use incorrectly (#1977) 
* feat: make persistence windows interface harder to use incorrectly

* chore: review feedback

* chore: update comment

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-14 13:03:18 +00:00
Raphael Taylor-Davies 5a0caeab44
feat: skip over fully persisted partitions (#1962) (#1973) 
* feat: skip over fully persisted partitions (#1962)

* chore: review feedback
 2021-07-13 10:40:45 +00:00
Andrew Lamb d35b74c226
fix: Fix doc build warnings (#1945) 
* fix: Fix doc build warnings

* refactor: add deny bare_urls to crates

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-13 08:03:42 +00:00
Raphael Taylor-Davies a54cd04b30
feat: log reason for triggering persistence (#1957) (#1967) 
* feat: log reason for triggering persistence (#1957)

Don't panic on no eligible chunks to persist (#1966)

* fix: fix conditions and logging

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-12 18:28:00 +00:00
Paul Dix 0c8c81a321 refactor: remove mutable_linger_seconds from lifecycle 
The interplay between mutable_linger_seconds, late_arrive_window and persist_age_threshold_seconds can be tricky to reason about. I realized that the lifecycle rules can be simplified by removing mutable_linger_seconds and instead using late_arrive_window_seconds for the same purpose. Semantically, they basically mean the same thing. We want to give data around this amount of time to arrive before the system persists it, which gives it more of an opportunity to persist non-overlapping data.

When a partition goes cold for writes, after we've waiting past this window, we should compact and persist that partition. This removes one unnecessary knob from the lifecycle configuration and also removes the potential for conflicting configuration options.
 2021-07-10 08:04:33 -04:00
Raphael Taylor-Davies 7af560aa99
feat: Persist lifecycle action (#1888) 
* feat: add split and persist operation

* docs: Improve doc strings

* refactor: use for loop rather than map

* refactor: Make it clear that the lifecycle policy picks the split timestamp

* fix: race condition

* docs: improve comments

* fix: logical merge conflict

* fix: clippy

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
 2021-07-09 13:21:46 +00:00
Paul Dix e41fd2a821 refactor: remove unused mutable_minimum_age_seconds lifecycle setting 
Closes #1878
 2021-07-08 18:34:01 -04:00