Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
7,650 Commits
188 Branches
453 Tags
379 MiB
Commit Graph

1416 Commits (4b47d723b105a87ef1e4c850de07469f5b75336a)

Author SHA1 Message Date
Marco Neumann 11df3091a1 fix: misleading comment 2021-12-15 12:41:30 +01:00
Marco Neumann 496043a767 fix: PR feedback 
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
 2021-12-15 12:41:30 +01:00
Marco Neumann 10a51bac61 refactor: allow database init to recover from errors 
Closes #3335.
 2021-12-15 12:41:30 +01:00
Nga Tran 7783149af1 fix: typos 2021-12-13 10:06:17 -05:00
Nga Tran e76157f0e4 chore: cleanup 2021-12-10 12:08:04 -05:00
Nga Tran 5253c71e03 test: replay test for compact OS chunks 2021-12-10 12:04:00 -05:00
kodiakhq[bot] 2e3aa2e1ab
Merge branch 'main' into ntran/test_compact_os_delete_propagation 2021-12-10 15:11:11 +00:00
kodiakhq[bot] dc152abb55
Merge branch 'main' into alamb/update_state_machine 2021-12-10 09:51:40 +00:00
Nga Tran 7d53fad427
Merge branch 'main' into ntran/test_compact_os_delete_propagation 2021-12-09 17:27:22 -05:00
Nga Tran 0d6fefdd09 test: test concurrent deletes and OS compaction 2021-12-09 17:13:18 -05:00
Nga Tran c0ba69f09e chore: marge main to branch and resolve conflict 2021-12-09 15:40:33 -05:00
Andrew Lamb 191e743ce0
fix: Update server/src/database.rs 
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2021-12-09 15:04:39 -05:00
Andrew Lamb b2f7306d5a docs: Update database startup machine diagram 2021-12-09 14:22:29 -05:00
Carol (Nichols || Goulding) 471c3181bb
feat: Make a top-level influxdb_iox feature for kafka functionality 
Thread the feature through router and server to the write buffer crate.
Move an end-to-end test that uses Kafka behind the feature flag.
 2021-12-09 08:49:34 -05:00
Nga Tran 4e86d0ef30 test: propogate delete tests for compact OS chunks 2021-12-08 17:50:51 -05:00
Nga Tran efbfbb1a0b feat: compact all object store chunks of a given partition 2021-12-08 16:06:03 -05:00
Marco Neumann a8bc5b3b6d fix: allow setting the same server ID twice 
This is important for idempotence and simplifies clients and helper
scripts a lot.
 2021-12-08 19:03:30 +01:00
Nga Tran 43e3e76de2 chore: Merge branch 'main' into ntran/compact_os_sql_tests 2021-12-08 12:07:02 -05:00
Nga Tran 428851b467
chore: Apply suggestions from code review 
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
 2021-12-08 11:42:25 -05:00
Andrew Lamb 218042784f
feat: add `system.queries `system table (#3328) 
* feat: Add query log and system table

* docs: Add docstrings for test normalization
 2021-12-08 16:26:24 +00:00
Andrew Lamb c6a3765d76
feat: Add force flag to RebuildCatalog (#3292) 
* feat: Add force flag to RebuildCatalog

* fix: small cleanups

* docs: Update comments and add WARNING
 2021-12-08 15:36:07 +00:00
Edd Robinson 2a2c5889c3 perf: loading chunks doesn't require sort 2021-12-08 12:03:56 +00:00
Nga Tran a13c3120a6 test: sql tests and checkpoint test for compact OS chunks 2021-12-07 17:32:48 -05:00
Nga Tran c992c82582 chore: Merge branch 'main' into ntran/compact_os_tests 2021-12-07 11:08:12 -05:00
Marco Neumann d9e8bafeb3 fix: improve state machine for write buffer creation retries 2021-12-07 10:29:08 +01:00
Marco Neumann e558f4c708 fix: address review comments 2021-12-07 09:43:47 +01:00
Marco Neumann ef9c49087e fix: retry write buffer creation during database init 2021-12-07 09:23:03 +01:00
kodiakhq[bot] 75cd1d24f4
Merge branch 'main' into ntran/grpc_compact_os_chunks 2021-12-07 03:26:06 +00:00
Nga Tran 3834435620 chore: Merge branch 'ntran/grpc_compact_os_chunks' of https://github.com/influxdata/influxdb_iox into ntran/grpc_compact_os_chunks 2021-12-06 12:06:22 -05:00
Nga Tran d8ed8a3bf7 refactor: address review comments 2021-12-06 12:06:07 -05:00
Carol (Nichols || Goulding) 7499eac067
fix: Disable uuid serde feature; we're not actually serializing any UUIDs 
Connects to #3117.
 2021-12-06 09:37:31 -05:00
Carol (Nichols || Goulding) 16d8ae5e04
fix: Match tokio features to what's actually in use in each crate 
Some crates listed features they don't use; other crates ware relying on
feature flags enabled by something else. I tested these changes by
disabling the workspace hack crate and testing each crate.
 2021-12-06 09:37:16 -05:00
Carol (Nichols || Goulding) 02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost 2021-12-06 09:37:15 -05:00
Carol (Nichols || Goulding) 30c4da7ca7
fix: Be consistent with regex version range specification 2021-12-06 09:37:15 -05:00
Carol (Nichols || Goulding) 4e95e1bb75
fix: Remove redundant once_cell race feature 
`race` is part of once_cell's default features and we aren't turning off
default features.
 2021-12-06 09:37:15 -05:00
Carol (Nichols || Goulding) 3e505951c1
fix: Always enable once_cell's parking_lot dependency to save a bit of space 2021-12-06 09:37:15 -05:00
Carol (Nichols || Goulding) 7a241882c2
fix: Make version reqs on futures-related crates consistent 2021-12-06 09:37:14 -05:00
Carol (Nichols || Goulding) 39862e3886
fix: Disable default chrono features 
Connects to #3117. Won't affect the workspace-hack crate until changes
to some other crates can be upstreamed.
 2021-12-06 09:36:49 -05:00
dependabot[bot] 573ee632b1
chore(deps): bump crc32fast from 1.2.2 to 1.3.0 
Bumps [crc32fast](https://github.com/srijs/rust-crc32fast) from 1.2.2 to 1.3.0.
- [Release notes](https://github.com/srijs/rust-crc32fast/releases)
- [Commits](https://github.com/srijs/rust-crc32fast/compare/v1.2.2...v1.3.0)

---
updated-dependencies:
- dependency-name: crc32fast
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-12-06 01:22:09 +00:00
Nga Tran 65660dace5
Merge branch 'main' into ntran/grpc_compact_os_chunks 2021-12-03 18:16:42 -05:00
Nga Tran 5f3706e0ee feat: grpc call for copact object store chunks 2021-12-03 18:01:28 -05:00
Nga Tran 8e86cadf64
Merge branch 'main' into ntran/compact_os_tests 2021-12-03 17:59:40 -05:00
Andrew Lamb 7dea2ef362
feat: Add force flag to ClaimDatabase (#3284) 
* feat: Add force flag to ClaimDatabase

* fix: Update name of test

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* docs: Update influxdb_iox/tests/end_to_end_cases/management_cli.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
 2021-12-03 21:11:23 +00:00
Nga Tran 86f9fe0bcb refactor: no longer need to create and test no-row-groups parquet files 2021-12-03 15:14:04 -05:00
Nga Tran 152281e428 fix: Capture the right 'no data' while parquet has no data 2021-12-03 12:19:48 -05:00
Nga Tran 21f3306a3f chore: Merge branch 'ntran/chunkid_catalogchunk' of https://github.com/influxdata/influxdb_iox into ntran/chunkid_catalogchunk 2021-12-03 10:02:36 -05:00
Nga Tran dd4e8884be refactor: modify comments 2021-12-03 10:02:22 -05:00
kodiakhq[bot] 0dffcad109
Merge branch 'main' into ntran/chunkid_catalogchunk 2021-12-03 14:21:04 +00:00
Nga Tran a03716cd06 chore: make the message clearer 2021-12-02 15:05:21 -05:00
Nga Tran ffc970a60f feat: add to-be-create-chunk-id for compacting OS chunks in CatalogChunk 2021-12-02 14:49:06 -05:00
Edd Robinson a08246e3c7
Merge branch 'main' into er/refactor/debug_rub_query 2021-12-02 19:42:46 +00:00
Carol (Nichols || Goulding) 0683f851e9
fix: Derive Default instead of doing an equivalent manual impl 
Identified by clippy. The impl did have `#[cfg(test)]` on it, but the
entire struct does too, so this is fine.

https://rust-lang.github.io/rust-clippy/master/index.html#derivable_impls
 2021-12-02 11:52:02 -05:00
Carol (Nichols || Goulding) 5d0fd1c603
fix: Allow dead code on fields that are now detected as never read 2021-12-02 11:52:01 -05:00
Edd Robinson 4a4e11f08b refactor: adjust debug logging 2021-12-02 12:19:51 +00:00
Nga Tran f085af034e
refactor: not persist empty chunk resulting from deleting & deduplicating (#3274) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-12-01 20:57:30 +00:00
kodiakhq[bot] e8edab0b43
Merge branch 'main' into ntran/compact_os_preserved_catalog 2021-12-01 18:26:33 +00:00
Nga Tran de54d2843c chore: remove unused error 2021-12-01 13:24:12 -05:00
Nga Tran 60807c2d95 refactor: address review comments 2021-12-01 13:11:17 -05:00
Nga Tran 254d35a6c2 refactor: address review comment 2021-12-01 12:25:42 -05:00
Nga Tran 878c59c0d2 chore: cleanup 2021-12-01 10:25:35 -05:00
Nga Tran 1ac32f1d90 refactor: avoid lock gap while updating in-memory catalog 2021-12-01 10:24:58 -05:00
Raphael Taylor-Davies 05064eba62
fix: don't error on delete to non-existent table (#3259) (#3263) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-12-01 00:40:05 +00:00
Nga Tran f53cdca010 feat: handling empty compacted stream 2021-11-30 18:13:36 -05:00
Nga Tran 6bb7a796e7 chore: Merge branch 'main' into ntran/compact_os_preserved_catalog 2021-11-30 16:55:38 -05:00
Nga Tran e8034cdc53 refactor: handle delete predicates for compacting os 2021-11-30 16:45:04 -05:00
Raphael Taylor-Davies c161d1176a
refactor: cleanup Db write path error handling (#3258) 2021-11-30 15:05:02 +00:00
Raphael Taylor-Davies 1e515a1dec
feat: load RUB from object store (#3224) (#3250) 2021-11-30 14:39:52 +00:00
Andrew Lamb 20da62eb87
chore: Update `arrow`, `arrow-flight`, and `parquet` to 6.3.0 (#3254) 
* chore: Update `arrow`,  `arrow-flight`, and `parquet` to 6.3.0

* fix: update metadata size
 2021-11-30 12:30:17 +00:00
Raphael Taylor-Davies 88acf3788e
feat: rebuild catalog (#3225) (#3253) 
* feat: rebuild catalog (#3225)

* chore: fix doc

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-30 12:17:27 +00:00
Nga Tran 7afc20142b test: add tests that do actual compacting chunks 2021-11-29 22:59:34 -05:00
Nga Tran 37c4bb402b feat: update os compacted chunk in both in memory and preserved catalog 2021-11-29 15:17:00 -05:00
Raphael Taylor-Davies 837e029ed8
refactor: cleanup database recovery (#3225) (#3245) 
* refactor: cleanup database recovery (#3225)

* chore: fix errors

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-29 18:49:46 +00:00
Raphael Taylor-Davies 0517adf752
feat: test interaction of write buffer and lifecycle (#2811) (#3211) 
* feat: test interaction of write buffer and lifecycle (#2811)

* chore: more docs
 2021-11-29 14:16:43 +00:00
kodiakhq[bot] 23dffefcf8
Merge branch 'main' into crepererum/remove_routing_from_database_mode_7 2021-11-29 11:38:12 +00:00
Raphael Taylor-Davies 197634ed50
feat: reload chunk back into read buffer (#3209) (#3216) 
* feat: reload chunk back into read buffer (#3209)

* chore: fix logical conflict

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-29 11:34:55 +00:00
Marco Neumann 4e043ecb55 refactor: remove old routing / sharding config 
This is superseded by the new router subsystem.
 2021-11-29 12:33:48 +01:00
dependabot[bot] 3bedf6e62a
chore(deps): bump crc32fast from 1.2.1 to 1.2.2 
Bumps [crc32fast](https://github.com/srijs/rust-crc32fast) from 1.2.1 to 1.2.2.
- [Release notes](https://github.com/srijs/rust-crc32fast/releases)
- [Commits](https://github.com/srijs/rust-crc32fast/compare/v1.2.1...v1.2.2)

---
updated-dependencies:
- dependency-name: crc32fast
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-29 09:00:01 +00:00
Raphael Taylor-Davies a286960285
refactor: remove delete predicates from read filter interface (#3212) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-26 12:15:29 +00:00
Marco Neumann 7f2e4f4342 refactor: remove write buffer direction 
The direction was required when a database could read or write from/to a
write buffer. Now it is clear from the usage context of a write buffer
context which of the two applications is meant (databases read, routers
write) so the direction flag is no longer required.
 2021-11-26 12:38:40 +01:00
Marco Neumann ac20fecf6f refactor: remove write buffer producer logic from `Db` 
As a side effect, writing to database no longer requires `async`.

Closes #2243.
 2021-11-26 12:02:27 +01:00
Marco Neumann e5bfa08910 feat: disallow unsequenced deletes when reading from write buffer 
Similar to how writes to a databases that reads from a write buffer are
forbidden deletes should be rejected as well.
 2021-11-26 11:40:08 +01:00
Marco Neumann 22936abb23 refactor: remove connection manager and resolver from `Server` 2021-11-26 11:00:19 +01:00
Marco Neumann 0e06026fbd refactor: remove write routing logic from `Server` 
API users should use `Database` instead, there's no need any longer to mirror
this API in `Server`. Note that `Database` is better than `Db` in this
case, because the former can also check if we're writing from a write
buffer and can easily reject unsequenced writes. The `pub` modifiers
were adjusted to make it impossible to write through `Db` directly.
 2021-11-26 09:26:43 +01:00
Nga Tran 4e7e9f6fc9 chore: Merge branch 'ntran/compact_os_blocks' of https://github.com/influxdata/influxdb_iox into ntran/compact_os_blocks 2021-11-23 16:56:59 -05:00
Nga Tran b2b033f00c chore: rerun fmt after merging 2021-11-23 16:56:38 -05:00
kodiakhq[bot] 1d1622efa3
Merge branch 'main' into ntran/compact_os_blocks 2021-11-23 21:47:05 +00:00
Nga Tran ade7ea6cfb chore: Merge branch 'ntran/compact_os_blocks' of https://github.com/influxdata/influxdb_iox into ntran/compact_os_blocks 2021-11-23 16:44:48 -05:00
Nga Tran a0fb6a4758 chore: cleanup 2021-11-23 16:44:30 -05:00
Nga Tran 4fc09484ed
chore: Apply suggestions from code review 
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
 2021-11-23 16:43:12 -05:00
Nga Tran 4e28b4c43b refactor: address review comments 2021-11-23 16:39:45 -05:00
Nga Tran e64ac10c40 refactor: address review comments and compute max checkpoints 2021-11-23 15:32:46 -05:00
Raphael Taylor-Davies 42e3a41825
fix: flaky test_chunk_access_time (#3197) (#3200) 
* fix: flaky test_chunk_access_time (#3197)

* chore: fix lint

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-23 17:39:57 +00:00
Raphael Taylor-Davies bfa4bfe735
refactor: lifecycle observability cleanup (#2286) (#3189) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-23 15:42:24 +00:00
Marco Neumann f9110eecad test: migrate `DistibutedTest` to use proper router 2021-11-23 15:39:53 +01:00
Nga Tran 124f5bb4c5 chore: make comments clearer 2021-11-22 23:30:18 -05:00
Nga Tran 8788dd6d39 chore: fix comment 2021-11-22 23:24:28 -05:00
Nga Tran 3182e935ed chore: cleanup 2021-11-22 23:22:34 -05:00
Nga Tran 06aba60068 feat: another piece: tests for contiguous chunks and read IOx metadata to get checkpoints 2021-11-22 23:15:20 -05:00
kodiakhq[bot] 29aaafe808
Merge branch 'main' into ntran/compact_os_catalog 2021-11-22 19:19:57 +00:00
Nga Tran 5e45c5a349 refactor: add meessage to assert 2021-11-22 13:28:08 -05:00