Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
5,248 Commits
188 Branches
453 Tags
378 MiB
Commit Graph

190 Commits (42d4ad61e1aaad2f903cb1e75b451d724d84a5b7)

Author SHA1 Message Date
dependabot[bot] d1f5209869
chore(deps): bump arrow from 5.4.0 to 5.5.0 
Bumps [arrow](https://github.com/apache/arrow-rs) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0)

---
updated-dependencies:
- dependency-name: arrow
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-04 08:55:38 +00:00
Marco Neumann a9662b0edb chore: remove unused `mutable_buffer` => `influxdb_line_protocol` dep 2021-09-22 10:56:15 +02:00
Raphael Taylor-Davies c66095cad1
feat: remove metrics crate (#2552) 2021-09-15 19:43:33 +00:00
Raphael Taylor-Davies 44918e4afc
feat: migrate chunk metrics (#2491) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-09 16:02:16 +00:00
Marco Neumann 6d5fb8b53d refactor: simplify nested code 2021-09-07 12:01:06 +02:00
Marco Neumann 31cbb646b9 feat: skip individual rows during replay based on timestamp 2021-09-07 11:44:52 +02:00
dependabot[bot] b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 09:11:38 +00:00
dependabot[bot] b1bb390893
chore(deps): bump parking_lot from 0.11.1 to 0.11.2 
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/Amanieu/parking_lot/releases)
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2)

---
updated-dependencies:
- dependency-name: parking_lot
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 01:18:24 +00:00
Marco Neumann 3c968ac092 feat: correctly account MUB sizes 
Fixes #1565.
 2021-09-03 09:15:49 +02:00
Dom 3de6b44e23
build: use new rustdoc lint name (#2261) 
* fix: nocache feature code rot

The MBChunk::snapshot code when using the "nocache" option no longer
compiles - this commit updates it to match the not(nocache) code.

* build: use updated broken_intra_doc_links name

The broken_intra_doc_links lint was renamed
rustdoc::broken_intra_doc_links

https://doc.rust-lang.org/rustdoc/lints.html
 2021-08-11 19:48:51 +00:00
Andrew Lamb 98a19b5ca6
fix(mutable_buffer): Statistics incorrect when subset of columns is inserted (#2249) 
* fix(mutable_buffer): Statistics incorrect when subset of columns is inserted

* docs: remove wild newline

* fix: clippy

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-11 13:00:14 +00:00
Carol (Nichols || Goulding) 9d15798288 fix: Address or allow Clippy warnings new with Rust 1.54 2021-07-30 09:59:59 -04:00
Carol (Nichols || Goulding) 4689b5e4e5 refactor: Remove first/last write times from MUB chunks 2021-07-28 11:02:57 -04:00
Andrew Lamb 5fb3e00f2a
fix: Properly record total_count and null_count in statistics (#2103) 
* fix: Properly record total_count and null_count in statistics

* fix: fix statistics calculation in mutable_buffer

* refactor: expose null counts in read_buffer

* refactor: expose null_count in parquet_file

* fix: update server crate tests

* fix: update query_tests tests

* docs: tweak comments

* refactor: Use storage_stats rather than adding `null_count`

* refactor: rename test data field for clarity

* fix: fixup merge conflicts

* refactor: rename initial_non_null_count to initial_total_count

* refactor: caculate null_count as row_count - to_add
 2021-07-26 18:13:36 +00:00
Raphael Taylor-Davies 0b88deea43
refactor: don't pass sequence to MUB (#2107) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-26 14:40:39 +00:00
Raphael Taylor-Davies 446af5eb15
fix: consistent write timestamps (#2104) 
* fix: consistent write timestamps

* chore: fix benchmarks
 2021-07-23 18:04:15 +00:00
kodiakhq[bot] 71f3f1aba2
Merge branch 'main' into cn/refactorings 2021-07-22 19:44:18 +00:00
Andrew Lamb 01c79f1a1a
fix: Print all timestamps using RFC3339 format (#2098) 
* fix: Use IOx pretty printer rather than arrow pretty printer

* chore: update tests in the query crate

* chore: update influxdb_iox tests

* chore: Update end to end tests

* chore: update query_tests

* chore: update mutable_buffer tests

* refactor: update parquet_file tests

* refactor: update db tests

* chore: update kafka integration test output

* fix: merge conflict
 2021-07-22 19:04:52 +00:00
Carol (Nichols || Goulding) 799406be72 refactor: Have MUB chunk snapshots hold a TableSummaryAndTimes rather than only ColumnSummaries 
This way we can have the first/last write times on MUB chunk snapshots
without needing to deconstruct/reconstruct TableSummaries as it was
doing.
 2021-07-22 11:35:23 -04:00
Carol (Nichols || Goulding) 956647d966 docs: Fix an outdated comment 2021-07-22 11:35:23 -04:00
Carol (Nichols || Goulding) 0a724878e6 refactor: Organize uses 2021-07-22 11:15:18 -04:00
Andrew Lamb 4da8a16c18
chore: update to arrow 5.0 and master datafusion (#2049) 
* chore: update to arrow 5.0 and master datafusion

* fix: Update test for change in object size
 2021-07-19 12:49:51 +00:00
Raphael Taylor-Davies 1d00fa2fd8
refactor: track memory metrics in catalog (#1995) 
* refactor: track memory metrics in catalog

* chore: update comment
 2021-07-14 16:23:00 +00:00
Carol (Nichols || Goulding) 7ccbab8c90 feat: Make a TableSummaryAndTimes to use to slowly replace TableSummary 
And use TableSummaryAndTimes with the mutable buffer chunks when turning
them into catalog chunks.

It's proving too big to switch over everything using TableSummary at
once, so this will let us switch over more incrementally.
 2021-07-14 10:28:12 -04:00
Andrew Lamb d35b74c226
fix: Fix doc build warnings (#1945) 
* fix: Fix doc build warnings

* refactor: add deny bare_urls to crates

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-13 08:03:42 +00:00
Carol (Nichols || Goulding) b9a6a11b34 feat: Record time of first/last write on MBChunk 2021-07-08 16:39:35 -04:00
Carol (Nichols || Goulding) 22495dd355 fix: Take a TableBatch in the MBChunk constructor 
Thus ensuring all MBChunks will have data in them.
 2021-07-08 16:39:35 -04:00
Carol (Nichols || Goulding) 57501e7f59 refactor: Remove a test function that's only slightly different from another test function 2021-07-08 16:39:35 -04:00
Carol (Nichols || Goulding) 2f9cf857af fix: MBChunk.write_columns doesn't need to be pub 2021-07-08 16:39:35 -04:00
Carol (Nichols || Goulding) 74c0a6cb00 fix: Arrange use statements so rustfmt can manage their order 2021-07-08 16:39:02 -04:00
Marco Neumann 4ca2d3e148 chore: move persistence windows related code into own crate 
The entire persistence windows data structures (including the
checkpoints) have nothing to do with the mutable buffer per se. So lets
move them into their own crate. This also makes `parquet_file` not
longer depend on `mutable_buffer`.
 2021-07-05 10:23:58 +02:00
Marco Neumann cdab1bed05 feat: persist part+db checkpoint in parquets and catalog 
This will be required for replay on server startup.
 2021-07-05 09:42:46 +02:00
Raphael Taylor-Davies 5fe49aa017
feat: add flush guard to PersistenceWindows (#1883) 
* feat: add flush guard to PersistenceWindows

* docs: Update comments based on code review

* fix: fmt

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
 2021-07-02 20:15:33 +00:00
Jacob Marble 0779b0d9bd
feat: add gRPC listener for new write protocol (#1842) 
* feat: add gRPC listener for new write protocol

* chore: clippy happy

* chore: lint

* chore: cargo fmt --all

* chore: cargo clippy

* chore: protobuf-lint

* chore: more formatting

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-07-01 16:15:12 +00:00
Marco Neumann 043890369f refactor: make `MinMaxSequence` safer to use 2021-06-30 16:37:48 +02:00
kodiakhq[bot] e10173124f
Merge branch 'main' into crepererum/mb_checkpoint 2021-06-29 15:25:24 +00:00
Andrew Lamb 3ee96c4618
fix: Do not sequence local writes (avoid panic under load) (#1826) 
* fix: Do not sequence local writes

* fix: Update server/src/db.rs

Co-authored-by: Edd Robinson <me@edd.io>

* fix: review comments

* fix: restore passing sequence information down to mutable buffer

* fix: store min/max times even when there are no sequence numbers

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-06-29 10:39:37 +00:00
Raphael Taylor-Davies f145c5d760
feat: add ability to optimize record batch dictionaries (#1827) (#1830) 
* feat: add ability to optimize record batch dictionaries (#1827)

* chore: correct capacity

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-06-29 09:23:56 +00:00
Raphael Taylor-Davies 32c683ddc1
fix: remove shared dictionary concept from ChunkSnapshot (#1570) (#1820) (#1825) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-06-28 15:26:49 +00:00
Marco Neumann 7b12426602 feat: implement persistence window checkpointing 2021-06-28 11:34:41 +02:00
Marco Neumann 25a912d4eb docs: fix mutable buffer intro docs 2021-06-28 11:34:41 +02:00
Paul Dix af93bb6573 chore: PR cleanup 2021-06-25 15:45:00 -04:00
Paul Dix de236c5a6f feat: update persistence windows to support late arrival less than 30 seconds 2021-06-25 15:34:11 -04:00
Paul Dix 435b4b6a94 feat: add persistence windows to partition and update on write 
This brings the persistence windows into the catalog partition. It adds a helper method on TableBatch to get the min and max times for a given write. Finally, it adds this logic to the db to update persistence windows on every write while the partition write lock is being held.
 2021-06-25 15:34:11 -04:00
Andrew Lamb bed6ec8c31
feat: Handle merging chunks that have different schemas (#1761) 
* feat: Handle merging chunks that have different schemas

* test: print out original (non deduplicated) data in tests
 2021-06-21 15:52:13 +00:00
Andrew Lamb 634c9b4c66
refactor: delete unused code (#1759) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-06-19 08:15:33 +00:00
Andrew Lamb de67bd3efe
refactor: Remove PartitionChunk::table_schema (#1756) 
* refactor: Remove PartitionChunk::table_schema

* docs: update comments
 2021-06-18 16:13:16 +00:00
Paul Dix ecfabe3408 chore: address PR feedback, changed closed persistence window to dequeue. 2021-06-16 15:28:37 -04:00
Paul Dix 6bc4b383a4 chore: update based on PR feedback, add more comments 2021-06-16 15:28:37 -04:00
Paul Dix 802ecacf61 feat: Add persistence windows ingestion tracking 
This adds a new module, persistence_windows, to the mutable buffer crate. Later PRs will add this into the mutable buffer chunk where it can be used to track when the lifecycle for persistence should be triggered.
 2021-06-16 15:28:37 -04:00