Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
7,170 Commits
189 Branches
453 Tags
379 MiB
Commit Graph

7170 Commits (f01382f578703c3203936e13014e1827607ca826)

Author SHA1 Message Date
Dom f01382f578
docs: fix typos 
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2022-03-14 11:36:49 +00:00
Dom Dwyer dc8a708fb7 refactor: use DummyObjectStore for error tests 2022-03-11 14:36:34 +00:00
Dom Dwyer bfc473b8cd feat: object store instrumentation 
Implements a decorator of the ObjectStoreAPI trait, recording:

    * Bytes uploaded / downloaded through the instrumented API
    * Call latencies, broken down by operation & success / error state

All the current implementations that return a Stream from the get
operation actually return a "fake" stream containing all the data in one
go rather than streaming chunks from the upstream. I've instrumented the
Stream to future-proof any actual streaming impls in the future.
 2022-03-11 13:38:48 +00:00
Marco Neumann d46de98183
feat: extract "backend" from querier cache (#4015) 
* feat: extract "backend" from querier cache

The backend will implement pruning policies like LRU and TTL as well as
where/how the data is stored. Having a proper interface for that
simplifies the implementation since we don't need to have one massive
`Cache` object with a super complex mechanism.

This is for #3985.

* refactor: `Backend` -> `CacheBackend`

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-11 11:49:57 +00:00
dependabot[bot] bd2824aa59
chore(deps): Bump pprof from 0.6.2 to 0.7.0 (#4004) 
* chore(deps): Bump pprof from 0.6.2 to 0.7.0

Bumps [pprof](https://github.com/tikv/pprof-rs) from 0.6.2 to 0.7.0.
- [Release notes](https://github.com/tikv/pprof-rs/releases)
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tikv/pprof-rs/commits)

---
updated-dependencies:
- dependency-name: pprof
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: `pprof` features for version 0.7

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-11 11:08:59 +00:00
kodiakhq[bot] 7f7c672ac5
Merge pull request #3992 from influxdata/dom/ingester-lock-fix 
refactor(ingester): eliminate unnecessary lock
 2022-03-11 10:40:40 +00:00
kodiakhq[bot] 30fc29c296
Merge branch 'main' into dom/ingester-lock-fix 2022-03-11 10:32:26 +00:00
Dom Dwyer 4a7364d63f refactor: rename lifecycle_manager args 2022-03-11 10:31:58 +00:00
kodiakhq[bot] 95f461b907
Merge pull request #4012 from influxdata/cn/parquet-file-params 
refactor: Extract a ParquetFileParams catalog type for use in ParquetFileRepo create
 2022-03-10 19:17:21 +00:00
Carol (Nichols || Goulding) ecd06c6ec3
fix: ParquetFileRepo create should be responsible for setting INITIAL_COMPACTION_LEVEL 
When created in the catalog, parquet files should always have compaction
level 0. Updating the compaction level should always happen in the
compactor.

Only the catalog should need to know about the initial compaction level
value.
 2022-03-10 13:51:18 -05:00
Carol (Nichols || Goulding) ff31407dce
refactor: Extract a ParquetFileParams type for create 
This has the advantages of:

- Not needing to create fake parquet file IDs or fake deleted_at
  values that aren't used by create before insertion
- Not needing too many arguments for create
- Naming the arguments so it's easier to see what value is what
  argument, especially in tests
- Easier to reuse arguments or parts of arguments by using copies of
  params, which makes it easier to see differences, especially in tests
 2022-03-10 13:51:18 -05:00
Nga Tran f03ebd79ab
refactor: move querier's test utils to a new crate to get reused by tests in other crates (#4013) 
* refactor: move querier's test utils to a new crate to be able resued by tests in other crates

* chore: remove unused import
 2022-03-10 18:17:58 +00:00
Paul Dix 27999ff72f
feat: add compaction_level and created_at to parquet_file (#3972) 2022-03-10 15:56:57 +00:00
dependabot[bot] 67209fce7c
chore(deps): Bump clap from 3.1.5 to 3.1.6 (#4010) 
Bumps [clap](https://github.com/clap-rs/clap) from 3.1.5 to 3.1.6.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.5...v3.1.6)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 14:38:53 +00:00
dependabot[bot] e1344f7ca8
chore(deps): Bump sqlparser from 0.14.0 to 0.15.0 (#4007) 
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases)
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.14.0...v0.15.0)

---
updated-dependencies:
- dependency-name: sqlparser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 14:28:48 +00:00
dependabot[bot] 68e09264f2
chore(deps): Bump pbjson from 0.2.2 to 0.2.3 (#4006) 
Bumps [pbjson](https://github.com/influxdata/pbjson) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases)
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3)

---
updated-dependencies:
- dependency-name: pbjson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 14:20:10 +00:00
dependabot[bot] f2834c4101
chore(deps): Bump handlebars from 4.2.1 to 4.2.2 (#4009) 
Bumps [handlebars](https://github.com/sunng87/handlebars-rust) from 4.2.1 to 4.2.2.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases)
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.2.1...v4.2.2)

---
updated-dependencies:
- dependency-name: handlebars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 14:11:19 +00:00
dependabot[bot] c051e944f2
chore(deps): Bump tracing from 0.1.31 to 0.1.32 (#4008) 
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.31 to 0.1.32.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.31...tracing-0.1.32)

---
updated-dependencies:
- dependency-name: tracing
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 14:01:00 +00:00
dependabot[bot] cef4090bce
chore(deps): Bump pbjson-types from 0.2.2 to 0.2.3 (#4005) 
Bumps [pbjson-types](https://github.com/influxdata/pbjson) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases)
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3)

---
updated-dependencies:
- dependency-name: pbjson-types
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 13:52:08 +00:00
dependabot[bot] ed84b453d3
chore(deps): Bump tracing-core from 0.1.22 to 0.1.23 (#4003) 
Bumps [tracing-core](https://github.com/tokio-rs/tracing) from 0.1.22 to 0.1.23.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.22...tracing-core-0.1.23)

---
updated-dependencies:
- dependency-name: tracing-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 13:43:15 +00:00
dependabot[bot] 0756642137
chore(deps): Bump pbjson-build from 0.2.2 to 0.2.3 (#4002) 
Bumps [pbjson-build](https://github.com/influxdata/pbjson) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases)
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3)

---
updated-dependencies:
- dependency-name: pbjson-build
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-10 13:34:00 +00:00
Andrew Lamb 2c3d30ca32
chore: Update datafusion, arrow, flight and parquet (#4000) 
* chore: Update datafusion, arrow, flight and parquet

* fix: api change

* fix: fmt

* fix: update test metadata size

* fix: Update sizes in parquet test

* fix: more metadata size update
 2022-03-10 12:24:47 +00:00
Andrew Lamb 12771a7061
fix: allow manual merge messages in semantic commit checker (#4011) 2022-03-10 12:14:45 +00:00
Andrew Lamb dcdd9b079d
chore: Run dependabot more frequently (#3998) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-10 11:40:09 +00:00
Marco Neumann 30d1c77d36
feat: querier test system, ground work (#3991) 
* feat: querier test system, ground work

See #3985 for the motivation.

This introduces a cache system for the querier which can later be
extended to support the remaining features listed in #3985 (e.g.
metrics, LRU/TTL).

All current caches are wired up to go throw the new cache system. Once
we move away from (ab)using `db`, the set of caches will be different
but the system will remain.

* test: explain it

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>

* refactor: simplify cache result broadcast

* refactor: introduce `Loader` crate

* fix: docs

* docs: explain why we manually drop removed hashmap entries

* docs: fix intra-doc link

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
 2022-03-10 11:27:24 +00:00
kodiakhq[bot] 59d6bee531
Merge pull request #3990 from influxdata/dom/log-spam 
refactor: drop trace log verbosity
 2022-03-09 20:18:02 +00:00
kodiakhq[bot] cec6af2949
Merge branch 'main' into dom/log-spam 2022-03-09 20:10:05 +00:00
Carol (Nichols || Goulding) 944f628e29
fix: Remove data_types as a dependency of ng compactor (#3993) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-09 17:03:02 +00:00
kodiakhq[bot] a9512e0fbe
Merge pull request #3905 from influxdata/cn/ng-end-to-end 
test: Add some end-to-end ish NG tests
 2022-03-09 16:54:25 +00:00
Carol (Nichols || Goulding) 1f474bfbf0
test: Create the test database before running postgres iox_catalog tests 2022-03-09 10:43:30 -05:00
Carol (Nichols || Goulding) c891f3c4f2
fix: Print an error with sample env var variable if unset 2022-03-09 09:55:43 -05:00
Carol (Nichols || Goulding) 8af2f60b59
fix: Run catalog setup as part of end-to-end test setup 2022-03-09 09:55:43 -05:00
Carol (Nichols || Goulding) 93b0cdbcc4
fix: Create the test database as part of ng server fixture startup 2022-03-09 09:55:43 -05:00
Carol (Nichols || Goulding) e4fb227c6e
feat: Set the catalog URL explicitly in the test config 2022-03-09 09:55:43 -05:00
Carol (Nichols || Goulding) 465fb0272d
fix: Remove unneeded server id const 2022-03-09 09:55:43 -05:00
Carol (Nichols || Goulding) 6d086705a8
fix: Remove ng create_shared server fixture method for now 2022-03-09 09:55:42 -05:00
Carol (Nichols || Goulding) e315012fe3
fix: Switch NG tests to use TEST_INFLUXDB_IOX_CATALOG_DSN too 2022-03-09 09:55:42 -05:00
Carol (Nichols || Goulding) a14e642f39
refactor: Extract NG end-to-end tests from OG end-to-end tests 2022-03-09 09:55:42 -05:00
Carol (Nichols || Goulding) 1536bdeca0
test: Add an end-to-end NG test 2022-03-09 09:55:42 -05:00
Carol (Nichols || Goulding) 880344494a
fix: Remove reference to AWS from postgres test comment 2022-03-09 09:55:42 -05:00
Carol (Nichols || Goulding) 66a5649258
test: Print out the server type with server log messages 
So that when you have more than one server running in a test, it's
easier to see which one is saying what
 2022-03-09 09:55:42 -05:00
Carol (Nichols || Goulding) ae45d9f750
test: Add Router2 as a supported ServerFixture type 2022-03-09 09:55:42 -05:00
kodiakhq[bot] b6eb61831b
Merge pull request #3944 from influxdata/cn/not-database-url 
fix: Use more consistent env vars for database connections
 2022-03-09 13:42:12 +00:00
kodiakhq[bot] caba70f871
Merge branch 'main' into cn/not-database-url 2022-03-09 13:32:02 +00:00
Andrew Lamb d2c0acdd46
refactor: Remove serving readiness gate (#3986) 
* refactor: Remove serving_readiness

* fix: remove more

* fix: remove test

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-09 12:17:44 +00:00
Dom Dwyer 874be0097d refactor: eliminate unnecessary lock 
This commit splits the API of the LifecycleManager into two:

    * LifecycleManager: singleton responsible for evaluating partitions
        and running persist tasks.

    * LifecycleHandle: a handle for each sequencer ingester(s) to update
        the global LifecycleManager state when applying ops.

This keeps the accessible API & responsibilities of each caller distinct
and allows us to leverage the type system to enforce linearisation of
calls to LifecycleManager::maybe_persist() without resorting to an
(unnecessary) mutex guard for serialisation.
 2022-03-09 11:16:44 +00:00
Dom Dwyer 8af9330cd7 refactor: drop trace log verbosity 
This is very shouty, logging one INFO per HTTP request - the router logs
are almost all this one line repeated.
 2022-03-09 10:38:20 +00:00
Nga Tran c6cab3538f
refactor: move parquet chunk's new and decode to parquet_file crate (#3987) 2022-03-08 22:04:32 +00:00
Paul Dix 96100635c3
feat: ingester seeks kafka partition on initialization (#3940) 
Fixes #3851
 2022-03-08 19:53:35 +00:00
Nga Tran 09fba1d2c0
feat: NG Compactor - main function for finding and compacting parquet files (#3973) 
* feat: main function for finding and compacting parquet files

* chore: Apply suggestions from code review

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* refactor: rename file and struct

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-08 16:34:43 +00:00