Commit Graph

769 Commits (8ed55e72b85b0807c97d3d1d7d12e9f07c9737e9)

Author SHA1 Message Date
William Baker 88d6487f4a
feat: extended downgrade command to include sql migrations (#22813)
* feat: extended downgrade command to include sql migrations

* feat: remove short forms in downgrade command
2021-11-02 10:42:02 -06:00
William Baker f7573f43a7
feat: sql migrator can do down migrations (#22806)
* feat: sql down migrations

* refactor: different name for up migrations

* chore: update migrations ref in svc tests

* build: add lint step to verify sql migration names match
2021-11-01 14:30:18 -06:00
Daniel Moran eedd84671b
feat: initialize `influxd downgrade` command to run `Down()` migrations on metadata (#22800) 2021-10-29 16:34:01 -04:00
Daniel Moran 335b74b25f
feat: take a backup of KV/SQL metadata stores before applying migrations (#22765) 2021-10-28 16:02:15 -04:00
mcfarlm3 8825cd5d50
feat: replication apis durable queue management (#22719)
* feat: added durable queue management to replications service

* refactor: improved mapping of replication streams to durable queues

* refactor: modified replication stream durable queues to use user-specified engine path

* chore: generated test mocks for replications DurableQueueManager

* chore: add test coverage for replications durable queue manager

* refactor: made changes based on code review, added mutex to durableQueueManager, improved error logging

* chore: ran make fmt

* refactor: further improvements to error logging
2021-10-26 12:14:29 -07:00
Daniel Moran 58139c47b2
feat: add auth to remotes & replications APIs (#22744) 2021-10-26 11:32:35 -04:00
Daniel Moran 7171e0752a
test: wait for time.Now to change before updating stack in pkger test (#22735)
Do this so comparisons between creation time and update time are realistic.
2021-10-22 16:20:17 -04:00
Dane Strandboge de7f052e5a
chore: fix `dump_tsi` deadlock (#22705) 2021-10-20 12:43:02 -05:00
Daniel Moran 401af4b3ae
feat(flux): enable writing to remote hosts via `to()` and `experimental.to()` (#22634) 2021-10-12 09:35:44 -07:00
Daniel Moran 7b7d4f3856
fix(upgrade): don't drop shard-group durations when upgrading DBs (#22650)
Add KV migration to repair missing shard-group durations
2021-10-11 14:43:57 -07:00
Sam Arnold a02782c63f
feat: return new operator token during backup overwrite (#22629)
* feat: return new operator token during backup overwrite

* chore: update swagger

* test: e2e restore test with new token
2021-10-07 09:07:09 -04:00
Sam Arnold 1542d2404f
feat: new recovery subcommand allows creating recovery user/token (#22590)
* feat: influxd recovery-cli allows creating recovery user/token

Closes #12051

* test: do not assert order of users or orgs in list

* fix: code review suggestions from dan

Co-authored-by: Daniel Moran <danxmoran@gmail.com>

* fix: more fixes from dan's review

* chore: depend on tabwriter from influx-cli

* chore: revert onboarding refactor

Co-authored-by: Daniel Moran <danxmoran@gmail.com>
2021-10-06 12:37:36 -04:00
Daniel Moran 7c19225bed
feat: implement replication validation (#22581) 2021-10-05 14:34:38 -04:00
Daniel Moran 2795c600c1
feat: add `--storage-write-timeout` flag to set write request timeouts (#22617) 2021-10-04 15:38:09 -04:00
Daniel Moran 233f277c07
test: check post-compaction series file sizes before reopening (#22534) 2021-09-21 16:48:44 -04:00
Daniel Moran 1291b1a218
test: update various tests to pass on Windows (#22484)
* Generalize checks for no-such-file errors to pass on Windows
* Close temp tombstone file before trying to delete it in test
* Generalize upgrade tests to work on Windows
2021-09-20 08:25:11 -04:00
Daniel Moran 140148edfb
test: close temp sfile before running buildtsi compaction test (#22481) 2021-09-14 17:34:55 -04:00
Daniel Moran e2948d875c
test: skip pkger URI test when running on Windows (#22475) 2021-09-14 10:58:35 -04:00
Daniel Moran ccd96929c5
test: don't assume unix paths in reporttsi tests (#22461) 2021-09-13 15:24:10 -04:00
William Baker ec7841b355
feat: support for flux cardinality query (#22441)
* feat: works with custom iterator

* feat: works with existing iterators

* chore: cleanup

* test: consistent assertions for tests

* fix: better log message if trying to filter on the value of a field key

* fix: comment for handling boolean literal; handle false boolean as well

* fix: make time range checking inclusive
2021-09-13 13:20:56 -06:00
Sam Arnold 5015297d40
fix: more expressive errors (#22448)
* fix: more expressive errors

Closes #22446

* fix: server only logging for untyped errors

* chore: fix formatting
2021-09-13 15:12:35 -04:00
Daniel Moran 6f39255fc8
fix: ensure files are closed before they are deleted or moved in `deletetsm` (#22458)
* Close files before deleting parent dir in tests
* Use platform-independent function to rename output
* Close source TSM file before overwriting it
2021-09-13 14:39:50 -04:00
Daniel Moran f36960d263
fix: detect noninteractive prompt when displaying warning in buildtsi (#22442) 2021-09-13 11:26:58 -04:00
Daniel Moran 153a89dba0
feat: deleting a bucket also deletes all associated replications (#22424) 2021-09-09 15:22:36 -04:00
Daniel Moran 1fa0ccf24a
refactor: move interfaces for remotes & replication services out of root package (#22417) 2021-09-07 16:21:29 -04:00
Daniel Moran 1c2d68b0cb
build: upgrade to go1.17 (#22363) 2021-09-01 16:09:02 -04:00
Daniel Moran 12c8fd28d2
feat: implement metadata management for replications (#22302) 2021-09-01 12:01:41 -04:00
Daniel Moran dc3b501298
refactor: rename Lock/Unlock on KV stores to RLock/RUnlock (#22357) 2021-08-31 17:03:54 -04:00
Daniel Moran 12fff64760
fix: make TSI index compact old and too-large log files (#22334)
*  TSI index should compact old or too-large log files
* Old tsl files should be compacted without new writes
* Add extra logging when disk size test fails


Co-authored-by: Sam Arnold <sarnold@influxdata.com>
2021-08-30 18:27:48 -04:00
Daniel Moran cc6accf106
chore: delete old-style DBRP mapping (#22339) 2021-08-30 18:27:11 -04:00
William Baker 74eacf63ba
feat: add `--flux-log-enabled` flag for detailed flux logs (#22072)
* feat: add `--flux-log-enabled` flag for detailed flux logs

* feat: set query to unknown if not available
2021-08-30 14:42:05 -06:00
Daniel Moran 6bb95ae6fe
refactor: clean up launcher code (#22330)
* Modify launcher shutdown to reduce field count
* Move tracing setup into its own method
* Move metastore setup into its own method
* Delete unused getter methods
* Fix imports
2021-08-30 14:46:25 -04:00
Daniel Moran 4dd2d7cc7f
fix: hard limit on field size while parsing line protocol (#22311)
Per https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/#string
we only support 64KB, but 1MB is a more realistic practical limit. Before this commit there was
no enforcement of field value size.


Co-authored-by: Sam Arnold <sarnold@influxdata.com>
2021-08-27 10:01:17 -04:00
mcfarlm3 337169f1fd
chore: added unit test coverage to inspect build-tsi 2.x command (#22262)
* chore: add test coverage for inspect build-tsi command

* refactor: moved repeated code into separate functions

* refactor: minor variable naming change

* refactor: added manifest check for index files and made minor flag naming changes to build-tsi

* fix: changed check from NotNil to NotZero
2021-08-24 12:46:02 -07:00
mcfarlm3 115822aeda
refactor: small changes to usage and flag naming for consistency (#22292) 2021-08-24 12:45:20 -07:00
Daniel Moran b37ad79e20
feat: add logging and metrics middlewares to replications API (#22291) 2021-08-24 14:56:56 -04:00
Daniel Moran 641c02f9a8
feat: add APIs for management of replication streams (#22287) 2021-08-24 14:19:03 -04:00
William Baker 467040a072
feat(annotations): add logging and metrics to annotations service (#22271) 2021-08-20 14:28:37 -06:00
William Baker e6c82b455e
feat(notebooks): add logging and metrics to notebooks service (#22266) 2021-08-19 13:22:52 -06:00
Daniel Moran 69988ae68d
feat: add logging and metrics middlewares to remotes API (#22258) 2021-08-18 17:51:40 -04:00
Daniel Moran 4af5c7bf61
feat: implement metadata management for remote connections (#22254) 2021-08-18 15:25:40 -04:00
Daniel Moran 5c7c6c2702
feat: add APIs for management of remote InfluxDB connections (#22219) 2021-08-17 15:15:05 -04:00
Daniel Moran 07d897d2f2
fix: switch flux formatter to one that preserves comments (#22186) 2021-08-12 14:17:18 -04:00
mcfarlm3 a160a1d47c
chore: add unit test coverage to inspect report-tsi command (#22153)
* chore: add unit test coverage to inspect report-tsi command

* chore: add additional test cases and data for inspect report-tsi command

* refactor: made small changes based on code review

* chore: added line protocol files for tsi test data

Co-authored-by: Michelle McFarland <michellemcfarland@Michelles-MacBook-Pro.local>
2021-08-12 08:55:42 -07:00
Dane Strandboge c2f6705008
chore: remove some flaky checks from `dump_tsm_test` (#22144)
* comment out and link to issue #22145
2021-08-10 14:43:52 -05:00
Dane Strandboge c960e11dc7
chore: fix dump_tsm test failing (#22130)
* chore: fix dump_tsm test missing flag
* chore: flush written data with multiple tested keys
2021-08-09 14:24:06 -05:00
William Baker 3f7b996a10
chore: delete the rest of chronograf (#21998)
* chore: deleted the rest of chronograf

* chore: tidy
2021-08-02 09:02:54 -06:00
mcfarlm3 3c1d841df6
chore: add unit test coverage to inspect dump-wal (#21994)
* chore: add unit test coverage to inspect dump-wal

* chore: ran make fmt

* refactor: small styling change in conditional statement

* refactor: passed cobra cmd as pointer instead of an object

Co-authored-by: Michelle McFarland <michellemcfarland@Michelles-MacBook-Pro.local>
2021-07-30 15:32:46 -07:00
Dane Strandboge 09f5f48aeb
chore: add test cases for `inspect dump-tsm` (#21956) 2021-07-29 16:35:01 -05:00
Dane Strandboge 52ccb238af
chore: deletetsm command tests (#21949) 2021-07-29 13:32:06 -05:00
Daniel Moran 9d81f4a096
fix: prevent silently dropped writes with overlapping shards (#21946)
Co-authored-by: Geoffrey Wossum <gwossum@influxdata.com>
2021-07-27 12:15:08 -04:00
mcfarlm3 9f13f1c309
feat: port influxd inspect build-tsi (#21936)
* feat: port influxd inspect build-tsi

* refactor: changes based on code review, not including logging changes

* refactor: improved logging based on code review

* chore: updated CHANGELOG.md

Co-authored-by: Michelle McFarland <michellemcfarland@Michelles-MacBook-Pro.local>
2021-07-27 08:26:22 -07:00
William Baker e7685fff01
feat: add `ui-disabled` flag to run server with UI disabled (#21910)
* feat: add `ui-disabled` flag to run server with UI disabled

* chore: update CHANGELOG
2021-07-21 19:52:57 -06:00
mcfarlm3 3b86c32468
feat: port `influxd inspect dump-wal` (#21888)
* feat: porting influxd inspect dump-wal from 1.x

* refactor: adapted 1.x logic to cobra command syntax

* refactor: improved clarity of printed output and help documentation

* chore: ran make fmt

* chore: updated CHANGELOG.md

* refactor: made small changes suggested in code review
2021-07-21 09:15:37 -07:00
Dane Strandboge 0136c7e63d
feat: port `influx inspect report-tsm` to 2.x (#21814) 2021-07-20 12:08:30 -05:00
Faith Chikwekwe 9ff953c400
build(flux): update flux to v0.123.0 (#21879)
* build(flux): update flux to v0.123.0

* fix(cmd/influxd/launcher): modify selector aggregate tests to use create empty false

Co-authored-by: Jonathan A. Sternberg <jonathan@influxdata.com>
2021-07-19 13:20:32 -07:00
mcfarlm3 5d8c0b4b17
feat: port `influxd inspect report-tsi` to 2.x (#21788)
* feat: port influxd inspect report-tsi

* chore: restored tsi1.IsIndexDir() functionality

* refactor: changed inspect report-tsi to work with 2.x file system structure

* chore: fixed output tab alignment

* chore: ran make fmt

* chore: ran make fmt on changed file after resolving merge conflicts

* refactor: made changes based on code review

* refactor: avoiding defer build-up in loop

* refactor: replaced path with filepath and added shardIdx closures

* chore: updated changelog

* refactor: handled shardIdx closures in error case

Co-authored-by: Michelle McFarland <michellemcfarland@Michelles-MacBook-Pro.local>
2021-07-19 08:37:28 -07:00
Dane Strandboge c62d4cdc24
refactor: move to `filepath.WalkDir` for slight performance gain in Go 1.16 (#21858) 2021-07-16 09:11:11 -05:00
Dane Strandboge 31780ba845
feat: create `influxd inspect verify-wal` command (#21828) 2021-07-15 11:38:20 -05:00
Daniel Moran fa1c352132
fix: don't assume bash when displaying interactive CLI prompts (#21839) 2021-07-14 18:12:56 -04:00
Sam Arnold 962b9d7d02
fix: simplify file set, remove series file member (#21831) 2021-07-12 10:43:20 -04:00
Dane Strandboge 60de82e40a
chore: move to os/io over ioutil for `influx inspect` code (#21827) 2021-07-09 15:12:00 -05:00
William Baker 0cb0da2060
chore: modify/delete some chronograf refs (#21802)
* chore: modify/delete some chronograf refs

* chore: update CHANGELOG
2021-07-08 09:14:03 -05:00
Daniel Moran 06b403ca57
build: delete `influx` CLI code, remove it from build (#21797) 2021-07-07 09:29:20 -04:00
Dane Strandboge 6a99416ab2
feat: port `influxd inspect deletetsm` to 2.x (#21786) 2021-07-06 12:20:30 -05:00
Dane Strandboge ba31a0e260
feat: port `influx inspect dumptsi` subcommand (#21784) 2021-07-06 11:40:21 -05:00
Dane Strandboge 7c529d929a
feat: port `influxd inspect dump-tsm` (#21761)
This differs from the 1.x implementation as the blocks and index in-depth printout will now only happen if explicitly stated, instead of being printed with filterKey set
2021-07-02 15:51:36 -05:00
Dane Strandboge 8e8953f2c5
refactor: reorganize inspect subcommands for better package hierarchy (#21763) 2021-06-30 11:46:47 -05:00
Dane Strandboge 3c715d6851
feat: Port influxd inspect verify-tombstone subcommand (#21646) 2021-06-28 15:14:30 -05:00
Daniel Moran b1b076f600
refactor: delete old backup/restore CLI commands, replace with dependency on new CLI repo (#21703) 2021-06-17 09:18:55 -04:00
William Baker 235366a603
feat: enable annotations backend (#21702)
* feat: enable annotations backend
2021-06-16 12:41:23 -04:00
William Baker 40c7d69415
feat: use go embed for sqlite migrations (#21653) 2021-06-10 18:40:57 -04:00
Dane Strandboge 8b90d23088
feat: Port influxd inspect verify-tsm (#21615)
* feat: port tsm-inspect command to dbv2

* test: add test code for tsm-verify

* chore: delete old tsm-verify code stubs

* docs: update changelog

* chore: test for broken checksums more thoroughly
2021-06-09 10:38:51 -05:00
Daniel Moran d747e7ec4e
feat: add config parameters to toggle WAL concurrency and timeouts (#21621)
* feat: add context parameter to Take() method on fixed limiter
* refactor: plumb context through to uses of Take()
* test: update tests to pass context as needed
* feat: add config toggles for setting WAL write concurrency & timeout
2021-06-09 11:03:53 -04:00
Dane Strandboge c665c749fe
feat: Port influxd inspect verify-seriesfile (#21635)
* feat: Port influxd inspect verify-seriesfile

* chore: clean up logic, use zaptest logger

* chore: better error handling

* chore: generic error return statement

* chore: collapse nil check into if-block

Co-authored-by: Daniel Moran <danxmoran@gmail.com>

* chore: update changelog

Co-authored-by: Daniel Moran <danxmoran@gmail.com>
2021-06-08 15:19:21 -05:00
William Baker 34a4a8e60e
feat: metadata backup returns bucket manifests (#21601)
* fix: properly omit empty time.Time that are optional

* fix: optimize slice creation in manifest fns
2021-06-04 09:26:17 -04:00
William Baker b0ea3b6675
feat: backup and restore sqlite database (#21584)
* feat: new metadata backup endpoint

* feat: added restore/sql API endpoint

* fix: content-type is multipart/mixed, part names are kv and sql

* fix: changed multipart manifest to buckets and made it .json

* feat: added lock for backing up sqlite and bolt dbs

* fix: use read lock instead of write lock on kv during backup

* fix: use filepath.Join for temp dirs
2021-06-02 19:07:53 -04:00
William Baker ed629bfebe
feat(notebooks): notebooks database implementation (#21573) 2021-05-28 09:29:00 -04:00
William Baker c267b31232
chore: move notebook service interface to root (#21570) 2021-05-26 12:27:56 -04:00
William Baker 19b0470ed7
feat: integrated sqlite db for metadata (#21543) 2021-05-25 12:12:07 -04:00
Daniel Moran 00420fb54c
fix(influxql): make meta queries respect query timeout (#21545)
Co-authored-by: davidby-influx <dbyrne@influxdata.com>
2021-05-24 21:10:53 -04:00
Daniel Moran b1e1125376
fix(cmd/influxd): Add logging work-around for absolute file paths on Windows (#21540) 2021-05-24 16:52:58 -04:00
William Baker 40897b9a98
feat(notebooks): authorization for notebooks (#21490)
* feat(notebooks): auth changes to top-level authz

* feat(notebooks): added authorization

* chore(notebooks): added tests for authz
2021-05-17 19:01:27 -04:00
William Baker cc4ebe231d
feat: notebooks api handlers & fake store (#21480)
* feat(notebooks): demo store functionality

* feat: demo store with context and middleware

* refactor: without context

* fix: update endpoint and status code for delete

* chore: changed name

* chore: added some tests

* fix: address review comments
2021-05-17 12:07:48 -04:00
Daniel Moran 4343d5ce07
fix: add logging to NATS streaming server (#21375) 2021-05-05 14:25:39 -04:00
Daniel Moran 942f7095e7
fix(query): make config validation for query controller less strict (#21324)
* fix(query): accept queue-size > 0 when concurrency = 0
* fix(influxd): revert defaults for query settings to avoid validation err
* test: lower the default query concurrency used by test launchers
2021-04-28 17:27:37 -04:00
Daniel Moran 91d59d9588
fix(upgrade): ensure query config written by `influxd upgrade` is valid (#21321)
* test: refactor upgrade test to cover the config upgrade
* fix: ensure upgraded query config is valid
2021-04-28 15:41:44 -04:00
William Baker 0106de9fa7
feat: notebooks HTTP handlers (#21316)
* feat: notebooks HTTP handlers
2021-04-28 11:06:13 -04:00
Timmy Luong 0c88854486
feat: add StaticLegend (#21218) 2021-04-15 15:06:01 -07:00
William Baker 1d431e8bc3
feat: structured logging for HTTP server ErrorLog (#21235)
* feat: structured logging for HTTP server ErrorLog

* chore: update CHANGELOG
2021-04-15 17:08:12 -04:00
Sam Arnold 7b2e122869
refactor: refactor for tasks backport (#21117)
* chore: remove dead code

* refactor: move FluxLanguageService interface to fluxlang

* chore: run fmt

* refactor: move task.go from top level to task/taskmodel

* chore: run formatter

* chore: fix up import ordering with gci
2021-04-07 14:42:55 -04:00
Daniel Moran ff6b55aca4
feat(flux): allow values of 0 to disable controller limits (#21127)
Co-authored-by: Sam Arnold <sarnold@influxdata.com>
2021-04-05 16:58:27 -04:00
Daniel Moran 00afd95cb7
refactor: automated move of errors and id from root to kit (#21101)
Co-authored-by: Sam Arnold <sarnold@influxdata.com>
2021-03-30 14:10:02 -04:00
Daniel Moran 4ef09e1ccb
feat(cmd/inflxud): add support for writing to stdout in `export-lp` (#21046) 2021-03-23 14:59:16 -04:00
Daniel Moran 3420ed53d3
refactor: modularize backup/restore behavior so it's possible to test using `go test` (#20991) 2021-03-23 07:28:38 -04:00
Daniel Moran c5edd9092f
feat(http): add config parameters for HTTP timeouts (#20971) 2021-03-16 16:54:46 -04:00
Daniel Moran 9922863185
fix(tls): Update TLS strict cipher suite to actually work (#20921) 2021-03-16 10:45:27 -04:00
Daniel Moran 316eaa24b4
feat(http): add config option to disable metrics endpoint in influxd (#20963) 2021-03-15 15:11:23 -04:00
J. Emrys Landivar 5df737cbc7
fix(cmd/influx): log error details when `influxd upgrade` fails to migrate databases (#20929)
Co-authored-by: Dan Moran <dmoran@influxdata.com>
2021-03-15 14:32:50 -04:00
Daniel Moran ba4099ddb9
feat(cmd/influx): allow setting shard-group durations for buckets via API and CLI (#20911)
Co-authored-by: Alexander Savinykh <asavinykh@nbrx.de>
2021-03-11 14:51:03 -05:00