Commit Graph

157 Commits (387d9007a767d052445e9b64ff4fc26d2d0677aa)

Author SHA1 Message Date
Johnny Steenbergen b4b52570b4 chore(pkger): handle edge cases for labels that manifest from user interaction
references: #17434
2020-05-06 15:26:01 -07:00
Johnny Steenbergen d4aa3be8db chore(pkger): handle edge cases for dashboards that manifest from user interaction
references: #17434
2020-05-06 15:12:30 -07:00
Johnny Steenbergen 485cc3ea54 chore(pkger): handle edge cases for checks that manifest from user interaction
references: #17434
2020-05-06 14:57:04 -07:00
Johnny Steenbergen 24dd13bcc6 chore(pkger): handle edge cases for buckets that manifest from user interaction
references: #17434
2020-05-06 14:19:44 -07:00
Johnny Steenbergen f575ea8d41 chore(pkger): handle edge cases that can manifest from users in the platform and stateful pkg applications
references: #17434
2020-05-06 13:01:50 -07:00
Johnny Steenbergen 3fd731511a chore(pkger): make integration stack tests indpendent 2020-05-05 11:01:13 -07:00
Johnny Steenbergen 95ad3c6582 feat(pkger): add the ability to remove a stack and all its associated resources
closes: #17554
2020-05-01 16:18:13 -07:00
Johnny Steenbergen c7e97f1625 feat(pkger): add service and http server integreations for stack lists
references: #17554
2020-04-30 13:24:15 -07:00
Johnny Steenbergen 097d761c3a feat(pkger): extend store with stack lists operation
references: #17544
2020-04-30 13:24:15 -07:00
Johnny Steenbergen ff27edbea9 feat(pkger): add support for using URLs associated with stack to apply/dry run 2020-04-29 16:48:00 -07:00
Johnny Steenbergen 72906a4484 feat(pkger): add support for stateful label mappings
references: #17434
2020-04-24 17:05:55 -07:00
Johnny Steenbergen 3acd860563 feat(pkger): add stateful management for notification rules
references: #17434
2020-04-23 11:47:32 -07:00
Johnny Steenbergen d82aa62fff feat(pkger): extend store to support associations
references: #17434
2020-04-22 16:50:40 -07:00
Johnny Steenbergen 47e5facc81 feat(pkger): add stateful management for telegraf configs 2020-04-21 15:47:49 -07:00
Johnny Steenbergen 596d8fde45 feat(pkger): add stateful management for tasks
notes on this commit. This commit was grueling ;-(. The task API is not a friendly
API to consume. There are a lot of non obvious things going on and almost every
one of them tripped me up. Things of note:

* the http.TaskService does not satisfy the influxdb.TaskService,
  making it impossible to use as a dependency if tasks service gets
  split out
* the APIs for create and update do not share common types. For example:
  creating a task takes every field as a string, but in the update it is
  taken as a options.Duration type. A step further and you'll notice that
  create does not need an option to be provided, but the update does. Its
  jarring trying to understand the indirection here. I struggled mightily
  trying to make sense of it all with the indirection and differing types.
  Made for a very difficult task (no pun intended) when it should have been
  trivial. Opportunity here to fix these up and make this API more uniform
  and remove unneccesary complexity like the options type.
* Nested IDs that get marshaled, are no bueno when you want to marshal a task
  that does not have an ID in it, for either user/org/or self IDs. Its a challenge
  just to do that.
* Lots of logs in the kv.Task portion where we hit errors and log and others where
  we return. It isn't clear what is happening. The kv implementation is also very
  procedural, and I found myself bouncing around like a ping pong ball trying to
  make heads or tails of it.
* There is auth buried deep inside the kv.Task implementation that kept throwing me
  off b/c it kept throwing errors, instead of warns. I assume, not sure if I'm
  correct on this, but that the stuff being logged is determined inconsequential
  to the task working. I had lots of errors from the auth buried in there, and hadn't
  a clue what to make of it....

leaving these notes here as a look back at why working with tasks is so
difficult. This API can improve dramatically. I spent 5x the time trying
to figure out how to use the task API, in procedural calls, than I did
writing the business logic to consume it.... that's a scary realization ;-(

references: #17434
2020-04-21 09:14:20 -07:00
Johnny Steenbergen c483aec2d2 feat(pkger): add stateful management for dashboards 2020-04-20 14:04:45 -07:00
Johnny Steenbergen 8251c362ee chore(pkger): refactor parser notification rule state out into stateful type
this PR also drops a bunc of dead code that was necessary for times of integration.

references: #17434
2020-04-17 14:02:29 -07:00
Johnny Steenbergen 21694416a3 chore(pkger): refactor parser dashboard state out into stateful type
references: #17434
2020-04-16 13:27:36 -07:00
Johnny Steenbergen f084a93fa2 chore(pkger): refactor parser telegraf config state out into stateful type
references: #17434
2020-04-16 12:05:07 -07:00
Johnny Steenbergen da2cf7e94a chore(pkger): refactor parser task state out into stateful type
references: #17434
2020-04-16 10:06:44 -07:00
Johnny Steenbergen 94ae519cc6 chore(pkger): refactor parser notification endpoint state out into stateful type
references: #17434
2020-04-15 14:26:25 -07:00
Johnny Steenbergen c8a79f5653 chore(pkger): refactor parser variable state out into stateful type
references: #17434
2020-04-14 16:48:56 -07:00
Johnny Steenbergen cb7f8ba521 chore(pkger): refactor parser check state out into stateful type
references: #17434
2020-04-14 15:43:50 -07:00
Johnny Steenbergen 68183bb156 chore(pkger): refactor embedded types out of stateful types 2020-04-14 14:12:23 -07:00
Johnny Steenbergen c718b676ef chore(pkger): refactor pkger data model to separate parsed and stateful datum 2020-04-14 09:28:09 -07:00
Johnny Steenbergen 67ccbd4490 fix(influx): add nil check for printing diff of endpoints and checks 2020-04-09 13:15:32 -07:00
Johnny Steenbergen dd432360cc feat(pkger): update notification rule with unique constraints 2020-04-08 10:36:21 -07:00
Johnny Steenbergen c245f3233b chore(pkger): add stack state management for notification endpoints 2020-04-06 14:56:56 -07:00
Johnny Steenbergen 1e19f38502 chore(pkger): add stack state management for variables 2020-04-06 10:22:55 -07:00
Jonathan A. Sternberg 0ae8bebd75
refactor: rewrite imports to include the /v2 suffix for version 2 2020-04-03 12:39:20 -05:00
Johnny Steenbergen 21d2bf5dc6 chore(pkger): add stack state management for checks 2020-04-02 16:32:20 -07:00
Johnny Steenbergen e6317a04ee chore(pkger): refactor summary/diff types for Endpints|Checks|Variables 2020-04-02 12:35:51 -07:00
Johnny Steenbergen 762e9a422e feat(pkger): add stack state management for labels 2020-04-02 09:03:59 -07:00
Johnny Steenbergen f5b9a30b1f feat(pkger): extend labels diff and summary types to include PkgName and Remove
the pkgName refers to the unqiue label resource within the pkg and the
Remove field indicates the label will be removed when applying the pkg
2020-04-01 14:13:45 -07:00
Johnny Steenbergen 34c3b35ca3 feat(pkger): add stack state management for buckets 2020-04-01 13:09:14 -07:00
Johnny Steenbergen 098fd27ab4 feat(pkger): extend DiffBucket with name for existing and new bucket values
provides mapping between pkgName and resource the diff represents. Now that
stacks are in place, the existing bucket may change names.
2020-03-31 13:34:40 -07:00
Johnny Steenbergen 37646464b3 feat(pkger): add stack init cmd to influx cli
closes: #17235
2020-03-27 08:32:58 -07:00
Johnny Steenbergen 8c70dc9b23 feat(pkger): add create stack endpoint for http server 2020-03-26 12:37:43 -07:00
Johnny Steenbergen 7a24e450c1 feat(pkger): add auth for pkger stack CRUDing
adds new AuthAgent type that can be reused across service middleware.
2020-03-26 10:11:59 -07:00
Johnny Steenbergen c926accb42 feat(pkger): add create stack implementation for service 2020-03-23 16:30:27 -07:00
Johnny Steenbergen aae629bffd feat(pkger): extend stack with name and description fields 2020-03-20 11:37:30 -07:00
Johnny Steenbergen e116ecaf5e feat(pkger): add CRUD store implmentation 2020-03-20 10:15:02 -07:00
Johnny Steenbergen 583e512451 feat(pkger): add stack foundation 2020-03-20 10:15:02 -07:00
Johnny Steenbergen 9d272b2229 chore(pkger): refactor exportCoordinator constructor 2020-03-18 17:02:35 -07:00
Johnny Steenbergen c31f321d11 chore(pkger): refactors export mechanics to provide unique metadata.name and spec.name
this is applied to all the resoruces that have had the spec.name field applied.
all resources that have not will work in the same way before this commit.
2020-03-18 15:14:48 -07:00
Johnny Steenbergen 136822da51 feat(pkger): extend endpoints to be able to define a endpoint and rename it 2020-03-17 11:31:33 -07:00
Johnny Steenbergen de32cf94d0 feat(pkger): extend labels to be able to define a label and rename it 2020-03-16 16:20:42 -07:00
Johnny Steenbergen de24c0e813 feat(influx): extend pkg export all cmd to allow for filtering of output
currently, support by filtering labelName and or by resource kind

closes: #17029
2020-03-09 09:27:01 -07:00
Johnny Steenbergen 1c9fd705a3 feat(pkger): add support for exporting org resources filtered by label 2020-03-06 14:51:01 -08:00
Johnny Steenbergen 6818925643 feat(pkger): add support for exporting all and filter by resource types
note: all associations are still included for resources that match the
filte criteria.
2020-03-06 11:51:38 -08:00
Johnny Steenbergen 46758e3370 fix(pkger): correct the export of tasks to include system type tasks
wrong assumption in original design. The system tasks are actually
the CRUD tasks. Name is a bit confusing for user generated tasks.

closes: #17038
2020-02-27 17:16:12 -08:00
Johnny Steenbergen 1a12ebc8ab fix(pkger): fix bug where tasks are exported for notification rules 2020-02-27 15:28:07 -08:00
Johnny Steenbergen 6ce0c3de7f feat(pkger): add support for combining pkgs 2020-02-06 12:40:30 -08:00
Johnny Steenbergen b2da311aa2 chore(pkger): simplify env refs to remove juggling state between validations 2020-02-06 10:31:43 -08:00
Johnny Steenbergen 7cf6167356 feat(influx): extend pkg command with providing env ref entries
this also extend dry run to provide env refs to it. the refactoring was
to enable that bit. Having the ability to dry run with the env ref entries
means we can dry run the pkg with the env ref values to see the impact before
the application takes place.
2020-02-06 08:48:05 -08:00
Johnny Steenbergen 036ba49e12 feat(pkger): add env ref fields for remaining resources 2020-02-05 07:47:12 -08:00
Johnny Steenbergen 64c8707594 feat(pkger): add support for env references for buckets 2020-02-04 17:00:11 -08:00
Johnny Steenbergen 93c8a2a104 feat(pkger): add service logging and tracing middlewares 2020-01-15 12:00:59 -08:00
Johnny Steenbergen 63fd1e2e9a feat(pkger): make pkger packages CRD compliant 2020-01-14 13:20:00 -08:00
Johnny Steenbergen fdc3e7658c feat(pkger): add ability to supply a pkg from a url
the following is now possible from the CLI(same in REST API):
  influx pkg -u 1717709ffa/bucket_pkg_json
2020-01-12 07:53:58 -08:00
Michael Desa 26e8ee879c
feat: add tool to migrate 1.x chronograf dashboards and variables
feat(pkger): export dashboard and variable *ToResource methods

fix(pkger): add empty selected _measurement to builder config

feat(chronograf): add note & note visibility to dashboard cell

The 1.x DashboardCell struct had migrated since we brought the code into
the InfluxDB codebase. This allows us to migrate cells that were created
since then.

feat(cmd/chronograf-migrator): add 1.x chronograf migrator tool

feat(chronograf-migrator): add function to transpile queries

fix: update spelling of todo comment pkger/models.go

Co-Authored-By: Deniz Kusefoglu <deniz@influxdata.com>

fix(chronograf): add type to DashboardQuery

The type has evolved since this code was moved over from chronograf.
Previously, we did not have access to flux as a type of query.

feat(chonograf-migrator): transpile influxql query to flux if possible

fix(chronograf): omit fields when empty on old chronograf structs

fix: make linter not mad at me

feat(chronograf-migrator): lowercase variable names

fix(pkger): add empty selected measurement to builder config

chore(chronograf-migrator): add basic readme

chore(pkger): export Variable and Dashboard ToResource methods

fix(chronograf-migrator): move flags out of init call
2020-01-10 09:05:19 -08:00
Johnny Steenbergen 5bdcfaee4d fix(pkger): make pkger skip system tasks when exporting all tasks 2019-12-27 14:26:18 -08:00
Johnny Steenbergen cce279a80e feat(pkger): add ability to provide secrets alongside package when applying
this provides the ability to add secret refs at runtime instead of adding
them to the pkg.
2019-12-27 12:41:42 -08:00
Johnny Steenbergen 33756ca7a8 fix(pkger): remove tasks created by checks from being exported when exporting by org 2019-12-27 09:36:23 -08:00
Johnny Steenbergen 1c2421cb9c feat(pkger): add export support for tasks 2019-12-23 15:41:14 -08:00
Johnny Steenbergen c9431bceb8 feat(pkger): add apply functionality for task resource 2019-12-23 13:56:42 -08:00
Johnny Steenbergen 8e87498317 feat(pkger): add dry run functionality for task resource 2019-12-23 11:40:28 -08:00
Johnny Steenbergen aef32b0a0a chore(pkger): improve error handling in service
this provides influxdb.Errors whenever possible from the pker service layer.
the behavior that caused the error is somewhat implicated in the errro code.
it is not perfect but is a big step forward. using the http server/client to
run pkger stuff makes it abundantly clear the errors need to be communicated
better.

closes: #16313
2019-12-23 09:36:25 -08:00
Johnny Steenbergen af5b7fd7d3 feat(pkger): add export support for notification rules 2019-12-21 14:58:02 -08:00
Johnny Steenbergen 61dceaa727 feat(pkger): add application functionality for the notification rules resource 2019-12-20 12:48:46 -08:00
Johnny Steenbergen 02aec6ed29 feat(pkger): add dry run functionality for notification rules 2019-12-19 16:31:22 -08:00
Johnny Steenbergen a8a4d3d0b2 feat(pkger): add export functionality to checks resource 2019-12-18 17:52:28 -08:00
Johnny Steenbergen 677e836bf4 feat(pkger): add apply functionality for checks
this work is to support pkger, but was able to add back in the
skipped tests. seeing failures upstream, and didn't catch it in
influxdb b/c the tests were being skipped.

closes: #14799
2019-12-18 15:02:03 -08:00
Johnny Steenbergen 1db3256980 feat(pkger): add check dry run functionality 2019-12-18 12:15:08 -08:00
Johnny Steenbergen d42bbb3c64 feat(pkger): add support for exporting notification endpoints 2019-12-17 11:33:34 -08:00
Johnny Steenbergen b8652ee178 feat(pkger): add support for secret references to notification endpoints parsing 2019-12-17 11:33:34 -08:00
Johnny Steenbergen 39e89eafdf chore(pkger): utilize http client for integration tests for pkger 2019-12-13 09:46:01 -08:00
Johnny Steenbergen 3daaa4d307 feat(pkger): add apply functionality for notification endpoints kind 2019-12-11 15:13:42 -08:00
Johnny Steenbergen 69d7eb4455 feat(pkger): add notification endpoint dry run functionality 2019-12-11 15:13:42 -08:00
Johnny Steenbergen 417d25537f chore(http): refactor org service to use httpc client 2019-12-10 10:49:22 -08:00
Johnny Steenbergen 0225160814 fix(http): provides a reusable http client instead of creating/GCing one each time 2019-12-06 18:15:35 -08:00
Johnny Steenbergen 51f1b73738 feat(pkger): speed up application of pkg with a touch of concurrency
governs the concurrency with a simple semaphore. Defaults to 5
concurrent reqs, anything greater, could put a lot of pressure
on the system as a whole (especially OSS/bolt store).
2019-12-06 18:15:35 -08:00
Deary Hudson III a6941648d2
feat(pkger): pkger to use updated dashboard params (#16142) 2019-12-06 11:13:06 -06:00
Johnny Steenbergen ea67663ab7 fix(pkger): cleanup leftover bits from telegraf and label mapping misses
telegraf was not being mapped correctly for applying the label mapppings
and touched up a bunch of other issues that came up along the way.
2019-12-06 08:51:32 -08:00
Johnny Steenbergen c76babdce6 feat(pkger): add support for exporting telegraf 2019-12-05 09:58:44 -08:00
Jacob Marble 5f19c6cace
chore: Remove several instances of WithLogger (#15996)
* chore: Remove several instances of WithLogger

* chore: unexport Logger fields

* chore: unexport some more Logger fields

* chore: go fmt

chore: fix test

chore: s/logger/log

chore: fix test

chore: revert http.Handler.Handler constructor initialization

* refactor: integrate review feedback, fix all test nop loggers

* refactor: capitalize all log messages

* refactor: rename two logger to log
2019-12-04 15:10:23 -08:00
Johnny Steenbergen 695211aeda feat(pkger): add dry run and apply functionality for telegraf resources 2019-12-04 13:50:42 -08:00
Johnny Steenbergen d0a336fbf8 chore(pkger): refactor nested labels and non unique by name resources
makes the label mapping more generic so we don't have to create N number
of label mapping funcs.
2019-12-03 11:17:20 -08:00
Johnny Steenbergen 988fd0ae80 chore(pkger): refactor diff into common types for easier access 2019-11-22 16:09:23 -08:00
Johnny Steenbergen 888baa2db6 feat(pkger): move bucket pkger model to reflect http APIs retention rules in palce of retention period 2019-11-22 11:13:06 -08:00
Johnny Steenbergen 9c525ad413 feat(pkger): add ability to export all and defined list of resources via influx cli to a package
refers: #15921
2019-11-21 10:55:12 -08:00
Johnny Steenbergen 1a28a3256e feat(pkger): extend apply HTTP API to return parse err with 422 resp 2019-11-14 12:11:13 -08:00
Johnny Steenbergen e26b21e261 chore(pkger): add integration tests for pkger service
these tests supply coverage for the happy path of the pkger service
behavior.
2019-11-12 14:05:07 -08:00
Johnny Steenbergen b58e7947b1 feat(pkger): add support for exporting resource associations
also adds some extra user friendliness. it sorts the pkg created via an
export by resource kinds. It also titles the kinds to make them match the
documentation even though the kind is case insensitive. Easier to read this
way.
2019-11-11 08:51:54 -08:00
Johnny Steenbergen a64b976561 feat(pkger): add export functonality to pkger for existing buckets/labels/dashboards
no associations included at this time. Also fixes http response to be just
the pkg without the envelope. Having that envelope makes the API icky to
work with from any shell script or just saving it to file. This feels more
organic to just drop that envelope.
2019-11-08 16:55:24 -08:00
Johnny Steenbergen 5eb29e9ed9 feat(pkger): add label associations to variables 2019-11-07 12:33:20 -08:00
Johnny Steenbergen 2090afa5fd chore(pkger): move pkger http server to http pkg 2019-11-07 09:44:24 -08:00
Johnny Steenbergen 515b3f542a feat(pkger): add support for the dry run and apply pkger http endpoints
not included is the pretty error handling for the pkger.ParseErr handling
2019-11-07 09:44:24 -08:00
Johnny Steenbergen 848875e64b feat(pkger): add create service and http functionality for a package
supports as of this just the basic metadata being setup on the initial post
2019-11-07 09:44:24 -08:00
Johnny Steenbergen 9d44bb02b4 feat(pkger): update pkg application to skip resources that impose no changes 2019-11-04 09:06:55 -08:00