Commit Graph

42 Commits (b08ea901f15e114704abdbaafb021dbf0ac43cc3)

Author SHA1 Message Date
Nathaniel Cook a75adf6c4b fix(http): separate InfluxQL and Flux services
Previously the APIBackend understood only a ProxyQueryService,
but it needs to understand that there are two implementations of the
ProxyQueryService one for handling InfluxQL queries and one for handling
Flux queries. The names of the fields have been updated to make this
clear. As well as the FluxBackend is now initialized using the
FluxService explicitly.
2019-02-22 10:31:20 -07:00
Lyon Hill c78477314a
Allow tasks to skip catchup (#12068) 2019-02-22 09:47:04 -07:00
Chris Goller c0266ba5cf refactor(cmd/influxd): use simple query rather than the flux service 2019-02-21 11:18:08 -08:00
Chris Goller 1ec7b69d85 fix(kv): error org search if neither name nor id is set 2019-02-21 11:18:08 -08:00
Stuart Carnie 2872233252 feedback: Add end-to-end write and read tests for multiple organizations 2019-02-21 11:18:08 -08:00
Lyon Hill 3665feef38
Populate organization name on task create action (#12004)
* Populate organization name on task create action

* Move the responsiblity of organization lookup to the platform adaptor
2019-02-20 13:36:35 -07:00
Michael Desa 6a8a1fce32 feat(kv): implemented key/value store with end-to-end integration tests
* feat(kv:inmem:bolt): implement user service in a kv

* refactor(kv): use consistent func receiver name

* feat(kv): add initial basic auth service

* refactor(passwords): move auth interface into own file

* refactor(passwords): rename basic auth files to passwords

* refactor(passwords): rename from BasicAuth to Passwords

* refactor(kv): copy bolt user test into kv

Co-authored-by: Michael Desa <mjdesa@gmail.com>

* feat(kv): add inmem testing to kv store

* fix(kv): remove extra user index initialization

* feat(kv): attempt at making errors nice

* fix(http): return not found error if filter is invalid

* fix(http): s/platform/influxdb/ for user service

* fix(http): s/platform/influxdb/ for user service

* feat(kv): initial port of telegraf configs to kv

* feat(kv): first pass at migrating bolt org service to kv

* feat(kv): first pass at bucket service

* feat(kv): first pass at migrating kvlog to kv package

* feat(kv): add resource op logs

* feat(kv): first pass at user resource mapping migration

* feat(kv): add urm usage to bucket and org services

* feat(kv): first pass at kv authz service

* feat(kv): add cascading auth delete for users

* feat(kv): first pass d authorizer.OrganizationService in kv

* feat(cmd/influxd/launcher): user kv services where appropriate

* fix(kv): initialize authorizations

* fix(influxdb): use same buckets while slowly migrating stuff

* fix(kv): make staticcheck pass

* feat(kv): add dashboards to kv

review: make suggestions from pr review

fix: use common bucket names for bolt/kv stores

* test(kv): add complete password test coverage

* chore(kv): fixes for staticcheck

* feat(kv): implement labels generically on kv

* feat(kv): implement macro service

* feat(kv): add source service

* feat(kv): add session service

* feat(kv): add kv secret service

* refactor(kv): update telegraf and urm with error messages

* feat(kv): add lookup service

* feat(kv): add kv onboarding service

* refactor(kv): update telegraf to avoid repetition

* feat(cmd/influxd): use kv lookup service

* feat(kv): add telegraf to lookup service

* feat(cmd/influxd): use kv telegraf service

* feat(kv): initial port of scrapers in bolt to kv

* feat(kv): update scraper error messaging

* feat(cmd/influxd): add kv scraper

* feat(kv): add inmem backend tests

* refactor(kv): copy paste errors

* refactor(kv): add code to password errors

* fix(testing): update error messages for incorrect passwords

* feat(kv:inmem:bolt): implement user service in a kv

* refactor(kv): use consistent func receiver name

* refactor(kv): copy bolt user test into kv

Co-authored-by: Michael Desa <mjdesa@gmail.com>

* feat(kv): add inmem testing to kv store

* fix(kv): remove extra user index initialization

* feat(kv): attempt at making errors nice

* fix(http): return not found error if filter is invalid

* fix(http): s/platform/influxdb/ for user service

* feat(kv): first pass at migrating bolt org service to kv

* feat(kv): first pass at bucket service

* feat(kv): first pass at migrating kvlog to kv package

* feat(kv): add resource op logs

* feat(kv): first pass at user resource mapping migration

* feat(kv): add urm usage to bucket and org services

* feat(kv): first pass at kv authz service

* feat(kv): add cascading auth delete for users

* feat(kv): first pass d authorizer.OrganizationService in kv

* feat(cmd/influxd/launcher): user kv services where appropriate

* feat(kv): add initial basic auth service

* refactor(passwords): move auth interface into own file

* refactor(passwords): rename basic auth files to passwords

* fix(http): s/platform/influxdb/ for user service

* fix(kv): initialize authorizations

* fix(influxdb): use same buckets while slowly migrating stuff

* fix(kv): make staticcheck pass

* feat(kv): add dashboards to kv

review: make suggestions from pr review

fix: use common bucket names for bolt/kv stores

* feat(kv): implement labels generically on kv

* refactor(passwords): rename from BasicAuth to Passwords

* test(kv): add complete password test coverage

* chore(kv): fixes for staticcheck

* feat(kv): implement macro service

* feat(kv): add source service

* feat(kv): add session service

* feat(kv): initial port of telegraf configs to kv

* feat(kv): initial port of scrapers in bolt to kv

* feat(kv): add kv secret service

* refactor(kv): update telegraf and urm with error messages

* feat(kv): add lookup service

* feat(kv): add kv onboarding service

* refactor(kv): update telegraf to avoid repetition

* feat(cmd/influxd): use kv lookup service

* feat(kv): add telegraf to lookup service

* feat(cmd/influxd): use kv telegraf service

* feat(kv): update scraper error messaging

* feat(cmd/influxd): add kv scraper

* feat(kv): add inmem backend tests

* refactor(kv): copy paste errors

* refactor(kv): add code to password errors

* fix(testing): update error messages for incorrect passwords

* feat(http): initial support for flushing all key/values from kv store

* feat(kv): rename macro to variable

* feat(cmd/influxd/launcher): user kv services where appropriate

* refactor(passwords): rename from BasicAuth to Passwords

* feat(kv): implement macro service

* test(ui): introduce cypress

* test(ui): introduce first typescript test

* test(ui/e2e): add ci job

* chore: update gitignore to ignore test outputs

* feat(inmem): in memory influxdb

* test(e2e): adding pinger that checks if influxdb is alive

* hackathon

* hack

* hack

* hack

* hack

* Revert "feat(inmem): in memory influxdb"

This reverts commit 30ddf032003e704643b07ce80df61c3299ea7295.

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* chore: lint ignore node_modules

* hack

* hack

* hack

* add user and flush

* hack

* remove unused vars

* hack

* hack

* ci(circle): prefix e2e artifacts

* change test to testid

* update cypress

* moar testid

* fix npm warnings

* remove absolte path

* chore(ci): remove /home/circleci proto mkdir hack

* wip: crud resources e2e

* fix(inmem): use inmem kv store services

* test(dashboard): add first dashboard crud tests

* hack

* undo hack

* fix: use response from setup for orgID

* chore: wip

* add convenience getByTitle function

* test(e2e): ui can create orgs

* test(e2e): add test for org deletion and update

* test(e2e): introduce task creation test

* test(e2e): create and update of buckets on org view

* chore: move types to declaration file

* chore: use route fixture in dashboard tests

* chore(ci): hack back

* test(ui): update snapshots

* chore: package-lock

* chore: remove macros

* fix: launcher rebase issues

* fix: compile errors

* fix: compile errors

* feat(cmd/influxdb): add explicit testing, asset-path, and store flags

Co-authored-by: Andrew Watkins <watts@influxdb.com>

* fix(cmd/influxd): set default HTTP handler and flags

Co-authored-by: Andrew Watkins <watts@influxdb.com>

* build(Makefile): add run-e2e and PHONY

* feat(kv:inmem:bolt): implement user service in a kv

* refactor(kv): use consistent func receiver name

* feat(kv): add initial basic auth service

* refactor(passwords): move auth interface into own file

* refactor(passwords): rename basic auth files to passwords

* refactor(passwords): rename from BasicAuth to Passwords

* refactor(kv): copy bolt user test into kv

Co-authored-by: Michael Desa <mjdesa@gmail.com>

* feat(kv): add inmem testing to kv store

* fix(kv): remove extra user index initialization

* feat(kv): attempt at making errors nice

* fix(http): return not found error if filter is invalid

* fix(http): s/platform/influxdb/ for user service

* fix(http): s/platform/influxdb/ for user service

* feat(kv): initial port of telegraf configs to kv

* feat(kv): initial port of scrapers in bolt to kv

* feat(kv): first pass at migrating bolt org service to kv

* feat(kv): first pass at bucket service

* feat(kv): first pass at migrating kvlog to kv package

* feat(kv): add resource op logs

* feat(kv): first pass at user resource mapping migration

* feat(kv): add urm usage to bucket and org services

* feat(kv): first pass at kv authz service

* feat(kv): add cascading auth delete for users

* feat(kv): first pass d authorizer.OrganizationService in kv

* feat(cmd/influxd/launcher): user kv services where appropriate

* fix(kv): initialize authorizations

* fix(influxdb): use same buckets while slowly migrating stuff

* fix(kv): make staticcheck pass

* feat(kv): add dashboards to kv

review: make suggestions from pr review

fix: use common bucket names for bolt/kv stores

* test(kv): add complete password test coverage

* chore(kv): fixes for staticcheck

* feat(kv): implement labels generically on kv

* feat(kv): implement macro service

* feat(kv): add source service

* feat(kv): add session service

* feat(kv): add kv secret service

* refactor(kv): update telegraf and urm with error messages

* feat(kv): add lookup service

* feat(kv): add kv onboarding service

* refactor(kv): update telegraf to avoid repetition

* feat(cmd/influxd): use kv lookup service

* feat(kv): add telegraf to lookup service

* feat(cmd/influxd): use kv telegraf service

* feat(kv): update scraper error messaging

* feat(cmd/influxd): add kv scraper

* feat(kv): add inmem backend tests

* refactor(kv): copy paste errors

* refactor(kv): add code to password errors

* fix(testing): update error messages for incorrect passwords

* feat(kv): rename macro to variable

* refactor(kv): auth/bucket/org/user unique checks return errors now

* feat(inmem): add way to get all bucket names from store

* feat(inmem): Buckets to return slice of bytes rather than strings

* feat(inmem): add locks around Buckets to avoid races

* feat(cmd/influx): check for unauthorized error in wrapCheckSetup

* chore(e2e): add video and screenshot artifcats to gitignore

* docs(ci): add build instructions for e2e tests

* feat(kv): add id lookup for authorized resources
2019-02-19 17:47:19 -06:00
Mark Rushakoff cbef811c63 refactor(task): remove explicit org from run/log lookup
Task ID is now a required value on run and log filters. It was
effectively required by all implementations before anyway, so now those
types reflect that requirement.

Organization ID was removed from those same fields. The TaskService
looks up the organization ID via the task in cases where we need it at a
lower layer.
2019-02-16 14:34:49 -08:00
Lyon Hill 48cc643992
Clean out user from the task store's (#11882)
* Clean out user from the task store's
2019-02-15 15:50:09 -07:00
Alirie Gray 5f524eb92d Rename all occurences of Macro to Variable 2019-02-14 13:21:57 -08:00
Mark Rushakoff d562d6bdde feat(task): pass authorizer to query service
Immediately before the executor calls out to the query service, the
executor loads the authorizer associated with the task, and associates
that authorizer with the context used to execute the query.
2019-02-14 08:53:40 -08:00
Mark Rushakoff caf08b5078 feat(task): store authorization ID with task
Accept token when creating or updating a task, but only report back the
authorization ID.

This means the executor and the platform adapter are now both aware of
an Authorization Service.
2019-02-14 08:53:40 -08:00
Mark Rushakoff 007f5059c7 refactor(task): extract TaskCreate struct
With the ongoing authorization work, creation arguments will differ from
what's returned on reads. More specifically, creation will accept a
token, but reads will report back a token ID.

This refactor facilitates that authorization work, and also brings the
code closer to the swagger definition, for the TaskCreateRequest type in
particular.
2019-02-12 15:10:03 -08:00
Michael Desa f79075deda fix(cmd/influxd/launcher): use http clients for testing launcher 2019-02-06 14:24:21 -05:00
Michael Desa d96787dd67 fix(cmd/influxd/launcher): remove platform alias for influxdb 2019-02-06 14:24:21 -05:00
docmerlin (j. Emrys Landivar) 0fec7aa6c4 feat(tasks): end-to-end tests for the task system 2019-01-25 13:28:25 -06:00
Chris Goller bb09a74732 fix(gather): add all subscribers to a single nats queue group
Previously, scrapers would scrape the target 10 times.  This was
because each scraper subscriber was not put into a queue group.

I've added the queue group "metrics" and now we the subcribers
will only scrape the target once.

Additionally, I moved to using nats memory instead of nats file
store.  We don't need durability for scraper runs across restarts.
2019-01-22 23:05:24 -06:00
Chris Goller e2a1e25707 feat(cmd/influxd): log build information 2019-01-22 21:39:40 -06:00
Chris Goller 31ab881c96
Merge pull request #11347 from influxdata/feat/telemetry
Add optional telemetry that reports every 8 hours
2019-01-22 15:48:24 -06:00
Chris Goller ae62edfa51 feat(cmd/influxd): scrapers should run every ten seconds
Previously, scrapers were running every one minute.
2019-01-22 12:49:13 -06:00
Chris Goller d9f5e84d6a feat(telemetry): add telemetry server handler 2019-01-21 13:05:58 -06:00
Chris Goller 2d00eb18c5 feat(telemetry): add telemetry handler 2019-01-21 13:05:58 -06:00
Chris Goller e690f4bfb3 chore(telemetry): move telemetry into its own package 2019-01-21 13:05:58 -06:00
Chris Goller af8f698490 feat(telemetry): add initial telemetry metrics 2019-01-21 13:05:58 -06:00
Michael Desa e2bc8afc04
Merge pull request #11301 from influxdata/feat/authorize-urm
authorize user resource mapping operations
2019-01-21 08:33:25 -08:00
Chris Goller 49782c3be4 feat(bolt): add metrics for resources stored in boltdb 2019-01-18 17:45:45 -06:00
Chris Goller dea64a3685 feat(proto): read JSON files in proto
We used go-bindata to embed all JSON files in proto to be shipped
with our released influxdb.

We are committing the bin_gen.go so that it simplifies the building
via make and go build.

If you would like to add another proto JSON file, run make
to generate a new bin_gen.go.

Co-Authored-By: Deniz Kusefoglu <deniz@influxdata.com>
2019-01-18 15:13:19 -06:00
Michael Desa cdc9146b78 feat(influxdb): add authorized user resource mappings
feat(bolt): add function to find a resources organization id

rename platform to influxdb

Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-authored-by: Michael Desa <mjdesa@gmail.com>

fix(bolt): rename FindResoureOrganization to FindResoureOrganizationID

feat(authorizer): add authorized user resource mapping service

Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-authored-by: Michael Desa <mjdesa@gmail.com>

feat(influxdb): wire up authorized user resource mapping

Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-authored-by: Michael Desa <mjdesa@gmail.com>

fix(authorizer): remove unused field from tests

Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-authored-by: Michael Desa <mjdesa@gmail.com>
2019-01-18 14:22:08 -05:00
Nathaniel Cook 3a2231be98 Merge branch 'master' into flux-staging 2019-01-11 13:47:49 -07:00
Kelvin Wang ce6004243c add(gather): add writer interface 2019-01-11 13:09:18 -05:00
jlapacik 5266cba346
deps: updates for packages and imports (#10962) 2019-01-10 14:33:25 -08:00
Mark Rushakoff d73d73c0d4 chore: rename imports from platform to influxdb
I did this with a dumb editor macro, so some comments changed too.

Also rename root package from platform to influxdb.

In interest of minimizing risk, anyone importing the root package has
now aliased it to "platform" so that no changes beyond imports were
necessary in those files.

Lastly, replace the old platform module to local path /dev/null so that
nobody can accidentally reintroduce a platform dependency while
migrating platform code to influxdb.
2019-01-09 20:51:47 -08:00
Jeff Wendling 703c3c15ca Hook up DeleteBucket to the tsm1 engine 2019-01-09 15:24:26 -07:00
Edd Robinson f0b0d33db1
Merge pull request #2366 from influxdata/er-bucket-delete
Wire up Storage Engine to API-layer BucketService
2019-01-09 18:44:37 +00:00
Edd Robinson 42ff769f1c Wire up storage.Engine to HTTP BucketService 2019-01-09 15:09:56 +00:00
Michael Desa 857b60150e feat(platform): add proto service for creating templated resources
feat(http): add http handler for proto service

feat(mock): add mock proto service

test(http): add proto handler tests

fix(platform): add view as option when adding a cell

feat(platform): add dashboard to proto struct

feat(fs): add filesystem implementation of proto

feat(http): add protos endpoints to api handler

feat(cmd/influxd/launcher): add protos path to server

doc(http): add protos to swagger

test(cmd/influxd/launcher): add --protos-path to launcher tests

fix(fs): remove unused args from test

fix(http): use platform.Error where appropriate
2019-01-09 09:55:15 -05:00
Michael Desa 338e0587d0 fix(platform): nest view beneath dashboard cell
feat(platform): add functional options for platform errors

fix(testing): set dashboard ids properly in dashboard tests

feat(bolt): add dashboard specific views

fix(bolt): delete view when cell is removed or dashboard is deleted
2019-01-08 18:11:13 -05:00
Adam fa35dea009 feat(query/functions/tests): end to end testing is configured to run against a live storage engine.
all tests use a unique bucket based on the test file name.  copied all tests over from flux repo

the tests are currently disabled due to engine consistency issues:  https://github.com/influxdata/flux/issues/613
2019-01-07 13:05:29 -05:00
Adam 9a561f8b53 manually merged addition of secrets to launcher 2019-01-02 15:42:55 -05:00
Adam 30b2d9ca00
refactor(cmd/influxd): move driver code for influxd main package to sub-package so it can be reused. (#2089) 2018-12-20 17:05:36 -05:00
Adam ad38ad8c3f Revert "refactor(cmd/influxd): move driver code for influxd main package to sub-package so it can be reused (#2077)"
This reverts commit c3c12d1517.
2018-12-20 16:29:28 -05:00
Adam c3c12d1517
refactor(cmd/influxd): move driver code for influxd main package to sub-package so it can be reused (#2077)
* refactor(cmd/influxd): move driver code for influxd main package to sub-package so it can be reused.

* chore(query/influxql): moved query_test.go and requisite files to the influxql dir to be closer to the code it tests. (#2013)
2018-12-20 16:15:40 -05:00