Commit Graph

196 Commits (6a8a1fce32ac4f9f5c6b13e51797ca4dcded46a9)

Author SHA1 Message Date
Michael Desa 6a8a1fce32 feat(kv): implemented key/value store with end-to-end integration tests
* feat(kv:inmem:bolt): implement user service in a kv

* refactor(kv): use consistent func receiver name

* feat(kv): add initial basic auth service

* refactor(passwords): move auth interface into own file

* refactor(passwords): rename basic auth files to passwords

* refactor(passwords): rename from BasicAuth to Passwords

* refactor(kv): copy bolt user test into kv

Co-authored-by: Michael Desa <mjdesa@gmail.com>

* feat(kv): add inmem testing to kv store

* fix(kv): remove extra user index initialization

* feat(kv): attempt at making errors nice

* fix(http): return not found error if filter is invalid

* fix(http): s/platform/influxdb/ for user service

* fix(http): s/platform/influxdb/ for user service

* feat(kv): initial port of telegraf configs to kv

* feat(kv): first pass at migrating bolt org service to kv

* feat(kv): first pass at bucket service

* feat(kv): first pass at migrating kvlog to kv package

* feat(kv): add resource op logs

* feat(kv): first pass at user resource mapping migration

* feat(kv): add urm usage to bucket and org services

* feat(kv): first pass at kv authz service

* feat(kv): add cascading auth delete for users

* feat(kv): first pass d authorizer.OrganizationService in kv

* feat(cmd/influxd/launcher): user kv services where appropriate

* fix(kv): initialize authorizations

* fix(influxdb): use same buckets while slowly migrating stuff

* fix(kv): make staticcheck pass

* feat(kv): add dashboards to kv

review: make suggestions from pr review

fix: use common bucket names for bolt/kv stores

* test(kv): add complete password test coverage

* chore(kv): fixes for staticcheck

* feat(kv): implement labels generically on kv

* feat(kv): implement macro service

* feat(kv): add source service

* feat(kv): add session service

* feat(kv): add kv secret service

* refactor(kv): update telegraf and urm with error messages

* feat(kv): add lookup service

* feat(kv): add kv onboarding service

* refactor(kv): update telegraf to avoid repetition

* feat(cmd/influxd): use kv lookup service

* feat(kv): add telegraf to lookup service

* feat(cmd/influxd): use kv telegraf service

* feat(kv): initial port of scrapers in bolt to kv

* feat(kv): update scraper error messaging

* feat(cmd/influxd): add kv scraper

* feat(kv): add inmem backend tests

* refactor(kv): copy paste errors

* refactor(kv): add code to password errors

* fix(testing): update error messages for incorrect passwords

* feat(kv:inmem:bolt): implement user service in a kv

* refactor(kv): use consistent func receiver name

* refactor(kv): copy bolt user test into kv

Co-authored-by: Michael Desa <mjdesa@gmail.com>

* feat(kv): add inmem testing to kv store

* fix(kv): remove extra user index initialization

* feat(kv): attempt at making errors nice

* fix(http): return not found error if filter is invalid

* fix(http): s/platform/influxdb/ for user service

* feat(kv): first pass at migrating bolt org service to kv

* feat(kv): first pass at bucket service

* feat(kv): first pass at migrating kvlog to kv package

* feat(kv): add resource op logs

* feat(kv): first pass at user resource mapping migration

* feat(kv): add urm usage to bucket and org services

* feat(kv): first pass at kv authz service

* feat(kv): add cascading auth delete for users

* feat(kv): first pass d authorizer.OrganizationService in kv

* feat(cmd/influxd/launcher): user kv services where appropriate

* feat(kv): add initial basic auth service

* refactor(passwords): move auth interface into own file

* refactor(passwords): rename basic auth files to passwords

* fix(http): s/platform/influxdb/ for user service

* fix(kv): initialize authorizations

* fix(influxdb): use same buckets while slowly migrating stuff

* fix(kv): make staticcheck pass

* feat(kv): add dashboards to kv

review: make suggestions from pr review

fix: use common bucket names for bolt/kv stores

* feat(kv): implement labels generically on kv

* refactor(passwords): rename from BasicAuth to Passwords

* test(kv): add complete password test coverage

* chore(kv): fixes for staticcheck

* feat(kv): implement macro service

* feat(kv): add source service

* feat(kv): add session service

* feat(kv): initial port of telegraf configs to kv

* feat(kv): initial port of scrapers in bolt to kv

* feat(kv): add kv secret service

* refactor(kv): update telegraf and urm with error messages

* feat(kv): add lookup service

* feat(kv): add kv onboarding service

* refactor(kv): update telegraf to avoid repetition

* feat(cmd/influxd): use kv lookup service

* feat(kv): add telegraf to lookup service

* feat(cmd/influxd): use kv telegraf service

* feat(kv): update scraper error messaging

* feat(cmd/influxd): add kv scraper

* feat(kv): add inmem backend tests

* refactor(kv): copy paste errors

* refactor(kv): add code to password errors

* fix(testing): update error messages for incorrect passwords

* feat(http): initial support for flushing all key/values from kv store

* feat(kv): rename macro to variable

* feat(cmd/influxd/launcher): user kv services where appropriate

* refactor(passwords): rename from BasicAuth to Passwords

* feat(kv): implement macro service

* test(ui): introduce cypress

* test(ui): introduce first typescript test

* test(ui/e2e): add ci job

* chore: update gitignore to ignore test outputs

* feat(inmem): in memory influxdb

* test(e2e): adding pinger that checks if influxdb is alive

* hackathon

* hack

* hack

* hack

* hack

* Revert "feat(inmem): in memory influxdb"

This reverts commit 30ddf032003e704643b07ce80df61c3299ea7295.

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* hack

* chore: lint ignore node_modules

* hack

* hack

* hack

* add user and flush

* hack

* remove unused vars

* hack

* hack

* ci(circle): prefix e2e artifacts

* change test to testid

* update cypress

* moar testid

* fix npm warnings

* remove absolte path

* chore(ci): remove /home/circleci proto mkdir hack

* wip: crud resources e2e

* fix(inmem): use inmem kv store services

* test(dashboard): add first dashboard crud tests

* hack

* undo hack

* fix: use response from setup for orgID

* chore: wip

* add convenience getByTitle function

* test(e2e): ui can create orgs

* test(e2e): add test for org deletion and update

* test(e2e): introduce task creation test

* test(e2e): create and update of buckets on org view

* chore: move types to declaration file

* chore: use route fixture in dashboard tests

* chore(ci): hack back

* test(ui): update snapshots

* chore: package-lock

* chore: remove macros

* fix: launcher rebase issues

* fix: compile errors

* fix: compile errors

* feat(cmd/influxdb): add explicit testing, asset-path, and store flags

Co-authored-by: Andrew Watkins <watts@influxdb.com>

* fix(cmd/influxd): set default HTTP handler and flags

Co-authored-by: Andrew Watkins <watts@influxdb.com>

* build(Makefile): add run-e2e and PHONY

* feat(kv:inmem:bolt): implement user service in a kv

* refactor(kv): use consistent func receiver name

* feat(kv): add initial basic auth service

* refactor(passwords): move auth interface into own file

* refactor(passwords): rename basic auth files to passwords

* refactor(passwords): rename from BasicAuth to Passwords

* refactor(kv): copy bolt user test into kv

Co-authored-by: Michael Desa <mjdesa@gmail.com>

* feat(kv): add inmem testing to kv store

* fix(kv): remove extra user index initialization

* feat(kv): attempt at making errors nice

* fix(http): return not found error if filter is invalid

* fix(http): s/platform/influxdb/ for user service

* fix(http): s/platform/influxdb/ for user service

* feat(kv): initial port of telegraf configs to kv

* feat(kv): initial port of scrapers in bolt to kv

* feat(kv): first pass at migrating bolt org service to kv

* feat(kv): first pass at bucket service

* feat(kv): first pass at migrating kvlog to kv package

* feat(kv): add resource op logs

* feat(kv): first pass at user resource mapping migration

* feat(kv): add urm usage to bucket and org services

* feat(kv): first pass at kv authz service

* feat(kv): add cascading auth delete for users

* feat(kv): first pass d authorizer.OrganizationService in kv

* feat(cmd/influxd/launcher): user kv services where appropriate

* fix(kv): initialize authorizations

* fix(influxdb): use same buckets while slowly migrating stuff

* fix(kv): make staticcheck pass

* feat(kv): add dashboards to kv

review: make suggestions from pr review

fix: use common bucket names for bolt/kv stores

* test(kv): add complete password test coverage

* chore(kv): fixes for staticcheck

* feat(kv): implement labels generically on kv

* feat(kv): implement macro service

* feat(kv): add source service

* feat(kv): add session service

* feat(kv): add kv secret service

* refactor(kv): update telegraf and urm with error messages

* feat(kv): add lookup service

* feat(kv): add kv onboarding service

* refactor(kv): update telegraf to avoid repetition

* feat(cmd/influxd): use kv lookup service

* feat(kv): add telegraf to lookup service

* feat(cmd/influxd): use kv telegraf service

* feat(kv): update scraper error messaging

* feat(cmd/influxd): add kv scraper

* feat(kv): add inmem backend tests

* refactor(kv): copy paste errors

* refactor(kv): add code to password errors

* fix(testing): update error messages for incorrect passwords

* feat(kv): rename macro to variable

* refactor(kv): auth/bucket/org/user unique checks return errors now

* feat(inmem): add way to get all bucket names from store

* feat(inmem): Buckets to return slice of bytes rather than strings

* feat(inmem): add locks around Buckets to avoid races

* feat(cmd/influx): check for unauthorized error in wrapCheckSetup

* chore(e2e): add video and screenshot artifcats to gitignore

* docs(ci): add build instructions for e2e tests

* feat(kv): add id lookup for authorized resources
2019-02-19 17:47:19 -06:00
Mark Rushakoff abd6fee757 fix(task): use HTTP 404 for no results when listing runs, logs 2019-02-19 11:00:43 -08:00
Lyon Hill c7f60271fd
remove task owner from task struct (#11968)
* remove task owner from the payload

* clean user references in store and tests
2019-02-19 11:19:06 -07:00
Mark Rushakoff cbef811c63 refactor(task): remove explicit org from run/log lookup
Task ID is now a required value on run and log filters. It was
effectively required by all implementations before anyway, so now those
types reflect that requirement.

Organization ID was removed from those same fields. The TaskService
looks up the organization ID via the task in cases where we need it at a
lower layer.
2019-02-16 14:34:49 -08:00
Lyon Hill 48cc643992
Clean out user from the task store's (#11882)
* Clean out user from the task store's
2019-02-15 15:50:09 -07:00
Mark Rushakoff 12292a4983 fix(task): recover when call to execute fails
This was a missed case from #11817.

This case currently occurs when creating a task through the UI, using a
session rather than a full-fledged authorization. It doesn't fix that
case yet, but at least it will log an informative message.
2019-02-15 08:32:04 -08:00
Brandon Farmer e8290276d3 Allow session authorization for tasks 2019-02-14 14:36:18 -08:00
Mark Rushakoff d562d6bdde feat(task): pass authorizer to query service
Immediately before the executor calls out to the query service, the
executor loads the authorizer associated with the task, and associates
that authorizer with the context used to execute the query.
2019-02-14 08:53:40 -08:00
Mark Rushakoff caf08b5078 feat(task): store authorization ID with task
Accept token when creating or updating a task, but only report back the
authorization ID.

This means the executor and the platform adapter are now both aware of
an Authorization Service.
2019-02-14 08:53:40 -08:00
Mark Rushakoff 007f5059c7 refactor(task): extract TaskCreate struct
With the ongoing authorization work, creation arguments will differ from
what's returned on reads. More specifically, creation will accept a
token, but reads will report back a token ID.

This refactor facilitates that authorization work, and also brings the
code closer to the swagger definition, for the TaskCreateRequest type in
particular.
2019-02-12 15:10:03 -08:00
Mark Rushakoff 4b80fc43c4 test(task): deflake test for CreatedAt, UpdatedAt
The earliest time needed to be captured before creating the task, not
after; otherwise a test running close to a second boundary would fail.

Also make the failure message slightly more readable.
2019-02-12 11:35:55 -08:00
Mark Rushakoff 3a9b3a521a fix(task): clean up run state after a run fails
Previously the DesiredState would not be cleared after an individual run
failed.

Fixes #11816.
2019-02-12 10:53:37 -08:00
Lyon Hill 512af6e4b3
Add createdAt and updatedAt to tasks api (#11794)
fixes #11061
2019-02-12 11:06:23 -07:00
Jonathan A. Sternberg e0e3a17ed9
Merge branch 'master' into flux-staging 2019-02-07 10:26:21 -06:00
Lorenzo Affetti c0ba465604 fix: multiple compatibility issues with flux@master 2019-02-05 17:46:54 +01:00
Mark Rushakoff 211d28567d fix(task): set run status to failed if execution fails
Previously a run that started successfully and then failed, would be
incorrectly marked as a success.
2019-02-04 15:51:11 -08:00
Mark Rushakoff 559cbc776e fix(task): return ErrRunNotFound when finding run by ID
This is necessary to return 404 from the HTTP handler.
2019-01-31 12:04:06 -08:00
Mark Rushakoff a0ff60c075 fix(task): respect limit when finding runs 2019-01-24 15:32:16 -08:00
Lyon Hill bdf65f5dfe
Remove owner from task json responses (#11504)
* Remove owner from task json responses

* allow for crud to run in parallel with other tests

To be parallel we just cant assume we only have 1 task.
2019-01-24 16:05:24 -07:00
Mark Rushakoff 882750c751 fix(task): populate created task response, the same as finding
The response from a created task was missing the name field; now a test
ensures that response matches what is returned when listing tasks.
2019-01-22 11:32:17 -08:00
Mark Rushakoff 6bda17c673 fix(task): use formal errors during task creation
The client expects influxdb.Error, so use them on the server.

Also clarify the message when the task option is missing.
2019-01-22 09:40:51 -08:00
Chris Goller 49782c3be4 feat(bolt): add metrics for resources stored in boltdb 2019-01-18 17:45:45 -06:00
Lyon Hill 523e52d662
Show organization names in task api actions (#11223)
* Show organization names on task actions.

closes #10876
2019-01-18 09:10:14 -07:00
Mark Rushakoff f93ecaa137 refactor(task): look up runs more efficiently
This switches run status from a tag to a field. This is likely a
breaking change to existing task logs.

Using a one-off local query, for 250 records, the previous approach took
around 10 seconds and the new approach is about 30 milliseconds. At 1000
records, the previous approach was roughly 110 seconds and the new
approach is around 70 milliseconds.
2019-01-16 13:25:28 -08:00
Lyon Hill 382de51683
Make task validation errors more clear (#11093)
* fix task validation errors to be more clear
fixes #10987
2019-01-15 13:35:04 -07:00
Michael Desa c60031a6b7 fix(influxdb): change resource to resource type 2019-01-15 12:57:57 -05:00
Nathaniel Cook d0603457b7 refactor(flux): make packages mirror Flux namespaces 2019-01-14 18:00:45 -07:00
Nathaniel Cook 622106caaa Merge branch 'master' into flux-staging 2019-01-14 17:03:57 -07:00
Lyon Hill dfbbfc53cd
Merge pull request #11015 from influxdata/feat/task-validation
feat(task): Add validations for all task actions
2019-01-14 08:16:27 -07:00
Lyon Hill bc9755ae75 fix a naming typo 2019-01-12 10:02:57 -07:00
Jorge Landivar f8f1c1e7b6
Merge pull request #10954 from influxdata/feature/update_task_options
feat(tasks): update task options from api
2019-01-11 14:58:10 -06:00
Nathaniel Cook 3a2231be98 Merge branch 'master' into flux-staging 2019-01-11 13:47:49 -07:00
Lyon Hill d1fc5dc58c vet code 2019-01-11 13:34:30 -07:00
Nathaniel Cook 1708a41fa7 refactor: update query functions for Flux builtins 2019-01-11 13:11:57 -07:00
Lyon Hill 6967734cda feat(task): Add validations for all task actions 2019-01-11 13:09:31 -07:00
j. Emrys Landivar (docmerlin) a2674fa03d feat(tasks): update task options from api 2019-01-11 11:55:46 -06:00
Michael Desa 944de4157a fix(influxdb): add org id to permissions
filter out resources that have mission IDs

fix(influxdb): simplify auth check in PermissionAllowed

review(platform): update as noted in review

fix(influxdb): ensure permission has valid org id
2019-01-11 12:23:28 -05:00
jlapacik 5266cba346
deps: updates for packages and imports (#10962) 2019-01-10 14:33:25 -08:00
Jonathan A. Sternberg a59e6b8d25 refactor: rename DoArrow to Do (#2372)
See influxdata/flux#783 for details.
2019-01-10 10:30:25 -07:00
Jonathan A. Sternberg 6ccb8e8c75 refactor: convert usages of ColReader to ArrowColReader 2019-01-10 10:25:10 -07:00
Mark Rushakoff d73d73c0d4 chore: rename imports from platform to influxdb
I did this with a dumb editor macro, so some comments changed too.

Also rename root package from platform to influxdb.

In interest of minimizing risk, anyone importing the root package has
now aliased it to "platform" so that no changes beyond imports were
necessary in those files.

Lastly, replace the old platform module to local path /dev/null so that
nobody can accidentally reintroduce a platform dependency while
migrating platform code to influxdb.
2019-01-09 20:51:47 -08:00
Michael Desa ad75185629 fix(task): update the permission checked in CreateTask
CreateTasks now check that the user has the write permission to the
tasks resource belonging to an organization. This change comes after
https://github.com/influxdata/platform/pull/2157 modified the structure
of authorization.
2019-01-09 10:36:25 -05:00
Andrew Watkins 3429e8d0c6
feat(auth): rewrite authorization service (#2157)
* feat(view-token-overlay): add view token overlay

* test(tokens): update tests

* chore(auths): rename mock data file

* feat(token-view): clicking on description opens token view modal

* feat(token-view): add ability to close overlay

* feat(token-view): display token permissions with updated permissions shape

* feat(token-view): wip update authorization and permission shapes

* feat(auth): wip refactor auth permissions

* fix(auth): generate permissions via functions

* fix(auth): make Id ID

* chore(types): update generated client

* feat(auth): wip add user and org names to auth

* fix(user): didnt save rebase

* feat(auth): WIP refactor auth

* feat(auth): check for user existence during auth creation

* feat(auth): org must exist during auth creation

* fix(auth): pluralize telegrafs resource type

* docs(http): update swagger definition for the Authorization

* test(auth): fix broken tests

* docs(swagger): update cur_swagger Authrorizations

* fix(api): remove trace from cur_swag

* test(ui/token): update components with new generated type definitions

* feat(http): add lookup service adding names for permissions

* fix(http): remove debugging panics

* chore: go tidy

* fix: unsaved rebase

* test(idpe): add ids to Authorizations for log tests
2018-12-28 15:02:19 -08:00
Mark Rushakoff 0e41f240f2 feat(task): support forcing a run with arbitrary timestamp
Also rename RetryAlreadyQueuedError by running:

gorename -from '"github.com/influxdata/platform/task/backend".RetryAlreadyQueuedError' -to RequestStillQueuedError

and some further manual cleanup for comments.
2018-12-21 14:37:53 -05:00
Chris Goller 31518b7022
Merge pull request #1562 from bednar/task_service
Fixed the task id path, Added RunController instance to pAdapter
2018-12-21 08:48:01 -06:00
Nathaniel Cook 61e36cbee6 chore(Makefile): add target to check generated files are accurate
A standard Makefile is used now in all subdirs that run go generate.
Make will only generate the file if its source files changed.
The checkgenerate target runs clean to ensure all targets a generated
fresh.
2018-12-18 12:54:17 -07:00
Lyon Hill f7ae8c3524
feat(api/task): Add validation for both tasks and buckets on task create (#1900)
fixes #1622
2018-12-12 16:07:40 -07:00
Mark Rushakoff 2e81dd31b4 refactor(task): enable high-frequency WithTicker option
Previously, the WithTicker option would call TickScheduler.Tick every
time the underlying time.Ticker sent a time on its channel. This meant
we used a 1s period, which meant that in the worst case, we would see a
tick at about 999ms after the second rollover.

This change increases the underlying time.Ticker frequency, but only
calls TickScheduler.Tick after a second rolls over. Since we now use a
tick frequency of 100ms, during normal operation, TickScheduler.Tick
will be called within 0.1s after the second rolls over.
2018-12-07 08:38:59 -08:00
Lyon Hill 678acd5307
feat(task): Allow the most recent run time to be shown in the api. (#1765)
* feat(task): Allow the most recent run time to be shown in the api.

When showing tasks in the api latest_completed will now show in the api if it has been run.

fixes #1705
2018-12-06 16:46:16 -07:00
j. Emrys Landivar (docmerlin) 0cb888afc0 Change delay to offset in tasks to match the flux spec 2018-12-06 11:00:07 -06:00