Commit Graph

126 Commits (7fb015cc7f88391f9e7ffcc3ad492b1b09d0d536)

Author SHA1 Message Date
Michael Desa 81d1520dc6 fix(platform): keep authorization struct on proxy query request
As a result of pr https://github.com/influxdata/platform/pull/1494, idpe
broke. This PR undoes some of the work done from that PR, but fixes the
underlying issue with #1494.
2018-11-20 15:20:51 -08:00
Michael Desa 407c01cb17 fix(platform): use authorizer instead of authorizion for endpoints
feat(platform): add ToPermissions method to user resource mapping

The ToPermissions method returns a set of permissions that is granted
via a user resource mapping.

feat(bolt): resolve sessions permissions on lookup

feat(http): use authorizer instead of authorization service for write api

feat(bolt): create user resource mappings for org users in bucket create

feat(bolt): create user resource mapping for first org/user

fix(platform): use authorizer for query endpoint instead of authorization

test(http): use cmp instead of reflect for decode test
2018-11-20 10:56:58 -08:00
Brandon Farmer 3c1112d31a Use task status default constant 2018-11-20 09:49:40 -08:00
Brandon Farmer 0bdb3414ae Add updated swagger docs for Task Creation 2018-11-20 09:43:21 -08:00
Chris Goller a8c1155b21 fix(Makefile): remove old clean targets 2018-11-15 20:21:30 -06:00
Mark Rushakoff b0b0d965ff feat(task): integrate task logs with system buckets
- Brought over enterprise's QueryLogReader, with small adjustments
  - Time filters are for the run's ScheduledFor field, per spec
- Adjusted run log timestamps for consistent formatting:
  - ScheduledFor is RFC3339 because it's a whole-second timestamp
  - StartedAt, FinishedAt use RFC3339Nano for high precision
- Several test adjustments to use relative time, for easier integration
  with storage retention
2018-11-15 08:59:15 -08:00
docmerlin (j. Emrys Landivar) 8f8c56ea8d opentracing 2018-11-12 16:34:06 -06:00
Jonathan A. Sternberg fadf4f3e4f
deps: update flux dependency 2018-11-08 14:49:35 -06:00
Mark Rushakoff 188db01b19 fix(task): consume table iterators when executing query
Leaving tables unconsumed seems to cause a variety of problems,
including deadlocks and queries that don't actually execute.
2018-11-05 14:10:37 -08:00
Mark Rushakoff fe84ff7357 chore(task): fix megacheck errors 2018-11-01 12:46:33 -07:00
Mark Rushakoff d9e10ff652 test(task): deflake scheduler test 2018-11-01 10:37:14 -07:00
Mark Rushakoff 39e115cebe fix(task): fix closure mistake when reclaiming tasks
Also improve some the coordinator tests a little bit and improve the
executor logging while we're here.
2018-10-31 19:00:02 -07:00
Nathaniel Cook 2a1d76eb2d chore: Update flux dependency for the type-inference changes 2018-10-31 17:11:11 -06:00
Mark Rushakoff a7e6a6634e test(task): deflake TaskService shared test
What happened here is kind of a long story.

The TaskService tests exercise different parts of the TaskService
concurrently, in hopes of exercising code that may be racy. This worked
fine so long as we were able to use arbitrary user and organization IDs
picked during test.

Then we added tests for the http-client-backed TaskService. The instance
of http.TaskService takes a single token, so the TaskService tests were
updated to use an optional override to get the user and org ID from the
TaskService implementation.

That meant our parallel tests were using one user and org ID in some
cases. One particular test to exercise concurrency would create tasks
and then retrieve them and delete one. Because this test was filtering
by user and org, sometimes it would delete a task created in a different
test.

The fix was to maintain a whitelist of tasks created in that particular
test, so that it wouldn't delete tasks created in a different test.
2018-10-30 18:52:21 -07:00
docmerlin (j. Emrys Landivar) 64883a5dad Use snowflake for bolt task ids 2018-10-30 18:20:59 -05:00
Mark Rushakoff cd80e41220 feat(task): add ability to retry runs
This also adjusts the TaskService interface's RetryRun method to accept
a task ID rather than an org ID. Internally, we still look up runs by
organization, and maybe that will change later, but this is a more
natural way for clients to look it up.
2018-10-30 09:20:40 -07:00
docmerlin (j. Emrys Landivar) 6e2620418c Now you can cancel runs 2018-10-29 16:08:50 -05:00
Mark Rushakoff 3fb86665eb test(task): fix check for setting task RequestedAt 2018-10-26 09:49:45 -07:00
Lyon Hill 751a53dd24
Allow coordinator to pick up existing tasks on startup (#1187) 2018-10-25 13:50:41 -06:00
Mark Rushakoff 8442acd437 refactor(task): emit "ticked" log messages at debug level
The ticked message every second is more annoying than helpful as-is.
Drop it to debug level so it's available if needed.
2018-10-25 11:23:28 -07:00
Mark Rushakoff 43b7617316 test(task): check task status in more methods
This TODO was meant to be removed as part #1179, but was mistakenly
overlooked.
2018-10-24 11:04:07 -07:00
Mark Rushakoff 75c5d91cec feat(task): include task status when listing tasks 2018-10-24 09:38:02 -07:00
Adam 98be8bcd32
moved to_http and to_kafka to flux and update to latest flux (#1175) 2018-10-24 12:33:43 -04:00
Lyon Hill 19b7709bad
Add tasks to the influx cli (#1163) 2018-10-23 11:51:13 -06:00
Mark Rushakoff 8c0aec7975 fix(task): propagate status updates when modifying tasks
This also changes the backend.Store API to remove the EnableTask and
DisableTask methods, merging their functionality into ModifyTask, which
has been named to UpdateTask to keep closer to the CRUD acronym.
2018-10-19 16:49:18 -07:00
Mark Rushakoff 3cf88009c4 test(task): fix incorrect test assertion
We changed the semantics of finding a task that doesn't exist, from
returning `nil, nil`, to `nil, ErrTaskNotFound`; this test didn't
reflect that change until this commit.
2018-10-15 10:24:23 -07:00
Jonathan A. Sternberg 5c8318d181
Merge pull request #1080 from influxdata/js-go-generate-with-go-run
refactor: modify tooling to take advantage of go run
2018-10-15 11:27:13 -05:00
Mark Rushakoff ad4942fe36 refactor(task): expose status when finding from TaskService
And consistently return ErrTaskNotFound, rather than nil, nil, when
searching an ID that doesn't exist.
2018-10-15 09:20:47 -07:00
Mark Rushakoff cb0b54cfd9 feat(task): allow setting task status during creation
This renames TaskEnabled and TaskDisabled to TaskActive and
TaskInactive, to keep in line with the swagger and other parts of the
system. But I left the EnableTask and DisableTask methods on the Store
interface for now. I could see eliminating those methods if we adjust
the signature of the UpdateTask method.
2018-10-15 09:20:47 -07:00
Mark Rushakoff 56f32b4780 refactor(task): extract CreateTaskRequest struct
We're about to add another parameter, so condensing the arguments into a
single struct makes sense at this point.
2018-10-15 09:20:47 -07:00
Jonathan A. Sternberg e9600b1f0b
refactor: modify tooling to take advantage of go run
The generate commands have been modified to take advantage of the new
functionality in Go 1.11 that allows `go run` to execute a package
instead of individual files.

This functionality combined with Go modules allows us to execute a
package directly out of our pinned dependencies rather than accidentally
picking up another binary outside of the build environment.

This also simplifies the Makefile because they no longer have to be
responsible for installing the correct tooling since the Go command
takes care of that logic. It also makes it so that the Makefiles with
file generation can now be invoked from their appropriate subdirectories
so they are contained within the directory itself rather than relying on
values in the top level Makefile.

It is now possible to generate all files within this project by using:

    go generate ./...

Or the Makefile can continue to be used.

This commit also copies over the special copy of `tmpl` that the storage
engine uses within the influxdb repository. It was never copied over so
using `go generate` on these packages did not work.
2018-10-15 11:01:27 -05:00
Lyon Hill e643c434f6
Allow the scheduler to Update tasks. (#1058)
* Allow the scheduler to Update tasks.

* update the coordinator to use the new update action

* remove unclean code
2018-10-12 08:24:22 -06:00
Chris Goller 2bea48f15a refactor(testing): rename MustIDFromString to MustIDBase16 2018-10-11 09:54:19 -05:00
Chris Goller ada7632f98 fix(task/backend): remove extra name by org 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 6725ae6f64 feat(platform/id): method to create an invalid id
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 016c7d04cb fix(task/servicetest): porting servicetest to uint64 IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato bfee4df046 fix(task/backend/bolt): using uint64 IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 5d4d8a7c04 fix(platform/task/backend/executor): porting tests to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 5db4276a71 fix(platform/id): improvements on code using uint64 IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 6587d1b13d fix(task/backend/coordinator): porting test to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 0a75951684 fix(task/backend/storetest): porting test functions to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato ca7ba33d27 fix(task/mock): porting to uint64 IDs 2018-10-11 09:54:18 -05:00
Leonardo Di Donato 9104771408 fix(task/backend): porting to uint64 IDs
Co-Authored-By: Lorenzo Fontana <lo@linux.com>
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato f5dfe0aecc fix(task/backend): create task returns a platform.ID + inmem store
refactoring

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-by: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Leonardo Di Donato 8d7f06cf4b feat(platform/id): using uint64 for platform.IDs
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-Authored-by: Lorenzo Fontana <lo@linux.com>
2018-10-11 09:54:18 -05:00
Jonathan A. Sternberg e7e17d6972
refactor: move the logger package from influxdb to platform
We are moving the necessary code for 2.0 from the influxdb 1.X
repository to the platform 2.0 repository. The logger is an unnecessary
dependency on the old influxdb that is making life more complicated.
2018-10-10 15:49:07 -05:00
Lyon Hill 610faf18e7
Update the scheduler to add a few task logs (#1029) 2018-10-10 12:37:11 -06:00
Mark Rushakoff 35383074f3 chore(task): remove task name uniqueness requirement 2018-10-09 11:46:12 -07:00
Lyon Hill f081ce95f0
allow run shows to work with an actual org id. (#981)
* allow run shows to work with an actual org id.
also necessary for using actual run lists
2018-10-08 14:07:08 -06:00
Brandon Farmer c1c4dc3393 Properly handle spaces in PATH with make tasks 2018-10-08 09:53:22 -07:00