This moves a few types and constants to the global package so it can be
used without importing the `task/backend` package. These constants are
referenced in non tasks-specific code.
This is needed to break a dependency chain where the task backend will
call into the flux runtime to perform parsing or evaluation of a script
and to prevent the http package from inheriting that dependency.
The tasks subsystem will now use the flux language service to parse and
evaluate flux instead of directly interacting with the parser or
runtime. This helps break the dependency on the libflux parser for the
base influxdb package.
This includes the task notification packages which were changed at the
same time.
* chore: Remove several instances of WithLogger
* chore: unexport Logger fields
* chore: unexport some more Logger fields
* chore: go fmt
chore: fix test
chore: s/logger/log
chore: fix test
chore: revert http.Handler.Handler constructor initialization
* refactor: integrate review feedback, fix all test nop loggers
* refactor: capitalize all log messages
* refactor: rename two logger to log
* feat(task): add limit function for task concurrency
The new task executor handles limit's differently then the old executor
instead of front loading limits by creating a runner for every task that might run
the new executor has a large worker pool and queue. This allow's us to have a unlimited
concurrency per task and helps us avoid a back log of task's execution based on a
arbitrary execution limit. This add's the ability to add an optional task execution limit
so a user can still have the advantages of limiting concurrency.
We needed the coordinator to be able to execute manual runs and resume runs.
These two functions have been added, but we also needed to allow for the executor to be
mocked out. To do that we needed to return a Promise interface instead of an actual
struct. Both these changes are to facilitate coordinator work and testing.
I chose to add a execute function that allow's the task executor to match expectation from
the scheduler but I left in the existing executor method that return's promises. This is
because I like to be able to have the accountablilty and visiblity inside what's happening
with each execution even though the promise isn't required for the scheduler. This function signature
will be used by the coordinator and potentially other's that want to ensure a 'execution' is completed.
Implementations of the backend.Executor produce errors limited to
querying the KV store. The remainder of the errors will be processed
in the implementation of a `RunPromise`.
Fixes#15161
When a task is told to execute it can be enqueued waiting for a worker.
This statistic will be superior to the existing delta based on scheduled for,
the current system can be effected by a user having slow queries or a long "delay" on the task.
This new way of measuring the same thing should allow us to accuratly measure when it is the task system's fault.
* feat(task): impersonate user on task execution
Passing tokens to tasks is cumbersome and we needed a way to more easily create tasks. With this change we no longer need a token on task create. We take the user that created the task and pass that in as the "owner". As far as the task is concerned the owner is the source of permissions.
This is done by adding an additional field on task create that is OwnerID. We will no longer respect the token passed in and it will be deprecated soon.
Things to do still:
Task updates need to allow for owners to be set.
* Report errors found when iterating over flux query in task
* Add failing test for tasks executor result iterator exhaust failure
* Ensure errors exhausting tasks query result iterator are surfaced as task failure
* Update CHANGELOG with task result iteration error surfacing fix
Brett Buddin
Jonathan A. Sternberg
jlapacik
Gavin Cabbage
Christopher M. Wolff
Lorenzo Affetti
Alirie Gray
George
Jacob Marble
Lyon Hill
docmerlin (j. Emrys Landivar)
Stuart Carnie
Nathaniel Cook
Sebastian Borza