Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/query/iterator_mapper.go

51 lines
1.4 KiB
Go
Raw Normal View History

Separate the query engine into a separate package This change provides a clear separation between the query engine mechanics and the query language so that the language can be parsed and dealt with separate from the query engine itself.
2017-08-15 19:24:22 +00:00
package query
Support subquery execution in the query language This adds query syntax support for subqueries and adds support to the query engine to execute queries on subqueries. Subqueries act as a source for another query. It is the equivalent of writing the results of a query to a temporary database, executing a query on that temporary database, and then deleting the database (except this is all performed in-memory). The syntax is like this: SELECT sum(derivative) FROM (SELECT derivative(mean(value)) FROM cpu GROUP BY *) This will execute derivative and then sum the result of those derivatives. Another example: SELECT max(min) FROM (SELECT min(value) FROM cpu GROUP BY host) This would let you find the maximum minimum value of each host. There is complete freedom to mix subqueries with auxiliary fields. The only caveat is that the following two queries: SELECT mean(value) FROM cpu SELECT mean(value) FROM (SELECT value FROM cpu) Have different performance characteristics. The first will calculate `mean(value)` at the shard level and will be faster, especially when it comes to clustered setups. The second will process the mean at the top level and will not include that optimization.
2016-11-23 20:32:42 +00:00
Modify the Select call to return a Cursor The Cursor returned will be capable of scanning rows into a structure. It replaces part of the function for why the Emitter existed. The Emitter would both join the resulting rows and then transform the values into a models.Row so it could be returned to the results. In the future, we will be able to use the Cursor directly to write out values which should be more memory efficient.
2018-03-01 20:12:22 +00:00
import (
"fmt"
"github.com/influxdata/influxql"
)
Support subquery execution in the query language This adds query syntax support for subqueries and adds support to the query engine to execute queries on subqueries. Subqueries act as a source for another query. It is the equivalent of writing the results of a query to a temporary database, executing a query on that temporary database, and then deleting the database (except this is all performed in-memory). The syntax is like this: SELECT sum(derivative) FROM (SELECT derivative(mean(value)) FROM cpu GROUP BY *) This will execute derivative and then sum the result of those derivatives. Another example: SELECT max(min) FROM (SELECT min(value) FROM cpu GROUP BY host) This would let you find the maximum minimum value of each host. There is complete freedom to mix subqueries with auxiliary fields. The only caveat is that the following two queries: SELECT mean(value) FROM cpu SELECT mean(value) FROM (SELECT value FROM cpu) Have different performance characteristics. The first will calculate `mean(value)` at the shard level and will be faster, especially when it comes to clustered setups. The second will process the mean at the top level and will not include that optimization.
2016-11-23 20:32:42 +00:00
Properly select a tag within a subquery Previously, subqueries would only be able to select tag fields within a subquery if the subquery used a selector. But, it didn't filter out aggregates like `mean()` so it would panic instead. Now it is possible to select the tag directly instead of rewriting the query in an invalid way. Some queries in this form will still not work though. For example, the following still does not function (but also doesn't panic): SELECT count(host) FROM (SELECT mean, host FROM (SELECT mean(value) FROM cpu GROUP BY host))
2017-02-13 22:23:10 +00:00
type IteratorMap interface {
Modify the Select call to return a Cursor The Cursor returned will be capable of scanning rows into a structure. It replaces part of the function for why the Emitter existed. The Emitter would both join the resulting rows and then transform the values into a models.Row so it could be returned to the results. In the future, we will be able to use the Cursor directly to write out values which should be more memory efficient.
2018-03-01 20:12:22 +00:00
Value(row *Row) interface{}
Properly select a tag within a subquery Previously, subqueries would only be able to select tag fields within a subquery if the subquery used a selector. But, it didn't filter out aggregates like `mean()` so it would panic instead. Now it is possible to select the tag directly instead of rewriting the query in an invalid way. Some queries in this form will still not work though. For example, the following still does not function (but also doesn't panic): SELECT count(host) FROM (SELECT mean, host FROM (SELECT mean(value) FROM cpu GROUP BY host))
2017-02-13 22:23:10 +00:00
}
type FieldMap int
Modify the Select call to return a Cursor The Cursor returned will be capable of scanning rows into a structure. It replaces part of the function for why the Emitter existed. The Emitter would both join the resulting rows and then transform the values into a models.Row so it could be returned to the results. In the future, we will be able to use the Cursor directly to write out values which should be more memory efficient.
2018-03-01 20:12:22 +00:00
func (i FieldMap) Value(row *Row) interface{} { return row.Values[i] }
Properly select a tag within a subquery Previously, subqueries would only be able to select tag fields within a subquery if the subquery used a selector. But, it didn't filter out aggregates like `mean()` so it would panic instead. Now it is possible to select the tag directly instead of rewriting the query in an invalid way. Some queries in this form will still not work though. For example, the following still does not function (but also doesn't panic): SELECT count(host) FROM (SELECT mean, host FROM (SELECT mean(value) FROM cpu GROUP BY host))
2017-02-13 22:23:10 +00:00
type TagMap string
Modify the Select call to return a Cursor The Cursor returned will be capable of scanning rows into a structure. It replaces part of the function for why the Emitter existed. The Emitter would both join the resulting rows and then transform the values into a models.Row so it could be returned to the results. In the future, we will be able to use the Cursor directly to write out values which should be more memory efficient.
2018-03-01 20:12:22 +00:00
func (s TagMap) Value(row *Row) interface{} { return row.Series.Tags.Value(string(s)) }
Properly select a tag within a subquery Previously, subqueries would only be able to select tag fields within a subquery if the subquery used a selector. But, it didn't filter out aggregates like `mean()` so it would panic instead. Now it is possible to select the tag directly instead of rewriting the query in an invalid way. Some queries in this form will still not work though. For example, the following still does not function (but also doesn't panic): SELECT count(host) FROM (SELECT mean, host FROM (SELECT mean(value) FROM cpu GROUP BY host))
2017-02-13 22:23:10 +00:00
Refactor the subquery code and fix outer condition queries This change refactors the subquery code into a separate builder class to help allow for more reuse and make the functions smaller and easier to read. The previous function that handled most of the code was too big and impossible to reason through. This also goes and replaces the complicated logic of aggregates that had a subquery source with the simpler IteratorMapper. I think the overhead from the IteratorMapper will be more, but I also believe that the actual code is simpler and more robust to produce more accurate answers. It might be a future project to optimize that section of code, but I don't have any actual numbers for the efficiency of one method and I believe accuracy and code clarity may be more important at the moment since I am otherwise incapable of reading my own code.
2017-02-28 21:40:43 +00:00
type NullMap struct{}
Modify the Select call to return a Cursor The Cursor returned will be capable of scanning rows into a structure. It replaces part of the function for why the Emitter existed. The Emitter would both join the resulting rows and then transform the values into a models.Row so it could be returned to the results. In the future, we will be able to use the Cursor directly to write out values which should be more memory efficient.
2018-03-01 20:12:22 +00:00
func (NullMap) Value(row *Row) interface{} { return nil }
Refactor the subquery code and fix outer condition queries This change refactors the subquery code into a separate builder class to help allow for more reuse and make the functions smaller and easier to read. The previous function that handled most of the code was too big and impossible to reason through. This also goes and replaces the complicated logic of aggregates that had a subquery source with the simpler IteratorMapper. I think the overhead from the IteratorMapper will be more, but I also believe that the actual code is simpler and more robust to produce more accurate answers. It might be a future project to optimize that section of code, but I don't have any actual numbers for the efficiency of one method and I believe accuracy and code clarity may be more important at the moment since I am otherwise incapable of reading my own code.
2017-02-28 21:40:43 +00:00
Modify the Select call to return a Cursor The Cursor returned will be capable of scanning rows into a structure. It replaces part of the function for why the Emitter existed. The Emitter would both join the resulting rows and then transform the values into a models.Row so it could be returned to the results. In the future, we will be able to use the Cursor directly to write out values which should be more memory efficient.
2018-03-01 20:12:22 +00:00
func NewIteratorMapper(cur Cursor, driver IteratorMap, fields []IteratorMap, opt IteratorOptions) Iterator {
Refactor the subquery code and fix outer condition queries This change refactors the subquery code into a separate builder class to help allow for more reuse and make the functions smaller and easier to read. The previous function that handled most of the code was too big and impossible to reason through. This also goes and replaces the complicated logic of aggregates that had a subquery source with the simpler IteratorMapper. I think the overhead from the IteratorMapper will be more, but I also believe that the actual code is simpler and more robust to produce more accurate answers. It might be a future project to optimize that section of code, but I don't have any actual numbers for the efficiency of one method and I believe accuracy and code clarity may be more important at the moment since I am otherwise incapable of reading my own code.
2017-02-28 21:40:43 +00:00
if driver != nil {
switch driver := driver.(type) {
case FieldMap:
Modify the Select call to return a Cursor The Cursor returned will be capable of scanning rows into a structure. It replaces part of the function for why the Emitter existed. The Emitter would both join the resulting rows and then transform the values into a models.Row so it could be returned to the results. In the future, we will be able to use the Cursor directly to write out values which should be more memory efficient.
2018-03-01 20:12:22 +00:00
switch typ := cur.Columns()[int(driver)].Type; typ {
case influxql.Float:
return newFloatIteratorMapper(cur, driver, fields, opt)
case influxql.Integer:
return newIntegerIteratorMapper(cur, driver, fields, opt)
case influxql.Unsigned:
return newUnsignedIteratorMapper(cur, driver, fields, opt)
case influxql.String:
return newStringIteratorMapper(cur, driver, fields, opt)
case influxql.Boolean:
return newBooleanIteratorMapper(cur, driver, fields, opt)
Refactor the subquery code and fix outer condition queries This change refactors the subquery code into a separate builder class to help allow for more reuse and make the functions smaller and easier to read. The previous function that handled most of the code was too big and impossible to reason through. This also goes and replaces the complicated logic of aggregates that had a subquery source with the simpler IteratorMapper. I think the overhead from the IteratorMapper will be more, but I also believe that the actual code is simpler and more robust to produce more accurate answers. It might be a future project to optimize that section of code, but I don't have any actual numbers for the efficiency of one method and I believe accuracy and code clarity may be more important at the moment since I am otherwise incapable of reading my own code.
2017-02-28 21:40:43 +00:00
default:
Modify the Select call to return a Cursor The Cursor returned will be capable of scanning rows into a structure. It replaces part of the function for why the Emitter existed. The Emitter would both join the resulting rows and then transform the values into a models.Row so it could be returned to the results. In the future, we will be able to use the Cursor directly to write out values which should be more memory efficient.
2018-03-01 20:12:22 +00:00
panic(fmt.Sprintf("unable to map iterator type: %s", typ))
Refactor the subquery code and fix outer condition queries This change refactors the subquery code into a separate builder class to help allow for more reuse and make the functions smaller and easier to read. The previous function that handled most of the code was too big and impossible to reason through. This also goes and replaces the complicated logic of aggregates that had a subquery source with the simpler IteratorMapper. I think the overhead from the IteratorMapper will be more, but I also believe that the actual code is simpler and more robust to produce more accurate answers. It might be a future project to optimize that section of code, but I don't have any actual numbers for the efficiency of one method and I believe accuracy and code clarity may be more important at the moment since I am otherwise incapable of reading my own code.
2017-02-28 21:40:43 +00:00
}
case TagMap:
Modify the Select call to return a Cursor The Cursor returned will be capable of scanning rows into a structure. It replaces part of the function for why the Emitter existed. The Emitter would both join the resulting rows and then transform the values into a models.Row so it could be returned to the results. In the future, we will be able to use the Cursor directly to write out values which should be more memory efficient.
2018-03-01 20:12:22 +00:00
return newStringIteratorMapper(cur, driver, fields, opt)
Refactor the subquery code and fix outer condition queries This change refactors the subquery code into a separate builder class to help allow for more reuse and make the functions smaller and easier to read. The previous function that handled most of the code was too big and impossible to reason through. This also goes and replaces the complicated logic of aggregates that had a subquery source with the simpler IteratorMapper. I think the overhead from the IteratorMapper will be more, but I also believe that the actual code is simpler and more robust to produce more accurate answers. It might be a future project to optimize that section of code, but I don't have any actual numbers for the efficiency of one method and I believe accuracy and code clarity may be more important at the moment since I am otherwise incapable of reading my own code.
2017-02-28 21:40:43 +00:00
default:
Modify the Select call to return a Cursor The Cursor returned will be capable of scanning rows into a structure. It replaces part of the function for why the Emitter existed. The Emitter would both join the resulting rows and then transform the values into a models.Row so it could be returned to the results. In the future, we will be able to use the Cursor directly to write out values which should be more memory efficient.
2018-03-01 20:12:22 +00:00
panic(fmt.Sprintf("unable to create iterator mapper with driver expression type: %T", driver))
Refactor the subquery code and fix outer condition queries This change refactors the subquery code into a separate builder class to help allow for more reuse and make the functions smaller and easier to read. The previous function that handled most of the code was too big and impossible to reason through. This also goes and replaces the complicated logic of aggregates that had a subquery source with the simpler IteratorMapper. I think the overhead from the IteratorMapper will be more, but I also believe that the actual code is simpler and more robust to produce more accurate answers. It might be a future project to optimize that section of code, but I don't have any actual numbers for the efficiency of one method and I believe accuracy and code clarity may be more important at the moment since I am otherwise incapable of reading my own code.
2017-02-28 21:40:43 +00:00
}
Support subquery execution in the query language This adds query syntax support for subqueries and adds support to the query engine to execute queries on subqueries. Subqueries act as a source for another query. It is the equivalent of writing the results of a query to a temporary database, executing a query on that temporary database, and then deleting the database (except this is all performed in-memory). The syntax is like this: SELECT sum(derivative) FROM (SELECT derivative(mean(value)) FROM cpu GROUP BY *) This will execute derivative and then sum the result of those derivatives. Another example: SELECT max(min) FROM (SELECT min(value) FROM cpu GROUP BY host) This would let you find the maximum minimum value of each host. There is complete freedom to mix subqueries with auxiliary fields. The only caveat is that the following two queries: SELECT mean(value) FROM cpu SELECT mean(value) FROM (SELECT value FROM cpu) Have different performance characteristics. The first will calculate `mean(value)` at the shard level and will be faster, especially when it comes to clustered setups. The second will process the mean at the top level and will not include that optimization.
2016-11-23 20:32:42 +00:00
}
Modify the Select call to return a Cursor The Cursor returned will be capable of scanning rows into a structure. It replaces part of the function for why the Emitter existed. The Emitter would both join the resulting rows and then transform the values into a models.Row so it could be returned to the results. In the future, we will be able to use the Cursor directly to write out values which should be more memory efficient.
2018-03-01 20:12:22 +00:00
return newFloatIteratorMapper(cur, nil, fields, opt)
Support subquery execution in the query language This adds query syntax support for subqueries and adds support to the query engine to execute queries on subqueries. Subqueries act as a source for another query. It is the equivalent of writing the results of a query to a temporary database, executing a query on that temporary database, and then deleting the database (except this is all performed in-memory). The syntax is like this: SELECT sum(derivative) FROM (SELECT derivative(mean(value)) FROM cpu GROUP BY *) This will execute derivative and then sum the result of those derivatives. Another example: SELECT max(min) FROM (SELECT min(value) FROM cpu GROUP BY host) This would let you find the maximum minimum value of each host. There is complete freedom to mix subqueries with auxiliary fields. The only caveat is that the following two queries: SELECT mean(value) FROM cpu SELECT mean(value) FROM (SELECT value FROM cpu) Have different performance characteristics. The first will calculate `mean(value)` at the shard level and will be faster, especially when it comes to clustered setups. The second will process the mean at the top level and will not include that optimization.
2016-11-23 20:32:42 +00:00
}