diff --git a/annotations/README.md b/annotations/README.md new file mode 100644 index 0000000000..9644528643 --- /dev/null +++ b/annotations/README.md @@ -0,0 +1,33 @@ +## Annotations + +This package provides an HTTP API for interacting with both annotations and +streams independently. The HTTP handlers are located in the `transport` folder. +The code for interacting with the sqlite datastore is located in the +`service.go` file. Definitions for the basic types & interfaces associated with +annotations and streams used throughout the platform are located in the +top-level `influxdb` package, in the `annotation.go` file. + +### Anatomy + +An annotation is, at its simplest, a textual note on a range of time. The start +and stop time of that range can be the same point in time, which represents an +annotation at a single instance. Annotations can also have "stickers". +"Stickers" allow users to "tag" the annotation with further granularity for +filtering in key-value pairs. Some examples of sticker key-value pairs are: +`"product: oss"`, `"product: cloud"`, or `"service: tasks"`, but keys and values +can be any string. + +Every annotation belongs to a single "stream". A "stream" represents a logical +grouping of annotated events. Some examples of stream names are: `"incidents"`, +`"deployments"`, or `"marketing"`, but can be any string. A stream can also have +a description to further clarify what annotated events may be expected in the +stream. + +### Use + +Requested annotations may be filtered by stream name, stickers, and/or time +range. Streams may also be retrieved, in order to view their description. If a +stream is deleted, all annotations associated with that stream are deleted as +well. Every annotation that is created must have a stream associated with it - +if a stream name is not provided when creating an annotation, it will be +assigned to the default stream. diff --git a/notebooks/README.md b/notebooks/README.md new file mode 100644 index 0000000000..1cf151abcc --- /dev/null +++ b/notebooks/README.md @@ -0,0 +1,25 @@ +## Notebooks + +This package provides an HTTP API for interacting with InfluxDB notebooks. The +HTTP handlers are located in the `transport` folder. The code for interacting +with the sqlite datastore is located in the `service.go` file. Definitions for +the basic types & interfaces associated with notebooks used throughout the +platform are located in the top-level `influxdb` package, in the `notebook.go` +file. + +### Anatomy + +The backend representation of a notebook is very simple: An object with an ID, +Name, associated organization ID, created/modified times, and a "spec". The +"spec" is a mechanism for storing a JSON string defined entirely by the frontend +UI. The notebook spec will likely be further defined in the future as the +notebooks feature is developed and more sophisticated backend behaviors are +required. + +### Use + +Basic CRUD actions are available for interacting with notebooks through the API. +Notebooks are persisted in the relational sqlite database, although they +currently do not make use of any relational features. Again, it is likely that +the more advanced features of the datastore will be utilized in the future as +the notebooks feature evolves. diff --git a/sqlite/README.md b/sqlite/README.md new file mode 100644 index 0000000000..d4942e7d8e --- /dev/null +++ b/sqlite/README.md @@ -0,0 +1,79 @@ +## SQlite + +### Purpose + +This `sqlite` package provides a basic interface for interacting with the +embedded sqlite database used by various InfluxDB services which require storing +relational data. + +The actual sqlite driver is provided by +[`mattn/go-sqlite3`](https://github.com/mattn/go-sqlite3). + +### Usage + +A single instance of `SqlStore` should be created using the `NewSqlStore` +function. Currently, this is done in the top-level `launcher` package, and a +pointer to the `SqlStore` instance is passed to services which require it as +part of their initialization. + +The [`jmoiron/sqlx`](https://github.com/jmoiron/sqlx) package provides a +convenient and lightweight means to write and read structs into and out of the +database and is sufficient for performing simple, static queries. For more +complicated & dynamically constructed queries, the +[`Masterminds/squirrel`](https://github.com/Masterminds/squirrel) package can be +used as a query builder. + +### Concurrent Access + +An interesting aspect of using the file-based sqlite database is that while it +can support multiple concurrent read requests, only a single write request can +be processed at a time. A traditional RDBMS would manage concurrent write +requests on the database server, but for this sqlite implementation write +requests need to be managed in the application code. + +In practice, this means that code intended to mutate the database needs to +obtain a write lock prior to making queries that would result in a change to the +data. If locks are not obtained in the application code, it is possible that +errors will be encountered if concurrent write requests hit the database file at +the same time. + +### Migrations + +A simple migration system is implemented in `migrator.go`. When starting the +influx daemon, the migrator runs migrations defined in `.sql` files using +sqlite-compatible sql scripts. These migration scripts include a statement to +set the `user_version` pragma in the database, and only scripts with a higher +`user_version` than the currently stored pragma are executed. + +When creating new migrations, follow the file naming convention established by +existing migration scripts, which should look like `00XX_script_name.sql`, where +`XX` is the version number. New scripts should have the version number +incremented by 1. + +### In-Memory Database + +When running `influxd` with the `--store=memory` flag, the database will be +opened using the `:memory:` path, and the maximum number of open database +connections is set to 1. Because of the way in-memory databases work with +sqlite, each connection would see a completely new database, so using only a +single connection will ensure that requests to `influxd` will return a +consistent set of data. + +### Backup & Restore + +Methods for backing up and restoring the sqlite database are available on the +`SqlStore` struct. These operations make use of the [sqlite backup +API](https://www.sqlite.org/backup.html) made available by the `go-sqlite3` +driver. It is possible to restore and backup into sqlite databases either stored +in memory or on disk. + +### Sqlite Features / Extensions + +There are many additional features and extensions available, see [the go-sqlite3 +package docs](https://github.com/mattn/go-sqlite3#feature--extension-list) for +the full list. + +We currently use the `sqlite_foreign_keys` and `sqlite_json` extensions for +foreign key support & JSON query support. These features are enabled using +build tags defined in the `Makefile` and `.goreleaser` config for use in +local builds & CI builds respectively.