903d30d658
* test: use `T.TempDir` to create temporary test directory
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* test: fix failing TestSendWrite on Windows
=== FAIL: replications/internal TestSendWrite (0.29s)
logger.go:130: 2022-06-23T13:00:54.290Z DEBUG Created new durable queue for replication stream {"id": "0000000000000001", "path": "C:\\Users\\circleci\\AppData\\Local\\Temp\\TestSendWrite1627281409\\001\\replicationq\\0000000000000001"}
logger.go:130: 2022-06-23T13:00:54.457Z ERROR Error in replication stream {"replication_id": "0000000000000001", "error": "remote timeout", "retries": 1}
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\circleci\AppData\Local\Temp\TestSendWrite1627281409\001\replicationq\0000000000000001\1: The process cannot access the file because it is being used by another process.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* test: fix failing TestStore_BadShard on Windows
=== FAIL: tsdb TestStore_BadShard (0.09s)
logger.go:130: 2022-06-23T12:18:21.827Z INFO Using data dir {"service": "store", "path": "C:\\Users\\circleci\\AppData\\Local\\Temp\\TestStore_BadShard1363295568\\001"}
logger.go:130: 2022-06-23T12:18:21.827Z INFO Compaction settings {"service": "store", "max_concurrent_compactions": 2, "throughput_bytes_per_second": 50331648, "throughput_bytes_per_second_burst": 50331648}
logger.go:130: 2022-06-23T12:18:21.828Z INFO Open store (start) {"service": "store", "op_name": "tsdb_open", "op_event": "start"}
logger.go:130: 2022-06-23T12:18:21.828Z INFO Open store (end) {"service": "store", "op_name": "tsdb_open", "op_event": "end", "op_elapsed": "77.3µs"}
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\circleci\AppData\Local\Temp\TestStore_BadShard1363295568\002\data\db0\rp0\1\index\0\L0-00000001.tsl: The process cannot access the file because it is being used by another process.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* test: fix failing TestPartition_PrependLogFile_Write_Fail and TestPartition_Compact_Write_Fail on Windows
=== FAIL: tsdb/index/tsi1 TestPartition_PrependLogFile_Write_Fail/write_MANIFEST (0.06s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\circleci\AppData\Local\Temp\TestPartition_PrependLogFile_Write_Failwrite_MANIFEST656030081\002\0\L0-00000003.tsl: The process cannot access the file because it is being used by another process.
--- FAIL: TestPartition_PrependLogFile_Write_Fail/write_MANIFEST (0.06s)
=== FAIL: tsdb/index/tsi1 TestPartition_Compact_Write_Fail/write_MANIFEST (0.08s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\circleci\AppData\Local\Temp\TestPartition_Compact_Write_Failwrite_MANIFEST3398667527\002\0\L0-00000003.tsl: The process cannot access the file because it is being used by another process.
--- FAIL: TestPartition_Compact_Write_Fail/write_MANIFEST (0.08s)
We must close the open file descriptor otherwise the temporary file
cannot be cleaned up on Windows.
Fixes:
|
||
|---|---|---|
| .. | ||
| migrations | ||
| test_migrations | ||
| README.md | ||
| migrator.go | ||
| migrator_test.go | ||
| sqlite.go | ||
| sqlite_helpers.go | ||
| sqlite_test.go | ||
README.md
SQlite
Purpose
This sqlite package provides a basic interface for interacting with the
embedded sqlite database used by various InfluxDB services which require storing
relational data.
The actual sqlite driver is provided by
mattn/go-sqlite3.
Usage
A single instance of SqlStore should be created using the NewSqlStore
function. Currently, this is done in the top-level launcher package, and a
pointer to the SqlStore instance is passed to services which require it as
part of their initialization.
The jmoiron/sqlx package provides a
convenient and lightweight means to write and read structs into and out of the
database and is sufficient for performing simple, static queries. For more
complicated & dynamically constructed queries, the
Masterminds/squirrel package can be
used as a query builder.
Concurrent Access
An interesting aspect of using the file-based sqlite database is that while it can support multiple concurrent read requests, only a single write request can be processed at a time. A traditional RDBMS would manage concurrent write requests on the database server, but for this sqlite implementation write requests need to be managed in the application code.
In practice, this means that code intended to mutate the database needs to obtain a write lock prior to making queries that would result in a change to the data. If locks are not obtained in the application code, it is possible that errors will be encountered if concurrent write requests hit the database file at the same time.
Migrations
A simple migration system is implemented in migrator.go. When starting the
influx daemon, the migrator runs migrations defined in .sql files using
sqlite-compatible sql scripts. Records of these migrations are maintained in a
table called "migrations". If records of migrations exist in the "migrations"
table that are not embedded in the binary, an error will be raised on startup.
When creating new migrations, follow the file naming convention established by
existing migration scripts, which should look like 00XX_script_name.up.sql &
00xx_script_name.down.sql for the "up" and "down" migration, where XX is the
version number. New scripts should have the version number incremented by 1.
The "up" migrations are run when starting the influx daemon and when metadata
backups are restored. The "down" migrations are run with the influxd downgrade
command.
In-Memory Database
When running influxd with the --store=memory flag, the database will be
opened using the :memory: path, and the maximum number of open database
connections is set to 1. Because of the way in-memory databases work with
sqlite, each connection would see a completely new database, so using only a
single connection will ensure that requests to influxd will return a
consistent set of data.
Backup & Restore
Methods for backing up and restoring the sqlite database are available on the
SqlStore struct. These operations make use of the sqlite backup
API made available by the go-sqlite3
driver. It is possible to restore and backup into sqlite databases either stored
in memory or on disk.
Sqlite Features / Extensions
There are many additional features and extensions available, see the go-sqlite3 package docs for the full list.
We currently use the sqlite_foreign_keys and sqlite_json extensions for
foreign key support & JSON query support. These features are enabled using
build tags defined in the Makefile and .goreleaser config for use in
local builds & CI builds respectively.