Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
5,920 Commits
189 Branches
453 Tags
379 MiB
Commit Graph

5920 Commits (b5a7bf03daae67081d43f58ecb57a1432d1aa685)

Author SHA1 Message Date
Marco Neumann 16fc610433 chore: faster and smaller IOx binaries 
Use `codegen-units = 1`, thin-LTO and debug section compression to make our binary smaller (which is good for deploy and
test times) and faster.

# Summary

The binary size of `influxdb_iox` after building with:

```console
$ cargo build --release --no-default-features --features="aws,gcp,azure,jemalloc_replacing_malloc"
```

The profile was:

```toml
[profile.release]
debug = true
```

The commit was:

```text
89ece8b493
```

The size results are:

| Method                                     | Size  |
| ------------------------------------------ | ----- |
| baseline                                   | 833MB |
| baseline + dbg compression                 | 222MB |
| baseline + strip                           |  49MB |
| codegen-units                              | 520MB |
| codegen-units + strip                      |  40MB |
| codegen-units + dbg compression            | 143MB |
| thin LTO                                   | 715MB |
| thin LTO + strip                           |  49MB |
| thin LTO + dbg compression                 | 199MB |
| codegen-units + thin LTO                   | 449MB |
| codegen-units + thin LTO + strip           |  40MB |
| codegen-units + thin LTO + dbg compression | 130MB |

For the methods that were successfully measured I couldn't really see any compile time differences on my laptop.

# Methods

## Strip
Remove debug symbols. We don't really want this, so this is just to get an idea of the size

```console
$ strip baseline
```

## Debug Sections compression
Debug sections make a large amount of our binary size (a stripped executable is 49MB instead of 833MB). Since we like to
have debug symbols we cannot just strip them. However these symbols are only used for:

- backtrace generation (something went wrong, not BAU)
- profiling
- debugging

So in normal operation and most test scenarios, we're just wasting memory. So we could compress them:

```console
$ objcopy --compress-debug-sections baseline baseline-dbg_compressed
```

There is also elfutils:

```console
$ eu-elfcompress test
```

Elfutils nearly ends up with the same size (220MB instead of 222MB that objcopy achieves), but takes more time and is
probably not worth it.

Note that compressed debug sections exist since many years. The Rust ecosystem supports reading them since over a year,
see:

- <https://github.com/gimli-rs/gimli/issues/195>
- <https://github.com/rust-lang/backtrace-rs/issues/342>

## Codegen Units
The rust compiler parallelizes codegen work. This split into units however means that optimizations are somewhat
limited. This can be change by:

```toml
[profile.release]
...
codegen-units = 1
```

As a nice side effect this should also make our code faster.

## Thin LTO
Get LLVM to run "thin" Link Time Optimization:

```toml
[profile.release]
...
lto = "thin"
```

As a nice side effect this should also make our code faster.

## Fat LTO
Get LLVM to run "fat" Link Time Optimization:

```toml
[profile.release]
...
lto = "fat"
```

There are no results for this because this took a massive amount of memory and CPU time and did not finish on my system.
 2021-11-10 15:49:19 +01:00
kodiakhq[bot] a439404e82
Merge pull request #3079 from influxdata/crepererum/fix_docker 
ci: fix usage of old docker version
 2021-11-10 09:50:38 +00:00
Marco Neumann 89ece8b493 ci: fix usage of old docker version 
`--all-tags` as introduced for version 20.10 but appearently we're
running an older version.
 2021-11-10 10:27:16 +01:00
kodiakhq[bot] 7261648ed7
Merge pull request #3078 from influxdata/crepererum/docker_latest_tag 
ci: also push `:main` tag in addition to SHA-based ones
 2021-11-10 09:03:38 +00:00
Marco Neumann ae58ae9c81 ci: also push `:main` tag in addition to SHA-based ones 2021-11-10 09:06:19 +01:00
kodiakhq[bot] 86ea00f53c
Merge pull request #3072 from influxdata/jgm-github-devex 
chore(github): new issue template for developer experience
 2021-11-09 13:28:33 +00:00
kodiakhq[bot] 83635ea612
Merge branch 'main' into jgm-github-devex 2021-11-09 13:20:43 +00:00
kodiakhq[bot] 05084aa722
Merge pull request #3074 from influxdata/crepererum/router_write_http_prep 
refactor: partially split up HTTP implementation and tests
 2021-11-09 12:19:48 +00:00
Marco Neumann 721011ab51 docs: improve wording 
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
 2021-11-09 13:11:32 +01:00
Marco Neumann 00b3282180 refactor: partially split up HTTP implementation and tests 
This should make it easier to to wire up the HTTP write interface for
the router. No functional change intended, this just moves code around.
 2021-11-09 12:22:31 +01:00
kodiakhq[bot] 17ecec6d37
Merge pull request #3067 from influxdata/crepererum/router_write_grpc 
feat: gRPC PB-write interface for new router
 2021-11-09 08:27:18 +00:00
kodiakhq[bot] c18acdcbcd
Merge branch 'main' into crepererum/router_write_grpc 2021-11-09 08:20:00 +00:00
Jacob Marble a4dd8b8e15
chore(github): new issue template for developer experience 
Just like it says
 2021-11-08 15:58:30 -08:00
kodiakhq[bot] e00435bc55
Merge pull request #3052 from influxdata/cn/assert-error-macro 
refactor: Extract a macro for matching errors in tests
 2021-11-08 16:33:36 +00:00
Carol (Nichols || Goulding) 9c40bd87fb
feat: Stringify and print error pattern on assert_error failure 2021-11-08 11:14:23 -05:00
Carol (Nichols || Goulding) 3dda694520
refactor: Extract a macro for matching errors in tests 2021-11-08 11:14:23 -05:00
kodiakhq[bot] b0c23df30f
Merge pull request #3069 from influxdata/ntran/order_output 
test: add group by tests
 2021-11-08 15:55:46 +00:00
Nga Tran 71731524c4 test: add group by tests 2021-11-08 10:46:40 -05:00
Marco Neumann 83dbfe1333 feat: gRPC PB-write interface for new router 2021-11-08 15:49:13 +01:00
kodiakhq[bot] 6fdfaef72c
Merge pull request #3054 from influxdata/tran/rub_timestamp 
fix: statistics for max/min(time) should have data type timstamp
 2021-11-08 14:36:29 +00:00
kodiakhq[bot] 2e7cf8fb2b
Merge branch 'main' into tran/rub_timestamp 2021-11-08 14:29:11 +00:00
Nga Tran abbfafcabd chore: merge main to branch 2021-11-08 09:28:29 -05:00
Marco Neumann d0df1a14b4
ci: fix quay login for image promotion (#3066) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-08 14:17:36 +00:00
kodiakhq[bot] 36cc592336
Merge pull request #3042 from influxdata/crepererum/router_core_logic 
feat: router core logic
 2021-11-08 14:05:39 +00:00
kodiakhq[bot] 4a529837e9
Merge branch 'main' into crepererum/router_core_logic 2021-11-08 13:58:28 +00:00
Raphael Taylor-Davies 6320ce6f55
refactor: move delete predicate proto to own package (#2731) (#3065) 
* refactor: move delete predicate proto to own package (#2731)

* chore: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-08 13:42:26 +00:00
kodiakhq[bot] 7277149bd1
Merge pull request #3064 from influxdata/crepererum/fix_get_deploy_tags 
ci: fix `get-deploy-tags.sh`
 2021-11-08 13:21:09 +00:00
Marco Neumann 6786fd5955 ci: fix `get-deploy-tags.sh` 2021-11-08 14:12:03 +01:00
Marco Neumann ea2604b7b6 docs: explain write sink test 2021-11-08 14:09:33 +01:00
Marco Neumann 4130d508c0 docs: document write buffer variants 2021-11-08 14:09:33 +01:00
Marco Neumann 1d34ba0183 docs: explain router name argument 2021-11-08 14:09:33 +01:00
Marco Neumann 12c684b546 test: extend `test_write` test 2021-11-08 14:09:33 +01:00
Marco Neumann dcc376456a docs: explain iteration order in `Router` 2021-11-08 14:09:33 +01:00
Marco Neumann 6a090b785a fix: use new `WriteMeta` constructor 2021-11-08 14:09:33 +01:00
Marco Neumann b226d629c1 feat: router write logic 2021-11-08 14:09:33 +01:00
Marco Neumann c745a0a94c feat: impl write sinks logic for router 2021-11-08 14:09:33 +01:00
Marco Neumann c1fa41196d feat: impl `Display` for `ShardId` 2021-11-08 14:09:33 +01:00
Marco Neumann fc4ea71921 feat: `RouterServer::router` 2021-11-08 14:09:33 +01:00
kodiakhq[bot] 897f5c9599
Merge pull request #3061 from influxdata/crepererum/new_quay_image 
ci: use new, public `quay.io/influxdb/iox` image
 2021-11-08 12:43:54 +00:00
kodiakhq[bot] 5f0a05bbb9
Merge branch 'main' into crepererum/new_quay_image 2021-11-08 12:36:36 +00:00
kodiakhq[bot] 34ebc37761
Merge pull request #3062 from influxdata/dependabot/cargo/libc-0.2.107 
chore(deps): bump libc from 0.2.106 to 0.2.107
 2021-11-08 11:32:18 +00:00
dependabot[bot] f6bbf7b617
chore(deps): bump libc from 0.2.106 to 0.2.107 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.106 to 0.2.107.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.106...0.2.107)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-08 11:18:48 +00:00
kodiakhq[bot] 65adc13c36
Merge pull request #3059 from influxdata/dependabot/cargo/nom-7.1.0 
chore(deps): bump nom from 7.0.0 to 7.1.0
 2021-11-08 11:15:29 +00:00
dependabot[bot] 9ce86f4f8e
chore(deps): bump nom from 7.0.0 to 7.1.0 
Bumps [nom](https://github.com/Geal/nom) from 7.0.0 to 7.1.0.
- [Release notes](https://github.com/Geal/nom/releases)
- [Changelog](https://github.com/Geal/nom/blob/main/CHANGELOG.md)
- [Commits](https://github.com/Geal/nom/commits)

---
updated-dependencies:
- dependency-name: nom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-08 11:06:18 +00:00
kodiakhq[bot] 8d9aeaae6d
Merge pull request #3058 from influxdata/dependabot/cargo/comfy-table-5.0.0 
chore(deps): bump comfy-table from 4.1.1 to 5.0.0
 2021-11-08 11:04:15 +00:00
dependabot[bot] e4d943d79c
chore(deps): bump comfy-table from 4.1.1 to 5.0.0 
Bumps [comfy-table](https://github.com/nukesor/comfy-table) from 4.1.1 to 5.0.0.
- [Release notes](https://github.com/nukesor/comfy-table/releases)
- [Changelog](https://github.com/Nukesor/comfy-table/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nukesor/comfy-table/compare/v4.1.1...v5.0.0)

---
updated-dependencies:
- dependency-name: comfy-table
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-08 10:55:57 +00:00
kodiakhq[bot] bfe7eddb4a
Merge pull request #3057 from influxdata/dependabot/cargo/handlebars-4.1.4 
chore(deps): bump handlebars from 4.1.3 to 4.1.4
 2021-11-08 10:54:03 +00:00
dependabot[bot] af6694299b
chore(deps): bump handlebars from 4.1.3 to 4.1.4 
Bumps [handlebars](https://github.com/sunng87/handlebars-rust) from 4.1.3 to 4.1.4.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases)
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.1.3...v4.1.4)

---
updated-dependencies:
- dependency-name: handlebars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-08 10:42:46 +00:00
kodiakhq[bot] ae5900b8ef
Merge pull request #3056 from influxdata/dependabot/cargo/serde_json-1.0.69 
chore(deps): bump serde_json from 1.0.68 to 1.0.69
 2021-11-08 10:40:41 +00:00
dependabot[bot] 7214fa9d6a
chore(deps): bump serde_json from 1.0.68 to 1.0.69 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.68 to 1.0.69.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.68...v1.0.69)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-08 10:31:33 +00:00