Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/iox_catalog/migrations
History
Marco Neumann 551e838db3
refactor: remove unused PG indices (#7905) 
Similar to #7859. To test index usage, execute the following query on
the writer replica:

```sql
SELECT
    n.nspname                                      AS namespace_name,
    t.relname                                      AS table_name,
    pg_size_pretty(pg_relation_size(t.oid))        AS table_size,
    t.reltuples::bigint                            AS num_rows,
    psai.indexrelname                              AS index_name,
    pg_size_pretty(pg_relation_size(i.indexrelid)) AS index_size,
    CASE WHEN i.indisunique THEN 'Y' ELSE 'N' END  AS "unique",
    psai.idx_scan                                  AS number_of_scans,
    psai.idx_tup_read                              AS tuples_read,
    psai.idx_tup_fetch                             AS tuples_fetched
FROM
    pg_index i
    INNER JOIN pg_class t               ON t.oid = i.indrelid
    INNER JOIN pg_namespace n           ON n.oid = t.relnamespace
    INNER JOIN pg_stat_all_indexes psai ON i.indexrelid = psai.indexrelid
WHERE
    n.nspname = 'iox_catalog' AND t.relname = 'parquet_file'
ORDER BY 1, 2, 5;
````

Data for eu-west-1 at `2023-05-31T16:30:00Z`:

```text
namespace_name |  table_name  | table_size | num_rows  |            index_name             | index_size | unique | number_of_scans |  tuples_read   | tuples_fetched
----------------+--------------+------------+-----------+-----------------------------------+------------+--------+-----------------+----------------+----------------
 iox_catalog    | parquet_file | 38 GB      | 146489216 | parquet_file_deleted_at_idx       | 6442 MB    | N      |      1693534991 | 21602734184385 |    21694365037
 iox_catalog    | parquet_file | 38 GB      | 146489216 | parquet_file_partition_delete_idx | 20 MB      | N      |        17854904 |     3087700816 |      384603858
 iox_catalog    | parquet_file | 38 GB      | 146489216 | parquet_file_partition_idx        | 2325 MB    | N      |      1627977474 | 12604272924323 | 11088781876397
 iox_catalog    | parquet_file | 38 GB      | 146489216 | parquet_file_pkey                 | 8290 MB    | Y      |       480767174 |      481021514 |      480733966
 iox_catalog    | parquet_file | 38 GB      | 146489216 | parquet_file_table_delete_idx     | 174 MB     | N      |         1006563 |    24687617719 |      385132581
 iox_catalog    | parquet_file | 38 GB      | 146489216 | parquet_file_table_idx            | 1905 MB    | N      |         9288042 |   351240529272 |          27551
 iox_catalog    | parquet_file | 38 GB      | 146489216 | parquet_location_unique           | 6076 MB    | Y      |       385294957 |         109448 |         109445
````

and at `2023-06-01T13:00:00Z`:

```text
 namespace_name |  table_name  | table_size | num_rows  |            index_name             | index_size | unique | number_of_scans |  tuples_read   | tuples_fetched
----------------+--------------+------------+-----------+-----------------------------------+------------+--------+-----------------+----------------+----------------
 iox_catalog    | parquet_file | 43 GB      | 152684560 | parquet_file_deleted_at_idx       | 6976 MB    | N      |      1693535032 | 21602834620294 |    21736731439
 iox_catalog    | parquet_file | 43 GB      | 152684560 | parquet_file_partition_delete_idx | 21 MB      | N      |        31468423 |     7397141567 |      677909956
 iox_catalog    | parquet_file | 43 GB      | 152684560 | parquet_file_partition_idx        | 2464 MB    | N      |      1627977474 | 12604272924323 | 11088781876397
 iox_catalog    | parquet_file | 43 GB      | 152684560 | parquet_file_pkey                 | 8785 MB    | Y      |       492762975 |      493017342 |      492729691
 iox_catalog    | parquet_file | 43 GB      | 152684560 | parquet_file_table_delete_idx     | 241 MB     | N      |         1136317 |    24735561304 |      429892231
 iox_catalog    | parquet_file | 43 GB      | 152684560 | parquet_file_table_idx            | 2058 MB    | N      |         9288042 |   351240529272 |          27551
 iox_catalog    | parquet_file | 43 GB      | 152684560 | parquet_location_unique           | 6776 MB    | Y      |       399142416 |         124810 |         124807
````

Due to #7842 and #7894, the following indices are no longer used:

- `parquet_file_partition_idx`
- `parquet_file_table_idx`
 		2023-06-01 13:45:05 +00:00
..
20210217134322_create_schema.sql refactor: do not specify schema in migrations 2022-02-17 14:15:58 +00:00
20211229171744_initial_schema.sql refactor: remove migration create schema 2022-02-17 14:41:32 +00:00
20220221145409_routing_limits.sql feat: routing service protection (#3807) 2022-02-22 17:26:37 +00:00
20220223192306_add_row_count_to_parquet_file.sql feat: Add row count to the parquet_file record in the catalog (#3847) 2022-02-24 15:20:50 +00:00
20220224080910_add_file_size_bytes.sql feat: extend catalog so we can recover `ParquetChunk`s from it (#3852) 2022-02-24 13:16:15 +00:00
20220224082147_add_parquet_metadata.sql feat: extend catalog so we can recover `ParquetChunk`s from it (#3852) 2022-02-24 13:16:15 +00:00
20220307201050_add_parquet_level_and_created_at.sql feat: add compaction_level and created_at to parquet_file (#3972) 2022-03-10 15:56:57 +00:00
20220308120917_ingester-missing-indexes.sql perf: get_table_persist_info indexes for joins 2022-03-08 12:12:47 +00:00
20220323150705_change_to_delete_to_timestamp.sql fix: Change to_delete column on parquet_files to be a time (#4117) 2022-03-23 18:47:27 +00:00
20220324152729_add_namespace_id_to_parquet_file.sql fix: Backfill namespace_id in schema migration (#4177) 2022-03-30 16:31:26 +00:00
20220328150925_add_index_to_to_delete.sql fix: Add an index to parquet_file to_delete 2022-03-29 08:15:26 -04:00
20220331222147_compactor-indexes.sql fix: add indexes to parquet_file (#4198) 2022-04-01 09:59:39 +00:00
20220401174403_add_sort_key_to_partition.sql feat: Add optional sort_key column to partition table 2022-04-01 15:45:51 -04:00
20220404120116_column_name-table_id-index.sql refactor: add table_id index on column_name 2022-04-04 13:04:25 +01:00
20220420142224_ids_bigint.sql fix: make all catalog IDs 64bit (#4418) 2022-04-25 16:49:34 +00:00
20220610102325_add_sort_key_array_column.sql chore: Incrementally migrate sort_key to array type (#4826) 2022-06-10 11:35:43 +00:00
20220610131325_rename_sort_key_arr.sql feat: Change data type of catalog partition's sort_key from a string to an array of string (#4801) 2022-06-10 13:31:31 +00:00
20220620160136_parquet_file_columns.sql refactor: store per-file column set in catalog (#4908) 2022-06-21 10:26:12 +00:00
20220623143430_non_null_parquet_file_columns.sql refactor: ensure that SQL parquet file column sets are not NULL (#4937) 2022-06-24 14:26:18 +00:00
20220624104030_remove_column_sort_key_old.sql chore: remove unused sort_key_old from catalog partition (#4944) 2022-06-24 15:02:38 +00:00
20220627092839_remove_parquet_metadata.sql refactor: parquet file metadata from catalog (#4949) 2022-06-27 15:38:39 +00:00
20220628085556_column_set_via_IDs.sql refactor: use IDs for `parquet_file.column_set` (#4965) 2022-06-30 15:08:41 +00:00
20220721195556_drop_min_sequence_number.sql chore: remove min sequence number from the catalog table as we no longer use it (#5178) 2022-07-21 20:47:55 +00:00
20220819154515_sequencer_to_shard.sql fix: Rename columns, tables, indexes and constraints in postgres catalog 2022-09-01 10:00:54 -04:00
20220902145700_compactor_index.sql feat: add index on parquet_file(shard_id, compaction_level, to_delete, created_at) (#5544) 2022-09-02 14:27:29 +00:00
20220909140048_create_skipped_compactions.sql feat: Record skipped compactions in memory 2022-09-09 15:31:07 -04:00
20220913162700_parquetfile_filesize_trigger.sql chore: renamed DB migration for billing trigger 2022-09-13 16:29:14 +01:00
20220915085930_per-partition-persist-offset.sql refactor(db): persist marker for partition table 2022-09-15 16:10:35 +02:00
20220915155941_nullable-partition-persist-offset.sql refactor(db): NULLable persisted_sequence_number 2022-09-15 18:19:39 +02:00
20220926182450_add_cols_to_skipped_compactions.sql feat: instead of adding num_files and memory budget into the reason text column, let us create differnt columns for them. We will be able to filter them easily (#5742) 2022-09-26 20:14:04 +00:00
20221014122742_lower-default-per-table-column-limit.sql refactor: reduce default column limit 2022-10-14 14:45:48 +02:00
20221101170001_add_cols_for_retention_policy.sql feat: add catalog columns needed for retention policy 2022-11-01 15:35:15 -04:00
20221111133939_drop_retention_duration.sql refactor: remove retention_duration field from namespace catalog table (#6124) 2022-11-11 20:30:42 +00:00
20221118135602_add_col_for_compaction_first_file_in_partition.sql feat: reintroduce compactor first file in partition exception (#6176) 2022-11-18 15:58:59 +00:00
20221215145057_namespace_delete_cascade.sql chore: delete ns postgres impl, test improvements, fix to mem impl 2022-12-16 10:23:50 +00:00
20230104114545_create_indexes_without_shard.sql chore: index for selecting partitions with parquet files created after a given time (#6496) 2023-01-04 18:07:07 +00:00
20230105120822_trigger_update_partition.sql fix: make trigger midification in different file (#6526) 2023-01-06 20:34:48 +00:00
20230106145820_modify_trigger.sql fix: make trigger midification in different file (#6526) 2023-01-06 20:34:48 +00:00
20230113075522_modify_trigger_new_file_at.sql perf: optimize not to update partitions with newly created level 2 files (#6590) 2023-01-13 14:46:58 +00:00
20230125103737_create_max_l0_crrated_at.sql feat: introduce a new way of max_sequence_number for ingester, compactor and querier (#6692) 2023-01-26 10:52:47 +00:00
20230207103859_namespace-soft-delete-column.sql refactor(catalog): soft delete namespace column 2023-02-09 11:35:27 +01:00
20230501173720_add_partition_templates.sql feat: Add columns to store the partition templates 2023-05-24 10:10:34 -04:00
20230522145017_remove_processed_tombstones_table.sql refactor: remove `processed_tombstone` table (#7840) 2023-05-22 15:56:23 +00:00
20230524095337_remove_unused_parquet_file_indices.sql refactor: remove ununused `parquet_file` indices 2023-05-24 12:10:22 +02:00
20230524151854_add_parquet_file_table_and_deleted_index.sql feat: add index for compactor (#7894) 2023-05-31 12:29:00 +00:00
20230530132809_add_parquet_file_partition_and_deleted_index.sql feat: add index for compactor (#7894) 2023-05-31 12:29:00 +00:00
20230601130730_remove_unused_parquet_file_indices_2.sql refactor: remove unused PG indices (#7905) 2023-06-01 13:45:05 +00:00